In python, is it possible to unpickle objects that were pickled as old-style python classes after being converted to new-style ones? (Ie, objects with different "class signatures"). *
For example, imagine some instances were saved as:
class foo: # old style
and then a few more objects were pickled after the class was changed to inherit from object:
class foo(object): # new style
Objects that are pickle.dump
'ed with one can be pickle.load
'ed by the same style class, but neither one will load both. I think that the strategy used by pickle changes based on the inheritance (the class that inherits from object has a __reduce__ method automatically defined, but the one without inheritance doesn't). When trying to load a with-inheritance old-style pickle from code without inheritance (old-style definition), I get the same argument error as seen in this SO question; even though the second argument is redundant, it still changes the "class signature" and expected arguments, and prevents loading. To solve this, I'd be happy to write an unpickler, although I'm afraid it may involve two separate sub-classed unpicklers a la the docs... If I knew how to correctly unpickle each one I'd be fine doing that.
A little background on my situation... I'm using a TrialHandler class to save and reload button presses and reaction times for behavioral psychology experiments. We refactored the TrialHandler class to inherit from a more abstract, baseTrialHandler, but in doing so temporarily changed the class signature to inherit from object. However, we couldn't unpickle older trial_handler files, so it was changed back. I'd like to look at data from the same experiment that was run with both versions of the trial handler, and so want to unpickle both types of saved log file in the same script.
Alternatively, if I can't write a custom unpickler that will unpickle both objects, is there another way to serialize them? I tried dumping straight to yaml, but it looks like I'll have to register the class as something that can be yaml'ized, anyway.
A full description of the problem with specific errors is on the PsychoPy mailing list. Any blog posts explaining the details intermediate pickling, or even python inheritance, would be most welcome; the closest I found was a good explanation of why pickling is insecure describing pickling as a "simple stack-based virtual machine", which is nice, but doesn't get me enough to figure out even the basics of unpicking myself.
Going through parts:
With that out of the way: unpickle will try to deserialize objects based on the class qualified name - as in <module>.<class>
, as seem on the object's __class__.__name__
attribute at pickling time. So it is possible to have a naive way that on an unpickle exception (it raises TypeError), swap the available class with the same name, and retry the unpickle operation
Again: your classes should inherit from "object"(more specifically, have a "type" based metaclass)
As for the example I mentioned, juust write something along:
try:
new_obj = pickle.load(data_stream)
except TypeError: # if you know this was caused due to the new version of "Foo" class:
current_foo = foomodule.Foo
foomodule.Foo = foomodule.OldFoo
new_obj = pickle.load(data_stream)
foomodule.Foo = current_foo
__name__
attribute to "Foo", and dynamically set the module "Foo" variable to one or another at unpickle time - jsbueno 2012-04-06 14:12
class Foo(object): # New Style
def __init__(self, ...args): # actual code
class OldFoo: # Old Style
def __init__(self, ...args): Foo(*args)
Where the OldFoo just calls the new one? By definition the old-style class can't inherit from the new style... Is there a better way where I don't need to leave a stubbed out old-style class - Erik Kastman 2012-04-06 14:14