Possible to Unpickle class instances after being converted from old to new style?

Go To StackoverFlow.com

6

In python, is it possible to unpickle objects that were pickled as old-style python classes after being converted to new-style ones? (Ie, objects with different "class signatures"). *

For example, imagine some instances were saved as:

class foo: # old style

and then a few more objects were pickled after the class was changed to inherit from object:

class foo(object): # new style

Objects that are pickle.dump'ed with one can be pickle.load'ed by the same style class, but neither one will load both. I think that the strategy used by pickle changes based on the inheritance (the class that inherits from object has a __reduce__ method automatically defined, but the one without inheritance doesn't). When trying to load a with-inheritance old-style pickle from code without inheritance (old-style definition), I get the same argument error as seen in this SO question; even though the second argument is redundant, it still changes the "class signature" and expected arguments, and prevents loading. To solve this, I'd be happy to write an unpickler, although I'm afraid it may involve two separate sub-classed unpicklers a la the docs... If I knew how to correctly unpickle each one I'd be fine doing that.

A little background on my situation... I'm using a TrialHandler class to save and reload button presses and reaction times for behavioral psychology experiments. We refactored the TrialHandler class to inherit from a more abstract, baseTrialHandler, but in doing so temporarily changed the class signature to inherit from object. However, we couldn't unpickle older trial_handler files, so it was changed back. I'd like to look at data from the same experiment that was run with both versions of the trial handler, and so want to unpickle both types of saved log file in the same script.

Alternatively, if I can't write a custom unpickler that will unpickle both objects, is there another way to serialize them? I tried dumping straight to yaml, but it looks like I'll have to register the class as something that can be yaml'ized, anyway.

A full description of the problem with specific errors is on the PsychoPy mailing list. Any blog posts explaining the details intermediate pickling, or even python inheritance, would be most welcome; the closest I found was a good explanation of why pickling is insecure describing pickling as a "simple stack-based virtual machine", which is nice, but doesn't get me enough to figure out even the basics of unpicking myself.

2012-04-05 21:54
by Erik Kastman


5

Going through parts:

  1. there is no thing called "class signature" in Python, although I got the idea from your question
  2. Not inheriting from "object" in a Python 2 program should be considered an error. Classes that do not inherit from object - also called "old style" classes where just kept on the language for backwards compatibility, when new style classes where introduced in Python 2.2 (circa year 2000/2001) - old style classes are a lot less consistent and lack many features that make today's python such a nice language.

With that out of the way: unpickle will try to deserialize objects based on the class qualified name - as in <module>.<class>, as seem on the object's __class__.__name__ attribute at pickling time. So it is possible to have a naive way that on an unpickle exception (it raises TypeError), swap the available class with the same name, and retry the unpickle operation Again: your classes should inherit from "object"(more specifically, have a "type" based metaclass)

As for the example I mentioned, juust write something along:

try:
    new_obj = pickle.load(data_stream)
except TypeError: # if you know this was caused due to the new version of "Foo" class:
    current_foo = foomodule.Foo
    foomodule.Foo = foomodule.OldFoo
    new_obj = pickle.load(data_stream)
    foomodule.Foo = current_foo
2012-04-06 03:49
by jsbueno
Good suggestions - I didn't realize this was the key distinction between declaring new and old style classes. Maybe I should change the title of the question to something like "How to unpickle an object after converting it from old to new-style?" I know that python doesn't' strictly have a "signature"/protocol, is there are better way to say it? At least I'll go back and edit to put signature in quotes - Erik Kastman 2012-04-06 13:59
I'd like to have posted a full fledged example, running it from the interactive interpreter - however, the simplified old style class I did wrote had no problem being unpickled as new style. But yes, I do usggest declare both, with "old" and "new" sufixes in their names ("Fooold" and "Foonew"), setting their __name__ attribute to "Foo", and dynamically set the module "Foo" variable to one or another at unpickle time - jsbueno 2012-04-06 14:12
I was thinking about rescuing based on name error, but I don't have both Foo and OldFoo available at the same time, since I converted one to the other. What's the best way to declare them side by side? Maybe something like?:

class Foo(object): # New Style
    def __init__(self, ...args): # actual code
class OldFoo: # Old Style
    def __init__(self, ...args): Foo(*args)

Where the OldFoo just calls the new one? By definition the old-style class can't inherit from the new style... Is there a better way where I don't need to leave a stubbed out old-style class - Erik Kastman 2012-04-06 14:14

Also, this question of old style vs new style can be related: http://stackoverflow.com/questions/10043963/class-classname-versus-class-classnameobject - jsbueno 2012-04-06 14:15
I think you'd better have a function (or other construct) to convert the objects from the old type to the new one. You do declare both classes - the old one with a different name - and use the name swapping, as above to unpickle the object. Then you restore teh name, and call the converter function to instantiate a new object, with the current class definition, and set its attributes according to the ones in the passed object - jsbueno 2012-04-06 14:18
That makes sense - I'll give it a go with a simple version and gist it if there are any problems. Thanks - Erik Kastman 2012-04-06 14:32
Although this is a kluge, I built some proof-of-concept code that will work. It may not be the way we leave it, but it solves this issue - Erik Kastman 2012-04-10 15:09
Ads