classification
Title: Python 2.6 can't read sets pickled with Python 3.0
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, alexandre.vassalotti, hagen, pitrou, tjreedy
Priority: critical Keywords: patch

Created on 2008-08-25 12:15 by hagen, last changed 2008-10-04 23:14 by alexandre.vassalotti.

Files
File name Uploaded Description Edit Remove
pickle.diff ajaksu2, 2008-08-28 17:51 Adds a 3.0:2.6 modules mapping and uses that in find_class
Messages
msg71916 (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-25 12:15
After pickling a set of ints with Python 3.0 and pickle protocol 2:

[hagenf@chage ~]$ python3.0
Python 3.0b3 (r30b3:65927, Aug 21 2008, 11:48:29)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "wb")
>>> pickle.dump({1,2,3}, f, 2)
>>> f.close()

I get the following error when trying to read this with Python 2.6:

[hagenf@chage ~]$ python
Python 2.6b3 (r26b3:65922, Aug 21 2008, 11:42:25)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "rb")
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1090, in
load_global
    klass = self.find_class(module, name)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1124, in
find_class
    __import__(module)
ImportError: No module named builtins
msg72026 (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-27 13:03
Well, this is obviously caused by renaming "__builtin__" to "builtins"
and the fact that set (as well as frozenset) doesn't have its own opcode
and therefore gets looked up in "builtins". The problem therefore
extends to all builtin objects without opcode special casing (e.g.
object, slice, property, ...) I'm afraid that means we have to pickle
"builtins" as "__builtin__" for backwards compatibility in protocols <= 2.

But aside from that, wouldn't it be more consistent to have opcodes for
set/frozenset in protocol 3?
msg72096 (view) Author: Daniel Diniz (ajaksu2) Date: 2008-08-28 16:40
Hagen,
does this simple patch (against 2.6) solve it for you?

Index: Lib/pickle.py
===================================================================
--- Lib/pickle.py       (revision 66050)
+++ Lib/pickle.py       (working copy)
@@ -1121,6 +1121,8 @@

     def find_class(self, module, name):
         # Subclasses may override this
+        if module == "builtins":
+            module = "__builtin__"
         __import__(module)
         mod = sys.modules[module]
         klass = getattr(mod, name)

I think a dict mapping the moved modules would work better, perhaps
having it in PyPI would be enough?
msg72097 (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-28 16:52
Well, Python <= 2.5 still wouldn't be able to unpickle those built in
objects.
msg72100 (view) Author: Daniel Diniz (ajaksu2) Date: 2008-08-28 17:51
FWIW, there's a mapping of 2.6:3.0 modules in lib2to3:

from lib2to3.fixes.fix_imports import MAPPING

The attached patch uses that for a quick and dirty way of loading 3.0
pickles in 2.6.
msg72164 (view) Author: Terry J. Reedy (tjreedy) Date: 2008-08-29 21:54
Was it really intended that 3.0 pickles unpickle on 2.6?
What about other changes like moving something from one module to
another (reduce from built-in to functools), changing all classes to new
style (several examples), or adding methods to a built-in class (floats)?
msg72165 (view) Author: Antoine Pitrou (pitrou) Date: 2008-08-29 22:02
> Was it really intended that 3.0 pickles unpickle on 2.6?

He used protocol 2, so he explicitly asked for something inpickleable
with 2.6. If it's not the intended behaviour, then protocols < 3 should
be deprecated.
msg74328 (view) Author: Alexandre Vassalotti (alexandre.vassalotti) Date: 2008-10-04 23:14
I agree with Antoine, protocols <= 2 should remain compatible with
Python 2.x or be deprecated. Keeping compatibility will require a hack,
in addition to the proposed patch, in Pickler.save_global to map Python
3's module names to the ones of Python 2.
History
Date User Action Args
2008-10-04 23:14:44alexandre.vassalottisetmessages: + msg74328
2008-08-29 22:02:11pitrousetnosy: + pitrou
messages: + msg72165
2008-08-29 21:54:36tjreedysetnosy: + tjreedy
messages: + msg72164
2008-08-28 17:51:42ajaksu2setfiles: + pickle.diff
keywords: + patch
messages: + msg72100
2008-08-28 16:52:05hagensetmessages: + msg72097
2008-08-28 16:41:00ajaksu2setnosy: + ajaksu2
messages: + msg72096
2008-08-27 13:03:24hagensetmessages: + msg72026
2008-08-25 15:58:24pitrousetpriority: critical
nosy: + alexandre.vassalotti
type: behavior
2008-08-25 12:15:41hagencreate