Protobuf 2.6.0 incompatibility issues with Python 3.x

I've run into a problem when using Python 3.4 and protobuf 2.6.0.
I get several errors including:
protobuf/python/google/protobuf/internal/decoder.py, line 167
_DecodeVarint = _VarintDecoder((1 << 64) - 1, long)
NameError: name 'long' is not defined

Blanket Python 3 support was unintentionally listed in these release notes. We need to fix that up for a 2.6.x release. The code itself is ready with some 2to3 use, the setup.py and deps are not done yet. Sorry about that!

Here's my summary:

Support for Python 3 is almost in place. Our setup.py does not yet do the right thing for 2to3 conversions with Python 3 but for the adventurous, manually run the following files through 2to3 and it should work in Python 3.3+:

protobuf/descriptor.py
protobuf/internal/cpp_message.py
protobuf/internal/decoder.py
protobuf/internal/python_message.py
protobuf/internal/type_checkers.py
protobuf/internal/message_cpp2_test.py
protobuf/internal/message_factory_test.py
protobuf/internal/message_factory_cpp2_test.py
protobuf/internal/message_factory_python_test.py
protobuf/internal/message_python_test.py
protobuf/internal/message_test.py
protobuf/internal/reflection_cpp2_generated_test.py
protobuf/internal/reflection_python_test.py
protobuf/internal/reflection_test.py
protobuf/internal/test_util.py
protobuf/internal/text_format_test.py
protobuf/message_factory.py
protobuf/text_encoding.py
protobuf/text_format.py

setup.py also specifies an external dep on google-apputils needed in order to run the unittests. We need to push out an updated version of that with Python 3 support.

Any specific reason for wanting to use 2to3 instead of something like https://github.com/python-modernize/python-modernize to do the transition once?

It says
"Modernizes Python code for eventual Python 3 migration. Build on top of
2to3"

and it's not apparent why it's better ;)

Also 2to3 support is in setuptools already.

On Thu, Sep 4, 2014 at 6:59 AM, Brett Cannon notifications@github.com
wrote:

Any specific reason for wanting to use 2to3 instead of something like
https://github.com/python-modernize/python-modernize to do the transition
once?

—
Reply to this email directly or view it on GitHub
#7 (comment).

Thanks,
--Mike

By unifying the code base to work on both Python 2 and 3 it makes development easier and installation faster (unless you release separate Python 2 and 3 packages). By not having to wait for 2to3 you eliminate a compile step during development. As for installation, you can forgo that step entirely. Plus the coding practices used in converted source is more in line with modern best practices and syntax.

I wonder if we can reuse code from my https://github.com/GreatFruitOmsk/protobuf-py3

@Kentzo

I'm +1 on @Kentzo's approach, and would volunteer to help review #66. As @brettcannon says, avoiding 2to3 is pretty much the "widely accepted" best practice for libraries which need to support both Python2 and Python3: e.g., see Lennart Regebro's excellent outline of strategies.

I would love to see a python 3 compatible official protobuf available by Christmas.

FWIW, using the current trunk of protobuf, the only change I had to make to the package after installation was change a number of str("", "utf-8") references in descriptor_pb2.py. Oddly enough, the protobuf I have installed from homebrew (2.6.1) generates _pb2.py files with a function that looks like this:

_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))

And it uses that to encode strings instead like this:

default_value=_b("").decode('utf-8')

So.. not sure why the descriptor_pb2.py file is out of date. If someone could push those changes into that file, then I believe trunk would at least work on python3 -- but obviously #66 would be even better.

That function should not be encoding literals as latin under Py3k, especially since they are going to be immediately decoded as utf-8. Use of latin1 in anything but edgiest cases (decoding wonky / unknown data, or encoding for antique wire protocols) is a sign of papered-over trouble: "mojibake ahead", it could read.

@tseaver

So, I stand corrected. When I install protobuf properly, it installs/runs from trunk without any noticable problems now. Python 3.4.2 on OSX. Now someone just needs to push an updated package to pypi... :)

It would be nice to get better python3 support in trunk to support the edge cases properly, as @tseaver mentions.

@gpshead

@gpshead would you be fine with only supporting python 2.6+? Current readme.txt mentions 2.4.

Single-source cross python support is, FTR, like 10x easier if you only support python 2.6+ and 3.3+.

Absolutely, 2.6 is a good minimum. Anything older is obsolete (I consider 2.6 obsolete as well, but it is easy enough to support).

One of the reasons we started with some files needing to be run through 2to3 is that we have a few internal things still running on the old App Engine 2.5 legacy runtime that needed proto2 code and using 2to3 made maintaining 2.5 compatibility easier. I believe we have dealt with those internally now but I need to double check. We should absolutely be able to make a single source 2.6, 2.7, >= 3.3+ compatible version in the future.

FWIW, #164 modernizes the python code to drop all the PY25 stuff.

@thanatos

@thanatos https://pypi.python.org/pypi/protobuf pip install protobuf==3.0.0b3 is python3-compatible.

While technically true, my understanding is that the major number here is bumped because it isn't backwards-compatible with compiled protocol buffers from protobuf v2; empirically, this seems to be the case; e.g., <a Message derived class>.INTEGER doesn't exist in protobuf==3.0.0b3.

(~~Did it change to VALUE_INTEGER_FIELD_NUMBER?~~ (Nope.) Perhaps if this is the only change that heka_py requries, I might be able to make a small patch to heka_py to make it protobuf==3.0.0b3 compatible? But the problem seems to still stand — if I pull in a library that relies on protobuf, that restricts me to legacy Python.)

@thanatos

On Fri, Jul 15, 2016 at 2:41 PM, Roy Wellington Ⅳ notifications@github.com
wrote:

@thanatos https://github.com/thanatos
https://pypi.python.org/pypi/protobuf pip install protobuf==3.0.0b3 is
python3-compatible.

While technically true, my understanding is that the major number here is
bumped because it isn't backwards-compatible with compiled protocol buffers
from protobuf v2; empirically, this seems to be the case; e.g., .INTEGER doesn't exist in protobuf==3.0.0b3.

We bumped the major version number because of the new proto3 syntax
support, not because it's incompatible. If you find incompatible issues
with old generated code, please help file a bug for it.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#7 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIJ6iiG4RJcNqEfQ0dyXn0benZXh40lIks5qV_6bgaJpZM4Cdsma
.

i have compiled caffe on Ubuntu 16.04 LTS both with python 2 and python, it is really a hectic process for beginners, for caffe with python 2.7 make sure that your /.local/lib/python2.7/site-packages only contains a version same as you protoc version. protoc version can be checked by entering
protoc --version
for me i got problems with python 2.7 because by mistake i installed protbuf 3.0 alpha in python 2.7 site packages, so there was a protobuf error after removing it from python 2.7 site packages everything worked fine
also install python-dev version
apt-get install the python3.x-dev
and for python 3.5 make sure you have python 3.0 alpha, if you have both python 2.7 and python 3.x in the system use pip3 otherwise use pip

pip3 install protobuf==3.0.0-alpha-3

or another protobuf such as 3.5 version of protobuf.
also do this
pip3 install python-dateutil --upgrade
after that do
make clean
make all
make runtest
make pycaffe
after performing make all make runtest steps don't forget to do that
sudo ldconfig

hope everything works OK for you.

Merge pull request protocolbuffers#7 from cfallin/master

Provide UT that reproduces issue protocolbuffers#7

Merge pull request protocolbuffers#9 from PierrickVoulet/testIssue7

Silence MSAN warning when running protoc

UX: Enhanced diagnostics, Type::Tiny integration, and Memory Hardening

Navigation Menu

Repository navigation

Protobuf 2.6.0 incompatibility issues with Python 3.x #7

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Protobuf 2.6.0 incompatibility issues with Python 3.x #7

Description

Activity

gpshead commented on Sep 3, 2014

brettcannon commented on Sep 4, 2014

mrovner commented on Sep 6, 2014

brettcannon commented on Sep 7, 2014

Kentzo commented on Sep 26, 2014

tseaver commented on Nov 18, 2014

virtuald commented on Nov 21, 2014

virtuald commented on Nov 30, 2014

tseaver commented on Nov 30, 2014

virtuald commented on Nov 30, 2014

Ivoz commented on Dec 2, 2014

gpshead commented on Dec 14, 2014

tseaver commented on Jan 30, 2015

3 remaining items

thanatos commented on Jul 15, 2016

xfxyjwf commented on Jul 15, 2016

ZubairKhan001 commented on Feb 9, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions