Skip to content

Linux wheels: manylinux2014 tracking issue #309

Closed
Closed
Linux wheels: manylinux2014 tracking issue#309

Description

@skvark
skvark
opened on Apr 5, 2020

The manylinux1 images are starting show their age and therefore make the maintenance of the Linux wheels quite troublesome. OpenCV 3.4.10 and 4.3.0 cannot be built on manylinux1 images due to too old glibc:

/io/opencv/modules/core/src/parallel.cpp:943:56: error: ‘CPU_COUNT’ was not declared in this scope

         unsigned cpu_count_cpu_set = CPU_COUNT(&cpu_set);

In practice this means that the manylinux1 (CentOS 5) needs to be updated to manylinux2010 (CentOS 6) or manylinux2014 (CentOS 7). Considering that the CentOS 6 will be deprecated later this year it's better move straight to manylinux2014. The manylinux2014 tag support is not yet fully rolled out (pypa/manylinux#338). However, there are only some minor issues remaining and they will get solved likely during the next few months.

This upgrade will most likely have positive effects to the OpenCV performance (see: #287). Additionally, external dependencies, such as Qt, can be updated to more recent versions.

What needs to be done:

  • Prepare new custom Dockerfiles which extend from manylinux2014 similarly to the current manylinux1 Dockerfiles
    • Install Qt 5 (preferably via package manager, currently Qt 4.8.7 is built from sources)
    • Check if Cmake is present. If not, it could be probably set as a Python dependency similarly to numpy
    • Check if custom OpenSSL build is needed
    • Build latest FFmpeg from sources
    • Check if other often requested 3rd party dependencies can be enabled easily by installing them via package manager (must check license terms) Edit: this needs more work, needs to be done later when first working versions of the new wheels have been released
    • Clean up other stuff from the Dockerfiles
  • Check setup.py custom rules and patches
    • Enable IPP if possible
    • Remove other custom rules which are not needed
    • Check if OpenEXR patch is still needed

Activity

  1. johnthagen commented on May 21, 2020

    @johnthagen
    johnthagen
    on May 21, 2020
    Contributor

    The manylinux2014 tag support is not yet fully rolled out

    It looks like as of this comment, the rollout has been completed.

  2. skvark commented on May 21, 2020

    @skvark
    skvark
    on May 21, 2020
    MemberAuthor

    Thanks for the info. I'll update the Docker images and will run some tests.

  3. skvark commented on May 23, 2020

    @skvark
    skvark
    on May 23, 2020
    MemberAuthor

    The current status is that Linux builds have an issue with Qt5. I get this on Ubuntu 20.04 when trying to run very simple script with imshow:

    Found metadata in lib /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 329991
}


Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/usr/bin/platforms" ...
loaded library "/home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/libqxcb.so"
[New Thread 0x7fffbb3b0700 (LWP 5905)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffbe3abe05 in _xcb_in_expect_reply () from /home/username/test/lib/python3.8/site-packages/cv2/qt/plugins/platforms/../../../../opencv_python.libs/libxcb-65da195c.so.1.1.0

    The Qt xcb plugin crashes on _xcb_in_expect_reply() method.

  4. skvark commented on May 23, 2020

    @skvark
    skvark
    on May 23, 2020
    MemberAuthor

    If someone would like to test the manylinux2014 wheels they can be downloaded by following this guide: https://github.com/skvark/opencv-python/releases/tag/34

  5. ychechik commented on Jun 7, 2020

    @ychechik
    ychechik
    on Jun 7, 2020

    Hi, any updates? looking forward to 4.3 :)

  6. skvark commented on Jun 7, 2020

    @skvark
    skvark
    on Jun 7, 2020
    MemberAuthor

    I'll have to compile Qt5 probably manually to get rid of that error but haven't had the time to do it yet.

  7. Red-Eyed commented on Jun 20, 2020

    @Red-Eyed
    Red-Eyed
    on Jun 20, 2020

    Regarding manual build, have u look at C++ package managers?
    this one is actively contributed https://github.com/microsoft/vcpkg

  8. Red-Eyed commented on Jun 20, 2020

    @Red-Eyed
    Red-Eyed
    on Jun 20, 2020

    BTW, I built opencv4.3 on ubuntu 20.04 for python 3.7.6 (from conda) and it works fine.
    Is the issue only on CentOS ?

  9. skvark commented on Jun 20, 2020

    @skvark
    skvark
    on Jun 20, 2020
    MemberAuthor

    Yeah, locally built OpenCV will work of course just fine but that's not what this repository is trying to do. Manylinux wheels must be built on very specific systems to make them compatible with different Linux distributions (https://www.python.org/dev/peps/pep-0513/). The problem is not in package managers but in binary level compatibility across different distributions.

    The OpenCV builds related to this issue work just fine, only a subset of the methods related to displaying GUI windows is broken. As we can see above, Qt5 segfaults when it tries to use the bundled xcb plugin which has been included from the CentOS Qt5 build.

    Additionally, building for example Qt5 manually is not a problem, it just takes time when it's done for the first time. After that it can be used from the Docker images without any build overhead.

  10. johnthagen commented on Jun 20, 2020

    @johnthagen
    johnthagen
    on Jun 20, 2020
    Contributor

    @skvark In case any of these resources are helpful:

    Another interesting thing I found: PyQt5 builds manylinux2014 wheels already.

  11. Red-Eyed commented on Jun 21, 2020

    @Red-Eyed
    Red-Eyed
    on Jun 21, 2020

    I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.
    I will try to build Qt5 and opencv with this docker

  12. johnthagen commented on Jun 21, 2020

    @johnthagen
    johnthagen
    on Jun 21, 2020
    Contributor

    I guess we don't need pyqt5 but instead we need native Qt5 with cmake and header files.

    Right, I just assumed that if PyQt5 managed to build a manylinux2014 wheel, as part of that they must have figured out how to built Qt5 on a CentOS 7 image.

  13. Red-Eyed commented on Jun 21, 2020

    @Red-Eyed
    Red-Eyed
    on Jun 21, 2020

    Oh, I see

  14. skvark commented on Jun 21, 2020

    @skvark
    skvark
    on Jun 21, 2020
    MemberAuthor

    I'm pretty sure that building Qt5 on the manylinux image is quite straightforward process (same as Qt4 in current images, probably mostly copy pasting from there). After that the next thing is to get OpenCV built against it and testing the resulting wheels in other systems. Again, the main issue is that all those things take multiple evenings (or weekends) of my time. Anyone can of course do that as this is a open source project, it does not have to be me. The Dockerfiles are in the docker directory in case someone is interested in contributing. Copying the Qt4 example from master branch Dockerfiles to manylinux2014 branch Dockerfiles and modifying it for Qt5 is a good starting point. https://doc.qt.io/qt-5/build-sources.html#linux-x11

  15. johnthagen commented on Jun 21, 2020

    @johnthagen
    johnthagen
    on Jun 21, 2020
    Contributor

    @skvark Would you consider setting up something like GitHub Sponsors for people who would like to say thank you for your work?

  16. skvark commented on Jun 22, 2020

    @skvark
    skvark
    on Jun 22, 2020
    MemberAuthor

    @johnthagen I'll have a look at it. There are some complicated things related to Finnish taxation system which I need to check before even applying for example to Github Sponsors.

  17. skvark commented on Jul 1, 2020

    @skvark
    skvark
    on Jul 1, 2020
    MemberAuthor

    I couldn't get the latest Qt5 work with OpenCV (cv2 import hanged forever in 64 bit builds and segfaulted in 32-bit builds) so downgrading back to Qt4 which hopefully solves the issue for now. I will test the builds this week. macOS has also some build failures which need to be solved.

  18. skvark commented on Jul 2, 2020

    @skvark
    skvark
    on Jul 2, 2020
    MemberAuthor

    Linux builds are now ok. Fixing macOS issues now.

  19. skvark commented on Jul 5, 2020

    @skvark
    skvark
    on Jul 5, 2020
    MemberAuthor

    Homebrew apparently fixed itself so the release builds can be now made. 3.4.10 is now in progress, 4.3.0 will come after they are done.

  20. skvark
    closed this as completedon Jul 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions