Repository navigation
Manylinux public container removal #1564
Description
Fixed by https://devdiv.visualstudio.com/DevDiv/_git/debugpy/pullrequest/586830
In release.yml we reference a public container for manylinux builds.
image: quay.io/pypa/manylinux2014_x86_64:latest
This violates the recent company security policies. We have to figure out a way to deal with this.
Activity
[-]Public container removal[/-][+]Manylinux public container removal[/+]AdamYoblick commentedon Apr 11, 2024
Steve Dower (@zooba) Karthik Nadig (@karthiknadig) Pavel Minaev (@int19h) Do you have any ideas for how we can get around this?
karthiknadig commentedon Apr 11, 2024
Adam Yoblick (@AdamYoblick) We can create a internal registry using Azure Container Registry and upload a custom built image for this based on the pypa spec for the container. Here is where we can find the definition: https://github.com/pypa/manylinux
int19h commentedon Apr 11, 2024
The reason why we want to build in a container is because that's the only reliable way to ensure that Linux wheels don't end up with dependencies on the wrong glibc version. This container is the official "blessed" one from PyPA that runs an old enough version of RedHat to produce binaries that run on anything newer.
But as I understand it, the problem here isn't the use of a container per se, but rather the fact that we don't own it? In which case we can just fork the container spec, review it, and build it ourselves.
int19h commentedon Apr 11, 2024
Side note, it doesn't have to be
manylinux2014specifically - we can take something newer, it all depends on which glibc versions (= which distros) we want to support. There's nothing in either debugpy or pydevd that requires anything newer thanmanylinux1, so the only time I update our build scripts to point to a newer container is when Azure build agent & tasks stop working on the older one. But we can always rev it for other reasons, if the newer version is more convenient in some other way.AdamYoblick commentedon Apr 11, 2024
Thank you! I don't know anything about this, but it looks like that link does have the instructions on how to generate the docker images from the source. I'm not sure if that is sufficient to satisfy the requirement, but it might not be since the source is public and can be an attack vector. So maybe we have to generate it from scratch based on the spec, like you said :)
zooba commentedon Apr 11, 2024
Unfortunately, there are other requirements about using up to date OS, so we're not going to win there.
Best shot might be to get as close as we can with a regular build and run a validation step in the container to make sure we aren't relying on anything too new.
int19h commentedon Apr 11, 2024
Steve Dower (@zooba) The problem is that when you link to glibc, the symbols get versioned. So e.g. if you use printf, your binary's import table will have
printf@GLIBC.x.y.z, and will refuse to run on anything lower. I'm not sure if anything changed in this department recently, but last I looked into it, the official answer was still "build against the lowest version of glibc that you intend to support" (either in container, or in chroot which is not fundamentally different in this case).zig cccan link to specific glibc versions on any platform (even when cross-compiling from Windows), but they jump through a lot of hoops to make it all work. But if that is a feasible option, it could greatly simplify our build, I think.AdamYoblick commentedon Apr 11, 2024
Ah, you're saying that using a too-old OS in the image is itself a violation? Hmm, sounds like a catch-22 if you want to support manylinux. Maybe we need to find other teams at MS that produce manylinux wheels and see what they are doing.
zooba commentedon Apr 11, 2024
Corporate compliance requirements are inspired by catch-22 😆
If we can figure a way to link against early glibc without actually running an OS based around that, it'll be easier to get the exception we need to run tests in the actual container. Otherwise, we need to justify getting an exception to build production artifacts in an obsolete OS, which is a much bigger ask (and the first question will be whether we can do what I suggested earlier, so let's run that one out before we go straight to begging forgiveness).
AdamYoblick commentedon Oct 21, 2024
Ok so I'm looking at this work again, and since the warning from the bots complained about not using azure container storage, that's where I'm going to start.
Here are the steps I believe need to get done:
Push an Imagefor instructionsdocker pull quay.io/pypa/manylinux2014_x86_64So we'll see how it goes :)