Repository navigation
EKS connectivity during tag to digest resolution #15877
Description
Hi, thanks for pushing this fix. I had this issue, and when I tried to upgrade to 1.18.0, I now see the following instead, which also appears to be a tag resolution issue:
Message: Revision "donny-helloworld-00001" failed with message: Unable to fetch image "111111111111.ecr.us-east-1.amazonaws.com/testing:dg": failed to resolve image to digest: Get "https://111111111111.ecr.us-east-1.amazonaws.com/v2/": context deadline exceeded.
Originally posted by @dongreenberg in #15778
Reacted by Tisham Dhar
Activity
dprotaso commentedon May 9, 2025
@dongreenberg Following up on your comment
Given
Are other pods in your cluster able to connect to your registry?
dongreenberg commentedon May 9, 2025
Indeed. If I create the pod manually with kubectl run it works fine, and if I create the ksvc with the exact sha256 it works as well. Only when I use a tag in the ksvc. When I try disabling resolution in the config-deployment configmap it still strangely doesn't work, with the same error.
dprotaso commentedon May 9, 2025
I don't have an AWS account to test with - we debugged scenarios like this before eg. AKS & GitLab
I dunno if you're able to modify this to help with debugging - #12761 (comment)
Alternatively, if you have a cluster I can poke at let me know in slack (CNCF Slack - #knative-serving)
My guess is it's not skipping the tag to digest resolution - how are you configuring it and what are you putting in the ksvc
dongreenberg commentedon May 19, 2025
For some reason creating an ImagePullSecret and attaching it to the service account used in my ksvc fixes the issue. However, the node IAM role attached to every node already has full ECR access, as does the IAM role attached the service account. Only adding the ImagePullSecret does the trick. Maybe this layering of permissions is creating issues, but thinking in terms of where the actual request is made to do the tag resolution, do you have a sense of why this would be? Which k8s service account is used to actually make the resolution request/s, and does it use the pull secrets to do it?
dprotaso commentedon May 19, 2025
The Knative Controller that runs as a pod is doing the tag to digest resolution. So the pod would need access to that metadata server running on the node.
One alternative that I've tested in the past was associating a fine grained policy and associating with the service account
#9477 (comment)
In the above there's a sample app you can run to help with debugging (I don't have access to an env). I'm guessing the metadata service isn't accessible by your Pod?
dprotaso commentedon May 20, 2025
@dongreenberg how are you setting up the EKS cluster, registry etc
treyhyde commentedon Jul 23, 2025
It hasn't been explicitly said in this PR how to disable the resolution.
Adding to your config-deployment is your workaround. (and works for me)
github-actions commentedon Oct 22, 2025
This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with
/reopen. Mark the issue asfresh by adding the comment
/remove-lifecycle stale.dprotaso commentedon Oct 22, 2025
/lifecycle frozen
Sadly still waiting on an AWS account from CNCF :/
A contributor also included some docs about using Pod Identity here: https://github.com/knative/docs/pull/6369/files
dprotaso commentedon Oct 22, 2025
@dongreenberg you mind trying out the Pod Identity approach as a work around?