sagemaker-core 2.15.0: role validation raises false-positive RoleValidationError under condition-based SCPs (IAM simulator can't evaluate conditional SCPs) #6019

Activity

1. hiSandog commented on Jul 14, 2026

   

   hiSandog

   on Jul 14, 2026

   More actions

   Could the exact RoleValidationError path from “sagemaker-core 2.15.0: role validation raises false-positive 'RoleValidationError' under condition-based SCPs (IAM…” be retained in the test suite when this is fixed? The report already isolates a useful boundary, and asserting its expected result would help prevent recurrence.
2. thejesterscap commented on Aug 5, 2026

   

   thejesterscap

   on Aug 5, 2026

   More actions

   A second trigger for the same RoleValidationError, with a different root cause

   This hits the same pre-flight and exception as the report above, but the evidence here doesn't point to an Organizations/SCP-conditional denial — nothing about IAM policy changed between a successful run and a failing one, only the caller's ability to call iam:SimulatePrincipalPolicy did. That isolates the cause to the fixed smoke-test action list itself:

   RoleValidationError: IAM role 'arn:aws:iam::…:role/…' cannot be used for 'training' workloads.
   Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
   ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
   ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions,
   ec2:DescribeNetworkInterfaces, ec2:DescribeSecurityGroups, ec2:DescribeSubnets,
   ec2:DescribeVpcs
   

   Every listed permission is for VPC-attached training. Our jobs set no VpcConfig, so none is needed, and the role deliberately does not carry them.

   The part that makes it hard to diagnose

   The check has three outcomes, and which one you get depends on the caller's permissions, not on the role being validated:

   caller can iam:SimulatePrincipalPolicy?	result
   no	verdict None → warn and proceed	fail-open
   yes, all allowed	proceed
   yes, any denied	RoleValidationError	fail-closed

   So the SDK logs Could not verify permissions for role … Proceeding with it. and works. Granting the permission that silences that warning is what breaks it. We granted iam:SimulatePrincipalPolicy specifically to make the pre-flight informative, and it took down every dispatch for 76 minutes.

   Cause

   sagemaker/core/helper/iam_role_resolver.py. _get_smoke_test_actions() collects actions from a bundled training policy template — the AmazonSageMakerFullAccess shape — and _evaluate_permissions() simulates those, not the permissions the submitted job actually requires. A least-privilege non-VPC role therefore cannot return "allowed".

   The function's own docstring states the intent this violates:

   Gating only on *-resource actions keeps validation a meaningful "is this a usable SageMaker execution role?" smoke test without false denials.

   Restricting to *-resource actions avoids one class of false denial (resource-scoped actions simulated without ResourceArns). It does not avoid this one: the VPC/CloudWatch actions are *-resource in the template, and are simply not required by every training job.

   Evidence it is a false denial

   Three facts, same role throughout:

   • a pipeline execution submitted before the grant trained to completion on that role;
   • PutRolePolicy adding iam:SimulatePrincipalPolicy landed at a known timestamp (CloudTrail);
   • the next submission, with a docstring-only diff to our pipeline code, died on RoleValidationError.

   The only variable was the caller's ability to simulate.

   On the pending fix (#6029)

   Read the diff before posting this: _is_unverifiable_denial treats an implicitDeny as unverifiable only when OrganizationsDecisionDetail.AllowedByOrganizations or PermissionsBoundaryDecisionDetail.AllowedByPermissionsBoundary comes back explicitly False. Our denial has neither in play — no SCP or permissions boundary is withholding these permissions, the role's own identity policy simply never granted them, by design. #6029 as written would still classify this as denied and still raise. It does add a validate_role=False / SAGEMAKER_VALIDATE_ROLE opt-out, which would give a supported bypass once merged — today the only remedies are widening the role beyond least privilege, revoking iam:SimulatePrincipalPolicy from the caller, or monkey-patching.

   Expected behaviour

   Do not fail closed on permissions the submitted job does not use. Options, roughly in order of preference: derive the required set from the job's own configuration (VPC actions only when VpcConfig is set); treat a denial as a warning and let the service be the authority; or use #6029's opt-out once it merges.

   Versions

   package	version
   sagemaker	3.18.0
   sagemaker-core	2.18.0
   sagemaker-train	1.18.0
   sagemaker-mlops	1.18.0
   sagemaker-serve	1.18.0
   boto3 / botocore	1.43.53
   pydantic	2.13.4
   Python	3.12.3

   Note: sagemaker.__version__ no longer exists on v3 — the 3.x wheel is a namespace shim — so version-report snippets that read it raise AttributeError. The table above is per-package.
3. sagemaker-bot

   added

   component: pysdk-teamRelated to SageMaker Python SDK Core IssuesRelated to SageMaker Python SDK Core Issues

   type: bug

   on Aug 24, 2026
4. sagemaker-bot

   added a commit that references this issue on Sep 30, 2026

   fix(aws#6019): sagemaker-core 2.15.0: role validation raises false-po…

   ...

   354da1b
5. mohamedzeidan2021

   added a commit that references this issue on Oct 9, 2026

   fix(core): prevent false-positive RoleValidationError under condition… (

   ...

   dcbcdf7
6. mohamedzeidan2021

   closed this as completedin #6029on Oct 9, 2026