Comet 1.1.0: FileSourceScanExec.copy NoSuchMethodError on EMR Spark 4.1.1 #7393

Describe the bug

Hey! We're running Comet 1.1.0 on EMR Serverless and hitting a NoSuchMethodError when caching a query with Parquet inputs.

The failure is in CometNativeScanExec.doCanonicalize(). It calls FileSourceScanExec.copy() with 10 arguments, but EMR's Spark build has a 12-argument version of that method.

Environment:

• EMR Serverless: emr-spark-8.1.0
• Spark: 4.1.1-amzn-0
• Comet: org.apache.datafusion:comet-spark-spark4.1_2.13:1.1.0
• Java 17, ARM64
• Comet enabled, native shuffle and native Iceberg writes disabled

The error:

java.lang.NoSuchMethodError: 'org.apache.spark.sql.execution.FileSourceScanExec org.apache.spark.sql.execution.FileSourceScanExec.copy(org.apache.spark.sql.execution.datasources.HadoopFsRelation, scala.Option, scala.collection.immutable.Seq, org.apache.spark.sql.types.StructType, scala.collection.immutable.Seq, scala.Option, scala.Option, scala.collection.immutable.Seq, scala.Option, boolean)'
    at org.apache.spark.sql.comet.CometNativeScanExec.doCanonicalize(CometNativeScanExec.scala:298)

Steps to reproduce

Our job joins Parquet inputs with Iceberg data, then calls:

changes.persist(StorageLevel.MEMORY_AND_DISK)

Spark fails while preparing the cached query. I haven't reduced the full query to a standalone PySpark reproducer yet, but the method mismatch can be checked directly against the public EMR image:

docker run --rm --entrypoint /bin/bash \
  public.ecr.aws/emr-serverless/spark4/emr-spark-8.1.0:20260827 \
  -c 'javap -classpath "/usr/lib/spark/jars/*" org.apache.spark.sql.execution.FileSourceScanExec | grep " copy("'

Compared with upstream Spark 4.1.1, EMR adds two fields after disableBucketedScan:

• enablePushdownAggregate: Boolean
• partitionOutput: Seq[Attribute]

The EMR JAR has the 12-argument copy() and no 10-argument overload. Comet's call here therefore can't resolve at runtime.

Expected behavior

Caching the query succeeds, or Comet falls back to Spark for this scan when the runtime is incompatible.

Additional context

I realize EMR changes Spark internals and isn't part of Comet's tested Spark matrix. I found #190 and the makeCopy() change in #207, which addressed a similar issue on Databricks. This looks like the same assumption in a different call site.

Would using makeCopy() with the runtime's existing field values be a reasonable fix here? We haven't tested that patch yet.