Enable the Comet in-memory cache (InMemoryTableScan) by default #7425

What is the problem the feature request solves?

Cached DataFrames are common in batch ETL. Comet's Arrow cache (spark.comet.exec.inMemoryCache.enabled) is experimental and off by default. With it off, a cached relation is read through Spark's InMemoryTableScanExec, and the operators above it can't run natively.

Describe the potential solution

Close the remaining correctness and stability items, then turn the cache on by default:

• the in-memory cache audit epic ([EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support #6834)
• the Spark 3.4 executor abort with the cache enabled (Executor JVM abort (NMT footer canary broken) when the in-memory cache is enabled on Spark 3.4 #7305)
• memory accounting and spill behavior for large cached relations
• a performance check against Spark's cache on wide and narrow schemas

Additional context

Because the serializer has to be installed at startup, the default has to be decided before the session starts, not per query.