Make datafusion-cli object store tracing mode work for local files #18119
Description
Is your feature request related to a problem or challenge?
@BlakeOrth and I have been working on an object store tracing mode which is great!
For exampele
DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559 |
+----------+
1 row(s) fetched.
Elapsed 0.836 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner: HttpStore
2025-10-17T09:24:25.641605+00:00 operation=Get duration=0.026670s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.668363+00:00 operation=Get duration=0.027705s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.767319+00:00 operation=Get duration=0.039514s size=15503 range: bytes=5120273-5135775 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.768711+00:00 operation=Get duration=0.085509s size=3895852 range: bytes=145483536-149379387 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.817364+00:00 operation=Get duration=0.039122s size=61815 range: bytes=46392516-46454330 path=hits_compatible/athena_partitioned/hits_1.parquet
However, when I try to use it on a local file it doesn't work
Describe the solution you'd like
I would like it also to trace requests when reading local files too
For example, create a local file:
COPY 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' to 'hits_1_index.parquet';And then run (note the lack of profiling):
DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'hits_1_index.parquet' where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559 |
+----------+
1 row(s) fetched.
Elapsed 0.225 seconds.
Object Store ProfilingDescribe alternatives you've considered
No response
Additional context
No response
Activity
BlakeOrth commentedon Oct 17, 2025
@alamb Have you tried this using a query on local files as well?
copyis not yet instrumented (again, coming very "soon") so I wouldn't expect the command in your example to work quite yet regardless of the underlying storage. I believe the instrumentation code, as written, should work with local storage as well, but it will definitely be good to validate that's actually the case (for better or for worse, my beliefs don't actually change reality).alamb commentedon Oct 17, 2025
Yes, I did and it didn't seem to work:
BlakeOrth commentedon Oct 17, 2025
@alamb I believe you may have been "bit" by the excellent performance improvements from @nuno-faria and the metadata cache here :) Disabling the cache shows some expected activity from the underlying store.
alamb commentedon Oct 17, 2025
🤦
Thank you @BlakeOrth