Add SchemaAdapterExec #2292

Activity

1. tustvold

   added

   enhancementNew feature or requestNew feature or request

   help wantedExtra attention is neededExtra attention is needed

   on Apr 20, 2022
2. alamb commented on Apr 20, 2022

   

   alamb

   on Apr 20, 2022

   Contributor

   More actions

   Note schema adapter is here: https://github.com/apache/arrow-datafusion/blob/9815ac6ecc2aee7fbbafa09c704ca81b0225221e/datafusion/core/src/physical_plan/file_format/mod.rs#L205

   IOx has most of the necessary code here for this logic here: https://github.com/influxdata/influxdb_iox/blob/5488c257d1bbb9a9b2f6882444b9e88098e53fdc/query/src/provider/adapter.rs#L45-L80

   👍React with 👍1Reacted by Raphael Taylor-Davies
3. thinkharderdev commented on Apr 21, 2022

   

   thinkharderdev

   on Apr 21, 2022

   Contributor

   More actions

   I have some concerns about this. The problem is that this sort of assumes that we actually know at planning time what the schema for each individual file is in a ListingScan. And if you infer the schemas at planning and merge then together to get the table schema then that is true. But since this happens during planning and can be quite expensive, I suspect that real world use cases will leverage some sort of metadata catalog to get the merged schema for a logical table instead of re-deriving it for each query. In that case we have no idea what the individual file schemas are.
4. tustvold commented on Apr 21, 2022

   

   tustvold

   on Apr 21, 2022

   ContributorAuthor

   More actions

   Thank you for bringing this up, to phrase it differently to check my understanding:

   • The ParquetExec, etc... needs to provide a schema at plan time
   • It needs to yield batches that match this schema
   • Depending on the catalog, the individual files might not match this schema, but must be compatible with it

   I agree that there doesn't appear to be a way around this without the file operator handling the schema adaption. I will close this and update the other tickets accordingly. Thank you 👍
5. tustvold

   closed this as completedon Apr 21, 2022
6. thinkharderdev commented on Apr 21, 2022

   

   thinkharderdev

   on Apr 21, 2022

   Contributor

   More actions

   • The ParquetExec, etc... needs to provide a schema at plan time
   • It needs to yield batches that match this schema
   • Depending on the catalog, the individual files might not match this schema, but must be compatible with it

   Yeah, exactly