[Epic] Extract catalog functionality from the core to make it more modular

@goldmedal

Is your feature request related to a problem or challenge?

As @goldmedal started trying to move the DynamicFileProvider so others could use it in #10745 I think it is clear that there is not a good way to add additional catalog support in the core without everything being intertwined.

Thus I think we should try and extract the different catalog providers out of datafusion core so it it easier

Describe the solution you'd like

I suggest the following final layout:

traits like CatalogProvider, SchemaProvider, etc in a new crate datafusion-catalog (since these traits rely on table provider, etc I think this can't be in datafusion-common or datafusion-expr)
The built in Memory* providers are in datafusion-catalog
The bult in InformationSchema providers are in datafusion-catalog
The newly proposed DynamicFileCatalog in datafusion-catalog
(eventually) the LIstingTableProvider (which is by far the most complicated) moved to its own crate datafusion-catalog-listing

Describe alternatives you've considered

No response

Additional context

No response

@andygrove

If this seems like a reasonable idea to people I will file tickets to break down the work

cc @andygrove @jayzhan211 @comphead @mustafasrepo for your thoughts

I agree with this direction. But now this seems hard to achieve because SchemaProvider depends on TableProvider and TableProvider depends on SessionState.

@alamb

Thanks @alamb for starting this discussion.
If I understood correctly there is a use case for using catalog abstractions/implementation but without datafusion core?

Like @lewiszlw correctly mentioned, we got some coupling between providers and the core. I'm just trying to understand the usecase when providers needed without the core

Is it due to the complexity of ListingTable so it has it's own crate? If they have common things then it is better to organize them into one crate. If ListingTable is so different than others, it is nice to have an independent crate

If I understood correctly there is a use case for using catalog abstractions/implementation but without datafusion core?

In my mind the real use usecase is to more easily use datafusion without having to bring in all the dependencies of LIstingTable (like parquet, avro, json, etc)

So the real usecase is getting ListingTable out of the core. But since the catalog API is in the core now there is no way to get ListingTable out of the core without also first moving the catalog API

Is it due to the complexity of ListingTable so it has it's own crate? If they have common things then it is better to organize them into one crate. If ListingTable is so different than others, it is nice to have an independent crate

I think both the complexity of ListingTable but also because if its dependency tree (e.g. parquet-rs and avro and json and object_store and ...)

For use cases like WASM it is quite messy to have the API split up like it currently is

@lewiszlw

I agree with this direction. But now this seems hard to achieve because SchemaProvider depends on TableProvider and TableProvider depends on SessionState.

I agree @lewiszlw -- well put. I made a first PR to start detangling things here: #10794 (it just splits SessionState into its own module)

Longer term we would have to figure out where SessionState would live (it still depends on several things in the core crate like datasource::provider and datasource::function 🤔

Maybe we could look into splitting out datafusion-datasource / datafusion-datasource-parquet / datafusion-datsource-avro, etc -- I don't have time to drive this at the moment but would be interested in helping anyone who did

If I understood correctly there is a use case for using catalog abstractions/implementation but without datafusion core?

In my mind the real use usecase is to more easily use datafusion without having to bring in all the dependencies of LIstingTable (like parquet, avro, json, etc)

So the real usecase is getting ListingTable out of the core. But since the catalog API is in the core now there is no way to get ListingTable out of the core without also first moving the catalog API

Hm... they probably thrive to have their own readers/writes perhaps other than arrow-rs implementation, that makes sense for me. And yes, if DF stands for extensibility we should make this happen. Not sure how difficult that can be though. We probably need to start with replacing core abstractions with traits instead of implementations to decouple it.

Hm... they probably thrive to have their own readers/writes perhaps other than arrow-rs implementation, that makes sense for me. And yes, if DF stands for extensibility we should make this happen. Not sure how difficult that can be though. We probably need to start with replacing core abstractions with traits instead of implementations to decouple it.

Yes something like this -- I think most of the traits already exist (e.g. CatalogProvider) but figuring out how to decouple SessionState (which is referred to by CatalogProvider is the trickiest bit I think)

I'm interesting on how datafusion can support vector index like Duckdb's VSS extension

Given the comment, it seems the issue here might be helpful 🤔 ?
I can work on this 👀

There is a dead lock that block us from extracting out catalog 😆

To move CatalogProvider out we need to move TableProvider to datafusion-datasource
To move TableProvider out we need to avoid dependency on SessionState for scan function, but there is CatalogProviderList in SessionState

SessionState contains tons of concept that are not all necessary for scan

pub struct SessionState {
    /// A unique UUID that identifies the session
    session_id: String,
    /// Responsible for analyzing and rewrite a logical plan before optimization
    analyzer: Analyzer,
    /// Provides support for customising the SQL planner, e.g. to add support for custom operators like `->>` or `?`
    expr_planners: Vec<Arc<dyn ExprPlanner>>,
    /// Responsible for optimizing a logical plan
    optimizer: Optimizer,
    /// Responsible for optimizing a physical execution plan
    physical_optimizers: PhysicalOptimizer,
    /// Responsible for planning `LogicalPlan`s, and `ExecutionPlan`
    query_planner: Arc<dyn QueryPlanner + Send + Sync>,
    /// Collection of catalogs containing schemas and ultimately TableProviders
    catalog_list: Arc<dyn CatalogProviderList>,
    /// Table Functions
    table_functions: HashMap<String, Arc<TableFunction>>,
    /// Scalar functions that are registered with the context
    scalar_functions: HashMap<String, Arc<ScalarUDF>>,
    /// Aggregate functions registered in the context
    aggregate_functions: HashMap<String, Arc<AggregateUDF>>,
    /// Window functions registered in the context
    window_functions: HashMap<String, Arc<WindowUDF>>,
    /// Deserializer registry for extensions.
    serializer_registry: Arc<dyn SerializerRegistry>,
    /// Holds registered external FileFormat implementations
    file_formats: HashMap<String, Arc<dyn FileFormatFactory>>,
    /// Session configuration
    config: SessionConfig,
    /// Table options
    table_options: TableOptions,
    /// Execution properties
    execution_props: ExecutionProps,
    /// TableProviderFactories for different file formats.
    ///
    /// Maps strings like "JSON" to an instance of  [`TableProviderFactory`]
    ///
    /// This is used to create [`TableProvider`] instances for the
    /// `CREATE EXTERNAL TABLE ... STORED AS <FORMAT>` for custom file
    /// formats other than those built into DataFusion
    ///
    /// [`TableProvider`]: crate::datasource::provider::TableProvider
    table_factories: HashMap<String, Arc<dyn TableProviderFactory>>,
    /// Runtime environment
    runtime_env: Arc<RuntimeEnv>,

    /// [FunctionFactory] to support pluggable user defined function handler.
    ///
    /// It will be invoked on `CREATE FUNCTION` statements.
    /// thus, changing dialect o PostgreSql is required
    function_factory: Option<Arc<dyn FunctionFactory>>,
}

@goldmedal

To move TableProvider out we need to avoid dependency on SessionState for scan function, but there is CatalogProviderList in SessionState

🤔 we could potentially move SessionState into the catalog crate

In fact the more I think about it, it seems to make some sort of sense to have a crate with the catalog traits / interfaces such as SessionState, Catalog, TableProvider that can be shared broadly.

Then we could make separate crates that actually had the implementations like

datafusion-catalog: contains traits
datafusion-catalog-basic (not a good name: MemoryCatalog, etc, InformationCatalog(potentiallyDynamicFileCatalog` / Introduce the DynamicFileCatalog in datafusion-catalog #11035 from @goldmedal )
datafusion-catalog-listing: ListingTable

And we could put the individual datasources like

datafusion-datasource-parquet
datafusion-datasource-csv
datafusion-datasource-json
...

🤔

I think #11320 and #11403 from @Omega359 makes this easier

datafusion-catalog: contains traits
datafusion-catalog-basic

~~datafusion-catalog~~ datafusion-catalog-common: contains traits
~~datafusion-catalog-basic~~ datafusion-catalog: actual implementation
🤔

Let's try moving SessionState, Catalog, TableProvider together into datafusion-catalog-common or common-runtime

Update: #11403 has been merged and I think building SessionState is much nicer now

If we're planning to replace SessionState, I'd like the minimum dependencies. This helps avoid potential circular dependency issues that might arise from other components, even if they're not necessary.

5. (eventually) the LIstingTableProvider (which is by far the most complicated) moved to its own crate datafusion-catalog-listing

This will require moving file formats (datafusion/core/src/datasource/file_format) too.
I think this might be a good fit for a yet another top level module. What do you think?

And we could put the individual datasources like
datafusion-datasource-parquet
datafusion-datasource-csv
datafusion-datasource-json
...

We could try to move them out to a crate datasource first and maybe individual data source crate if required

We could try to move them out to a crate datasource first and maybe individual data source crate if required

I think this would be ideal

The parquet one in particular is quite large. The others are not as large

@findepi

@findepi Is there any blocker of moving catalog / datasource out of core?

@jayzhan211

@jayzhan211 not aware of any blockers, but not working on this right now, if this is what you're asking about.

@jonathanc-n

@jonathanc-n
The main idea behind extracting a module is to identify dependencies by looking at which crates are imported.

For example, if we want to move CsvExec out, we’ll also need to move FileScanConfig and FileCompressionType. To extract FileScanConfig, we’ll need to move PartitionedFile as well. This process will reveal that we must move the entire datasource::listing module first, and whenever we need SessionState, replace it with trait Session

Oh i see, should we be putting priority on this soon? or should this be visited back another time?

Oh i see, should we be putting priority on this soon? or should this be visited back another time?

I will help to review with this.

@alamb

I went through the thread and I couldn't figure out if we had identified/agreed on a set of issues that compose this epic @alamb

@alamb

Oh i see, should we be putting priority on this soon? or should this be visited back another time?

In my opinion, this would qualify as "Things that make DataFusion easier to work with" (details here) and thus I will also try and find time to help this project along (both reviews as well as project planning)

I have been annoyed recently at the amount of time it takes to compile datafusion

For example, if we want to move CsvExec out, we’ll also need to move FileScanConfig and FileCompressionType. To extract FileScanConfig, we’ll need to move PartitionedFile as well. This process will reveal that we must move the entire datasource::listing module first, and whenever we need SessionState, replace it with trait Session

I went through the thread and I couldn't figure out if we had identified/agreed on a set of issues that compose this epic @alamb

Here is what I suggest:

Try and complete moving the items listed in this issues description.
Do NOT try and move ListingTableProider as part of this project (that is big enough that I think we should make its own epic for the reasons @jonathanc-n mentions)

I took a quick look and I think there are two things we could move to datafusion-catalog to complete this ticket:

The Memory* providers: https://github.com/apache/datafusion/blob/4c1ec807fb70ef8abe96299760dd2faaa556a49c/datafusion/core/src/catalog_common/memory.rs
The InformationSchema providers: https://github.com/apache/datafusion/blob/4c1ec807fb70ef8abe96299760dd2faaa556a49c/datafusion/core/src/catalog_common/information_schema.rs

Perhaps we can file tickets for those items and I can link them to this epic?

Hi, I am working on moving InformationSchema into the datafusion-catalog.
This would require moving core/src/datasource/streaming.rs (StreaminTable) to some place out of the core,
I think moving it to datafusion-catalog should work fine since that's where the orignal TableProvider resides. So, that's what I am going ahead with, in the meantime let me know if you think otherwise.
Thanks

Hi, I am working on moving InformationSchema into the datafusion-catalog. This would require moving core/src/datasource/streaming.rs (StreaminTable) to some place out of the core, I think moving it to datafusion-catalog should work fine since that's where the orignal TableProvider resides. So, that's what I am going ahead with, in the meantime let me know if you think otherwise. Thanks

I think it makes sense to move StreamingTable as it is a relatively thin wrapper around Arc<dyn PartitionStream> 👍

Navigation Menu

Repository navigation

[Epic] Extract catalog functionality from the core to make it more modular #10782

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Epic] Extract catalog functionality from the core to make it more modular #10782

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

alamb commented on Jun 3, 2024

lewiszlw commented on Jun 4, 2024

comphead commented on Jun 4, 2024

jayzhan211 commented on Jun 5, 2024

alamb commented on Jun 5, 2024

alamb commented on Jun 5, 2024

comphead commented on Jun 5, 2024

alamb commented on Jun 6, 2024

jayzhan211 commented on Jul 11, 2024

jayzhan211 commented on Jul 12, 2024

alamb commented on Jul 12, 2024

jayzhan211 commented on Jul 13, 2024

alamb commented on Jul 15, 2024

9 remaining items

jayzhan211 commented on Jul 17, 2024

findepi commented on Aug 20, 2024

jayzhan211 commented on Aug 20, 2024

alamb commented on Aug 20, 2024

jayzhan211 commented on Sep 2, 2024

findepi commented on Sep 2, 2024

jayzhan211 commented on Nov 9, 2024

jonathanc-n commented on Nov 9, 2024

jayzhan211 commented on Nov 9, 2024

edmondop commented on Nov 11, 2024

alamb commented on Nov 13, 2024

logan-keede commented on Jan 29, 2025

alamb commented on Jan 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions