Repository navigation
Catalog abstraction #126
Description
Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10902
This issue is intended to be a design discussion around the introduction of the notion of data catalog into DataFusion.
A first draft containing some observations can be found here: [https://docs.google.com/document/d/1dy2eiJDtmhEHKit1ekR-7_LguMO8z-72ORIPCCIHWOo/edit#|https://docs.google.com/document/d/1dy2eiJDtmhEHKit1ekR-7_LguMO8z-72ORIPCCIHWOo/edit?usp=sharing]
To add a bit of context, a higher level description of DataFusion's interfaces can be found here: [https://docs.google.com/document/d/1S5bR48Apzvp1cXOVrlk-YJnc-Aml-IapRrEs3cLvz6c/edit?usp=sharing]
Activity
alamb commentedon Apr 26, 2021
Comment from Remi Dettai(rdettai) @ 2020-12-14T15:51:33.144+0000:
Comment from Jorge Leitão(jorgecarleitao) @ 2020-12-14T16:00:50.035+0000:
Comment from Remi Dettai(rdettai) @ 2020-12-14T16:35:44.060+0000:
houqp commentedon Sep 15, 2021
I think this can be closed now?
alamb commentedon Sep 15, 2021
I defer to @rdettai -- we have a catalog abstraction in DataFusion but it is only part of what is described in the document linked in the description
rdettai commentedon Sep 15, 2021
There are different definitions of what a Catalog really is that are conflicting. I don't think that this issue format is the right way to track the topic I was interested in initially. Also the attached document is kind of outdated, this one is more relevant. I would say we can close this. Thanks @houqp 😃
alamb commentedon Sep 15, 2021
Good call -- we can also always open a new issue for follow on work.
feat: Minimize number of connections used by parallel reader (apache#126
build(deps): bump object_store from 0.5.2 to 0.5.3 (apache#126)