Repository navigation
FFI schema providers fail when views are registered #21155
Description
Describe the bug
When you have a FFI based schema provider and you register a view with it, it is unable to be executed. This is a regression from 51.0.0 to 52.3.0.
To Reproduce
Add this unit test to datafusion/ffi/tests/ffi_catalog.rs
#[tokio::test]
async fn test_view_on_schema_provider() -> datafusion_common::Result<()> {
let module = get_module()?;
let (ctx, codec) = super::utils::ctx_and_codec();
let ffi_catalog =
module
.create_catalog()
.ok_or(DataFusionError::NotImplemented(
"External catalog provider failed to implement create_catalog"
.to_string(),
))?(codec);
let foreign_catalog: Arc<dyn CatalogProvider + Send> = (&ffi_catalog).into();
let _ = ctx.register_catalog("fruit", foreign_catalog);
let df = ctx.table("fruit.apple.purchases").await?;
let df_view = df.into_view();
let _ = ctx
.catalog("fruit")
.unwrap()
.schema("apple")
.unwrap()
.register_table("sales".to_owned(), df_view);
let df = ctx.table("fruit.apple.sales").await?;
let results = df.collect().await?;
assert_eq!(results.len(), 2);
let num_rows: usize = results.into_iter().map(|rb| rb.num_rows()).sum();
assert_eq!(num_rows, 5);
Ok(())
}
Expected behavior
These views should work.
Additional context
Originally filed as apache/datafusion-python#1444
Activity
timsaucer commentedon Mar 25, 2026
The reason this worked in 51.0.0 is because we created a
SessionContextwithin the FFI providers. So when we calledcreate_physical_planinDataFrameTableProviderit had a localSessionContext(not the actual session context that registration was happening with) and that was able to do the appropriate downcasting toViewTablefor the table scan.The reason this does not work in 52.3.0 is because we now have a full
FFI_Sessioninstead of creating a temporary locally heldSessionContext. Thus when we have aArc<dyn TableProvider>in the protobuf serialization code it shows up as aForeignTableProviderand NOT aViewTable. Thus we try the "fall back" to a logical codec which gives the user error we end up with.timsaucer commentedon Mar 25, 2026
This is resolved in
mainand the fix is to here: https://github.com/apache/datafusion/blob/main/datafusion/ffi/src/table_provider.rs#L394C9-L396C10Open question of if we need to backport to (another) release of 52 since 53 is already out.
Support table views on our schema provider
alamb commentedon Apr 6, 2026
I am making a new 52 so I'll also make a backport of this one (well, really have an agent do it)
alamb commentedon Apr 6, 2026
Actually, after some research I see that this is not is not a trivial backport, so I'll hold off on doing anything and leave it to to experts
[BRANCH-52] fix: foreign inner ffi types (#21439)
[BRANCH-52] fix: foreign inner ffi types (#21439)
[branch-53] fix: foreign inner ffi types (#21439) (#21524)