Repository navigation
Dynamic Filter marked as not used #19715
Description
Describe the bug
For one user the DynamicFilterPhysicalExpr function is getting marked as unused. I'm openning this issue to discuss why this could be happening .
More context on #18566 (comment)
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
adriangb commentedon Jan 9, 2026
I'll note that if the using the Arc pointers, etc. is too much of an issue or we can't resolve this bug / usage we could always go for an explicit API to register "consumers" of a dynamic expression. But I think that'd be a worse user experience (things need to remember to call it, etc.).
LiaCastaneda commentedon Jan 9, 2026
👋 @tobixdev the
innerstruct ofDynamicFilterPhysicalExprgets updated (the ref count increases from 1 to 2) when we callwith_new_childrenduring the filter pushdown optimization rule.I'm trying to understand why
with_new_childrenis not being called for your query 🤔 this might be an edge case I overlooked. The reason why I didn't used the strong count ofDynamicFilterPhysicalExpris precisely because when with_new_children is called forDynamicFilterPhysicalExprit creates a newDynamicFilterPhysicalExprbut keeps the Arc of theinneralamb commentedon Jan 9, 2026
Could it be that https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.reset_state is being called (or should be called)?
tschwarzinger commentedon Jan 10, 2026
Thank you for opening the issue @LiaCastaneda ! I am not sure what the problem is but I guess it has something to do with our custom data source as our project is far away from production ready.
For example, we currently do not implement reset state in our custom data source. Maybe that's it.
I think the next step is that I'll create a reproducer. Hopefully, then we can identify and fix the problem. I'll try to do it sometime next week.
Thanks again and I'll be back with a reproducer!
LiaCastaneda commentedon Jan 10, 2026
I suspect this is happening because you have a custom
DataSource, that differently toFileScanConfig(used for parquet) doesn't callreassign_expr_columnsinsidetry_pushdown_filters(code), sowith_new_childrenis never called. We should fix this since it should work out of the box for any DataSource, I will try to think of something...adriangb commentedon Jan 10, 2026
Maybe we can say it’s used if either one of the outer or inner Arc have > 1 ref?
LiaCastaneda commentedon Jan 10, 2026
yep that works, if the dynamic filter is used the consumer will have either a ref >1 in the inner struct (because
with_new_childrenwas called and cloned inner) or outer struct (with_new_childrenwas not called), otherwise there is no way for the consumer to see updates on the DF.I have a repro similar to @tobixdev
DataSourcehere and verified @adriangb suggestion fixes it.LiaCastaneda commentedon Jan 10, 2026
(will open a PR shortly)
Fix dynamic filter is_used function (#19734)
Fix dynamic filter is_used function (apache#19734)
Fix dynamic filter is_used function (apache#19734)
Bring dynamic filtering cherry picks (#70)
Fix dynamic filter is_used function (apache#19734)