Repository navigation
Real-time streaming support #10895
Description
Is your feature request related to a problem or challenge?
Good day. Are there any plans to support real-time streaming of Arrow record batches? The use case I imagine would be that we could setup an Arrow Flight client on our side that will receive and process the stream from the Session Context upon executing the query. Otherwise, are there any recommendations that could enable this workflow?
Thanks as always.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Activity
alamb commentedon Jun 14, 2024
Hi @maronavenue --
I think several people use DataFusion for this - I think https://docs.rs/datafusion/latest/datafusion/datasource/streaming/struct.StreamingTable.html gives
@matthewmturner recently added an example in https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/file_stream_provider.rs in #10600
I think the documentation could still be improved, so if you have any time to help out that would be most appreciated 🙏
maronavenue commentedon Aug 20, 2024
@alamb - Sorry it took a while, but I wanted to say thanks for sharing the above code references. I am still in the early stages of learning and discovering more things with DataFusion, but I'm more than happy to contribute.
alamb commentedon Aug 22, 2024
Thanks @maronavenue -- all improvements to the documentation would be most welcome. I personally find when I try to explain something makes me undertand it.
Also, if you make a draft PR, even if it has gaps / TODOs, we can often find time to help fill them in.
maronavenue commentedon Sep 7, 2024
@alamb - Took my first stab at #12379. I have started looking at the docs setup. Hoping to contribute meaningfully on that front soon.
Will keep this open for potential documentation.
alamb commentedon Aug 11, 2026
We are now tracking streaming in a larger epic: