[EPIC] A collection of tickets for improved WASM support in DataFusion #13815

Activity

1. alamb

   added

   enhancementNew feature or requestNew feature or request

   on Dec 17, 2024
2. alamb

   changed the title [-][EPIC] A collection of tickets for supporting WASM in DataFusion[/-] [+][EPIC] A collection of tickets for improved WASM support in DataFusion[/+] on Dec 17, 2024
3. savaliyabhargav commented on Mar 14, 2025

   

   savaliyabhargav

   on Mar 14, 2025

   More actions

   Dear @alamb ,

   I hope you’re doing well.

   I’m very interested in contributing to the WASM support for DataFusion project as part of GSoC 2025. Enhancing embeddability and ensuring robust WASM integration aligns with my passion for building reliable and maintainable software. I’d greatly appreciate it if you could share more details on the project’s current challenges and key milestones. Understanding these aspects will help me prepare a thoughtful and well-structured proposal.

   Thank you for your time and guidance.

   Best regards,
   bhargav
4. alamb commented on Mar 17, 2025

   

   alamb

   on Mar 17, 2025

   ContributorAuthor

   More actions

   I’m very interested in contributing to the WASM support for DataFusion project as part of GSoC 2025. Enhancing embeddability and ensuring robust WASM integration aligns with my passion for building reliable and maintainable software

   Thank you @savaliyabhargav

   I think the key challenge is the same as in this ticket's description:

   DataFusion can be compiled today to WASM with some care. However it is somewhat tricky and the experience could be improved.

   I suggest some key milestones:

   1. Write a blog post explaining what WASM is, why it is important, and how to use DataFusion with it
   2. Add example / section to the documentation about compiling and building with WASM
   3. Make several example WASM apps with DataFusion -- for example Browser-accessible official DataFusion playground / DataFusion fiddle #13818
   4. Stretch goal -- prototype how we would support WASM user defined functions
5. matthewmturner commented on Mar 17, 2025

   

   matthewmturner

   on Mar 17, 2025

   Last edited by matthewmturner

   Contributor

   More actions

   4. Stretch goal -- prototype how we would support WASM user defined functions

   On this point, I was able to get WASM UDFs working in dft (im sure there is room for improvement but i got both WASM Native and Arrow IPC formats working) and as part of the next release I was planning to publish that crate so others could use it. @savaliyabhargav I would be happy to collaborate on that if you are interested in it.
6. savaliyabhargav commented on Mar 18, 2025

   

   savaliyabhargav

   on Mar 18, 2025

   More actions

   @matthewmturner yes sure i am interested can you please give me more detail about it
7. matthewmturner commented on Mar 19, 2025

   

   matthewmturner

   on Mar 19, 2025

   Contributor

   More actions

   For the WASM UDFs they just need some more real world testing / benchmarking. To be honest, the other points @alamb mentioned would probably better benefit the DataFusion ecosystem at this stage. But, if WASM UDFs are what interest you then you could make an issue on the dft repo and we could discuss some tangible next steps for that.
8. qstommyshu commented on Mar 19, 2025

   

   qstommyshu

   on Mar 19, 2025

   Contributor

   More actions

   Hi, I'm also interested in working on this project for Gsoc 2025. I'm currently a master student at Georgia Institute of Technology, I had some experiences with WASM and Rust in my undergrad study, and I would like to see how that can be incorporated with DataFusion!

   Are there any prerequisite for this project? I suppose the points below would be the key points for our proposal, right?

   I suggest some key milestones:

   Write a blog post explaining what WASM is, why it is important, and how to use DataFusion with it
   Add example / section to the documentation about compiling and building with WASM
   Make several example WASM apps with DataFusion -- for example #13818
   Stretch goal -- prototype how we would support WASM user defined functions

   I'm new to DataFusion but I've heard of the name for a long long time.

   1. Can you please give me some relative materials to look at for this project? or working on some "good first issue" should be enough for me to get familiar with DataFusion?
   2. Can you please give me some guidance on how to work on an precise and appropriate proposal?

   Thanks in advance!
9. savaliyabhargav commented on Mar 20, 2025

   

   savaliyabhargav

   on Mar 20, 2025

   More actions

   @matthewmturner Please provide me with some information on DFT so that I can learn and work on your project effectively
10. ilikepi63 commented on May 26, 2025

    

    ilikepi63

    on May 26, 2025

    More actions

    Hello 👋

    Not sure if this is the correct place, but I've been working on a very rough implementation of using Arrow in Wasmtime here: https://github.com/ilikepi63/arrow_wasmtime.

    My approach boils down to having the wasm module expose a _malloc function which is then used to reserve space for the Arrow pointers' data. I've adjusted some of the code from the arrow::ffi module to allow the copying of Arrays + Schema over the FFI boundary. It's not the greatest considering there are multiple calls to _malloc (that might be able to be done once in a similar fashion to arena allocation?), multiple copies to the wasm modules' data and that alignment is different (wasm usually is 32 whilst my machine is 64).

    For a simple example of doing this, please see here: https://github.com/ilikepi63/arrow_wasmtime/blob/main/tests/simple_addition.rs (you'll need to cargo build the example_wasm function)

    ❤️React with ❤️1Reacted by Andrew Lamb
11. alamb

    added

    EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks

    PROPOSAL EPICA proposal being discussed that is not yet fully underwayA proposal being discussed that is not yet fully underway

    and removed

    EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks

    on Aug 12, 2025
12. logankaser commented on Jul 10, 2026

    

    logankaser

    on Jul 10, 2026

    Last edited by logankaser

    More actions

    datafusion 54 regresses pretty far on wasm support, with default features requiring zstd and thus full Emscripten to compile.
    53.1 is the last version that it's reasonably possible to use with wasm
13. kylebarron commented on Jul 10, 2026

    

    kylebarron

    on Jul 10, 2026

    Member

    More actions

    Is it only the inclusion of the feature as default that changed? Did you try opting-in manually to specific features and excluding zstd?

    FWIW in other projects (e.g. parquet) I've been able to compile zstd to wasm32-unknown-unknown without any issue. If you're on macos you need to switch the clang you use to compile.
14. alamb commented on Jul 10, 2026

    

    alamb

    on Jul 10, 2026

    ContributorAuthor

    More actions

    datafusion 54 regresses pretty far on wasm support, with default features requiring zstd and thus full Emscripten to compile. 53.1 is the last version that it's reasonably possible to use with wasm

    @logankaser can you please provide a reproducer for this?

    We have a wasm test module https://github.com/apache/datafusion/tree/main/datafusion/wasmtest precisely designed to ensure DataFusion can be compiled under wasm. If it lacking coverage I think we should fix it.
15. kumarUjjawal

    added a commit that references this issue on Sep 10, 2026

    perf: reduce invariant-check code generation (apache#25073)

    ...

    423df6f