Repository navigation
Improvements to user guide for DataFrames #11388
Description
First of all, I'm not sure we need the distinction between "user guide" and "library user guide" when it comes to data frames. The only way you can use a data frame is if you are using it as library? I'm unsure why I should be reading one section or the other.
Second, I think you lose a lot of context by removing the table. The SessionContext and DataFrame structs both expose large API surfaces. I think they become much easier to digest once you understand that there is actually a fairly small number of categories of things being exposed. However, the API documentation doesn't provide any way of seeing this structure. Ideally, there would be something like a way to do something like tagging the methods into different categories.
But I think the important part is simply to note that there are transformations, methods that execute the frame and administrative methods. I might further break down the methods that execute the frame into those that return a new frame in some way and those that write to a data sink? That is, I'm not sure its necessary to list every method in each of these categories but it is helpful to identify the categories. That being said, I think a table, perhaps more granular, with links to the API documentation for each method and possibly even links to the SQL equivalent where appropriate would be a good long term goal. Is there some tooling / macros we could build to support this in a sustainable way?
Also, is it the case that I can only create a data frame via SessionContext? The typically in the introduction suggests there are other ways of doing it. I wonder if it would be better to be more precise and just enumerate the different ways you can create a data frame. I think it's something like: read from a file, read from a table (which really covers a lot of possibilities), execute SQL statements.
So - I suppose to make this executable within the context of this PR - perhaps reduce the tables to more of a summary? But also curious to hear from others.
Finally, not for this PR, I wonder if SessionContext warrants its own section. As with DataFrame I think it would benefit from a discussion of the different categories of things it can be used for. Related, it's becoming clear to me from poking around the documentation and methods its becoming clear that there is a great deal of flexibility in mixing and matching SQL and data frames if you want to but I'm not sure that's coming across in the guides? When I have time I can try drafting something to see how it might fit.
Originally posted by @efredine in #11324 (comment)
Activity
sm4rtm4art commentedon Oct 14, 2025
take
sm4rtm4art commentedon Oct 16, 2025
General thought
Is there a recommended length for documentation pages?
By “Do good and talk about it” (“Write cool software and tell people about it” – Paul Dix), I interpret “talk about it” as: catch interest quickly and then give readers a path toward deeper understanding—the why, the how, and also the how-not/what-not.
Based on that, I’d suggest keeping the “user” description broader, general reference-rich and more leaning towards SQL, and making the “library user guide” deeper and more relational, guiding readers toward docs.rs, leaning towards Rust.
(These are ballpark ranges, not hard limits.)
Proposal based on the above
Should we keep two documents as mentioned? If yes, I would write “A Gentle Introduction to DataFrames” for the User Guide—covering general ideas, connectivity, and usage—and a “Library User Guide” that is a deeper dive into DataFrames.
This split might also address the second part of the issue by giving a path from the basic user context of DataFrames to how SessionContext fits in.
Do you agree with this concept, any suggestions, or am I on the wrong track?
alamb commentedon Oct 17, 2025
It sound like a good split to me -- thank you
https://datafusion.apache.org/
I think specifically @efredine was talking about this page: https://datafusion.apache.org/library-user-guide/using-the-dataframe-api.html
maybe that would be a good place to start and see if it makes any sense / what could be done to improve it
sm4rtm4art commentedon Oct 30, 2025
Hi everyone,
I've been making significant progress on the DataFrame API documentation for issue #11388. As I delved into the topic (and fell down a rabbit hole 🌀), I found that a more comprehensive guide would be highly beneficial. To keep the content well-structured and easy to navigate, I've split it into the following interconnected files:
Given the total size of this contribution, I want to make the review process as smooth as possible for everyone.
Proposal for the review process:
My suggestion is to submit this as a series of smaller, sequential PRs rather than one massive one. I would start with concepts.md as the first PR and then follow up with the others.
Does this approach work for you, or would you prefer a single large pull request?
My goal is to create a thorough and welcoming resource.
Thanks for your guidance!
P.S. @alamb: On a related note, in your podcast with Developer Voices, you were asked about the RecordBatch size of ~8000 rows and explained that research papers show why 4k-16k is optimal. I'm very interested in this! If you could share a link to that research, I'd appreciate it. I don't plan to put it in the docs (yet), but I'm curious.
P.s.s : The unnofficial Topics (won't be in the Docs) is kind of taken from the interaction of light with matter :)
alamb commentedon Oct 30, 2025
The paper I had in mind is Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask
Specifically this chart
alamb commentedon Oct 30, 2025
This sounds like a good plan to me. Another alternative is to make one giant PR as a draft and then submit PRs one by one (referring people to the large PR if they want to see the overall context)
Thank you!
sm4rtm4art commentedon Nov 26, 2025
Hi all,
Just a quick heads up that I'm still actively working on this with all the spare time and energy I can provide. It's taken longer than expected, but I'm convinced it will add value to the project.
Sorry for being so brief—I'm truly grateful for the chance to contribute and supporting otheres with documentation.
Edit: Time estimate : Will try to open the PR end of the week or mid next week! ⏲️
docs: enhance DataFrame documentation (apache#11388)
docs: comprehensive improvements to DataFrame API documentation (apac…
docs(dataframe): execution patterns, show/show_limit semantics, write…
docs: refine DataFrame/LogicalPlan relationship section (intermediate…
docs: improve DataFrame/LogicalPlan relationship section
docs: Enhance DataFrame creation guide with production patterns
sm4rtm4art commentedon Dec 29, 2025
Hi all,
Quick update: I'm in the final refinement stages of the DataFrame documentation. I discovered Diátaxis and later how-to-write-exceptional-documentation_api-reference during the process and restructured parts of the docs to follow that framework.
Note about the linked commits: Those are from my working branch and reference this issue automatically via commit messages. They represent early work-in-progress, not final content — please disregard them for now. I feel actually ashamed of them.
Sorry for missing my original timeline — learning DataFusion internals, query engine concepts, and documentation best practices took longer than expected. But I'm confident the result will be worth it.
Looking forward to opening a PR soon!
sm4rtm4art commentedon Mar 4, 2026
Hey everyone,
Just wanted to give a quick update to let you know that I am still actively working on this issue! The scope expanded a bit more than I initially expected, but I’ve learned a ton and am really dedicated to adding value to the project.
I had a few personal and technical hurdles to overcome recently, but those are fully resolved now. I've put a lot of hours into researching and writing this, and I believe I've now covered almost all the methods the DataFrame API provides.
Right now, I'm in the final stretch. I am currently polishing the text and refactoring the structure (specifically, breaking down a massive monolithic markdown file into smaller, logical pieces in a new directory structure).
My goal is to make the final PR easy for you all to review and maintain in the future. Furthermore, I would like to give others the opportunity to learn about and use useful functions in the documentation.
I do have one minor technical question for the group:
Sphinx offers a great way to highlight specific lines in code blocks using emphasize-lines (see Sphinx docs). However, cargo test --doc doesn't seem to parse Sphinx's directives for testing, as it strictly expects standard markdown ```rust blocks.
This is not a blocker! Just a nice to have
Does anyone know a clean workaround for this?
For now, I will stick to standard markdown code blocks so that the doctests pass which is prioritized, but I'd love to use line highlights if possible.
Thanks again for your generous patience and for letting me tackle this. I'm looking forward to opening the PR soon!
alamb commentedon Mar 4, 2026
I think we use doc-comment to run the tests via the rust
cargo test --docdatafusion/Cargo.toml
Line 153 in 92d0a5c
And
datafusion/datafusion/core/src/lib.rs
Lines 933 to 936 in 92d0a5c
Maybe there is some setting there we could use (we are on the 0.3 release of doc-comment and there is a 0.4 release...)