Repository navigation
[EPIC] Improve examples to make them easier to navigate #11172
Description
Is your feature request related to a problem or challenge?
The current number of examples in https://github.com/apache/datafusion/tree/main/datafusion-examples is somewhat overwhelming and hard to navigate
Describe the solution you'd like
I would like fewer consolidated examples. For example a sql.rs example rather than SQL examples being spread out across (at least) 9 different files
- parqet_sql
- parqet_sql_multiple_files
- regexp
- to_char
- to_date
- to_timestamp
- make_date
- avro_sql.rs
Subtasks
- Consolidate Example: simplify_udaf_expression.rs into advanced_udaf.rs #13842
- Consolidate Example: dataframe_output.rs into dataframe.rs #13844
- Consolidate Example: simplify_udwf_expression.rs into advanced_udwf.rs #13842 #13843
- Replace
println!withassert!if possible in DataFusion examples #11230 - datafusion-examples CI run is failing:
final link failed: No space left on device#11210 - Automatically test doc examples in user guide #1813
- Consolidate Example: dataframe_subquery.rs into dataframe.rs #13911
- Consolidate Examples: memtable.rs and parquet_multiple_files.rs #13912
- Consolidate Examples: regexp.rs into dataframe.rs and sql_query.rs #13938
- Consolidate csv_opener.rs and json_opener.rs into a single example #13955
- Minor: consolidate ConfigExtension example into API docs #13954
Describe alternatives you've considered
No response
Additional context
Also, I am trying to add some new examples, but the CI in my PR #11088 is failing due to running out of space
Example failure: https://github.com/apache/datafusion/actions/runs/9715084012/job/26815819949
= note: /usr/bin/ld: final link failed: No space left on device
collect2: error: ld returned 1 exit status
I think it is because each example creates its own binary which fills up the disk
Thus I also want to reduce the number of examples so I can add new ones :)
Reacted by Jacob Sherin, Bruce Ritchie and Edmondo Porcu
Activity
alamb commentedon Jun 29, 2024
While working with @lewiszlw in #11173 (comment) I think I came up with a better idea -- that is to update the user guide so they code examples are run / tested
Then we can move many/all of the examples from rust to the user guide, which will make them easier to find and navigate. 🏃
alamb commentedon Jun 29, 2024
I played around with what we have and it turns out I think we can test examples in the user guide. I made a PR with docs of how to do it: #11178
I think with that framework we could add many of the simpler examples directly into the documentation where they would be eaiser to find. We can leave the more complex ones in datafusion-examples perhaps
Thoughts?
efredine commentedon Jun 29, 2024
As a newcomer I'll say that this seems right to me. Examples that are right in the documentation are much more discoverable.
[-]Consolidate examples to make them easier to navigate[/-][+][EPIC] Improve examples to make them easier to navigate[/+]alamb commentedon Jul 15, 2024
I plan do to this as a background task, but realistically can't devote enough focus time to it now
logan-keede commentedon Jan 22, 2025
take
alamb commentedon Jan 22, 2025
Thank you @logan-keede 🙏
After #14240 I don't think there is any example left for consolidation that I can see
logan-keede commentedon Jan 23, 2025
@alamb I think there is one more thing we can do, though perhaps it is only my preference.
We can consolidate them into folders, a big
sql.rsmight hinder ease of navigation, but if we are going to keep adding examples it might be better to have them in a directory, we can do that for other general domains likeparquetandcsvtoo.Except that, I think we can close this issue.
alamb commentedon Jan 23, 2025
I actually think keping them in a single directory, while somewhat overwhelming, is good for discoverability (aka knowing what is available) so until we get more I would recommend we keep it that way