Skip to content

Use the shared aggr_test_schema in CSV scan tests instead of a local copy #26306

Description

@andygrove

Is your feature request related to a problem or challenge?

The test module in core/src/datasource/physical_plan/csv.rs defines a private aggr_test_schema() that is a verbatim copy of the public datafusion::test_util::aggr_test_schema(). If the shared schema changes (for example new columns or metadata on aggregate_test_100.csv), the CSV tests keep using the stale copy.

The same module also repeats about 35 lines of scan setup in every rstest case (session context, partitioned_file_groups, CsvOptions, TableSchema, CsvSource, FileScanConfigBuilder), differing only in projection, limit and the expected snapshot (L130-L442). json.rs (L213-L372) does the same with prepare_store and JsonSource.

Describe the solution you'd like

  1. Delete the local aggr_test_schema in the csv.rs tests and import crate::test_util::aggr_test_schema. Keep aggr_test_schema_with_missing_col as a derivation of it.
  2. Optionally add a test helper such as csv_scan_config(file_compression_type, file_schema, |builder| ...) -> FileScanConfig that does the shared setup, so each rstest case only applies its projection or limit and asserts. Do the same for json.rs.

Describe alternatives you've considered

Do step 1 only.

Additional context

Step 1 is a small, self-contained change and a good first issue. Line references are pinned to commit 0d79252.

Activity

  1. added
    enhancementNew feature or request
    help wantedExtra attention is needed
    coreCore DataFusion crate
    tech debtCode complexity, duplication or structure that makes the codebase harder to change
    on Oct 14, 2026
  2. med9110 commented on Oct 14, 2026

    @med9110

    Hi @andygrove, I'd like to work on #26306. I'll start by replacing the duplicated schema with the shared test utility and verifying that the existing CSV tests pass. Would you prefer keeping the optional scan-setup refactoring separate?

  3. mohitgurav20 commented on Oct 14, 2026

    @mohitgurav20
    Contributor

    take

  4. Reza98Sh commented on Oct 17, 2026

    @Reza98Sh

    4. take

    You can't take all open issues !!!

  5. krishnagp97 commented on Oct 19, 2026

    @krishnagp97

    Hi @andygrove, I'm interested in contributing to this issue as my first DataFusion contribution. I noticed multiple contributors have expressed interest, so I wanted to check whether this task is still available.

    I'm happy to work on another good first issue if someone is already handling it. Thanks!

  6. added a commit that references this issue on Oct 24, 2026
    0a42aca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreCore DataFusion crateenhancementNew feature or requestgood first issueGood for newcomershelp wantedExtra attention is neededtech debtCode complexity, duplication or structure that makes the codebase harder to change

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions