Building project takes a *long* time (esp compilation time for datafusion core crate)

Is your feature request related to a problem or challenge?

Compiling the datafusion crate currently takes 40 seconds on my machine, far longer than any other crate

This slows down CI builds as well as my own local development workflow

For example, running

# start from clean checking
rm -rf target
cargo build --timings

Generates a chart as follows (attached here): carg-timings.zip

Describe the solution you'd like

I would like to speed up compilation somehow -- likely by decreasing the time required for datafusion-core

Describe alternatives you've considered

I think the first thing would be to figure out if possible what is taking up so much time when building the core crate

I suspect it has to do with listing table / some of the various file format support, but I don't have data to justify that

Additional context

No response

install sccache and tell rust to use it

export RUSTC_WRAPPER=sccache

alias cargo="RUSTFLAGS='-Z threads=8' cargo +nightly"

With the above two options I see a full build on my machine taking 100.8s (1m 40.8s).

https://corrode.dev/blog/tips-for-faster-rust-compile-times/

@Omega359

With the above two options I see a full build on my machine taking 100.8s (1m 40.8s).

This makes sense -- thank you @Omega359

However, I think it will not help CI where the use of such caching doesn't help (and since the intermediate rust files are so large trying to use cargo cache wasn't effective in the past when we tried)

I would love to figure out how to break the datafusion core crate into smaller pieces / crates that can be compiled in parallel

fwiw I tried to gather a bit more info with llvm-lines:

cargo llvm-lines -p datafusion --lib | head -20

Lines                  Copies               Function name
  -----                  ------               -------------
  1220519                36986                (TOTAL)
    44221 (3.6%,  3.6%)    423 (1.1%,  1.1%)  <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
    25426 (2.1%,  5.7%)    265 (0.7%,  1.9%)  alloc::vec::Vec<T,A>::extend_desugared
    22436 (1.8%,  7.5%)    142 (0.4%,  2.2%)  <arrow_array::array::primitive_array::PrimitiveArray<T> as core::iter::traits::collect::FromIterator<Ptr>>::from_iter
    21263 (1.7%,  9.3%)     45 (0.1%,  2.4%)  <core::iter::adapters::flatten::FlattenCompat<I,U> as core::iter::traits::iterator::Iterator>::size_hint
    16910 (1.4%, 10.7%)    151 (0.4%,  2.8%)  <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
    15949 (1.3%, 12.0%)    172 (0.5%,  3.2%)  alloc::vec::Vec<T,A>::extend_trusted
    12204 (1.0%, 13.0%)    116 (0.3%,  3.6%)  <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold::{{closure}}
    10585 (0.9%, 13.8%)     88 (0.2%,  3.8%)  <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
    10438 (0.9%, 14.7%)    295 (0.8%,  4.6%)  <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
     9702 (0.8%, 15.5%)     53 (0.1%,  4.7%)  alloc::vec::in_place_collect::from_iter_in_place
     8645 (0.7%, 16.2%)     13 (0.0%,  4.8%)  std::io::default_read_to_end
     8256 (0.7%, 16.9%)    116 (0.3%,  5.1%)  core::iter::adapters::try_process
     7939 (0.7%, 17.5%)    356 (1.0%,  6.0%)  alloc::boxed::Box<T>::new
     7673 (0.6%, 18.2%)    236 (0.6%,  6.7%)  core::iter::adapters::map::map_fold::{{closure}}
     7375 (0.6%, 18.8%)      1 (0.0%,  6.7%)  datafusion::physical_planner::DefaultPhysicalPlanner::map_logical_node_to_physical::{{closure}}
     7352 (0.6%, 19.4%)    241 (0.7%,  7.3%)  tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
     7317 (0.6%, 20.0%)    102 (0.3%,  7.6%)  core::iter::traits::iterator::Iterator::try_fold

I would love to figure out how to break the datafusion core crate into smaller pieces / crates that can be compiled in parallel

yes! and move around sqlparser dependency when doing so :)

FWIW the only reliable mechanism I've found to measure this is to comment out modules and measure the impact on compilation time. Llvm-lines and cargo-bloat can be informative, but not all lines are created equal. This is especially true for code with complex lifetimes and/or async.

Anything making extensive use of macros is likely a good place to start

BTW I did some profiling on this today and it seems a non trivial amount of time is spent type checking / resolving functions somehow...

might be compiler related?

IIRC that relates to type checking expressions, and therefore this would suggest the compiler is spending a lot of time resolving generics. At least historically non-boxed async has been a good source of this, as async "desugars" to quite complex generic code, and often has quite complex Send / lifetime constraints.

Edit: rust-lang/rust#87012 is the related issue I was thinking of

@tustvold

Thanks @tustvold it was nice attempt to solve it 3 years ago, hopefully someone is planning to close it out one day

@AdamGS

One thing that is strange is that we have moved most of the code out of datafusion-core now.

I am still hoping we can get https://github.com/apache/datafusion/tree/main/datafusion/core/src/datasource/physical_plan out of the core somehow (I know @AdamGS is thinking about that)

However, I don't think that is the problem either as all the formts are feature gated and when I build with no default feaatures

When I build with no default features:

cargo build --timings -p datafusion --no-default-features

I get

And then adding parquet (by far the largest / most complex), it goes up by only 1 seconds

cargo build --timings -p datafusion --no-default-features --features=parquet

I also tried removing all the doctests from lib.rs and that didn't help (I didn't expect it to but I wanted to try)

diff --git a/datafusion/core/src/lib.rs b/datafusion/core/src/lib.rs
index b4d5f9740..9e1f9795e 100644
--- a/datafusion/core/src/lib.rs
+++ b/datafusion/core/src/lib.rs
@@ -824,282 +824,3 @@ pub mod test;

 mod schema_equivalence;
 pub mod test_util;
-
-#[cfg(doctest)]
-doc_comment::doctest!("../../../README.md", readme_example_test);
-
-// Instructions for Documentation Examples

I also tried cutting out (ablating) the contents of b/datafusion/core/src/physical_planner.rs but that diidn't seem to make any difference either

@tustvold

using following command on release 42.0.0, 45.0.0, and current main.

cargo build -p datafusion-cli --timings -j 10

[42.0.0]

[45.0.0]

[46.0.0]

There is some improvement in build time (201sec -> 190 sec-->183sec), for datafusion crate(65->65->49).(some progress!!)

also, this article and the issue mentioned above by @tustvold(rust-lang/rust#87012) looks promising. The problem cause also seems to be similar between evaluate_obligation[1] and async_trait. Though I am not sure if we can apply the same tricks in our usecase. Is it technically possible[2] and worth a try?

[1]: I think evaluate_obligation and process_obligation are same, correct me if I am wrong.
[2]: I am guessing it should be possible but not too sure.

There is some improvement in build time (201sec -> 190 sec-->183sec), for datafusion crate(65->65->49).(some progress!!)

Pulling stuff out of core helped! Thank you !

I was wondering recently if the the probably could be related to all the re-exports (pub use ....) we do in DataFusion. Maybe we could see if reducing them helped 🤔

I think before we spend a lot of time engineering some solution we should try and do something quick and dirty to prove it will help. If it does then we can figure out how to make it work reasonably / get it in chunks

Docs: document manual catalog impls and how compilation got faster (a…

perf: Update `TableProvider` to take scan projection as `Option<&[usi…

perf(catalog): cut `datafusion-catalog` compile time ~6x (apache#24325)

perf(core): cut `datafusion` core compile time ~10x (apache#24329)

perf(catalog-listing): cut `datafusion-catalog-listing` compile time …

perf(session): cut `datafusion-session` compile time ~1.7x (apache#24326

perf(build): don't build the doc-generator binaries by default (apach…

perf(build): use line-tables-only debug info in the `dev` profile (ap…

Docs: document manual catalog impls and how compilation got faster (a…

perf: Update `TableProvider` to take scan projection as `Option<&[usi…

@crepererum

@crepererum pointed out recently that there is a Rust blog post about the new solver in the Rust compiler (and it mentions that datafusion specifically as something that gets 10x faster with the new solver):

https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/

I am not sure if this will be in addition to what @Dandandan did in #13814 (comment), but it is pretty cool to see the shoutout

@crepererum pointed out recently that there is a Rust blog post about the new solver in the Rust compiler (and it mentions that datafusion specifically as something that gets 10x faster with the new solver):

https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/

I am not sure if this will be in addition to what @Dandandan did in #13814 (comment), but it is pretty cool to see the shoutout

Yes - it is the same issue, so this will probably no longer give any speed up - in fact I measured compilation to be a bit slower in the next gen trait solver (I guess it probably will be optimized a bit further before stabilization).

10x (or 4x in the linked article) refers to optimizing the new solver (which was slower than the old one first), also it probably cargo check or something on that module only (i.e. not all dependent crates). The 10x happens to correspond with perf(core): cut datafusion core compile time ~10x #24329 as well.
I traced the issues in the old trait solver: Trait solver re-derives Send proofs per impl when an outlives where-bound is in scope rust-lang/rust#157595 (comment)

Possibly related: #24727 tracks reducing unnecessary monomorphization (generic functions that compile to many near-identical copies).

I think we should close this issue and open a new one for additional build time issues so that we can start a clean slate. I think the root cause here was fixed, and we can always improve build times

Navigation Menu

Repository navigation

Building project takes a long time (esp compilation time for `datafusion` core crate) #13814

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

36 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Building project takes a *long* time (esp compilation time for datafusion core crate) #13814

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

Omega359 commented on Dec 17, 2024

alamb commented on Dec 17, 2024

Omega359 commented on Dec 17, 2024

findepi commented on Dec 18, 2024

tustvold commented on Dec 19, 2024

alamb commented on Mar 10, 2025

comphead commented on Mar 10, 2025

tustvold commented on Mar 10, 2025

comphead commented on Mar 10, 2025

alamb commented on Mar 11, 2025

alamb commented on Mar 11, 2025

alamb commented on Mar 11, 2025

logan-keede commented on Mar 14, 2025

alamb commented on Mar 15, 2025

36 remaining items

alamb commented on Aug 24, 2026

Dandandan commented on Aug 24, 2026

alamb commented on Aug 27, 2026

alamb commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Building project takes a long time (esp compilation time for `datafusion` core crate) #13814