Repository navigation
datafusion-cli requires more than 8GB to build #18471
Description
Describe the bug
@pmcgleenon reports on #17721 (comment)
The instance type c6a.xlarge only has 8GB RAM (the clickbench dataset is 15GB) and there are a number of issues with it, including
datafusion compilation fails on this instance. We can workaround this by using brew install
To Reproduce
You can reproduce this locally on docker using this command:
cd .devcontainer
# create a docker container
docker build -t datafusion-build .
cd ..
docker run -m 4G -v `pwd`:/datafusion -it datafusion-build /bin/bashThen try to run
cargo build --release -p datafusion-cli
This will fail with an error like this:
Caused by:
process didn't exit successfully: `/usr/local/rustup/toolchains/1.90.0-aarch64-unknown-linux-gnu/bin/rustc --crate-name datafusion_functions_aggregate --edition=2021 datafusion/functions-aggregate/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=160 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 '--warn=clippy::used_underscore_binding' --deny=unused_qualifications '--warn=clippy::unnecessary_lazy_evaluations' '--warn=clippy::uninlined_format_args' --warn=unexpected_cfgs '--warn=clippy::or_fun_call' '--warn=clippy::large_futures' '--warn=clippy::inefficient_to_string' --check-cfg 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))' --check-cfg 'cfg(tarpaulin)' --check-cfg 'cfg(tarpaulin_include)' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=727ec34a83a3b23a -C extra-filename=-41481de704778bc6 --out-dir /datafusion/target/release/deps -C strip=symbols -L dependency=/datafusion/target/release/deps --extern ahash=/datafusion/target/release/deps/libahash-16bfcefaf019ce0a.rmeta --extern arrow=/datafusion/target/release/deps/libarrow-7a0c4e030b71b835.rmeta --extern datafusion_common=/datafusion/target/release/deps/libdatafusion_common-ca5a073002b6fd0e.rmeta --extern datafusion_doc=/datafusion/target/release/deps/libdatafusion_doc-8ab038afb63bc362.rmeta --extern datafusion_execution=/datafusion/target/release/deps/libdatafusion_execution-93c2cc956c650df0.rmeta --extern datafusion_expr=/datafusion/target/release/deps/libdatafusion_expr-87611f1ca5ce8f65.rmeta --extern datafusion_functions_aggregate_common=/datafusion/target/release/deps/libdatafusion_functions_aggregate_common-05b65785638f4212.rmeta --extern datafusion_macros=/datafusion/target/release/deps/libdatafusion_macros-0bed692e75dab347.so --extern datafusion_physical_expr=/datafusion/target/release/deps/libdatafusion_physical_expr-93e2a25460efd9b7.rmeta --extern datafusion_physical_expr_common=/datafusion/target/release/deps/libdatafusion_physical_expr_common-79bb18eb023feafb.rmeta --extern half=/datafusion/target/release/deps/libhalf-feba467d2af9eeb5.rmeta --extern log=/datafusion/target/release/deps/liblog-d456af582e58c114.rmeta --extern paste=/datafusion/target/release/deps/libpaste-5dba9edad0280e64.so -L native=/datafusion/target/release/build/zstd-sys-85e0b1f1ab4f39ce/out -L native=/datafusion/target/release/build/lzma-sys-d68250324f9032a4/out -L native=/datafusion/target/release/build/aws-lc-sys-441c9757ee7585c9/out -L native=/datafusion/target/release/build/ring-dcbccc510e078148/out -L native=/datafusion/target/release/build/psm-2382bc49ac467ad4/out` (signal: 9, SIGKILL: kill)
Note the signal: 9, SIGKILL: kill
Expected behavior
No response
Additional context
No response
Activity
alamb commentedon Nov 3, 2025
I tried limiting concurrency to work around the issue but this still fails with a SIGKILL:
I was able to build it using nonlto mode:
alamb commentedon Nov 3, 2025
BTW I am not sure this is a bug we need to solve necessairly -- using a pre-built binary might be a better option