Repository navigation
How to write csv file to disk from a empty dataframe? #16240
Description
I want write a csv file only include headers;
use datafusion::config::CsvOptions;
use datafusion::dataframe::DataFrameWriteOptions;
use datafusion::error::Result;
use datafusion::prelude::*;
use std::path::Path;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
let df = ctx.sql("SELECT 1 limit 0").await?;
let mut cfg1 = DataFrameWriteOptions::new();
cfg1 = cfg1.with_single_file_output(true);
let mut cfg2 = CsvOptions::default();
cfg2.has_header = Some(true);
df.write_csv("/tmp/t.csv", cfg1, Some(cfg2)).await?;
assert_eq!(Path::new("/tmp/t.csv").exists(), true); // panic!
Ok(())
}
Metadata
Metadata
Assignees
Labels
No labels
Activity
mmooyyii commentedon Jun 3, 2025
I want to do something like materialized view. I have to deal with this special case if datafusion can't write empty data frame;
alamb commentedon Jun 3, 2025
I am not sure -- it seems like maybe we should have an option in DataFrameWriteOptions that will force an empty file to be written even if there are no rows 🤔
chenkovsky commentedon Jun 4, 2025
take
alamb commentedon Jul 8, 2025
Unfortunately, we had to revert the original fix, see
So I am reopening the ticket to track
vadimpiven commentedon Jul 9, 2025
I can add that for Arrow format the empty DataFrame also does not create a file.
nlimpid commentedon Dec 14, 2025
take
fix: emit empty RecordBatch for empty file writes (#19370)