Default parquet reader to reading 64K footer  #4459

Activity

1. alamb commented on Dec 1, 2022

   

   alamb

   on Dec 1, 2022

   ContributorAuthor

   More actions

   Any thoughts @tustvold or @Ted-Jiang ?
2. Ted-Jiang commented on Dec 2, 2022

   

   Ted-Jiang

   on Dec 2, 2022

   Last edited by Ted-Jiang

   Member

   More actions

   Make sense to me, I think we have to notice the best practice to user keeping footer size less than 64k.
   And i can not find a tool to read parquet footer size 😂
3. thinkharderdev commented on Dec 2, 2022

   

   thinkharderdev

   on Dec 2, 2022

   Last edited by thinkharderdev

   Contributor

   More actions

   #!/bin/bash
   
   le=`xxd -p -s -8 -l 4 $1`;
   be=${le:6:2}${le:4:2}${le:2:2}${le:0:2};
   printf "Footer has size $le=%d\n" $((16#$be));

   ./paruqet_size.sh file.parquet

   😄
4. alamb commented on Dec 2, 2022

   

   alamb

   on Dec 2, 2022

   ContributorAuthor

   More actions

   And i can not find a tool to read parquet footer size 😂

   While the bash script is quite compelling from a dependencies point of view, I have been dreaming (though haven't found time) to contribute to @manojkarthick 's https://github.com/manojkarthick/pqrs -- I think with some more contributions that tool could be come "the parquet-tools I actually want to use"

   ❤️React with ❤️1Reacted by Dan Harris
5. Jefffrey

   closed this as a duplicateof Reduce number of object store requests when reading parquet files by default (set metadata_size_hint) #18118 on Nov 27, 2025