Should PyDataFrame.collect() return a Table? #23

Activity

1. matthewmturner commented on Feb 20, 2022

   

   matthewmturner

   on Feb 20, 2022

   Contributor

   More actions

   Asides from repr, do you see any other advantages?
2. houqp commented on Feb 20, 2022

   

   houqp

   on Feb 20, 2022

   Member

   More actions

   This is to keep the signature in sync with what we have in the Rust core. Perhaps it would be better to add a new method to return a pa.Table instead.
3. wjones127 commented on Feb 20, 2022

   

   wjones127

   on Feb 20, 2022

   Author

   More actions

   Asides from repr, do you see any other advantages?

   Mostly was just surprised coming from PyArrow, but it sounds like Rust usually just represents results as a sequence of record batches.

   Perhaps it would be better to add a new method to return a pa.Table instead.

   Yeah perhaps that's a better path. A to_table() method is common in PyArrow. If we eventually get the C Streaming data interface implemented in arrow-rs, we could also provide a to_reader().

   👍React with 👍2Reacted by Matthew Turner and QP Hou