You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think the answer here is that the API is fully compatible with implementations that use lazy evaluation, and that it should not have specific syntax. We should probably document that
Support for lazy evaluation
We have to design all APIs to support lazy evaluation, and I think so far there's nothing in the draft API that's incompatible with that. It's basically the same as in the array API, quoting from here: The same applies to software environments: it must be possible to create an array library adhering to this standard that runs efficiently independent of what compilers, build-time or run-time execution environment, or distribution and install method is employed. Parallel execution, JIT compilation, and delayed (lazy) evaluation must all be possible.
As a design rule, the syntax and semantics of the API must be independent of execution model. Which by definition rules out something like .collect(). Execution of code written in the API backed by a lazy implementation can probably be triggered by (a) library-specific implementation details where the implementation must graph break for some reasons (full Python branching like on if-else not supported for example), (b) interop with another library, like when __dataframe__ is called, and (c) the user explicitly asking for an execution step with .collect() or similar.
Syntax variations between libraries for lazy execution
This is very non-uniform and would be hard to standardize anyway. E.g.:
Dask is always lazy and has a .compute() method to trigger execution
Vaex offers both eager and lazy execution (see here), uses .execute() to trigger and delay=True or @delayed to choose the lazy mode over the eager one
Polars also offers both eager and lazy (see here), and uses .collect() to trigger execution and .lazy() or lazy-specific I/O functions like pl.scan_csv to choose lazy mode.
How this should work
importa_dataframe_lib# Create a dataframe and lazy mode, may be library-specific I/O or apply explicit syntax like @delayeddf=a_dataframe_lib.xxx# Use standard API
....
df_out= ...
# Now we have, for example, `df_out` as the result of use of standard-compliant API# This can now be materialized by something like:df_out.compute() # Dask-specific
plot the data, like ax.plot(df.get_column_by_name('x').to_array(), df.get_column_by_name('y').to_array())
By the time seaborn gets to the last step, the data would need to be materialised. But if we're decide that to_array should return something which can be iterated over, then perhaps the call to .collect can be taken care of inside to_array and can be seen as an implementation detail
Agreed, depending on the semantics of the to_array call and what guarantees it provides, if materialized data is required then the framework implementing the API would be responsible for handling that in its to_array implementation.
re-opening as scikit-learn have mentioned that they would like to collect separately from converting to ndarray
Can we have a specific use case here? Even a single code snippet would help. I don't think it's good practice to trigger .collect() explicitly in some random place in another library. Lazy computations should stay lazy as long as possible.
In fact, Dask devs have told me that their users sometimes accidentally trigger compute twice, because some of their methods do that under-the-hood for them.
I'm against anything other than an explicit .collect (or compute) triggering computation. If collect can't make its way into the standard, then OK, so be it, lazy computations will stay lazy and eager computations will stay eager
Activity
rgommers commentedon Mar 30, 2023
I think the answer here is that the API is fully compatible with implementations that use lazy evaluation, and that it should not have specific syntax. We should probably document that
Support for lazy evaluation
We have to design all APIs to support lazy evaluation, and I think so far there's nothing in the draft API that's incompatible with that. It's basically the same as in the array API, quoting from here: The same applies to software environments: it must be possible to create an array library adhering to this standard that runs efficiently independent of what compilers, build-time or run-time execution environment, or distribution and install method is employed. Parallel execution, JIT compilation, and delayed (lazy) evaluation must all be possible.
As a design rule, the syntax and semantics of the API must be independent of execution model. Which by definition rules out something like
.collect(). Execution of code written in the API backed by a lazy implementation can probably be triggered by (a) library-specific implementation details where the implementation must graph break for some reasons (full Python branching like on if-else not supported for example), (b) interop with another library, like when__dataframe__is called, and (c) the user explicitly asking for an execution step with.collect()or similar.Syntax variations between libraries for lazy execution
This is very non-uniform and would be hard to standardize anyway. E.g.:
.compute()method to trigger execution.execute()to trigger anddelay=Trueor@delayedto choose the lazy mode over the eager one.collect()to trigger execution and.lazy()or lazy-specific I/O functions likepl.scan_csvto choose lazy mode.How this should work
MarcoGorelli commentedon Mar 30, 2023
OK - thinking of seaborn as an example, it might:
df(which could be lazy or not)df.groupby([col]).mean()ax.plot(df.get_column_by_name('x').to_array(), df.get_column_by_name('y').to_array())By the time seaborn gets to the last step, the data would need to be materialised. But if we're decide that
to_arrayshould return something which can be iterated over, then perhaps the call to.collectcan be taken care of insideto_arrayand can be seen as an implementation detailrgommers commentedon Mar 30, 2023
That sounds about right to me.
kkraus14 commentedon Mar 31, 2023
Agreed, depending on the semantics of the
to_arraycall and what guarantees it provides, if materialized data is required then the framework implementing the API would be responsible for handling that in itsto_arrayimplementation.MarcoGorelli commentedon Jun 20, 2023
thanks all, closing then
MarcoGorelli commentedon Aug 2, 2023
re-opening as scikit-learn have mentioned that they would like to collect separately from converting to ndarray
rgommers commentedon Aug 2, 2023
Can we have a specific use case here? Even a single code snippet would help. I don't think it's good practice to trigger
.collect()explicitly in some random place in another library. Lazy computations should stay lazy as long as possible.MarcoGorelli commentedon Aug 2, 2023
Thanks, let's discuss tomorrow. I'm realising that there's far, far more to the topic of lazy dataframes, I'll add it to the agenda
MarcoGorelli commentedon Aug 21, 2023
closing as this discussion has effectively moved to #224
MarcoGorelli commentedon Aug 29, 2023
Revisiting this, I think it's sufficiently different from #224 that it needs to stay open
I no longer stand by this, this would potentially result in double-computation, e.g.:
In fact, Dask devs have told me that their users sometimes accidentally trigger compute twice, because some of their methods do that under-the-hood for them.
I'm against anything other than an explicit
.collect(orcompute) triggering computation. Ifcollectcan't make its way into the standard, then OK, so be it, lazy computations will stay lazy and eager computations will stay eagerMarcoGorelli commentedon Oct 20, 2023
I'll just add this to the pandas/polars implementations, and then users can do
if they need to force materialisation
Other suggestions are welcome - I tried with #249 but there wasn't support