APIs for both building pipelines and data analysis

From my own experience, there are (at least) two very different use cases to use dataframes:

Doing some real-time data analysis in notebooks
Building production pipelines

While I think pandas (and later Vaex, Dask, Modin...) did a very reasonable job at building a single tool that solves both use cases. There are trade offs, that IMO will bias any API towards one or the other.

Some specific examples:

Eager/lazy modes: In case 1 (data analysis) eager mode is probably preferable, while in case 2 (pipelines), lazy mode has more advantages
Automatic type inference/casting: I see the advantage of having all sorts of magic on inferring types and guessing when the user is able to to check the results of the operations at every step. But when instead of executing cell by cell, there is a big pipeline that is executed as a batch process, I see this problematic. I think it's worth having to be explicit and avoid any magic. It helps prevent bugs, and the errors are not propagated to later stages in the pipeline, making them difficult to identify.

I wrote a post about this that describes this point of view in more detail.

I think it can help the discussions to keep in mind that there are at least two main use cases, and that there will be trade offs among them.

Feedback here very welcome.

@datapythonista

This is a great point @datapythonista, and type casting behaviour is probably the best example of where you want different semantics for the same functionality.

Eager vs lazy really is a separate topic I think that's independent of API (it's about performance, not semantics). Also for developing production pipelines, you want to develop using eager - it's simple the more intuitive way to develop code. And then you want to flip a switch to get lazy behaviour for possible performance optimizations in production. This is exactly what the PyTorch/Tensorflow experience showed (Tensorflow 2 switched to eager by default, and both libraries have a graph compiled mode for production use cases). It's completely analogous for dataframes, except libraries are a little less mature. At this point I'd suggest keeping eager/lazy out of this topic. @teoliphant volunteered to write up a summary of eager/lazy as a standalone topic.

@maartenbreddels

I mostly agree, but I think the API may be affected. With an example:

import sql

query = sql.connect('foo.db')

query = query.select('*')
query = query.from('table1')
query.execute()

This is obviously a lazy API. In eager mode the line query = query.select('*') is meaningless. So, if you want to make this eager, the API needs to be changed.

So, IMO, a decision needs to be made before defining an API among:

Lazy
Eager
Hybrid (the same syntax can be executed both in lazy and eager modes)
Mixed (some things are executed in eager mode, some in lazy mode, see @maartenbreddels comment below)

May be we're already assuming hybrid is the way to go. But I don't think it's the only option (not saying is not the best).

I would like to add that vaex sits somewhere in between:

import vaex
df = vaex.example()
# lazy element wise operations
df['y'] = df.x**2  # just the expression gets added
display(df['y'])  # will compute/show only the the first and last 5 rows
# lazy filtering
dff = df[df['x'] > 1]  # no explicit compute() needed
# half lazy join
df.join(df, 'x', rsuffix='r', allow_duplication=True)  # eagerly calculates the lookup, lazily fetches the values

So in usage, it seems eager, but a lot is done lazily behind the scene.

I think my point is, that even if you have an eager acting system, it can do a lot of things lazily, and it could build computational graphs/DAG's/plans behind the scene to automatically generate pipelines for instance (like what we do in Vaex).

@maartenbreddels

Thanks @maartenbreddels, that's a very good point. I clarified my comment above to note that.

My main point is that there may be API implications on the decision made regarding laziness. If we totally forget about it, and decide once the API is finished, there will be limitations. May be they are not important enough to not postpone the discussion, and keep things simpler for now. But I don't think the user API and eagerness are fully independent topics.

@maartenbreddels

My personal opinion here is that adding explicit compute or execute leaks execution details to the user. Separating the execution details from the API is going to be important because systems may want to control when computation is actually triggered, like in the case @maartenbreddels showed with Vaex.

I think we can get away without needing some explicit computation trigger word, lazy systems can just trigger computation when an API calls for the dataframe to be converted to an external format (e.g. array, csv file, __repr__, etc.). This is how laziness is handled in Modin (preliminarily).

In Ibis automatic lazy execution is configurable:

https://github.com/ibis-project/ibis/blob/master/ibis/expr/types.py#L25

If you __repr__ an expression object in a Jupyter notebook, for example, then it executes automatically (rather than having to type expr.execute()). I think it's important to support to support both modes since you don't want your library firing off expensive queries to BigQuery by accident.

Interesting to hear Devin & Wes. I wasn't totally sure we could get away with it, but if 3 libraries are doing this already (Vaex, Modin and Ibis), I guess it is demonstrated to be possible.

Then I guess the next question is: Should lazy execution be something we should consider explicitly in the API design? Or can this be an implementation detail of the libraries?

This is one of the central questions of this whole project: does the API expose execution semantics or not? The pandas API in general exposes eager execution semantics. This is a significant decision with many implications. My design work with Ibis was an exploration of a data frame library without exposing any eager execution semantics.

In R people use both kinds of APIs, ones that expose execution semantics (base R, data.table) and ones that don't (dplyr).

I don't necessarily agree that the pandas API itself is what exposes eager execution semantics, instead it is the implementation. It is not too difficult to intercept API calls under __getattribute__ and build a DAG (this is what Modin does). Doing a print(df.expensive().head()), users don't care if the entire expensive is fully executed before they see the top 5 lines, especially in EDA/data cleaning.

I don't think we should support an explicit execute or compute or persist in the API. It leaks too much to the user. I would prefer to have full control over when things are executed, not rely on the user to give their best guess about when there is enough information. Ideally users should not have to think about execution, it should just work.

It is not too difficult to intercept API calls under getattribute and build a DAG (this is what Modin does).

IMHO, just because you can do this does not mean that you should. If the barrier to entry of implementing the API is a significant about of metaprogramming (regardless of whether there is a example implementation of it or not), that does not strike me necessarily as a good thing.

@devin-petersohn

I don't think we should support an explicit execute or compute or persist in the API. It leaks too much to the user. I would prefer to have full control over when things are executed, not rely on the user to give their best guess about when there is enough information. Ideally users should not have to think about execution, it should just work.

@devin-petersohn What about the pipeline execution? Don't you think that having the API that allows more granular control, or even execution scheduling of various portions of DAG is beneficial - should the DAG be exposed as a work item, like, e.g. in Airflow?

@wesm

IMHO, just because you can do this does not mean that you should. If the barrier to entry of implementing the API is a significant about of metaprogramming (regardless of whether there is a example implementation of it or not), that does not strike me necessarily as a good thing.

@wesm I think you misunderstand the point I was trying to make. The __getattribute__ approach I mention is just one way to handle building the DAG, you can also handle this on an API to API basis (as @maartenbreddels mentioned he does with Vaex) or implement each API to add itself to the DAG or be completely eager. My point was the pandas API itself does not imply eager.

@devin-petersohn What about the pipeline execution? Don't you think that having the API that allows more granular control, or even execution scheduling of various portions of DAG is beneficial - should the DAG be exposed as a work item, like, e.g. in Airflow?

@aregm I am imagining each dataframe system as a black box, similar to SQL systems. Exposing the actual DAG also comes with a new set of requirements and would require some additional API standards. I expect there is another can of worms involved in exposing this level of execution detail as well because some systems may not want to do it at all. I'm interested to hear your thoughts.

My point was the pandas API itself does not imply eager.

I don't know about this. By the same logic, no library has eager evaluation so long as API calls can be intercepted and translated into an AST.

Consider this pandas code

df[col][df[col].isnull()] = 5
result = df.groupby(col)[col].sum()

Yes, you can transform this into:

df <- Mutate(
  df,
  AssignWhere(
    ColumnRef(df, col),
    IsNull(ColumnRef(df, col))
    Literal(5)
  )
)             

result <- GroupBy(
  df, ColumnRef(df, col),
  Aggregate(ColumnRef(df, col)))

But was that the intention of the library creators? The execution semantics (especially when dealing with very large data sets) influence a great deal how people write and think about the code.

just because you can do this does not mean that you should

I agree those two are separate questions. The can question is answered with yes.

I don't think we should support an explicit execute or compute or persist in the API.

For a data science-user, I agree. Aas a programmer/engineer-user I am not sure which is the right way.

@datapythonista

Yes, the "can" vs. "should" distinction is a good one :)

So I think we're back to Ralph's original point that eager vs. lazy is primarily about performance, rather than API? Does anyone object to that?

@datapythonista you noted in #5 (comment)

My main point is that there may be API implications on the decision made regarding laziness. If we totally forget about it, and decide once the API is finished, there will be limitations.

Can we delve into that a bit? In my experience, this is primarily an issue when we have value dependent behavior, i.e. the output dtype / shape depends on the values in the dataframe, rather than just the input dtype / shape. We've been removing this type of behavior in pandas wherever possible, but in some cases it's unavoidable (e.g. pivot(df, index="a", columns="b") will for a DataFrame whose columns are the values in b). In practice, this just means that lazy systems will need to evaluate the values in b before proceeding with the rest of the computation.

I think that if we agree that eager vs. lazy is (primarily) about performance then we can

Rule out a .compute / .persist API as part of the standard
Try to avoid APIs that aren't sensible in either a lazy or eager system

@rgommers

@rgommers there's plenty of iterative algorithms in scikit-learn in pure numpy. As I said above, if we don't allow iterative algorithms, we basically say scikit-learn is not in scope. So there is no adoption of the standard by scikit-learn [in the sense that there's basically nothing of sklearn left that could adopt the standard].

@amueller

@amueller I don't believe those will be a problem, but let's make sure. Can you point to a concrete algorithm you have in mind, and we'll assess if there's a potential issue with it given this eager/lazy discussion?

Let's say NMF with multiplicative updates:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/decomposition/_nmf.py#L710

I guess what I don't really understand is whether computation of the stopping criterion will be lazy or not.

Another maybe more simple example would be gaussian mixture models (thought the fit is actually in a parent class):
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/mixture/_gaussian_mixture.py#L434

We also have a pure numpy neural net (not super relevant in practice but maybe a good example as well):
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/_multilayer_perceptron.py#L625

Thanks.

I guess what I don't really understand is whether computation of the stopping criterion will be lazy or not.

It should not be - the lazy implementations normally know when they need to evaluate, and that then happens without any change to the API. (previous_error - error) / error_at_init < tol cannot be kept lazy for something like Dask.

For dask you need to explicitly add a sync point from what I remember, but that might have changed?

From my perspective, iterative algorithms are much more natural in arrays than in dataframes (though they are used). An API for going from dataframe API standard -> array API standard might solve this issue.

to_array_standard() would always trigger computation, we can define this in the spec. Of course this could be abused as well so some thought has to go into not incentivizing

type(df)(df.to_array_standard())

to manually trigger computation in purely lazy systems.

Part of the reason to let libraries control execution is that optimizing for each individual implementation can end up removing the benefit of the standard API. This is my main concern with exposing these execution semantics.

@devin-petersohn

Part of the reason to let libraries control execution is that optimizing for each individual implementation can end up removing the benefit of the standard API. This is my main concern with exposing these execution semantics.

@devin-petersohn totally agreed! I'm just trying to see how this would work in practice. We all come from very different directions and so I think it's important to get on the same page. I think the lazy vs eager is probably one of core issues in defining this API.

I read your suggestion as saying that arrays would be always eager while dataframes would not be.
That indeed would probably solve my concerns for sklearn. However, there are several lazy array libraries that probably wouldn't like that.

It should not be - the lazy implementations normally know when they need to evaluate, and that then happens without any change to the API.

Yeah, I agree.

So that every method that must return a concrete Python type, would trigger computation.

So for [this line](https://github.com/scikit-learn/scikit-learn/blob/a1cc566605e61bcac5cd9d9f676ca430b5f90a1d/sklearn/decomposition/_nmf.py#L831:

            if (previous_error - error) / error_at_init < tol:

It would end up calling __bool__ on the value emit by the __lt__, which is the point at which it would force it to not be lazy anymore and would force computation.

For dask you need to explicitly add a sync point from what I remember, but that might have changed?

My assumption is then that Dask array would not fulfil the API as is, because it doesn't force computation, AFAIK, when using it in Python control flow. However, I imagine that there could be a compatible wrapper object written on top of the Dask array that does fulfil it.

So for [this line](scikit-learn/scikit-learn:sklearn/decomposition/_nmf.py@a1cc566#L831:

            if (previous_error - error) / error_at_init < tol:

It would end up calling bool on the value emit by the lt, which is the point at which it would force it to not be lazy anymore and would force computation.

I'm not sure I agree that it would force the values to be materialized. It can be much cheaper to know __lt__ than __eq__ and since the implementation has control over both, as long as they return booleans, materialization of any part of the result is not required or guaranteed. Some computation would be required, but materialization is not a prerequisite to either from my perspective.

Perhaps the problem here is that in performance critical code, you don't want to both evaluate some estimation for really fast __lt__ evaluation and evaluate a value later when you need it. Is this the problem or do I misunderstand?

Is this the problem or do I misunderstand?

I agree that this you don't need to necessarily materialize the full computation, just to get access to a single check like this. But I think that's all a backend implementation detail and not anything this spec should concern itself with.

IMHO all that sklearn know is that if x is some array that follow the sec, then code likeif (x.mean() < 0): ... will work. So to make this work, any lazy backend will need to compute enough to return the right return value for this control flow to proceed in Python.

Yes, it means that that call won't be lazy itself, but that's fine, because it's out of scope of this proposal to try to convert Python control flow to a lazy form.

We could possibly try to add lazy versions of control flow, so instead it could be written in a form that lazy backends could take advantage of, like (x.mean() < 0).if_(..., ...), but IMHO this is probably out of scope for at least the first draft here.

EDIT: So to clarify, I think all this spec needs to be clear about is the return values of the different methods and functions, and specify that things like __iter__ and __bool__ should be present to fulfil the spec. Then, sklearn can build it's API around this spec and know it can use Python control flow. If something like Dask Arrays wants to fulfil the spec, they can as long as they properly implement those methods. ~~This would be different then the current behavior of Dask Array, or Ibis, at the moment, because they usually don't execute until you tell them too, not implicitly AFAIK.~~ See comment below for for how they currently.

Yes, it means that that call won't be lazy itself, but that's fine, because it's out of scope of this proposal to try to convert Python control flow to a lazy form.

Agreed.

This would be different then the current behavior of Dask Array, or Ibis, at the moment, because they usually don't execute until you tell them too, not implicitly AFAIK.

Just to clarify, dask.array & dataframe will implicitly compute things when requested, where "requested" means something like a __bool__, __iter__, __array__

In [1]: import numpy as np

In [2]: import dask.array as da

In [3]: a = da.ones((4, 4))

In [5]: a.mean() > 0  #lazy
Out[5]: dask.array<gt, shape=(), dtype=bool, chunksize=(), chunktype=numpy.ndarray>

In [6]: if a.mean() > 0:  # executes (x.mean() > 0).__bool__()
   ...:     print("hi")
   ...:
hi

In [7]: np.asarray(a)
Out[7]:
array([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]])

I'm not sure if it's worth looking at, but dask-glm does have some examples of distributed optimization algorithms. They aren't really in use today though. I would say that we haven't really figured out a good model here yet.

https://blog.dask.org/2018/08/07/incremental-saga and https://blog.dask.org/2017/04/19/dask-glm-2 have some discussions as well.

Just to clarify, dask.array & dataframe will implicitly compute things when requested

Ah, thank you for clarifying this! I was incorrect above then, I updated my comment.

As long as the framework / contract handles triggering the computation in these places then I'm happy :) It might leave a couple of sklearn related issues but they are probably not broadly important.

Navigation Menu

Repository navigation

APIs for both building pipelines and data analysis #5

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

APIs for both building pipelines and data analysis #5

Description

Activity

rgommers commented on May 18, 2020

datapythonista commented on May 18, 2020

maartenbreddels commented on May 18, 2020

datapythonista commented on May 18, 2020

devin-petersohn commented on May 18, 2020

wesm commented on May 18, 2020

maartenbreddels commented on May 18, 2020

wesm commented on May 18, 2020

devin-petersohn commented on May 18, 2020

wesm commented on May 18, 2020

aregm commented on May 18, 2020

devin-petersohn commented on May 18, 2020

wesm commented on May 18, 2020

maartenbreddels commented on May 18, 2020

TomAugspurger commented on May 18, 2020

15 remaining items

amueller commented on May 22, 2020

rgommers commented on May 22, 2020

amueller commented on May 22, 2020

rgommers commented on May 22, 2020

amueller commented on May 22, 2020

devin-petersohn commented on May 22, 2020

amueller commented on May 22, 2020

saulshanabrook commented on May 22, 2020

devin-petersohn commented on May 22, 2020

saulshanabrook commented on May 22, 2020

TomAugspurger commented on May 22, 2020

saulshanabrook commented on May 22, 2020

amueller commented on May 27, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions