Repository navigation
Study on the pandas API: What is the most commonly used? #3
Description
I have spent a lot of time trying to understand users and their behaviors in order to optimize for them. As a part of this work, I have done numerous studies on what gets used in pandas.
This will be extremely useful when it comes to defining a dataframe standard, because what people are using can help inform us on what behaviors to support.
For this study, we scraped the top 6000 notebooks from Kaggle by upvote.
Repo here, reproduction script included: https://github.com/modin-project/study_kaggle_usage
Results here: results.csv
Reacted by Ralf Gommers, Andreas Mueller, Ruifeng Zheng and lipi
Activity
datapythonista commentedon May 15, 2020
This is really cool, thanks a lot for sharing!
amueller commentedon May 18, 2020
This is awesome! I'm somewhat surprised at the common use of
valuestbh.Probably obvious but maybe worth mentioning: kaggle is a very biased source. Basically by design everything is read from CSV files, for example, and it's unlikely anyone reads sql. Also, the workloads are quite specific. Still really cool!
datapythonista commentedon May 18, 2020
I use
valuesto "export" pandas data to numpy to train scikit-learn models. Not sure if that's the reason, but it doesn't surprise me. I guess it's asread_csv, biased in that kaggle users will be converting data to numpy to use in scikit-learn.amueller commentedon May 18, 2020
At some point a pandas dev told me to use
np.arrayinstead ofvalues, but I guess that's not very common. Also, you don't need to do any conversion for using sklearn ;)datapythonista commentedon May 18, 2020
I agree with both points (I think they meant
DataFrame.array, which is somehow recent). But I think that's still the reason why the notebooks use.valuesfrequently.devin-petersohn commentedon May 18, 2020
We have done the same thing over 1 million
allpublic GitHub repos and gists, published here: https://arxiv.org/pdf/2001.00888.pdf in Section 3.6. I thought it better to share some raw data we have collected rather than the summary in that paper. The GitHub scripts still need some cleaning up to be made public.I do not think the bias of Kaggle diminishes the value of this data (not that you are implying this).
TomAugspurger commentedon May 28, 2020
Here are the top 40ish pandas methods by pageviews on pandas' docs
I view the number of page views as some
function(usefulness, complexity), hence read_csv with its 50+ parameters at the top :)tdimitri commentedon Jun 13, 2020
I love this list. Below are some decisions we made in riptide to work with existing pandas users while trying to eliminate duplicate methods or sometimes too much being put into the same method (like sort and merge).
na: fill_na and drop_na. We broke out fill_na to fill_forward, fill_backward
sort: we did sort_copy, sort_inplace, sort_view <-- new one that pandas does not have
dtypes/describe/stats: all related to information about the columns
apply: (big topic!) apply_reduce, apply_nonreduce, apply_numba (last one not yet implemented)
note: apply_reduce has a transform=True (and this got rid of .transform)
columns: for all column operations we started all methods with col_. Thus 'col_rename' and 'col_map' are the most used.
join/merge: (another big topic!) this got broken out into merge_enrich, and a few more merge_ methods. concat went to 'stack_rows' we started using stack_rows as a generic API to stack anything.
set_index/reset_index: we eliminated these (this will end up being a big discussion topic and is related to row labels -- i think?)
drop_duplicates: this is really 'first', 'last' and 'nth' and is related to groups or groupings (which is going back to categoricals)
groupby: this goes back to groups which goes back to categoricals
.loc/[,:], etc: These are related to row and column indexing which we reduced into always using [,] syntax
pivot_table: we have this also (was a fight as I tried to eliminate it) -- we often use another API to do what pivot does (specifically set_index, followed by pivot can be done in a new method).
filter: this needs more discussion -- we have filter= kwarg in many methods
value_counts: this became .count() everywhere
to/from: from_dict -- surprised to see this on the list. not surprisingly we have to_pandas, from_pandas
save/load: not on the list but related
kgryte commentedon Dec 10, 2020
To add to this discussion, we've done some analysis of downstream library usage of pandas APIs, which can be found here.