Repository navigation
Implicit alignment in operations #12
Description
In #2 there seems to be some agreement that row-labels are an important component of a dataframe. Pandas takes this a step further by using them for alignment in many operations involving multiple dataframes.
In [10]: a = pd.DataFrame({"A": [1, 2, 3]}, index=['a', 'b', 'c'])
In [11]: b = pd.DataFrame({"A": [2, 3, 1]}, index=['b', 'c', 'a'])
In [12]: a
Out[12]:
A
a 1
b 2
c 3
In [13]: b
Out[13]:
A
b 2
c 3
a 1
In [14]: a + b
Out[14]:
A
a 2
b 4
c 6In the background there's an implicit a.align(b), which reindexes the dataframes to a common index. The resulting index will be the union of the two indices.
A few other places this occurs
- Indexing a DataFrame / Series with an integer or boolean series
pd.concat- DataFrame constructor
Do we want to adopt this behavior for the standard?
Reacted by Andreas Mueller
Activity
rgommers commentedon Jun 5, 2020
Eh, just to make sure, can you summarize that agreement? As far as I can see you suggested that including row labels was inappropriate, and @devin-petersohn was in favor but also noted that columnar dataframes like Vaex and R tidyverse do not support row labels. Hence my impression was that row labels should be optional or a "level 1" feature.
devin-petersohn commentedon Jun 6, 2020
@rgommers I do not believe that the presence of row labels affects this conversation directly because
a + bcould be reasonably done with the position instead of row labels, and in that case the position is the row's label. Side note: let's figure out this "levels" thing. It is hard to have meaningful conversations without the concrete levels.@TomAugspurger This brings up an interesting discussion about joining/manipulating the row labels (or order) and how that interacts with the data in a dataframe.
If we dissect the
a + boperation, we are effectively doing ajoinalong both axes, and adding (or another binary operation) on label collisions in both axes. This is a bit unusual from the database perspective, but it can be done (though it is tedious).So there is this ability to treat labels as data that can be joined on, or manipulating the order of the rows with an
align-style join. It's a very nice property for visualization.TomAugspurger commentedon Jun 8, 2020
Re-reading #2, it does seem that I overstated the level of support for row labels.
As Devin notes, there's a positional version of label alignment:
So do we expect that operation to raise (different shapes) or align (by position). My recommendation would be to align.
amueller commentedon Jun 8, 2020
This might be a silly questions but I guess we agree/assume that there's alignment for column names, right? There's also a question whether to raise there on misalignment or drop or create nan columns...
TomAugspurger commentedon Jun 8, 2020
TomAugspurger commentedon Jun 9, 2020
For reference, it seems like vaex raises when the lengths of the "column" (expression) don't match
It seems like this doesn't come up for vaex, which AFAICT doesn't implement binary operators (can you confirm that @maartenbreddels?)
Do other dataframe implementors want to weigh in on what's desired / feasible here?
From Dask's perspective, alignment is doable. We partition by divisions on the index. When those divisions aren't available a full shuffle is needed to do the operation.
datapythonista commentedon Jun 19, 2020
Trying to structure a bit the discussion, this is how I see the different components of what is being discussed here (with an example):
Basic case, same size, same index, in the same order (I guess whatever we do, it will work):
Same size and index, but index in different order. With row labels and automatic alignment, what we have is:
Without row labels (or without automatic alignment), I guess we would operate by row id, and rely on sorting for the alignment
df.sort_values('country_id').When the size of the dataframes is different, with automatic alignment, pandas fills with
NAafter aligning, and then operates:Without row labels, I guess the best solution would probably be to fail if the size is different, and rely on a join / reindex to force the user to make the alignment explicitly
df1 + df1.join(on='country_id', how='left').So, correct me if I'm wrong, but I think the decisions that need to be made regarding alignment are:
NArows if the index values don't match?TomAugspurger commentedon Jun 22, 2020
maartenbreddels commentedon Jun 25, 2020
Yes, basically vaex does not have row labels, so both operations do not make sense in the current state. There is a branch which lets the dataframe behave like a 2d array (nep13/nep18), meaning implicit row labels that are row numbers. In the case of a binary operator it will ignore the column labels, and only use the column index, similar to a 2d array.
MarcoGorelli commentedon Dec 4, 2024
In case it's of interest, in Narwhals we solved this by following the left-hand rule for index alignment: https://narwhals-dev.github.io/narwhals/pandas_like_concepts/pandas_index/
An unexpected benefit of this was that, for Plotly, moving to Narwhals ended up solving some existing bugs for free