Dictionary GroupBy renaming aggregation can't insert group names to the frame (`as_index=False`) in case of aggregation against 'by' columns

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Any
• Modin version (modin.__version__): f1f3aabd0ecd75086dfe7fb70be6a040958e0045
• Python version: 3.7.5
• Code we can use to reproduce:

import modin.pandas as pd
import pandas

pd_df = pandas.DataFrame({"a": [1, 1, 2, 2], "b": [3, 3, 5, 5], "c": [3, 4, 5, 6]})
md_df = pd.DataFrame(pd_df)

pd_res = pd_df.groupby(["a", "b"], as_index=False).agg(max=("a", max))
md_res = md_df.groupby(["a", "b"], as_index=False).agg(max=("a", max))

print(f"pandas:\n{pd_res}\n")
print(f"modin:\n{md_res}\n")

Output:

pandas:
   a  b  max
0  1  3    1
1  2  5    2

modin:
   b  max
0  3    1
1  5    2

Since #3592 it starts printing a warning in advance:

UserWarning: `GroupBy.aggregate(**dictionary_renaming_aggregation)` implementation has mismatches with pandas:
intersection of the columns to aggregate and 'by' is not yet supported when 'as_index=False', columns with group names of the intersection will not be presented in the result. To achieve the desired result rewrite the original code from:
df.groupby('by_column', as_index=False).agg(agg_func=('by_column', agg_func))
to the:
df.groupby('by_column').agg(agg_func=('by_column', agg_func)).reset_index().
pandas:
   a  b  max
0  1  3    1
1  2  5    2

modin:
   b  max
0  3    1
1  5    2

Describe the problem

There is almost a similar issue (#3376) related to the non-renaming aggregation. Although the issues look identical the root causes are different.

The flow of processing renaming aggregation is the following:

1. Firstly, it converts renaming aggregation dict into a non-renaming one: https://github.com/modin-project/modin/blob/7a8158873e77cb5f1a5a3b89be4ddac89f576269/modin/pandas/groupby.py#L450-L453 In the case of related reproducer the non-renaming aggregation is:

(Pdb) func_dict
{'a': ['max']}

2. Second step is to process groupby aggregation with non-renaming dict as usual, and this part is causing us trouble. When processing this aggregation as is with as_index=False parameter, we got a naming conflict of 'by' and aggregated columns:

(pdb) result # result with `as_index` parameter unhandled
      a
    max
a b
1 3   1
2 5   2
(pdb) handle_as_index(result) # conflicting name 'a' was dropped from index levels before insertion
   b   a
     max
0  3   1
1  5   2

And it's the place where we're losing the 'a' column with the group names.

It seems that there is no way of solving this unless moving the handling of as_index parameter to the front-end, where we know, that this particular naming conflict will be resolved later after final relabeling.

Start in modin/pandas/groupby.py around the renaming-aggregation conversion at lines 450-453 and the final relabeling at lines 487-503. Reproduce the case with groupby(["a", "b"], as_index=False).agg(max=("a", max)) and inspect how handle_as_index drops the conflicting group column. Done means the result includes both group columns, matching pandas, without the documented workaround.