apache / apache/beam

[Feature Request]: Implement 'DataFrame.loc.setitem' for deferred DataFrame

Open
#22,267 2 comments 0 reactions 0 assignees View on GitHub
dataframe dsl new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

We would like to use the DataFrame API to perform operations on certain columns.

As an example:

```
# Initialize pipeline
p = beam.Pipeline(InteractiveRunner())

# Create a deferred Beam DataFrame
beam_df = p | beam.dataframe.io.read_csv('test.csv', splittable=True)

# Perform operation on specific columns
columns = ['col_1','col_2']
beam_df.loc[:,columns] = beam_df.loc[:, columns] - beam_df.loc[:, columns].mean()
```

This is currently not supported since the `loc.setitem()` method is not implemented yet .

### Issue Priority

Priority: 2

### Issue Component

Component: dsl-dataframe

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.