Kotlin / Kotlin/dataframe

GroupBy.sort and sortBy vague error reporting

Open
#780 1 comment 0 reactions 1 assignee Claimed by @Allex-Nik View on GitHub
bug
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Given https://www.kaggle.com/datasets/ruchi798/data-science-job-salaries and
```kt
@DataSchema
interface DsSalaries {
@ColumnName("company_location")
val companyLocation: String
@ColumnName("company_size")
val companySize: String
@ColumnName("employee_residence")
val employeeResidence: String
@ColumnName("employment_type")
val employmentType: String
@ColumnName("experience_level")
val experienceLevel: String
@ColumnName("job_title")
val jobTitle: String
@ColumnName("remote_ratio")
val remoteRatio: Int
val salary: Int
@ColumnName("salary_currency")
val salaryCurrency: String
@ColumnName("salary_in_usd")
val salaryInUsd: Int
val untitled: Int
@ColumnName("work_year")
val workYear: Int
}
```

This doesn't fail
```kt
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortBy(pathOf("group", "salaryInUsd")).print()
```
This does with "can't apply sort flag to column group":
```kt
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortByDesc(pathOf("group", "salaryInUsd")).print()
```

In reality, both snippets are wrong and it should be `sortBy(pathOf("group", "group", "salary_in_usd"))` (which itself is confusing).
These functions should report that pathOf("group", "salaryInUsd") doesn't exist in both cases

======================================================================================

## Problem

`GroupBy.sortBy` and `GroupBy.sortByDesc` handle an invalid nested path inconsistently.

Given a dataframe where `salaryInUsd` is grouped into a column group:

```kotlin
df.group { salaryInUsd }
.into("group")
.groupBy { companyLocation }
```

The following call does not fail, even though the path is incorrect:

```kotlin
.sortBy(pathOf("group", "salaryInUsd"))
```

But the descending version fails with a misleading error:

```kotlin
.sortByDesc(pathOf("group", "salaryInUsd"))
```

Current error:

```text
can't apply sort flag to column group
```

In reality, both calls use a wrong path. The expected path is:

```kotlin
pathOf("group", "group", "salary_in_usd")
```

The current behavior is confusing because:

- `sortBy` silently accepts an invalid path;
- `sortByDesc` fails, but with an error that points to the wrong problem;
- neither function clearly reports that `pathOf("group", "salaryInUsd")` does not exist.

## Expected Behavior

Both `GroupBy.sortBy` and `GroupBy.sortByDesc` should validate the selected path consistently.

If the path does not exist, both functions should report a clear error, for example:

```text
Column path 'group.salaryInUsd' does not exist in the grouped dataframe.
```

The error should preferably include enough context to help users understand the actual available nested path.

## Definition of Done

- Add regression tests for invalid nested paths in `GroupBy.sortBy`.
- Add regression tests for invalid nested paths in `GroupBy.sortByDesc`.
- Ensure both functions fail consistently for the same invalid path.
- Replace the misleading `can't apply sort flag to column group` error with a clearer missing-path diagnostic when appropriate.
- Verify that valid nested paths, such as `pathOf("group", "group", "salary_in_usd")`, still work.
- Avoid changing sorting semantics beyond validation/error reporting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.