GroupBy.sort and sortBy vague error reporting
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Given https://www.kaggle.com/datasets/ruchi798/data-science-job-salaries and
```kt
@DataSchema
interface DsSalaries {
@ColumnName("company_location")
val companyLocation: String
@ColumnName("company_size")
val companySize: String
@ColumnName("employee_residence")
val employeeResidence: String
@ColumnName("employment_type")
val employmentType: String
@ColumnName("experience_level")
val experienceLevel: String
@ColumnName("job_title")
val jobTitle: String
@ColumnName("remote_ratio")
val remoteRatio: Int
val salary: Int
@ColumnName("salary_currency")
val salaryCurrency: String
@ColumnName("salary_in_usd")
val salaryInUsd: Int
val untitled: Int
@ColumnName("work_year")
val workYear: Int
}
```
This doesn't fail
```kt
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortBy(pathOf("group", "salaryInUsd")).print()
```
This does with "can't apply sort flag to column group":
```kt
df.group { salaryInUsd }.into("group").groupBy { companyLocation }.sortByDesc(pathOf("group", "salaryInUsd")).print()
```
In reality, both snippets are wrong and it should be `sortBy(pathOf("group", "group", "salary_in_usd"))` (which itself is confusing).
These functions should report that pathOf("group", "salaryInUsd") doesn't exist in both cases
======================================================================================
## Problem
`GroupBy.sortBy` and `GroupBy.sortByDesc` handle an invalid nested path inconsistently.
Given a dataframe where `salaryInUsd` is grouped into a column group:
```kotlin
df.group { salaryInUsd }
.into("group")
.groupBy { companyLocation }
```
The following call does not fail, even though the path is incorrect:
```kotlin
.sortBy(pathOf("group", "salaryInUsd"))
```
But the descending version fails with a misleading error:
```kotlin
.sortByDesc(pathOf("group", "salaryInUsd"))
```
Current error:
```text
can't apply sort flag to column group
```
In reality, both calls use a wrong path. The expected path is:
```kotlin
pathOf("group", "group", "salary_in_usd")
```
The current behavior is confusing because:
- `sortBy` silently accepts an invalid path;
- `sortByDesc` fails, but with an error that points to the wrong problem;
- neither function clearly reports that `pathOf("group", "salaryInUsd")` does not exist.
## Expected Behavior
Both `GroupBy.sortBy` and `GroupBy.sortByDesc` should validate the selected path consistently.
If the path does not exist, both functions should report a clear error, for example:
```text
Column path 'group.salaryInUsd' does not exist in the grouped dataframe.
```
The error should preferably include enough context to help users understand the actual available nested path.
## Definition of Done
- Add regression tests for invalid nested paths in `GroupBy.sortBy`.
- Add regression tests for invalid nested paths in `GroupBy.sortByDesc`.
- Ensure both functions fail consistently for the same invalid path.
- Replace the misleading `can't apply sort flag to column group` error with a clearer missing-path diagnostic when appropriate.
- Verify that valid nested paths, such as `pathOf("group", "group", "salary_in_usd")`, still work.
- Avoid changing sorting semantics beyond validation/error reporting.
Contributor guide
Assessment
This issue has not been assessed yet.