Add a User Guide for the Parquet file reading
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
## Problem
Kotlin DataFrame already has a Parquet documentation page and a realistic example project:
- `docs/StardustDocs/topics/dataSources/Parquet.md`
- `samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt`
- `examples/projects/spark-parquet-dataframe`
However, the current docs are closer to an API reference for `DataFrame.readParquet(...)`. We should add or expand a user-facing guide that explains how to read Parquet data in practical workflows, especially when Parquet files are produced by tools such as Apache Spark.
## Goal
Create a user guide for reading Parquet files with Kotlin DataFrame, using the existing Spark + Parquet example as the main realistic scenario.
The guide should help users understand:
- when Parquet is useful;
- how to set up the required dependency/module;
- how to read one or multiple Parquet files with `DataFrame.readParquet(...)`;
- how to read Spark-generated Parquet output, including `part-*.parquet` files;
- current limitations and platform constraints.
## Suggested Scope
The guide should cover:
1. **Setup**
- Mention that Parquet reading is provided through the `dataframe-arrow` module.
- Clarify that it is included in the general `dataframe` artifact and in `%use dataframe`.
2. **Basic reading examples**
- Read from a file path/string.
- Read from `Path`.
- Read from `File`.
- Read from `URL`.
- Read multiple Parquet files:
```kotlin
val df = DataFrame.readParquet(file1, file2, file3)
```
3. **Spark-produced Parquet**
- Explain that Spark commonly writes a Parquet dataset as a directory containing `part-*.parquet` files.
- Show that Kotlin DataFrame currently reads concrete Parquet files, so users may need to pass the actual part files rather than the directory.
- Reuse the existing example project:
`examples/projects/spark-parquet-dataframe`.
4. **Realistic workflow**
- Spark reads/prepares data.
- Spark writes Parquet.
- Kotlin DataFrame reads the exported Parquet files.
- Kotlin DataFrame explores/transforms/prints the result.
5. **Limitations**
- Kotlin DataFrame currently supports reading Parquet, not writing Parquet.
- Android is not supported because Apache Arrow is not supported there.
- Structured/nested Arrow types such as `Struct` are not supported yet.
- Predicate pushdown / column pruning are not exposed through `readParquet`; users should `select` / `filter` after reading.
## Acceptance Criteria
- The guide references or reuses validated snippets from `samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt`.
- The guide links to the existing Spark example project.
- The guide explains how to handle Spark output directories with `part-*.parquet` files.
- The limitations are clearly documented.
- The page is added to the docs navigation if a new guide page is created.
- Code snippets are validated through the existing docs sample mechanism where possible.
Contributor guide
Assessment
This issue has not been assessed yet.