Kotlin / Kotlin/dataframe

Add a User Guide for the Parquet file reading

Open
#1,452 0 comments 0 reactions 1 assignee Claimed by @zaleslaw View on GitHub
documentation
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

## Problem

Kotlin DataFrame already has a Parquet documentation page and a realistic example project:

- `docs/StardustDocs/topics/dataSources/Parquet.md`
- `samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt`
- `examples/projects/spark-parquet-dataframe`

However, the current docs are closer to an API reference for `DataFrame.readParquet(...)`. We should add or expand a user-facing guide that explains how to read Parquet data in practical workflows, especially when Parquet files are produced by tools such as Apache Spark.

## Goal

Create a user guide for reading Parquet files with Kotlin DataFrame, using the existing Spark + Parquet example as the main realistic scenario.

The guide should help users understand:

- when Parquet is useful;
- how to set up the required dependency/module;
- how to read one or multiple Parquet files with `DataFrame.readParquet(...)`;
- how to read Spark-generated Parquet output, including `part-*.parquet` files;
- current limitations and platform constraints.

## Suggested Scope

The guide should cover:

1. **Setup**
- Mention that Parquet reading is provided through the `dataframe-arrow` module.
- Clarify that it is included in the general `dataframe` artifact and in `%use dataframe`.

2. **Basic reading examples**
- Read from a file path/string.
- Read from `Path`.
- Read from `File`.
- Read from `URL`.
- Read multiple Parquet files:
```kotlin
val df = DataFrame.readParquet(file1, file2, file3)
```

3. **Spark-produced Parquet**
- Explain that Spark commonly writes a Parquet dataset as a directory containing `part-*.parquet` files.
- Show that Kotlin DataFrame currently reads concrete Parquet files, so users may need to pass the actual part files rather than the directory.
- Reuse the existing example project:
`examples/projects/spark-parquet-dataframe`.

4. **Realistic workflow**
- Spark reads/prepares data.
- Spark writes Parquet.
- Kotlin DataFrame reads the exported Parquet files.
- Kotlin DataFrame explores/transforms/prints the result.

5. **Limitations**
- Kotlin DataFrame currently supports reading Parquet, not writing Parquet.
- Android is not supported because Apache Arrow is not supported there.
- Structured/nested Arrow types such as `Struct` are not supported yet.
- Predicate pushdown / column pruning are not exposed through `readParquet`; users should `select` / `filter` after reading.

## Acceptance Criteria

- The guide references or reuses validated snippets from `samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt`.
- The guide links to the existing Spark example project.
- The guide explains how to handle Spark output directories with `part-*.parquet` files.
- The limitations are clearly documented.
- The page is added to the docs navigation if a new guide page is created.
- Code snippets are validated through the existing docs sample mechanism where possible.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.