TFRecords File is too big! 10X the size of parquet

Open

#47 opened on Mar 9, 2022 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language: Scala
Stars: 300
Forks: 57
PR merge metrics: No merged PRs in 30d

Description

This how I'm writing a PySpark dataframe to tf-records to an S3 bucket:---

s3_path = "s3://Shuks/dataframe_tf_records"   
df.write.mode("overwrite").format("tfrecord").option("recordType", "Example").save(s3_path)

This creates a new key/"directory" on S3 with the following path : s3://Shuks/dataframe_tf_records/
And under this directory are all the tf-records.

How do I specify compression type during conversion?

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start at the PySpark write entry point using format("tfrecord"), option("recordType", "Example"), and save to S3; trace how these options reach the TFRecord writer. Determine how compression could be selected for this path, then verify that the resulting records are readable and smaller when stored in the referenced S3 location.

Written by the indexing model from the issue text.

Assessment

Tech stack: scala, spark
Domain: data-engineering
Issue type: Feature
Difficulty: 3/5
Estimated time: 1-2 days
Activity status: Stale
Clarity: Mostly clear
Newbie friendliness: 35/100

A short digest of beginner-friendly GitHub issues.