google-deepmind / google-deepmind/gemma

Issue: Fix/parquet multipath support

Open
#533 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Description :
This PR updates the ParquetDataSource class in: gemma/gm/data/_parquet.py to correctly handle a list of file paths. Previously, passing a list of paths would cause the initialization/reading to fail or behave unexpectedly as it was not iterating through the paths. The new implementation iterates through provided paths, reads each table individually, and concatenates them using pyarrow.concat_tables.

## Changes:
- Modified ParquetDataSource.table to normalize self.path to a list.
- Iterate through each path in the list, open the file, and read the parquet table.
- Concatenate all read tables into a single PyArrow table.

## Verification:
Verified manually using a reproduction script (mocked) demonstrating that read_table is now called for each path in the list and concat_tables is utilized.

## #524 PR Raised for the issue, lmk if there any any iterations . Love contributing to Google-Deepmind

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.