deepseek-ai / deepseek-ai/smallpond

Example partition in the readme create 3 files, 1 empty

Open
#6 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5k
Forks
452
PR merge metrics
No merged PRs in 30d

Description

```sql
SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker
```
from the readme, when run on my m2 pro macbook pro running the latest macos and duckdb 1.2 using this sample code:

```python
import smallpond

# Initialize session
sp = smallpond.init()

# Load data
df = sp.read_parquet("prices.parquet")

# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)

# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())
```

creates 3 files but one is empty and the third has 2 rows instead of the expected 1.

```shell
/tmp/foo/output via 🐍 v3.11.11 (af)
❯ ls -lrth
total 24
-rw-r--r--@ 1 alex.narayan staff 1.1K Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000007.0.0-0.0.0.parquet
-rw-r--r--@ 1 alex.narayan staff 489B Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000008.0.0-0.0.0.parquet
-rw-r--r--@ 1 alex.narayan staff 1.1K Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000009.0.0-0.0.0.parquet
```

output:

```shell
/tmp/foo/output via 🐍 v3.11.11 (af)
❯ duckdb -c "select * from 'SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000008.0.0-0.0.0.parquet'";
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ticker β”‚ min(price) β”‚ max(price) β”‚
β”‚ varchar β”‚ int64 β”‚ int64 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ 0 rows β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

```shell
/tmp/foo/output via 🐍 v3.11.11 (af)
❯ duckdb -c "select * from 'SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000009.0.0-0.0.0.parquet'";
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ticker β”‚ min(price) β”‚ max(price) β”‚
β”‚ varchar β”‚ int64 β”‚ int64 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ APPL β”‚ 1 β”‚ 3 β”‚
β”‚ MSFT β”‚ 1 β”‚ 3 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

I guess it's expected that there'd be 1 row in each file.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.