deepseek-ai / deepseek-ai/smallpond
Example partition in the readme create 3 files, 1 empty
- Dominant language
- Python
- Stars
- 5k
- Forks
- 452
- PR merge metrics
- No merged PRs in 30d
Description
```sql
SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker
```
from the readme, when run on my m2 pro macbook pro running the latest macos and duckdb 1.2 using this sample code:
```python
import smallpond
# Initialize session
sp = smallpond.init()
# Load data
df = sp.read_parquet("prices.parquet")
# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)
# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())
```
creates 3 files but one is empty and the third has 2 rows instead of the expected 1.
```shell
/tmp/foo/output via π v3.11.11 (af)
β― ls -lrth
total 24
-rw-r--r--@ 1 alex.narayan staff 1.1K Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000007.0.0-0.0.0.parquet
-rw-r--r--@ 1 alex.narayan staff 489B Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000008.0.0-0.0.0.parquet
-rw-r--r--@ 1 alex.narayan staff 1.1K Feb 28 11:29 SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000009.0.0-0.0.0.parquet
```
output:
```shell
/tmp/foo/output via π v3.11.11 (af)
β― duckdb -c "select * from 'SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000008.0.0-0.0.0.parquet'";
βββββββββββ¬βββββββββββββ¬βββββββββββββ
β ticker β min(price) β max(price) β
β varchar β int64 β int64 β
βββββββββββ΄βββββββββββββ΄βββββββββββββ€
β 0 rows β
βββββββββββββββββββββββββββββββββββββ
```
```shell
/tmp/foo/output via π v3.11.11 (af)
β― duckdb -c "select * from 'SqlEngineTask.000003-2e8354b4-0ba2-498c-ab93-d38860160337.000009.0.0-0.0.0.parquet'";
βββββββββββ¬βββββββββββββ¬βββββββββββββ
β ticker β min(price) β max(price) β
β varchar β int64 β int64 β
βββββββββββΌβββββββββββββΌβββββββββββββ€
β APPL β 1 β 3 β
β MSFT β 1 β 3 β
βββββββββββ΄βββββββββββββ΄βββββββββββββ
```
I guess it's expected that there'd be 1 row in each file.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.