spotify/scio

View on GitHub

A Scala API for Apache Beam and Google Cloud Dataflow.

Stars: 2.6k
Forks: 534
Open beginner issues: 0
Indexed issues: 111
Avg merge: 6d 38m
Merged PRs (30d): 7

Dominant language: Scala
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue

Beginner friendly Everything

Open All

111 open issues indexed

BigQuery IO site page needs updating with magnolify page Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

spotify/scio#5884 · Opened Feb 13, 2026
java.io.FileNotFoundException in NativeFileSorter Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5765 · 1 comment · Opened Sep 12, 2025
Support EmptyMatchTreatment param for SMB reads Open

SMB

Difficulty 3/5 1-2 days Newbie friendliness 42/100

spotify/scio#5759 · 1 comment · Opened Sep 2, 2025
Documentation fixes for scio parquet testing harness Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

spotify/scio#5675 · Opened Apr 24, 2025
Upgrade to gcs-connector 3.x Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

spotify/scio#5673 · Opened Apr 23, 2025
Inconsistent null-value behavior between Caffeine and Guava cache Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

spotify/scio#5659 · Opened Apr 1, 2025
BigQuery writes with GenericRecord format don't support overridden non-String types Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

spotify/scio#5644 · 1 comment · Opened Mar 17, 2025
More bugs with Avro format support in BigQueryType writes Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

spotify/scio#5642 · Opened Mar 11, 2025
Assert SMB Transform test input is not re-iterated Open

enhancement SMB

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

spotify/scio#5603 · Opened Feb 12, 2025
BigQuery: fails with Joda LocalDateTime when using default method Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

spotify/scio#5596 · Opened Feb 4, 2025
ElasticsearchIO and JobTest Integration Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5590 · Opened Jan 30, 2025
Add ErrorHandling in BigQuery Open

gcp io

Difficulty 3/5 1-2 days Newbie friendliness 35/100

spotify/scio#5530 · Opened Nov 27, 2024
Benchmark BigTable maxPendingRequests Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5514 · Opened Oct 16, 2024
Null Pointer Error on Avro to BigQuery load Open

bug gcp io

Difficulty 4/5 3-5 days Newbie friendliness 28/100

spotify/scio#5512 · 2 comments · Opened Oct 9, 2024
Increased memory consumption after upgrade from 13.6 to 14.8 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

spotify/scio#5499 · Opened Sep 19, 2024
Improve data colocation support for Parquet writes Open

parquet

Difficulty 5/5 Over a week Newbie friendliness 30/100

spotify/scio#5466 · Opened Aug 29, 2024
Add Errorhandling to BigTableIO Open

gcp io

Difficulty 4/5 3-5 days Newbie friendliness 45/100

spotify/scio#5440 · 1 comment · Opened Aug 6, 2024
Support for Kafka Open

enhancement io

Difficulty 5/5 Over a week Newbie friendliness 30/100

spotify/scio#5433 · 2 comments · Opened Jul 23, 2024
Benchmark Vectored IO support for scio-parquet/scio-smb Open

blocked parquet

Difficulty 4/5 3-5 days Newbie friendliness 28/100

spotify/scio#5430 · 1 comment · Opened Jul 15, 2024
Label SMB Reads with metadata about SortedBucketIO class Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

spotify/scio#5389 · Opened Jun 7, 2024
Allow largeHash* and sparkey methods to set a byte size target Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

spotify/scio#5355 · Opened May 2, 2024
Improve memory usage in scio-extra's sortValues API Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5338 · Opened Apr 16, 2024
Better projection support for Parquet SMB reads Open

enhancement parquet

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5303 · Opened Mar 14, 2024
Improve error message when sparkey hits array-size limits Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5300 · Opened Mar 13, 2024
Add option for SMB joins to fall back to regular joins Open

enhancement SMB

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5266 · 1 reaction · Opened Feb 21, 2024
Could we add `SchemaUpdateOption` param to BQ WriteParam? Open

enhancement gcp

Difficulty 4/5 3-5 days Newbie friendliness 38/100

spotify/scio#5168 · 1 comment · Opened Jan 15, 2024
Support setting numShards > 1 in SMB transforms Open

enhancement SMB

Difficulty 5/5 Over a week Newbie friendliness 30/100

spotify/scio#5151 · Opened Jan 10, 2024
Using sparkey largeHashIntersectByKey blocks pipeline on Dataflow v1 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5112 · Opened Dec 12, 2023
Document BigTableDoFn Open

documentation gcp

Difficulty 3/5 1-2 days Newbie friendliness 35/100

spotify/scio#5105 · 1 reaction · Opened Dec 4, 2023
Support projections in ParquetAvroFileOperations/ParquetAvroSortedBucketIO Open

io parquet

Difficulty 3/5 1-2 days Newbie friendliness 38/100

spotify/scio#5082 · 1 comment · 1 reaction · Opened Nov 17, 2023
new StateBaseAsyncDoFn with State and Timers Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

spotify/scio#5055 · 6 comments · Opened Nov 8, 2023
ParquetTypeSortedBucketIO writes and reads don't work with CaseMapper Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#5017 · 1 comment · Opened Sep 27, 2023
Enhance saveAsCustomOutput and provide access to POutput Open

enhancement io

Difficulty 4/5 3-5 days Newbie friendliness 45/100

spotify/scio#4995 · 4 comments · Opened Sep 9, 2023
Improve file metadata lookup in Parquet SDF Open

enhancement parquet

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#4979 · Opened Sep 5, 2023
Beam JDBC partitioning Open

enhancement io

Difficulty 4/5 3-5 days Newbie friendliness 30/100

spotify/scio#4955 · Opened Aug 17, 2023
Capture "job graph is too large" and provide suggestions Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

spotify/scio#4939 · Opened Jul 28, 2023
The suffix parameter doesn't seem to work when reading using parquetAvroFile Open

io

spotify/scio#4936 · 1 assignee · Opened Jul 25, 2023
Add option/env var indicating job should be cancelled on interrupt Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#4922 · 1 comment · Opened Jul 13, 2023
Support PythonExternalTransform with conveniences for standard model types Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

spotify/scio#4908 · Opened Jul 6, 2023
Beam Resource Hints API support Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#4897 · Opened Jun 22, 2023
Add dead-letter queue support Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

spotify/scio#4881 · Opened Jun 14, 2023
Evaluate moving to ApproximateCountDistinct Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

spotify/scio#4872 · Opened Jun 9, 2023
Add async lookup variant for JDBC Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

spotify/scio#4852 · Opened Jun 3, 2023
Add typed variant of JdbcSelect Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

spotify/scio#4851 · Opened Jun 3, 2023
Provide standard redisLookup convenience method Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

spotify/scio#4849 · Opened Jun 2, 2023
RedisDoFn should use `BaseAsyncDoFn` Open

Difficulty 3/5 1-2 days Newbie friendliness 30/100

spotify/scio#4848 · Opened Jun 2, 2023
Issue with encoding Map with complex types Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

spotify/scio#4812 · Opened May 18, 2023
Bigtable ensureTables* calls remove existing GC rules if gcRules param is not passed Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

spotify/scio#4797 · Opened May 4, 2023
Label ParquetIO Read Transforms with `filePattern` Open

enhancement parquet

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

spotify/scio#4794 · 1 comment · Opened May 3, 2023
Re-implement SMB read transforms using SplittableDoFn? Open

blocked enhancement SMB

Difficulty 5/5 Over a week Newbie friendliness 28/100

spotify/scio#4793 · Opened May 3, 2023

spotify/scio

Get new issues in your inbox