connectors: Delta Sink - produciton feasibility improvements
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
As mentioned a number of time on Discord, for the last few weeks I was working on trying to make Delta sink run on a real EC2 instance writing to S3. The implementation turned out to have a substantial amount of issues that render the sink not practical to use in production:
1. S3 storage type mandates access key and secret key, use of which is highly discouraged by AWS. The modern practice is to assign a role to an instance that is going to perform the write, for which you should simply write into the destination bucket without any credentials.
2. The existence of the destination delta table is not checked, which causes obscure errors when the table doesn't exist in the destination S3 URL. The table is not automatically created either, but this is likely to be a future feature.
3. When set in production, `aws_s3_endpoint_url` is confusing S3 API to understand the region setting which doesn't allow the code to open the table for writing. This setting should be documented better (and studied as there seem to be a couple of other quirks) as removing it usually fixes the problem.
4. If a connectors runtime instantiated with a config containing topics that do not exist yet, and then tries to connect to the iggy-server, it stalls indefinitely and the sink is not writing anymore, which seems to be an offset issue, but the cause of this is yet to be studied in more detail.
5. Current implementation allows to have a setup when no fields from the JSONs inside of the topic and the fields (columns) of the table schema overlap, causing the system to write a table full of nulls. This needs to be studied further.
6. The defaults for `batch_length` and `poll_interval` have to be adjusted for Delta tables. Writing the data with the standard `batch_length == 100` and `poll_interval == 5ms` is going to create lots and lots of small files in Delta, which is causing very slow reading on the side of Delta readers. The default of `batch_length == 10000` and `poll_interval == 3s` is much more adequate for most use cases as it's going to create appropriately-sized files.
7. The underlying IggyError ID is not propagated for the user to understand what is the problem which leads to cryptic error messages: `iggy-connect | 2026-09-10T14:17:06.835367Z ERROR iggy_connectors::sink: Failed to initialize sink connector with ID: 1: "Failed to set up sink consumers: Iggy error". Skipping...`
All of these will likely be bundled in a number of different PRs but the grouping is still TBD.
### Affected area / component
Connectors
### Proposed solution
Items are corresponding to the enumeration in the issue description.
PR 1:
1. Stop mandating the S3 access and secret keys. Mention the two access options in the docs. Suggest the sufficient policy configuration to be given to the instance.
2. Check the table existence after opening, write a test for it.
3. Study and document the usage of `aws_s3_endpoint_url` configuration setting.
6. Change defaults in the documentation. One possibility could also be to have a cutoff by bytes as the recommended size of an individual file for a Delta table is 128 - 1024 MB, but this is a larger feature that needs to be thought through way more.
7. Make the error message more elaborate in case of IggyError returned from the client.
Future PRs (exact split is TBD)
4. The stalling after failure to find a stream / topic by the connectors runtime has to be studied and most likely there needs to be an error with a retry period or something like this.
5. Study the schema coercion process further to see the exact nature of the problem. I think it would make sense to implement an error on no fields overlapping as this results in all-null records made into the delta table.
### Alternatives considered
_No response_
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.