apache / apache/iceberg-rust

Partitions with type UUID fail to create transactions

Open
#2,913 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Apache Iceberg Rust version

None

### Describe the bug

When making a table with a UUID partition, using the UUID type, transactions to that table crash.

I'm not sure if this is a case that UUIDs just aren't supported. However the type system suggests I should be able to have a UUID type, and for my needs where I work this would end up being extremely common. As a workaround I'm currently working around this by using a String type instead.

If it helps:
* I have this issue against both version `0.10.0` and `main`.
* I am using the S3 backend.

Btw I am able to work on this issue full time to help get this resolved. I will need some guidance on where, primarily to avoid going off on tangents.

Also thank you guys for this library. It's already been a huge help!

### To Reproduce

I don't have a full reproducible example I'm afraid, as the code I am using is split across a large project.

The steps are:
1. Create a table with a UUID field, which uses the type `Type::Primitive(PrimitiveType::Uuid))`.
2. Add a partition using the UUID field
3. When writing data, use a UUID partition key.
4. Start a transaction.
5. Commit the transaction.

## Pseudo code

The pseudo code for this is along the lines of:
```rust
// 1. Create a table with a UUID field
let table = TableCreation::builder()
.name("ExampleTable".to_string())
.schema(
IcebergSchema::builder()
.with_fields([
// This is a UUID here
NestedField::required(1, "uuid", Type::Primitive(PrimitiveType::Uuid))
.into(),
NestedField::optional(2, "value", Type::Primitive(PrimitiveType::String))
.into(),
])
.build()
.unwrap(),
)
// 2. Add a partition using the UUID field
.partition_spec(
UnboundPartitionSpec::builder()
.with_spec_id(0)
.add_partition_fields([UnboundPartitionField::builder()
.source_id(1)
.field_id(1)
.name("uuid".to_string())
.transform(Transform::Identity)
.build()])?
.build(),
)
.build();

// 3. When writing data, use a UUID partition key.
let uuid_key = Some(Literal::uuid(Uuid::new_v4()));
let data = Struct::from_iter([uuid_key]);
let partition_key = PartitionKey::new(spec, schema, data);

let rolling_builder = todo!(" ... omitted ...");
let data_writer = DataFileWriterBuilder::new(rolling_builder)
.build(Some(partition_key))
.await?;

// 4. Start a transaction.
let data_files = todo!(" ... omitted ...");
let catalog = todo!(" ... omitted ...");
let transaction = Transaction::new(&table);
transaction
.fast_append()
.add_data_files(data_files)
.apply(transaction)?
// 5. Start a transaction.
.commit(catalog.as_ref())
.await?; // <-- it is failing here
```

## Stack trace

Here is part of the stack trace. I have excluded the work code stack trace below, and only included the Iceberg parts:
```
called `Result::unwrap()` on an `Err` value: Iceberg(DataInvalid => Failure in conversion with avro

Source: Could not find matching type in UnionSchema { schemas: [Null, Uuid], variant_index: {Null: 0, Uuid: 1} } for Bytes([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

Backtrace:
0: std::backtrace_rs::backtrace::libunwind::trace
at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
1: std::backtrace_rs::backtrace::trace_unsynchronized::<::create::{closure#0}>
at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
2: ::create
at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/backtrace.rs:331:13
3: ::new::<&str>
at /iceberg-rust/crates/iceberg/src/error.rs:234:24
4: >::from
at ~/iceberg-rust/crates/iceberg/src/error.rs:340:17
5: as core::ops::try_trait::FromResidual>>::from_residual
at ~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2189:27
6: ::write_manifest_file::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/spec/manifest/writer.rs:498:21
7: ::write_added_manifest::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:340:38
8: ::produce_manifests::::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:367:62
9: ::commit::::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:494:14
10: ::commit::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/append.rs:108:14
11: > + core::marker::Send>> as core::future::future::Future>::poll
at ~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:133:9
12: ::do_commit::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:233:75
13: ::commit::{closure#0}::{closure#0}::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:195:48
14: ::commit::{closure#0}::{closure#0}::{closure#0}, ::commit::{closure#0}::{closure#0}, tokio::time::sleep::sleep, ::commit::{closure#0}::{closure#1}, for<'a> fn(&'a iceberg::error::Error, core::time::Duration)> as core::future::future::Future>::poll
at ~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backon-1.6.0/src/retry_with_context.rs:327:58
15: ::commit::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:202:10
16: my_library::iceberg::iceberg_data_writer::IcebergDataWriter>::flush::{closure#0}
at ... this and below is our code ...
```

I would be happy to write a failing test within the Iceberg repo. I'm not 100% sure where the best place would be to recreate this, so some guidance on that would help.

### Expected behavior

This is what I would expect:

* It should not error
* I should get a partition on S3 with a hyphen formatted UUID. i.e. `s3://.../uuid=4bc971ed-7590-4552-8408-d60928ba76f0/...`

### Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/spec/manifest/writer.rs, especially write_manifest_file, and trace the UUID partition value through crates/iceberg/src/transaction/snapshot.rs using the reported transaction steps. Add a focused regression test for a UUID identity partition and transaction commit. Done means the commit succeeds and produces the expected hyphen-formatted UUID partition path.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.