apache / apache/texera

Optimized Python silently truncates materialization inputs

Open
#8,203 1 comment 0 reactions 1 assignee Claimed by @carloea2 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Input materialization validates URI and partitioning counts with an assert. Python removes asserts in optimized mode, then zip silently truncates mismatched lists and creates fewer readers than requested.

Before: mismatched configuration -> zip truncates -> materialized input is silently dropped

After: mismatched configuration -> ValueError identifies the invalid configuration

Reproduction evidence:

1. Run the Python worker with optimization enabled.
2. Configure an input port with two storage URIs and one partitioning.
3. Set up the materialization readers.
4. Observe that only one reader is created and no error is raised.

A direct run on main produced:

```text
optimized= True
uris= 2
partitionings= 1
readers_created= 1
error_raised= False
```

Version and commit evidence:

1.3.0-incubating-SNAPSHOT (main)

**Commit Hash**

70c21145887920528d7d5540e3fb790b43e8b759

**What browsers are you seeing the problem on?**

Not browser-specific.

**Relevant log output**

None.

### Proposed Solution or Design

After: mismatched configuration -> ValueError identifies the invalid configuration

### Affected Area

Workflow Engine (Amber)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.