apache / apache/texera

Large dataset file downloads are silently truncated (~15s Envoy Gateway route timeout on the MinIO presigned-URL path)

Open
#8,445 0 comments 0 reactions 1 assignee Claimed by @aicam View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### What happened?

Downloading a large file from a dataset through the UI **silently produces a truncated file**. There is no error in the browser, no error toast in Texera, and no error in the file-service logs — the browser reports the download as *completed*, but the resulting file is only a small fraction of the real object.

Observed on a Kubernetes deployment (Helm chart in `bin/k8s`) with a dataset containing multi-GB Zeiss `.czi` microscopy files:

| File | Size reported by `rootFileNodes` | Bytes actually downloaded | % |
|---|---|---|---|
| `A.czi` | 1,416,092,224 | 106,168,320 | 7.5% |
| `B.czi` | 2,424,775,680 | 92,546,841 | 3.8% |
| `-v1.zip` (Download Version) | ~9.39 GB | 51,367,792 | 0.5% |

The downloaded files are structurally valid at the start — the `.czi` still begins with the `ZISRAWFILE` magic bytes — they just end abruptly. The version zip has no central directory (`unzip -t` → "cannot find zipfile directory"), so it cannot be opened at all. Downstream tools (Bio-Formats / Fiji, `unzip`) fail on the results.

Note that the truncation points are **not** a constant byte count and are not round numbers. They correspond to a constant *wall-clock* window of roughly 15 seconds at the throughput each transfer happened to get (~7.1, ~6.2 and ~3.4 MB/s respectively).

**Root cause:** dataset file downloads are served by redirecting the browser to a **presigned S3 URL pointing at the MinIO hostname**, and that hostname is routed through Envoy Gateway by `texera-minio-route` in `bin/k8s/templates/gateway-routes.yaml`:

```yaml
# MinIO Route
{{- if .Values.minio.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: texera-minio-route
...
spec:
...
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: {{ .Release.Name }}-minio
port: 9000
{{- end }}
```

This rule declares **no `timeouts` block**, so Envoy Gateway applies its default HTTP route request timeout of **15s**. Any object that takes longer than 15s to transfer gets cut off mid-stream. Because the presigned `GET` response is streamed and the connection is then closed cleanly, the browser does not treat it as a failed download — it just writes a short file. The result is **silent data corruption** rather than a visible error.

`kubectl -n get httproute texera-minio-route -o yaml` on the live cluster confirms there is no `spec.rules[].timeouts` field.

**Expected:** downloading a dataset file should yield the complete object regardless of size/duration, or fail loudly if it cannot.

**Impact:** effectively every dataset file larger than ~15 seconds of transfer time is silently unusable. For imaging/genomics datasets (the multi-GB case Texera datasets are meant to hold) this means the download feature does not work at all, and users get corrupted data without knowing it.

**Suggested fix:**
1. Set an explicit long (or disabled) timeout on the MinIO `HTTPRoute` in `bin/k8s/templates/gateway-routes.yaml`, ideally configurable via `values.yaml`:
```yaml
timeouts:
request: 0s # disable; Envoy Gateway treats 0s as no timeout
backendRequest: 0s
```
The same should be reviewed for the `file-service` route, which serves the `/{did}/versionZip` streaming endpoint and has the identical problem.
2. Independently of the gateway config, the frontend should verify the downloaded byte count against the known object size and surface an error instead of handing the user a truncated file.

### How to reproduce?

1. Deploy Texera on Kubernetes using the Helm chart in `bin/k8s` with `minio.gateway.enabled=true` (i.e. presigned downloads served through Envoy Gateway).
2. Create a dataset and upload a large file — ~2 GB is enough; anything whose transfer takes more than ~15s will do.
3. Open the dataset page in the UI, select the file, and click the download button.
4. Observe the browser reports the download as complete with no error.
5. Compare the size of the downloaded file with the size shown in the dataset file list — the local file is a small fraction of it.
6. Same for "Download Version" (the zip): the resulting archive is truncated and `unzip -t` reports a missing central directory.

### Version/Branch

1.3.0-incubating-SNAPSHOT (main)

### What browsers are you seeing the problem on?

Chrome

### Relevant log output

No error is logged — that is a core part of the problem. The file-service reports a successful presign, MinIO reports a normal GET, and the browser reports a completed download.

Evidence of the truncation, taken locally after the "successful" download:

$ stat -c%s '.czi'
106168320 # dataset file list reports 1416092224

$ head -c 16 '.czi' | xxd
00000000: 5a49 5352 4157 4649 4c45 0000 0000 0000 ZISRAWFILE......
# valid CZI header - the stream is real data, it just stops early

$ unzip -t '-v1.zip'
cannot find zipfile directory in one of -v1.zip ...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.