cockroachdb / cockroachdb/cockroach

roachtest: run import tests on AWS with S3 fixtures

Open
#164,462 0 comments 0 reactions 0 assignees View on GitHub
A-import C-enhancement O-postmortem P-2 T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

[Technical advisory a161317](https://www.cockroachlabs.com/docs/advisories/a161317) (silent data loss in AVRO OCF import, [#161317](https://github.com/cockroachdb/cockroach/issues/161317)) identified a gap in storage backend test coverage. The existing import roachtests (`pkg/cmd/roachtest/tests/import.go`) only use GCS fixtures (`gs://cockroach-fixtures-us-east1/tpch-csv/`), so the S3 storage path is never exercised end-to-end by import tests.

We should get the import roachtests running on the existing AWS roachtest runs. This requires:
- Uploading the TPC-H fixtures (and AVRO fixtures from #164461, if available) to an S3 bucket
- Teaching the import roachtest how to discover the correct fixture URLs based on which cloud it's running in (GCS vs S3)

Jira issue: CRDB-60824

Epic CRDB-62881

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.