basalt-26.3.2-rc-20260914: storage/fs: TestBasaltMetrics failed

storage/fs.TestBasaltMetrics [failed](https://mesolite.cluster.engflow.com/invocations/default/d9a3408e-1577-4a9c-bff1-dd3df430d2df?testReportRun=2&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvc3RvcmFnZS9mczpmc190ZXN0) on basalt-26.3.2-rc-20260914 @ [d3eb08ebdd1dacc8d3dac0b107a7db5929e09969](https://github.com/cockroachdb/cockroach/commits/d3eb08ebdd1dacc8d3dac0b107a7db5929e09969):

```
I260916 09:46:28.166290 2130 (gostd) log.go:56 [-] 369 INFO rpc component=rpc method=Mkdir tenant=test-tenant dir=7ca9aa20-2d6b-43ce-a7fa-30385c7840a9 name=store-1 dur=9.568496ms code=OK
I260916 09:46:28.166779 1996 (gostd) logger.go:37 [-] 370 INFO basaltfs: opened dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 zone=test-zone-1
I260916 09:46:28.172060 2134 (gostd) log.go:56 [-] 371 INFO rpc component=rpc method=Lock tenant=test-tenant object=fa837d12-9d17-4f51-ba09-4731a68d67f1 dur=4.757602ms code=OK
I260916 09:46:28.172439 1996 (gostd) logger.go:37 [-] 372 INFO basaltfs: locked dir=fa837d12-9d17-4f51-ba09-4731a68d67f1
I260916 09:46:28.206299 2135 (gostd) log.go:56 [-] 373 INFO rpc component=rpc method=List tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 dur=33.418121ms code=OK
I260916 09:46:28.210055 2136 (gostd) log.go:46 [-] 374 INFO rpc component=rpc method=StatByPath tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 name=STORAGE_MIN_VERSION dur=2.957281ms code=NotFound err="rpc error: code = NotFound desc = entry \"STORAGE_MIN_VERSION\" not found"
I260916 09:46:28.212698 2137 (gostd) log.go:56 [-] 375 INFO rpc component=rpc method=List tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 dur=1.941001ms code=OK
I260916 09:46:28.218300 2138 (gostd) log.go:56 [-] 376 INFO rpc component=rpc method=SetSetting tenant=test-tenant key=tenant_usage.interval value=10ms restore_default=false dur=4.709232ms code=OK
I260916 09:46:28.235372 2140 (gostd) log.go:56 [-] 377 INFO rpc component=rpc method=Create tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 name=metrics-test-file dur=11.678708ms code=OK
I260916 09:46:28.268539 2154 (gostd) seal_direct.go:721 [-] 378 INFO CopyFrom to materialize direct upload replica component=seal object=5ec94f09-d7b9-4aaf-be62-6e19f045f6cf disk=1 size=29 dur=12.032037ms
I260916 09:46:28.268736 2153 (gostd) seal_direct.go:721 [-] 379 INFO CopyFrom to materialize direct upload replica component=seal object=5ec94f09-d7b9-4aaf-be62-6e19f045f6cf disk=2 size=29 dur=12.523747ms
I260916 09:46:28.288076 2147 (gostd) log.go:56 [-] 380 INFO rpc component=rpc method=Seal tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 name=metrics-test-file size=29 dur=50.659751ms code=OK
I260916 09:46:28.398250 2193 (gostd) log.go:56 [-] 381 INFO rpc component=rpc method=Mkdir tenant=test-tenant dir=7ca9aa20-2d6b-43ce-a7fa-30385c7840a9 name=store-2 dur=4.653133ms code=OK
I260916 09:46:28.398683 1996 (gostd) logger.go:37 [-] 382 INFO basaltfs: opened dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb zone=test-zone-1
I260916 09:46:28.402420 2197 (gostd) log.go:56 [-] 383 INFO rpc component=rpc method=Lock tenant=test-tenant object=e0815fcf-b5ab-4103-8e56-49a37e168ebb dur=3.250792ms code=OK
I260916 09:46:28.402791 1996 (gostd) logger.go:37 [-] 384 INFO basaltfs: locked dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb
I260916 09:46:28.410286 2198 (gostd) log.go:56 [-] 385 INFO rpc component=rpc method=List tenant=test-tenant dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb dur=7.074645ms code=OK
I260916 09:46:28.415742 2199 (gostd) log.go:46 [-] 386 INFO rpc component=rpc method=StatByPath tenant=test-tenant dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb name=STORAGE_MIN_VERSION dur=4.736963ms code=NotFound err="rpc error: code = NotFound desc = entry \"STORAGE_MIN_VERSION\" not found"
I260916 09:46:28.418334 2200 (gostd) log.go:56 [-] 387 INFO rpc component=rpc method=List tenant=test-tenant dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb dur=1.927191ms code=OK
I260916 09:46:28.426528 2201 (gostd) log.go:56 [-] 388 INFO rpc component=rpc method=Create tenant=test-tenant dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb name=metrics-test-file-2 dur=7.476605ms code=OK
I260916 09:46:28.441505 2211 (gostd) seal_direct.go:721 [-] 389 INFO CopyFrom to materialize direct upload replica component=seal object=9544ab93-8479-4e18-99e4-be3c826ba105 disk=1 size=29 dur=2.670421ms
I260916 09:46:28.441828 2210 (gostd) seal_direct.go:721 [-] 390 INFO CopyFrom to materialize direct upload replica component=seal object=9544ab93-8479-4e18-99e4-be3c826ba105 disk=2 size=29 dur=3.127533ms
I260916 09:46:28.452888 2208 (gostd) log.go:56 [-] 391 INFO rpc component=rpc method=Seal tenant=test-tenant dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb name=metrics-test-file-2 size=29 dur=24.492456ms code=OK
I260916 09:46:28.461137 1996 (gostd) logger.go:37 [-] 392 INFO basaltfs: unlocked dir=e0815fcf-b5ab-4103-8e56-49a37e168ebb
I260916 09:46:28.461434 1996 (gostd) logger.go:37 [-] 393 INFO basaltfs: closed
I260916 09:46:28.478612 2220 (gostd) log.go:56 [-] 394 INFO rpc component=rpc method=Create tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 name=metrics-test-file-archived dur=11.676557ms code=OK
I260916 09:46:28.499093 2225 (gostd) seal_direct.go:721 [-] 395 INFO CopyFrom to materialize direct upload replica component=seal object=a5a08ba4-9d94-4c2a-99f6-2f8b4d39e2c7 disk=1 size=29 dur=2.952532ms
I260916 09:46:28.499462 2226 (gostd) seal_direct.go:721 [-] 396 INFO CopyFrom to materialize direct upload replica component=seal object=a5a08ba4-9d94-4c2a-99f6-2f8b4d39e2c7 disk=2 size=29 dur=3.192042ms
I260916 09:46:28.512446 2223 (gostd) log.go:56 [-] 397 INFO rpc component=rpc method=Seal tenant=test-tenant dir=fa837d12-9d17-4f51-ba09-4731a68d67f1 name=metrics-test-file-archived size=29 dur=31.99053ms code=OK
basalt_test.go:1414:
Error Trace: pkg/storage/fs/basalt_test.go:1414
Error: Not equal:
expected: fs.archiveMetrics{staleUnsealedEntries:0, uploads:1, uploadRetries:0, uploadResigns:0, uploadFailures:0, bytesUploaded:0, finishPut:1, finishPartUpload:0, seal:1, putResponseWait:1, completeMultipartUpload:0, adoptHead:1, materialization:1}
actual : fs.archiveMetrics{staleUnsealedEntries:0, uploads:1, uploadRetries:0, uploadResigns:0, uploadFailures:0, bytesUploaded:0, finishPut:1, finishPartUpload:0, seal:1, putResponseWait:0, completeMultipartUpload:0, adoptHead:1, materialization:1}

Diff:
--- Expected
+++ Actual
@@ -10,3 +10,3 @@
seal: (int64) 1,
- putResponseWait: (int64) 1,
+ putResponseWait: (int64) 0,
completeMultipartUpload: (int64) 0,
Test: TestBasaltMetrics
I260916 09:46:28.521198 1996 (gostd) logger.go:37 [-] 398 INFO basaltfs: unlocked dir=fa837d12-9d17-4f51-ba09-4731a68d67f1
I260916 09:46:28.521865 1996 (gostd) logger.go:37 [-] 399 INFO basaltfs: closed
I260916 09:46:28.522047 1996 (gostd) logger.go:37 [-] 400 INFO basaltfs: closed
I260916 09:46:28.524257 2066 (gostd) tenant_usage.go:313 [-] 401 ERROR unable to query tenant usage aggregates component=tenant_usage err="context canceled"
I260916 09:46:28.524478 2066 (gostd) tenant_usage.go:313 [-] 402 ERROR unable to query tenant usage aggregates component=tenant_usage err="context canceled"
--- FAIL: TestBasaltMetrics (9.80s)
```

Parameters:
- attempt=1
- race=true
- run=2
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #175307 storage/fs: TestBasaltMetrics failed [A-storage C-test-failure O-robot P-1 T-storage branch-master]

/cc @cockroachlabs/storage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestBasaltMetrics.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Start with pkg/storage/fs/basalt_test.go at line 1414 and run TestBasaltMetrics with the reported race=true settings. Compare the expected and actual archiveMetrics values, focusing on putResponseWait, and use the failure log to investigate the differing count. Done means the test passes reliably with consistent metric behavior.

Written by the indexing model from the issue text.