apache / apache/opendal

services/gcs: XML multipart writer cannot honor write_with_if_not_exists

Open
#8,040 1 comment 1 reaction 0 assignees View on GitHub
bug enhancement services/gcs
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

## Summary

GCS supports `write_with_if_not_exists` for single-shot uploads via `ifGenerationMatch=0`, but OpenDAL's multipart writer path uses the XML API multipart upload APIs, which do not support request preconditions.

This was exposed by `test_writer_write_with_if_not_exists` after #8028 added chunked-writer precondition coverage. Against GCS the multipart write with `if_not_exists(true)` succeeds and overwrites an existing object instead of returning `ConditionNotMatch`.

## Evidence from GCS docs

From [Request preconditions](https://docs.cloud.google.com/storage/docs/request-preconditions):

> Preconditions cannot be used in XML API multipart uploads. Attempting to do so results in a `400 NotImplemented` error.

`If-None-Match` is also not a substitute here: GCS documents it for retrieve/GET/HEAD style requests, and the Complete Multipart Upload request headers do not list precondition headers.

## Current OpenDAL behavior

- Single-shot `write_with(...).if_not_exists(true)` works on GCS.
- Capability currently reports `write_with_if_not_exists: true` and `write_can_multi: true`.
- Chunked `writer_with(...).if_not_exists(true)` therefore looks supported, but the multipart path cannot enforce the condition.

## Possible directions

1. Keep skipping `test_writer_write_with_if_not_exists` on GCS until there is a supported upload path.
2. Move GCS multi-write onto an API that can enforce `ifGenerationMatch=0` (for example resumable upload), if that can preserve current writer semantics.
3. Narrow the advertised capability so multipart conditional writes are not implied when only single-shot conditional writes are possible.

## References

- Failing main Behavior Test: https://github.com/apache/opendal/actions/runs/31186355721
- Related conditional-write tracking: #7889
- Azblob multipart precondition fix that introduced the coverage: #8028

Contributor guide

Open the contributing guide

Research direction

Start with test_writer_write_with_if_not_exists and the GCS XML multipart writer and capability implementation; compare the behavior added in #8028 with GCS request-precondition documentation. Determine whether to skip the test, switch to a supported upload path, or narrow the capability, then verify that writes to existing objects return ConditionNotMatch without overwriting.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.