apache / apache/opendal

Tracking Issue: Complete object_store Integration Implementation

Open
#6,446 2 comments 3 reactions 0 assignees View on GitHub
enhancement integrations/object_store releases-note/feat rust
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

## Overview

This issue tracks the remaining work needed to fully implement the `object_store` trait in OpenDAL's integration layer. While the core functionality is working, several methods return `NotSupported` errors and some implementations could be enhanced.

## Current Status

### ✅ Implemented
- `put_opts` with full PutMode support (Overwrite, Create, Update)
- `put_multipart` (basic implementation)
- `get_opts` with GetOptions support (range, conditionals, version)
- `delete`
- `list`, `list_with_offset`, `list_with_delimiter`
- `MultipartUpload` trait
- Error mapping between OpenDAL and object_store
- Amazon S3 builder support

### ❌ Not Implemented

The following methods currently return `NotSupported` errors:

1. **`put_multipart_opts`** ([store.rs:165](integrations/object_store/src/store.rs:165))
- Required for advanced multipart upload options
- Currently returns: "put_multipart_opts is not implemented so far"

2. **`copy`** ([store.rs:248](integrations/object_store/src/store.rs:248))
- Server-side copy operation
- Currently returns: "copy is not implemented so far"

3. **`rename`** ([store.rs:255](integrations/object_store/src/store.rs:255))
- Server-side rename operation
- Currently returns: "rename is not implemented so far"

4. **`copy_if_not_exists`** ([store.rs:262](integrations/object_store/src/store.rs:262))
- Conditional server-side copy
- Currently returns: "copy_if_not_exists is not implemented so far"

### ⚠️ Incomplete Implementations

1. **PutResult metadata**
- Always returns `e_tag: None, version: None`
- Should return actual ETag and version when available from backend

2. **GetResult attributes**
- Always returns `Default::default()` for attributes
- Should include object metadata/attributes from backend

3. **Test coverage**
- Missing behavior tests for list operations
- No tests for `list_with_delimiter` and `list_with_offset`

## Tasks

### High Priority
- [ ] Implement `copy` method using OpenDAL's copy operation
- [ ] Implement `rename` method using OpenDAL's rename operation
- [ ] Return proper ETag in `PutResult` when available
- [ ] Return proper version in `PutResult` when backend supports versioning

### Medium Priority
- [ ] Implement `put_multipart_opts` with proper options handling
- [ ] Implement `copy_if_not_exists` with conditional logic
- [ ] Populate `GetResult.attributes` with object metadata

### Low Priority
- [ ] Add comprehensive tests for list operations
- [ ] Add integration tests with different storage backends
- [ ] Document any limitations compared to native object_store implementations

## Implementation Notes

1. **Copy/Rename Operations**: Check if OpenDAL's backend supports native copy/rename. If not, consider fallback to client-side implementation with appropriate warnings.

2. **ETag/Version Support**: Different backends have varying support for ETags and versioning. Implementation should gracefully handle backends that don't support these features.

3. **Attributes Mapping**: Need to define how OpenDAL's metadata maps to object_store's attributes.

## References

- [object_store documentation](https://docs.rs/object_store/latest/object_store/)
- [OpenDAL object_store integration](integrations/object_store/)

## Acceptance Criteria

- All object_store trait methods return meaningful results instead of `NotSupported`
- Metadata (ETag, version, attributes) is properly populated when available
- Comprehensive test coverage for all implemented methods
- Documentation clearly states any limitations or backend-specific behavior

Contributor guide

Open the contributing guide

Research direction

Start in integrations/object_store/src/store.rs, focusing on put_multipart_opts, copy, rename, copy_if_not_exists, and the PutResult and GetResult conversions. Review the object_store trait requirements alongside OpenDAL's supported operations and existing integration behavior. Done means unsupported methods are addressed where possible, metadata is mapped when available, and the listed list and backend integration tests cover the behavior and limitations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.