developmentseed / developmentseed/obstore

Databricks Unity Catalog. store.copy() GenericError: Function not implemented

Open
#748 2 comments 0 reactions 0 assignees View on GitHub
upstream
Dominant language
Python
Stars
810
Forks
42
Avg merge
1d 15h
Merged PRs (30d)
6

Description

Hello, I'm seeing unexpected behavior with `LocalStore.copy()` when using a Databricks Unity Catalog Volume.

Environment:
- obstore: 0.11
- Databricks Runtime: 17.3
- Storage: Unity Catalog Volume (`/Volumes/...`)

It works on a local temporary directory:

```
import tempfile
from obstore.store import LocalStore

tmpdir = tempfile.mkdtemp()
store = LocalStore(tmpdir)

store.put("a.txt", b"test")
store.copy("a.txt", "b.txt")
```

But it fails on a Unity Catalog Volume:

```
from obstore.store import LocalStore

store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)

store.copy(
"file.csv",
"copy.csv"
)

Error:

GenericError: Generic LocalFileSystem error:
Unable to copy file from
/Volumes/conf/hydrologie/datasets/file.csv
to
/Volumes/conf/hydrologie/datasets/copy.csv:
Function not implemented (os error 38)
```

Read operations work correctly:

```
from obstore.store import LocalStore

store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)

obj = store.get("file.csv")
print(len(obj.bytes()))
```

I also verified that copying the file using Databricks utilities works correctly:

```
dbutils.fs.cp(
"dbfs:/Volumes/conf/hydrologie/datasets/file.csv",
"dbfs:/Volumes/conf/hydrologie/datasets/copy.csv"
)
```

This suggests that:
- Access to the Unity Catalog Volume is working.
- Read and write permissions are available.
- The issue appears to be specific to the implementation used by `LocalStore.copy()` on the `/Volumes` filesystem.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the LocalStore.copy() entry point and reproduce the difference between a local temporary directory and the Databricks Unity Catalog Volume path shown in the report. Compare the failing copy with the working dbutils.fs.cp behavior; done means copying file.csv to copy.csv succeeds on /Volumes without the GenericError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.