developmentseed / developmentseed/obstore
Databricks Unity Catalog. store.copy() GenericError: Function not implemented
- Dominant language
- Python
- Stars
- 810
- Forks
- 42
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 6
Description
Hello, I'm seeing unexpected behavior with `LocalStore.copy()` when using a Databricks Unity Catalog Volume.
Environment:
- obstore: 0.11
- Databricks Runtime: 17.3
- Storage: Unity Catalog Volume (`/Volumes/...`)
It works on a local temporary directory:
```
import tempfile
from obstore.store import LocalStore
tmpdir = tempfile.mkdtemp()
store = LocalStore(tmpdir)
store.put("a.txt", b"test")
store.copy("a.txt", "b.txt")
```
But it fails on a Unity Catalog Volume:
```
from obstore.store import LocalStore
store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)
store.copy(
"file.csv",
"copy.csv"
)
Error:
GenericError: Generic LocalFileSystem error:
Unable to copy file from
/Volumes/conf/hydrologie/datasets/file.csv
to
/Volumes/conf/hydrologie/datasets/copy.csv:
Function not implemented (os error 38)
```
Read operations work correctly:
```
from obstore.store import LocalStore
store = LocalStore(
"/Volumes/conf/hydrologie/datasets/"
)
obj = store.get("file.csv")
print(len(obj.bytes()))
```
I also verified that copying the file using Databricks utilities works correctly:
```
dbutils.fs.cp(
"dbfs:/Volumes/conf/hydrologie/datasets/file.csv",
"dbfs:/Volumes/conf/hydrologie/datasets/copy.csv"
)
```
This suggests that:
- Access to the Unity Catalog Volume is working.
- Read and write permissions are available.
- The issue appears to be specific to the implementation used by `LocalStore.copy()` on the `/Volumes` filesystem.
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the LocalStore.copy() entry point and reproduce the difference between a local temporary directory and the Databricks Unity Catalog Volume path shown in the report. Compare the failing copy with the working dbutils.fs.cp behavior; done means copying file.csv to copy.csv succeeds on /Volumes without the GenericError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100