man-group / man-group/ArcticDB
Fine-grained GIL management for modification operations (write/append/update)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 219
- Avg merge
- 5d 13m
- Merged PRs (30d)
- 33
Description
Describe the bug
Hello,
I am running articdb in a child thread (using Python threading module) and I notice that if the child thread invokes arcticdb lib.append(symbol, df, prune_previous_versions=True) method, the child thread will block the main thread from running until the lib.append() method finishes.
I have a while loop in main thread which constantly logs out some information to the console and the logging will be stopped once lib.append() method is executed in the child thread.
I am running arcticdb with AWS S3 and from my testing it often takes a few seconds for the lib.append() method to finish.
Further investigation suggests that the arcticdb code is holding the GIL (Global Interpreter Lock) which prevents any other thread from running.
My question is why did arcticdb need to hold the GIL for so long (a few second)?
Does arcticdb code even hold the GIL while wating for IO?
Steps/Code to Reproduce
N/A
Expected Results
articdb should not hold the GIL for a long time
OS, Python Version and ArcticDB Version
Python: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OS: Linux-5.15.0-76-generic-x86_64-with-glibc2.29
ArcticDB: 1.4.0
Backend storage used
AWS S3
Additional Context
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the lib.append entry point and trace its GIL handling through the AWS S3 I/O path; the issue names no files or tests. Done means a threaded append no longer blocks the main Python thread during long operations, with regression coverage for the reported case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, python
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100