man-group / man-group/ArcticDB

Fine-grained GIL management for modification operations (write/append/update)

Open
#577 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
2.5k
Forks
219
Avg merge
5d 13m
Merged PRs (30d)
33

Description

Describe the bug

Hello,

I am running articdb in a child thread (using Python threading module) and I notice that if the child thread invokes arcticdb lib.append(symbol, df, prune_previous_versions=True) method, the child thread will block the main thread from running until the lib.append() method finishes.

I have a while loop in main thread which constantly logs out some information to the console and the logging will be stopped once lib.append() method is executed in the child thread.

I am running arcticdb with AWS S3 and from my testing it often takes a few seconds for the lib.append() method to finish.

Further investigation suggests that the arcticdb code is holding the GIL (Global Interpreter Lock) which prevents any other thread from running.
My question is why did arcticdb need to hold the GIL for so long (a few second)?
Does arcticdb code even hold the GIL while wating for IO?

Steps/Code to Reproduce

N/A

Expected Results

articdb should not hold the GIL for a long time

OS, Python Version and ArcticDB Version

Python: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
OS: Linux-5.15.0-76-generic-x86_64-with-glibc2.29
ArcticDB: 1.4.0

Backend storage used

AWS S3

Additional Context

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the lib.append entry point and trace its GIL handling through the AWS S3 I/O path; the issue names no files or tests. Done means a threaded append no longer blocks the main Python thread during long operations, with regression coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, python
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.