simonsobs / simonsobs/socs

Error in smurf-sync and timestream-sync dockers

Open
#504 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

agent: suprsync bug
Dominant language
Python
Stars
17
Forks
20
Avg merge
1h 22m
Merged PRs (30d)
2

Description

In both the smurf-sync and timestream-sync dockers on smrfso3 I'm seeing errors that look like this:

2023-08-14T04:12:08+0000 Exception during reset or similar
2023-08-14T04:12:08+0000 Traceback (most recent call last):
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 739, in _finalize_fairy
2023-08-14T04:12:08+0000     fairy._reset(pool)
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 988, in _reset
2023-08-14T04:12:08+0000     pool._dialect.do_rollback(self)
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 682, in do_rollback
2023-08-14T04:12:08+0000     dbapi_connection.rollback()
2023-08-14T04:12:08+0000 sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140261117826816 and this is thread id 140262513887040.

It seems like afterwards it continues to copy new files, but this could be related to why finalization files are not appearing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the smurf-sync and timestream-sync Docker configurations and the sync entry points that use SQLite. Reproduce the SQLAlchemy thread error from the reported traceback, then trace finalization-file handling. Done means the errors are resolved and finalization files are confirmed to appear while new files continue copying.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python, sqlalchemy, sqlite
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.