Sqlite issues on rzvernal when using single GPU

There are some issues with sqlite when using GPUs on rzvernal.

Note: The machine by default has 8 visible devices which ends up leading to issues similar to what we saw with the multithreading, but I figure documenting the single GPU case will be a good first step.

I mitigate the multiple device issue by explicitly set CUDA_AVAILABLE_DEVICES=1. From here we end up with 1 device being seen, but I still get some low level issues.

Here is the traceback of the log:

```bash
(no-flux-000) [bgarcia@rzvernal10:test]$ pytest -c pytest-gpu.ini -m includes_gpu -vv tests/
================================================================================== test session starts ===================================================================================
platform linux -- Python 3.9.12, pytest-8.2.0, pluggy-1.5.0 -- /usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.9.12', 'Platform': 'Linux-4.18.0-513.24.1.1toss.t4.x86_64-x86_64-with-glibc2.28', 'Packages': {'pytest': '8.2.0', 'pluggy': '1.5.0'}, 'Plugins': {'xdist': '3.5.0', 'order': '1.2.1', 'metadata': '3.1.1', 'html': '4.1.1', 'anyio': '3.6.2', 'cov': '4.0.0', 'typeguard': '2.13.3', 'jaxtyping': '0.2.14', 'hypothesis': '6.23.1'}}
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/test/.hypothesis/examples')
rootdir: /usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/test
configfile: pytest-gpu.ini
plugins: xdist-3.5.0, order-1.2.1, metadata-3.1.1, html-4.1.1, anyio-3.6.2, cov-4.0.0, typeguard-2.13.3, jaxtyping-0.2.14, hypothesis-6.23.1
collected 103 items / 94 deselected / 9 selected

tests/train/test_autoencoder.py::test_autoencoder_mnist_000 PASSED [ 11%]
tests/train/test_autoencoder.py::test_autoencoder_mnist_001 PASSED [ 22%]
tests/train/test_autoencoder.py::test_autoencoder_mnist_002 PASSED [ 33%]
tests/train/test_autoencoder.py::test_autoencoder_pbcgs_load_000 PASSED [ 44%]
tests/train/test_autoencoder.py::test_autoencoder_pbcgs_000 PASSED [ 55%]
tests/train/test_autoencoder.py::test_autoencoder_pbcgs_001 PASSED [ 66%]
tests/train/test_autoencoder.py::test_autoencoder_pbcgs_002 PASSED [ 77%]
tests/train/test_deltauq.py::test_reproduce_deltauq_figure PASSED [ 88%]
tests/train/test_deltauq.py::test_deltauq_regressor_000 PASSED [100%][rank0]: Traceback (most recent call last):
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[rank0]: self.dialect.do_execute(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[rank0]: cursor.execute(statement, parameters)
[rank0]: sqlite3.OperationalError: no such table: Record

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/bin/pytest", line 8, in
[rank0]: sys.exit(console_main())
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/config/__init__.py", line 206, in console_main
[rank0]: code = main()
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/config/__init__.py", line 178, in main
[rank0]: ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
[rank0]: return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
[rank0]: return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
[rank0]: raise exception.with_traceback(exception.__traceback__)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
[rank0]: res = hook_impl.function(*args)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main
[rank0]: return wrap_session(config, _main)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/main.py", line 320, in wrap_session
[rank0]: config.hook.pytest_sessionfinish(
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
[rank0]: return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
[rank0]: return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
[rank0]: raise exception.with_traceback(exception.__traceback__)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 122, in _multicall
[rank0]: teardown.throw(exception) # type: ignore[union-attr]
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/logging.py", line 872, in pytest_sessionfinish
[rank0]: return (yield)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 122, in _multicall
[rank0]: teardown.throw(exception) # type: ignore[union-attr]
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/terminal.py", line 867, in pytest_sessionfinish
[rank0]: result = yield
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 122, in _multicall
[rank0]: teardown.throw(exception) # type: ignore[union-attr]
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/_pytest/warnings.py", line 140, in pytest_sessionfinish
[rank0]: return (yield)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
[rank0]: res = hook_impl.function(*args)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/test/tests/conftest.py", line 251, in pytest_sessionfinish
[rank0]: coordinator.quick_connect(pytest.store_path)
[rank0]: File "/usr/WS2/bgarcia/repos/nn4n/worktrees/eas-wf/no-flux-000/lib/python3.9/site-packages/nn4n/coordinator/utils.py", line 62, in quick_connect
[rank0]: store = kosh.connect(store, **kargs)
[rank0]: File "/usr/WS1/idesign/idesign/arch/toss_4_x86_64_ib_cray/3.9.12/lib/python3.9/site-packages/kosh/store.py", line 101, in connect
[rank0]: update_store_and_get_info_record(sina_store.records)
[rank0]: File "/usr/WS1/idesign/idesign/arch/toss_4_x86_64_ib_cray/3.9.12/lib/python3.9/site-packages/kosh/utils.py", line 343, in update_store_and_get_info_record
[rank0]: store_info = list(records.find_with_type("__kosh_storeinfo__"))
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sina/datastores/sql.py", line 503, in _do_get_all_of_type
[rank0]: filtered_ids = (str(x[0]) for x in query.all())
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2773, in all
[rank0]: return self._iter().all()
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
[rank0]: result = self.session.execute(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
[rank0]: result = conn._execute_20(statement, params or {}, execution_options)
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
[rank0]: return meth(self, args_10style, kwargs_10style, execution_options)
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
[rank0]: return connection._execute_clauseelement(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
[rank0]: ret = self._execute_context(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
[rank0]: self._handle_dbapi_exception(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
[rank0]: util.raise_(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
[rank0]: raise exception
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[rank0]: self.dialect.do_execute(
[rank0]: File "/usr/WS1/aml_cs/splash/envs/hermit-trainer/toss_4_x86_64_ib_cray/hermit-trainer-rocm6.0/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[rank0]: cursor.execute(statement, parameters)
[rank0]: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: Record
[rank0]: [SQL: SELECT "Record".id AS "Record_id"
[rank0]: FROM "Record"
[rank0]: WHERE "Record".type IN (?)]
[rank0]: [parameters: ('__kosh_storeinfo__',)]
[rank0]: (Background on this error at: https://sqlalche.me/e/14/e3q8)
```

Start by running the issue's pytest command in the single-GPU environment, then trace pytest_sessionfinish in test/tests/conftest.py through coordinator.quick_connect and kosh.connect. Compare the Sina SQL path that queries the Record table. Done means the GPU test run completes without the reported sqlite3.OperationalError.

Written by the indexing model from the issue text.