dbt-labs / dbt-labs/dbt-common

[Bug] dbt Core 1.9 - Intermittent error "OSError: [Errno 116] Stale file handle" during parsing

Open
#360 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
40
Forks
42
PR merge metrics
No merged PRs in 30d

Description

### Is this a new bug in dbt-core?

- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

We are experiencing intermittent OSError: [Errno 116] Stale file handle errors when running dbt commands inside a Google Cloud Run Job. The error occurs during the dbt parsing phase, specifically when dbt is trying to read the project's source files.

Our setup involves running dbt from the official [dbt-labs/dbt-bigquery:1.9.0](https://github.com/dbt-labs/dbt-bigquery/pkgs/container/dbt-bigquery/319602272?tag=1.9.0) public Docker image. The dbt project source code is stored in a Google Cloud Storage (GCS) bucket, which is mounted as a volume into the Cloud Run instance for each execution.

Crucially, we are already isolating our artifacts by providing a unique `--target-path` and `--log-path` for every single dbt execution, so the issue is not related to concurrent writes to the target directory.

### Expected Behavior

dbt should reliably parse all project files from the GCS mounted volume without encountering a filesystem error. The process should be robust to high-concurrency environments, especially given that mutable artifacts (`target/`, `logs/`) are already isolated per run.

### Steps To Reproduce

The environment is key to reproducing this issue:

Store a dbt project's source files in a Google Cloud Storage bucket.

Configure a Google Cloud Run Job to use the [dbt-labs/dbt-bigquery:1.9.0](https://github.com/dbt-labs/dbt-bigquery/pkgs/container/dbt-bigquery/319602272?tag=1.9.0) image.

In the Cloud Run Job configuration, mount the GCS bucket containing the dbt project as a volume.

Set the job to execute a dbt command, such as `dbt run/test/build/compile --select my_model`. Ensure the command uses a unique `--target-path` for this specific execution.

Trigger multiple jobs to run with high concurrency.

Intermittently, one of the jobs will fail during startup with the [Errno 116] Stale file handle traceback.

### Relevant log output

```shell
11:02:24 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [, , ]}
11:02:24 Running with dbt=1.9.0
11:02:24 running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'fail_fast': 'False', 'log_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'debug': 'False', 'profiles_dir': '/mnt/gcs/dbt', 'use_colors': 'False', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'None', 'log_format': 'json', 'static_parser': 'True', 'invocation_command': 'dbt --log-format-file=text --log-format=json --warn-error-options {"include": ["NothingToDo"]} --no-use-colors test --select source:my_first_source source:my_second_source --vars {"AIRFLOW_DAG": "my_example_dag"} --target-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64 --log-path /mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'warn_error_options': "WarnErrorOptions(include=['NothingToDo'], exclude=[])", 'target_path': '/mnt/gcs/target/dbt-run-b9454326-3a59-4f8a-804c-66482fc28a64', 'introspect': 'True', 'send_anonymous_usage_stats': 'True'}
11:02:27 Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': []}
11:02:27 Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': []}
11:02:27 Registered adapter: bigquery=1.9.0
11:02:28 checksum: b46e93efb52f67a19bdb6e7abf8f24e437648676c7bd1d5db6ccdaa72cc84c4c, vars: {'AIRFLOW_DAG': 'my_example_dag'}, profile: , target: , version: 1.9.0
11:02:28 Unable to do partial parsing because saved manifest not found. Starting full parse.
11:02:28 Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': 'ec4be0c9-09e6-46a7-934a-2b5de26cdda6', 'context': []}
11:02:35 Encountered an error:
[Errno 116] Stale file handle
11:02:35 Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 153, in wrapper
result, success = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 235, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 264, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 311, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 327, in wrapper
setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
File "/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py", line 351, in setup_manifest
ctx.obj["manifest"] = parse_manifest(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 2037, in parse_manifest
manifest = ManifestLoader.get_full_manifest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 310, in get_full_manifest
manifest = loader.load()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/manifest.py", line 357, in load
file_reader.read_files()
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 212, in read_files
self.read_files_for_project(project, file_types)
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 219, in read_files_for_project
project_files[file_type_info["parser"]] = read_files_for_parser(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 167, in read_files_for_parser
source_files = get_source_files(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 156, in get_source_files
file = load_source_file(fp, parse_file_type, project.project_name, saved_files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt/parser/read_files.py", line 84, in load_source_file
file_contents = load_file_contents(path.absolute_path, strip=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dbt_common/clients/system.py", line 178, in load_file_contents
to_return = handle.read().decode("utf-8")
^^^^^^^^^^^^^
OSError: [Errno 116] Stale file handle
11:02:35 Resource report: {"command_name": "test", "command_success": false, "command_wall_clock_time": 11.851658, "process_in_blocks": "0", "process_kernel_time": 0.763955, "process_mem_max_rss": "197116", "process_out_blocks": "32", "process_user_time": 2.880488}
11:02:35 Command `dbt test` failed at 11:02:35.484429 after 11.86 seconds
11:02:35 Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [, , ]}
11:02:35 Flushing usage events
11:02:45 An error was encountered while trying to flush usage events
```

### Environment

```markdown
- Image: dbt-labs/dbt-bigquery:1.9.0
- OS: Debian 12 (Bookworm) (derived from the python:3.11-slim base image)
- Python: 3.11
- dbt: 1.9.0
```

### Which database adapter are you using with dbt?

bigquery

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.