mandiant / mandiant/capa

elffile: crash with `ValueError: max() iterable argument is empty` when `DT_GNU_HASH` points at a removed `.gnu.hash` section

Open
#3,170 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.2k
Forks
726
Avg merge
11d 11h
Merged PRs (30d)
7

Description

### Description

capa crashes with `ValueError: max() iterable argument is empty` on an ELF whose `.gnu.hash` section has been removed while the `DT_GNU_HASH` dynamic tag remains, for example after `strip --remove-section=.gnu.hash`.

`.gnu.hash` is `SHF_ALLOC` and sits inside a `PT_LOAD`, so `strip` cannot excise it without shifting vaddrs: it drops the section header and zero-fills the bytes in place, leaving the tag with its original `d_ptr`. pyelftools parses that zeroed region as a hash table with `nbuckets == 0`, and `GNUHashTable.get_number_of_symbols()` does `max([])`. Nothing catches the exception, so it aborts the run: capa reports no results at all for a file that is otherwise perfectly analyzable.

Confirmed with `readelf`, independently of pyelftools: after stripping, `readelf -S` reports no `.gnu.hash` section while `readelf -d` still reports `GNU_HASH 0x390` unchanged, `DT_SYMTAB` and `DT_STRTAB` are unmoved, and the target region is all zeroes.

### Steps to Reproduce

```console
$ cat > hello.c <<'EOF'
#include
int main(void) { printf("hello world\n"); return 0; }
EOF

$ gcc -o hello hello.c
$ cp hello hello_stripped
$ strip --remove-section=.gnu.hash hello_stripped

$ capa ./hello # baseline: works, exit 0
$ capa ./hello_stripped # crashes
```

(A `pip install flare-capa` ships no rule set, so add `-r /path/to/capa-rules` or capa exits 10 before reaching the crash. The rule set is irrelevant to it: the crash happens during file feature extraction.)

**Expected behavior:** capa analyzes the file. The dynamic symbols are still fully present: `DT_SYMTAB` is untouched and `.gnu.hash` is only a lookup accelerator, so the file's 3 imports (`__cxa_finalize`, `__libc_start_main`, `puts`) remain recoverable.

**Actual behavior:** `Unexpected exception raised: `, exit code 1. With `-d`:

```
File "capa/capa/features/extractors/elffile.py", line 71, in extract_file_export_names
logger.debug("Dynamic segment contains %s symbols: ", segment.num_symbols())
File "site-packages/elftools/elf/dynamic.py", line 321, in _num_symbols
return gnu_hash_section.get_number_of_symbols()
File "site-packages/elftools/elf/hash.py", line 160, in get_number_of_symbols
max_idx = max(self.params['buckets'])
ValueError: max() iterable argument is empty
```

### Versions

- capa from a source checkout at 497120f, the latest commit on master as of now (`capa/version.py` reports 9.4.0)
- Python 3.12.12, pyelftools 0.33 (reproduced identically on 0.31 and 0.32; `pyproject.toml` requires `>=0.31`)
- Ubuntu 26.04 LTS x86-64, samples built with gcc 15.2.0 / GNU strip (binutils) 2.46

### Additional Information

**Where the behavior changed.** #2142 (merged 2024-06-18 for #2096, first shipped in v7.2.0) added a dynamic-segment path that works without section headers, and that path trusts `DT_GNU_HASH` unconditionally. capa 7.1.0 does not crash on the stripped sample and extracts 0 exports / 3 imports from it; the v7.2.0 standalone crashes on the same file. It is not a pyelftools version effect: `DynamicSegment.num_symbols()` raises on 0.31, 0.32 and 0.33 alike. #2142 is not a regression overall, since its gain is real for binaries stripped of section headers whose `.gnu.hash` is intact, which 7.1.0 cannot read at all. The crash is its side effect.

**Affected call sites**, all in `capa/features/extractors/elffile.py`: `segment.num_symbols()` at line 71, which is where a default run lands because a `logger.debug` argument is evaluated regardless of log level, and `segment.iter_symbols()` at lines 73 and 101. `iter_symbols` calls `num_symbols` itself, so all three fail the same way.

**Why capa's error handling misses it.** `main.py:777` already catches `(ELFError, OverflowError)` and turns it into a clean `E_CORRUPT_FILE` exit 13. This one escapes for a mechanical reason: the count comes from an unguarded `max(self.params['buckets'])` (`hash.py:160`), so an empty bucket list surfaces as a builtin `ValueError` rather than the `ELFError` pyelftools raises for malformed input elsewhere.

**A stale tag fails in four distinct ways.** Three raise, each a different type, and `struct.error` derives straight from `Exception`, so no one catch covers them:

| stale `DT_GNU_HASH` points at | result |
|---|---|
| a zero-filled region (`strip --remove-section`) | `ValueError` from `max([])`, `hash.py:160` |
| junk (`0xff` fill, tag retargeted into `.text`) | `ELFParseError`, a subclass of `ELFError` |
| a chain walk running past EOF | `struct.error`, `hash.py:172` |

The fourth does not raise at all. `ELFHashTable.get_number_of_symbols` returns the SysV table's `nchains` field directly (`hash.py:75`), so removing the `.hash` section of a `--hash-style=sysv` binary reads as zero symbols and `iter_symbols()` returns an empty list silently. Any handling keyed on exceptions alone misses this case, and it is not exotic: Go binaries link `.hash` rather than `.gnu.hash`.

**The data is recoverable, but not through pyelftools' own fallbacks.** `DT_SYMTAB` still points at an intact `.dynsym`, and its entries read back in order and match the names the unstripped original yields. `DynamicSegment._num_symbols` (`dynamic.py:315`) has two fallbacks for bounding that table, the nearest `DT_*` pointer above `DT_SYMTAB` and then the end of the containing `PT_LOAD`. The stale tag short-circuits the first branch so neither is reached, and both are unreliable when they are. Measured against ground truth (`.dynsym` `sh_size / sh_entsize`) on 925 binaries from `/usr/bin` and `/usr/lib/x86_64-linux-gnu`:

- The pointer bound is exact on 225 of them, 24%. `d_ptr` and `d_val` are a union and the loop compares every tag as though it were an address: in `/usr/bin/apt-config`, `.dynsym` sits at `0x3c8` and `DT_PLTRELSZ` is `0x3f0`, a byte count rather than a place, so a 54-entry table is bounded at 1 symbol. Across the 922 whose `DT_STRTAB` does sit above `DT_SYMTAB`, every value intruding between `.dynsym` and `.dynstr` belongs to a non-address tag (`DT_NEEDED` x1668, `DT_STRSZ` x531, `DT_PLTRELSZ` x420, `DT_SONAME` x224, `DT_RELASZ` x198, `DT_RELACOUNT` x27, `DT_RUNPATH` x13), and not one to an address tag.
- The `PT_LOAD` bound never runs. The tag scan finds something above `DT_SYMTAB` on all 925, so the second fallback is unreachable in practice. Where the first bound errs the other way, an address tag is what does it: Go binaries lay `.dynstr` *below* `.dynsym`, so `DT_STRTAB` no longer caps the scan and the nearest pointer above `DT_SYMTAB` is `DT_PLTGOT`. `/usr/bin/docker-proxy`, whose `.dynsym` holds 39 entries, yields 13,118 that way, and the surplus carries whatever the following bytes spell (`'\x08'`, `'Rh'`, `'ead_getattr_np'`), which would surface as file-scope `export:` features. The `PT_LOAD` bound would have given exactly 39 here, so it is the pointer bound, not the segment bound, that overshoots.

Two properties did hold across that corpus and bear on any recovery. The `.dynsym` section header gives the size of the table by definition and survives `strip --remove-section=.gnu.hash`; on healthy binaries it agrees with the hash-derived count in all 925, so the two corroborate each other and a disagreement means one of them is wrong. Which one is not decidable in general: a section header is not load bearing, so a file can carry a false one and still run, while the loader itself reads the dynamic tags. And the first nameless entry is an exact end marker. No `.dynsym` entry after index 0 lacked a name anywhere in the 925. I also read the table the other way, unbounded through `DT_SYMTAB` and `DT_STRTAB` with no section headers consulted, stopping at the first nameless entry after index 0: that lands on `sh_size / sh_entsize` for all 925, never short and never long, and it works on the three Go binaries that every pointer bound overshoots. The entry past the end reads as nameless either way, through an out-of-range `st_name` where `.dynstr` follows `.dynsym` (922 of them) or through `st_name == 0` in the zero padding that follows it (the 3 Go binaries).

**How widespread the input class is.** I mutation-fuzzed the extractor: 3000 samples of a hello-world binary, each with one to four random bytes overwritten in one of four structured regions drawn uniformly (ELF header and head of the program header table; `.gnu.hash` through `.rela.plt`; the `.dynamic` tag array; the section header table). 2554 analyzed cleanly and 161 produced unhandled tracebacks: 2 `ValueError` (`hash.py:160`) and 34 `struct.error` (`hash.py:172`) from this path, plus 125 `RuntimeError: generator raised StopIteration` from an unrelated `DT_RELA`-without-`DT_RELAENT` path. The remaining 285 were already reported properly as corrupt (`ELFParseError` 186, `ELFError` 98, `OverflowError` 1). That `RuntimeError` is a separate root cause and I will file it separately; the symbol-name path also raises a bare `AssertionError` from `dynamic.py:183` when `DT_STRTAB` points out of range and no section headers remain, which belongs with that report. (The bare `assert` arrived with 0.33's type hints, but 0.31 and 0.32 are no better off: there `_get_stringtable` returns `None` and the same input raises `AttributeError` at `dynamic.py:333`. Either way a non-`ELFError` builtin escapes, so the defect is not confined to one version, and `pyproject.toml` allows `>=0.31`.)

**Scope of the consequence.** In a default run the extractor's symbols feed only the pre-analysis limitation check, whose rules use no `import:`/`export:` features, so losing them changes nothing observable: vivisect derives `api:`, `import:` and `export:` features from its own ELF parser. The symbols matter for the binexport2 backend, which builds file-scope `import:`/`export:` features from this extractor, and there the difference is confined to the 27 rule files that reference those features. The crash, by contrast, happens in the pre-analysis pass and so loses everything on every backend.

**Upstream component.** The deeper problem is in pyelftools: `GNUHashTable.get_number_of_symbols` derives the symbol count without checking that the bytes are a hash table at all, and neither `ValueError` nor `struct.error` is an `ELFError`, so consumers that handle malformed ELF by catching `ELFError` miss them. I found no existing report for this specific defect; the closest is eliben/pyelftools#612, which raises the general problem of non-`ELFError` exceptions escaping on inputs `llvm-readelf` parses happily, but does not reach this function. The code is unchanged on master (`hash.py:155`). Validation there has to be on the table's contents rather than on whether a `.gnu.hash` section exists, since `GNUHashTable` documents itself as serving super-stripped binaries that have no section headers. Validating and then falling through to the existing `_num_symbols` fallbacks would only trade the crash for silent truncation on three binaries in four, by the measurements above, so the bound wants fixing in the same change. Restricting the tag scan to address-valued tags is necessary but not sufficient on its own: it removes the 697 undercounts and still leaves `/usr/bin/docker-proxy` at 13,118, because the tag that overshoots there is `DT_PLTGOT`. Capping the scan at `DT_STRTAB` as well, and treating the table as unbounded when `DT_STRTAB` sits below `DT_SYMTAB`, is exact on all 922 bounded binaries in the corpus and leaves only the 3 Go binaries to be bounded some other way. Either way capa needs its own handling, since `pyproject.toml` allows pyelftools `>=0.31`.

**Prior art.** angr's loader hit this exact traceback (`hash.py:160`, `max()` on an empty bucket list) and is fixing it in angr/cle#792, opened 2026-08-26 and still open: it catches the pyelftools exception and re-bounds `DT_SYMTAB` using only address-valued dynamic tags, cross-checked against `DT_STRTAB - DT_SYMTAB`. The same PR measured 8,424 distro binaries and found no linker emits `nbuckets == 0` (GNU ld 2.46 and LLD 21 emit `nbuckets == 1` for a `.so` that exports nothing), so a zero-bucket table is a corruption signature rather than a build artifact. elfutils already handles this input: `find_dynsym` in `libdwfl/dwfl_module_getdwarf.c` guards the GNU hash walk with `symndx < nbuckets`, which is false for a zeroed table, and falls through to the `DT_STRTAB` bound, so `eu-readelf -D --dyn-syms` recovers every symbol from the stripped sample. GNU `readelf` and `llvm-readelf` report zero dynamic symbols on it without error, since neither sizes the table from `DT_GNU_HASH`. glibc's `ld.so` loads and runs the file: `do_lookup_x` in `elf/dl-lookup.c` skips objects with `l_nbuckets == 0`. This is a runnable binary that capa currently refuses.

**Proposal.** I am happy to work on a PR for capa's ELF extractor, with tests. The strategy I have in mind:

- Read the symbols the hash table yields, but never trust its count: cut the list at the first entry whose `st_name` falls outside `DT_STRSZ`. This is also the only thing that reaches the SysV case, which raises nothing at all: a removed `.hash` reads as zero symbols, and handling keyed on exceptions cannot see that.
- Bound `DT_SYMTAB` independently, from the `.dynsym` section header when there is one and from a tag scan restricted to address-valued tags and capped at `DT_STRTAB`, then read the table up to the larger of the two. Neither source is trustworthy alone: a section header is not load bearing, so a file can carry a false one and still run, while the tags are what the loader reads. Taking the larger bound over-reads when the section header lies upward, and the extra entries are only filtered by the `st_name` check and the type/value conditions the extractor already applies. Taking the smaller would silently drop symbols, and an early stop is something a sample can steer, so I prefer the direction whose failure is visible.
- When neither bounds it, as in a Go binary with `.dynstr` below `.dynsym` and no section headers, read until the first nameless entry after index 0. Exact on every binary in the corpus above, including the Go ones the pointer bound overshoots.
- Treat `ValueError`, `struct.error`, `AttributeError` and `AssertionError` alongside `ELFError` as "this ELF is malformed", scoped to the pyelftools calls themselves rather than wrapped around extractor logic. Log the fallback at debug, like the rest of the extractor.

Does that approach look right to you, or would you rather solve it a different way? Glad to adjust before writing anything.

Related but not a duplicate: #1704 (closed, different code path, pre-dates #2142).

### Note on tooling

This issue was written with AI tooling (Claude Code). The investigation, the test samples, the corpus measurements, the prior-art survey and the text above were produced with AI assistance. I reviewed and double checked the AI's findings, and reproduced the crash and the measurements myself before filing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at capa/features/extractors/elffile.py, especially segment.num_symbols() and iter_symbols(), and reproduce the failure with strip --remove-section=.gnu.hash. Read pyelftools' dynamic.py and hash.py to understand the stale DT_GNU_HASH path; done means the stripped ELF completes analysis without a traceback while symbol extraction remains correctly bounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
reverse-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.