Evaluate feasibility of automated file content pattern creation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
The gist of this idea is that with some package repositories we have links to the source code used to build the binary packages. And we can leverage our work with source code parsing to automatically extract features that should appear in the compiled binary, and possibly use AI to automatically create regular expressions for recognizing version strings.
Potential issues:
- Identifying good strings (or other data in binaries) -- what is unique, and ideally has version identifiers that will work cross-platform
- Symbol names could be interesting, but need good demangler that can handle multiple decompilers (this might be more useful for a check once we think we know a particular library is present to confirm that there is actual code in a file)
- Having a way to efficiently check thousands of patterns (10k+ minimum, potentially 100k+)
- A fast pattern matching implementation was tested in Surfactant -- it is able to recognize 15k+ patterns in around 5.9 sec, and 100k+ in about 35 seconds. Most of the increase in time is building the Aho-Corasick automaton, which could be optimized/cached for subsequent runs. For comparison checking 15k+ regexes one at a time took 5min 18sec (100k+ was not tested because it would be over 30min).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified in the issue. Start by reviewing the source-parsing work referenced here and Surfactant's fast pattern-matching implementation, including its Aho-Corasick results. Done would be a documented feasibility assessment covering pattern extraction, symbol handling, scalable matching, and possible AI-assisted regular expressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, reverse-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100