trailofbits / trailofbits/polyfile

Settle and document the public Python API before 1.0

Open
#3,577 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Python
Stars
390
Forks
31
Avg merge
7h 52m
Merged PRs (30d)
72

Description

Summary

polyfile/__init__.py exports a public Python API — Matcher, Parser, PARSERS,
register_parser, Submatch, Match, InvalidMatch — but nothing states what a caller may rely
on. There is no py.typed marker, no documented contract, and no deprecation policy. Before
declaring 1.0, decide what that surface is and write it down.

This is an entry criterion for the v1.0.0 milestone, not a defect.

Why now

The v0.6.0 cycle changed that surface repeatedly, which is the evidence that it is not yet settled:

  • try_all_offsets was removed from Matcher.__init__ and Analyzer.__init__ (#3465). It had been
    accepted and ignored since it was introduced.
  • Matcher.match changed behavior substantially for an already-open stream (#3463): it previously
    returned 1 match where a path returned 51.
  • FileStream.__init__'s length parameter had two contradictory meanings depending on which
    branch ran (#3561). One was chosen and documented; nothing had recorded which was intended.
  • Both "Extending PolyFile" examples in CLAUDE.md were non-functional, and the register_parser
    example documented the wrong shape entirely — it decorated a class, where register_parser wraps
    its argument in a function wrapper (#3467). Anyone following the guide got
    TypeError: MyParser() takes no arguments.

A 1.0 version number promises that breaking changes wait for 2.0. That promise is only meaningful if
the surface it covers is defined.

What this needs

  1. Decide what is public. polyfile/__init__.py's exports are the current de facto answer, but
    Analyzer, MagicMatcher, MagicTest, FileStream and MatchContext are all reachable and
    used by the documented extension points. Either bring them in deliberately or state that they are
    internal.
  2. Write the contract. docs/extending_polyfile.md exists and should carry it: what a custom
    matcher and a custom parser must implement, what they may rely on, and what the library
    guarantees across minor versions. The corrected examples from #3467 are the starting point.
  3. Add a py.typed marker so type checkers see the annotations the package already carries, and
    decide whether the annotated signatures are part of the contract.
  4. Settle #3532 first, or state that it is out. Implementing all-offsets scanning would change
    Matcher's behavior and possibly its constructor. Deciding that after 1.0 means either breaking
    the promise or shipping it awkwardly.
  5. State a deprecation policy. One minor release with a warning is the usual shape; any policy
    is better than none.

Related

  • #3532 — all-offsets scanning, which would change Matcher
  • #3467 — the documentation defects that revealed the extension API was undocumented in practice
  • #3463, #3465, #3561 — the v0.6.0 changes to this surface

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the exports in polyfile/init.py and read docs/extending_polyfile.md alongside the corrected examples from #3467. Review the public extension points and settle the relationship with #3532 before documenting the supported API, version guarantees, deprecation policy, and adding the py.typed marker. Done means the 1.0 contract and its scope are explicit in the documentation and package metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.