Optional ingest-time gating for untrusted document sources

Offen
#63 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
30/100
Issue-Typ
Feature
Klarheit
Muss geklärt werden
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
database, security

Rechercherichtung

No implementation is selected yet. Read _ingest_location, _resolve_local, _locations_from_files_table, and add_docs to understand the current trust behavior, then review the DID-matlab companion issue and bridge sync requirements. Done would require an agreed optional gating design, matching Python and MATLAB changes, and tests for trusted and attacker-authored sources.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Context

Follow-up to #58 and #60. After #60 (PR #62), _ingest_location refuses only unsafe uid values via _is_safe_uid — the ingest source location is trusted verbatim (_resolve_local, no containment check), on the reasoning that the caller who says "add this file to my DB" chose the source. This unblocks legitimate ingest workflows (NDI-python stages sources under /tmp/ndi-vhsb-*/ and adds them to a DB whose .ndi lives elsewhere) and keeps the destination fully constrained under <FileDir>/<uid>.

The read-side _is_safe_local_location filter (_locations_from_files_table) still defends orig_location, so a crafted stored location cannot steer open_doc to read outside db_dir through the orig_location branch.

The gap

There is one code path that ingestion isn't gated against, and it only matters when the document JSON is attacker-controlled (a cloud pull, not a locally authored document):

  • A document with location='../../etc/passwd' (or any traversal / absolute-outside path pointing at a readable file) now ingests successfully on POSIX. shutil.copyfile resolves the traversal, reads the target file, and writes its contents into <FileDir>/<uid>.
  • On a later open_doc, the read-side filter refuses orig_location, but by then the cache candidate at <FileDir>/<uid> — built from the safe uid — already exists and wins the earlier loop; the smuggled contents are served back.

There is no such issue when the source of the document is trusted (the local NDI-python call, say). It is specifically the cloud-pull path where the JSON is not the caller's.

What we're not doing

For now, no change. The trade-off from #60 stands: reintroducing a source-side containment check breaks the legitimate workflow, and this residual surface only matters for the "attacker-authored document" case — one downstream packages can also mitigate by not blindly ingesting cloud-pulled documents.

What an optional gate might look like

If we do decide to close this later:

  • Narrow the guard to relative locations with a traversal segment (a relative path whose _resolve_local result escapes db_dir) — this refuses the ../../etc/passwd shape without refusing an absolute path that legitimately lives outside db_dir. It's stricter than "no traversal", weaker than "must be inside db_dir".
  • Or make the guard opt-in via add_docs(..., trust_sources=True|False), defaulting to True (today's behavior) and set to False by callers who pull documents from a cloud store. Downstream (NDI cloud pull) then sets trust_sources=False and gets the strict containment check.

Either would need matching changes on the MATLAB side and a bridge sync note. See the DID-matlab companion issue for the parity version.

Vorherrschende Sprache
Python
Sterne
0
Forks
1
Ø Merge
2 Std. 15 Min.
Gemergte PRs (30 T.)
40

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.