sillsdev / sillsdev/python-sil-lift
Honor SOURCE_DATE_EPOCH as the clock source for generated timestamps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 10d 11h
- Merged PRs (30d)
- 6
Description
Problem
sil-lift currently generates no timestamps of its own. dateCreated/dateModified are read from the source and written back verbatim; the only other root attributes are version (fixed at 0.13) and producer (caller-supplied). Serialization is therefore fully deterministic today — the same input plus the same in-memory edits always produce the same bytes, which is what makes the byte-exact fidelity contract testable.
Any feature that generates a timestamp from the wall clock breaks that. A dateModified stamped with datetime.now(...) means two runs of the same script over the same input produce different bytes, which:
- makes byte-exact round-trip tests of the stamping feature itself unwritable without monkeypatching the clock;
- defeats diff-based CI gates that assert an export is unchanged;
- makes container/Action runs non-reproducible, which matters because sil-lift ships both a
Dockerfileand a GitHub Action (action.yml) explicitly aimed at pipeline use.
Proposed solution
Honor SOURCE_DATE_EPOCH as the default clock source anywhere the library generates a timestamp, with explicit arguments always taking precedence.
Precedence: an explicit when= argument > SOURCE_DATE_EPOCH > wall clock.
def _default_now() -> datetime:
"""The clock for generated timestamps: SOURCE_DATE_EPOCH if usable, else now."""
raw = os.environ.get("SOURCE_DATE_EPOCH")
if raw is not None:
try:
return datetime.fromtimestamp(int(raw.strip()), tz=timezone.utc)
except (ValueError, OverflowError, OSError):
pass # a malformed value is not honored; fall through to the wall clock
return datetime.now(timezone.utc).replace(microsecond=0)
Two properties make this a clean fit:
SOURCE_DATE_EPOCHis integer POSIX seconds, so the result is naturally seconds-precision — no truncation needed, and no fractional-second output to worry about._fmt_dateinsrc/sil_lift/_writer.pyalready rewrites+00:00toZ, so an aware UTC datetime serializes asYYYY-MM-DDTHH:MM:SSZ. That is the shape real FieldWorks exports use: across the seven FLEx 8.3–9.0 exports in The Combine'sBackend.Tests/Assets, all 70,636dateCreated/dateModifiedliterals are exactly that 20-character form, with no bare dates, numeric offsets, or fractional seconds.
The variable is already set by Debian, Nix, Buildah, setuptools, and Sphinx among others, so pipelines that set it for other tools get sil-lift determinism for free rather than needing sil-lift-specific configuration.
Dependency
This is only observable once something in sil-lift generates a timestamp; nothing does today. It is a prerequisite or companion to any timestamp-stamping API rather than an independently visible change — and arguably a prerequisite rather than a follow-up, since it is what makes byte-exact tests of such a feature possible without clock monkeypatching.
Open questions
- Malformed values: the spec permits a consumer to either error or ignore. The sketch above ignores and falls through, which keeps a library from raising on an environment variable the caller may not control — but it means a typo silently reintroduces nondeterminism. An alternative is to ignore it but surface a
Problem-style warning, which fits the existing validation vocabulary. - Scope of "generated timestamp":
dateModifiedonly, ordateCreatedtoo? Both are generated in a from-scratch export, so both presumably. - Forcing the wall clock: with the env var set, is passing
when=datetime.now(timezone.utc)explicitly the sanctioned escape hatch, or is a dedicated override wanted? - CLI surface: whether the
validate/exportcommands should also accept an explicit--nowvalue, which would document the behavior in--helpwhere an environment variable is invisible. - Documentation home: the reproducibility guarantee belongs next to the fidelity contract in
docs/en/fidelity.md, since that is where readers go for byte-exactness claims.
Alternatives considered
- A sil-lift-specific variable (e.g.
SIL_LIFT_NOW). Rejected: it duplicates an established cross-ecosystem standard, and a pipeline already pinningSOURCE_DATE_EPOCHwould have to learn a second name for the same concept. - An injectable clock parameter and nothing else. A
when=parameter should exist regardless, but it does not help when the timestamp is generated inside a call the pipeline does not control — the CLI subcommands and the Docker/Action entrypoint being the concrete cases here. - Making generated timestamps opt-in only, and declaring reproducibility the caller's problem. Viable, but it pushes every pipeline that wants both stamping and determinism into monkeypatching or post-processing, and it leaves the library's own test suite without a supported way to exercise stamping byte-exactly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading src/sil_lift/_writer.py, especially _fmt_date, then locate the timestamp-generation entry points; the issue states that none exist yet. Review the validate/export CLI paths, Dockerfile, action.yml, and docs/en/fidelity.md while resolving the open questions about scope, overrides, malformed values, and documentation before defining done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, python
- Domain
- backend, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100