sillsdev / sillsdev/python-sil-lift

Honor SOURCE_DATE_EPOCH as the clock source for generated timestamps

Open
#9 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1
Forks
0
Avg merge
10d 11h
Merged PRs (30d)
6

Description

Problem

sil-lift currently generates no timestamps of its own. dateCreated/dateModified are read from the source and written back verbatim; the only other root attributes are version (fixed at 0.13) and producer (caller-supplied). Serialization is therefore fully deterministic today — the same input plus the same in-memory edits always produce the same bytes, which is what makes the byte-exact fidelity contract testable.

Any feature that generates a timestamp from the wall clock breaks that. A dateModified stamped with datetime.now(...) means two runs of the same script over the same input produce different bytes, which:

  • makes byte-exact round-trip tests of the stamping feature itself unwritable without monkeypatching the clock;
  • defeats diff-based CI gates that assert an export is unchanged;
  • makes container/Action runs non-reproducible, which matters because sil-lift ships both a Dockerfile and a GitHub Action (action.yml) explicitly aimed at pipeline use.

Proposed solution

Honor SOURCE_DATE_EPOCH as the default clock source anywhere the library generates a timestamp, with explicit arguments always taking precedence.

Precedence: an explicit when= argument > SOURCE_DATE_EPOCH > wall clock.

def _default_now() -> datetime:
    """The clock for generated timestamps: SOURCE_DATE_EPOCH if usable, else now."""
    raw = os.environ.get("SOURCE_DATE_EPOCH")
    if raw is not None:
        try:
            return datetime.fromtimestamp(int(raw.strip()), tz=timezone.utc)
        except (ValueError, OverflowError, OSError):
            pass  # a malformed value is not honored; fall through to the wall clock
    return datetime.now(timezone.utc).replace(microsecond=0)

Two properties make this a clean fit:

  • SOURCE_DATE_EPOCH is integer POSIX seconds, so the result is naturally seconds-precision — no truncation needed, and no fractional-second output to worry about.
  • _fmt_date in src/sil_lift/_writer.py already rewrites +00:00 to Z, so an aware UTC datetime serializes as YYYY-MM-DDTHH:MM:SSZ. That is the shape real FieldWorks exports use: across the seven FLEx 8.3–9.0 exports in The Combine's Backend.Tests/Assets, all 70,636 dateCreated/dateModified literals are exactly that 20-character form, with no bare dates, numeric offsets, or fractional seconds.

The variable is already set by Debian, Nix, Buildah, setuptools, and Sphinx among others, so pipelines that set it for other tools get sil-lift determinism for free rather than needing sil-lift-specific configuration.

Dependency

This is only observable once something in sil-lift generates a timestamp; nothing does today. It is a prerequisite or companion to any timestamp-stamping API rather than an independently visible change — and arguably a prerequisite rather than a follow-up, since it is what makes byte-exact tests of such a feature possible without clock monkeypatching.

Open questions

  • Malformed values: the spec permits a consumer to either error or ignore. The sketch above ignores and falls through, which keeps a library from raising on an environment variable the caller may not control — but it means a typo silently reintroduces nondeterminism. An alternative is to ignore it but surface a Problem-style warning, which fits the existing validation vocabulary.
  • Scope of "generated timestamp": dateModified only, or dateCreated too? Both are generated in a from-scratch export, so both presumably.
  • Forcing the wall clock: with the env var set, is passing when=datetime.now(timezone.utc) explicitly the sanctioned escape hatch, or is a dedicated override wanted?
  • CLI surface: whether the validate/export commands should also accept an explicit --now value, which would document the behavior in --help where an environment variable is invisible.
  • Documentation home: the reproducibility guarantee belongs next to the fidelity contract in docs/en/fidelity.md, since that is where readers go for byte-exactness claims.

Alternatives considered

  • A sil-lift-specific variable (e.g. SIL_LIFT_NOW). Rejected: it duplicates an established cross-ecosystem standard, and a pipeline already pinning SOURCE_DATE_EPOCH would have to learn a second name for the same concept.
  • An injectable clock parameter and nothing else. A when= parameter should exist regardless, but it does not help when the timestamp is generated inside a call the pipeline does not control — the CLI subcommands and the Docker/Action entrypoint being the concrete cases here.
  • Making generated timestamps opt-in only, and declaring reproducibility the caller's problem. Viable, but it pushes every pipeline that wants both stamping and determinism into monkeypatching or post-processing, and it leaves the library's own test suite without a supported way to exercise stamping byte-exactly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading src/sil_lift/_writer.py, especially _fmt_date, then locate the timestamp-generation entry points; the issue states that none exist yet. Review the validate/export CLI paths, Dockerfile, action.yml, and docs/en/fidelity.md while resolving the open questions about scope, overrides, malformed values, and documentation before defining done.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, python
Domain
backend, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.