larksuite / larksuite/cli

API registry data is not reproducible: version the api_definition endpoint or ship meta_data.json in releases

Open
#2,398 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
17.3k
Forks
1.4k
Avg merge
2d 4h
Merged PRs (30d)
105

Description

Summary

scripts/fetch_meta.py fetches meta_data.json from a live endpoint on every build, and internal/registry/remote.go refreshes it at runtime. The endpoint serves continuously changing content while reporting a constant data.version of \"1.0.0\", so:

  • builds of the same git tag at different dates embed different registries (official release binaries are not reproducible),
  • the runtime data_version conditional never detects changes,
  • downstream packagers that pin the endpoint by hash (e.g. nixpkgs, see NixOS/nixpkgs#554000) end up with pins that die the moment the content drifts, because the previously served bytes are gone forever.
Evidence

All of the following is reproducible against the live endpoint:

  1. Content drifts while data.version stays constant. Wayback Machine snapshots of GET https://open.feishu.cn/api/tools/open/api_definition?protocol=meta:

    snapshot services resources methods normalized sha256 (of jq -S \".data\")
    2026-06-26 14 77 237 4d772886…
    2026-07-01 15 78 239 a00d5ba6…
    2026-08-11 15 80 245 2f45412c…
    today 15 81 247 244b769f…

    In every snapshot data.version is the literal \"1.0.0\".

  2. data_version is an exact-match conditional against that constant, not a version selector.

    • &data_version=1.0.0 returns a 38-byte empty "not modified" body;
    • &data_version=0.0.1, 1.0.1, 2.0.0, 9999.0.0, 2026-07-01, abc, … all return the current full document with version: \"1.0.0\".

    There is no way to request a historical version.

  3. client_version is ignored: the v1.0.58 and v1.0.88 URLs return byte-identical data today.

  4. No HTTP cache anchors: the response has no ETag, Last-Modified, or Cache-Control.

Consequences
  • Runtime refresh is effectively dead for binaries with embedded data. fetchRemoteMerged(embeddedVersion) sends data_version=1.0.0 (the embedded registry version), the server compares it against its own constant \"1.0.0\", finds an exact match, and answers "not modified". Embedded baselines therefore never receive updates, even though the content has changed several times since.
  • Same-tag builds are not reproducible. build.sh runs fetch_meta.py before go build, so rebuilding a tag at a different date embeds a different registry than the official binaries for that tag.
  • Downstream hash pins die silently. In nixpkgs the registry is a fixed-output derivation pinned by sha256. When the endpoint drifted, the bytes behind the old hash became unfetchable forever: the 1.0.58 pin can no longer be fetched from the endpoint or from cache.nixos.org, so that nixpkgs revision cannot build the CLI from source anymore.
Workaround currently used downstream

nixpkgs now extracts the registry that upstream already embeds in each release binary (go:embed via loader_embedded.go) from the official linux-amd64 release tarball, which is immutable per version and checksummed. This was verified byte-exact against the official 1.0.88 and 1.0.58 binaries (the two releases even use different JSON key orders). It works, but it is a workaround for missing versioned distribution of this data.

Requests

Any one of these would fix reproducibility; the first two together would be ideal:

  1. Bump data.version whenever the registry content changes, so the existing data_version conditional actually works (this would also revive the runtime refresh protocol).
  2. Support fetching a specific data_version (historical snapshot), so builds can pin the data version alongside the CLI version.
  3. Ship meta_data.json in the git tag / release assets. The file is already embedded in every release binary, so publishing it alongside the binaries is zero extra work and gives every downstream packager an immutable, versioned source.

Happy to provide any further traces or test results if useful.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/fetch_meta.py, internal/registry/remote.go, build.sh, and loader_embedded.go to trace how registry data is fetched, embedded, refreshed, and released. Reproduce the documented endpoint behavior and compare the three proposed distribution or versioning approaches. Done means release builds use an immutable, reproducible registry source and the runtime refresh behavior or release asset distribution matches the chosen approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
build-system, release
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.