iotexproject / iotexproject/iotex-bootstrap

Snapshot tarballs are flat: setup_fullnode.sh and README Option 1 extract them to the wrong directory

Open
#336 0 comments 0 reactions 1 assignee Claimed by @envestcc View on GitHub
bug
Dominant language
Shell
Stars
97
Forks
43
Avg merge
1h 7m
Merged PRs (30d)
2

Description

## Summary

The published snapshot tarballs are **flat** — members sit at the archive root (`chain-00000001.db`, `bloomfilter.index.db`, …) with no `data/` prefix. They therefore have to be extracted with `-C $IOTEX_HOME/data/`.

Two places in this repo still extract them somewhere else. Both look like leftovers from when the snapshot was a single `data.tar.gz`; they were not updated when the URLs moved to the split core/gateway format in eec89cb.

## 1. `scripts/setup_fullnode.sh` / `scripts/setup_fullnode_marketplace.sh` — extracts one level too high

```bash
tar xvf $SAVE_DIR/data-core.tar.gz -C $IOTEX_HOME # setup_fullnode.sh L514
tar xvf $SAVE_DIR/data-gateway.tar.gz -C $IOTEX_HOME # setup_fullnode.sh L525
```

With a flat tarball this lands the DB files in `$IOTEX_HOME/`, not `$IOTEX_HOME/data/`.

The rest of the same script assumes `$IOTEX_HOME/data/`:

- install-state gate: `if [ ! -f "${IOTEX_HOME}/data/chain.db" ]; then _NEED_INSTALL_=1` (L221), same check again at L669
- container mount: `-v=$IOTEX_HOME/data:/var/data:rw`
- cleanup path: `rm -f data/bloomfilter.index.db` (run after `pushd ${IOTEX_HOME}`)
- the call site's own comment: `# Extract snapshot after container is stopped (writes to mounted data dir)`

**Effect of `setup_fullnode.sh --auto --snapshot`:** the ~190 GB download extracts to the wrong directory, the script's own `chain.db` check then fails, the node is treated as a fresh install and starts syncing from genesis against an empty `/var/data` — while the snapshot files sit unused one level up. To the user this reads as "the snapshot silently did nothing".

`scripts/all_in_one_mainnet.sh` in the same commit is correct (`-C $IOTEX_HOME/data/`), which is what makes this look like a missed line rather than an intentional difference.

## 2. `README.md` Option 1 (gateway snapshot) — no `-C` at all

```bash
aria2c -x16 -s16 -c --file-allocation=none \
-d $IOTEX_HOME -o data_index.tar.gz \
https://t.iotex.me/mainnet-data-snapshot-gateway-latest
tar -xzf data_index.tar.gz
```

Step 5 just above it correctly uses `-C $IOTEX_HOME/data/`. Option 1 extracts into the current working directory instead, and the file path is unqualified so it also only resolves if `cwd == $IOTEX_HOME`. Same in `README_CN.md` / `README_testnet.md` if they carry the same block.

## 3. Dead link in `changelog/v1.2-instruction.md`

`https://t.iotex.me/mainnet-data-with-idx-latest` 302s to `https://storage.iotex.io/mainnet-data-with-idx-latest.tar.gz`, which now returns **404**. It is an outdated doc, but it is still reachable and people do follow it — worth pointing at `mainnet-data-snapshot-gateway-latest` or marking the file as superseded.

## How the flat layout was verified

- Read the first tar member of the current mainnet and testnet core and gateway tarballs (streamed, two different network paths): all flat, no `data/` prefix.
- Cross-checked against a running mainnet API node bootstrapped from these same tarballs with `-C /`: the DB files sit directly in the data dir, with no nested `data/data/` and nothing stray one level up — which could not happen if the archives carried a prefix.

## Suggested fix

```diff
- tar xvf $SAVE_DIR/data-core.tar.gz -C $IOTEX_HOME
+ tar xvf $SAVE_DIR/data-core.tar.gz -C $IOTEX_HOME/data
```
same for the gateway line and for `setup_fullnode_marketplace.sh`, and add `-C $IOTEX_HOME/data/` to the README Option 1 extract.

Worth a quick end-to-end run of `setup_fullnode.sh --auto --snapshot` on a scratch host afterwards, since that path does not appear to be covered by CI.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.