ChainSafe / ChainSafe/lodestar

Use binary diff to store checkpoint state

Open
#7,535 2 comments 0 reactions 0 assignees View on GitHub
meta-feature-request
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Problem description

Right now the finalized state slot 3710944 takes 257.61 MB. The [holesky-branch](https://github.com/ChainSafe/lodestar/pull/7501/files#diff-63998979cd5f7d4f4c9b934f8b20191f3403ed8f9c43a60aabed9c4f07a860cbR64) has to only keep the last 100 epochs for the user which is not great

### Solution description

instead of storing the whole state, we only store `validators` diff and state (without validators) diff using @chainsafe/xdelta3-node

```typescript
import { describe, expect, it } from "vitest";
import fs from "fs";
import { formatBytes } from "@lodestar/utils";
import {decodeSync, encodeSync} from "@chainsafe/xdelta3-node";
import {holeskyChainConfig} from "@lodestar/config/networks";
import { createChainForkConfig } from "@lodestar/config";
import { getForkFromStateBytes, getStateTypeFromBytes } from "../src/util/sszBytes";
import { ssz } from "@lodestar/types";

describe("State diff", () => {
const dir = ".";
const statePath = `${dir}/holesky_state_3768287.ssz`;
const finalizedStatePath = `${dir}/holesky_finalized_state_Feb_25.ssz`;
const config = createChainForkConfig(holeskyChainConfig);
it("load state and check validators size", () => {
const stateBytes = fs.readFileSync(statePath);
console.log("@@@ state bytes", formatBytes(stateBytes.length));
const fork = getForkFromStateBytes(config, stateBytes);
const forkName = config.getForkName(fork);
console.log("@@@ fork", forkName);
const stateType = getStateTypeFromBytes(config, stateBytes);
const state = stateType.deserializeToViewDU(stateBytes);
console.log("@@@ state", state.slot);
const validatorsBytes = state.validators.serialize();
console.log("@@@ validators bytes", formatBytes(validatorsBytes.length));

const finalizedStateBytes = fs.readFileSync(finalizedStatePath);
console.log("@@@ finalized state bytes", formatBytes(finalizedStateBytes.length));
const finalizedStateType = getStateTypeFromBytes(config, finalizedStateBytes);
const finalizedState = finalizedStateType.deserializeToViewDU(finalizedStateBytes);
console.log("@@@ finalized state", finalizedState.slot);
const finalizedValidatorsBytes = finalizedState.validators.serialize();
console.log("@@@ finalized validators bytes", formatBytes(finalizedValidatorsBytes.length));

const fullStateDelta = encodeSync(finalizedStateBytes, stateBytes);
console.log("@@@ state delta", formatBytes(fullStateDelta.length));

let start = Date.now();
const validatorDelta = encodeSync(finalizedValidatorsBytes, validatorsBytes);
console.log("@@@ validator delta", formatBytes(validatorDelta.length), Date.now() - start, "ms");
start = Date.now();
const validatorsBytes2 = decodeSync(finalizedValidatorsBytes, validatorDelta);
console.log("@@@ apply delta in", Date.now() - start, "ms");
expect(Buffer.compare(validatorsBytes, validatorsBytes2)).toBe(0);

state.validators = ssz.phase0.Validators.defaultViewDU();
state.commit();
finalizedState.validators = ssz.phase0.Validators.defaultViewDU();
finalizedState.commit();
const stateWoValidatorsBytes = state.serialize();
const finalizedStateWoValidatorsBytes = finalizedState.serialize();
console.log("@@@ state without validators bytes", formatBytes(stateWoValidatorsBytes.length));
console.log("@@@ finalized state without validators bytes", formatBytes(finalizedStateWoValidatorsBytes.length));
const stateDelta = encodeSync(finalizedStateWoValidatorsBytes, stateWoValidatorsBytes);
console.log("@@@ state delta", formatBytes(stateDelta.length));
})
});

```

on our typical infrastructure it prints out
```
@@@ state bytes 257.65 MB
@@@ fork bellatrix
@@@ state 3768288
@@@ validators bytes 221.86 MB
@@@ finalized state bytes 257.61 MB
@@@ finalized state 3710944
@@@ finalized validators bytes 221.85 MB
@@@ state delta 21.69 MB
@@@ validator delta 8.17 MB 2749 ms
@@@ apply delta in 289 ms
@@@ state without validators bytes 35.79 MB
@@@ finalized state without validators bytes 35.77 MB
@@@ state delta 13.36 MB
```

so validator delta is just < 8.2 MB while state delta is < 13.36 MB, in total we can store < 22MB instead of 257MB
the down side is that it takes almost 3s to compute delta, need to consider worker thread or any native solution to avoid that

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.