HarperFast / HarperFast/harper

Bytes-column export (export_local json) is not re-importable (numeric-keyed object, not base64)

Open
#1,467 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## Summary

A `Bytes`-typed column cannot survive an `export_local` → re-import round-trip. `export_local(format: json)` serializes a `Bytes` field as a **numeric-keyed JS object** (`{"0":65,"1":65,"2":72,...}` — the byte values), not as a base64 string. The import path (`csv_data_load` / ops `insert`) then rejects that shape ("must be a Buffer or Uint8Array"), so a full backup → restore silently cannot reconstruct any binary column.

Confirmed on **main `541a3d33d` (v5.1.11)**, both storage engines (the serialization is engine-agnostic / pure JSON).

## Reproduction

```
npm run test:integration -- "integrationTests/qa-scratch/qa258-backup-restore.test.ts"
```

The test log prints, verbatim:

```
DEFECT — Bytes field serialization in export: bioBytes exported as
numeric-keyed object {"0":65,...} len=12 — NOT importable
```

The suite passes 6/6 only because the harness works around the broken form before calling import; the raw export JSON is unimportable as-is.

## What works (scope, for contrast)

Every other scalar round-trips exactly through export/restore — `Long` (incl. > 2^53), `Float`, `Boolean`, `Int` boundaries, unicode/emoji strings, `null`. Indexes rebuild; cross-engine restore is clean (it's plain JSON). The gap is specifically the `Bytes` serialization shape.

## Impact

Backup/restore is a data-durability operation; silently producing an export that cannot restore a whole column type is a real fidelity gap. A deployment with binary columns relying on `export_local` for DR loses them on restore (or must hand-transform every record).

## Suggested fix

Serialize `Bytes` as base64 (matching the supported write encoding) so the round-trip closes, or teach the import path to accept the numeric-keyed-object form.

---
*Surfaced by the QA-explorer campaign (scenario QA-258), re-confirmed on `main` v5.1.11. Severity: medium. Filed for @kris.*

— Claude (Opus 4.8, 1M)

Contributor guide

Open the contributing guide

Research direction

Run integrationTests/qa-scratch/qa258-backup-restore.test.ts and inspect the export_local path alongside csv_data_load and the ops insert import path. Trace how Bytes becomes the numeric-keyed object and how import validates it; done means a raw JSON export can restore Bytes without the test harness workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, typescript
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.