HarperFast / HarperFast/harper
Bytes-column export (export_local json) is not re-importable (numeric-keyed object, not base64)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
## Summary
A `Bytes`-typed column cannot survive an `export_local` → re-import round-trip. `export_local(format: json)` serializes a `Bytes` field as a **numeric-keyed JS object** (`{"0":65,"1":65,"2":72,...}` — the byte values), not as a base64 string. The import path (`csv_data_load` / ops `insert`) then rejects that shape ("must be a Buffer or Uint8Array"), so a full backup → restore silently cannot reconstruct any binary column.
Confirmed on **main `541a3d33d` (v5.1.11)**, both storage engines (the serialization is engine-agnostic / pure JSON).
## Reproduction
```
npm run test:integration -- "integrationTests/qa-scratch/qa258-backup-restore.test.ts"
```
The test log prints, verbatim:
```
DEFECT — Bytes field serialization in export: bioBytes exported as
numeric-keyed object {"0":65,...} len=12 — NOT importable
```
The suite passes 6/6 only because the harness works around the broken form before calling import; the raw export JSON is unimportable as-is.
## What works (scope, for contrast)
Every other scalar round-trips exactly through export/restore — `Long` (incl. > 2^53), `Float`, `Boolean`, `Int` boundaries, unicode/emoji strings, `null`. Indexes rebuild; cross-engine restore is clean (it's plain JSON). The gap is specifically the `Bytes` serialization shape.
## Impact
Backup/restore is a data-durability operation; silently producing an export that cannot restore a whole column type is a real fidelity gap. A deployment with binary columns relying on `export_local` for DR loses them on restore (or must hand-transform every record).
## Suggested fix
Serialize `Bytes` as base64 (matching the supported write encoding) so the round-trip closes, or teach the import path to accept the numeric-keyed-object form.
---
*Surfaced by the QA-explorer campaign (scenario QA-258), re-confirmed on `main` v5.1.11. Severity: medium. Filed for @kris.*
— Claude (Opus 4.8, 1M)
Contributor guide
Research direction
Run integrationTests/qa-scratch/qa258-backup-restore.test.ts and inspect the export_local path alongside csv_data_load and the ops insert import path. Trace how Bytes becomes the numeric-keyed object and how import validates it; done means a raw JSON export can restore Bytes without the test harness workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs, typescript
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100