freeCodeCamp / freeCodeCamp/scripts

[feat]: ultimate normalization script

Open
#222 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
17
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Going forward, there are likely many times we may want to normalize records. As such, we need a smart script:

**Musts**

- [ ] not bring down production to run script
- it should be a script we can tune to **not** hammer the db, and quietly run in the background
- [ ] easy to define/add transformations
- [ ] fault tolerant
- [ ] keep track of versioning
- must not repeat work - the script should be stopped/started without hesitation, and start off where it left off
- [ ] fearlessly concurrent
- the api will write a `lastUpdated` field to the user record which should allow all ops to be done without losing data

**Niceties**

- [ ] dynamically asks database how loaded it is, and accordingly adjusts ops/rps

**Notes**

Currently, the idea is to add a `schemaVersion` field to the `user` record.

I would prefer to write this in Rust:
1) Ease of distribution
2) Strict typing

That said, this might go against _"easy to define/add transformations"_.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by clarifying the database and API interfaces, the proposed user.schemaVersion behavior, and how transformations are defined and versioned. Done should include resumable, fault-tolerant, concurrency-safe execution that can be throttled without repeated work or production impact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.