Complete tiny model training, evaluation, custom corpus, and learning experiments

What

Complete the learning program established by the bootstrap in README.md: reproduce the pinned Marin TinyStories/Llama Nano baseline, inspect the system before training, collect comparable training/evaluation evidence, curate and train on a licensed code/technical-text dataset, run controlled experiments, and publish the resulting graphs, failures, lessons, and end-to-end explanation.

Bootstrap commit eb04045 provides the protocol, wrapper, ADR, code maps, and evidence templates; it does not claim that Marin installation, plan lowering, data download, training, checkpoint retention, or evaluation has completed. The execution boundary and current evidence state are documented in ADR 0001, the architecture map, and known concerns.

Why

The project succeeds through understanding and reproducible evidence—not model quality. The unchanged baseline must be established before later recipe changes have meaning. By the end, the learner should be able to connect observed artifacts across:

directory of text → provenance manifest → tokenizer/token IDs → packed batches → Transformer forward pass/logits → next-token loss → gradients/Adam updates → checkpoint → fixed evaluation/inference

The final result must demonstrate a measurable before/after training difference while distinguishing training-data fit from generalization or usefulness.

How

Phase 0 — Review and land the bootstrap as needed

• Confirm bootstrap commit eb04045 is present on main.
• Review the bootstrap wrapper, documentation, templates, ignore rules, setup/resume scripts, local links, and executable bits.
• Land only corrections required to execute the documented plan; keep Marin external and pinned as decided by ADR 0001.
• Record any deliberate pin/baseline revision as an explicit decision rather than silently following Marin main.

Phase 1 — Install and verify the pinned Marin sibling checkout

• Create/verify sibling checkout ../marin at e9e77ee6d9603aa2fe665a3f0e0a2db7ed950fda.
• Verify Python 3.12+, pinned uv (0.12.5 in project setup), and Marin's CPU environment (uv sync --all-packages --extra=cpu).
• Confirm experiments/phase1_tinystories.sh rejects the wrong Marin revision and defaults to non-training plan mode.
• Confirm offline W&B behavior and document cold-cache network/Hugging Face requirements without recording credentials.
• Check available disk/runtime and choose durable artifact/log locations before downloading or training.

Phase 2 — Run the Phase 1 plan and reproduce TinyStories/Llama Nano

• Run MARIN_DIR=../marin ./experiments/phase1_tinystories.sh plan.
• Verify the lowered plan matches the pinned source: TinyStories, Marin tokenizer, context 512, hidden dimension 32, intermediate dimension 128, 2 attention/KV heads, 2 layers, CPU batch size 4, Adam LR 6e-4, weight decay 0.1, and 100 steps.
• Record immutable run/dataset versions, source commits, command, seed/config, machine, artifact prefix, and plan observations in a dated copy of experiments/phase1-tinystories.md.
• Investigate exact step-25/50 checkpoint retention before training. If unchanged Marin cannot retain them, document the limitation and the smallest reproducible comparison method without misrepresenting it as the unchanged baseline.

Phase 3 — Capture untrained model/tokenizer inspection

• Complete notes/model-inspection.md before optimizer updates.
• Record tokenizer identity/revision, representative tokenization/round-trip behavior, special tokens, vocabulary observations, and sequence packing/truncation behavior.
• Record model parameter shapes, total parameter count, initialization statistics, architecture/config, and fixed random seed.
• Save untrained generations for the frozen evaluation prompts/settings and identify the untrained checkpoint/state unambiguously.
• Explain batch → forward pass → logits/loss → gradients → optimizer update in the learner's own words.

Phase 4 — Execute the 100-step baseline training

• Run the explicit run command only after the plan, inspection, prompts, storage, and checkpoint-retention strategy are recorded.
• Capture required metadata and metrics using notes/training-metrics.md: step, train loss, actual LR, grad norm when emitted, tokens/examples seen, throughput, elapsed time, interruptions, and warnings.
• Preserve the final checkpoint and every available partial checkpoint with exact IDs/checksums; do not claim step-25/50 artifacts until verified.
• Record raw log/W&B-offline locations and checksums outside Git; keep a compact tracked summary.
• Confirm actual steps, batch size, sequence length, source revisions, and dataset identity match the plan; record all deviations and failures.

Phase 5 — Evaluate untrained, partial, and final states

• Freeze prompts, held-out input, tokenizer, inference implementation, seed, decoding settings, max tokens, and stop conditions before comparing outputs.
• Compare untrained (0), partial (target 25 and 50), and final (100) states using evals/checkpoint-comparison.md.
• Report exact checkpoint IDs/checksums, train loss, held-out loss where implemented, and other predeclared measurements.
• Compare generations on observable criteria (repetition, local syntax, continuation, determinism, suspicious overlap), not subjective prose quality alone.
• State what changed, what may be sampling noise, and what the evidence does not establish about generalization or usefulness.

Phase 6 — Create a licensed 1–10 MB TypeScript/JavaScript + technical-text corpus

• Define corpus composition, target scale, inclusion/exclusion rules, and intended evaluation split before collection.
• Use only owned data or sources with explicit compatible/permissive terms; public availability alone is not permission.
• For every source, record stable URL/repository commit/owned path, exact license and terms, permission rationale, selected files, and redistribution/training constraints as required by datasets/README.md.
• Implement deterministic extraction, cleanup, normalization, deduplication, concatenation, and train/evaluation split instructions.
• Record source/output revisions, cryptographic checksums, bytes, documents, lines, and tokenizer token count.
• Verify output is 1–10 MB and manually audit a sample for secrets, credentials, personal data, generated/vendor/minified files, incompatible licenses, and train/evaluation leakage.
• Keep corpus bytes out of Git; track only manifests, provenance, licenses, build instructions, configs, and compact statistics.

Phase 7 — Train on the custom dataset

• Define a custom-data baseline that reuses the Phase 1 tokenizer/model/training settings wherever applicable and clearly identifies necessary data-path changes.
• Record a falsifiable prediction and frozen code-domain evaluation prompts/measurements before training.
• Run plan/inspection/training with immutable dataset identity and the same evidence standards as the TinyStories baseline.
• Preserve metrics, available checkpoints, fixed-setting generations, held-out measurements, failures, and a narrow comparison against TinyStories.

Phase 8 — Run controlled one-variable experiments

For every run, use notes/experiment-template.md: state one falsifiable hypothesis, name the control, predeclare measurements, hold seed/data/tokenizer and all unrelated settings fixed, record source/config/artifact identity, and conclude narrowly.

• Learning rate — vary only LR.
• Dataset size — vary only the amount of training data/exposure definition and compare on a fair axis.
• Layers — vary only Transformer depth.
• Hidden dimension — vary only model width (record dependent parameter-count effects explicitly).
• Context length — vary only sequence length (record batching/token-exposure and compute implications explicitly).
• Record failed, unstable, interrupted, and null-result experiments in notes/failures-and-lessons.md; do not retain only favorable runs.

Phase 9 — Publish final README evidence and teach back

• Add readable README graphs for baseline/custom/experiment metrics with labeled axes, run IDs, comparable scales, and links to source summaries; do not imply causal conclusions from uncontrolled comparisons.
• Summarize reproducible settings, results, generation comparisons, failures, operational constraints, licensing decisions, and lessons.
• Update the code maps if ownership, execution, storage, or evaluation boundaries changed.
• Write the final teach-back explaining how a directory of text becomes a trained checkpoint, including provenance, tokenization, batching, forward/loss/backpropagation, Adam updates, checkpoint contents, evaluation, and inference.
• Demonstrate at least one measurable before/after difference between the untrained and trained states using fixed inputs/settings (for example held-out loss and/or a predeclared generation metric), with exact checkpoint identities and uncertainty/limitations.

Dependencies

• Git and a sibling Marin checkout pinned to e9e77ee6d9603aa2fe665a3f0e0a2db7ed950fda.
• Python 3.12+, uv 0.12.5, Marin's CPU extras, JAX/Levanter, and enough CPU time/RAM/disk.
• Network access for cold dependency, TinyStories, and tokenizer downloads; HF_TOKEN may help with rate limits but must never be committed.
• W&B may remain offline; raw logs still require a documented durable location.
• A verified checkpoint-retention/comparison method before relying on exact 25%/50% artifacts.
• Explicitly compatible source licenses and deterministic corpus build/provenance records.
• Storage outside Git for datasets, checkpoints, generated evals, caches, and logs.

Acceptance criteria

• The pinned Marin baseline is runtime-reproduced with plan, exact source/config identity, metrics, logs, and final checkpoint evidence.
• Tokenizer and random initialization are inspected before training, with untrained outputs captured under frozen settings.
• Available 0/partial/final states are compared reproducibly; exact 25/50/100 states are used when retention is verified, otherwise the limitation and comparison method are explicit.
• Metrics and graphs distinguish training fit, held-out behavior, throughput/cost, and generation observations; lower train loss is not presented as model usefulness.
• A deterministic, license-reviewed, checksum-identified 1–10 MB TypeScript/JavaScript and technical-text corpus is built without committing corpus bytes.
• The custom-data run has complete provenance, metrics, checkpoint, and evaluation evidence.
• Learning rate, dataset size, layers, hidden dimension, and context length each have a controlled one-variable experiment record.
• README/docs contain results, graphs, failures, lessons, storage/licensing constraints, and links to evidence.
• Final teach-back accurately explains text directory → trained checkpoint and demonstrates a measurable fixed-protocol before/after difference.

Explicit non-goals

• Producing a useful, competitive, leaderboard-quality, or production-deployable model.
• Broad hyperparameter optimization, multi-variable tuning, benchmark chasing, scaling work, serving, or deployment.
• Treating lower training loss or more plausible-looking samples as proof of generalization, correctness, code ability, or quality.
• Copying/vendoring Marin internals, replacing Marin as the execution engine, or silently updating the pinned baseline.
• Collecting arbitrary public code, books, documentation, web text, restricted data, personal data, or secrets.
• Committing raw/generated corpora, model checkpoints, full logs, caches, W&B state, or large generated evaluation artifacts to Git.

Data, artifact, and storage safeguards

• Follow the dataset provenance/license contract; retain exact terms and permission rationale for every source.
• Never put access tokens, .env files, credentials, personal data, or restricted source content in tracked files, logs, manifests, examples, or generated outputs.
• Dataset artifacts and generated evaluation outputs remain untracked. Publish only reviewable metadata, deterministic build instructions, checksums, compact results, and license-compatible excerpts when needed.
• Follow checkpoints/README.md: checkpoints/Marin artifact trees can be large, are ignored by Git, and are fragile if stored only locally. Choose durable external/local storage, record paths/URIs plus checksums, and verify recoverability before deleting caches or workspaces.
• Do not assume Git/GitHub can store checkpoints, and do not add Git LFS or upload artifacts to a shared service without an explicit storage/licensing decision.
• Treat exact quarter checkpoints as unverified until their artifact IDs/checksums exist; never backfill or label a different state as step 25/50 without clear disclosure.

Comienza con README.md, docs/adr/0001-use-marin-as-a-pinned-external-execution-engine.md y .planning/codebase/ARCHITECTURE.md; después revisa experiments/phase1_tinystories.sh y ejecuta su modo de planificación. Revisa las notas, evals y plantillas de datasets enlazadas, preservando la identidad fijada de Marin y la evidencia. Se considera terminado cuando la línea base, el corpus personalizado, los experimentos controlados, las evaluaciones, los gráficos, los fallos, los registros de licencias y el teach-back cumplen los criterios de aceptación.