Prioritize encoder refinement after decoder-capacity saturation

Motivation

The Phase-2 price decoder results and the baseline epoch sweeps raise one connected question:

Are downstream results limited by decoder capacity, or are larger decoders simply overfitting a persistence-dominated target using frozen representations that have already lost important current-price information?

This issue combines two observations:

1. Increasing decoder capacity from D0 to residual, gated, recurrent, and attention-based designs has not produced a reliable price-prediction improvement.
2. Several models achieve their strongest generalization at an earlier epoch even while training loss continues to decrease.

Observation 1: decoder refinement has not reliably improved price prediction

All completed Phase-2 decoder runs use the same frozen five-branch features, contract-safe next-close labels, K=8 temporal row contract, and 27,100 test rows. D0--D2 have three seeds; D3 currently has two complete seeds because the interrupted third seed is intentionally being preserved rather than rerun.

Mean test RMSE across available seeds:

Decoder	Structure	e15	e50	e100	Current interpretation
D0	shallow concat MLP	0.0934	0.0824	0.0836	strongest reliable RMSE result
D1	branch-aware residual MLP	0.1008	0.0849	0.0897	extra capacity does not reliably beat D0
D2	task-trained gated fusion + MLP	0.1087	0.1023	0.1042	worse and seed-sensitive
D3	temporal LSTM, K=8	0.1089	0.0960	0.1042	no temporal gain on two complete seeds

D1 sometimes improves MAE, particularly around epoch 50, while worsening RMSE. This suggests a possible center-versus-tail trade-off rather than a uniform improvement.

Preliminary D4 status

D4 is being completed for price only. At the time of this update, seed 0 is complete:

D4 seed 0	e15	e50	e100
MAE	0.0682	0.0549	0.0556
RMSE	0.1115	0.0890	0.0870

This single seed improves with training but still does not beat the simpler D0 aggregate. It is not a completed multi-seed D4 result; seed 1 is partial and seed 2 is pending.

Observation 2: longer training has mixed and often negative effects

Model / task	Earlier checkpoint	Epoch 50	Epoch 100	Observed pattern
Stacked LSTM price (RMSE)	0.0157 at e20	0.0185	0.0167	early optimum; late instability
Raw-OHLCV MLP price (RMSE)	0.1067 at e15	0.0805	0.0683	continues improving
Phase-1 framework price (RMSE)	0.0908 at e15	0.0993	0.1009	degrades while train loss falls
Raw-OHLCV MLP volatility (RMSE)	0.0931 at e15	0.0948	0.1001	degrades
Raw LSTM volatility (RMSE)	0.1086 at e15	0.0981	0.1031	intermediate budget strongest
GARCH--LSTM stack volatility (RMSE)	0.0890 at e15	0.0865	0.0834	continues improving

The TA-MLP classification sweep is nearly flat: macro-F1 remains approximately 0.45--0.47 from 15 to 100 epochs. These results do not support one universal best epoch count.

Why the observations may be connected

Frozen-representation bottleneck

The decoder only receives frozen Phase-1 branch representations. If they do not preserve the exact current close or the information needed to forecast the next movement, a deeper decoder cannot reconstruct information absent from its input.

Persistence-dominated target

The task predicts the absolute next 4-hour close. Approximately 52% of evaluated rows have an unchanged next close, and only a small fraction make large moves. The persistence reference (prediction = current close) achieves approximately MAE 0.00458 / RMSE 0.01473, stronger than the trained price models.

The raw stacked LSTM sees the latest close directly and learns a persistence-like mapping. The frozen framework representations have no explicit current-close skip path, likely explaining much of the visual and metric gap.

Capacity may fit representation noise

D1--D4 add projections, residual layers, gating, recurrence, or attention. Because K=8 contexts contain heavily overlapping windows, much of the temporal input may be redundant. Additional capacity may fit branch-specific or contract-specific noise instead of adding forecast information. Falling training loss with flat or worse test metrics is compatible with overfitting, although multiple seeds and paired intervals are needed for a causal claim.

Raw epoch counts are not comparable

On approximately 109,840 training rows:

• Stacked LSTM: batch 64, learning rate 1e-3; 50 epochs is about 85,800 optimizer updates.
• Raw-OHLCV MLP: batch 128, learning rate 1e-4; 100 epochs is about 85,800 updates.
• Framework/Phase-2 decoders: batch 512, learning rate 1e-4; 100 epochs is only about 21,500 updates.

Equal epoch numbers do not equalize optimization exposure or compute. Capacity and epoch effects cannot be separated cleanly without recording and matching optimizer steps.

Methodological constraint

The current 15,50,100 results have already been evaluated on the locked test split. They must remain characterization evidence and be reported in full. We must not retrospectively choose the lowest-test-error decoder, epoch, regularization value, or learning-rate schedule as the official model. Any follow-up below must be declared as a new experiment before test evaluation.

Revised research direction

Decoder refinement will be closed after preserving the evidence already produced. No additional decoder architectures, decoder-width searches, temporal-context searches, or decoder-specific regularization sweeps are planned. The next research focus is encoder refinement.

The decoder experiment remains useful as a negative diagnostic result: adding downstream capacity did not reliably recover the missing predictive information. The follow-up should therefore ask whether the frozen representations preserve the information required by the downstream tasks.

Proposed plan

Stage 1: close and document decoder refinement

• Preserve all existing D0--D4 artifacts under experiments/framework/phase2/decoder_refinement_1/4h_k8/.
• Do not rerun or overwrite the interrupted D3 seed 2.
• Do not launch volatility or classification decoder matrices.
• Allow only the already-launched D4 price work to finish; do not add further decoder runs afterward.
• Aggregate the available price results with the exact seed count stated for every cell.
• Record the conclusion conservatively: increased decoder capacity did not provide a reliable improvement over D0 and sometimes increased overfitting or seed sensitivity.

This stage is archival/reporting work, not another decoder-development cycle.

Stage 2: diagnose what the current encoders preserve

Keep the downstream probe fixed and lightweight. Before designing a new encoder, measure representation sufficiency using training-only probes and the existing locked evaluation contract:

• reconstruct the current close from each frozen branch and from the five-branch concat representation with a linear probe;
• probe recent price change, next-step delta, volatility, and movement class separately;
• run single-branch and leave-one-branch-out ablations;
• measure embedding variance, redundancy, collapse, and temporal smoothness;
• inspect whether contrastive/BYOL augmentations remove absolute price level or short-horizon movement information;
• report persistence beside every next-close result.

These diagnostics should identify whether the problem is level information, short-horizon dynamics, branch redundancy, or a particular neural encoder objective.

Stage 3: define encoder-refinement candidates

Refine representation learning rather than decoder complexity. Candidate directions should be predeclared after the diagnostics, for example:

• endpoint-aware reconstruction so the embedding preserves the latest OHLC values;
• masked-timestep or masked-patch prediction to retain local temporal structure;
• multi-scale temporal objectives covering level, short movement, and volatility;
• revised contrastive/BYOL augmentations that do not erase economically relevant price scale;
• an encoder with separate level-sensitive and scale-invariant subspaces;
• a compact temporal Transformer/masked autoencoder as a new frozen representation branch, if supported by the literature and compute budget.

Any auxiliary targets must be derived only from the training sequence itself or from training rows. They must not use test labels or test-selected hyperparameters.

Stage 4: isolate encoder quality with a fixed decoder

• Use D0/concat plus the existing shallow task head as the sole primary probe for encoder comparisons.
• Keep label bundles, aligned rows, feature scaling, head architecture, optimizer policy, and seeds unchanged across encoder candidates.
• Compare each refined encoder against the current five-branch representation and appropriate raw-input baselines.
• Use the same frozen encoder across price, volatility, and movement classification to test transferability.
• Report absolute metrics, multi-seed variation, paired contract-level intervals, parameter count, pretraining time, and inference cost.

Using one fixed probe ensures that any improvement is attributable to the representation rather than to a stronger downstream decoder.

Stage 5: revise training budgets without test-driven selection

The epoch-count observation remains relevant to encoder and probe training:

• record cumulative optimizer steps, examples seen, batch size, and learning-rate trajectory;
• define short/medium/long optimizer-step budgets before executing each new study;
• inspect training-only loss and embedding diagnostics for convergence;
• report every predeclared test snapshot instead of selecting the lowest test error;
• treat the existing 15/50/100 results only as characterization evidence motivating the new protocol.

Stage 6: evaluate task formulation after encoder diagnostics

For price prediction, retain the completed absolute next-close task for continuity, but add a separately declared diagnostic for future delta/return information. An explicit current-close residual formulation may be considered later:

predicted next close = current close + predicted delta

This is a task/input-contract change, not decoder refinement, and should not replace the completed experiment retroactively.

Start by reviewing the preserved artifacts under experiments/framework/phase2/decoder_refinement_1/4h_k8/ and the existing D0/concat probe protocol. Close the already-launched D4 price work, retain exact seed counts, and use the proposed representation diagnostics to define encoder candidates. Done means decoder refinement is archived and a predeclared, fixed-probe encoder study protocol is documented without test-driven selection.