TraderX State 015 Proposal: Historical Synthetic Market Data and Charting

# TraderX State 015 Proposal: Historical Synthetic Market Data and Charting

## Context

We are working in the FINOS TraderX repository and Spec-Kit state model.

The current state graph includes:

- 008 Pricing Awareness and Market Data Streaming
- 009 Order Management and Matcher (C2 Advanced Demo)
- 010 Kubernetes Runtime
- 011 Tilt Kubernetes Dev Loop
- 012 Platform Convergence C3
- 013 Radius Platform (optional)
- 014 FDC3 Intent Interoperability

This proposal should become a new functional state:

**015 Historical Synthetic Market Data and Charting**

The new state should branch from:

```text
012-platform-convergence-c3
```

similar to how 014 branches from 012.

This feature is intended to improve the realism, educational value, and demonstration capabilities of TraderX by introducing historical market data, charting, volatility analytics, and synthetic price history generation.

The implementation must follow the existing TraderX state-generation architecture and should be implemented as a proper Spec-Kit state rather than a one-off code change.

---

# High-Level Goals

TraderX currently publishes synthetic market prices and supports pricing-aware order management.

We want to introduce:

1. Historical price storage
2. Intraday charting
3. Historical charting
4. Volatility calculations
5. High/low statistics
6. Deterministic synthetic history generation
7. Optional seeding from public market data
8. Market-data services that are architecturally separated from the live price publisher
9. Future support for more advanced execution and fill simulations

The result should feel much closer to a realistic trading platform while remaining completely synthetic and safe for educational/demo use.

---

# State Placement

Create a new state:

```text
015-historical-synthetic-market-data
```

Suggested metadata:

```yaml
track: functional
previous:
- 012-platform-convergence-c3
primaryLineageRole: canonical
convergenceLevel: none
isConvergence: false
```

This should not modify the intent of state 009.

Instead, it should layer new functionality on top of the converged platform.

---

# Architectural Principles

## Principle 1: Do Not Put Persistence Inside The Price Publisher

The existing price publisher should remain responsible for:

- generating synthetic prices
- publishing price events
- broadcasting market activity

The price publisher must NOT become responsible for:

- database writes
- historical persistence
- chart generation
- analytics calculations
- backfill generation

---

## Principle 2: Use NATS As The Boundary

All market-data persistence should occur downstream of NATS.

Recommended architecture:

```text
price-publisher
|
v
NATS
|
+--------------------+
| |
v v

live consumers market-data-recorder
|
v
market-data-db
```

The recorder should subscribe to the same pricing events already emitted by the system.

---

## Principle 3: Historical Data Must Be Optional And Non-Invasive

If market-data persistence fails:

- pricing should continue
- order matching should continue
- the platform should remain operational

Historical storage is additive.

It must not become a dependency of the trading path.

---

# New Components

## Market Data Recorder

Create a new service:

```text
market-data-recorder
```

Responsibilities:

- Subscribe to market price events
- Batch writes
- Persist ticks
- Generate OHLC bars
- Calculate high/low statistics
- Calculate volatility metrics
- Serve as source-of-truth for chart data

Non-responsibilities:

- Publishing live prices
- Matching orders
- Managing accounts

---

## Market Data Bootstrapper

Create a new startup component:

```text
market-data-bootstrapper
```

Responsibilities:

- Execute on clean startup
- Load the TraderX security universe
- Obtain baseline prices
- Generate synthetic historical data
- Populate the market-data database

The bootstrapper should be idempotent.

Repeated startups should not endlessly duplicate data.

---

## Market Data API

Create a new API surface:

```text
market-data-api
```

Responsibilities:

- Serve charts
- Serve history
- Serve statistics
- Serve metadata
- Serve provenance information

---

# Storage Design

Evaluate the best implementation.

## Option A (Preferred)

Separate TimescaleDB instance.

Reasons:

- PostgreSQL compatibility
- Time-series optimization
- Easy chart queries
- Easy future scaling

## Option B

Separate PostgreSQL instance.

Design schema so migration to TimescaleDB is trivial later.

---

# Data Model

## price_tick

Fields:

```text
symbol
asset_class
timestamp
price
volume
bid
ask
source
synthetic
generated_run_id
```

## price_bar_1m

```text
symbol
bucket_time
open
high
low
close
volume
```

## price_bar_5m

Optional.

Can be materialized or generated from 1m bars.

## price_bar_1d

```text
symbol
date
open
high
low
close
volume
```

## security_price_profile

```text
symbol
baseline_price
current_price
all_time_high
all_time_low
volatility
last_seeded_at
baseline_source
```

---

# Historical Generation Requirements

## Clean Startup

When a clean environment starts:

1. Load securities
2. Seed baseline prices
3. Generate synthetic history
4. Start live publishing

---

## Public Baseline Pricing

Attempt to retrieve realistic recent prices.

Examples:

- Yahoo Finance compatible feeds
- AlphaVantage
- Stooq
- Other public/open market data providers

Implementation should be pluggable.

Failure to reach a provider must not break startup.

---

## Offline Mode

Support fully offline operation.

Provide:

```text
seed-prices.json
```

or equivalent reference data.

If no public source is available:

- use local seed prices
- continue startup

---

## Deterministic Mode

Support:

```text
TRADERX_MARKETDATA_SEED=12345
```

or equivalent.

The same seed should generate identical synthetic history.

This is valuable for:

- demos
- workshops
- tutorials
- automated tests

---

# Synthetic Price Generation

Generate 2–5 days of synthetic history.

The generated history should:

- trend realistically
- exhibit volatility
- avoid obviously random behavior
- preserve continuity

Suggested inputs:

```text
baseline price
volatility factor
daily drift
random seed
```

Avoid pure uniform random walks.

Prefer realistic market-like movement.

Potential enhancements:

- geometric Brownian motion
- volatility clustering
- mean reversion
- regime changes

---

# Future Asset Classes

Design for future expansion:

```text
equities
etfs
indices
crypto
fx
bonds
commodities
```

Do not hardcode equity assumptions into storage schemas.

---

# Charting Requirements

Add chart support to the Angular UI.

Every priced security should support:

```text
View Chart
```

or equivalent.

## Intraday View

Display:

- live price
- intraday history
- high
- low

## Historical View

Display:

- daily history
- multi-day history
- trend

## Statistics

Display:

- volatility
- high
- low
- current price

---

# Disclaimer

Every chart and market-data view should clearly display:

> Prices shown are synthetic simulation data. Initial baselines may be seeded from public market data, but generated prices do not represent actual market prices, executable quotes, or investment information.

---

# Retention Strategy

## Raw Ticks

Retain:

```text
7–30 days
```

## 1 Minute Bars

Retain long-term.

## Daily Bars

Retain indefinitely.

---

# Performance Requirements

Historical persistence must never become a bottleneck.

Requirements:

- Batch inserts
- Back-pressure handling
- Async persistence
- Recorder failures isolated from trading services
- Recorder restart safe
- Bootstrap restart safe

---

# Future Roadmap

Design this state so future states can support:

## Advanced Fill Simulation

Examples:

- VWAP fills
- TWAP fills
- Slippage
- Market impact
- Liquidity curves

## Strategy Simulation

Examples:

- Trend following
- Mean reversion
- Momentum

## Market Replay

Ability to replay historical synthetic sessions.

---

# Deliverables

Produce:

1. State 015 specification
2. State graph update
3. ADR describing architecture decisions
4. Market-data-recorder service
5. Market-data-bootstrapper service
6. Market-data-api service
7. Database schema
8. Docker Compose changes
9. NATS integration
10. Angular chart integration
11. Synthetic price generation design
12. Public-price seeding design
13. Test plan
14. Documentation
15. Migration strategy

---

# Expected Design Review Output

Before implementation:

1. Inspect existing state-generation conventions.
2. Identify the minimal set of generated assets required.
3. Propose architecture.
4. Identify risks.
5. Present implementation plan.
6. Explain why the chosen database architecture is appropriate.
7. Explain how the design remains compatible with future execution simulation enhancements.
8. Only then begin implementation.

---

# Success Criteria

A newly started TraderX environment should:

- Display realistic-looking charts immediately after startup.
- Have 2–5 days of historical synthetic pricing available.
- Continue generating live synthetic prices.
- Persist history without affecting trading-path performance.
- Support future expansion into execution simulation and market replay.
- Remain fully functional when external market-data providers are unavailable.
- Clearly indicate that all displayed prices are synthetic.

Start by inspecting the existing state-generation conventions and compare state 014 with the 012-platform-convergence-c3 branch. Then map the requested market-data-recorder, market-data-bootstrapper, market-data-api, Angular chart integration, schema, and Docker Compose changes. Done means a reviewed State 015 specification, architecture decision record, implementation plan, tests, documentation, and migration strategy.

Written by the indexing model from the issue text.