cardano-foundation / cardano-foundation/cardano-rosetta-java

refactor: merge API and yaci-indexer into a single "rosetta-app" module

Open
#731 0 comments 0 reactions 0 assignees View on GitHub
Non Functional Requirement Tech Excellence
Dominant language
Java
Stars
26
Forks
15
Avg merge
5d 3h
Merged PRs (30d)
2

Description

# refactor: merge API and yaci-indexer into a single "rosetta-app" module

Ref: https://github.com/cardano-foundation/cardano-rosetta-java/issues/731

---

## Problem

Today cardano-rosetta-java runs as two Spring Boot services:

- **`yaci-indexer`** — owns the JPA entities + repositories for `block`, `transaction`, `address_utxo`, `metadata_reference_nft`, `ft_offchain_metadata`, and all other yaci-store tables. This is the **write side** — it consumes Cardano node events and persists them via yaci-store's storage modules.
- **`api`** — the Rosetta-spec HTTP endpoints. Reads the same tables the indexer writes, but because it doesn't pull in yaci-store's storage jars, it **re-defines its own JPA entities and repositories** that map to the same column names.

This means every yaci-store schema used by a Rosetta endpoint is duplicated:

- `AddressUtxoEntity` (api) ↔ yaci-store's utxo module entity
- `BlockEntity` / `TxnEntity` / `TxInputEntity` (api) ↔ yaci-store's blocks module entities
- `PoolRegistrationEntity` / `PoolDelegationEntity` / `WithdrawalEntity` (api) ↔ yaci-store staking module
- `TokenMetadataEntity` / `TokenLogoEntity` / `MetadataReferenceNftEntity` (api) ↔ yaci-store assets-ext entities — added in #730
- And so on for epoch params, governance, etc.

Plus duplicate `*Repository` interfaces and, in some cases, duplicate custom native-SQL queries (e.g. the CIP-68 window-function batch query added in #730 exists both downstream in `MetadataReferenceNftRepositoryCustom` and upstream in `yaci-store-assets-ext`).

---

## Why we can kill the duplication

yaci-store supports **read-only mode** via `store.read-only-mode=true` — listeners/processors/cron jobs are skipped, repositories are still registered, storage reader beans are still created. That's exactly the shape the Rosetta API needs.

With that knob, a **single Spring Boot application — `rosetta-app`** — can:

- Run yaci-store in **write mode** (sync blocks from the node, populate all tables) on the indexer node(s)
- Run the **same application** in **read-only mode** on the API node(s) — consuming yaci-store's `Cip26StorageReader`, `Cip68StorageReader`, `AddressUtxoStorageReader`, etc. directly without its own entity/repository layer

No behavioral difference for operators — the mode is a single config flag. Horizontal scaling is unchanged (you still run N api replicas + M indexer replicas, they just share the same jar).

---

## Benefits

1. **No more duplicate JPA entities.** Schema changes in yaci-store stop being "downstream code must be manually updated" — they flow through the jar dependency.
2. **No more duplicate queries.** The CIP-68 batch window-function query in #730 was written twice (custom fragment downstream + `@Query` upstream). Post-merge we reuse upstream's `findBySubjects` directly.
3. **Single build, single image.** Operators deploy one `rosetta-app` jar with two config profiles instead of two separate artifacts.
4. **Smaller surface for drift bugs.** Right now a rename or column-type change in yaci-store-assets-ext can silently break the API's custom entities until runtime — there's no compile-time or build-time guardrail. After the merge, compilation against the yaci-store jar catches it.
5. **Simpler CI.** One module to test, not two.

---

## Naming

The merged module will be called **`rosetta-app`** — a single Spring Boot application that runs either in Rosetta HTTP read mode (default) or in yaci-store write mode (indexer profile).

---

## What PR #746 already completed (groundwork for this merge)

PR #746 moved index creation natively into `yaci-indexer` and established clean
ownership boundaries. The following is already done:

- **`indexes` package** in yaci-indexer owns the full index lifecycle:
`IndexService`, `PgIndexService`, `NoOpIndexService`, `IndexCatalog`,
`IndexesEndpoint`, `IndexStallIndicator`
- **`index-applier` sidecar deleted** — Docker Compose file, Helm Job + ConfigMap
templates, `apply-indexes.sh` shell script, `jq`/`yq` from Postgres Dockerfile
- **Single `db-indexes.yaml`** in yaci-indexer drives both index creation and the
api's index readiness check (kept identical via `IndexCatalogParityTest`)

---

## Index service migration (specific steps at merge time)

The api has its own parallel index monitoring stack that duplicates logic already
in the yaci-indexer's `indexes` package. At merge time, consolidate as follows:

### 1. Wire `IndexService` into `SyncStatusService`

```java
// BEFORE — api's SyncStatusService
@RequiredArgsConstructor
public class SyncStatusService {
private final IndexCreationMonitor indexCreationMonitor;
...
boolean indexesNotReady = indexCreationMonitor.isCreatingIndexes();
}

// AFTER — direct injection of IndexService
@RequiredArgsConstructor
public class SyncStatusService {
private final IndexService indexService;
...
boolean indexesNotReady = indexService.getState() != IndexLifecycleState.READY;
}
```

`getIndexCreationProgress()` (used for detailed status) maps to
`indexService.getIndexStatus()` → `List`, which is richer than
the old `List` (4 always-null numeric fields).

### 2. Delete from api

| Class / file | Replacement |
|---|---|
| `IndexCreationMonitor` (interface) | `IndexService` |
| `PostgreSQLIndexCreationMonitor` | `PgIndexService` |
| `H2IndexCreationMonitor` | `NoOpIndexService` |
| `RosettaIndexConfig` | `IndexCatalog` |
| `api/.../config/db-indexes.yaml` | Single copy in `yaci-indexer` |
| `IndexCatalogParityTest` | No longer two files to compare |

Also remove the `spring.config.import` line from api's `application.yaml`:
```yaml
# REMOVE this line
- classpath:config/db-indexes.yaml
```

### 3. Profile alignment

Both modules already use the same profile names — no changes needed:

| Profile | yaci-indexer (keep) | api (delete) |
|---------|---------------------|--------------|
| `h2`, `test-integration` | `NoOpIndexService` | `H2IndexCreationMonitor` |
| everything else | `PgIndexService` | `PostgreSQLIndexCreationMonitor` |

---

## Scope of work

- [ ] Create the new `rosetta-app` module by merging `api` and `yaci-indexer`
- [ ] Add `com.bloxbean.cardano:yaci-store-*` dependencies to `rosetta-app`
- [ ] Introduce a Spring profile `indexer` that enables write-mode yaci-store; default `rosetta-app` profiles set `store.read-only-mode=true`
- [ ] **Index service migration** (detail above):
- [ ] Wire `IndexService` into `SyncStatusService`
- [ ] Delete `IndexCreationMonitor` + both implementations
- [ ] Delete `RosettaIndexConfig` from api (`getIndexCommands()` was already dead code)
- [ ] Delete `api/src/main/resources/config/db-indexes.yaml`
- [ ] Delete `IndexCatalogParityTest`
- [ ] Replace api-side `*Entity` + `*Repository` classes with references to yaci-store's equivalents, module by module:
- [ ] account / address_utxo → `yaci-store-utxo`
- [ ] block / transaction → `yaci-store-blocks` + `yaci-store-transaction`
- [ ] staking (pool, delegation, withdrawal) → `yaci-store-staking`
- [ ] governance → `yaci-store-governance`
- [ ] epoch params → `yaci-store-epoch`
- [ ] token metadata (CIP-26 / CIP-68) → `yaci-store-assets-ext` (**blocked on #730 release**)
- [ ] Move yaci-indexer's `application.properties` contents into `rosetta-app` under the `indexer` profile
- [ ] Delete the `api` and `yaci-indexer` modules
- [ ] Update Docker Compose / Helm charts to run one `rosetta-app` image with two different `SPRING_PROFILES_ACTIVE` values (`indexer` vs default)
- [ ] Update the README deployment docs
- [ ] Regression test: full sync → full Rosetta API test pass against the merged `rosetta-app` binary

---

## Dependencies

- `yaci-store-assets-ext` must be released before the token-metadata portion can be cleaned up (currently consumed as a pre-release SNAPSHOT via #730)
- Any custom api queries that beat yaci-store's built-in query shape should be reviewed — if they're genuinely faster, upstream them (same model as the CIP-68 batch query in #730)

---

## Related

- #730 — introduces `yaci-store-assets-ext` + duplicate JPA entities for CIP-26/CIP-68 metadata tables. Flagged in that PR as tech debt to be cleaned up by this refactor.
- #746 — replaces `index-applier` sidecar with native `yaci-indexer` index lifecycle; cleans up package/class naming. Groundwork for this merge.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the api and yaci-indexer module build files and their application configuration, then review SyncStatusService, the indexes package, and the duplicated entity and repository classes listed in the scope. Check the yaci-store module dependencies and the blocked yaci-store-assets-ext release before planning the merge. Done means one rosetta-app binary passes full-sync and Rosetta API regression tests, with Docker Compose, Helm, and deployment documentation updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, helm, java, postgresql, spring-boot
Domain
backend, build-system, databases, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.