Proposal to move out GraphQL API from `fuel-core` to `fuel-indexer`
- Dominant language
- Rust
- Stars
- 56.8k
- Forks
- 2.9k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 5
Description
# Overview
The proposal was discussed with the `fuel-indexer` team and accepted. You can find the proposal's description at the end of this description. The beginning of its description tracks the progress of the implementation.
---
# The progress of the implementation
- [x] Migrate all internal graphql query resolver db logic into a plain rust api for the FuelService. The graphql resolvers should just be a minimal wrapper around our Rust APIs and perform minimal logic.
- [x] https://github.com/FuelLabs/fuel-core/pull/858
- [x] https://github.com/FuelLabs/fuel-core/pull/909
- [x] #1069
- [x] https://github.com/FuelLabs/fuel-core/pull/875
- [ ] Extract related folders and field into a separate crate.
- [x] Clean up the `fuel-core-client` by removing any graphql/cynic types in the publicly facing client interface and replacing them with the plain and ergonomic domain types in `fuel-core-types` https://github.com/FuelLabs/fuel-core/issues/378
- [ ] Implement simple rpc-based API for `fuel-core` with minimal functionality.
- [ ] Re-use the `fuel-core-graphql-api` in the `fuel-indexer`.
- [ ] Move `fuel-core-client` into `fuel-indexer` repository.
- [ ] Update all downstream crates to work with `fuel-indexer`.
## Unresolved questions
- Do we need to move debugger API into `fuel-indexer`?
## Child tickets:
- https://github.com/FuelLabs/fuel-core/issues/1127
- https://github.com/FuelLabs/fuel-core/issues/1128
- https://github.com/FuelLabs/fuel-core/issues/1126
- https://github.com/FuelLabs/fuel-core/issues/1122
- https://github.com/FuelLabs/fuel-core/issues/1123
- https://github.com/FuelLabs/fuel-core/issues/1124
- https://github.com/FuelLabs/fuel-core/issues/1125
- https://github.com/FuelLabs/fuel-core/issues/1129
- https://github.com/FuelLabs/fuel-core/issues/1130
---
# The initial proposal
## Summary
The `fuel-core` implements CRUD functionality, interaction with the user, indexation, subscription on events(not implemented yet), and implementation of some specific queries. All this API functionality is not **required** for the blockchain.
This issue is a place where we can discuss which parts of the API functionality we want to move out and what not. Where do we want to implement moved-out functionality, how, and who?
It may affect other teams(indexer team, dev-ops team, documentation team), so they also may, and **should** participate in the final decision.
## Motivation
Moving out API functionality brings benefits along several different axes:
* **Performance**: The client team can focus on the architecture and optimization of the blockchain itself. The API code also will have its architecture for optimal code in a separate repository.
* **Code-complexity**: The code doesn't mix blockchain functionality with API. It makes the code clear and entirely focuses on the main task - smart contract execution.
* **Parallelisation of work**: Changes can be done in parallel by different teams with some synchronization steps regarding breaking changes(p2p protocol, entity layout).
### Performance
Right now, many supported queries are done in a non-optimal way(`O(N)`, `O(N^2)`, `O(N^2 * log(N))`). It is because we don't have indexes for some fields or pairs of fields. It is a vulnerability for the mainnet because anyone with a few requests can shut down/block/overload the node(you can create 1000 dust UTXO and send several queries to get a spendable balance).
Almost all `input` queries require iteration over all owner's entities in the database with a load of all fields and filtering(by asset id or spent status) them in runtime.
Example of owned_coins_by_asset_id

We can manually add indexes into `RocksDB`, but it:
- Makes the development slower and forces us to create our database with indexes(and we still will not be able to make them so performance as indexes in the RDB).
- This means introducing the index support on the `Database` layer, decreasing the node's performance.
- Each **exotic** query can require more work and architecture changes to work optimally.
Using the database that supports the creation of indexes solves many problems, but it will be a pain for us to manage all schemas for different databases.
### Code-complexity
Integration of API into other parts of the blockchain breaks the abstraction and requires creating workarounds or writing non-optimal code.
If the API functionality is part of the blockchain, it creates the following issues:
- The architecture should worry about information(transactions/blocks/entities) propagation to the API service from other services.
- Database management is more complex and requires more abstractions(because we need to connect `RocksDB` with another database).
- The whole API module should be configurable.
If the API is a separate node/binary/daemon, it can have its logic on how to process each block and transaction. It can, optimally, update all tables/entities in the database and use proper indexes(and manage them) to support all required queries.
The entry point may be the typical `fuel-p2p` service. But except for the p2p service, it also will contain Rest API to work with users.
Another good point is that API is independent of the blockchain node, and it is not a problem to make changes in the DB schema to support a new query or data format.
### Parallelisation of work
The proper and actual API is essential for a good development experience and requires the introduction of many endpoints. The client team is more focused on the blockchain changes first, so unable to deliver it in time. With a separate API node/indexer, we can have another team that is entirely focused on that.
It makes the development parallelizable from the beginning. Of course, those two teams should sync periodically about breaking changes(p2p protocol, entity layout), and the API team will depend on changes in the `fuel-core`. It forces the development of `fuel-core` to be more modular and act as a crate more than the package. But the dependency will be only on several services, so we can minimize the number of conflicts.
## Possible solution
Fuel already has a particular product [indexer](https://github.com/FuelLabs/fuel-indexer), that indexes blocks, transactions, and events, and the team grows(human resources). The indexer uses SQL database under the hood and supports indexation. Also, in the future, it can be our [full history node](https://github.com/FuelLabs/fuel-core/issues/129) and support timeline queries.
The indexer auto-generate DB schemas based on the entities description from handlers. But it doesn't have schemas for common data used by the blockchain. Moving API to them also means starting indexation and storing blocks/transactions/data required for the queries. As a consequence, support of additional schemes in the database with manually specified indexes based on the list of supported queries(as an example [`coinsToSpend`](https://github.com/FuelLabs/fuel-core/issues/586)).
#### Possible problems
- The `fuel-core` should be modular enough to easily integrate with the indexer. Right now, indexers use `fuel-core` to query blocks and receipts. Maybe it should receive all information from the p2p service in the future as any peer node in the network. Also, traits used in the `fuel-vm`(one of the indexer's dependencies) should be as generic as possible to cover the work with different databases.
- We need to move most queries to the indexer, and maybe the current architecture is not ready. It requires additional work on the indexer side(maybe someone from the client team should do that).
- The relationship between indexer and `fuel-core` will be tight because data layout will affect them too. Someone from the client team should be involved in the indexer's processes to notify them/us about upcoming/required changes.
- It makes UX for developers harder because they need to install the `fuel-core` node together with the indexer. Maybe we need to think about one binary that will include the functionality of both products for a simple setup.
- It may require updating the integration tests, maybe infrastructure/deployment/CI.
#### Unresolved questions
1. Which GraphQL queries do we want to process in the `fuel-core`, and which do we want to move out?
1. Do we need to support **some** GraphQL queries on the `fuel-core`? Or better make it a fully p2p node without any endpoint?
1. How better to start the transition, and who will work on it?
1. Who is responsible for coordinating future work with the indexer team(like breaching change notifications/discussions)? Do we need a coordinator?
1. May it somehow affect future integration with the bridge and relayer?
1. What is the approximate scope of the work? Can we finish it before the mainnet and update all repositories/documentation?
Contributor guide
Research direction
Start by reviewing the implementation progress, unresolved questions, and child tickets #1122–#1130; no specific files or tests are named. Map the remaining checklist items across fuel-core and fuel-indexer, then confirm the migration is complete when the RPC API, shared GraphQL API, client move, and downstream updates are finished.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, rust
- Domain
- api, backend-api-design, blockchain
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100