OpenRailAssociation / OpenRailAssociation/osrd
editoast: create high performance batched pathfinding endpoints
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 660
- Forks
- 87
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 269
Description
The osrd-data script responsible for generating reference trains (used by the similar trains feature in stdcm) makes a lot of requests (~100 000) to 2 endpoints:
- /infra/{infra_id}/pathfinding/blocks
- /infra/{infra_id}/path_properties
This process is very slow, taking multiple hours. One way to speed it up would be to batch these requests, requiring a batched version of these endpoints (or of a single endpoint grouping these 2?).
This process also leads to continuous ram scaling of the postgres processes, which makes it very ressource intensive (and leads to crash when running the script locally). If possible, it would be nice to limit the postgres usage of these new high performance endpoints.
The only output we actually use in osrd-data currently is the list of op ids encountered. Perhaps limiting the output to this could help?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the osrd-data script that generates reference trains and trace its calls to /infra/{infra_id}/pathfinding/blocks and /infra/{infra_id}/path_properties. Review how the script uses the returned operation IDs and measure PostgreSQL resource usage; done means a batched endpoint or grouped endpoints complete this workflow with substantially fewer requests and controlled memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres
- Domain
- backend-api-design, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100