OpenRailAssociation / OpenRailAssociation/osrd

editoast: create high performance batched pathfinding endpoints

Open
#14,768 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:editoast kind:technical-enhancement module:stdcm
Dominant language
TypeScript
Stars
660
Forks
87
Avg merge
3d 4h
Merged PRs (30d)
269

Description

The osrd-data script responsible for generating reference trains (used by the similar trains feature in stdcm) makes a lot of requests (~100 000) to 2 endpoints:

  • /infra/{infra_id}/pathfinding/blocks
  • /infra/{infra_id}/path_properties

This process is very slow, taking multiple hours. One way to speed it up would be to batch these requests, requiring a batched version of these endpoints (or of a single endpoint grouping these 2?).

This process also leads to continuous ram scaling of the postgres processes, which makes it very ressource intensive (and leads to crash when running the script locally). If possible, it would be nice to limit the postgres usage of these new high performance endpoints.

The only output we actually use in osrd-data currently is the list of op ids encountered. Perhaps limiting the output to this could help?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the osrd-data script that generates reference trains and trace its calls to /infra/{infra_id}/pathfinding/blocks and /infra/{infra_id}/path_properties. Review how the script uses the returned operation IDs and measure PostgreSQL resource usage; done means a batched endpoint or grouped endpoints complete this workflow with substantially fewer requests and controlled memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres
Domain
backend-api-design, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.