mapbox / mapbox/nepomuk

Implement Data Provider

Open
#154 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Backend Feature Request
Dominant language
C++
Stars
24
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Depending on the amount of data / processing required, starting up a server can be a long process.
Especially for testing our server infrastructure, we would like to avoid unnecessary overheads.

To do so (and as already outlined in https://github.com/mapbox/nepomuk/issues/3), we can think about implementing a data provider to supply data to our engine without a need to recreate the datasets after a failure / for any new startup of the engine.

As outlined in https://github.com/mapbox/nepomuk/issues/3, we should seriously consider a process that does not used shared memory but rather communication via zeromq. The reasoning here is that this process would allow for us to avoid any locking problems that come along with shared memory regions and to allow distribution of workloads onto different machine types.

A data provider should take the role of the what is currently handled in the `master-service::dataset`. Instead of having a dataset and allowing the creation of different data structures, the master service should hand this responsibility to a `data provider` that can be located anywhere. The master service handles the communication with the `data provider` and returns structures as we are used to.

To do so, we need to serialise all structures into `PBF` and deserialised them from `PBF`. The Provider should offer the functionality to load a raw GTFS feed from disk and put it into `PBF`. The `PBF` has then to be transferred via ZeroMQ to the `MasterService` which, in turn, returns the access as usual, hiding all the ZeroMQ shenanigans from the rest of the project.

- [ ] add PBF serialisation / deserialisation to all timetable data structures / look-up data structures
- [ ] create data provider to be started up before any server can be started (think shared memory loading in OSRM), loading data and creating pbf-forms of all structures
- start off with a create-all approach?
- or still create on demand (might need to add additional pbf -> datastructure steps)
- [ ] add communication possibilities to data provider
- [ ] replace master service data-loading and creation with request + pbf->datastructure conversion

/cc @daniel-j-h

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading issue #3 and the current master-service::dataset implementation to understand the existing loading and lookup flow. Then map the timetable and lookup structures that need PBF serialization, the raw GTFS loading path, and the proposed ZeroMQ boundary. Done means the provider can load data, communicate with MasterService, and preserve the existing access behavior without shared-memory setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.