Implement Data Provider
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 24
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Depending on the amount of data / processing required, starting up a server can be a long process.
Especially for testing our server infrastructure, we would like to avoid unnecessary overheads.
To do so (and as already outlined in https://github.com/mapbox/nepomuk/issues/3), we can think about implementing a data provider to supply data to our engine without a need to recreate the datasets after a failure / for any new startup of the engine.
As outlined in https://github.com/mapbox/nepomuk/issues/3, we should seriously consider a process that does not used shared memory but rather communication via zeromq. The reasoning here is that this process would allow for us to avoid any locking problems that come along with shared memory regions and to allow distribution of workloads onto different machine types.
A data provider should take the role of the what is currently handled in the `master-service::dataset`. Instead of having a dataset and allowing the creation of different data structures, the master service should hand this responsibility to a `data provider` that can be located anywhere. The master service handles the communication with the `data provider` and returns structures as we are used to.
To do so, we need to serialise all structures into `PBF` and deserialised them from `PBF`. The Provider should offer the functionality to load a raw GTFS feed from disk and put it into `PBF`. The `PBF` has then to be transferred via ZeroMQ to the `MasterService` which, in turn, returns the access as usual, hiding all the ZeroMQ shenanigans from the rest of the project.
- [ ] add PBF serialisation / deserialisation to all timetable data structures / look-up data structures
- [ ] create data provider to be started up before any server can be started (think shared memory loading in OSRM), loading data and creating pbf-forms of all structures
- start off with a create-all approach?
- or still create on demand (might need to add additional pbf -> datastructure steps)
- [ ] add communication possibilities to data provider
- [ ] replace master service data-loading and creation with request + pbf->datastructure conversion
/cc @daniel-j-h
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #3 and the current master-service::dataset implementation to understand the existing loading and lookup flow. Then map the timetable and lookup structures that need PBF serialization, the raw GTFS loading path, and the proposed ZeroMQ boundary. Done means the provider can load data, communicate with MasterService, and preserve the existing access behavior without shared-memory setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100