joeroe / joeroe/rintchron

Optimise intchron()

Open
#3 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
R
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

We should try to make IntChron queries as fast as possible. Currently, retrieving the entire database (without tabulation) takes:

```r
system.time(intchron("all", tabulate = FALSE))
#> user system elapsed
#> 48.587 0.579 436.211
```

`user`+`system` time is low (but could still be improved), so we can assume most of the time is spent waiting for responses to IntChron requests.

Ideas for optimisation:
- [ ] Group batches of requests into vectorised calls to `intchron_request()` as often as possible
- [ ] Parallelise requests:
- [ ] At the least it should be safe to execute recursive calls to `intchron_crawl()` in parallel, since crawling each branch is independent of other parts of the tree.
- [ ] Also investigate using ... in `intchron_request()` and possibly elsewhere.
- [ ] But need to ensure these don't place too much load on the IntChron server, since we already get timeouts when running the relatively modest queries in the examples and vignettes simultaneously as R CMD checks.
- [ ] Improve filtering/pattern-matching options in `intchron_crawl()` and use them aggressively in `intchron()`
- [ ] Investigate where the `user` CPU time is being spent

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.