tidyverts / tidyverts/fabletools
Memory issue from mable and possibly model definition
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 98
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
This is more of a question than an issue if this behaviour is expected:
library(fable)
#> Loading required package: fabletools
library(tsibble)
#>
#> Attaching package: 'tsibble'
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, union
mdl <- model(tourism, SNAIVE(Trips))
lobstr::obj_size(tourism)
#> 1.11 MB
lobstr::obj_size(mdl)
#> 3.65 MB
Created on 2022-10-14 with reprex v2.0.2
The size of mable is triple of the data itself, even if I'm only using SNAIVE. Isn't this a little big?
The model definitions themselves are quite big, so maybe this is normal:
lobstr::obj_size(fable::SNAIVE())
#> 1.15 MB
lobstr::obj_size(fable::ARIMA())
#> 1.57 MB
lobstr::obj_size(fable::ETS())
#> 1.25 MB
Created on 2022-10-14 with reprex v2.0.2
But the real issue is the size scales quite seriously to the data
> mdl <- model(df, SNAIVE(value))
> lobstr::obj_size(df)
68.25 MB
> lobstr::obj_size(mdl)
147.51 MB
It looks the model definition is stored separately for every key group?
> obj_size(mdl$`SNAIVE(value)`[[1]])
1.82 MB
> obj_size(mdl$`SNAIVE(value)`[[1]]$model)
1.81 MB
> mdl$`SNAIVE(value)`[[1]]$model
<RW model definition>
With my much larger dataset, my RAM quickly explodes. I tried to find out why but couldn't dig deeper with model definition being R6 with so many environments linking each other so still didn't find out what is it that the model is storing that causes the size to be big. All I can tell is the model definition seems like a bunch of harmless functions. Is there any way to avoid this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported sizes with tourism, model(tourism, SNAIVE(Trips)), and the larger df example. Inspect the resulting mable and its SNAIVE(value) model definition, then determine whether the per-key memory growth is expected; done means documenting the cause or identifying a focused correction and validating the memory measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100