elastic / elastic/ml-cpp

[ML] Include memory required for persistence in data frame analytics memory estimates

Offen
#1,900 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
:ml/DataFrameAnalysis
Vorherrschende Sprache
C++
Sterne
157
Forks
67
Ø Merge
17 Std. 52 Min.
Gemergte PRs (30 T.)
20

Beschreibung

The memory estimates used in data frame analytics should include an allowance for the extra memory required during persistence and restoration. Otherwise during persistence or restoration on a maxed-out node there is a risk that some process (not necessarily the job doing persistence or restoration) gets killed because there isn't enough memory to accommodate both the model and the extra data structures used during persistence and restoration.

It would be reasonable that this is just a simple multiplier of the estimated model size. If the main extra memory usage is in compressed form then maybe 10% would be enough, or maybe 10% plus a capped amount for a single uncompressed chunk, for example, `reported memory estimate = raw memory estimate * 1.1 + min(16MB, raw memory estimate)`.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.