JuliaText / JuliaText/Embeddings.jl

Lazily loading embeddings

Open
#25 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
84
Forks
19
PR merge metrics
No merged PRs in 30d

Description

in #24 @robertfeldt said

> I also wonder if it would be possible to do the loading also of the Word2Vec default embeddings lazily since that could take down the time when first executing using Embeddings. Would simplify testing and use in "downstream" packages which might only optionally use the embeddings.

In theory it is possible that rather than storing the array,
we could store some lazy array that is only instantiated when it is accessed.
We'ld still need to process the whole file to get the vocabulary.
For Word2Vec I don't think it would gain much as those we have in a binary format.
But for some of the others like FastText we have them in a text format,
and so parsing takes some time.

Note: this would not change how long `using Embeddings` takes.
As no embeddings are actually loaded when you do that -- you need to call `load_embeddings` before anything is loaded.

I won't have time to work on this any time soon but would review PRs

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.