Considerations for language model inclusion in default package or download them later

This issue works as a note on the size of different language models PyThaiNLP currently use.

• Some of them are included in the package - will be immediately available after the installation of package.
• Some of them are not included in the package - will be downloaded automatically on the first call, during runtime.

To include or not include: Pros and cons

• Having language models included in the standard package
  • Pros:
    • Less dependable to the network connection, more predictable behaviour
    • May be easier to manage and cache in continuous integration/testing environment
  • Cons:
    • Larger package size
    • May waste user's disk space with files they never use
• Download language models at the point of its first usage
  • Pros:
    • Smaller package size
    • User only download what they really use
  • Cons:
    • More dependable to the network connection, less predictable behaviour
    • Can slow down test (multiple separated file downloads, in sequence, is slower than one big file download)

Use pip to download language models

Optionally, we can also consider create a new package, upload them to PyPI, and using pip to facilitate downloads.

User can do something like pip install pythainlp-models-pos or pip install pythainlp-models[ner] or pip install pythainlp-models[all] during their environment setup, and then will never have to worry about them being downloaded during runtime.

This way, we can use PyPI as our data host and also benefit from any possible proxy and cache CI platforms/ISPs may have for PyPI. This can be more secure than our self-manage system as well.

PyPI standard package size limit is 60MB. But more can be requested.

pip also support offline operation by using pip download to keep the files and use pip install --no-index --find-links <dir_of_dependencies>

Size and Hosting

(clearly, we need some standard naming convention here as well)

Training data and training scripts

See #344

Model card

Related to this, in terms of model description, see #471

Model auto-download

See discussion about pythainlp.corpus.get_corpus_path() at #385

Review the model inventory and the related discussions in #344, #471, and #385, especially the notes on corpus auto-download and model packaging. The issue does not name files, tests, or a settled implementation target, so a contributor would first need agreement on the packaging, hosting, naming, and download approach.