[Proposal] Codebase refactoring
- Dominant language
- Jupyter Notebook
- Stars
- 434
- Forks
- 84
- PR merge metrics
- No merged PRs in 30d
Description
To organize the code and introduce testing and continuous integration, it would be beneficial to refactor the entire codebase.
## TL;DR
- Re-organizing the codebase to follow best practices and to introduce testing and continuous integration.
- Separating `logic` to import the package as a separate module, `scripts` to localize scripts that were used for train/inference of the logic, `notebooks` to localize demos and simple scripts that were written as notebooks, and `tests` to test the logic
- Adding GitHub Actions to test `build`, `logic` of the package, auto-generate `docs`, and to `publish` the package to `pypi`
- Moving from `pip` and `requirements.txt` setup to `conda` for environment management and [`poetry`](https://python-poetry.org/) for packages management. This will ease the development as the project scales.
## Codebase refactoring
Mapping
file/dir | action | placement
-- | -- | --
FastSpeed2/* | moved | kaalm/external/FastSpeed2/*
dialect_speech_corpus | moved | klaam/speech_corpus/dialect.py
egy_speech_corpus | moved | klaam/speech_corpus/egy.py
mor_speech_corpus | moved | klaam/speech_corpus/mor.py
samples | moved | samples
.gitignore | moved | .gitignore
LICENSE | moved | LICENSE
README.md | moved | README.md
audio_utils.py | moved | klaam/utils/audio.py
demo.ipynb | moved | notebooks/demo.ipynb
demo_with_mic.ipynb | moved | notebooks/demo_with_mix.ipynb
inference.ipynb | moved | notebooks/inference.ipynb
klaam.py | moved | klaam/run.py
klaam_logo.PNG | moved | misc/klaam_logo.png
models.py | moved | klaam/models/wav2vec.py
processors.py | moved | klaam/processors/custom_wave2vec.py
requirements.txt | removed |
run.sh | moved | scripts/run.sh
run_classifier.py | moved | scripts/run_classifier.py
run_common_voice.py | moved | scripts/run_common_voice.py
run_mgb3.py | moved | scripts/run_mgb3.py
run_mgb5.py | moved | scripts/run_mgb5.py
sample_run.sh | moved | scripts/sample_run.sh
utils.py | moved | klaam/utils/utils.py
| added | docs
| added | tests
| added | .github
| added | output
| added | environment.yml
| added | install.sh
| added | mypi.ini
| added | pyproject.toml
| added | pytest.ini
| added | ckpts
Tree Structure
root | level 1 | level2 | description
-- | -- | -- | --
.github | | | github stuff (e.g. github issue templates, github actions workflows, etc.)
| workflows | |
| | build.yml | to test building of the package
| | publish.yml | to publish the package to `pypi`
| | tests.yml | to run tests
| | docs.yml | to generate documentation
klaam | | | the logic for the package
| utils | |
| | audio.py |
| | utils.py |
| models | |
| | wav2vec.py |
| processors | |
| | wave2vec.py |
| external | |
| | FastSpeed2/* |
| speech_corpus | |
| | dialect.py |
| | egy.py |
| | mor.py |
| run.py | |
notebooks | | |
| demo.ipynb | |
| demo_with_mix.ipynb | |
| inference.ipynb | |
| | |
scripts | | | set of scripts to be used to train/evaluate or anything external from the logic of the package
| run.sh | |
| run_classifier.py | |
| run_common_voice.py | |
| run_mgb3.py | |
| run_mgb5.py | |
| sample_run.sh | |
tests | | | set of tests to test logics within `klaam`
| test_*.py | |
| conftest.py | |
misc | | |
| klaam_logo.png | |
samples | | |
| demo.wav | |
ckpts | ... | | checkpoints of pre-trained models that were downloaded
docs | ... | | documentation files
output | ... | |
environment.yml | | | `conda` environment definition
install.sh | | | installing script to setup `conda` environment and install dependecies using `poetry`
mypy.ini | | | `pylint` configuration
pyproject.toml | | | package definition and list of dependecies to be installed
pytest.ini | | | `pytest` configuration
LICENSE | | |
README.md | | |
.gitignore | | |
## Environment/dependencies packages
- [`conda`](https://docs.conda.io/en/latest/) is used to manage the environment and install essential libraries that are big/core to the package, e.g. TensorFlow, PyTorch, cudatools, etc.
- [`poetry`](https://python-poetry.org/) is used to manage dependencies and setup the package
- [`pytest`](https://docs.pytest.org/en/7.1.x/) is used to enable unit/integration testing of the codebase
## Commands
- `poetry add PACKAGE` - to add a package (this will append to `pyproject.toml`)
- If the package installation failed and couldn't find another way to add the package, then install it using `conda` and add to `enviroment.yml` manually. (leave a comment next to the line)
- Check on the web for the right channels when install packages using `conda`
- `poetry install` - to install the package (`package_name`)
- `pytest tests` - to run all tests manually
- `pytest tests/TEST_PATH` - to run a specific test file (check [pytest documentation for more information](https://docs.pytest.org/en/stable/contents.html))
---
Edit - added the following sections: env/dep packages and commands
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the current repository layout with the proposed tree, then review the referenced package files, notebooks, scripts, and tests. Read pyproject.toml, environment.yml, pytest.ini, and the planned .github/workflows files to understand the packaging, environment, testing, and automation scope. Done means the code is organized as mapped, tests run with pytest tests, and the documented build, test, docs, and publish workflows are present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, jupyter-notebook, python
- Domain
- build-system, ci-cd, developer-experience, testing, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100