awslabs / awslabs/graphrag-toolkit
[BUG] Remove nltk hardlink workaround once upstream (nltk/llama-index) resolves it
- Dominant language
- Python
- Stars
- 442
- Forks
- 106
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 52
Description
### Package version
3.18.5
### Package
lexical-graph
### Python version
3.10, 3.11, 3.12 (CI)
### Operating System
Linux
### Description
As a stopgap for #468, we worked around nltk 3.10.3's `pathsec` hardlink check (CWE-59), which refuses to open `uv`-hardlinked NLTK data files and breaks sentence splitting. The workaround(s) currently in the codebase:
- `UV_LINK_MODE: copy` in `.github/workflows/lexical-graph-tests.yml`
- `UV_LINK_MODE: copy` in `.github/workflows/byokg-rag-tests.yml`
- `nltk<3.10.3` pin in `lexical-graph/src/graphrag_toolkit/lexical_graph/requirements.txt`,
These are intentional temporary measures. The proper fix belongs upstream, not in our code.
Revisit and remove the workarounds once upstream fix lands.
- [ ] Remove the `nltk` pin from `requirements.txt` (if present), or widen it to allow the fixed nltk range.
- [ ] Remove `UV_LINK_MODE: copy` from both test workflows (verify CI still green).
- [ ] Confirm `test_chunking.py::TestChunkingBasicBehavior` tests pass with the fixed nltk on a `uv`-based Linux install.
### Steps to reproduce
```python
On Linux, in a uv-created venv (uv hardlinks from its cache):
uv venv --seed .venv
uv pip install --python .venv/bin/python -e '.[test]' # resolves nltk 3.10.3
PYTHONPATH=src .venv/bin/python -m pytest tests/unit/indexing/extract/test_chunking.py
```
### Error output / stack trace
```shell
```
Contributor guide
Research direction
First confirm that the upstream NLTK pathsec fix is available. Then inspect lexical-graph/src/graphrag_toolkit/lexical_graph/requirements.txt, .github/workflows/lexical-graph-tests.yml, and .github/workflows/byokg-rag-tests.yml; remove or widen the pin and remove both UV_LINK_MODE settings, then run tests/unit/indexing/extract/test_chunking.py::TestChunkingBasicBehavior on a uv-based Linux install.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100