awslabs / awslabs/graphrag-toolkit

[BUG] Remove nltk hardlink workaround once upstream (nltk/llama-index) resolves it

Open
#471 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
442
Forks
106
Avg merge
2d 17h
Merged PRs (30d)
52

Description

### Package version

3.18.5

### Package

lexical-graph

### Python version

3.10, 3.11, 3.12 (CI)

### Operating System

Linux

### Description

As a stopgap for #468, we worked around nltk 3.10.3's `pathsec` hardlink check (CWE-59), which refuses to open `uv`-hardlinked NLTK data files and breaks sentence splitting. The workaround(s) currently in the codebase:
- `UV_LINK_MODE: copy` in `.github/workflows/lexical-graph-tests.yml`
- `UV_LINK_MODE: copy` in `.github/workflows/byokg-rag-tests.yml`
- `nltk<3.10.3` pin in `lexical-graph/src/graphrag_toolkit/lexical_graph/requirements.txt`,

These are intentional temporary measures. The proper fix belongs upstream, not in our code.

Revisit and remove the workarounds once upstream fix lands.
- [ ] Remove the `nltk` pin from `requirements.txt` (if present), or widen it to allow the fixed nltk range.
- [ ] Remove `UV_LINK_MODE: copy` from both test workflows (verify CI still green).
- [ ] Confirm `test_chunking.py::TestChunkingBasicBehavior` tests pass with the fixed nltk on a `uv`-based Linux install.

### Steps to reproduce

```python
On Linux, in a uv-created venv (uv hardlinks from its cache):

uv venv --seed .venv
uv pip install --python .venv/bin/python -e '.[test]' # resolves nltk 3.10.3
PYTHONPATH=src .venv/bin/python -m pytest tests/unit/indexing/extract/test_chunking.py
```

### Error output / stack trace

```shell

```

Contributor guide

Open the contributing guide

Research direction

First confirm that the upstream NLTK pathsec fix is available. Then inspect lexical-graph/src/graphrag_toolkit/lexical_graph/requirements.txt, .github/workflows/lexical-graph-tests.yml, and .github/workflows/byokg-rag-tests.yml; remove or widen the pin and remove both UV_LINK_MODE settings, then run tests/unit/indexing/extract/test_chunking.py::TestChunkingBasicBehavior on a uv-based Linux install.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.