abetlen / abetlen/llama-cpp-python
Implementation of LlamaDiskCache lacks capacity setting.
- 主要言語
- Python
- スター
- 10.6k
- フォーク
- 1.4k
- PR マージ指標
- PR 指標を取得中
説明
LlamaDiskCache utilizes diskcache module to hold cache.
While LlamaDiskCache.\_\_init\_\_ can have capacity setting, it's not passed to diskcache.Cache.\_\_init\_\_, which is causing capacity mismatch, and it fails to store data of size greater than 1GB (default capacity of diskcache).
Current implementation:
```python
class LlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""
def __init__(
self, cache_dir: str = ".cache/llama_cache", capacity_bytes: int = (2 << 30)
):
super().__init__(capacity_bytes)
self.cache = diskcache.Cache(cache_dir)
```
This I guess is required:
```python
class LlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""
def __init__(
self, cache_dir: str = ".cache/llama_cache", capacity_bytes: int = (2 << 30)
):
super().__init__(capacity_bytes)
self.cache = diskcache.Cache(cache_dir, size_limit=capacity_bytes) # Added size_limit configuration.
```
コントリビューションガイド
評価
この issue はまだ評価されていません。