abetlen / abetlen/llama-cpp-python

Implementation of LlamaDiskCache lacks capacity setting.

未關閉
#1,402 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

LlamaDiskCache utilizes diskcache module to hold cache.
While LlamaDiskCache.\_\_init\_\_ can have capacity setting, it's not passed to diskcache.Cache.\_\_init\_\_, which is causing capacity mismatch, and it fails to store data of size greater than 1GB (default capacity of diskcache).

Current implementation:
```python
class LlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""

def __init__(
self, cache_dir: str = ".cache/llama_cache", capacity_bytes: int = (2 << 30)
):
super().__init__(capacity_bytes)
self.cache = diskcache.Cache(cache_dir)
```

This I guess is required:
```python
class LlamaDiskCache(BaseLlamaCache):
"""Cache for a llama.cpp model using disk."""

def __init__(
self, cache_dir: str = ".cache/llama_cache", capacity_bytes: int = (2 << 30)
):
super().__init__(capacity_bytes)
self.cache = diskcache.Cache(cache_dir, size_limit=capacity_bytes) # Added size_limit configuration.
```

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。