crewAIInc / crewAIInc/crewAI

[BUG] LanceDBStorage.list_records() returns oldest records instead of newest first due to premature query truncation

Open
#7,394 4 comments 0 reactions 2 assignees View on GitHub

@Rohitkanithi is already working on this.

Since Sep 11, 2026.

bug
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

In LanceDBStorage (lib/crewai/src/crewai/memory/storage/lancedb_storage.py), list_records() is documented to return records ordered by created_at descending:

"List records in a scope, newest first. Returns: List of MemoryRecord, ordered by created_at descending."

However, list_records() currently returns the oldest records in the table rather than the newest.

Root Cause:
In lancedb_storage.py lines 507–510:

rows = self._scan_rows(scope_prefix, limit=limit + offset)
records = [self._row_to_record(r) for r in rows]
records.sort(key=lambda r: r.created_at, reverse=True)
return records[offset : offset + limit]

### Steps to Reproduce

```text
1. Initialize LanceDBStorage in a temporary directory.
2. Insert 10 records with sequential created_at timestamps (rec_0 oldest, rec_9 newest).
3. Call storage.list_records(limit=3).
4. Inspect the returned IDs.

### Expected behavior

list_records(limit=3) should return the 3 most recently created records:
['rec_9', 'rec_8', 'rec_7']

### Screenshots/Code snippets

import tempfile
from datetime import datetime, timezone, timedelta
from crewai.memory.storage.lancedb_storage import LanceDBStorage
from crewai.memory.types import MemoryRecord

with tempfile.TemporaryDirectory() as tmpdir:
    storage = LanceDBStorage(path=tmpdir)
    base_time = datetime.now(timezone.utc)
    
    # Insert 10 records: rec_0 is oldest, rec_9 is newest
    records = [
        MemoryRecord(
            id=f"rec_{i}",
            content=f"content {i}",
            created_at=base_time + timedelta(minutes=i),
            embedding=[0.1] * 1536
        )
        for i in range(10)
    ]
    storage.save(records)

    # Request the 3 newest records
    results = storage.list_records(limit=3)
    print("Returned IDs:", [r.id for r in results])
    # Expected: ['rec_9', 'rec_8', 'rec_7']
    # Actual:   ['rec_2', 'rec_1', 'rec_0']

### Operating System

macOS Sonoma

### Python Version

3.12

### crewAI Version

1.15.21

### crewAI Tools Version

1.15.21

### Virtual Environment

Venv

### Evidence

Running the reproduction script outputs:

Returned IDs: ['rec_2', 'rec_1', 'rec_0']

The 3 oldest records (rec_0, rec_1, rec_2) were fetched and sorted descending, while the actual newest records (rec_9, rec_8, rec_7) were truncated at the storage layer and never returned.

### Possible Solution

In `LanceDBStorage.list_records`:
Do not truncate `_scan_rows` before sorting. Scan the candidate rows in the scope (using the default `_SCAN_ROWS_LIMIT` of 50,000, identical to how `get_scope_info` works and how `QdrantEdgeStorage.list_records` works), sort by `created_at` descending, and then apply the pagination slice:

```python
def list_records(
    self, scope_prefix: str | None = None, limit: int = 200, offset: int = 0
) -> list[MemoryRecord]:
    rows = self._scan_rows(scope_prefix)
    records = [self._row_to_record(r) for r in rows]
    records.sort(key=lambda r: r.created_at, reverse=True)
    return records[offset : offset + limit]

### Additional context

```text
I have verified this reproduction locally and have the fix and regression unit test ready to submit as a PR if approved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.