ai-cfia / ai-cfia/ailab-db

Search functions return wrong content in `ailab-db`

Aberta
#91 0 comentários 1 reação 1 responsável Reivindicada por @k-allagbe Ver no GitHub
Linguagem predominante
Jupyter Notebook
Estrelas
1
Forks
0
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

## Description

Search functions now return funny values for the `content` field.

```python
def search(cursor, query_embedding):
"""Search matching documents with a given query and return a dict."""
data = {
'text': ' '.join(sys.argv[1:]),
'query_embedding': query_embedding,
'match_threshold': 0.5,
'match_count': 10,
'weights': json.dumps(FINESSE_JSON_PARSED_WEIGHTS)
}

cursor.execute("""
SELECT *
FROM search(%(text)s, %(query_embedding)s::vector, %(match_threshold)s,
%(match_count)s, %(weights)s::JSONB)
""", data)
# turn into list of dict now to preserve dictionaries
results = cursor.fetchall()
from pprint import pprint
pprint(results)
return [dict(r) for r in results[0]["search"]]
```
Printed value for "who's the president?"
```json
[
{
"search": [
{
"content": "2004",
"id": "1bc6def7-339e-49cf-996c-8559c8f074b0",
"last_updated": "2022-08-02",
"query_id": "b685ff7a-7bc3-40c7-88de-8f6183a13f2a",
"score": 0.43899683490982105,
"scores": {
"current": 1,
"recency": 0.8294036718602102,
"similarity": 0.778211130010486,
"traffic": 0.08711416212970638,
"typicality": 0.0012760527435133986,
},
"subtitle": "2004",
"title": "L'ACIA : beaucoup de chemin parcouru depuis 1997 - "
"Agence canadienne d'inspection des aliments",
"url": "https://inspection.canada.ca/inspecter-et-proteger/salubrite-des-aliments/chemin-parcouru/fra/1655235733366/1655496070310",
},
{
"content": "2004",
"id": "9a8ba37c-8b15-4b3f-9f5d-49ca50f92d35",
"last_updated": "2022-08-02",
"query_id": "b685ff7a-7bc3-40c7-88de-8f6183a13f2a",
"score": 0.43874722259501314,
"scores": {
"current": 1,
"recency": 0.8294036718602102,
"similarity": 0.778211130010486,
"traffic": 0.08586610055566671,
"typicality": 0.0012760527435133986,
},
"subtitle": "2004",
"title": "L'ACIA : beaucoup de chemin parcouru depuis 1997 - "
"Agence canadienne d'inspection des aliments",
"url": "https://inspection.canada.ca/inspecter-et-proteger/sante-des-vegetaux/chemin-parcouru/fra/1655235733366/1655496024138",
}
]
}
]

```

1. The chunk's full text or snippet is expected to be received in the `content` field.
2. The subtitle values also look funny, although I'm not sure what is supposed to be received there.
3. I suggest we receive the value (list of results) of the `search` field instead of `[ { "search": [...] } ]`

For comparison we receive this from azure-db:
```json
[
{
"content": "Kochhar\n \n \n CFIA President\n Dr.",
"id": "ZGI2M2VmNjktNTVmMC00ODQ2LThlZWItZDljYTYwZDMwNTI10",
"last_updated": "2023-04-18T00:00:00Z",
"score": 8.465198,
"title": "Dr. Harpreet S. Kochhar - Canadian Food Inspection Agency",
"url": "https://inspection.canada.ca/about-cfia/organizational-structure/cfia-president/eng/1681496883837/1681496884212",
},
{
"content": "The CFIA is headed by a "
"President, who has the rank "
"and all the powers of a Deputy Head of a "
"Department.",
"id": "ZDg1YzNlMjEtMzhjMS00NTQ2LWFhYTEtM2ZjOWUyOTZmZWFm0",
"last_updated": "2017-08-28T00:00:00Z",
"score": 8.322858,
"title": "Canadian Food Inspection Agency (CFIA) - Quarterly Financial "
"Report (QFR) for the Quarter ended June 30, 2017 - Canadian Food "
"Inspection Agency",
"url": "https://inspection.canada.ca/about-cfia/transparency/corporate-management-reporting/reports-to-parliament/financial-reporting/quarter-ended-june-30-2017/eng/1502989987656/1502989988316",
},
]
```

## Acceptance criteria
- Full text or relevant snippet is received in the `content` field

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.