Mission: Multilingual Semantic Theme Discovery and Retrieval Quality Audit

Mission/open goal Description

I selected the Data Scientist mission, A Day as Zava's Principal Data Scientist.

The goal was to create a semantic theme discovery and retrieval quality study using multilingual customer feedback stored in the PromptathonDb database. The available feedback included reviews and support chats in English, Spanish, and French.

The agent used the precomputed document embeddings and vector similarity search to identify semantically related customer feedback across languages and source types. It then manually labelled retrieved document pairs as Relevant, Partially relevant, or Not relevant and used those labels to calculate Precision at 5.

The study also examined recurring customer themes, successful cross language retrieval examples, false positives, and cases where documents shared similar words but represented different customer intentions.

The final output includes a reproducible Jupyter notebook, a concise Markdown study summary, a model card style assessment, limitations of the small unlabeled corpus, and recommendations for improving semantic retrieval quality.

Harness and model

Harness: GitHub Copilot Chat in Agent mode running inside GitHub Codespaces. Model: MAI Code 1 Flash, as displayed in the Copilot session. Environment and tools: The study used the preconfigured sqlMcpServer MCP server connected to the PromptathonDb SQL Server database. The agent used read only SQL MCP tools such as describe_entities, read_records, aggregate_records, and find_similar_docs_by_doc_id. Python, pandas, matplotlib, JSON processing, and Jupyter Notebook were used to label retrieved document pairs, calculate Precision at 5, create visualizations, and produce the final retrieval quality audit.

Turn-by-turn journey

1. I asked the agent to read missions/data-scientist.md, SQL_DATABASE.md, and README.md and create a read only investigation plan for multilingual semantic theme discovery and retrieval quality evaluation.

2. The plan proposed selecting diverse seed documents from multiple languages and source types, retrieving vector neighbours, manually labelling the results, calculating Precision at 5, and analysing false positives.

3. I initially had missions/data-analyst.md open by mistake. I corrected the context and explicitly instructed the agent to ignore the Data Analyst mission and continue only with missions/data-scientist.md.

4. The agent inspected the Docs corpus using the configured sqlMcpServer tools and examined feedback from reviews and support chats.

5. It selected seed documents representing different languages, customer themes, and source types.

6. For each seed document, the agent used find_similar_docs_by_doc_id to retrieve semantically related documents.

7. Retrieved seed and neighbour pairs were labelled as Relevant, Partially relevant, or Not relevant. Each label included a short rationale based on semantic meaning rather than only shared keywords.

8. Python and pandas were used to organise the labelled evidence and calculate strict and inclusive Precision at 5 results.

9. The study compared retrieval behaviour across languages and between Review and SupportChat documents.

10. The agent identified recurring semantic customer themes and documented successful cross language retrieval examples.

11. It also analysed false positives, including cases where documents contained similar words but represented different intentions or customer problems.

12. During verification, the agent incorrectly used the singular entity name Doc, which caused an EntityNotFound error. I instructed it to use the configured plural entity Docs and resume the existing work.

13. The original analysis session stopped before completing every verification step. I resumed the mission from the two existing artifacts instead of restarting and asked the agent to recheck unsupported or incomplete claims.

14. The first notebook version required correction because it did not open as valid Jupyter Notebook JSON. The agent used a Python validation process to rebuild and verify the notebook structure.

15. The agent audited document identifiers, language and source claims, relevance labels, Precision at 5 calculations, charts, model card content, confidence, and limitations.

16. Audit and Reproducibility Notes were added so another data scientist could repeat, challenge, or improve the study.

17. The completed artifacts were saved as data-scientist-retrieval-audit.ipynb, data-scientist-study-summary.md, and promptathon-data-scientist-submission.md.

18. Each artifact was committed and pushed to the participant's GitHub fork.

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

I went beyond the basic mission requirements by:

• Creating a valid and reproducible Jupyter Notebook containing the complete semantic retrieval quality audit.
• Creating a separate Markdown study summary for reviewers who want a concise overview.
• Evaluating both strict and inclusive definitions of Precision at 5.
• Manually labelling vector search results with written relevance rationales.
• Comparing multilingual retrieval and Review versus SupportChat retrieval behaviour.
• Documenting successful semantic matches as well as false positives and retrieval failure modes.
• Adding visualisations based on the verified labelled evidence.
• Including a model card style section describing appropriate and inappropriate uses of the system.
• Adding Audit and Reproducibility Notes so another data scientist can repeat and critique the study.
• Documenting actual agent errors and corrections, including the accidental Data Analyst context, the incorrect Doc entity name, the interrupted analysis, and the invalid notebook format.
• Creating a submission ready Markdown document containing a Mermaid workflow diagram.
• Committing and pushing the three final artifacts separately to the GitHub fork.

Working notebook:
https://github.com/prithvisaini1222-a11y/sql-ai-promptathon/blob/main/data-scientist-retrieval-audit.ipynb

Study summary:
https://github.com/prithvisaini1222-a11y/sql-ai-promptathon/blob/main/data-scientist-study-summary.md

Submission documentation:
https://github.com/prithvisaini1222-a11y/sql-ai-promptathon/blob/main/promptathon-data-scientist-submission.md

The completed work is in data-scientist-retrieval-audit.ipynb, data-scientist-study-summary.md, and promptathon-data-scientist-submission.md. Start by reviewing the notebook and its validation notes, then compare its documented Docs queries, labels, Precision@5 calculations, charts, and model-card claims with the linked artifacts. Done means the audit is reproducible and the summary matches the verified evidence.