Mission: Principal Data Scientist - Semantic Theme Discovery and Retrieval Quality Study

Mission/open goal Description

This analysis builds a semantic theme-discovery and retrieval-quality study over Zava's customer feedback corpus (89 documents across English, Spanish, and French). Using vector embeddings and the find_similar_docs_by_doc_id tool, I identified 4 major themes: product quality defects, fit/finish issues, refund/return intent, and style undermined by quality problems. The retrieval quality audit showed Precision@5 of 80% for complaint-focused reviews, with one notable false positive where shared keywords masked opposite sentiment. Key limitations include: the system strongly segments by document type (reviews only, no SupportChats appeared), cross-language retrieval works but favors the seed language, and the small corpus limits generalizability. Future work should test SupportChat seeds, expand the corpus, and add explicit sentiment signals alongside embeddings.

Harness and model

GitHub Copilot Agent mode with GPT-4

Turn-by-turn journey

Turn 1: Discovery and Schema Understanding

Prompt: "I'm the Principal Data Scientist for Zava. I need to understand our customer feedback across English, Spanish, and French. Please help me: 1. First, tell me what SQL MCP tools are available 2. Then show me the schema of the Docs table and how the vector similarity search works 3. Finally, give me some sample documents to see what's in our data"

Agent Response: The Agent described 4 MCP tools (describe_entities, read_records, aggregate_records, find_similar_docs_by_doc_id) and explained the Docs schema including the VECTOR(1536) embedding column. It provided SQL query examples but didn't execute them.

Result: Learned about the available tools, schema structure, and vector similarity approach.

Dead End: The Agent provided SQL examples but didn't execute them. I realized I needed to explicitly request tool execution.

Insight: Vector embedding approach should help with cross-language theme discovery. TagsJson field is key for language filtering.

Turn 2: Data Discovery and Language Distribution

Prompt: "Great, now let me see actual data. Use the SQL MCP tools to: 1. Show me 3 sample documents in English 2. Show me 3 sample documents in Spanish 3. Show me 3 sample documents in French 4. Tell me how many total documents we have and what the language distribution looks like. Please show me the actual SQL queries you're running so I can see the evidence."

Agent Response: Connected to SQL Server (had to retry with SSL trust), discovered language tags are in language:en format, adapted queries to use LIKE '%language:en%', and returned sample documents and distribution.

SQL Query:

sql
SELECT TOP 3 DocId, SourceType, Title, Body, TagsJson
FROM dbo.Docs
WHERE TagsJson LIKE '%language:en%'
ORDER BY DocId;
Result: 89 total documents (vs 83 in documentation). English: 59, Spanish: 19, French: 11. Found reviews with ratings 1-5 stars covering quality, fit, and comfort.

Dead End/Correction: First query using JSON_VALUE failed because TagsJson is an array format. Switched to LIKE '%language:en%' pattern matching.

Insight: English dominates the corpus (66%). Data is primarily product reviews with wide sentiment range. TagsJson is array-based, not key-value.

Turn 3: Vector Similarity Retrieval Quality Audit

Prompt: "Now let me test the vector similarity search. Pick DocId = 2 (the English negative review about quality) and find its 5 most similar documents using find_similar_docs_by_doc_id. Show me: 1. The full text of the query document 2. The full text of each of the 5 similar documents 3. What languages are in the results 4. Based on reading them, are these actually similar in meaning or just sharing keywords?"

Agent Response: Executed dbo.FindSimilarDocsByDocId stored procedure, retrieved all document texts, analyzed for semantic relevance.

SQL Query:

sql
EXEC dbo.FindSimilarDocsByDocId @DocId = 2, @TopN = 5;
Result: Query was 1-star review about product quality. 4/5 neighbors were genuinely relevant (negative reviews about product quality). 1 false positive (5-star review with "small quality issue"). All results English.

Insight: Vector search is good at grouping English product quality complaints (80% precision). However, it can be fooled by shared keywords/products when sentiment is opposite. No cross-language retrieval.

Turn 4: Theme Discovery Across Customer Feedback

Prompt: "Now help me discover themes across customer feedback. 1. Looking at the 5 similar documents from DocId 2, what themes emerge? 2. Now expand this - find 10 similar documents to DocId 2 and identify 3-4 broader themes across all feedback 3. For each theme, tell me: Which documents support it, Whether the theme appears in reviews, support chats, or both, Whether it appears across languages"

Agent Response: Expanded to TopN=10, retrieved all document texts, conducted thematic analysis, categorized into 4 themes with supporting evidence.

SQL Query:

sql
EXEC dbo.FindSimilarDocsByDocId @DocId = 2, @TopN = 10;
Result: Identified 4 major themes: 1) Product Quality Defects (9/10 docs), 2) Fit/Finish & Premium Feel Issues (6/10), 3) Refund/Return Intent (4/10), 4) Good Style + Quality Issues (3/10). All 10 documents were reviews (no support chats), all in English.

Dead End/Correction: The analysis revealed no SupportChat documents appeared, and all results were English. Cross-language and cross-source theme discovery not yet tested.

Insight: Quality defects are the dominant theme across multiple product lines. There's a pattern of "good aesthetics, poor durability." Need to test Spanish/French and SupportChat data.

Turn 5: Cross-Language and Cross-Source Retrieval Analysis

Prompt: "Now let me test cross-language and cross-source retrieval: 1. Pick a Spanish document (DocId = 9) and find its 5 most similar documents. What languages appear? 2. Pick a French document (DocId = 1) and find its 5 most similar documents. What languages appear? 3. Check if ANY support chat documents appear in the top results. If not, why? 4. For the Spanish seed, are the similar documents actually about the same themes?"

Agent Response: Executed similarity searches for Spanish (DocId=9) and French (DocId=1) seeds with TopN=20, checked SupportChat corpus size, analyzed results.

SQL Query:

sql
EXEC dbo.FindSimilarDocsByDocId @DocId = 9, @TopN = 20;
EXEC dbo.FindSimilarDocsByDocId @DocId = 1, @TopN = 20;
Result: Spanish seed: 4 Spanish + 1 English; themes aligned (fit/comfort, small defects). French seed: 3 French + 2 Spanish; themes aligned. 45 SupportChats exist (50.6% of corpus) but ZERO appeared in top 20 results.

Dead End/Correction: Cross-source retrieval failed. Review and SupportChat texts are structurally and semantically different, causing embeddings to group by document type.

Insight: Cross-language retrieval works to some degree, but language preference favors the seed language. More importantly, the system strongly segments by document type - reviews with reviews, chats with chats. This means a complete understanding requires separate analysis of each document type.

Turn 6: Retrieval Quality Audit and Precision Calculation

Prompt: Provided relevance labels for 9 neighbor documents and requested precision metrics at k=1, 3, 5, and 10.

Agent Response: Applied labels to ranked list, computed precision metrics, explained false positive case.

Result: Precision@1: 100%, Precision@3: 66.7%, Precision@5: 80%, Precision@10: 66.7% (normalized to 9 docs). DocId 18 identified as false positive due to shared keywords but opposite sentiment.

Insight: System performs well for finding similar complaint documents (4/5 relevant in top 5). False positives occur when reviews share keywords but opposite sentiment - a common limitation of vector similarity.

semantic_theme_analysis.ipynb

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

I went beyond the basic requirements by:

Cross-language analysis: Tested both Spanish and French seeds to evaluate multilingual retrieval

Cross-source investigation: Discovered and documented the significant gap between Reviews and SupportChats (50.6% of corpus but zero cross-retrieval)

Rigorous false positive analysis: Identified why DocId 18 fails (shared product/defect keywords but opposite sentiment)

Honest model card: Included what the system is good for AND its limitations

Evidence documentation: Included all SQL queries and tool calls in the notebook

Start by reviewing semantic_theme_analysis.ipynb and the documented FindSimilarDocsByDocId queries. Compare the reported language, document-type, and precision findings with the notebook evidence. The mission is already marked complete, so no unimplemented change or additional definition of done is stated.