Use multilingual labels for embeddings and LLM mapping prompts

Summary

Once multilingual labels with language information are available (see #106), we can leverage them to improve both embedding quality and LLM mapping prompt context.

Context

Currently:

• Embeddings: Positions use paraphrase-multilingual-MiniLM-L12-v2 for semantic search, but only embed the primary entity name (single label)
• LLM mapping prompts (Stage 2): Candidate entities show only name and description, not multilingual label variations
• Labels: Stored in wikidata_entity_labels table, soon to include language_id per #106

Proposed Enhancements

1. Multilingual Labels in Mapping Prompts (Stage 2)

Current state (poliloom/enrichment.py:288-296):

candidate_entities = [
    {
        "qid": entity.wikidata_id,
        "name": entity.name,
        "description": entity.description,
    }
    for entity in similar_entities
]

Enhancement: Include relevant language labels in candidate context:

candidate_entities = [
    {
        "qid": entity.wikidata_id,
        "name": entity.name,
        "description": entity.description,
        "labels": {
            lang.iso1_code: label.label 
            for label in entity.language_labels
        }
    }
    for entity in similar_entities
]

Benefits:

• LLM can see entity names in multiple languages during mapping
• Better disambiguation for entities with similar English names but distinct local names
• Matches user's source language context (if source is in French, showing French labels helps)

2. Multilingual Embeddings

Current state (poliloom/enrichment.py:369-375):

texts = [free_item.name for free_item in free_form_results]
embeddings = generate_embeddings_batch(texts)

Enhancement: Include all language labels in embedding input:

texts = [
    f"{entity.name} | {' | '.join(entity.labels)}"
    for entity in entities
]

This allows the multilingual embedding model to capture semantic meaning across all language variations, improving cross-lingual matching.

Implementation Steps

1. Depends on #106: Multilingual labels with language_id must be implemented first
2. Update candidate entity serialization to include language-specific labels
3. Modify mapping prompts to leverage multilingual context
4. Update embedding generation to include all labels
5. Consider re-generating embeddings for existing entities (migration)

Files to Modify

• poliloom/enrichment.py - Update candidate serialization and embedding generation
• poliloom/prompts.py - Update mapping prompts to show multilingual labels
• poliloom/models/wikidata.py - Add method to get labels with language info

Start with dependency #106, then inspect poliloom/enrichment.py at the candidate serialization and embedding-generation sections, poliloom/prompts.py, and poliloom/models/wikidata.py. Confirm how language labels expose their language information before choosing the prompt and embedding inputs. Done means multilingual labels are included in Stage 2 mapping context and embedding generation, with existing embeddings and any migration implications addressed.