Add optional retrieval confidence to retriever outputs

Context

Opening this issue to continue the discussion from PR #11130, based on maintainer feedback to discuss the approach before implementation.

Problem

Currently, retrievers return documents with scores, but there is no standardized way to expose a retrieval confidence signal that downstream components (e.g., RAG pipelines, evaluators) can use.

Proposal

Introduce an optional "retrieval_confidence" metadata field in retriever outputs.

For example:

• Normalize scores to a [0, 1] range
• Attach confidence to each "Document.meta"

doc.meta["retrieval_confidence"] = normalized_score

This would:

• Provide a consistent confidence signal across retrievers
• Help downstream components make better decisions (filtering, ranking, fallback logic)
• Improve explainability in RAG systems

Questions

• Should this be handled at the retriever level or pipeline level?
• Is adding it to "Document.meta" the preferred approach?
• Should this be optional/configurable?

Happy to implement this once the approach is aligned.

Start by reading the discussion from PR #11130 and the maintainer feedback that prompted this issue. Resolve whether confidence belongs at the retriever or pipeline level, whether Document.meta is appropriate, and how normalization and configurability should work. Done means the approach is agreed before implementation begins.