[FEATURE] Second look at OTel logs and traces schema

Is your feature request related to a problem?

The OpenSearch OTel index mappings (otel-v1-apm-span-*, logs-otel-v1-*, metrics-otel-v1-*) were originally defined in Data Prepper ~6 years ago and have not been formally revisited. A discussion with Yuri Shkuro (OTel/Jaeger co-founder) raised significant concerns about the current schema design. Specifically, dynamic mapping of span attributes causes field explosion and type conflicts at scale (thousands of microservices writing to the same index). Yuri's assessment is that the Jaeger-style nested array with selectively uplifted common tags remains more robust than the current otel-v1 approach of dynamically mapped attribute sub-fields.

Meanwhile, three independent projects now consume or reimplement these mappings:

1. Data Prepper - the original author of the OTel v1 / SS4O schemas.
2. OpenSearch Exporter (OTel Collector Contrib) - adding an otel-v1 mapping mode with index-template management and ISM rollover directly in the Collector.
3. SAP OpenSearch OTLP Receiver - a PoC plugin that embeds native OTLP gRPC/HTTP endpoints inside OpenSearch (3.6.0) and writes SS4O-compatible documents, eliminating the Data Prepper intermediary.

With no single source of truth, each project risks diverging on field names, types, attribute handling, and index conventions, creating a fragmented experience for users who switch between ingestion paths.

What solution would you like?

1. Revisit the OTel v1 mappings - Audit the current schemas against the latest OTel semantic conventions and evaluate the attribute-handling strategy. Key questions:

   • Should attributes remain dynamically mapped sub-fields (current), move to a nested array model (Jaeger-style), or support a hybrid with selectively uplifted common tags?
   • Are date_nanos timestamps, integer status.code/severity.number, and typed resource/scope attributes correctly represented?
   • How should field explosion and type conflicts be mitigated at scale without sacrificing query performance (avoiding the read-amplification of per-service indices and the full-scan cost of stringified JSON)?

2. Establish a canonical, centralized mapping package - Publish authoritative index templates and component templates in a single location under opensearch-project so that Data Prepper, the OTel Collector OpenSearch Exporter, and the SAP OTLP Receiver can all consume them as a shared dependency rather than maintaining independent copies.

3. Define a versioning strategy - The current schemas are named otel-v1-*. If breaking changes are introduced (e.g., a new attribute storage model), determine whether v2 is the right version identifier or whether a different scheme (e.g., tied to OTel schema versions or a semver contract) would better communicate compatibility to users and downstream consumers.

What alternatives have you considered?

• Per-service indices (otel-v1-apm-span-<service>) - Bounds field cardinality per index but introduces ~1000x read amplification for cross-service queries (resolved indices x shards x segments), plus massive coordinator-node overhead for the gather phase. Not viable at scale per Yuri's feedback.
• Stringified JSON attributes expanded at query time (PPL spath) - Eliminates dynamic mapping entirely but is equivalent to a full table scan where the coordinator must receive and parse full documents. Not viable for production query performance.
• Do nothing - Each project independently maintains its own copy. Guarantees drift and forces users to reconcile differences when switching ingestion paths.
• Informal cross-project alignment - Periodically sync schemas via issues/PRs. Doesn't scale as new consumers appear and gives users no single version to pin against.

Do you have any additional context?

• Full OTel Collector discussion: open-telemetry/opentelemetry-collector-contrib#48585 - includes Yuri's feedback on field explosion, per-service index pitfalls, and the Jaeger nested-array precedent.
• SAP OTLP Receiver: SAP/opensearch-otlp-receiver - PoC plugin for OpenSearch 3.6.0; writes to otel-v1-apm-span-*, logs-otel-v1-*, metrics-otel-v1-* with auto-provisioned SS4O templates.
• Current index naming convention: otel-v1-apm-span-*, otel-v1-apm-service-map, logs-otel-v1-*, metrics-otel-v1-*.
• Jaeger mapping reference: Nested tag array with key/value/type fields + selectively uplifted common attributes (e.g., http.status_code, service.name) as top-level indexed fields. A 10-year proven approach for avoiding mapping explosion at scale.