[BUG] When agent response is too long, failed to update task index

What is the bug?
When testing using agent execute with async mode, if the final response is long, it failed to update the task index and cannot get the final response.

How can one reproduce the bug?

POST /_plugins/_ml/agents/p9gsPpwBJscUAg_qA0Gr/_execute?async=true
{
    "parameters": {
        "question": "TRy search on opensearch-release index. Research the complete history of OpenSearch from version 1.0 onwards, specifically detailing every major architectural upgrade and new machine learning capability introduced (e.g., each significant iteration of the ML Commons Agent framework).\n Requirement: For every major ML capability update, provide a detailed description of at least four (4) key technical features. The final report must also fully quote the functional descriptions for all key Agent Tools you encounter during your research?",
          "context_management_name": "interaction_summarizations",
         "verbose": true
    }
}

GET  _plugins/_ml/tasks/S5VkPpwB84_tbshlbP3B

the task is always created but NOT success

{
  "task_type": "AGENT_EXECUTION",
  "function_name": "AGENT",
  "state": "CREATED",
  "worker_node": [
    "7WgoNASBT76ZQv6HMV9pGg"
  ],
  "create_time": 1770573294732,
  "last_update_time": 1770573294732,
  "is_async": true,
  "response": {
    "memory_id": "SZVkPpwB84_tbshlbP2N",
    "parent_interaction_id": "SpVkPpwB84_tbshlbP2n"
  }
}

Looking at traces, the task is already got an final response

GET _plugins/_ml/memory/message/SpVkPpwB84_tbshlbP2n/traces
{
      "memory_id": "SZVkPpwB84_tbshlbP2N",
      "message_id": "U5VkPpwB84_tbshl__1_",
      "create_time": "2026-02-08T17:55:32.348746963Z",
      "updated_time": "2026-02-08T17:55:32.348746963Z",
      "input": """TRy search on opensearch-release index. Research the complete history of OpenSearch from version 1.0 onwards, specifically detailing every major architectural upgrade and new machine learning capability introduced (e.g., each significant iteration of the ML Commons Agent framework).
 Requirement: For every major ML capability update, provide a detailed description of at least four (4) key technical features. The final report must also fully quote the functional descriptions for all key Agent Tools you encounter during your research?""",
      "response": """The search results provide helpful details on OpenSearch release history and major ML capability upgrades like:

OpenSearch 2.13.0 (April 2024):
- Made agents/tools and OpenSearch Assistant generally available
- Introduced vector quantization
- Added LLM guardrails and hybrid search with aggregations 

OpenSearch 2.12.0 (February 2024): 
- Made concurrent segment search and conversational search generally available
- Provided experimental OpenSearch Assistant Toolkit with agents, tools, workflow automation
- Added match-only text field type, query insights for top N queries, k-NN on nested fields

Some key ML highlights from the release notes:

ML Commons Agent Framework Upgrades:
- 2.13.0 introduced agents/tools, OpenSearch Assistant 
- 2.12.0 had experimental agents, tools, workflow automation

Vector Search Capabilities:
- 2.13.0 added vector quantization 
- 2.12.0 enabled k-NN search on nested fields
- 2.11.0 added multimodal, sparse neural search

I'll provide functional descriptions of the major Agent Tools identified:

<tool_descriptions>

Agent Tool:
"This tool enables developers to rapidly build and extend agents with custom capabilities by creating modular snippets of code that can be composed together to form more complex behaviors."

Connector Tool: 
"Provides a standardized interface for connecting agents to external systems and services, enabling bidirectional data exchange and integration with third-party APIs and databases."

Create Tool:
"Allows agents to dynamically generate new content such as text, images, or other multimedia based on user inputs and contextual information from their knowledge base."

Query Planning Tool:
"Helps agents formulate optimal query plans by analyzing the user's information need, understanding context, and selecting the most relevant data sources and techniques to retrieve high-quality results efficiently."

RAG Tool (Retrieval-Augmented Generation):
"Combines information retrieval from large knowledge bases with natural language generation capabilities, allowing agents to provide substantive responses that are grounded in factual evidence while maintaining conversational fluency."

Search Alerts Tool: 
"Enables agents to continuously monitor search results and data streams for predefined patterns, anomalies or events of interest, triggering notifications or actions when relevant information is detected."

Search Index Tool:
"Provides unified access to search across multiple indices and data sources, with capabilities for structuring queries, filtering, ranking and processing results based on the agent's intelligence layer."

Vector DB Tool:
"Allows efficient storage, indexing and querying of high-dimensional vector embeddings, a core requirement for many modern AI/ML workloads ranging from semantic search to recommendation systems."

Visualization Tool:
"Generates dynamic visualizations and renders them in multiple formats based on raw data inputs and higher-level insights produced by the agent's reasoning processes."

Web Search Tool:
"Provides a structured interface for accessing and processing results from web search engines, enabling agents to gather relevant information from the open internet while applying techniques to enhance relevance and filter out low-quality sources."

</tool_descriptions>

The release notes covered major architectural upgrades like the ML Commons Agent framework iterations, new vector search capabilities (indexing, filtering, quantization), and AI workflow enhancements (conversational agents, RAG, multimodal search). I provided detailed descriptions of 10 core Agent Tools that empower developers to build sophisticated AI agents by composing reusable modules for capabilities like information retrieval, query planning, data visualization, internet access and more.""",
      "origin": "LLM",
      "parent_message_id": "SpVkPpwB84_tbshlbP2n",
      "trace_number": 7
    }

SO it seems the agent already got a response, but not returnning back to update the task id.

looking at logs, found these exceptions

org.opensearch.OpenSearchStatusException: Failed to update data object in index .plugins-ml-task	at org.opensearch.remote.metadata.client.impl.LocalClusterIndicesClient.lambda$updateDataObjectAsync$2(LocalClusterIndicesClient.java:253)	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)	at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:124)	at org.opensearch.action.update.TransportUpdateAction.lambda$innerExecute$0(TransportUpdateAction.java:207)	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)	at org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleException(TransportInstanceSingleOperationAction.java:262)	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:442)	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1607)	at org.opensearch.transport.NativeMessageHandler.lambda$handleException$0(NativeMessageHandler.java:495)	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341)	at org.opensearch.transport.NativeMessageHandler.handleException(NativeMessageHandler.java:493)	at org.opensearch.transport.NativeMessageHandler.handlerResponseError(NativeMessageHandler.java:485)	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:195)	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:149)	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:152)	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:144)	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:804)	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137)	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77)	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124)	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113)	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95)	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:354)	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:354)	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1540)	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1398)	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1449)	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356)	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429)	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:172)	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445)	at io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)	at io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)	at io.netty.channel.nio.NioIoHandler.processSelectedKeysPlain(NioIoHandler.java:541)	at io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:514)	at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)	at io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)	at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)	at java.base/java.lang.Thread.run(Thread.java:1474)

Caused by: java.lang.IllegalArgumentException: Document contains at least one immense term in field="response._value" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[123, 34, 95, 105, 110, 100, 101, 120, 34, 58, 34, 111, 112, 101, 110, 115, 101, 97, 114, 99, 104, 45, 114, 101, 108, 101, 97, 115, 101, 34]...', original message: bytes can be at most 32766 in length; got 80204	at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1318)	at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1196)	at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:745)	at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:618)	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)	at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1573)	at org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1890)	at org.opensearch.index.engine.LuceneIndexWriter.softUpdateDocument(LuceneIndexWriter.java:165)	at org.opensearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1387)	at org.opensearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1174)	at org.opensearch.index.engine.InternalEngine.index(InternalEngine.java:959)	at org.opensearch.index.shard.IndexShard.index(IndexShard.java:1303)	at org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1242)	at org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:1133)	at org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:655)	at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)	at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)	at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)	at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)	at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:977)	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)	at java.lang.Thread.run(Thread.java:1474)
--
1770573332539

Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 80204	at org.apache.lucene.util.BytesRefBlockPool.addBytesRef(BytesRefBlockPool.java:88)	at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:348)	at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:195)	at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1300)	... 26 more
--

org.opensearch.OpenSearchStatusException: Failed to update data object in index .plugins-ml-task
	at org.opensearch.remote.metadata.client.impl.LocalClusterIndicesClient.lambda$updateDataObjectAsync$2(LocalClusterIndicesClient.java:253)
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
	at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:124)
	at org.opensearch.action.update.TransportUpdateAction.lambda$innerExecute$0(TransportUpdateAction.java:207)
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
	at org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleException(TransportInstanceSingleOperationAction.java:262)
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:442)
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1607)
	at org.opensearch.transport.NativeMessageHandler.lambda$handleException$0(NativeMessageHandler.java:495)
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341)
	at org.opensearch.transport.NativeMessageHandler.handleException(NativeMessageHandler.java:493)
	at org.opensearch.transport.NativeMessageHandler.handlerResponseError(NativeMessageHandler.java:485)
	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:195)
	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:149)
	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:152)
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:144)
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:804)
	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137)
	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77)
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124)
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113)
	at

What is the expected behavior?

the response is expected to be

{
  "task_type": "AGENT_EXECUTION",
  "function_name": "AGENT",
  "state": "COMPLETED",
  "worker_node": [
    "q5yAqa75RM-rv0I67V1VVQ"
  ],
  "create_time": 1746148548710,
  "last_update_time": 1746148706345,
  "is_async": false,
  "response": {
    "memory_id": "bzWQjpYBKhItn1nNYHtu",
    "inference_results": [
      {
        "output": [
          {
            "result": "bzWQjpYBKhItn1nNYHtu",
            "name": "memory_id"
          },
          {
            "result": "cDWQjpYBKhItn1nNYHuS",
            "name": "parent_interaction_id"
          },
          {
            "result": "dTWQjpYBKhItn1nNbHsw",
            "name": "executor_agent_memory_id"
          },
          {
            "result": "YjWQjpYBKhItn1nN6oYk",
            "name": "executor_agent_parent_interaction_id"
          },
          {
            "name": "response",
            "dataAsMap": {
              "response": """# Comprehensive Analysis 
...}
 }
        ]
      }
    ]
  }
}

*** proposed solution:
Check the task index for the response field , if the llm response is string, when the response is too long, long than max_length 32766, try

1. truncate the response saving to response field not longer than 32766
2. for conversational agent, when final response is long, try invoke another llm call before confirming the final response

What is your host/environment?

• OS: linux
• Version 3.5
• Plugins