aws / aws/aws-appsync-community
[Feature Request] Support bulk API (NDJSON) request to OpenSearch
- Dominant language
- HTML
- Stars
- 507
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
I have a pipeline resolver for an AppSync Mutation. It contains two functions, the first one is a Lambda sending updates to RDS, the second one should take the result from `$ctx.prev.result` and index it via OpenSearch datasource.
In the request resolver mapping template of the second one, I am composing the bulk body in NDJSON similar to the following mannar:
```vtl
#set($bulk = $util.toJson({ "index": { "_id": "${$ctx.prev.result.id}" } }))
#set($bulk = "${bulk}
${util.toJson($ctx.prev.result)}")
{
"version": "2017-02-28",
"operation": "POST",
"path": "/_bulk",
"params": {
"body": $util.toJson($bulk)
}
}
```
Lacking proper debugging tools, I have been using `$util.error` as a logging method to get my `$bulk` contents. And it looks like the following format, which seems correct.
```ndjson
{"index":{"_id":"A8DEF210-C342-48CB-9A4A-DA7D1E4D6AF1"}}
{"foo":123,"bar":999,"baz":1234567}
```
But when I actually runs the mutation via AppSync, I got a `MappingTemplate` error `Unable to transform for the body: $[params][body].`.
I took a look at [[re:Post] Appsync HTTP resolver supported content types](https://repost.aws/questions/QUFqBom4iFQ-6ePM1z11Sthw/appsync-http-resolver-supported-content-types), which inspired me to take another look at [Resolver Mappping Template for OpenSearch (params)](https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-elasticsearch.html#params-field). It seems that POST body only accepts a single JSON object.
1. Am I correct that NDJSON is not currently supported yet?
2. If so, is supporting the bulk API in the upcoming plans?
3. What is the currently recommended way to index multiple "normalized" documents from the same resolver?
Contributor guide
Research direction
No repository files or tests are named. Start with the request resolver mapping template shown in the issue and the linked OpenSearch resolver-parameter documentation, then verify how the /_bulk NDJSON body is transformed. Done would require a clear determination of bulk-request support and the recommended approach for indexing multiple documents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, graphql
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100