JanusGraph / JanusGraph/janusgraph

cannot update list property when value contains special characters

Open
#841 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

I have a list property with composite index and mixed index built.

```
// build composite and mixed indexes

JanusGraphTransaction tx = graph.newTransaction();
JanusGraphVertex vertex1 = tx.addVertex(T.label, "Person");
vertex1.property("Person_emailList", "\"aa@qq.com\t");
long id = vertex1.longId();
System.out.println("id="+id);
tx.commit();
tx = graph.newTransaction();
Vertex vertex2 = tx.getVertex(id);
// str ends with a tab character
String str = "\"bb@qq.com ";
vertex2.property("Person_emailList", str);
tx.commit();
```

```
java.io.IOException: Failure(s) in Elasicsearch bulk request: {type=illegal_argument_exception, reason=failed to execute script, caused_by={type=script_exception, reason=compile error, script_stack=[... ["Person_emailList"].add("\"bb@qq.com\t");, ^---- HERE], script=if(ctx._source["Person_emailList"] == null){ctx._source["Person_emailList"] = [];}ctx._source["Person_emailList"].add("\"bb@qq.com\t");, lang=painless, caused_by={type=illegal_argument_exception, reason=unexpected character ["\"bb@qq.com\t]. The only valid escape sequences in strings starting with ["] are [\\] and [\"]., caused_by={type=lexer_no_viable_alt_exception, reason=null}}}}
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:274)
```

in getAdditionScript method, my update script is like this:

```
if(ctx._source["Person_emailList"] == null){ctx._source["Person_emailList"] = [];}ctx._source["Person_emailList"].add("\"bb@qq.com\t");
```
in bulkRequest method, after `outputStream.write(mapWriter.writeValueAsBytes(request.getSource()));` the script goes like this:

```
{"update":{"_index":"janusgraph_vertices","_type":"vertices","_id":"39s"}}
{"script":{"source":"if(ctx._source[\"Person_emailList\"] == null){ctx._source[\"Person_emailList\"] = [];}ctx._source[\"Person_emailList\"].add(\"\\\"bb@qq.com\\t\");","lang":"painless"},"upsert":{"Person_emailList":["\"bb@qq.com\t"]}}
```
I thought it was an ES bug. But they say it's not.
[painless cannot parse escaped characters correctly](https://github.com/elastic/elasticsearch/issues/27789)

I cannot step into outputStream.write method, so I changed getAdditionScript method like this and then I got the script I wanted.:

```
script.append("if(ctx._source[\"" + e.field + "\"] == null){ctx._source[\"" + e.field + "\"] = [];}");
script.append("ctx._source[\"").append(e.field).append("\"].add(").append(convertToPainlessType(e.value, compat.scriptLang(), Mapping.getMapping(keyInformation))).append(");");
if (hasDualStringMapping(keyInformation)) {
String mappingName = getDualMappingName(e.field);
script.append("if(ctx._source[\"" + mappingName + "\"] == null){ctx._source[\"" + mappingName + "\"] = [];}");
script.append("ctx._source[\"").append(mappingName).append("\"].add(").append(convertToPainlessType(e.value, compat.scriptLang(), Mapping.getMapping(keyInformation))).append(");");
}
```
```
private static String convertToPainlessType(Object value, String scriptLang, Mapping mapping) throws PermanentBackendException {
final String esValue;
try {
if(value instanceof String) {
esValue = "'"+value+"'";
} else {
esValue = mapWriter.writeValueAsString(convertToEsType(value, mapping));
}
} catch (final Exception e) {
throw new PermanentBackendException("Could not write json");
}
return scriptLang.equals("groovy") ? esValue.replace("$", "\\$") : esValue;
}
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the Elasticsearch backend's getAdditionScript and convertToPainlessType methods, then inspect bulkRequest where the update request is serialized. Reproduce the issue with a list property containing quotes and a tab, and compare the generated Painless script with the request sent to Elasticsearch. Done means the update succeeds for these special characters and the relevant regression coverage passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.