JanusGraph / JanusGraph/janusgraph
cannot update list property when value contains special characters
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 5.8k
- Forks
- 1.2k
- Avg merge
- 13h 53m
- Merged PRs (30d)
- 6
Description
I have a list property with composite index and mixed index built.
```
// build composite and mixed indexes
JanusGraphTransaction tx = graph.newTransaction();
JanusGraphVertex vertex1 = tx.addVertex(T.label, "Person");
vertex1.property("Person_emailList", "\"aa@qq.com\t");
long id = vertex1.longId();
System.out.println("id="+id);
tx.commit();
tx = graph.newTransaction();
Vertex vertex2 = tx.getVertex(id);
// str ends with a tab character
String str = "\"bb@qq.com ";
vertex2.property("Person_emailList", str);
tx.commit();
```
```
java.io.IOException: Failure(s) in Elasicsearch bulk request: {type=illegal_argument_exception, reason=failed to execute script, caused_by={type=script_exception, reason=compile error, script_stack=[... ["Person_emailList"].add("\"bb@qq.com\t");, ^---- HERE], script=if(ctx._source["Person_emailList"] == null){ctx._source["Person_emailList"] = [];}ctx._source["Person_emailList"].add("\"bb@qq.com\t");, lang=painless, caused_by={type=illegal_argument_exception, reason=unexpected character ["\"bb@qq.com\t]. The only valid escape sequences in strings starting with ["] are [\\] and [\"]., caused_by={type=lexer_no_viable_alt_exception, reason=null}}}}
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.bulkRequest(RestElasticSearchClient.java:274)
```
in getAdditionScript method, my update script is like this:
```
if(ctx._source["Person_emailList"] == null){ctx._source["Person_emailList"] = [];}ctx._source["Person_emailList"].add("\"bb@qq.com\t");
```
in bulkRequest method, after `outputStream.write(mapWriter.writeValueAsBytes(request.getSource()));` the script goes like this:
```
{"update":{"_index":"janusgraph_vertices","_type":"vertices","_id":"39s"}}
{"script":{"source":"if(ctx._source[\"Person_emailList\"] == null){ctx._source[\"Person_emailList\"] = [];}ctx._source[\"Person_emailList\"].add(\"\\\"bb@qq.com\\t\");","lang":"painless"},"upsert":{"Person_emailList":["\"bb@qq.com\t"]}}
```
I thought it was an ES bug. But they say it's not.
[painless cannot parse escaped characters correctly](https://github.com/elastic/elasticsearch/issues/27789)
I cannot step into outputStream.write method, so I changed getAdditionScript method like this and then I got the script I wanted.:
```
script.append("if(ctx._source[\"" + e.field + "\"] == null){ctx._source[\"" + e.field + "\"] = [];}");
script.append("ctx._source[\"").append(e.field).append("\"].add(").append(convertToPainlessType(e.value, compat.scriptLang(), Mapping.getMapping(keyInformation))).append(");");
if (hasDualStringMapping(keyInformation)) {
String mappingName = getDualMappingName(e.field);
script.append("if(ctx._source[\"" + mappingName + "\"] == null){ctx._source[\"" + mappingName + "\"] = [];}");
script.append("ctx._source[\"").append(mappingName).append("\"].add(").append(convertToPainlessType(e.value, compat.scriptLang(), Mapping.getMapping(keyInformation))).append(");");
}
```
```
private static String convertToPainlessType(Object value, String scriptLang, Mapping mapping) throws PermanentBackendException {
final String esValue;
try {
if(value instanceof String) {
esValue = "'"+value+"'";
} else {
esValue = mapWriter.writeValueAsString(convertToEsType(value, mapping));
}
} catch (final Exception e) {
throw new PermanentBackendException("Could not write json");
}
return scriptLang.equals("groovy") ? esValue.replace("$", "\\$") : esValue;
}
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the Elasticsearch backend's getAdditionScript and convertToPainlessType methods, then inspect bulkRequest where the update request is serialized. Reproduce the issue with a list property containing quotes and a tab, and compare the generated Painless script with the request sent to Elasticsearch. Done means the update succeeds for these special characters and the relevant regression coverage passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100