data races in org.apache.lucene.queryParser.QueryParserTokenManager [LUCENE-5286]

Hi,

We ran a dynamic detector on Lucene and found quite a number of races in org.apache.lucene.queryParser.QueryParserTokenManager when an instance of this class is shared by multiple threads. Almost all field variables of this class are racey, e.g., curChar, curLexState jjnewStateCnt jjmatchedPos jjmatchedKind.

For example, methods "jjMoveStringLiteralDfa0_3" and "getNextToken" can be executed by different threads at the same time with no synchronization. The read and write to "curChar" at lines 65 and 1174 may corrupt the parser.

63 private int More jjMoveStringLiteralDfa0_3()
64 {
65 switch(curChar)
66 {

1164public Token More getNextToken()
1165{
1166 Token matchedToken;
1167 int curPos = 0;
1168
1169 EOFLoop :
1170 for (;;)
1171 {
1172 try
1173 {
1174 curChar = input_stream.BeginToken();
1175 }

---
Migrated from [LUCENE-5286](https://issues.apache.org/jira/browse/LUCENE-5286) by Jeff Huang

Start with org.apache.lucene.queryParser.QueryParserTokenManager, especially getNextToken and jjMoveStringLiteralDfa0_3, and inspect how their shared fields and input_stream are used across threads. Determine how concurrent use should be made safe and define completion around preventing corruption when one instance is shared by multiple threads.

Written by the indexing model from the issue text.