Make synonyms_path setting dynamic for updatable synonym filters

Background

The synonym filters defined with "updateable": true are exclusively created for query time synonym analysis and unlike other filters, have a way to be reloaded into memory after the underlying synonym file has been updated. For example, analysis/synonym.txt (as shown in example below) file can be updated with new synonyms and dynamically reloaded into memory to return results based on the new synonym list

         "filter": {
          "synonym": {
            "type": "synonym_graph",
            "synonyms_path": "analysis/synonym.txt",  
            "updateable": true                        
          }
        }

The idea of having dynamic reload functionality(via refresh_search_analyzers api) is to avoid any downtime for query time analysis as by making these filters dynamically reloadable, we are able to prevent 1) re-indexing of index data or 2. close/open index, if the underlying synonyms list needs to be updated.

However, the synonyms_path setting, once created, cannot be changed (without close/open index) because the setting is not dynamic. Thus if synonyms list needs to be updated, the only way is to update the same underlying file (analysis/synonym.txt in the above example) and then invoke refresh_search_analyzers API.

However, refresh_search_analyzers API is not the only way to reload this updated analyzers into memory, it also happens inadvertently due to node restarts or opensearch process restarts. Also opensearch does not have a way to distinguish the updated file from the previous one as there is no versioning support internally. These things create a few problems as described below -

1. Query consistency issue - Unexpected opensearch process restarts can cause automatic reloading of updated synonym files (rebuilding of synonymMap) without user explicitly calling the refresh_search_analyzers API. What would happen in this case is requests going to shard on the restarted node would return different results from the request going to the same shard(replica) on a different node because of different synonymMap in memory. This breaks the query consistency as only the restarted node would return results based on updated synonyms list while the other nodes would still return results based on previous synonym list.
2. Problem with partial synonym file deployment - If the updated synonym file could not be deployed successfully to all nodes, the reload (either user triggered or from node restarts) still succeeds as it only cares for the presence of the file, irrespective of any version. This happens because opensearch does not have a way to distinguish the updated file from the existing one. This would again cause inconsistent results for queries as different nodes will have different synonymMap in memory.
3. No way for monitoring this behavior - Since opensearch does not know which version of the file is in memory used for synonym analysis, it becomes hard for user to successfully detect and monitor for such inconsistencies in production systems.
4. Index time analyzers could also be accidentally updated - Since opensearch does not block user from using the same underlying file for query time analysis on one index while index time analysis on other, if a user updates the file, the node restarts can lead to index time analysis also gets impacted as the new ingested documents would use the updated synonym list from the reloaded synonymMap. This breaks the immutability construct for index time analysis. (Just to be clear, the proposal below does not intend to solve this problem but provides a way to circumvent this, refer to Reusability point)

Proposal

The proposal is to make the synonyms_path setting, a dynamic setting, when defined with "updateable": true flag. The synonyms_path setting change only succeeds if the new file is present on all the nodes, thereby insuring from failed/partial deployments. This also eliminates the need for users to maintain file versioning at their end as they could simply upload a new file, that contains the updated synonym list, and point the synonyms_path to the new file.

curl -X PUT "localhost:9200/my_index/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index.analysis.filter.synonym.synonyms_path": "synonyms_v2.txt"
}'

By allowing user to change the synonyms_path, we can remove the need of file versioning as the files on the node become immutable.

How this proposal solves the problems highlighted with current behavior -

1. Guarantees all nodes have same updated version of the file - as user is not allowed to change the synonyms_path setting if the new file is missing on any node.
2. Query correctness guaranteed - Node restarts during partial/failed deployments would not lead to synonymMap reload as the approach requires user to explicitly change the synonyms_path to point to new synonym file on opensearch node.
3. No versioning concept needed - opensearch does not support the concept of versioning the files, so it would be better to keep them immutable so that there is no confusion for user as to what is being used for analysis in memory.
4. Reusability - same file can be used for query time analysis on one index and index time analysis on another (if need be) and if a user wishes to update the file later, the query time analysis synonym_path can be updated without impacting the index time analysis.

Start by tracing the index analysis settings for updateable synonym filters and the refresh_search_analyzers API described in the issue. The change is complete when an updateable filter can switch synonyms_path only after the new file is available on every node, while non-updateable filters retain their existing behavior.