cityofaustin / cityofaustin/techstack

Implement ElasticSearch on Janis

Open
#4,919 1 comment 0 reactions 0 assignees View on GitHub
DevOps Epic Janis Alpha Team: Dev
Dominant language
No language data
Stars
11
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Right now our elasticsearch service is hardcoded to be turned off in our static.config.js.

```
const withElasticsearch = false
const indexName = `local_${langCode}_${Date.now()}`
```

If you're running locally, you can manually set `withElasticsearch = true` and run `yarn run elasticsearch`. This will run a local elasticsearch server that searchIndexBuilder will submit a searchIndex to. That's as far as I've gotten with development. I didn't implement querying elasticsearch from the search page yet. So that's one issue.

To make this work on any environment, not just locally, there will be some devops tasks to accomplish. If a Janis wants to use elasticsearch it will need to:
- have an elasticsearch server (this will have to be created in AWS or Elasticsearch-as-a-service)
- know the address of that elasticsearch server (to know where to send the searchIndex during build time, and where to query it during search time)
- know which index to use on that elasticsearch server
- That's why our indexName is coded by `${branchName}_${langCode}_${Date.now()}`.
- We don't want just 1 index per Janis.
- Why? Because then our search results won't be 100% in sync. Let's say Janis PR#500 version A is deployed, but we're now building Janis PR#500 version B. Janis version A and B both get their search results from the same outside elasticsearch server address (rather than a searchIndex embedded on the Search page). If we only had 1 Janis index per Janis branch, then version B's index would be created halfway through version B's build process. So for 5 minutes or so, the current Janis version A's searchIndex would be using data for version B, which is inaccurate. And what if the build fails? Then version A will be stuck searching version B's searchIndex.
- So with an environment variable, we tell Janis version B's search page to only query Janis version B's searchIndex. That way there is never any downtime/inaccuracy during builds.
- Once Janis version B is fully deployed, then version A no longer exists. So we can create a callback within publisher to go into elasticsearch and delete version A's searchIndexes. (Multiple indexes because there's one for every language.)
- We'll know which index to delete because we set a timestamp for each indexName. So the oldest index for a ${branchName} can be deleted if there's a more recent one present.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.