apache / apache/couchdb

[Bug]: Flaky Elixir Tests

Open
#5,901 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

### Version

unknown

### Describe the problem you're encountering

Noticed a lot of elixir tests are flaky. Almost every CI run has at least a few failed elixir tests. They retry but sometimes even 3 retries are not enough. It's not an urgent issue, of course, as retries cover it up but it does make the tests run a bit longer

```
05:36:45 1) test limit (BasicFindTest)
05:36:45 test/elixir/test/mango/02_basic_find_test.exs:177
05:36:45 ** (FunctionClauseError) no function clause matching in Access.get/3
05:36:45
05:36:45 The following arguments were given to Access.get/3:
05:36:45
05:36:45 # 1
05:36:45 {"error", "not_found"}
05:36:45
05:36:45 # 2
05:36:45 "id"
05:36:45
05:36:45 # 3
05:36:45 nil
05:36:45
05:36:45 Attempted function clauses (showing 6 out of 6):
05:36:45
05:36:45 def get(-%module{} = container-, key, default)
05:36:45 def get(map, key, default) when -is_map(map)-
05:36:45 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:36:45 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:36:45 def get(list, key, _default) when -is_list(list)-
05:36:45 def get(-nil-, _key, default)
05:36:45
05:36:45 stacktrace:
05:36:45 (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:36:45 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:36:45 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:36:45 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:36:45 (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
05:36:45 test/elixir/test/mango/02_basic_find_test.exs:13: BasicFindTest.__ex_unit__/2
05:36:45
```

```
DiskMonitorTest [test/elixir/test/disk_monitor_test.exs]
07:33:38 * test block interactive view indexing [L#30]
07:33:38 * test block interactive view indexing (30.9ms) [L#30]
07:33:38
07:33:38 1) test block interactive view indexing (DiskMonitorTest)
07:33:38 test/elixir/test/disk_monitor_test.exs:30
07:33:38 Assertion with == failed
07:33:38 code: assert resp.status_code == 507
07:33:38 left: 200
07:33:38 right: 507
07:33:38 stacktrace:
07:33:38 test/elixir/test/disk_monitor_test.exs:41: (test)
```

In this case all 3 retries failed in a row on two separate worker (so 6 total elixir failures): https://ci-couchdb.apache.org/job/jenkins-cm1/job/FullPlatformMatrix/job/main/1871/pipeline-console/?selected-node=374

```
2) test choose index most accurate (ChooseCorrectIndexForDocs)
04:56:17 test/elixir/test/mango/12_use_correct_index_test.exs:81
04:56:17 ** (FunctionClauseError) no function clause matching in Access.get/3
04:56:17
04:56:17 The following arguments were given to Access.get/3:
04:56:17
04:56:17 # 1
04:56:17 {"error", "not_found"}
04:56:17
04:56:17 # 2
04:56:17 "id"
04:56:17
04:56:17 # 3
04:56:17 nil
04:56:17
04:56:17 Attempted function clauses (showing 6 out of 6):
04:56:17
04:56:17 def get(-%module{} = container-, key, default)
04:56:17 def get(map, key, default) when -is_map(map)-
04:56:17 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:56:17 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:56:17 def get(list, key, _default) w
```

```
ChooseCorrectIndexForDocs [test/elixir/test/mango/12_use_correct_index_test.exs]
04:58:13 * test choose index with id [L#128]
04:58:13 * test choose index with id (41.1ms) [L#128]
04:58:14 * test chooses idxA [L#105]
04:58:14 * test chooses idxA (19.5ms) [L#105]
04:58:14
04:58:14 1) test chooses idxA (ChooseCorrectIndexForDocs)
04:58:14 test/elixir/test/mango/12_use_correct_index_test.exs:105
04:58:14 ** (FunctionClauseError) no function clause matching in Access.get/3
04:58:14
04:58:14 The following arguments were given to Access.get/3:
04:58:14
04:58:14 # 1
04:58:14 {"error", "not_found"}
04:58:14
04:58:14 # 2
04:58:14 "id"
04:58:14
04:58:14 # 3
04:58:14 nil
04:58:14
04:58:14 Attempted function clauses (showing 6 out of 6):
04:58:14
04:58:14 def get(-%module{} = container-, key, default)
04:58:14 def get(map, key, default) when -is_map(map)-
04:58:14 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:58:14 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:58:14 def get(list, key, _default) when -is_list(list)-
04:58:14 def get(-nil-, _key, default)
04:58:14
04:58:14 stacktrace:
04:58:14 (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:58:14 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:58:14 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:58:14 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:58:14 test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
04:58:14 test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2
04:58:14
```

```
05:01:11 1) test warn on full db scan (ChooseCorrectIndexForDocs)
05:01:11 test/elixir/test/mango/12_use_correct_index_test.exs:97
05:01:11 ** (FunctionClauseError) no function clause matching in Access.get/3
05:01:11
05:01:11 The following arguments were given to Access.get/3:
05:01:11
05:01:11 # 1
05:01:11 {"error", "not_found"}
05:01:11
05:01:11 # 2
05:01:11 "id"
05:01:11
05:01:11 # 3
05:01:11 nil
05:01:11
05:01:11 Attempted function clauses (showing 6 out of 6):
05:01:11
05:01:11 def get(-%module{} = container-, key, default)
05:01:11 def get(map, key, default) when -is_map(map)-
05:01:11 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:01:11 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:01:11 def get(list, key, _default) when -is_list(list)-
05:01:11 def get(-nil-, _key, default)
05:01:11
05:01:11 stacktrace:
05:01:11 (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:01:11 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:01:11 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:01:11 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:01:11 test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
05:01:11 test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2
```

```
04:51:48 1) test json range (BeginsWithOperator)
04:51:48 test/elixir/test/mango/25_beginswith_test.exs:60
04:51:48 ** (FunctionClauseError) no function clause matching in Access.get/3
04:51:48
04:51:48 The following arguments were given to Access.get/3:
04:51:48
04:51:48 # 1
04:51:48 {"error", "not_found"}
04:51:48
04:51:48 # 2
04:51:48 "id"
04:51:48
04:51:48 # 3
04:51:48 nil
04:51:48
04:51:48 Attempted function clauses (showing 6 out of 6):
04:51:48
04:51:48 def get(-%module{} = container-, key, default)
04:51:48 def get(map, key, default) when -is_map(map)-
04:51:48 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:51:48 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:51:48 def get(list, key, _default) when -is_list(list)-
04:51:48 def get(-nil-, _key, default)
04:51:48
04:51:48 stacktrace:
04:51:48 (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:51:48 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:51:48 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:51:48 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:51:48 test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:51:48 test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
```

```
) test use index without fallback fails for empty index (JSONIndexSelectionTest)
04:53:33 test/elixir/test/mango/05_index_selection_test.exs:284
04:53:33 ** (FunctionClauseError) no function clause matching in Access.get/3
04:53:33
04:53:33 The following arguments were given to Access.get/3:
04:53:33
04:53:33 # 1
04:53:33 {"error", "not_found"}
04:53:33
04:53:33 # 2
04:53:33 "id"
04:53:33
04:53:33 # 3
04:53:33 nil
04:53:33
04:53:33 Attempted function clauses (showing 6 out of 6):
04:53:33
04:53:33 def get(-%module{} = container-, key, default)
04:53:33 def get(map, key, default) when -is_map(map)-
04:53:33 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:53:33 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:53:33 def get(list, key, _default) when -is_list(list)-
04:53:33 def get(-nil-, _key, default)
04:53:33
04:53:33 stacktrace:
04:53:33 (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:53:33 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:53:33 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:53:33 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:53:33 (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
04:53:33 test/elixir/test/mango/05_index_selection_test.exs:274: JSONIndexSelectionTest.__ex_unit__/2
```

```
04:57:04 1) test json range (BeginsWithOperator)
04:57:04 test/elixir/test/mango/25_beginswith_test.exs:60
04:57:04 ** (FunctionClauseError) no function clause matching in Access.get/3
04:57:04
04:57:04 The following arguments were given to Access.get/3:
04:57:04
04:57:04 # 1
04:57:04 {"error", "not_found"}
04:57:04
04:57:04 # 2
04:57:04 "id"
04:57:04
04:57:04 # 3
04:57:04 nil
04:57:04
04:57:04 Attempted function clauses (showing 6 out of 6):
04:57:04
04:57:04 def get(-%module{} = container-, key, default)
04:57:04 def get(map, key, default) when -is_map(map)-
04:57:04 def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:57:04 def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:57:04 def get(list, key, _default) when -is_list(list)-
04:57:04 def get(-nil-, _key, default)
04:57:04
04:57:04 stacktrace:
04:57:04 (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:57:04 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:57:04 (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:57:04 (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:57:04 test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:57:04 test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
04:57:04
04:57:04
```

I think a common theme is around `{error, not_found}`. Maybe setup needs a tweak or need an extra clause just for not found with an internal retry?

### Expected Behaviour

Flaky failures would be a bit more rare.

### Steps to Reproduce

_No response_

### Your Environment

_No response_

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failures in test/elixir/test/support/mango_database.exs and the named Mango and DiskMonitor tests, especially the Access.get/3 errors and status-code assertion. Trace the failing setup and test conditions across repeated runs. Done means the reported Elixir tests no longer fail intermittently or require repeated CI retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.