epam / epam/Indigo

Bingo NoSQL `enumerateId` and `delete` methods apparently out of sync

Open
#1,176 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
406
Forks
134
Avg merge
2d 11h
Merged PRs (30d)
24

Description

**Steps to Reproduce**
Using Bingo NoSQL, python 3.x epam.indigo library v1.11.0. My built indices were indicating that there were duplicate entries across all 16 databases, I attempted to enumerate and compare the set of IDs present in each.

Running the following script on a bingo nosql database (split across 16 directories), attempting to delete any entries which were already present in a previously-processed index:

```
def deduplicate(indigo, path):
ids = {}
for idx in tqdm(range(16), total=16, desc=f"Checking ID uniqueness for {path}"):
db = bingo.Bingo.loadDatabaseFile(indigo, f"{path}/db{idx+1:02}")
current_idx = idx + 1
ids[current_idx] = current_ids = set()
result = db.enumerateId()
while result.next():
current_ids.add(result.getCurrentId())
for other_idx in range(idx):
if other_idx < idx:
overlap = current_ids & ids[other_idx+1]
if overlap:
print(f"Remove {len(overlap)} from db{current_idx:02} duplicated with db{other_idx+1:02}")
for id in tqdm(overlap, desc=f"Pruning db{current_idx:02}", leave=False):
try:
db.delete(id)
current_ids.remove(id)
except BingoException as e:
print(f"delete {id} failed: {e}")
print(f"After removal db{current_idx:02} {len(current_ids)}")
db.optimize()
updated = set()
result = db.enumerateId()
while result.next():
updated.add(result.getCurrentId())
print(f"After optimization: {len(updated)}")
db.close()
```

**Expected behavior**
I expected that after "delete", the list generated by `enumerateId` would no longer contain the deleted IDs.

**Actual behavior**
The enumerated IDs still include deleted IDs. An attempt to delete IDs contained in the enumerated IDs list results in a BingoException, `delete xxxxx failed: There is no object with this id`. I have not tested whether the deleted ID still shows up in the queries that were showing duplicates.

**Indigo/Bingo version**
python epam.indigo 1.11.0

**Additional context**
`enumerateId` is not documented nor is its expected usage.
`optimize` is not documented nor is its intended usage.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the supplied deduplicate script with Python epam.indigo 1.11.0, focusing on enumerateId(), delete(), optimize(), and the post-optimization enumeration. Compare the returned IDs with deletion results, then determine the intended documented behavior for these methods and add or update coverage and documentation so the observed mismatch is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.