algolia / algolia/search-bundle

Some entities imported via `search:import` are not indexed (missing records)

Open
#372 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PHP
Stars
193
Forks
74
PR merge metrics
No merged PRs in 30d

Description

- Symfony version: `v6.2.7`
- Algolia Search Bundle version: `6.0.0`
- Algolia Client Version: N/A
- Language Version: `PHP 8.1.14 (cli)`

### Description

When importing entities with `search:import`, the logs display correct index counts, but when browsing the index, some are missing.

Here is the command output:
```
> bin/console search:import
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 500 / 500 App\Entity\MediaTranslation entities into quentin_media index
Indexed 160 / 160 App\Entity\MediaTranslation entities into quentin_media index
Done!
```

I'd then expect my index to contain `14 * 500 + 160 = 7160` items, but only `5216` exist:

![image](https://user-images.githubusercontent.com/349104/224959124-d21933e8-ccf9-48a7-87cc-7de7fa716ee0.png)

But clearing the index and importing again yields another record count (+/-5%).

Here's my configuration:

```yaml
algolia_search:
prefix: '%algolia_search_prefix%'
indices:
- name: media
class: App\Entity\MediaTranslation
```

And here's the index settings file (created with `search:settings:backup`)

```json
{
"minWordSizefor1Typo": 4,
"minWordSizefor2Typos": 8,
"hitsPerPage": 20,
"maxValuesPerFacet": 100,
"version": 2,
"searchableAttributes": [
"unordered(media.id)",
"unordered(title)",
"unordered(tags)",
"unordered(description)",
"unordered(features)",
"unordered(goals)",
"unordered(more)"
],
"numericAttributesToIndex": null,
"attributesToRetrieve": null,
"unretrievableAttributes": null,
"optionalWords": null,
"attributesForFaceting": [
"locale",
"media.type",
"status",
"filterOnly(tags)",
"filterOnly(title)"
],
"attributesToSnippet": null,
"attributesToHighlight": null,
"paginationLimitedTo": 1000,
"attributeForDistinct": null,
"exactOnSingleWordQuery": "attribute",
"ranking": [
"typo",
"geo",
"words",
"filters",
"proximity",
"attribute",
"exact",
"custom"
],
"customRanking": null,
"separatorsToIndex": "",
"removeWordsIfNoResults": "none",
"queryType": "prefixLast",
"highlightPreTag": "",
"highlightPostTag": "<\/em>",
"snippetEllipsisText": "",
"alternativesAsExact": [
"ignorePlurals",
"singleWordSynonym"
],
"sortFacetValuesBy": "count",
"renderingContent": {
"facetOrdering": {
"facets": {
"order": [
"locale",
"media.type",
"status"
]
},
"values": {
"locale": {
"sortRemainingBy": "alpha"
},
"media.type": {
"sortRemainingBy": "alpha"
},
"status": {
"sortRemainingBy": "alpha"
}
}
}
}
}
```

I tried changing the `batchSize` but the issue remained.
I used to have a `index_if` in there, but removed it and the issue remained.

When running the `search:import` command and regularly refreshing the index on the Algolia dashboard, the "No. records" evolves like so (that's only an example, values change if I re-run this on a clear index):

- 500
- 1000
- 1,500
- 2,000
- 2,253
- 2,525
- 3,025
- (...)

As you can see, thinks looks OK at first, but then get a bit crazy around the 2000/2500 mark.

### Steps To Reproduce

Unfortunately this is hard to reproduce, because I can't pinpoint the origin of the issue (and the randomness makes it even stranger) 🙁

I tried looking at the Symfony logs to see if some error appeared there, but found nothing.

**What could prevent records from appearing in my index?**

Contributor guide

Open the contributing guide

Research direction

Start with the `search:import` command and the Algolia Search Bundle configuration for `App\Entity\MediaTranslation`; compare the reported batch counts with the records in the Algolia dashboard while reviewing Symfony logs. Done means identifying why records are intermittently missing and preventing the import from completing with an incomplete index.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.