algolia / algolia/search-bundle
[RFC] Split Large Records Support
- Dominant language
- PHP
- Stars
- 193
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
## Background
Algolia recommends splitting large records (e.g. blog posts) into smaller chunks for better search relevance. There seems to be no support for this in the Symfony bundle
## Suggestion
I think that functionality should be implemented in a flexible way, allowing anyone to define decoupled business logic around that. In order to do that from my perspective couple basic principles has to be met:
- object ids should contain FQCN as well as original ID (similar to aggregators)
- chunk splitting should not collide with aggregations
- original ID should be persisted in a separate field in the index (configurable per index)
- entities should not contain logic for splitting
- each index should be able to define field used for splitting
- chunks should be easily invalidated
## Architecture
So far this is more a rough idea than a plan. First of all I believe the chunk splitting should happen after all normalizers were executed. The bundle should not interfere with normalization, especially since AFAIK you cannot emit multiple objects for a single normalized object.
I suggest that configuration gets a format similar to:
```yaml
algolia_search:
prefix: '%kernel.environment%_'
settingsDirectory: /config/algolia_search
chunk_id_transformer: app.foo.id_transformer # by default set to a service provided by the bundle, can be customized
indices:
- name: foos
class: App\Entity\Foo
chunking:
enabled: true
id_transformer: app.foo.id_transformer # by default set to 'algolia_search.chunk_id_transformer'
body_transformer: app.foo.chunk_transformer
context:
- custom_marker
- something more
```
Additionally two new interfaces should be introduced:
**ChunkTransformerInterface**
```php
Contributor guide
Research direction
No files or tests are named. Start by tracing the bundle's normalization, aggregation, indexing, and invalidation flows, then compare them with the proposed configuration and transformer interfaces. Done means an agreed design or implementation that supports configurable chunking without entity logic, preserves original IDs, avoids aggregation collisions, and allows chunks to be invalidated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php, symfony
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100