algolia / algolia/search-bundle

[RFC] Split Large Records Support

Open
#293 4 comments 0 reactions 0 assignees View on GitHub
Feature Request
Dominant language
PHP
Stars
193
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Background
Algolia recommends splitting large records (e.g. blog posts) into smaller chunks for better search relevance. There seems to be no support for this in the Symfony bundle

## Suggestion
I think that functionality should be implemented in a flexible way, allowing anyone to define decoupled business logic around that. In order to do that from my perspective couple basic principles has to be met:
- object ids should contain FQCN as well as original ID (similar to aggregators)
- chunk splitting should not collide with aggregations
- original ID should be persisted in a separate field in the index (configurable per index)
- entities should not contain logic for splitting
- each index should be able to define field used for splitting
- chunks should be easily invalidated

## Architecture
So far this is more a rough idea than a plan. First of all I believe the chunk splitting should happen after all normalizers were executed. The bundle should not interfere with normalization, especially since AFAIK you cannot emit multiple objects for a single normalized object.

I suggest that configuration gets a format similar to:
```yaml
algolia_search:
prefix: '%kernel.environment%_'
settingsDirectory: /config/algolia_search
chunk_id_transformer: app.foo.id_transformer # by default set to a service provided by the bundle, can be customized

indices:
- name: foos
class: App\Entity\Foo
chunking:
enabled: true
id_transformer: app.foo.id_transformer # by default set to 'algolia_search.chunk_id_transformer'
body_transformer: app.foo.chunk_transformer
context:
- custom_marker
- something more
```

Additionally two new interfaces should be introduced:
**ChunkTransformerInterface**
```php

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing the bundle's normalization, aggregation, indexing, and invalidation flows, then compare them with the proposed configuration and transformer interfaces. Done means an agreed design or implementation that supports configurable chunking without entity logic, preserves original IDs, avoids aggregation collisions, and allows chunks to be invalidated.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, symfony
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.