loopbackio / loopbackio/loopback-next
Support `inq` splitting in `findByForeignKeys`
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 5.1k
- Forks
- 1.1k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 27
Description
Under the hood, inclusion resolvers are implemented using `inq` operator:
1. Gather PK/FK values from source models.
2. Query target models using `inq` and PK/FK values from step 1.
3. Assign target models to navigational property in source models.
This can be problematic when the number of source instances is large, we don't
know if all databases support `inq` with arbitrary number of items.
To address this issue, LB3 is implementing "inq splitting", where a single query
with arbitrary-sized `inq` condition is split into multiple queries where each
query has a reasonably-sized `inq` condition.
Connectors are allowed to specify the maximum `inq` size supported by the
database via `dataSource.settings.inqLimit` option. By default, `inqLimit` is
set to 256.
In this task, we need to improve `findByForeignKeys` (see https://github.com/strongloop/loopback-next/issues/3443) to handle the maximum size of `inq` parameter supported by the target database (data-source). When the list of provided FK values is too long, then we should split it into smaller chunks and execute multiple queries.
However, because our `Repository` interface is generic and does not assume that
a repository has to be backed by a data-source, I am proposing to expose
`inqLimit` via a new property of the `Repository` interface instead of accessing
the parameter via DataSource settings.
```ts
/**
* Description of capabilities offered by the connector backing the given
* datasource.
*/
export interface RepositoryCapabilities {
/**
* Maximum number of items allowed for `inq` operators.
* This value is used to split queries when resolving related models
* for a large number of source instances.
*/
inqLimit?: number;
}
```
To preserve backwards compatibility with existing repository implementation, we
cannot add `RepositoryCapabilities` directly to the `Repository` class. We need
to introduce a new interface instead that Repositories can (or may not)
implement.
```ts
export interface RepositoryWithCapabilities {
capabilities: RepositoryCapabilities;
}
```
See #3387 for more details & a prototype implementation.
## Acceptance criteria
To allow the helper to detect `inqLimit`, we need to extend Repository interfaces.
- [ ] Introduce `RepositoryCapabilities` interface (called `ConnectorCapabilities` in the spike), this interface will have a single property `inqLimit` (for now).
- [ ] Introduce `RepositoryWithCapabilities` interface (called `WithCapabilities` in the spike),= this interface should define `capabilities` property.
- [ ] Implement `isRepositoryWithCapabilities` type guard
- [ ] Implement `getRepositoryCapabilities` helper
The rest should be straightforward:
- [ ] Modify `findByForeignKeys` to obtain `inqLimit` from repository capabilities and implement query splitting (see the spike implementation).
- [ ] Write unit-level tests where we verify what (and how many) queries are called by `findByForeignKeys`.
- [ ] Write integration-level tests (in `repository-tests`) to verify that connectors can handle `inqLimit` they are advertising. For example, create a test that runs a query returning 1000 records.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with findByForeignKeys and the repository interfaces, then review the prototype referenced in #3387. Trace how repository capabilities are detected and inspect the repository-tests area before adding unit and integration coverage. Done means capability helpers exist, large foreign-key lists are split according to inqLimit, and the expected number of queries is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100