loopbackio / loopbackio/loopback-next

Support `inq` splitting in `findByForeignKeys`

Open
#3,444 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature needs discussion Relations Repository
Dominant language
TypeScript
Stars
5.1k
Forks
1.1k
Avg merge
2d 21h
Merged PRs (30d)
27

Description

Under the hood, inclusion resolvers are implemented using `inq` operator:

1. Gather PK/FK values from source models.
2. Query target models using `inq` and PK/FK values from step 1.
3. Assign target models to navigational property in source models.

This can be problematic when the number of source instances is large, we don't
know if all databases support `inq` with arbitrary number of items.

To address this issue, LB3 is implementing "inq splitting", where a single query
with arbitrary-sized `inq` condition is split into multiple queries where each
query has a reasonably-sized `inq` condition.

Connectors are allowed to specify the maximum `inq` size supported by the
database via `dataSource.settings.inqLimit` option. By default, `inqLimit` is
set to 256.

In this task, we need to improve `findByForeignKeys` (see https://github.com/strongloop/loopback-next/issues/3443) to handle the maximum size of `inq` parameter supported by the target database (data-source). When the list of provided FK values is too long, then we should split it into smaller chunks and execute multiple queries.

However, because our `Repository` interface is generic and does not assume that
a repository has to be backed by a data-source, I am proposing to expose
`inqLimit` via a new property of the `Repository` interface instead of accessing
the parameter via DataSource settings.

```ts
/**
* Description of capabilities offered by the connector backing the given
* datasource.
*/
export interface RepositoryCapabilities {
/**
* Maximum number of items allowed for `inq` operators.
* This value is used to split queries when resolving related models
* for a large number of source instances.
*/
inqLimit?: number;
}
```

To preserve backwards compatibility with existing repository implementation, we
cannot add `RepositoryCapabilities` directly to the `Repository` class. We need
to introduce a new interface instead that Repositories can (or may not)
implement.

```ts
export interface RepositoryWithCapabilities {
capabilities: RepositoryCapabilities;
}
```

See #3387 for more details & a prototype implementation.

## Acceptance criteria

To allow the helper to detect `inqLimit`, we need to extend Repository interfaces.

- [ ] Introduce `RepositoryCapabilities` interface (called `ConnectorCapabilities` in the spike), this interface will have a single property `inqLimit` (for now).
- [ ] Introduce `RepositoryWithCapabilities` interface (called `WithCapabilities` in the spike),= this interface should define `capabilities` property.
- [ ] Implement `isRepositoryWithCapabilities` type guard
- [ ] Implement `getRepositoryCapabilities` helper

The rest should be straightforward:

- [ ] Modify `findByForeignKeys` to obtain `inqLimit` from repository capabilities and implement query splitting (see the spike implementation).
- [ ] Write unit-level tests where we verify what (and how many) queries are called by `findByForeignKeys`.
- [ ] Write integration-level tests (in `repository-tests`) to verify that connectors can handle `inqLimit` they are advertising. For example, create a test that runs a query returning 1000 records.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with findByForeignKeys and the repository interfaces, then review the prototype referenced in #3387. Trace how repository capabilities are detected and inspect the repository-tests area before adding unit and integration coverage. Done means capability helpers exist, large foreign-key lists are split according to inqLimit, and the expected number of queries is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.