Load multiple associations in one single roundtrip
- Dominant language
- Go
- Stars
- 17.2k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
In https://github.com/ent/ent/issues/977#issuecomment-733077372 @a8m mentioned
> as mentioned in the [docs](https://entgo.io/docs/eager-load/#implementation): since a query-builder can load more than one association, it's not possible to load them using one `JOIN` operation. Therefore, ent executes additional queries for loading associations. One query for M2O/O2M and O2O edges, and 2 queries for loading M2M edges.
>
> In some cases, executing multiple queries reduces the number of rows returned from the database and the total time of the query - even if there are multiple roundtrips.
>
> One optimization I want to add is to use `JOIN` when loading only one association (edge), or at least give users an SQL specific API to tell ent how to load the edges (e.g. use `JOIN` even if the query builder contains more than 2 association).
Another thing that could reduce the number of roundtrips, possibly shaving off quite some time especially for large number of associations, could be to use (for RDBMSes that support it, like MySQL or PostgreSQL) the capability to [execute multiple statements in one single request](https://dev.mysql.com/doc/c-api/8.0/en/c-api-multiple-queries.html) (see also the [corresponding support in go-sql-driver/mysql](https://github.com/go-sql-driver/mysql#multistatements)) or to otherwise [pipeline queries](https://www.postgresql.org/docs/current/libpq-pipeline-mode.html). This would basically reduce the number of serialized application-level roundtrips to just one regardless of the number of associations.
There is obviously the question of, if needed, how to forward resultsets from one query to the next. The most generic way is clearly just that of (assuming we are inside a transaction) reexecuting the first query as a subquery of the next one, but ideally it would be great also to offer the option to "materialize" the resultset of the first query into a temporary table, and use the temporary table as source for the next query.
So instead of
```sql
SELECT col1, col2, ... FROM tbl1 WHERE ;
# wait for response
SELECT colA, colB, ... FROM tbl2 WHERE colA IN ();
# wait for response
SELECT colC, colD, ... FROM tbl3 WHERE colC IN ();
# wait for response
```
we could do
```sql
# note: assumes multi-statement queries or pipelining are enabled
SELECT col1, col2, ... FROM tbl1 WHERE ;
SELECT colA, colB, ... FROM tbl2 WHERE colA IN (SELECT col1 FROM tbl1 WHERE );
SELECT colC, colD, ... FROM tbl3 WHERE colC IN (SELECT col2 FROM tbl1 WHERE );
# wait for all responses
```
or, using a temporary table to avoid the repeated subqueries:
```sql
# note: assumes multi-statement queries or pipelining are enabled
CREATE TEMPORARY TABLE t1 AS SELECT col1, col2, ... FROM tbl1 WHERE ;
SELECT col1, col2, ... FROM t1;
SELECT colA, colB, ... FROM tbl2 WHERE colA IN (SELECT col1 FROM t1);
SELECT colC, colD, ... FROM tbl3 WHERE colC IN (SELECT col2 FROM t1);
DROP TEMPORARY TABLE t1;
# wait for all responses
```
An even more exotic solution, for RDBMSes that support it and possibly only for smallish resultsets, could be to load the associations as composite objects (e.g. JSON array of objects) directly in synthesized columns of the first query. Or, bringing the approach to the extreme, directly return the whole response as a single composite JSON value. This would potentially allow not only to reduce the roundtrips to one, but even to reduce the statement count to a single one, potentially allowing the RDBMS engine to optimize away identical subqueries, and executing association subqueries in parallel - but with the potential downside of requiring in some cases to transfer more data for M2O/M2M associations.
```sql
SELECT JSON_ARRAY(
(SELECT JSON_ARRAYAGG(JSON_ARRAY(col1, col2, ...)) FROM tbl1 WHERE ),
(SELECT JSON_ARRAYAGG(JSON_ARRAY(colA, colB, ...)) FROM tbl2 WHERE colA IN (SELECT col1 FROM tbl1 WHERE )),
(SELECT JSON_ARRAYAGG(JSON_ARRAY(colC, colD, ...)) FROM tbl3 WHERE colC IN (SELECT col2 FROM tbl1 WHERE ))
);
# wait for response
```
(these last more exotic approaches also avoid the need to explicitly start a read-only transaction, as everything happens in a single statement)
---
The reason I'm opening this issue is that we are actually running into relatively high latencies due to the many roundtrips when we operate on heavily normalized schemas. We have instances where queries require up to 7 associations to be loaded, and if the additional roundtrips were eliminated the whole operation would take half of the time (for small queries the roundtrip latency is comparable to the query execution time on the database).
Contributor guide
Assessment
This issue has not been assessed yet.