jackc / jackc/pgx

Provide improved support for loading types

Open
#2,030 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

I am using pgx with a database which defines a very large number of custom types. Virtually every stored procedure returns a custom type, and any given application needs at least 50-100 types, often more.
The current `LoadType` connection method is helpful, but is limited:
- `LoadType` generates two queries to the database when one would be sufficient.
- it can only be given one type at a time, and then requiring at least one round-trip to the database to complete. Coupled with the impracticality of running multiple `LoadType` calls concurrently, this adds significant overhead to an application's startup, before the dataclass connection/pool can be used at all.
- it is painful to have to manually calculate the types which will be required in the application. It is not only necessary to specify the desired type, but to recursively calculate all the other types these depend on, and to load them in, in the correct order.

I would like to create a PR to resolve these issues, either in separate PRs or together, if I get the green light. Are these seen as valid concerns, and would improvements in these areas be accepted?

To briefly describe what I am doing locally, and which would form the basis of my PR(s):
- (the first point above is quite trivial: the first `QueryRow` calls made in `LoadType` is used in the second query)
- add a new method named `LoadTypes` which takes a `[]string` of type names. Modify the (now) single SQL query to use `= ANY($1)` to find all OIDs matching the provided type names, and return `[]*pgtype.Type`.
- add a helper method `GetTypeDependencies` which, given a connection and a list of type names, recursively identifies the sub-types they implicitly also need registered. This can either be called internally by `LoadType` and the new `LoadTypes` or if that is considered too "magicial", can be kept separate for the user to invoke. e.g. `conn.LoadTypes(ctx, conn.GetTypeDependencies(ctx, "my_type1", "my_type_2", ...))`

The SQL I am using to assist in the above:
```sql
WITH RECURSIVE typeinfo AS (
SELECT c.relname AS parent, regexp_replace(pg_catalog.format_type(a.atttypid, a.atttypmod), '\[\]$', '') as child, 'f'::bool AS has_array
FROM pg_catalog.pg_class c
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
JOIN pg_catalog.pg_attribute a ON (a.attrelid = c.oid)
JOIN pg_type ON (pg_type.typname = regexp_replace(pg_catalog.format_type(a.atttypid, a.atttypmod), '\[\]$', ''))
WHERE pg_catalog.pg_table_is_visible(c.oid)
AND a.attnum > 0 AND NOT a.attisdropped
UNION ALL
SELECT typname AS parent, regexp_replace(pg_catalog.format_type(typbasetype, typtypmod) , '\[\]$', '') AS child, (pg_type.typarray > 0)::bool AS has_array
FROM pg_type
),
relationships(parent, child, has_array, depth) AS (
SELECT typeinfo.parent, typeinfo.child, typeinfo.has_array, 0
FROM typeinfo
WHERE parent = ANY($1)
UNION ALL
SELECT typeinfo.parent, typeinfo.child, typeinfo.has_array, relationships.depth + 1
FROM typeinfo
JOIN relationships ON (typeinfo.parent = relationships.child)
)
SELECT child, has_array
FROM relationships
INNER JOIN pg_type ON (pg_type.typname = relationships.child)
WHERE typtype != 'b'
GROUP BY child, has_array
ORDER BY max(depth) DESC, child;
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing LoadType connection method and compare its current queries with the proposed LoadTypes and GetTypeDependencies entry points. Review the recursive PostgreSQL query and resolve the accepted API and dependency-loading behavior; done means the agreed loading approach supports the described custom-type use case without the current startup overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.