tortoise / tortoise/tortoise-orm

Tortoise queries with long JOIN aliases raise exceptions when using Postgres as the backend database

Open
#1,902 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Describe the bug
Hey guys, I'm using tortoise in production and I found a Postgres specific bug.
Postgres has a limitation where identifiers must be at most 63 character long. This means in practice that if you create a column name or alias which is longer than 63 bytes long, Postgres will truncate it.
This limit can be configured but it requires re-compiling Postgres which is not practical when using cloud-managed solutions.

Here's an example

SELECT 1 AS "my_very_very_very_long_unpractical_and_almost_comical_column_name_hehe";

returns

my_very_very_very_long_unpractical_and_almost_comical_column_nam
---
1

This is a problem for tortoise which relies on columns aliases to resolve field names and build related models when doing joins, or even if any field has a very long name.

To Reproduce
This is a kind of comical example that abuses the length of field names. But note, in production models tend to have descriptive names and it is not uncommon to require many joins when executing a queryset with .select_related(), which generates very long aliases by joining field names using double underscores.

# file models.py
import tortoise
import uuid


class Author(tortoise.models.Model):
	id = tortoise.fields.UUIDField(primary_key=True, default=uuid.uuid4)


class Book(tortoise.models.Model):
	id = tortoise.fields.UUIDField(primary_key=True, default=uuid.uuid4)
	author_model_relation_with_long_name: tortoise.fields.ForeignKeyRelation[Author] = tortoise.fields.ForeignKeyField(
		"models.Author"
	)


class Chapter(tortoise.models.Model):
	id = tortoise.fields.UUIDField(primary_key=True, default=uuid.uuid4)
	book_model_relation_with_long_name: tortoise.fields.ForeignKeyRelation[Book] = tortoise.fields.ForeignKeyField(
		"models.Book"
	)
#file main.py
import models
import tortoise

async def main():
	await tortoise.Tortoise.init(
		db_url="postgres://user:password@host/dbname",
		modules={"models": ["models"]},
	)
	await tortoise.Tortoise.generate_schemas()

	author = await models.Author.create()
	book = await models.Book.create(author_model_relation_with_long_name=author)
	chapter = await models.Chapter.create(book_model_relation_with_long_name=book)

	# Joins generate colums aliases with lenghth >63 chars
	queryset = models.Chapter.all().select_related(
		"book_model_relation_with_long_name",
		"book_model_relation_with_long_name__author_model_relation_with_long_name",
	)
	print(queryset.sql(), end="\n\n")


	# upon selecting, column aliases get truncated to 63 chars in Postgres response records
	# tortoise does not expect return aliases to be truncated
	join = await queryset.latest("id")  # BOOM!

tortoise.run_async(main())

For executing

python main.py

Which raises

IndexError: list index out of range

in

tortoise/backends/base/executor.py", line 120, in execute_select
    obj = model._init_from_db(**{k.split(".")[1]: v for k, v in related_items})

Note that the issue is having an alias which is very long

SELECT
    ....
    ,"chapter__book_model_relation_with_long_name__author_model_relation_with_long_name"."id" "chapter__book_model_relation_with_long_name__author_model_relation_with_long_name.id"
    ...
FROM ...

and the returned alias from Postgres does not include the dot . that tortoise needed to split the field name from the model reference alias.

Expected behavior
I expect this not to raise an exception and work as any other select with joins.

Additional context
AS tortoise generates the SQL to be executed, it knows in which order the rows will be returned from asyncpg. Tortoise could probably use the column index of each row and map that index to the long alias name instead of relying in the alias returned by Postgres.
I'm not sure how hard this is to implement. If you guys could provide any comments I could give a try to solve this problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tortoise/backends/base/executor.py around execute_select line 120, then reproduce the failure using the models.py and main.py example against PostgreSQL. Inspect how joined rows are mapped when PostgreSQL truncates aliases. Done means the select_related query completes without IndexError and related models are built correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.