tortoise / tortoise/tortoise-orm

select_related not preserved in ValuesQuery

Open
#2,004 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Hello!

I noticed that the optimization done in ValuesQuery is a bit too agressive when it comes to related selects.

Take the following example:

from tortoise.models import Model
from tortoise import Tortoise, fields

class Author(Model):
    age = fields.IntField()

class Book(Model):
    author = fields.ForeignKeyField("models.Author")

async def main():
    await Tortoise.init(db_url='sqlite://:memory:', modules={"models": ["example"]})
    query = (
        Book.all()
        .select_related("author")
        .annotate(custom=RawSQL("book__author.age * 2 :: BIGINT"))
        .order_by("custom")
    )
    print(query.sql())
    # SELECT "book"."id","book"."author_id",book__author.age * 2 :: BIGINT "custom","book__author"."id" "book__author.id","book__author"."age" "book__author.age" FROM "book" LEFT OUTER JOIN "author" "book__author" ON "book__author"."id"="book"."author_id" ORDER BY "custom" ASC

    print(query.values("id").sql())
    # SELECT "id" "id" FROM "book" ORDER BY book__author.age * 2 :: BIGINT ASC

As you can see in the second query, the JOIN is removed which results in an operational error, as book__author doesn't exist anymore. It seems Tortoise isn't aware that I need the join, despite having requested it explicitely via select_related.

I can imagine it's hard for the library to understand which related fields are needed when RawSQL is being used (I'm forced to use it for the cast, there's no way to my knowledge to cast types ORM-style), is there a way for me to force Tortoise to make that join?

Edit: I'd rather avoid having more than one column in the .values, I'm actually using .values_list("id", flat=True), which I can't do with two values.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ValuesQuery optimization involved in values() and values_list(), then reproduce the provided select_related, RawSQL, and order_by example against SQLite. Trace why the related join is removed and verify that the resulting SQL preserves the join needed by the RawSQL ordering while still returning only the requested value column.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.