tortoise / tortoise/tortoise-orm

low performance on insensitive case

Open
#1,407 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Describe the bug
There's a performance Bug in the filters __iendswith, istartswith, iexact, icontains

To Reproduce
I have a table with 24M records,

class Company(Model):
    vat_number = fields.CharField(20, pk=True)
    name = fields.CharField(200, null=True)
    creation_date = fields.DateField(null=True)
    legal_category = fields.CharField(10, null=True)
    business_activity = fields.CharField(10, null=True)
    headquarters_nic = fields.CharField(5, null=True)
    created_at = fields.DateField(auto_now_add=True, null=True)

    class Meta:
        indexes = [
            GinIndex(fields=("name",))
        ]

when i use:
companies = await Company.filter(name__istartswith="something)

it takes about 25 seconds to retrieve data from the database

after checking the generated sql

print(await Company.filter(name__istartswith="something).as_query())

# SELECT "headquarters_nic","name","creation_date","created_at","vat_number","business_activity","legal_category" FROM "company" WHERE UPPER(CAST("name" AS VARCHAR)) LIKE UPPER('something%') ESCAPE '\' LIMIT 50

so the problem is the UPPER(CAST("name" AS VARCHAR)) if sql makes upper to all records to make a compare, it will take a lot of time.

Expected behavior
the solution, at least for postgres is to generate the follow sql:

SELECT "headquarters_nic","name","creation_date","created_at","vat_number","business_activity","legal_category" FROM "company" WHERE CAST("name" AS VARCHAR) ILIKE something%' ESCAPE '\' LIMIT 50

with this option the time to retrieve a response passed to about 300ms

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the Company.filter(name__istartswith=...) entry point and trace how the ORM generates SQL for insensitive filters. Compare the PostgreSQL query shown in the issue with the expected ILIKE form, then identify the relevant tests or database-specific query path. Done means the PostgreSQL filters avoid the reported UPPER expression without breaking other database backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.