tortoise / tortoise/tortoise-orm
low performance on insensitive case
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 516
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 9
Description
Describe the bug
There's a performance Bug in the filters __iendswith, istartswith, iexact, icontains
To Reproduce
I have a table with 24M records,
class Company(Model):
vat_number = fields.CharField(20, pk=True)
name = fields.CharField(200, null=True)
creation_date = fields.DateField(null=True)
legal_category = fields.CharField(10, null=True)
business_activity = fields.CharField(10, null=True)
headquarters_nic = fields.CharField(5, null=True)
created_at = fields.DateField(auto_now_add=True, null=True)
class Meta:
indexes = [
GinIndex(fields=("name",))
]
when i use:
companies = await Company.filter(name__istartswith="something)
it takes about 25 seconds to retrieve data from the database
after checking the generated sql
print(await Company.filter(name__istartswith="something).as_query())
# SELECT "headquarters_nic","name","creation_date","created_at","vat_number","business_activity","legal_category" FROM "company" WHERE UPPER(CAST("name" AS VARCHAR)) LIKE UPPER('something%') ESCAPE '\' LIMIT 50
so the problem is the UPPER(CAST("name" AS VARCHAR)) if sql makes upper to all records to make a compare, it will take a lot of time.
Expected behavior
the solution, at least for postgres is to generate the follow sql:
SELECT "headquarters_nic","name","creation_date","created_at","vat_number","business_activity","legal_category" FROM "company" WHERE CAST("name" AS VARCHAR) ILIKE something%' ESCAPE '\' LIMIT 50
with this option the time to retrieve a response passed to about 300ms
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the Company.filter(name__istartswith=...) entry point and trace how the ORM generates SQL for insensitive filters. Compare the PostgreSQL query shown in the issue with the expected ILIKE form, then identify the relevant tests or database-specific query path. Done means the PostgreSQL filters avoid the reported UPPER expression without breaking other database backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100