SQL The problem of long tables
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1.1k
- Avg merge
- 12d 2h
- Merged PRs (30d)
- 1
Description
**What's wrong?**
Unexpected shrinking of a table when loading from a database. Longer than > 1 000 000
I know that there is a limit on the maximum table length = 1 000 000. But I think we can be a little friendlier.
We have different behavior depending on the database.
### MS SQL
If table length > 1 000 000 then nothing happens.
I think we can display an error message if the data could not be loaded.
### Postge SQL
If the length of the table is > 1 000 000, then it greatly decreases when loading.
For example, when loading a sequence of numbers from 0 to 1,000,100. We get a table of length 10.400 with a minimum value of 39.220 and a maximum value of 979.019. This is a very strange result.
I propose to discuss solutions:
1. Display an error message
2. [Slicing ranges](https://pandas.pydata.org/docs/user_guide/indexing.html#slicing-ranges) Similar to pandas based on size
**How can we reproduce the problem?**
[jupyter notebook](https://github.com/Mikhaylov-yv/Open_source/blob/main/df_to_db.ipynb)
**What's your environment?**
- Operating system: win11
- Orange version: '3.35.0.dev0+3cc58b4'
- How you installed Orange: from source code
Contributor guide
Research direction
Start with the linked df_to_db.ipynb and reproduce the loading behavior against MS SQL and PostgreSQL using a table longer than 1,000,000 rows. Trace where the database result is limited or reduced; done should mean the data is not silently truncated and the chosen behavior—an error message or documented range slicing—is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100