biolab / biolab/orange3

SQL The problem of long tables

Open
#6,427 1 comment 0 reactions 0 assignees View on GitHub
bug report
Dominant language
Python
Stars
5.7k
Forks
1.1k
Avg merge
12d 2h
Merged PRs (30d)
1

Description

**What's wrong?**
Unexpected shrinking of a table when loading from a database. Longer than > 1 000 000

I know that there is a limit on the maximum table length = 1 000 000. But I think we can be a little friendlier.
We have different behavior depending on the database.

### MS SQL

If table length > 1 000 000 then nothing happens.
I think we can display an error message if the data could not be loaded.

### Postge SQL

If the length of the table is > 1 000 000, then it greatly decreases when loading.
For example, when loading a sequence of numbers from 0 to 1,000,100. We get a table of length 10.400 with a minimum value of 39.220 and a maximum value of 979.019. This is a very strange result.

I propose to discuss solutions:

1. Display an error message
2. [Slicing ranges](https://pandas.pydata.org/docs/user_guide/indexing.html#slicing-ranges) Similar to pandas based on size

**How can we reproduce the problem?**
[jupyter notebook](https://github.com/Mikhaylov-yv/Open_source/blob/main/df_to_db.ipynb)

**What's your environment?**

- Operating system: win11
- Orange version: '3.35.0.dev0+3cc58b4'
- How you installed Orange: from source code

Contributor guide

Open the contributing guide

Research direction

Start with the linked df_to_db.ipynb and reproduce the loading behavior against MS SQL and PostgreSQL using a table longer than 1,000,000 rows. Trace where the database result is limited or reduced; done should mean the data is not silently truncated and the chosen behavior—an error message or documented range slicing—is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.