biolab / biolab/orange3

TAB/CSV files read differently on different operating systems

Open
#6,067 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.7k
Forks
1.1k
Avg merge
12d 2h
Merged PRs (30d)
1

Description

**What's wrong?**
Data from TAB/CSV files with `'` characters after delimiter (e.g. at the beginning of the sentence) are sometimes misread on Linux/macOS. Look at the next section for details.

**How can we reproduce the problem?**
1. Open File widget
2. Load https://github.com/biolab/orange3-text/blob/master/orangecontrib/text/datasets/book-excerpts.tab file
3. Connect the Data Table and observe that in row 133 (in the Text column), the `'` character at the beginning of the text is missing (it is present in the TAB file). The error only appears on Linux/macOS; it works correctly on Windows (but there might be the case that it would fail on Windows too).

The reason is that Sniffer (https://github.com/biolab/orange3/blob/99b196651b206b448a9c3120a34954179578d876/Orange/data/io.py#L147-L152) recognizes `'` as quote character of CSV and hence this character is removed while reading. In the same case,`"` is recognized as a delimiter on Windows (so it works correctly there).

**What's your environment?**
- Operating system: macOS/Linux
- Orange version: master/3.32

Contributor guide

Open the contributing guide

Research direction

Start in Orange/data/io.py around lines 147-152, where Sniffer determines the CSV dialect. Reproduce the issue by loading the linked book-excerpts.tab file through the File widget and checking row 133 in Data Table on Linux or macOS. Done means the leading apostrophe remains present while tab-delimited data continues to load correctly across the reported operating systems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.