openml / openml/OpenML

Column names with '\%' are renamed

Open
#1,124 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
755
Forks
128
PR merge metrics
No merged PRs in 30d

Description

Description

Datasets with column names that contain '%' are escaped by pandas, but not in the list of feature names.
That creates errors whenever you first get the list of features and then try to look up some of them.
Not sure what the best action is. One possible solution is to always escape them, also in the feature list created by get_dataset.

I only found this to happen on one dataset so far, but there may be more. It's not hard to work around this, but it breaks automated tests.

Steps/Code to Reproduce

Example:

import openml
d = openml.datasets.get_dataset(70)
df, *_ = d.get_data(dataset_format="dataframe",include_row_id=True, include_ignore_attribute=True)
print(df.columns)
print([f.name for f in d.features.values()])
Expected Results

Identical lists of feature names

Actual Results

Some feature names are different, e.g. bw\%2Fme, blue\%2Fbright\%2Fvarn\%2Fclean
versus bw%2Fme, blue%2Fbright%2Fvarn%2Fclean

Versions

Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Python 3.7.11 (default, Jul 3 2021, 18:01:19)
[GCC 7.5.0]
NumPy 1.19.5
SciPy 1.4.1
Scikit-Learn 0.22.2.post1
OpenML 0.12.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the get_dataset and get_data entry points shown in the reproduction, then compare the returned dataframe columns with the names in d.features. Reproduce the mismatch using dataset 70 and verify that both lists contain identical feature names, including names containing escaped percent sequences.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.