huggingface / huggingface/data-measurements-tool

Handle the case where there are multiple label columns

Open
#36 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
75
Forks
18
Avg merge
6m
Merged PRs (30d)
1

Description

I would like to use data-measurements-tool for my NER dataset. However, the label field `ner_tags` consists of a list of tags.

Consequently, `python3 run_data_measurements.py` crashes with a TypeError` (see details below).

I traced to `labels.py`'s `def _prepare_labels(self)` comment
```
# TODO: Handle the case where there are multiple label columns.
# The logic throughout the code assumes only one.
```
It seems `DMTHelper` can already handle multiple label names, so would it make sense to use the labels `ner_tags_1` to `ner_tags_x` with `x` being the sequence length? I haven't yet investigated the statistics reports you create (since `python3 run_data_measurements.py` crashes), but guess having `x` independent labels doesn't make sense because the order of elements in the sequence is of little relevance for a statistics report.

When do you think `_prepare_labels()` will be capable of handling NER datasets?

Stacktrace:
```
[data_measurements_tool] {/home/davef/git/data-measurements-tool/run_data_measurements.py:328} run_data_measurements ERROR - int() argument must be a string, a bytes-like object or a number, not 'list'
Traceback (most recent call last):
File "/home/davef/git/data-measurements-tool/run_data_measurements.py", line 304, in main
pass_args_to_DMT(
File "/home/davef/git/data-measurements-tool/run_data_measurements.py", line 155, in pass_args_to_DMT
load_or_prepare(dataset_args, calculation=calculation, use_cache=use_cache)
File "/home/davef/git/data-measurements-tool/run_data_measurements.py", line 97, in load_or_prepare
dstats.load_or_prepare_labels()
File "/home/davef/git/data-measurements-tool/data_measurements/dataset_statistics.py", line 389, in load_or_prepare_labels
label_obj.run_DMT_processing()
File "/home/davef/git/data-measurements-tool/data_measurements/labels/labels.py", line 122, in run_DMT_processing
self.label_results = self._prepare_labels()
File "/home/davef/git/data-measurements-tool/data_measurements/labels/labels.py", line 161, in _prepare_labels
label_results = label_obj.prepare_labels(label_field, self.label_names)
File "/home/davef/git/data-measurements-tool/data_measurements/labels/labels.py", line 205, in prepare_labels
label_measurement = label_distribution.compute(data=label_list)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/evaluate/module.py", line 432, in compute
self.add_batch(**inputs)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/evaluate/module.py", line 480, in add_batch
self.current_features = self._infer_feature_from_batch(batch)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/evaluate/module.py", line 551, in _infer_feature_from_batch
return self._infer_feature_from_example(example)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/evaluate/module.py", line 560, in _infer_feature_from_example
features.encode_example(example)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/datasets/features/features.py", line 1579, in encode_example
return encode_nested_example(self, example)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/datasets/features/features.py", line 1148, in encode_nested_example
{
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/datasets/features/features.py", line 1149, in
k: encode_nested_example(sub_schema, sub_obj, level=level + 1)
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/datasets/features/features.py", line 1203, in encode_nested_example
return schema.encode_example(obj) if obj is not None else None
File "/home/davef/anaconda3/envs/data309/lib/python3.9/site-packages/datasets/features/features.py", line 465, in encode_example
return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
[data_measurements_tool] {/home/davef/git/data-measurements-tool/run_data_measurements.py:336} run_data_measurements WARNING - Data measurements not computed. ☹️
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with `python3 run_data_measurements.py`, then read `data_measurements/labels/labels.py`, especially `_prepare_labels()` and `prepare_labels()`, alongside the traceback. Determine how sequence-valued NER labels should be represented in the statistics reports; done means the command completes for the reported dataset and produces valid measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.