biocore / biocore/empress

Identify common (sample) metadata values, and replace them with unique integers

Open
#337 6 comments 0 reactions 1 assignee Claimed by @fedarko View on GitHub
performance
Dominant language
JavaScript
Stars
56
Forks
32
PR merge metrics
No merged PRs in 30d

Description

@kwcantrell brought this up in this morning's meeting. Currently sample metadata is stored as follows in the HTML (this is from the moving pictures dataset, formatting modified for ease of reading):

```js
["barcode-sequence", "body-site", "year", "month", "day", "subject", "reported-antibiotic-usage", "days-since-experiment-start"],
[["AGTGCGATGCGT", "gut", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["ATGGCAGCTCTA", "gut", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["CTGAGATACGCG", "gut", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CCGACTGAGATG", "gut", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["CCTCTCGTGATC", "gut", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["ACACACTATGGC", "gut", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["ACTACGTGTGGT", "gut", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["AGCTGACTAGTC", "gut", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["ACGATGCGACCA", "left palm", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["AGCTATCCACGA", "left palm", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["ATGCAGCTCAGT", "left palm", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CACGTGACATGT", "left palm", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["CATATCGCAGTT", "left palm", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["CGTGCATTATCA", "left palm", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CTAACGCAGTCA", "left palm", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["CTCAATGACTCA", "left palm", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["ACAGTTGCGCGA", "right palm", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["CACGACAGGCTA", "right palm", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["AGTGTCACGGTG", "right palm", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["CAAGTGAGAGAG", "right palm", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CATCGTATCAAC", "right palm", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["ATCGATCTGTGG", "right palm", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["GCGTTACACACA", "right palm", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["GAACTGTATCTC", "right palm", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["CTCGTGGAGTAG", "right palm", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CAGTGTCAGGAC", "tongue", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["ATCTTAGACTGC", "tongue", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["CAGACATTGCGT", "tongue", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["CGATGCACCAGA", "tongue", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CTAGAGACTCTT", "tongue", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["CTGGACTCATAG", "tongue", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["GAGGCTCATCAT", "tongue", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["GATACGTCCTGA", "tongue", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["GATTAGCACTCT", "tongue", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"]]
```

It would be useful to identify common string values in the metadata, map these to unique integers, and then replace these values in the metadata. Then, when looking up sample metadata info in the BIOM table or something, numeric values would be replaced with their original string value. (This'd work because all metadata is stored as strings in Empress right now.)

An example of what this might look like, by just replacing ten nonunique values I arbitrarily picked:

```js
{"gut": 0, "left palm": 1, "right palm": 2, "tongue": 3, "2008.0": 4, "2009.0": 5, "subject-1": 6, "subject-2": 7, "No": 8, "Yes": 9},
["barcode-sequence", "body-site", "year", "month", "day", "subject", "reported-antibiotic-usage", "days-since-experiment-start"],
[["AGTGCGATGCGT", 0, 5, "3.0", "17.0", 6, 8, "140.0"],
["ATGGCAGCTCTA", 0, 4, "10.0", "28.0", 7, 9, "0.0"],
["CTGAGATACGCG", 0, 5, "1.0", "20.0", 7, 8, "84.0"],
["CCGACTGAGATG", 0, 5, "3.0", "17.0", 7, 8, "140.0"],
["CCTCTCGTGATC", 0, 5, "4.0", "14.0", 7, 8, "168.0"],
["ACACACTATGGC", 0, 5, "1.0", "20.0", 6, 8, "84.0"],
["ACTACGTGTGGT", 0, 5, "2.0", "17.0", 6, 8, "112.0"],
["AGCTGACTAGTC", 0, 4, "10.0", "28.0", 6, 9, "0.0"],
["ACGATGCGACCA", 1, 5, "1.0", "20.0", 6, 8, "84.0"],
["AGCTATCCACGA", 1, 5, "2.0", "17.0", 6, 8, "112.0"],
["ATGCAGCTCAGT", 1, 5, "3.0", "17.0", 6, 8, "140.0"],
["CACGTGACATGT", 1, 5, "4.0", "14.0", 6, 8, "168.0"],
["CATATCGCAGTT", 1, 4, "10.0", "28.0", 7, 9, "0.0"],
["CGTGCATTATCA", 1, 5, "1.0", "20.0", 7, 8, "84.0"],
["CTAACGCAGTCA", 1, 5, "3.0", "17.0", 7, 8, "140.0"],
["CTCAATGACTCA", 1, 5, "4.0", "14.0", 7, 8, "168.0"],
["ACAGTTGCGCGA", 2, 4, "10.0", "28.0", 6, 9, "0.0"],
["CACGACAGGCTA", 2, 5, "1.0", "20.0", 6, 8, "84.0"],
["AGTGTCACGGTG", 2, 5, "2.0", "17.0", 6, 8, "112.0"],
["CAAGTGAGAGAG", 2, 5, "3.0", "17.0", 6, 8, "140.0"],
["CATCGTATCAAC", 2, 5, "4.0", "14.0", 6, 8, "168.0"],
["ATCGATCTGTGG", 2, 4, "10.0", "28.0", 7, 9, "0.0"],
["GCGTTACACACA", 2, 5, "3.0", "17.0", 7, 8, "140.0"],
["GAACTGTATCTC", 2, 5, "4.0", "14.0", 7, 8, "168.0"],
["CTCGTGGAGTAG", 2, 5, "1.0", "20.0", 7, 8, "84.0"],
["CAGTGTCAGGAC", 3, 4, "10.0", "28.0", 6, 9, "0.0"],
["ATCTTAGACTGC", 3, 5, "1.0", "20.0", 6, 8, "84.0"],
["CAGACATTGCGT", 3, 5, "2.0", "17.0", 6, 8, "112.0"],
["CGATGCACCAGA", 3, 5, "3.0", "17.0", 6, 8, "140.0"],
["CTAGAGACTCTT", 3, 5, "4.0", "14.0", 6, 8, "168.0"],
["CTGGACTCATAG", 3, 4, "10.0", "28.0", 7, 9, "0.0"],
["GAGGCTCATCAT", 3, 5, "1.0", "20.0", 7, 8, "84.0"],
["GATACGTCCTGA", 3, 5, "3.0", "17.0", 7, 8, "140.0"],
["GATTAGCACTCT", 3, 5, "4.0", "14.0", 7, 8, "168.0"]]
```

The metadata looks _a lot_ smaller (and I haven't even replaced nonunique stuff in the month/day/etc. fields). For massive datasets with lots of metadata (e.g. the EMP) this could be really useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.