Identify common (sample) metadata values, and replace them with unique integers
- Dominant language
- JavaScript
- Stars
- 56
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
@kwcantrell brought this up in this morning's meeting. Currently sample metadata is stored as follows in the HTML (this is from the moving pictures dataset, formatting modified for ease of reading):
```js
["barcode-sequence", "body-site", "year", "month", "day", "subject", "reported-antibiotic-usage", "days-since-experiment-start"],
[["AGTGCGATGCGT", "gut", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["ATGGCAGCTCTA", "gut", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["CTGAGATACGCG", "gut", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CCGACTGAGATG", "gut", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["CCTCTCGTGATC", "gut", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["ACACACTATGGC", "gut", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["ACTACGTGTGGT", "gut", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["AGCTGACTAGTC", "gut", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["ACGATGCGACCA", "left palm", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["AGCTATCCACGA", "left palm", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["ATGCAGCTCAGT", "left palm", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CACGTGACATGT", "left palm", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["CATATCGCAGTT", "left palm", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["CGTGCATTATCA", "left palm", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CTAACGCAGTCA", "left palm", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["CTCAATGACTCA", "left palm", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["ACAGTTGCGCGA", "right palm", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["CACGACAGGCTA", "right palm", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["AGTGTCACGGTG", "right palm", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["CAAGTGAGAGAG", "right palm", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CATCGTATCAAC", "right palm", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["ATCGATCTGTGG", "right palm", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["GCGTTACACACA", "right palm", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["GAACTGTATCTC", "right palm", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"],
["CTCGTGGAGTAG", "right palm", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["CAGTGTCAGGAC", "tongue", "2008.0", "10.0", "28.0", "subject-1", "Yes", "0.0"],
["ATCTTAGACTGC", "tongue", "2009.0", "1.0", "20.0", "subject-1", "No", "84.0"],
["CAGACATTGCGT", "tongue", "2009.0", "2.0", "17.0", "subject-1", "No", "112.0"],
["CGATGCACCAGA", "tongue", "2009.0", "3.0", "17.0", "subject-1", "No", "140.0"],
["CTAGAGACTCTT", "tongue", "2009.0", "4.0", "14.0", "subject-1", "No", "168.0"],
["CTGGACTCATAG", "tongue", "2008.0", "10.0", "28.0", "subject-2", "Yes", "0.0"],
["GAGGCTCATCAT", "tongue", "2009.0", "1.0", "20.0", "subject-2", "No", "84.0"],
["GATACGTCCTGA", "tongue", "2009.0", "3.0", "17.0", "subject-2", "No", "140.0"],
["GATTAGCACTCT", "tongue", "2009.0", "4.0", "14.0", "subject-2", "No", "168.0"]]
```
It would be useful to identify common string values in the metadata, map these to unique integers, and then replace these values in the metadata. Then, when looking up sample metadata info in the BIOM table or something, numeric values would be replaced with their original string value. (This'd work because all metadata is stored as strings in Empress right now.)
An example of what this might look like, by just replacing ten nonunique values I arbitrarily picked:
```js
{"gut": 0, "left palm": 1, "right palm": 2, "tongue": 3, "2008.0": 4, "2009.0": 5, "subject-1": 6, "subject-2": 7, "No": 8, "Yes": 9},
["barcode-sequence", "body-site", "year", "month", "day", "subject", "reported-antibiotic-usage", "days-since-experiment-start"],
[["AGTGCGATGCGT", 0, 5, "3.0", "17.0", 6, 8, "140.0"],
["ATGGCAGCTCTA", 0, 4, "10.0", "28.0", 7, 9, "0.0"],
["CTGAGATACGCG", 0, 5, "1.0", "20.0", 7, 8, "84.0"],
["CCGACTGAGATG", 0, 5, "3.0", "17.0", 7, 8, "140.0"],
["CCTCTCGTGATC", 0, 5, "4.0", "14.0", 7, 8, "168.0"],
["ACACACTATGGC", 0, 5, "1.0", "20.0", 6, 8, "84.0"],
["ACTACGTGTGGT", 0, 5, "2.0", "17.0", 6, 8, "112.0"],
["AGCTGACTAGTC", 0, 4, "10.0", "28.0", 6, 9, "0.0"],
["ACGATGCGACCA", 1, 5, "1.0", "20.0", 6, 8, "84.0"],
["AGCTATCCACGA", 1, 5, "2.0", "17.0", 6, 8, "112.0"],
["ATGCAGCTCAGT", 1, 5, "3.0", "17.0", 6, 8, "140.0"],
["CACGTGACATGT", 1, 5, "4.0", "14.0", 6, 8, "168.0"],
["CATATCGCAGTT", 1, 4, "10.0", "28.0", 7, 9, "0.0"],
["CGTGCATTATCA", 1, 5, "1.0", "20.0", 7, 8, "84.0"],
["CTAACGCAGTCA", 1, 5, "3.0", "17.0", 7, 8, "140.0"],
["CTCAATGACTCA", 1, 5, "4.0", "14.0", 7, 8, "168.0"],
["ACAGTTGCGCGA", 2, 4, "10.0", "28.0", 6, 9, "0.0"],
["CACGACAGGCTA", 2, 5, "1.0", "20.0", 6, 8, "84.0"],
["AGTGTCACGGTG", 2, 5, "2.0", "17.0", 6, 8, "112.0"],
["CAAGTGAGAGAG", 2, 5, "3.0", "17.0", 6, 8, "140.0"],
["CATCGTATCAAC", 2, 5, "4.0", "14.0", 6, 8, "168.0"],
["ATCGATCTGTGG", 2, 4, "10.0", "28.0", 7, 9, "0.0"],
["GCGTTACACACA", 2, 5, "3.0", "17.0", 7, 8, "140.0"],
["GAACTGTATCTC", 2, 5, "4.0", "14.0", 7, 8, "168.0"],
["CTCGTGGAGTAG", 2, 5, "1.0", "20.0", 7, 8, "84.0"],
["CAGTGTCAGGAC", 3, 4, "10.0", "28.0", 6, 9, "0.0"],
["ATCTTAGACTGC", 3, 5, "1.0", "20.0", 6, 8, "84.0"],
["CAGACATTGCGT", 3, 5, "2.0", "17.0", 6, 8, "112.0"],
["CGATGCACCAGA", 3, 5, "3.0", "17.0", 6, 8, "140.0"],
["CTAGAGACTCTT", 3, 5, "4.0", "14.0", 6, 8, "168.0"],
["CTGGACTCATAG", 3, 4, "10.0", "28.0", 7, 9, "0.0"],
["GAGGCTCATCAT", 3, 5, "1.0", "20.0", 7, 8, "84.0"],
["GATACGTCCTGA", 3, 5, "3.0", "17.0", 7, 8, "140.0"],
["GATTAGCACTCT", 3, 5, "4.0", "14.0", 7, 8, "168.0"]]
```
The metadata looks _a lot_ smaller (and I haven't even replaced nonunique stuff in the month/day/etc. fields). For massive datasets with lots of metadata (e.g. the EMP) this could be really useful.
Contributor guide
Assessment
This issue has not been assessed yet.