blekhmanlab / blekhmanlab/compendium_website
Subsample user input samples
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I just realized there's another another relatively unusual operation we'll need to figure out for the projectionist app: In the user's input table, each row is a sample, and the sum of each row is the total read count for that sample. It's almost always less than 100,000 reads per sample, but a few big projects end up having 1 million+. Here's what we found in the original compendium dataset:
To minimize the effect of some samples/projects having way more reads, we need to *rarefy* each sample down to 3000 reads. Starting with a pool of, say, 100,000 reads in a microbiome sample, we need to select a random subset of 3000 reads (without replacement). Unfortunately, I'm not aware of a specific tool for doing this when the items are counted already, rather than just living in an array of length 10,000.
In case I've managed to make this even more confusing, here's a sketch of what it might look like:
**Original data**
| Sample | Tax 1 | Tax 2 | Tax 3 | Tax 4 | Tax 5 | READ TOTAL |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| s123 | 400 | 200 | 1000 | 1100 | 1900 | 4600 |
| s124 | 1000 | 5000 | 8400 | 200 | 10 | 14610 |
| s125 | 0 | 1 | 8000 | 7000 | 1400 | 16401 |
**Subsampled**
| Sample | Tax 1 | Tax 2 | Tax 3 | Tax 4 | Tax 5 | READ TOTAL |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| s123 | 300 | 150 | 600 | 650 | 1300 | 3000 |
| s124 | 250 | 1000 | 1700 | 50 | 0 | 3000 |
| s125 | 0 | 0 | 1400 | 1300 | 300 | 3000 |
These subsampled values would be the ones that are then CLR-transformed.
For what it's worth, this step has an R implementation in [vegan](https://www.rdocumentation.org/packages/vegan/versions/2.7-2/topics/rarefy) and will be implemented in microbiomap to generate production-quality figures. If it would help to set some kind of limit in the browser—rejecting files with any samples with >250,000 reads, say—it would be trivial to add a helper function that takes the user input file and generates a new, rarefied one.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source files, tests, or entry points are named. Start by reviewing the issue's vegan rarefy reference and decide whether the browser should perform random subsampling or reject oversized inputs; done means the chosen scope produces or enforces 3000-read samples before CLR transformation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data, web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100