Save / Load with multiple data inputs / outputs
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 228
- Forks
- 81
- Avg merge
- 1d 29m
- Merged PRs (30d)
- 1
Description
I'm frequently dealing with a really weird tabular data structure called a PCL file - basically, the first n rows contain various forms of metadata (first column for these rows is metadata name), then below that it has a sample x feature matrix. So for example, I might have my_data.pcl:
| age | 12 | 14 | 18 |
|---|---|---|---|
| gender | m | f | f |
| sampleID | a | b | c |
| feature1 | 0.2 | 0.4 | 0.3 |
| feature2 | 0.3 | 0.2 | 0.3 |
| feature3 | 0.1 | 0.6 | 0.7 |
Since most of the operations occur on the numerical table part, and storing this all in a single dataframe (or whatever) would generally lead to columns with type Any, what I'd like to be able to do is have load/save functions that make/take two iterable tables, that share the sampleID row, eg the two tables would be:
metadatadf:
| sampleID | a | b | c |
|---|---|---|---|
| age | 12 | 14 | 18 |
| gender | m | f | f |
featuredf:
| sampleID | a | b | c |
|---|---|---|---|
| feature1 | 0.2 | 0.4 | 0.3 |
| feature2 | 0.3 | 0.2 | 0.3 |
| feature3 | 0.1 | 0.6 | 0.7 |
And I'd like to be able to do something like:
(x, y) = load("my_data.pcl", id_row="sampleID")
metadatadf = DataFrame(x)
featuredf = DataFrame(y)
save("new_table.pcl", metadatadf, featuredf)
Can I get some guidance on whether this is possible / makes sense to use the FileIO framework for this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review FileIO's load/save interface against the proposed two-table PCL examples and the id_row="sampleID" option. Determine whether multi-input/output dispatch and this metadata/feature split fit the framework, then document a concrete API decision and its expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100