NatLabRockies / NatLabRockies/chronify
Decide on supported input formats
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
Database vs User Schemas
We plan to always store data in the database with the same format.
- One or more timestamp columns, usually date time
- One or more columns that designate unique time arrays. This can be an integer ID or set of dimension columns.
- One value column. The concept of a pivoted table is not supported internally, but can be passed as input data.
Database format
| timestamp | id | value |
|---|---|---|
| 2020-01-01 01:00:00 | 1 | 1.0 |
| 2020-01-01 02:00:00 | 1 | 2.0 |
| 2020-01-01 03:00:00 | 1 | 3.0 |
| 2020-01-01 01:00:00 | 2 | 11.0 |
| 2020-01-01 02:00:00 | 2 | 12.0 |
| 2020-01-01 03:00:00 | 2 | 13.0 |
Input Formats
- Create a view from data stored in files matching the format above. No copying required.
- Create a table from data stored in files matching the format above.
- Array of floats, optional timestamps and IDs. Start time and frequency are required without timestamps. Add rows to existing table or create a new one. Auto-generate IDs and timestamps as necessary.
- Create a table from files with data stored with a pivoted dimension. Unpivot the data for final storage. Timestamps are optional. If they exist in Parquet files must have timestamp type. If they existing in CSV files, str-format is required unless it matches the default ISO format.
| timestamp | device1 | device2 |
|---|---|---|
| 2020-01-01 01:00:00 | 1.0 | 11.0 |
| 2020-01-01 02:00:00 | 1.0 | 12.0 |
| 2020-01-01 03:00:00 | 3.0 | 13.0 |
That would be converted into
| timestamp | device_name | value |
|---|---|---|
| 2020-01-01 01:00:00 | device1 | 1.0 |
| 2020-01-01 02:00:00 | device1 | 2.0 |
| 2020-01-01 03:00:00 | device1 | 3.0 |
| 2020-01-01 01:00:00 | device2 | 11.0 |
| 2020-01-01 02:00:00 | device2 | 12.0 |
| 2020-01-01 03:00:00 | device2 | 13.0 |
Required file formats
- Parquet
- Arrow
- CSV
TBD
- Units
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are named. Start with the issue's database schema and four proposed input formats—Parquet, Arrow, CSV, and arrays—and define the supported scope, conversion rules, timestamp and ID requirements, and units; done is a documented decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100