graze / graze/sprout

Support different data file types

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
PHP
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

We should support different data file types: mainly: `csv`, `json`, and `yaml`.

When importing we should support:

- allow different sets of key/values between each 'row'
- group rows based on their common sets of keys, to reduce time when inserting

## Potential Solutions

1. Convert each into `.sql` and import using `mysql` command
1. Read the data into memory and write to the database within code

### CSV

```csv
"id","country_code","country_code_iso2","country_code_iso3","name","decimal_point","thousands_separator","added","updated","deleted"
1,'AF','AF','AFG','Afghanistan','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
2,'AX','AX','ALA','Aland Islands','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
3,'AL','AL','ALB','Albania','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
4,'DZ','DZ','DZA','Algeria','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
5,'AS','AS','ASM','American Samoa','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
```

### Yaml

```yaml
schema: the_schema
table: country

data:
- id: 1
country_code: AF
country_code_iso2: AF
country_code_iso3: AFG
name: Afghanistan
decimal_point: '.'
thousands_separator: ','

- id: 2
country_code: AX
country_code_iso2: AX
country_code_iso3: ALA
name: Aland Island
decimal_point: '.'
thousands_separator: ','

- id: 3
country_code: AL
country_code_iso2: AL
country_code_iso3: ALB
name: Albania
decimal_point: '.'
thousands_separator: ','

- id: 4
country_code: DZ
country_code_iso2: DZ
country_code_iso3: DZA
name: Algeria
decimal_point: '.'
thousands_separator: ','

- id: 5
country_code: AS
country_code_iso2: AS
country_code_iso3: ASM
name: American Samoa
decimal_point: '.'
thousands_separator: ','
```

### Json

```json
{
"schema": "the_schema",
"table": "country",
"data": [
{
"id": 1,
"country_code": "AF",
"country_code_iso2": "AF",
"country_code_iso3": "AFG",
"name": "Afghanistan",
"decimal_point": ".",
"thousands_separator": ","
},
{
"id": 2,
"country_code": "AX",
"country_code_iso2": "AX",
"country_code_iso3": "ALA",
"name": "Aland Island",
"decimal_point": ".",
"thousands_separator": ","
},
{
"id": 3,
"country_code": "AL",
"country_code_iso2": "AL",
"country_code_iso3": "ALB",
"name": "Albania",
"decimal_point": ".",
"thousands_separator": ","
},
{
"id": 4,
"country_code": "DZ",
"country_code_iso2": "DZ",
"country_code_iso3": "DZA",
"name": "Algeria",
"decimal_point": ".",
"thousands_separator": ","
},
{
"id": 5,
"country_code": "AS",
"country_code_iso2": "AS",
"country_code_iso3": "ASM",
"name": "American Samoa",
"decimal_point": ".",
"thousands_separator": ","
}
]
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.