Keeping file size reasonable and schema independance

Main issue

Currently each version has its own schema.

What would be nice is having a common schema for every version.

Problem is : new things will be added and older versions didn't have as much data.

An other problem : some files, for examples blocks.json contain too many things which is annoying when creating extraction scripts (and possibly when reading it too)

One solution : keep things meaningful, split big files.

For example we might have a blockVariation.json instead of storing all the variations in blocks.json.

That would allow to have one common schema for all the versions. Some version just wouldn't have some of the data files.

Related/ideas

• database schema are usually split when putting it in https://en.wikipedia.org/wiki/Database_normalization#Normal_forms
• knowledge bases avoid that problem by putting data in its atomic form : triples

The knowledge base way is close to what we are doing (for example <1,name,stone>)
and it's even possible to convert all our data to a knowledge base forma (like n triples) by using json ld (http://json-ld.org/)

Also see http://wikiba.se/

Start by comparing the per-version schemas described in the issue and inspect blocks.json, including the proposed blockVariation.json split. Determine which data is meaningful and common across versions before choosing a normalized structure. Done means the version data follows one maintainable schema and oversized files are split, but the issue does not define the target structure or tests.