Schema extraction
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
@janl:
> I have half an (old) patch that extracts top level fields from a document and stores them with a hash in an “attachment” to the database header. So we only end up storing doc values and the schema hash. First of all this trades storage for CPU time (I haven’t measured anything yet), but more interestingly, we could use that schema data to do smart things like auto-generating a validation function / mango expression based on the data that is already in the database. And other fun things like easier schema migration operations that are native in CouchDB and thus a lot faster than external ones. For the curious ones, I’ve got the idea from [V8’s property access optimisation](https://github.com/v8/v8/wiki/Design%20Elements#fast-property-access) strategy.
@kocolosk:
> Cloudant has some work on a metadata system that computes the schemas for various clusters of documents in a database. First use case for us was schlepping the data into a relational data warehouse for analytics. Not sure if we can open source the code but agreeing on a schema format would be good.
Contributor guide
Research direction
The issue does not name files, tests, or an entry point. Start by locating CouchDB's database-header metadata and document storage code, then assess the proposed schema format and storage-versus-CPU tradeoff. Done would require an agreed design for schema extraction, storage, and possible validation or migration use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100