Distributed streams-based static-site-generator ideas

Hi @maxogden

We've met briefly before at CampJS a few years ago, I'm the author of http://docpad.org among many other things.

I've been brainstorming the last year on several things, one of those things being a distributed streams-based SSG that I'd like to get your feedback on as you are the node database wizard that I am aware of.

DocPad as it is, is incredibly powerful with its in-memory database; essentially it parses all the files on the file system into a in-memory database powered by backbone.js models and collections, it then uses https://github.com/bevry/query-engine to query the data as well as to build fast live collections that always stay up to date.

This is great for most things, but is slow for huge websites, requires a lot of memory, and is difficult to move away from the file system being the only source. Which makes it quite difficult for web dev companies to add a full-featured admin interface on-top of DocPad for their small-business clients.

I've been contemplating building a distributed streams based ssg for a while, something that would work something like this (the code is merely demonstration code, probably wouldn't work that way):

// Setup a mongo database to be the export target of our consolidated data
var db = require('tinkle-mongodb').create('mongodb://db1.example.net:2500')

// Import prismic.io files, wordpress files, website source files, tumblr data, and soundcloud data into our database
require('tinkle-prismic').create('balupton').pipe(db)
require('tinkle-wordpress').create('balupton.wordpress.com').pipe(db)
require('tinkle-fs').create('./my-website/source').pipe(db)
require('tinkle-tumblr').create('balupton.tumblr.com').pipe(db)
require('tinkle-soundcloud').create('balupton').pipe(db)

// Create a collection for our blog
var collections = {
 'blog-listing': db.pipe(require('tinkle-collection').create({directory: 'blog', cacheAttributes:['title']}))
}

// When changes occur, render the changes, and save the rendered results back into the database
// Give it access to the database so for instance eco templates can query the database to display things like blog listings
db.pipe(require('tinkle-renderer').create({db, collections})).pipe(db)

// When changes occur, update our static website file output
db.pipe(require('tinkle-fs').create('./my-website/output'))

To render say posts.html, we would have:

---
render: ['eco', 'markdown', 'html']
usesCollections: 'blog-listing'

---

# Blog Posts

<ol>
<% for post in @getCollection('blog-listing').items: %>
  <li><a href="<%- post.url %>"><%- post.title %></a></li>
<% end %>
</ol>

Which would re-render anytime a change is made to the blog collection, always keeping posts.html up to date with the most recently injected and updated blog content.

From my knowledge, no one is tackling such a consolidation service for distributed data (which by itself is very useful, e.g. homemade zapier and revert.io like services), nor do I know of any like service with a ssg.

It seems that http://dat-data.com could be an option for this, but I can't find that much information on it at all. http://noflojs.org may also be an option.

Keen to pick your brain about this, perhaps over a google hangout on air or whatever, about what some options could be.

This is an exploratory proposal rather than an implementation task, with no repository files, tests, or entry points named. Start by reviewing the proposed streams-based topology and the references to dat-data and noflojs. A contribution would first need an agreed scope, implementation target, and acceptance criteria.