Hydrating browser-side Documents with populated SubDocuments using isomorphic schemas

In reaction to #3774, here's what I am trying to do. Currently I'm using a combination of riot.js, feathers.js and Primus to transport data from Node/Mongoose to the browser and back. But the case I am describing is generic. Client-side I am trying to fully instantiate/hydrate documents so I can use validation, embedded document and subdocument modifications, using isomorphic techniques as much as possible.

``` javascript
var categorySchema = new Schema({
name: {type: String, maxlength: 20}
});

var thingTemplateSchema = new Schema({
name: {type: String, maxlength: 50},
unit: {type: String, enum: ['x', 'm2', 'm', 'kg']},
category: {type: ObjectId, ref: 'categorySchema'}
});

var thingEstimatedSchema = new Schema({
thingTemplate: {type: ObjectId, ref: 'thingTemplateSchema'},
quantity: {type: Number, min: 0},
cost: {type: Number, min: 0}
});

var thingCostsSchema = new Schema({
hourlyRate: {type: Number, min: 0},
categories: [{
category: {type: ObjectId, ref: 'categorySchema'},
thingsEstimated: [ thingEstimatedSchema ]
}]
});
```

This is only a part of the data structure and simplified. `Category` and `ThingTemplate` are used in a couple of other places, so I think I need db-refs for these. Please note that I am pretty new to Mongoose & I wonder if I'm sticking enough to the 'No' in NoSQL with my approach. Creation, modification and validation of these isomorphic schemas is performed at both the client-side and the server-side.
Moving on to the server-side query. The actual query is initiated client-side, but results in something like this on the server, the only non-isomorphic part of the code:

``` javascript
var thingCosts = mongoose.model('thingCosts', thingCostsSchema);

thingCosts
.findById(someId)
.populate('categories.category categories.thingsEstimated.thingTemplate')
.lean()
.exec(function(error, data) {
thingService.sendDataToClient('things', data);
});
```

The data transmitted might look something like this:

``` javascript
{
"_id": "56908565acaef75c0f046504",
"hourlyRate": 85,
"categories": [
{
"category": /* populated subdoc */ {
"name": "Category number one",
"_id": "568f15b6c348ed0000e6135f"
},
"_id": "5690a139803b810000c4fc65",
"thingsEstimated": [
{
"thingTemplate": /* populated subdoc */ {
"category": "568f15b6c348ed0000e6135f",
"unit": "x",
"name": "Cat1 Unit1",
"_id": "5690a139803b810000c4fc63"
},
"quantity": 2,
"cost": 234,
"_id": "5690a139803b810000c4fc64"
}
]
},
{
"category": /* populated subdoc */ {
"name": "The second category",
"_id": "568f1590c348ed0000e6135c"
},
"_id": "5690ac5c85b80e0000cd7146",
"thingsEstimated": [
{
"cost": 3,
"quantity": 1,
"thingTemplate": /* populated subdoc */ {
"category": "568f1590c348ed0000e6135c",
"unit": "x",
"name": "Little Unit",
"_id": "5690ac5c85b80e0000cd7144"
},
"_id": "5690ac5c85b80e0000cd7145"
},
{
"cost": 89,
"quantity": 5,
"thingTemplate": /* populated subdoc */ {
"name": "Heavy Unit",
"unit": "kg",
"category": "568f1590c348ed0000e6135c",
"_id": "5698574c371b7100003eefbe"
},
"_id": "5698574c371b7100003eefbf"
}
]
}
]
}
```

On the client-side this is then received, instantiated and used in various ways:

``` javascript
thingService.on('things', function(err, data) {
// make a proper Mongoose Document out of it, with Embedded Documents + SubDocuments
// from populated data
var thingCosts = mongoose.document(data, thingCostsSchema);
// this document would then be used to render the page
irrelevantRenderingEngine('thingCostsView', thingCosts);

// and would want to use things like this directly:
console.log(thingCosts.categories[0].category.name);
thingCosts.categories[3].thingsEstimated[1].quantity = -1;
thingCosts.categories[3].thingsEstimated[1].validate(...); // -1 should not validate ok
thingCosts.categories[3].thingsEstimated[2].thingTemplate.name += ' (old)';
thingCosts.categories[4].remove();

// using a Model-like constructor, a wrapper around mongoose.document(obj, xSchema)
var thingTpl = new ThingTemplate({
name: 'Some Thing',
unit: 'x',
category: currentCategory._id
});
var thingEst = new ThingEstimated({
thingTemplate: thingTpl,
quantity: 42,
cost: 4242
});
thingCosts.categories[3].thingsEstimated.push(thingEst);
});
```

Saving of the modified document is left out of scope for brevity. Brief - a foreign concept, indeed :grinning:
Save to say (pun intended) that the modification semantics are meant to be used for efficient saving at a later time.

For instantiation with subdocuments from populated data I've managed to modify `Document`'s `function init()` to check for `schema.options.ref` and pass in the actual subdoc `schema` to cast to explicitly using `self.set()`.

Then, for later modification with `.push()` or `.set()` I was able to fill `val.constructor.modelName` (from Model-land) by explicitly setting it on `doc.constructor` after construction in a custom wrapper around `mongoose.document(obj, xSchema)` for each schema.

I suspect that @robschley's proposed BrowserModel in #3774 will take care of most of these challenges. Plus the possibility of plugins to elegantly handle the actual queries/saving.

Yes, I am aware of the hackiness, just wanted to see if I could, it is semi-working at the moment. Still seeing a lot of `$__` and `schema` properties in the `toObject()` result, probably due to my coercion of various objects. Is there a better way to do this currently?

Start by reading the proposed BrowserModel discussion in #3774 and the Document init() path described here, then compare it with mongoose.document(obj, xSchema) and the populated nested schemas. Done would be a supported, documented way to hydrate populated subdocuments in the browser while preserving validation and modification behavior without leaking internal properties.

Written by the indexing model from the issue text.