Major Directions

Babylon Release

Given the Hackathon submission experience of using Hydra with Kusama, we are now laser-focused on solving some salient last-mile obstacles to deploying the first version solely for Joystream purposes.

Here is the issue currently used to summarise Hydra team meeting summaries

https://github.com/Joystream/hydra/issues/10

✅ Faster Blockchain Synchronization

Due to a number of factors, it took some 10-14 days to synch up to 3M blocks on Kusama. This is a very long time when combined with the fact that changing your schema and/or mappings in any way currently would require resynching from scratch. There are many obvious ways of radically reducing this time.

✅ One-Time Blockchain Synchronization

As mentioned in the prior issue, when doing development, you are likely to change your schema and/or mappings. If you did not have to resynch the chain, but instead build a local database of all relevant events and transactions, then reprocessing them when you make changes would be much faster, and the development experience would be radically improved. It would also significantly help with reproducing failures during processing due to bugs.

✅ Separate Databases for Blockchain and API Indexes

Currently, the local database mentioned in the prior point is the same database that holds the search index. This prevents a single such database from servicing distinct APIs, or even just API nodes with the same API, from the same database. This means every developer working against the same chain has to rebuild this database locally, or in the cloud when deploying, from scratch every time.

Post Babylon Release

Statically Type-Safe Mappings

Input to mappings, which include the event parameter values and possible originating extrinsic, is dynamically typed. This means that the developer has to do lots of manual type conversions and checks, any one of which could easily have a mistake. By moving to statically typed mapping signatures, it will be easier for developers to write, maintain, test and document mappings.

Mapping Manifest

Currently, we are just using a naming convention to capture what mappings should run for what event/transaction. This is very brittle, a standalone manifest makes more sense.

Transaction Processing

Only event processing is possible at the moment. This is often very inconvenient, because of two reasons.

1. Unless the module developer has followed the convention of replicating all extrinsic parameters in the corresponding event, which is rarely the case, then just about all mappings require that one recovers the originating extrinsic associated with an event. A single event could in principle originate from more than one extrinsic, or even from a "dynamic" extrinsic like that found in SUDO or Utility pallets. Being forced to write mappings that deal with this heterogeneity is difficult, and the Hydra node deployed for Kusama failed a number of times because of this. Tracking down the exact details of each oversight is also time-intensive.

2. This heterogeneity makes it next to impossible to autogenerate the types required for statically type-safe mappings (see above) because there is no way to automatically determine what extrinsics could generate each event. A better compromise would be to have type-safe mappings, where event mappings only depend on event parameters, and extrinsic mappings only depend on input parameters. In both cases, all required static types can be easily autogenerated from runtime metadata.

Improved Hosted Deployment

1. A starter pack to quickly deploy a Hydra stack to Heroku
2. Sample CircleCI scripts

Long-Term

There are a few major directions we are evaluating after this short term faze.

FRAME Paletts Support

Joystream uses some important palletts that are part of FRAME, such as balances, staking, etc. These are likely also of great interest to other runtime developers. Beyond this, it could also make sense to implement support for many of the other prominent FRAME palletts in order to encourage the adoption of Hydra.

Reusable & Stitchable Schemas and Mappings

Schemas and mappings written for one module or runtime should ideally be conveniently reusable in another. For example, if a module developer writes a new module, it would be nice if a corresponding set of schemas and mappings for Hydra could be shipped and reused as easily as the module itself.

Graceful Handling of Runtime Upgrades

Currently, we have no good ideas for how the query infrastructure could gracefully keep working across a runtime upgrade. This could involve a range of changes, including on-chain storage migrations, changes to types repertoire, etc. Some best practices, and possibly explicit tools and functionality, are required to handle this in Hydra.

Attributable Malicious Query Results

A Hydra node serving a DApp instance can compromise user funds if replying with malicious results. For example, the user may be provided an incorrect account to send their funds to for some purpose.

One way to attempt to deal with this is to write the Dapp in a way where it attempts to explicitly double-check certain key state variables by talking to a full node during critical steps with a light client. However, this can get quite complex for the DApp developer, and may not always be feasible.

Another model, which appears to be the most popular one, is effectively to nearly fully trust the DApp developer that owns and maintains the app or website that serves the client-side application the user runs to interact with the chain. Under this model, it's not a lot of extra risk to also trust the developer to run an honest Hydra node.

A third model is to require all query responses to include a signature from the originating Hydra node, and also to have a commitment in the result to a specific block and the initial query. This signature would serve as proof that a given node operator returned a given result, at a given block height, to a given query. If the node is staked, then such a signature of a malicious query could be submitted a DAO, or some incentive aligned actor set, which can then individually confirm the validity of the result, and vote to slash or not slash the node operator. This model is closest to what we would like to have in Joystream.

Improved Integration Tests

There is currently very little integration testing of Hydra.

Improved Documentation and Tutorials

We have something now, but having more non-Hydra developers start to use it internally is likely to reveal much room for improvement.

No files, tests, or entry points are named. Start by reading the linked Hydra team meeting summary in issue 10 and choose one direction to scope; the issue does not define a specific implementation target or completion criteria.