Aggregator misses data due to losing contact with crossbar

At site we are now regularly troubled by instances where the Aggregators lose contact with crossbar. Crossbar reports a ping-pong error, and drops the aggregator client; then the aggregator sees the dropped connection and reconnects a few seconds later. During the outage, aggregator doesn't get data on its subscribed feeds, and data is lost.

This is likely to be related to disk i/o causing something to hang in the aggregator and foiling the ping checks.

If a more direct solution cannot be found, I propose making ocs more robust to such dropouts by adding the following:

• Agents may publish their data multiple times. Each bundle they send will be tagged with some identifier (so duplicates can be removed/ignored by the aggregator).
• The Aggregator will monitor identifers on feeds and (a) warn / alert when a bundle is dropeed (b) quietly accept (and not record) any duplicate packets.

The scheme could be made backwards compatible -- only some agents (such as ACU) will need to enable this safety function.

Start by tracing the Aggregator's connection handling with crossbar and investigating whether disk I/O can block ping checks. Reproduce the dropped-connection scenario and review how subscribed-feed data is received during reconnects. Done means the data-loss cause is identified and a decided fix prevents missing bundles or safely detects and handles them.