Extracting results

Hello

I'm trying to grasp how the timely dataflow is meant to be used kind of end to end. The documentation concentrates on how to do and structure the computation and its good at explaining that. But I kind of still don't get how to connect it to the rest of the world.

To illustrate what I mean, let's say I have a huge source of data ‒ let's say live stream of log messages or metrics or something like that. So, the whole thing will do something like this:

• I have bunch of worker processes across machines.
• They either read a partition of the stream each, or one „master“ reads them and inserts them into the system and it'll distribute through the workers.
• It performs the computation, producing let's say events of style „this service entered alert state“ and „this service left alert state“.

However, let's say I want a static web page containing all the services in alert state at the currently newest timestamp. This sounds challenging, because:

• The information about the state of different machines is scattered through the cluster, not at one place.
• They may be coming out of order (OK, that probably can be somehow worked around by buffering in a custom operator).

I guess I could somehow force my way through it, but all the ideas I have in mind look like huge hacks. Is there a correct way to do such things? Should it be part of the book? Because all the „exporting“ of the results is done by println ‒ that'll end up in a random worker, right?

No file or test is named; start by reading the project's documentation or book and the existing discussion about exporting results from a distributed computation. Done would require a clear, maintainable explanation of how to extract and aggregate results for an external consumer, including how worker output should be handled.