Question: behavior for large messages and/or histories

Netherite does not impose any hard limits on the size of messages or histories. But of course, the question remains what happens as messages or histories get very large, i.e. what breaks first. I created this issue to track discussion, testing, and documentation around this question.

Some thoughts on this:

• Everything in the system (i.e. not only the specific orchestrations which contain large messages or histories) is likely to slow down substantially when storage bandwidth, or inter-partition bandwidth (e.g. event hub throughput) becomes a bottleneck. The system should keep working under such circumstances but may be too slow to meet its intended purpose.

• All in-flight messages are kept in memory (in the outboxes and the session buffers on the source and destination partitions respectively), so we may hit OOM when using large messages and not processing them quickly.

• Netherite keeps the in-memory instance caching in line with the specified cache limits. If workers need to process histories or messages that exceed the memory limits of the cache, the result is thrashing which makes a system perform horribly. It is therefore important to increase the cache size when trying to handle such situations.

• Page blobs have a maximum size of 1TB, and all of the data in the task hub partition, including all instance states, histories, and in-flight messages, have to fit into that. Also, the FASTER log can be quite a bit larger than the data it stores because it may contain multiple versions of orchestration states. FASTER does run compaction periodically but it remains to be determined what expansion factors would be typical. I would guess something like 3x.

A lot of that is just my guesses, we need experiments to validate these statements.

Start with the issue's hypotheses about in-flight message memory, cache limits, page-blob capacity, and FASTER log expansion. Design experiments to validate what fails or slows first for large messages and histories, then document the observed limits, bottlenecks, and relevant cache guidance.