ClusteringStorageAwareResourceLoader might run out of of memory

The ClusteringStorageAwareResourceLoader might (from our experience it will) run out of of memory if the resource set gets to big to be held in memory.

This is most likely to happen if DisabledClusteringPolicy is used, as in org.eclipse.xtext.build.IncrementalBuilder.build(BuildRequest, Function1<? super URI, ? extends IResourceServiceProvider>). Even if org.eclipse.xtext.resource.clustering.DynamicResourceClusteringPolicy is used, out of memory will happen, as it is not possible to know in advance how much memory a resource processing will consume (it can be big if there is a complex derived state or cross-references are resolved). Our experience showed than the memory limit computed DynamicResourceClusteringPolicy.continueProcessing(ResourceSet, URI, int) is also not working correctly, as we show memory consumption exceeding the limit by far.

A better solution would be to have a resource set that uses WeakResourceReference, so that the garbage collector can clean them up when memory is tight, and so avoid out of memory errors. Clustering is also not needed, as the resource set might grow as big as the memory allows it, and it is cleared on demand.

Would you be interested that we contribute such a ResourceSet using WeakResourceReferences? We have been using it extensively and are very happy with it.

Start with ClusteringStorageAwareResourceLoader and the resource-processing paths used by IncrementalBuilder.build(BuildRequest, Function1<? super URI, ? extends IResourceServiceProvider>). Compare DisabledClusteringPolicy and DynamicResourceClusteringPolicy, then review the proposed WeakResourceReference-based ResourceSet; done means resource processing avoids out-of-memory failures and clears resources on demand.