Warning about large numbers of databases on Enterprise Clusters

Large numbers of databases on a cluster can indeed cause performance issues. The primary driver of this behavior is total shard count although there are other contributing factors. As a general rule, it is best to stay below 1000 shards per node on a cluster. Let’s take a look at a few possible arrangements to see how database count affects shard count.

The number of databases/(RP) is a direct multiple for the number of shards

```
2 node, 5 databases, RF 2, 52 Week retention, 1 week shard duration = ~520 shards per node
6 node, 5 databases, RF 2, 52 Week retention, 1 week shard duration = ~520 shards per node

2 node, 50 databases, RF 2, 52 Week retention, 1 week shard duration = ~5200 shards per node
6 node, 50 databases, RF 2, 52 Week retention, 1 week shard duration = ~5200 shards per node
```

Longer shard group durations will help but generally not enough. They also bring higher costs to query the data, a larger memory footprint, higher disk IOPS/throughput usage, and more difficult compactions.

```
2 node, 5 databases, RF 2, 52 Week retention, 2 week shard duration = ~260 shards per node
2 node, 5 databases, RF 2, 52 Week retention, 4 week shard duration = ~130 shards per node

2 node, 50 databases, RF 2, 52 Week retention, 2 week shard duration = ~2600 shards per node
2 node, 50 databases, RF 2, 52 Week retention, 4 week shard duration = ~1300 shards per node
```

Greater replication will also hurt as the size of shards on each node will increase. Unbalanced replication (5 node cluster with RF=2) will cause a dramatic increase in shard counts.
Large amounts of shards also cause a significant issue in the performance of TSI. At counts of greater than 1000, we should expect any performance gained by TSI to be outweighed by the overhead.

The effects of a high shard count can be mitigated by adding more resources but this will inevitably hit bounds imposed by either hardware availability, hosting costs, or license limitations.

In short - Lots of shards can increase the resource requirements needed to run the cluster.

one more thing to mention - hot shards are more impactful than cold shards. More databases = more hot shards at any given moment

from InfluxData Support