opt: epsilon selectivity is inconsistently applied when estimating row counts

When using a histogram to estimate row counts with a filter that constrains a column to a value(s) outside the max and min bounds of the histogram, a selectivity [`epsilon`](https://github.com/cockroachdb/cockroach/blob/9f61a38b670e57873509568f67bb665e50cd1935/pkg/sql/opt/props/selectivity.go#L17) of `1e-10` is used as a minimum selectivity. This selectivity is applied to the input row count, regardless of whether or not that row count was already reduced by other filters/histograms. The effect is that expressions which should have the same row counts of ~0 actually have row counts that differ by several orders of magnitude. This can lead to poor query plans.

One way to reproduce this issue is with a partial index. Consider the example:

```sql
CREATE TABLE t (
k INT PRIMARY KEY,
a INT,
b INT,
c INT,
INDEX a_idx (a) STORING (b),
INDEX b_idx (b, a) WHERE a IN (0, 1, 2)
);

-- Insert stats for 100,000 rows where k, a, and b are distinct values in the
-- range (0, 100,000].
ALTER TABLE t INJECT STATISTICS '[
{
"columns": ["k"],
"created_at": "2018-01-01 1:00:00.00000+00:00",
"row_count": 100000,
"distinct_count": 100000,
"null_count": 0,
"avg_size": 4,
"histo_col_type": "int",
"histo_buckets": [
{"num_eq": 0, "num_range": 0, "distinct_range": 0, "upper_bound": "0"},
{"num_eq": 1, "num_range": 99999, "distinct_range": 99999, "upper_bound": "100000"}
]
},
{
"columns": ["a"],
"created_at": "2018-01-01 1:00:00.00000+00:00",
"row_count": 100000,
"distinct_count": 100000,
"null_count": 0,
"avg_size": 4,
"histo_col_type": "int",
"histo_buckets": [
{"num_eq": 0, "num_range": 0, "distinct_range": 0, "upper_bound": "0"},
{"num_eq": 1, "num_range": 99999, "distinct_range": 99999, "upper_bound": "100000"}
]
},
{
"columns": ["b"],
"created_at": "2018-01-01 1:00:00.00000+00:00",
"row_count": 100000,
"distinct_count": 100000,
"null_count": 0,
"avg_size": 4,
"histo_col_type": "int",
"histo_buckets": [
{"num_eq": 0, "num_range": 0, "distinct_range": 0, "upper_bound": "0"},
{"num_eq": 1, "num_range": 99999, "distinct_range": 99999, "upper_bound": "100000"}
]
}
]';

EXPLAIN (OPT, VERBOSE)
SELECT k, a, b, c FROM t WHERE a = 0;
-- index-join t
-- ├── columns: k:1 a:2 b:3 c:4
-- ├── stats: [rows=2e-05, distinct(2)=2e-05, null(2)=0]
-- │ histogram(2)= 0 0
-- │ <--- 0
-- ├── cost: 18.0401446
-- ├── key: (1)
-- ├── fd: ()-->(2), (1)-->(3,4)
-- ├── distribution: us-east1
-- ├── prune: (1,3,4)
-- └── scan t@a_idx
-- ├── columns: k:1 a:2 b:3
-- ├── constraint: /2/1: [/0 - /0]
-- ├── stats: [rows=2e-05, distinct(2)=2e-05, null(2)=0]
-- │ histogram(2)= 0 0
-- │ <--- 0
-- ├── cost: 18.0200224
-- ├── key: (1)
-- ├── fd: ()-->(2), (1)-->(3)
-- └── distribution: us-east1

EXPLAIN (OPT, VERBOSE)
SELECT k, a, b, c FROM t@b_idx WHERE a = 0;
-- index-join t
-- ├── columns: k:1 a:2 b:3 c:4
-- ├── stats: [rows=2e-05, distinct(2)=2e-05, null(2)=0]
-- │ histogram(2)= 0 0
-- │ <--- 0
-- ├── cost: 20.3300535
-- ├── key: (1)
-- ├── fd: ()-->(2), (1)-->(3,4)
-- ├── distribution: us-east1
-- ├── prune: (1,3,4)
-- └── select
-- ├── columns: k:1 a:2 b:3
-- ├── stats: [rows=4.00002e-10, distinct(2)=4.00002e-10, null(2)=0]
-- │ histogram(2)=
-- ├── cost: 20.3100113
-- ├── key: (1)
-- ├── fd: ()-->(2), (1)-->(3)
-- ├── distribution: us-east1
-- ├── scan t@b_idx,partial
-- │ ├── columns: k:1 a:2 b:3
-- │ ├── flags: force-index=b_idx
-- │ ├── stats: [rows=2.00001, distinct(1)=2.00001, null(1)=0, distinct(2)=2, null(2)=0]
-- │ │ histogram(1)= 0 0 2 2e-05
-- │ │ <--- 0 --- 100000
-- │ │ histogram(2)= 0 1 0 1
-- │ │ <--- 1 --- 2
-- │ ├── cost: 20.2600112
-- │ ├── key: (1)
-- │ ├── fd: (1)-->(2,3)
-- │ └── distribution: us-east1
-- └── filters
-- └── a:2 = 0 [outer=(2), constraints=(/2: [/0 - /0]; tight), fd=()-->(2)]
```

Notice that the `scan t@a_idx` in the first query and the `select` in the second query are logically equivalent (even though they are not in the same memo group). They should have the same row count estimate of ~0 since `a=0` is outside the max/min of the histogram. But the row counts are wildly different: `2e-05` rows for the former and `4.00002e-10` rows for the latter. In this example, the optimizer manages to pick the better plan, despite the inconsistent row counts. But we have seen real-world examples where this leads to a bad plan.

The different row counts are due to the inconsistent application of the `epsilon` selectivity. In the first query it is applied to the base table's row count of 100,000. In the second query it is applied to the input row count of 2, double counting selectivities, and effectively creating a "zero-er zero". This is like some fundamental property of histogram filtering and row count estimation that we are breaking! Shouldn't the cardinality of `A ∩ ∅` be the same as the cardinality of `A ∩ B ∩ ∅`? And `A ∩ B ∩ C ∩ ∅` and `A ∩ B ∩ C ∩ D ∩ ∅`, and so on? We're effectively creating different levels of ~0 each time we apply `epsilon` to a row count that has been filtered by a different set of filters or histograms.

**NOTE:** This problem doesn't occur when a filter constrains a column to values within the histogram, because the selectivity will be at least `1/input_row_count`—if the value is within the histogram then it must be at least _one_ of the values, no less.

**NOTE:** I assume this can be reproduced without the use of partial indexes, but I haven't yet come up with such a reproduction.

### Possible solutions and workarounds

Setting the `optimizer_min_row_count` setting to something other than 0, like 1, will help avoid this problem. It "normalizes" all the different levels of "zero rows" into the same number. Here's the effect on the query plans above:

```
index-join t
├── columns: k:1 a:2 b:3 c:4 d:5
├── stats: [rows=1, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 25.33
├── key: (1)
├── fd: ()-->(2), (1)-->(3-5)
├── distribution: us-east1
├── prune: (1,3-5)
└── scan t@a_idx
├── columns: k:1 a:2 b:3 d:5
├── constraint: /2/1: [/0 - /0]
├── flags: force-index=a_idx
├── stats: [rows=1, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 19.18
├── key: (1)
├── fd: ()-->(2), (1)-->(3,5)
└── distribution: us-east1

index-join t
├── columns: k:1 a:2 b:3 c:4 d:5
├── stats: [rows=1, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 26.4800113
├── key: (1)
├── fd: ()-->(2), (1)-->(3-5)
├── distribution: us-east1
├── prune: (1,3-5)
└── select
├── columns: k:1 a:2 b:3
├── stats: [rows=1, distinct(2)=1, null(2)=0]
│ histogram(2)=
├── cost: 20.3100113
├── key: (1)
├── fd: ()-->(2), (1)-->(3)
├── distribution: us-east1
├── scan t@b_idx,partial
│ ├── columns: k:1 a:2 b:3
│ ├── flags: force-index=b_idx
│ ├── stats: [rows=2.00001, distinct(1)=2.00001, null(1)=0, distinct(2)=2, null(2)=0]
│ │ histogram(1)= 0 0 2 2e-05
│ │ <--- 0 --- 100000
│ │ histogram(2)= 0 1 0 1
│ │ <--- 1 --- 2
│ ├── cost: 20.2600112
│ ├── key: (1)
│ ├── fd: (1)-->(2,3)
│ └── distribution: us-east1
└── filters
└── a:2 = 0 [outer=(2), constraints=(/2: [/0 - /0]; tight), fd=()-->(2)]
```

Notice that `scan t@a_idx` and `select` now have the same row count: 1.

However, because `epsilon` is `1 / 10 billion`, the row counts will start to diverge again when the table contain 10+ billion rows. Here's the same example above but with 10 billion rows injected into the stats:

```
index-join t
├── columns: k:1 a:2 b:3 c:4 d:5
├── stats: [rows=2, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 32.62
├── key: (1)
├── fd: ()-->(2), (1)-->(3-5)
├── distribution: us-east1
├── prune: (1,3-5)
└── scan t@a_idx
├── columns: k:1 a:2 b:3 d:5
├── constraint: /2/1: [/0 - /0]
├── flags: force-index=a_idx
├── stats: [rows=2, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 20.34
├── key: (1)
├── fd: ()-->(2), (1)-->(3,5)
└── distribution: us-east1

index-join t
├── columns: k:1 a:2 b:3 c:4 d:5
├── stats: [rows=2, distinct(2)=1, null(2)=0]
│ histogram(2)= 0 0
│ <--- 0
├── cost: 29.76
├── key: (1)
├── fd: ()-->(2), (1)-->(3-5)
├── distribution: us-east1
├── prune: (1,3-5)
└── select
├── columns: k:1 a:2 b:3
├── stats: [rows=1, distinct(2)=1, null(2)=0]
│ histogram(2)=
├── cost: 21.44
├── key: (1)
├── fd: ()-->(2), (1)-->(3)
├── distribution: us-east1
├── scan t@b_idx,partial
│ ├── columns: k:1 a:2 b:3
│ ├── flags: force-index=b_idx
│ ├── stats: [rows=3, distinct(1)=3, null(1)=0, distinct(2)=2, null(2)=0]
│ │ histogram(1)= 0 0 3 3e-10
│ │ <--- 0 --- 10000000000
│ │ histogram(2)= 0 1 0 1
│ │ <--- 1 --- 2
│ ├── cost: 21.38
│ ├── key: (1)
│ ├── fd: (1)-->(2,3)
│ └── distribution: us-east1
└── filters
└── a:2 = 0 [outer=(2), constraints=(/2: [/0 - /0]; tight), fd=()-->(2)]
```

2 rows are estimated for the `scan t@a_idx` and 1 row for the `select`. And the respective costs are 32.62 and 29.76. So we've flipped to the worse plan! So with really large tables `optimizer_min_row_count` is not enough.

Jira issue: CRDB-52420