Ask-and-tell interface outputting repeated points

I was simulating a optimization which has four workers (GPUs), which runs a function (`fake_training`) asynchronously over 20 days. Each new day, the four workers would get four new points to study. However, after a few days, the optimizer starts outputting the same point for two workers.

There is no noise in the target-function. I am using nevergrad version 0.4.2_post5.

## Steps to reproduce

1. Run the code below
2. Check 1st and 3rd lines of day 9

## Observed Results

On day 9, the results for worker 0 and 2 are the same, as seen below (full log attached)

```
Day: 9
Point for worker 0: {'learning_rate': 0.05191107915156722, 'batch_size': 2, 'learning_decay': 0.8634180716999259, 'learning_decay_steps': 4500} . Simulated loss: 0.2328946307866975
Point for worker 1: {'learning_rate': 0.05191107915156722, 'batch_size': 2, 'learning_decay': 0.8634180716999259, 'learning_decay_steps': 4242} . Simulated loss: 66564.2328946308
Point for worker 2: {'learning_rate': 0.05191107915156722, 'batch_size': 2, 'learning_decay': 0.8634180716999259, 'learning_decay_steps': 4500} . Simulated loss: 0.2328946307866975
Point for worker 3: {'learning_rate': 0.05191107915156722, 'batch_size': 1, 'learning_decay': 0.8634180716999259, 'learning_decay_steps': 4500} . Simulated loss: 1.2328946307866975
```

*Full log:* [log.txt](https://github.com/facebookresearch/nevergrad/files/5733843/log.txt)

## Expected Results

I would have expected that each worker receives a completely independent phase-space point.

## Relevant Code

```python
#! /usr/bin/env python

import nevergrad as ng
import numpy as np

def fake_training(learning_rate: float, batch_size: int, learning_decay: float, learning_decay_steps: int) -> float:
return 10*(learning_rate - 0.2)**2 + (batch_size - 2)**2 + (learning_decay_steps - 4500)**2 + (learning_decay-0.98)**2

np.random.seed(320)
parametrization = ng.p.Instrumentation(
learning_rate=ng.p.Log(lower=0.0001, upper=1.0),
batch_size=ng.p.Scalar(lower=1, upper=2).set_integer_casting(),
learning_decay = ng.p.Scalar(lower=0.5, upper=1),
learning_decay_steps = ng.p.Scalar(lower=4000, upper=5000).set_integer_casting()
)

gpus = 4
budget_days = 20
bug_days = 9
budget = gpus*budget_days
optim = ng.optimizers.NGOpt(parametrization=parametrization, budget=budget)

for day in range(bug_days):
# Get new points
inputs = []
losses = []
print(f"Day: {day+1}")
for g in range(gpus):
x = optim.ask()
y = fake_training(**x.kwargs)
inputs.append(x)
losses.append(y)

print(f"Point for worker {g}:", x.kwargs, ". Simulated loss:", y)

for x, y in zip(inputs, losses):
optim.tell(x,y)
```

Start by running the supplied Python reproduction with four workers through optim.ask() and optim.tell(), focusing on day 9 and the repeated points. Trace the NGOpt ask-and-tell behavior to determine why distinct workers receive the same point; done means the reproduction no longer produces duplicate worker points under the stated setup.

Written by the indexing model from the issue text.