[BUG] Clearing aggregation cache triggers too much overhead of mysterious CUDA API calls

In groupby aggregation, we use a hash table to store the aggregation results, such that dependent aggregations can look up each other. The hash table is implemented as:
```
std::unordered_map>,
std::pair, std::unique_ptr>,
pair_column_aggregation_hash,
pair_column_aggregation_equal_to>
_cache;
```

At the end of the aggregation process, we extract all the output columns from the cache object (holding this hash table). Since then, this hash table should be empty and deleting it should be very cheap. However, we detected that deleting it actually triggers some mysterious calls to CUDA API, the number of calls proportional to the number of stored columns. With larger number of stored columns, the API calls accumulate into a significant overhead:

Zooming in, these calls are `cudaEventRecord_ptsz`: