[Feature request] Quantisation of floating point arguments in MaxLIPO+TR optimiser
- Dominant language
- C++
- Stars
- 14.4k
- Forks
- 3.4k
- PR merge metrics
- No merged PRs in 30d
Description
I am writing in interface between XGBoost and the MaxLIPO+TR optimiser in dlib, using the latter to tune the hyperparameters of the former. I have found a situation in which it would be useful to have quantisation of the floating point arguments being optimised, or alternatively be able to switch on the Trust Region (TR) part of the optimiser for integer arguments. I will attempt to explain below.
Some hyperparameters of XGBoost, including `subsample` and `colsample_by*` represent the proportion of the input data to be (in these cases) subsampled. They therefore effectively take discrete quantised values between 0 and 1, quantised to `1/N` where `N` is the extent of the input data in the appropriate dimension.
When attempting to optimise using these parameters, I have two choices with the MaxLIPO+TR optimiser in its current configuration:
1. Treat these arguments as true floating point, or
2. Treat these arguments as integers, and include a divisor to quantise the values actually sent to XGBoost to appropriate values.
Here's what appears to happen in the two cases:
1. TR is turned on for these arguments (✔ VG), but the TR part of the optimiser eventually wastes function calls to XGBoost by making tiny changes to these arguments - smaller than the quantisation size - which result in _exactly_ the same function output due to me specifying a seed value to XGBoost for reproducibility. In the cases I have looked at so far, the correct 'valley' is apparently found relatively quickly, but TR makes tiny changes to no avail in an attempt to refine an unrefinable solution.
2. TR is turned off for these arguments as they are integers. MaxLIPO still does a good job, but for a set number of function calls, the optimisation is not as good as in 1.
Ideally, I would like to be able to try a combination of the two methods, i.e. be able to use quantisation _with_ TR - if the quantisation is sufficiently fine - as my suspicion is that the lack of TR in case 2 means that its performance is not as good as case 1.
My C++ is not up to attempting to implement this, and I have no idea how much work it would be. I would be grateful if you could consider this as an enhancement, and if you would like to see example output to support my case, I will be happy to provide some.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.