VL53L0X doesn't work under esp-idf framework

### The problem

The Sensor Component VL53L0X Time Of Flight Distance Sensor doesn't work when the esp-idf framework is used, but works under Arduino framework.
There are no errors, it just doesn't give any values. But it is found by the I2C scan.

### Which version of ESPHome has the issue?

2023.12.9

### What type of installation are you using?

Home Assistant Add-on

### Which version of Home Assistant has the issue?

2024.1.5

### What platform are you using?

ESP32

### Board

Wemos S2 mini

### Component causing the issue

_No response_

### Example YAML snippet

```yaml
Doesn't work:

esp32:
board: lolin_s2_mini
framework:
type: esp-idf
version: recommended
# Custom sdkconfig options
sdkconfig_options:
COMPILER_OPTIMIZATION_SIZE: y
# Advanced tweaking options
advanced:
ignore_efuse_mac_crc: false

Works:

esp32:
board: lolin_s2_mini
```

### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_