Negative timestamps occur by using a Notifier to record CAN messages from multiple channels

### Describe the bug

Negative time occurs when recording many channles CAN messages and channel changes

### To Reproduce

# 创建自定义日志记录器
log_writer = can.Logger('test.asc')

def handle_message(msg):
print(msg)
log_writer.on_message_received(msg)

_can = Can_interface(driver_list=['kvaser', 'vector'])
buses = {
'1': _can.open_driver(3),
'2': _can.open_driver(4),
}
notifiers = list()
for channel, bus in buses.items():
# 注意：先添加过滤器，再添加处理函数
notifier = Notifier(bus, [])
notifier.add_listener(handle_message)
notifiers.append(notifier)

### Expected behavior

I hope to record the CAN messages of different channels in a way that is consistent with the actual received message, and can be kept consistent with 'canoe'

### Additional context

OS and version:win10
Python version:python 3.9.13
python-can version:4.2.0
python-can interface/s (if applicable): vector

Traceback and logs

```python
def func():
return "hello, world!"
```