[Bug]: --listen flag made lora refresh button not work

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

If use --listen flag, lora refresh button will not work.

### Steps to reproduce the problem

1. Setting --listen flag
2. Go to lora tab
3. press "refresh" button

### What should have happened?

Lora list should be refreshed.

### What browsers do you use to access the UI ?

Microsoft Edge

### Sysinfo

[Uploading sysinfo-2024-03-08-22-44.json…]()

### Console logs

```Shell
No error message
```

### Additional information

_No response_

Start by reproducing the failure with the --listen flag, then use the LoRA tab's refresh button as the entry point for tracing the request. Compare its behavior with a local launch and check the browser and console behavior. Done means the LoRA list refreshes successfully when --listen is enabled.

Written by the indexing model from the issue text.