ML backend can not process data from s3

If a dataset resides on s3 with custom endpoint - ml backend can not correctly get an image for predict method.
First it requires credentials which is strange cos all that data is setup in ml studio and ml studio can provide presigned url for downloading the data. And if credentials are setup it gets the bucket name but ignores endpoint url.
Detailed trace below:
```
[2022-12-08 15:59:13,619] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:13] "GET /health HTTP/1.1" 200 -
[2022-12-08 15:59:13,623] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670515090 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
[2022-12-08 15:59:13,623] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670513565 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
Load new model from: /home/testuser/share/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[2022-12-08 15:59:13,968] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:13] "POST /setup HTTP/1.1" 200 -
[2022-12-08 15:59:14,320] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 92, in predict
image_path = self.get_local_path(image_url)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 323, in get_local_path
return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
File "/home/testuser/.local/lib/python3.8/site-packages/label_studio_tools/core/utils/io.py", line 92, in get_local_path
r.raise_for_status()
File "/home/testuser/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
**requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bucket-name.s3.amazonaws.com/dataset1065/worker10_000003.jpg?AWSAccessKeyId=user&Signature=ttttttmYJ19ABLxEY%3D&Expires=1670518754**

Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 92, in predict
image_path = self.get_local_path(image_url)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 323, in get_local_path
return get_local_path(url, project_dir=project_dir, hostname=self.hostname, access_token=self.access_token)
File "/home/testuser/.local/lib/python3.8/site-packages/label_studio_tools/core/utils/io.py", line 92, in get_local_path
r.raise_for_status()
File "/home/testuser/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
**requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bucket-name.s3.amazonaws.com/dataset1065/worker10_000003.jpg?AWSAccessKeyId=user&Signature=ttttttmYJ19ABLxEY%3D&Expires=1670518754**

[2022-12-08 15:59:14,322] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:14] "POST /predict HTTP/1.1" 500 -
[2022-12-08 15:59:32,772] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:32] "GET /health HTTP/1.1" 200 -
[2022-12-08 15:59:32,777] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670515090 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
[2022-12-08 15:59:32,777] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670513565 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
Load new model from: /home/testuser/share/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[2022-12-08 15:59:33,124] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:33] "POST /setup HTTP/1.1" 200 -
[2022-12-08 15:59:33,146] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 89, in predict
assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 89, in predict
assert len(tasks) == 1
AssertionError

[2022-12-08 15:59:33,146] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:33] "POST /predict HTTP/1.1" 500 -
[2022-12-08 15:59:33,207] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:33] "GET /health HTTP/1.1" 200 -
[2022-12-08 15:59:33,209] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670515090 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
[2022-12-08 15:59:33,210] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1670513565 job returns exception:
Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError
Load new model from: /home/testuser/share/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /home/testuser/share/mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[2022-12-08 15:59:33,529] [INFO] [werkzeug::_log::225] 192.168.7.25 - - [08/Dec/2022 15:59:33] "POST /setup HTTP/1.1" 200 -
[2022-12-08 15:59:33,618] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 89, in predict
assert len(tasks) == 1
AssertionError

Traceback (most recent call last):
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/exceptions.py", line 39, in exception_f
return f(*args, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/api.py", line 51, in _predict
predictions, model = _manager.predict(tasks, project, label_config, force_reload, try_fetch, **params)
File "/home/testuser/share/label-studio-ml-backend/label_studio_ml/model.py", line 617, in predict
predictions = cls._current_model.model.predict(tasks, **kwargs)
File "/home/testuser/share/label-studio-ml-backend/coco-detector/mmdetection.py", line 89, in predict
assert len(tasks) == 1
AssertionError
```
The very line is :
**requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://bucket-name.s3.amazonaws.com/dataset1065/worker10_000003.jpg?AWSAccessKeyId=user&Signature=ttttttmYJ19ABLxEY%3D&Expires=1670518754**