ModuleNotFoundError: No module named 'sam_vit_h_encode'

livello@archlinux ~/PROG/segment-anything-services (git)-[main] % sudo bash start_serve_encode_gpu.sh :(
Warning: TorchServe is using non-default JVM parameters: -Xmx25g
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2025-05-08T08:51:50,952 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties
2025-05-08T08:51:50,953 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2025-05-08T08:51:51,002 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml
2025-05-08T08:51:51,100 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.8.2
TS Home: /home/venv/lib/python3.9/site-packages
Current directory: /home/model-server
Temp directory: /home/model-server/volume/tmp
Metrics config path: /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml
Number of GPUs: 1
Number of CPUs: 12
Max heap size: 25600 M
Python executable: /home/venv/bin/python
Config file: ./config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://127.0.0.1:8082
Model Store: /home/model-server/volume/model-store
Initial Models: all
Log dir: /home/model-server/logs
Metrics dir: /home/model-server/logs
Netty threads: 1
Netty client threads: 0
Default workers per model: 1
Blacklist Regex: N/A
Maximum Response Size: 17797905
Maximum Request Size: 17797905
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: true
Enable metrics API: true
Metrics mode: log
Disable system metrics: false
Workflow Store: /home/model-server/volume/model-store
Model config: N/A
2025-05-08T08:51:51,105 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin...
2025-05-08T08:51:51,123 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: sam_vit_h_encode.mar
2025-05-08T08:51:51,126 [WARN ] main org.pytorch.serve.archive.model.ModelArchive - Model archive version is not defined. Please upgrade to torch-model-archiver 0.2.0 or higher
2025-05-08T08:51:51,127 [WARN ] main org.pytorch.serve.archive.model.ModelArchive - Model archive createdOn is not defined. Please upgrade to torch-model-archiver 0.2.0 or higher
2025-05-08T08:51:51,128 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model sam_vit_h_encode
2025-05-08T08:51:51,128 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model sam_vit_h_encode
2025-05-08T08:51:51,129 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model sam_vit_h_encode loaded.
2025-05-08T08:51:51,129 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: sam_vit_h_encode, count: 1
2025-05-08T08:51:51,138 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/venv/bin/python, /home/venv/lib/python3.9/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /home/model-server/volume/tmp/.ts.sock.9000, --metrics-config, /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml]
2025-05-08T08:51:51,139 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2025-05-08T08:51:51,177 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2025-05-08T08:51:51,178 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2025-05-08T08:51:51,178 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2025-05-08T08:51:51,179 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2025-05-08T08:51:51,179 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2025-05-08T08:51:51,359 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2025-05-08T08:51:51,923 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:33.3|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,923 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:587.4694633483887|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,924 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:365.1214942932129|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,924 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:38.3|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,924 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUtilization.Percent:3.6702473958333335|#Level:Host,DeviceId:0|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,924 [INFO ] pool-3-thread-1 TS_METRICS - GPUMemoryUsed.Megabytes:902.0|#Level:Host,DeviceId:0|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,925 [INFO ] pool-3-thread-1 TS_METRICS - GPUUtilization.Percent:0.0|#Level:Host,DeviceId:0|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,925 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:26514.0859375|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,925 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:4506.04296875|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:51,925 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:17.3|#Level:Host|#hostname:ba230fe86caa,timestamp:1746694311
2025-05-08T08:51:52,613 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - s_name_part0=/home/model-server/volume/tmp/.ts.sock, s_name_part1=9000, pid=44
2025-05-08T08:51:52,614 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:52,620 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Successfully loaded /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml.
2025-05-08T08:51:52,620 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - [PID]44
2025-05-08T08:51:52,620 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Torch worker started.
2025-05-08T08:51:52,620 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Python runtime: 3.9.18
2025-05-08T08:51:52,621 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change null -> WORKER_STARTED
2025-05-08T08:51:52,625 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:52,629 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/volume/tmp/.ts.sock.9000.
2025-05-08T08:51:52,631 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD to backend at: 1746694312631
2025-05-08T08:51:52,653 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - model_name: sam_vit_h_encode, batchSize: 1
2025-05-08T08:51:52,654 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Backend worker process died.
2025-05-08T08:51:52,654 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2025-05-08T08:51:52,654 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_loader.py", line 100, in load
2025-05-08T08:51:52,655 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler)
2025-05-08T08:51:52,655 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_loader.py", line 145, in _load_handler_file
2025-05-08T08:51:52,655 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name)
2025-05-08T08:51:52,655 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2025-05-08T08:51:52,655 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
2025-05-08T08:51:52,655 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1030, in _gcd_import
2025-05-08T08:51:52,655 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1007, in _find_and_load
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 972, in _find_and_load_unlocked
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 228, in _call_with_frames_removed
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1030, in _gcd_import
2025-05-08T08:51:52,656 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1007, in _find_and_load
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 984, in _find_and_load_unlocked
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'sam_vit_h_encode'
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG -
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - During handling of the above exception, another exception occurred:
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG -
2025-05-08T08:51:52,657 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Traceback (most recent call last):
2025-05-08T08:51:52,658 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_service_worker.py", line 253, in
2025-05-08T08:51:52,658 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - worker.run_server()
2025-05-08T08:51:52,658 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_service_worker.py", line 221, in run_server
2025-05-08T08:51:52,658 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket)
2025-05-08T08:51:52,658 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_service_worker.py", line 184, in handle_connection
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg)
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_service_worker.py", line 131, in load_model
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - service = model_loader.load(
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_loader.py", line 102, in load
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - module = self._load_default_handler(handler)
2025-05-08T08:51:52,659 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/home/venv/lib/python3.9/site-packages/ts/model_loader.py", line 151, in _load_default_handler
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name, "ts.torch_handler")
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level)
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1030, in _gcd_import
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1007, in _find_and_load
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 972, in _find_and_load_unlocked
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 228, in _call_with_frames_removed
2025-05-08T08:51:52,660 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1030, in _gcd_import
2025-05-08T08:51:52,661 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 1007, in _find_and_load
2025-05-08T08:51:52,661 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - File "", line 984, in _find_and_load_unlocked
2025-05-08T08:51:52,661 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - ModuleNotFoundError: No module named 'ts.torch_handler./home/model-server/volume/model-store/sam_vit_h_encode'
2025-05-08T08:51:52,656 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679) ~[?:?]
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435) ~[?:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:213) [model-server.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2025-05-08T08:51:52,666 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: sam_vit_h_encode, error: Worker died.
2025-05-08T08:51:52,666 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2025-05-08T08:51:52,666 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1746694312666
2025-05-08T08:51:52,667 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stderr
2025-05-08T08:51:52,667 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:52,667 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2025-05-08T08:51:52,683 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:52,683 [INFO ] W-9000-sam_vit_h_encode_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stderr
2025-05-08T08:51:53,668 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/venv/bin/python, /home/venv/lib/python3.9/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /home/model-server/volume/tmp/.ts.sock.9000, --metrics-config, /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml]
2025-05-08T08:51:54,567 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - s_name_part0=/home/model-server/volume/tmp/.ts.sock, s_name_part1=9000, pid=85
2025-05-08T08:51:54,568 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:54,574 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Successfully loaded /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml.
2025-05-08T08:51:54,574 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - [PID]85
2025-05-08T08:51:54,574 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2025-05-08T08:51:54,574 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Torch worker started.
2025-05-08T08:51:54,575 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:54,575 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Python runtime: 3.9.18
2025-05-08T08:51:54,576 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2025-05-08T08:51:54,576 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/volume/tmp/.ts.sock.9000.
2025-05-08T08:51:54,576 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2025-05-08T08:51:54,576 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2025-05-08T08:51:54,577 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2025-05-08T08:51:54,577 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2025-05-08T08:51:54,577 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stderr
2025-05-08T08:51:54,578 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:54,578 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.
2025-05-08T08:51:54,594 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:54,594 [INFO ] W-9000-sam_vit_h_encode_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stderr
2025-05-08T08:51:55,578 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/home/venv/bin/python, /home/venv/lib/python3.9/site-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /home/model-server/volume/tmp/.ts.sock.9000, --metrics-config, /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml]
2025-05-08T08:51:56,472 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - s_name_part0=/home/model-server/volume/tmp/.ts.sock, s_name_part1=9000, pid=100
2025-05-08T08:51:56,473 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Listening on port: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:56,479 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Successfully loaded /home/venv/lib/python3.9/site-packages/ts/configs/metrics.yaml.
2025-05-08T08:51:56,479 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - [PID]100
2025-05-08T08:51:56,479 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Torch worker started.
2025-05-08T08:51:56,480 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change WORKER_STOPPED -> WORKER_STARTED
2025-05-08T08:51:56,480 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Python runtime: 3.9.18
2025-05-08T08:51:56,480 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/volume/tmp/.ts.sock.9000
2025-05-08T08:51:56,481 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2025-05-08T08:51:56,481 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2025-05-08T08:51:56,481 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout MODEL_LOG - Connection accepted: /home/model-server/volume/tmp/.ts.sock.9000.
2025-05-08T08:51:56,481 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException: null
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
at org.pytorch.serve.wlm.Model.pollBatch(Model.java:276) ~[model-server.jar:?]
at org.pytorch.serve.wlm.BatchAggregator.getRequest(BatchAggregator.java:34) ~[model-server.jar:?]
at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:186) [model-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
2025-05-08T08:51:56,482 [DEBUG] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-sam_vit_h_encode_1.0 State change WORKER_STARTED -> WORKER_STOPPED
2025-05-08T08:51:56,482 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again
2025-05-08T08:51:56,483 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stderr
2025-05-08T08:51:56,483 [WARN ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:56,483 [INFO ] W-9000-sam_vit_h_encode_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds.
2025-05-08T08:51:56,499 [INFO ] W-9000-sam_vit_h_encode_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stdout
2025-05-08T08:51:56,499 [INFO ] W-9000-sam_vit_h_encode_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-sam_vit_h_encode_1.0-stderr
^CTraceback (most recent call last):
File "/home/venv/bin/torchserve", line 8, in
sys.exit(start())
File "/home/venv/lib/python3.9/site-packages/ts/model_server.py", line 203, in start
process.wait()
File "/usr/lib/python3.9/subprocess.py", line 1189, in wait
return self._wait(timeout=timeout)
File "/usr/lib/python3.9/subprocess.py", line 1933, in _wait
(pid, sts) = self._try_wait(0)
File "/usr/lib/python3.9/subprocess.py", line 1891, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt