project-codeflare / project-codeflare/codeflare-sdk

job.status and job.log returns error for certain domain name pattern

Open
#150 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
35
Forks
66
Avg merge
48m
Merged PRs (30d)
7

Description

The job.status() and job.logs() Are failing, they're showing

gaierror                                  Traceback (most recent call last)
File /opt/app-root/lib64/python3.8/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:
....
ConnectionError: Failed to connect to Ray at address: http://ray/.

Cause:
In torchx The app_handle is a string. On my cluster it looks like
ray://torchx/ray-dashboard-mnisttest-default.tedchang-codeflare-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-south.containers.appdomain.cloud-mnisttest-trwmgjb2s041jc

This method uses a regx that not parse the app_id(mnisttest-trwmgjb2s041jc) and ray dashboard(ray-dashboard-mnisttest-default.tedchang-codeflare-test-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-south.containers.appdomain.cloud) property
":\d+-|.com-|.org-|.net-|.gov-|.io-"

/opt/app-root/lib64/python3.8/site-packages/torchx/schedulers/ray_scheduler.py

330         def _parse_app_id(self, app_id: str) -> Tuple[str, str]:
331             # find index of '-' in the first :\d+-
332             m = re.search(r":\d+-|.com-|.org-|.net-|.gov-|.io-" , app_id)
333             if m:
334                 sep = m.span()[1]
335                 addr = app_id[: sep-1]
336                 app_id = app_id[sep:]
337                 return addr, app_id
338 ~~~~~~~~~~~~~~~~
339             addr, _, app_id = app_id.partition("-")
340             return addr, app_id

something like this worked for my cluster but may not be robust
":\d+-|.com-|.org-|.net-|.gov-|.io-|.cloud-"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in torchx/schedulers/ray_scheduler.py at _parse_app_id, then trace how job.status() and job.logs() use the parsed address and application ID. Reproduce the failure with the shown cloud hostname pattern and verify both operations connect successfully without the Ray connection error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.