microsoft / microsoft/pai

sometimes submit job failed.

Open
#5,720 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

Organization Name:HIT

Short summary about the issue/question:
sometimes submit job failed.
Brief what process you are following:
when I submit ad job, it occurs error as follow:

[Exit Trigger Info]

ExitTriggerMessage: FailedTaskCount 1 has reached MinFailedTaskCount 1 in the TaskRole
ExitTriggerTaskRole: taskrole
ExitTriggerTaskIndex: 0

--------------------------------------------------------------------------------

[Exit Spec]

code: 1
phrase: PAIRuntimeExitAbnormally
issuer: PAI_RUNTIME
causer: PAI_RUNTIME
type: PLATFORM_FAILURE
stage: UNKNOWN
behavior: UNKNOWN
reaction: RETRY_TO_MAX
reason: 'PAI Runtime exit abnormally with undefined exitcode, it may have bugs'
repro:
  - PAI Runtime exits with exitcode 1
solution:
  - Contact PAI Dev to fix PAI Runtime bugs


--------------------------------------------------------------------------------

[Exit Diagnostics]

Pod failed: PodPattern unmatched:
containers:
  - name: init
    reason: Completed
    code: 0
  - name: app
    reason: Error
    message: >
      standard_init_linux.go:228: exec user process caused: no such file or
      directory
    code: 1

what 's the reason about this error? I need help, thanks!

PLATFORM_FAILURE

How to reproduce it:
submit a new job.

OpenPAI Environment:

  • OpenPAI version: v1.8.0

  • Cloud provider or hardware configuration:

  • OS (e.g. from /etc/os-release):

  • Kernel (e.g. uname -a): Linux rsgpuserver154 4.15.0-166-generic 174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  • Hardware (e.g. core number, memory size, storage size, GPU type etc.): A40

  • Others:

Anything else we need to know:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/user/troubleshooting_job.md and the job-debugging section of docs/job_tutorial.md, then reproduce the failure by submitting a new job. Use the reported pod diagnostics, especially the app container's exit message, to identify the cause; done means documenting or fixing the confirmed cause and verifying that a job submits successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.