microsoft / microsoft/pai

Volume healthy monitor for Azureblob/NFS and other remote storage

Open
#4,659 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pai-dev
Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

When job using blob storage, if blobfuse crash when job is running, The mount point in container will be inaccessible, and the error is: "Transport endpoint is not connected"

Currently, we don’t have a way to monitor k8s volume healthy. Same issue will happen for NFS/AzureFile and other remote storage. ( when server failure or client failure)

One solution to solve this issue is detecting such error automatically and do some action when this issue occur.
(We can use sidecar to detect volume failure and leverage current errorSpec to define actions for such error. For this case retry the job will solve this issue.)

Refer: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1432-volume-health-monitor/README.md

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Kubernetes volume-health-monitor KEP and trace how this project currently represents volume failures and errorSpec actions. The issue names no files or tests; done would require an agreed design that detects inaccessible remote mounts and defines the resulting action for blob storage, NFS, AzureFile, and similar storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.