Azure / Azure/AKSFlexNode

install-arc verification fails on azcmagent v1.62+ (gcarcservice renamed to gcad)

Open
#178 2 comments 1 reaction 2 assignees Claimed by @bcho View on GitHub
Dominant language
Go
Stars
14
Forks
28
Avg merge
4d 13h
Merged PRs (30d)
12

Description

## Summary

`install-arc` verification fails on hosts running **Azure Connected Machine Agent (`azcmagent`) v1.62+** because the service `gcarcservice` was renamed to `gcad` upstream, but `pkg/arc/consts.go` still hardcodes the old name. The Arc connection itself succeeds (`azcmagent show` reports `Agent Status: Connected`); only flex-node's verification check fails.

## Environment

- AKSFlexNode: `v0.1.0` (and `main` at time of writing — same code)
- Host: Raspberry Pi 5, Ubuntu 24.04.4 LTS, arm64
- Azure Connected Machine Agent: `1.62.03365.964`
- AKS managed cluster: K8s 1.34.7, AAD + Azure RBAC enabled, westus2

## Reproduction

1. Onboard a fresh Ubuntu 24.04 host with the latest `azcmagent` (`curl -sSL https://aka.ms/azcmagent-linux | bash` pulls v1.62+ today).
2. Pre-grant the resulting Arc identity Reader on subscription + User Access Administrator on subscription (so `assignRBACRoles` can succeed).
3. Run `aks-flex-node start --config /etc/aks-flex-node/config.json`.

## Observed

`install-arc` reports all four RBAC roles assigned, RBAC propagated, then immediately:

```
level=ERROR msg=failed task=install-arc duration=23.4s status=failed
error="arc installation completed but verification failed"
```

Meanwhile on the host:

```
$ azcmagent show | grep "Agent Status"
Agent Status : Connected

$ systemctl is-active himdsd gcad extd
active
active
active

$ systemctl is-active gcarcservice
inactive # service does not exist under this name on azcmagent v1.62+
```

## Root cause

`pkg/arc/consts.go`:

```go
// Arc services that may be present (not all are guaranteed to exist on every installation)
arcServices = []string{"himdsd", "gcarcservice", "extd"}
```

…combined with `pkg/arc/helpers.go::isArcServicesRunning`, which requires **all** services to be active:

```go
for _, service := range arcServices {
if !utilexec.IsServiceActive(ctx, logger, service) {
return false
}
}
```

The comment says *"not all are guaranteed to exist"*, but the implementation is strict-AND, so a single rename upstream breaks the verification gate.

`arcServiceFiles` in the same file (used for uninstall/cleanup) has the same stale path: `/lib/systemd/system/gcarcservice.service`.

## Workaround

```bash
sudo ln -s /lib/systemd/system/gcad.service /etc/systemd/system/gcarcservice.service
sudo systemctl daemon-reload
```

After this symlink, `systemctl is-active gcarcservice` returns `active` and `install-arc` verification passes.

## Suggested fix options

1. **Trust `azcmagent show`.** `isCompleted` already parses `Agent Status: Connected`. Make `isArcServicesRunning` a soft check (log + warn) and let the `azcmagent show` line be authoritative. This is the most robust — service names will keep drifting as the Arc agent evolves.
2. **Accept either name.** Replace `"gcarcservice"` with `[]string{"gcarcservice", "gcad"}` and require at least one to be active.
3. **Match the comment.** Make the loop tolerant: warn on missing services rather than fail. Combined with the `azcmagent show` parse, this is sufficient.

Also update `arcServiceFiles` for uninstall to include `gcad.service`, otherwise cleanup will leak the service file on hosts using the new agent.

## Impact

Anyone onboarding flex-node on a freshly-installed host today will hit this, since `aka.ms/azcmagent-linux` serves v1.62+. The error message *"arc installation completed but verification failed"* doesn't point at the cause and looks like an RBAC/connectivity problem, so it's easy to chase for an hour before locating it in source.

Happy to send a PR if helpful — just want to confirm which of the three approaches you prefer.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.