install-arc verification fails on azcmagent v1.62+ (gcarcservice renamed to gcad)
- Dominant language
- Go
- Stars
- 14
- Forks
- 28
- Avg merge
- 4d 13h
- Merged PRs (30d)
- 12
Description
## Summary
`install-arc` verification fails on hosts running **Azure Connected Machine Agent (`azcmagent`) v1.62+** because the service `gcarcservice` was renamed to `gcad` upstream, but `pkg/arc/consts.go` still hardcodes the old name. The Arc connection itself succeeds (`azcmagent show` reports `Agent Status: Connected`); only flex-node's verification check fails.
## Environment
- AKSFlexNode: `v0.1.0` (and `main` at time of writing — same code)
- Host: Raspberry Pi 5, Ubuntu 24.04.4 LTS, arm64
- Azure Connected Machine Agent: `1.62.03365.964`
- AKS managed cluster: K8s 1.34.7, AAD + Azure RBAC enabled, westus2
## Reproduction
1. Onboard a fresh Ubuntu 24.04 host with the latest `azcmagent` (`curl -sSL https://aka.ms/azcmagent-linux | bash` pulls v1.62+ today).
2. Pre-grant the resulting Arc identity Reader on subscription + User Access Administrator on subscription (so `assignRBACRoles` can succeed).
3. Run `aks-flex-node start --config /etc/aks-flex-node/config.json`.
## Observed
`install-arc` reports all four RBAC roles assigned, RBAC propagated, then immediately:
```
level=ERROR msg=failed task=install-arc duration=23.4s status=failed
error="arc installation completed but verification failed"
```
Meanwhile on the host:
```
$ azcmagent show | grep "Agent Status"
Agent Status : Connected
$ systemctl is-active himdsd gcad extd
active
active
active
$ systemctl is-active gcarcservice
inactive # service does not exist under this name on azcmagent v1.62+
```
## Root cause
`pkg/arc/consts.go`:
```go
// Arc services that may be present (not all are guaranteed to exist on every installation)
arcServices = []string{"himdsd", "gcarcservice", "extd"}
```
…combined with `pkg/arc/helpers.go::isArcServicesRunning`, which requires **all** services to be active:
```go
for _, service := range arcServices {
if !utilexec.IsServiceActive(ctx, logger, service) {
return false
}
}
```
The comment says *"not all are guaranteed to exist"*, but the implementation is strict-AND, so a single rename upstream breaks the verification gate.
`arcServiceFiles` in the same file (used for uninstall/cleanup) has the same stale path: `/lib/systemd/system/gcarcservice.service`.
## Workaround
```bash
sudo ln -s /lib/systemd/system/gcad.service /etc/systemd/system/gcarcservice.service
sudo systemctl daemon-reload
```
After this symlink, `systemctl is-active gcarcservice` returns `active` and `install-arc` verification passes.
## Suggested fix options
1. **Trust `azcmagent show`.** `isCompleted` already parses `Agent Status: Connected`. Make `isArcServicesRunning` a soft check (log + warn) and let the `azcmagent show` line be authoritative. This is the most robust — service names will keep drifting as the Arc agent evolves.
2. **Accept either name.** Replace `"gcarcservice"` with `[]string{"gcarcservice", "gcad"}` and require at least one to be active.
3. **Match the comment.** Make the loop tolerant: warn on missing services rather than fail. Combined with the `azcmagent show` parse, this is sufficient.
Also update `arcServiceFiles` for uninstall to include `gcad.service`, otherwise cleanup will leak the service file on hosts using the new agent.
## Impact
Anyone onboarding flex-node on a freshly-installed host today will hit this, since `aka.ms/azcmagent-linux` serves v1.62+. The error message *"arc installation completed but verification failed"* doesn't point at the cause and looks like an RBAC/connectivity problem, so it's easy to chase for an hour before locating it in source.
Happy to send a PR if helpful — just want to confirm which of the three approaches you prefer.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.