Azure / Azure/azure-cli

`-o tsv` emits CRLF on Windows, breaking `for x in $(az ... -o tsv)` shell loops with misleading 400 errors

Open
#33,385 5 comments 1 reaction 2 assignees Assigned to @a0x1ab View on GitHub
act-observability-squad Auto-Assign Auto-Resolve bug customer-reported Output Possible-Solution Service Attention Service Bus Similar-Issue
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

On Windows, `az ... --output tsv` writes each row terminated by **CRLF** (`\r\n`) instead of LF (`\n`). When that output is consumed by a POSIX shell (Git Bash / MSYS2 / WSL-on-Windows-files / bash-from-PowerShell) via `for x in $(...)`, every token except the last carries a stray `\r`. That `\r` gets interpolated into the URL path of the next `az` call, and ARM responds with `HTTP 400 Bad Request`. The CLI's error message — `Operation returned an invalid status 'Bad Request'` — does not reveal the underlying cause.

Net effect: a common shell pattern shown in many Microsoft Learn tutorials silently fails on Windows for every iteration **except the last one** in the loop.

### To Reproduce

Any namespace/topic combination works. Example using Service Bus:

```bash
RG=
NS=

for topic in $(az servicebus topic list --resource-group "$RG" --namespace-name "$NS" --query "[].name" -o tsv); do
count=$(az servicebus topic subscription list --resource-group "$RG" --namespace-name "$NS" --topic-name "$topic" --query "length(@)" -o tsv 2>&1)
printf "%-40s %s\n" "$topic" "$count"
done
```

Actual output (14 topics, only the alphabetically last works):

```
topic1 ERROR: Operation returned an invalid status 'Bad Request'
topic2 ERROR: Operation returned an invalid status 'Bad Request'
```

### Environment summary

```
azure-cli 2.80.0

core 2.80.0
telemetry 1.1.0

Extensions:
account 0.2.5
authV2 1.0.1
azure-devops 1.0.2
redisenterprise 1.4.0
subscription 0.1.3

Dependencies:
msal 1.34.0b1
azure-mgmt-resource 23.3.0

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'

OS: Windows 11 Pro 10.0.26200
Shell: Git Bash (MSYS2 / mintty); also reproducible from PowerShell-invoked bash subshells
```

### Proof of root cause

`az ... -o tsv` writes CRLF row terminators on Windows:

```bash

The bash `$(...)` substitution then carries `\r` on every token except the last (bash strips a trailing `\n` and the `\r` just before it in that single edge case — the other tokens keep their `\r` because the `\r` precedes a non-terminal `\n`).

When the next `az` call receives `--topic-name "$topic"`, the inner CLI inserts the `\r` into the ARM REST URL path. ARM responds 400. The CLI surfaces only `Operation returned an invalid status 'Bad Request'` and does not echo the ARM error body, so the cause is invisible to the user.

### Workaround

Strip `\r` from the captured output:

```bash
for topic in $(az servicebus topic list --resource-group "$RG" --namespace-name "$NS" --query "[].name" -o tsv | tr -d '\r'); do
count=$(az servicebus topic subscription list --resource-group "$RG" --namespace-name "$NS" --topic-name "$topic" --query "length(@)" -o tsv)
printf "%-40s %s\n" "$topic" "$count"
done
```

### Proof of root cause

`az ... -o tsv` writes CRLF row terminators on Windows:

```bash
$ az servicebus topic list --resource-group "$RG" --namespace-name "$NS" --query "[].name" -o tsv | head -3 | od -c
0000000 t o p i c 1 \r \n t o p i c 2 \r \n
0000020 t o p i c 3 \r \n
0000030
```
After applying `| tr -d '\r'` the loop returns the expected counts for all 14 topics.

### Comparison of output formats on Windows

| Output mode | Line ending | Safe in shell `for` loops? |
|---|---|---|
| `-o tsv` | **CRLF** | **No — this bug** |
| `-o json` | LF | n/a (not row-delimited) |
| `-o jsonc` | LF | n/a |
| `-o yaml` | LF | n/a |
| `-o table` | CRLF (decorative) | n/a |

Only the **machine-parseable, row-delimited** format `tsv` is affected. `csv` likely has the same issue but I have not retested it for this report.

### Suggested fix

The CLI's `tsv` formatter inherits Python's default text-mode line translation on Windows. Two options:

1. **Reconfigure stdout for tsv/csv output** — at format-dispatch time, call `sys.stdout.reconfigure(newline='\n')` when the user selected `tsv` or `csv`. Smallest change, easiest to review.
2. **Write tsv as bytes** — `sys.stdout.buffer.write(line.encode() + b'\n')`. More invasive but eliminates any platform-specific newline handling.

Option 1 should be a one-line change in the format dispatch (`azure-cli-core/azure/cli/core/commands/__init__.py` or wherever the formatter lookup happens).

### Bonus suggestion (separate issue territory)

When the underlying ARM call returns 4xx, the CLI should print the ARM error body, not just `Operation returned an invalid status 'Bad Request'`. ARM almost always includes a `code` and `message` field that would have pointed at the bad URL character immediately. The current behavior of swallowing those fields turns a 30-second diagnosis into a 30-minute one.

### Why this matters

The pattern `for x in $(az ... -o tsv); do ...; done` appears verbatim in Microsoft Learn tutorials, Azure samples, and many blog posts. Every Windows user copying those snippets hits this bug and assumes their script, their credentials, their permissions, or their resource state is wrong. Few think "the CLI's tsv output has the wrong line endings." A one-line formatter fix removes a recurring scripting trap.

### Anything else

Confirmed not present in PowerShell — PowerShell splits `tsv` output on `[\r\n]+`, masking the bug. The issue is bash-on-Windows-specific in observed effect, but the **root cause** (CRLF emission by the CLI on Windows) is the CLI's responsibility, not bash's.
```

### Related command

```bash
RG=
NS=

for topic in $(az servicebus topic list --resource-group "$RG" --namespace-name "$NS" --query "[].name" -o tsv); do
count=$(az servicebus topic subscription list --resource-group "$RG" --namespace-name "$NS" --topic-name "$topic" --query "length(@)" -o tsv 2>&1)
printf "%-40s %s\n" "$topic" "$count"
done

### Errors

Actual output (14 topics, only the alphabetically last works):

```
topic1 ERROR: Operation returned an invalid status 'Bad Request'
topic2 ERROR: Operation returned an invalid status 'Bad Request'

### Issue script & Debug output

Actual output (14 topics, only the alphabetically last works):

```
topic1 ERROR: Operation returned an invalid status 'Bad Request'
topic2 ERROR: Operation returned an invalid status 'Bad Request'

### Expected behavior

### Comparison of output formats on Windows

| Output mode | Line ending | Safe in shell `for` loops? |
|---|---|---|
| `-o tsv` | **CRLF** | **No — this bug** |
| `-o json` | LF | n/a (not row-delimited) |
| `-o jsonc` | LF | n/a |
| `-o yaml` | LF | n/a |
| `-o table` | CRLF (decorative) | n/a |

Only the **machine-parseable, row-delimited** format `tsv` is affected. `csv` likely has the same issue but I have not retested it for this report.

### Environment Summary

```
azure-cli 2.80.0

core 2.80.0
telemetry 1.1.0

Extensions:
account 0.2.5
authV2 1.0.1
azure-devops 1.0.2
redisenterprise 1.4.0
subscription 0.1.3

Dependencies:
msal 1.34.0b1
azure-mgmt-resource 23.3.0

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'

OS: Windows 11 Pro 10.0.26200
Shell: Git Bash (MSYS2 / mintty); also reproducible from PowerShell-invoked bash subshells
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.