aws / aws/aws-cli

ddb expression lexer: base64 literals silently truncated, escaped backslashes not unescaped

Open
#10,543 0 comments 0 reactions 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
bug customization dynamodb investigating p2
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

The `aws ddb` expression lexer (`awscli/customizations/dynamodb/lexer.py`) mishandles quoted strings in three ways. The first one silently corrupts data.

**1. Base64 literals are only validated at their start, so invalid input is silently truncated**

`_consume_base64_string` validates with `VALID_BASE64.match(...)`. `re.match` anchors only at the beginning of the string, so any invalid characters after a valid base64 prefix pass validation. `base64.b64decode` then discards characters outside the alphabet by default (`validate=False`), so the value is silently decoded to a truncated result and written to DynamoDB with no error.

The comment directly above that check states this is exactly what the manual validation is meant to prevent:

```python
# Python will simply ignore invalid characters, so we have to
# validate manually.
if raw_string['value'] and not VALID_BASE64.match(raw_string['value']):
```

Note the check *does* work when the invalid characters come first (`b"!!!!AAAA"` is correctly rejected), which is likely why this went unnoticed.

**2. An escaped backslash inside a quoted string or identifier is not unescaped**

`_consume_until` preserves `\X` verbatim, and the callers only strip `\"` and `\'`. An escaped backslash therefore stays as two characters, so `"C:\\path"` produces `C:\\path` instead of `C:\path`. There is no way to express a single literal backslash in a `ddb` expression string.

**3. The `end` offset recorded for quoted tokens is a length, not an offset**

`_consume_quoted_identifier` and `_consume_string_literal` set `'end': token_len`. Everywhere else `end` is an offset, and `awscli/customizations/dynamodb/exceptions.py` computes `token_length = token['end'] - token['start']` to size the `^^^` underline in error messages. For any quoted token that does not start at offset 0 this goes negative and the underline collapses to a single `^` of the wrong width.

### Expected Behavior

1. `data = b"AAAA!!!!"` is rejected as an invalid base64 literal.
2. `path = "C:\\Users"` produces the string `C:\Users`.
3. `expression[token['start']:token['end']]` yields the token's source text.

### Current Behavior

1. `data = b"AAAA!!!!"` is accepted and decodes to `b'\x00\x00\x00'` — the `!!!!` is silently discarded and a truncated binary value is sent to DynamoDB.
2. `path = "C:\\Users"` produces `C:\\Users` (two backslashes).
3. `end` is a length, so the error-message underline is mis-sized.

### Reproduction Steps

Against `aws-cli/2.36.20`, this shows the values that get substituted into the request actually sent to DynamoDB:

```python
from awscli.customizations.dynamodb.extractor import AttributeExtractor

e = AttributeExtractor()
print(e.extract('data = b"AAAA!!!!"')['values'])
print(e.extract('path = "C:\\\\Users"')['values'])
```

Output on 2.36.20:

```
{':n1': Binary(b'\x00\x00\x00')} # expected: an "Invalid base64 string" error
{':n1': 'C:\\\\Users'} # expected: 'C:\\Users', i.e. C:\Users
```

These reach DynamoDB through any expression argument that accepts a literal, e.g.:

```
aws ddb select my-table --filter 'data = b"AAAA!!!!"'
aws ddb put my-table '{...}' --condition 'path = "C:\\Users"'
```

For the third issue:

```python
from awscli.customizations.dynamodb.lexer import Lexer

tokens = list(Lexer().tokenize('bar = "foo"'))
tok = [t for t in tokens if t['type'] == 'literal'][0]
print(tok['start'], tok['end']) # 6 4 -- end < start
```

### Possible Solution

1. Validate the entire literal with `fullmatch` instead of `match`.
2. Resolve escape sequences while consuming the string, so that `\\` collapses to a single backslash and `\"`/`\'` continue to yield the delimiter. A backslash that escapes neither the delimiter nor another backslash should stay verbatim, to preserve current behaviour for sequences like `\n`.
3. Record `end` as an offset.

I have a branch with these three fixes plus regression tests, and will open a PR referencing this issue.

### Additional Information/Context

Found by reading the code rather than by hitting it in production, so I do not have a report of real-world data loss — but (1) fails silently and writes wrong data, which seemed worth reporting regardless.

### CLI version used

aws-cli/2.36.20 Python/3.12.8 Darwin/25.2.0 source/arm64

### Environment details (OS name and version, etc.)

macOS 15 (Darwin 25.2.0), arm64, Python 3.12.8

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.