ddb expression lexer: base64 literals silently truncated, escaped backslashes not unescaped
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
The `aws ddb` expression lexer (`awscli/customizations/dynamodb/lexer.py`) mishandles quoted strings in three ways. The first one silently corrupts data.
**1. Base64 literals are only validated at their start, so invalid input is silently truncated**
`_consume_base64_string` validates with `VALID_BASE64.match(...)`. `re.match` anchors only at the beginning of the string, so any invalid characters after a valid base64 prefix pass validation. `base64.b64decode` then discards characters outside the alphabet by default (`validate=False`), so the value is silently decoded to a truncated result and written to DynamoDB with no error.
The comment directly above that check states this is exactly what the manual validation is meant to prevent:
```python
# Python will simply ignore invalid characters, so we have to
# validate manually.
if raw_string['value'] and not VALID_BASE64.match(raw_string['value']):
```
Note the check *does* work when the invalid characters come first (`b"!!!!AAAA"` is correctly rejected), which is likely why this went unnoticed.
**2. An escaped backslash inside a quoted string or identifier is not unescaped**
`_consume_until` preserves `\X` verbatim, and the callers only strip `\"` and `\'`. An escaped backslash therefore stays as two characters, so `"C:\\path"` produces `C:\\path` instead of `C:\path`. There is no way to express a single literal backslash in a `ddb` expression string.
**3. The `end` offset recorded for quoted tokens is a length, not an offset**
`_consume_quoted_identifier` and `_consume_string_literal` set `'end': token_len`. Everywhere else `end` is an offset, and `awscli/customizations/dynamodb/exceptions.py` computes `token_length = token['end'] - token['start']` to size the `^^^` underline in error messages. For any quoted token that does not start at offset 0 this goes negative and the underline collapses to a single `^` of the wrong width.
### Expected Behavior
1. `data = b"AAAA!!!!"` is rejected as an invalid base64 literal.
2. `path = "C:\\Users"` produces the string `C:\Users`.
3. `expression[token['start']:token['end']]` yields the token's source text.
### Current Behavior
1. `data = b"AAAA!!!!"` is accepted and decodes to `b'\x00\x00\x00'` — the `!!!!` is silently discarded and a truncated binary value is sent to DynamoDB.
2. `path = "C:\\Users"` produces `C:\\Users` (two backslashes).
3. `end` is a length, so the error-message underline is mis-sized.
### Reproduction Steps
Against `aws-cli/2.36.20`, this shows the values that get substituted into the request actually sent to DynamoDB:
```python
from awscli.customizations.dynamodb.extractor import AttributeExtractor
e = AttributeExtractor()
print(e.extract('data = b"AAAA!!!!"')['values'])
print(e.extract('path = "C:\\\\Users"')['values'])
```
Output on 2.36.20:
```
{':n1': Binary(b'\x00\x00\x00')} # expected: an "Invalid base64 string" error
{':n1': 'C:\\\\Users'} # expected: 'C:\\Users', i.e. C:\Users
```
These reach DynamoDB through any expression argument that accepts a literal, e.g.:
```
aws ddb select my-table --filter 'data = b"AAAA!!!!"'
aws ddb put my-table '{...}' --condition 'path = "C:\\Users"'
```
For the third issue:
```python
from awscli.customizations.dynamodb.lexer import Lexer
tokens = list(Lexer().tokenize('bar = "foo"'))
tok = [t for t in tokens if t['type'] == 'literal'][0]
print(tok['start'], tok['end']) # 6 4 -- end < start
```
### Possible Solution
1. Validate the entire literal with `fullmatch` instead of `match`.
2. Resolve escape sequences while consuming the string, so that `\\` collapses to a single backslash and `\"`/`\'` continue to yield the delimiter. A backslash that escapes neither the delimiter nor another backslash should stay verbatim, to preserve current behaviour for sequences like `\n`.
3. Record `end` as an offset.
I have a branch with these three fixes plus regression tests, and will open a PR referencing this issue.
### Additional Information/Context
Found by reading the code rather than by hitting it in production, so I do not have a report of real-world data loss — but (1) fails silently and writes wrong data, which seemed worth reporting regardless.
### CLI version used
aws-cli/2.36.20 Python/3.12.8 Darwin/25.2.0 source/arm64
### Environment details (OS name and version, etc.)
macOS 15 (Darwin 25.2.0), arm64, Python 3.12.8
Contributor guide
Assessment
This issue has not been assessed yet.