GoogleCloudPlatform / GoogleCloudPlatform/cloud-spanner-emulator
Change stream ALTER operations incorrectly accumulate internal state causing false limit violations
- Dominant language
- C++
- Stars
- 334
- Forks
- 77
- Avg merge
- 8m
- Merged PRs (30d)
- 2
Description
The Spanner emulator enforces a limit that "no more than 3 change streams can track the same non-key column" (as documented in the [quotas page](https://cloud.google.com/spanner/quotas)). However, when a change stream undergoes multiple `ALTER CHANGE STREAM` operations, the emulator appears to accumulate internal state incorrectly, causing false violations of this limit.
Specifically, after performing multiple ALTER operations on a change stream that tracks a column (e.g., `numericId`) in 3 tables, subsequent attempts to add additional tables to the change stream fail with a "more than 3 Change Streams" error, even when:
1. There is still only ONE change stream in the database
2. The new table being added does NOT contain the restricted column
**Environment:**
- Emulator version: 1.5.28 (tested via `gcr.io/cloud-spanner-emulator/emulator:1.5.28`)
- Testing method: Docker container + gcloud CLI
**Expected Behavior:**
According to the [Spanner quotas documentation](https://cloud.google.com/spanner/quotas), the limit is:
> **Change streams watching any given non-key column: 3**
This means up to **3 separate change streams** can track the same column. A single change stream tracking a column in multiple tables should count as **1 change stream** watching that column, regardless of how many ALTER operations have been performed on it.
**Actual Behavior:**
After performing multiple `ALTER CHANGE STREAM` operations on a change stream that tracks `numericId` in 3 tables, attempting to add a 4th table (even one without `numericId`) fails with:
```
ERROR: (gcloud.spanner.databases.ddl.update) HTTPError 400: {"code":9, "message":"Failed to create or alter Change Stream S : because it is not allowed to have more than 3 Change Streams tracking the same table or non-key column or ALL: numericId."}
```
**Root Cause Analysis:**
The bug appears to be that each `ALTER CHANGE STREAM` operation creates new internal state tracking the column usage, but doesn't properly clean up the old state. After multiple ALTERs, the emulator incorrectly believes the limit has been reached.
**Minimal Reproduction Steps:**
1. Start the Spanner emulator:
```bash
docker run -d --rm --name spanner-emulator -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator:1.5.28
```
2. Configure gcloud to use the emulator:
```bash
gcloud config configurations create emulator --no-activate
gcloud config set --configuration=emulator auth/disable_credentials true
gcloud config set --configuration=emulator project local-project
echo "y" | gcloud config set --configuration=emulator api_endpoint_overrides/spanner http://localhost:9020/
gcloud config configurations activate emulator
```
3. Create instance and database:
```bash
gcloud spanner instances create local-instance --config=emulator-config --description="Local Instance" --nodes=1
gcloud spanner databases create local-database --instance=local-instance
```
4. Create 3 tables with `numericId` column and 1 table without it:
```bash
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="CREATE TABLE T1 (id STRING(36) NOT NULL, numericId INT64 NOT NULL, col1 STRING(MAX), col2 STRING(MAX)) PRIMARY KEY (id)"
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="CREATE TABLE T2 (id STRING(36) NOT NULL, numericId INT64 NOT NULL, col1 STRING(MAX), col2 STRING(MAX)) PRIMARY KEY (id)"
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="CREATE TABLE T3 (id STRING(36) NOT NULL, numericId INT64 NOT NULL, col1 STRING(MAX), col2 STRING(MAX)) PRIMARY KEY (id)"
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="CREATE TABLE T4 (id STRING(36) NOT NULL, status STRING(MAX)) PRIMARY KEY (id)"
```
5. Create a change stream tracking `numericId` in 3 tables:
```bash
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="CREATE CHANGE STREAM S FOR T1(numericId, col1), T2(numericId, col1), T3(numericId, col1) OPTIONS (value_capture_type = 'NEW_ROW')"
```
6. Verify there is only ONE change stream:
```bash
gcloud spanner databases execute-sql local-database --instance=local-instance --sql="SELECT CHANGE_STREAM_NAME FROM INFORMATION_SCHEMA.CHANGE_STREAMS"
```
**Expected output:** Only `S`
7. Perform multiple ALTER operations on the change stream (modifying which columns are tracked):
```bash
# ALTER #1 - add col2 to all tables
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="ALTER CHANGE STREAM S SET FOR T1(numericId, col1, col2), T2(numericId, col1, col2), T3(numericId, col1, col2)"
# ALTER #2 - remove col2 from all tables
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="ALTER CHANGE STREAM S SET FOR T1(numericId, col1), T2(numericId, col1), T3(numericId, col1)"
# ALTER #3 - add col2 back to all tables
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="ALTER CHANGE STREAM S SET FOR T1(numericId, col1, col2), T2(numericId, col1, col2), T3(numericId, col1, col2)"
# ALTER #4 - remove col2 again
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="ALTER CHANGE STREAM S SET FOR T1(numericId, col1), T2(numericId, col1), T3(numericId, col1)"
```
All 4 ALTER operations should succeed.
8. Now attempt to add T4 (which does NOT have `numericId`) to the change stream:
```bash
gcloud spanner databases ddl update local-database --instance=local-instance --ddl="ALTER CHANGE STREAM S SET FOR T1(numericId, col1), T2(numericId, col1), T3(numericId, col1), T4(status)"
```
**Expected:** Should succeed because:
- There is still only 1 change stream in the database
- T4 does not contain the `numericId` column
- We should be able to have up to 3 change streams tracking `numericId`
**Actual:** Fails with:
```
ERROR: (gcloud.spanner.databases.ddl.update) HTTPError 400: {"code":9, "message":"Failed to create or alter Change Stream S : because it is not allowed to have more than 3 Change Streams tracking the same table or non-key column or ALL: numericId."}
```
9. Verify there is still only ONE change stream:
```bash
gcloud spanner databases execute-sql local-database --instance=local-instance --sql="SELECT CHANGE_STREAM_NAME FROM INFORMATION_SCHEMA.CHANGE_STREAMS"
```
**Output:** Still only `S`
**Cleanup:**
```bash
docker rm -f spanner-emulator
gcloud config configurations activate default
```
**Key Observations:**
1. If you skip steps 7 (the multiple ALTERs) and go directly from step 6 to step 8, the operation **succeeds**
2. The bug is specifically triggered by performing multiple ALTER operations before attempting to add a new table
3. The number of ALTER operations needed to trigger the bug appears to be around 4
4. Each ALTER operation seems to create internal state that incorrectly accumulates toward the limit
**Analysis:**
The quota limit states "3 change streams watching any given non-key column". The emulator should be counting:
- ✅ Number of distinct CHANGE STREAM objects tracking `numericId`: 1
But instead appears to be counting something like:
- ❌ Number of ALTER operations performed × number of tables tracking `numericId`: 4 ALTERs × 3 tables = 12 (or similar accumulation)
This suggests that `ALTER CHANGE STREAM` operations are not properly cleaning up or replacing the old internal state, but instead accumulating new state that incorrectly counts toward the limit.
**Impact:**
This bug prevents normal schema evolution patterns for change streams. In production use cases, change streams often need to be modified over time to add/remove columns or tables. After several such modifications, users hit this artificial limit even though they have not violated the actual documented quota.
Contributor guide
Assessment
This issue has not been assessed yet.