firebase / firebase/firebase-tools

[firestore-emulator: datastore-mode] No concurrent transaction manages to write successfully

Open
#8,300 1 comment 3 reactions 0 assignees View on GitHub
emulators: firestore reproducible type: bug
Dominant language
TypeScript
Stars
4.5k
Forks
1.3k
Avg merge
1d 12h
Merged PRs (30d)
84

Description

### [REQUIRED] Environment info

firebase-tools: Using the gcloud cli version 511.0.0, firestore emulator: 1.19.9
Platform: macOS (arm), python client library

### [REQUIRED] Test case

```python
import os
import time
import itertools

from google.cloud import datastore
import concurrent.futures

def get_new_client():
return datastore.Client(
project=os.environ.get("GCLOUDC_PROJECT_ID", "test"),
namespace=None,
_http=None,
)

def sdk_increment_integer_entity(integer_pk):
client = get_new_client()
key = client.key("Integer", integer_pk)
with client.transaction():
entity = client.get(key=key)
entity["value"] += 1
client.put(entity)

def run_test():
PORT = 10901
os.environ["DATASTORE_EMULATOR_HOST"] = "127.0.0.1:%s" % PORT
os.environ["DATASTORE_PROJECT_ID"] = "test"

initial_value = 0
concurrent_writes = 10
futures = []

local_client = get_new_client()

key = local_client.key("Integer", 1)
entity = datastore.Entity(key)
entity["value"] = initial_value
local_client.put(entity)

with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_writes) as executor:
for _ in range(concurrent_writes):
futures.append(executor.submit(sdk_increment_integer_entity, 1))

concurrent.futures.wait(futures)

futures_exceptions = [future.exception() for future in futures]
number_successful_writes = len(list(itertools.filterfalse(None, futures_exceptions)))
if number_successful_writes == 0:
print(futures_exceptions)
else:
print(f"All good, successful writes: {number_successful_writes}")

from_db = local_client.get(key)

assert number_successful_writes != 0
assert from_db["value"] == number_successful_writes

def main():
for _ in range(1, 50):
run_test()
time.sleep(2)

if __name__ == "__main__":
main()

```

### [REQUIRED] Steps to reproduce

1. Start the emulator with gcloud emulators firestore start --host-port=127.0.0.1:10901 --database-mode=datastore-mode
2. Install the python deps: pip install google-cloud-datastore==2.19.0
3. Run the above python script

### [REQUIRED] Expected behavior

Runs successfully, without errors, every time

### [REQUIRED] Actual behavior

Command fails with

```
[Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.'), Aborted('Transaction lock timeout.')]
Traceback (most recent call last):
File "/Users/artoale/dev/gcloudc/./example.py", line 69, in
main()
File "/Users/artoale/dev/gcloudc/./example.py", line 64, in main
run_test()
File "/Users/artoale/dev/gcloudc/./example.py", line 58, in run_test
assert number_successful_writes != 0
```

I've tested the same script against two real firestore in datastore mode instances (one configured with pessimistic concurrency, one with optimistic) - running the script multiple time.

Transaction failing because of lock timeout make complete sense, but I would expect at least one transaction (typically the first to acquire the lock with a pessimistic model) to pass - but the test shows that they do not

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.