googleapis / googleapis/ruby-spanner-activerecord

Floating point serialization issues in JSON fields

Aperta
#376 0 commenti 1 reazione 1 assegnatario Rivendicata da @olavloite Vedi su GitHub
api: spanner priority: p2 type: bug
Lingua principale
Ruby
Stelle
105
Fork
37
Merge medio
10h 2m
PR unite (30g)
5

Descrizione

We've recently run into some issues serializing specific floating point numbers. We have hit these before, and our understanding is that they stem from Ruby using a different floating point serialization algorithm from Spanner (which spanner runs some API-layer checks to validate).

Prior to this, we were able to work around them by monkeypatching `Float#to_s` to use a matching grisu2 implementation from [2].

Unfortunately after `v2.11.0` of `ruby/json`[1], this workaround no longer works. `ruby/json` no longer delegates the serialization back to `Float#to_s` and instead uses it's own C implementation of `grisu2`. It's worth noting this isn't a new issue, but has become much more challenging to monkeypatch away now that it doesn't call back to the Ruby layer.

Our understanding is Spanner also uses grisu2 internally, however there seems to be some inconsistencies with implementations in [2] and [1]. In an ideal scenario, the SDK with should maintain a consistent serialization interface with the validations performed by the API layer.

#### Steps to reproduce

1. Add the following to `acceptance/cases/type/json_test.rb`:
```
def test_float_serialization
record = TestTypeModel.create! details: { test: 41.021725 }
record.reload

assert_equal 41.021725, record.details[:test]
end
```
2. Run the test against a remote spanner instance (it passes on the emulator).
3. The test should `error` with with:
```
1) Error:
ActiveRecord::Type::DateTest#test_float_serialization:
ActiveRecord::StatementInvalid: Google::Cloud::FailedPreconditionError: 9:Invalid value for column details in table test_types: Expected JSON.. debug_error_string:{UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:4005:80e::200a%5D:443 {grpc_message:"Invalid value for column details in table test_types: Expected JSON.", grpc_status:9}}
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1813:in `rescue in commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1775:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-2.28.0/lib/google/cloud/spanner/service.rb:557:in `commit'
lib/spanner_client_ext.rb:29:in `commit_transaction'
lib/activerecord_spanner_adapter/transaction.rb:105:in `commit'
lib/activerecord_spanner_adapter/connection.rb:286:in `commit_transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:305:in `block in commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract_adapter.rb:1142:in `log'
lib/active_record/connection_adapters/spanner_adapter.rb:327:in `log'
lib/active_record/connection_adapters/spanner/database_statements.rb:304:in `commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:400:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:514:in `block in commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:503:in `commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:565:in `ensure in block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:564:in `block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:239:in `transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/transactions.rb:212:in `transaction'
lib/activerecord_spanner_adapter/base.rb:28:in `create!'
acceptance/cases/type/json_test.rb:42:in `test_float_serialization'
```

The problematic value here is `41.021725` -- passing that in as a root node to the JSON (which is valid JSON) also fails.

[1] https://github.com/ruby/json/commit/7d77415108c1fd0f55543cf0705abcb7c679d17f
[2] https://github.com/nlohmann/json

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.