googleapis / googleapis/ruby-spanner-activerecord

Floating point serialization issues in JSON fields

Abierto
#376 0 comentarios 1 reacción 1 asignado Reclamado por @olavloite Ver en GitHub
api: spanner priority: p2 type: bug
Lenguaje dominante
Ruby
Estrellas
105
Forks
37
Merge medio
10 h 2 min
PR fusionados (30 d)
5

Descripción

We've recently run into some issues serializing specific floating point numbers. We have hit these before, and our understanding is that they stem from Ruby using a different floating point serialization algorithm from Spanner (which spanner runs some API-layer checks to validate).

Prior to this, we were able to work around them by monkeypatching `Float#to_s` to use a matching grisu2 implementation from [2].

Unfortunately after `v2.11.0` of `ruby/json`[1], this workaround no longer works. `ruby/json` no longer delegates the serialization back to `Float#to_s` and instead uses it's own C implementation of `grisu2`. It's worth noting this isn't a new issue, but has become much more challenging to monkeypatch away now that it doesn't call back to the Ruby layer.

Our understanding is Spanner also uses grisu2 internally, however there seems to be some inconsistencies with implementations in [2] and [1]. In an ideal scenario, the SDK with should maintain a consistent serialization interface with the validations performed by the API layer.

#### Steps to reproduce

1. Add the following to `acceptance/cases/type/json_test.rb`:
```
def test_float_serialization
record = TestTypeModel.create! details: { test: 41.021725 }
record.reload

assert_equal 41.021725, record.details[:test]
end
```
2. Run the test against a remote spanner instance (it passes on the emulator).
3. The test should `error` with with:
```
1) Error:
ActiveRecord::Type::DateTest#test_float_serialization:
ActiveRecord::StatementInvalid: Google::Cloud::FailedPreconditionError: 9:Invalid value for column details in table test_types: Expected JSON.. debug_error_string:{UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:4005:80e::200a%5D:443 {grpc_message:"Invalid value for column details in table test_types: Expected JSON.", grpc_status:9}}
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1813:in `rescue in commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1775:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-2.28.0/lib/google/cloud/spanner/service.rb:557:in `commit'
lib/spanner_client_ext.rb:29:in `commit_transaction'
lib/activerecord_spanner_adapter/transaction.rb:105:in `commit'
lib/activerecord_spanner_adapter/connection.rb:286:in `commit_transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:305:in `block in commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract_adapter.rb:1142:in `log'
lib/active_record/connection_adapters/spanner_adapter.rb:327:in `log'
lib/active_record/connection_adapters/spanner/database_statements.rb:304:in `commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:400:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:514:in `block in commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:503:in `commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:565:in `ensure in block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:564:in `block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:239:in `transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/transactions.rb:212:in `transaction'
lib/activerecord_spanner_adapter/base.rb:28:in `create!'
acceptance/cases/type/json_test.rb:42:in `test_float_serialization'
```

The problematic value here is `41.021725` -- passing that in as a root node to the JSON (which is valid JSON) also fails.

[1] https://github.com/ruby/json/commit/7d77415108c1fd0f55543cf0705abcb7c679d17f
[2] https://github.com/nlohmann/json

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.