tarantool / tarantool/crud

operation_data usability

Open
#396 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Lua
Stars
43
Forks
17
Avg merge
28m
Merged PRs (30d)
2

Description

*_many operation provides operation_data with each error so it would be possible to

  • track which exact record had been failed and
  • retry failed records afterward.

But operation_data that may be returned is inconsistent, may lose some info or contain additional values that may change the behavior (like bucket_id).

./doc/playground.lua
tarantool> crud.insert_many('customers', {{'asd'}})
---
- null
- - line: 58
    class_name: CallError
    err: 'Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned an error:
      Tuple field 1 (id) type does not match one required by operation: expected unsigned,
      got string'
    file: '.../crud/crud/common/map_call_cases/batch_postprocessor.lua'
    stack: "stack traceback:\n\t.../crud/crud/common/map_call_cases/batch_postprocessor.lua:58:
      in function 'collect'\n\t...y/Development/github/tarantool/crud/crud/common/call.lua:132:
      in function 'map'\n\t...y/Development/github/tarantool/crud/crud/insert_many.lua:179:
      in function 'method'\n\t...ment/github/tarantool/crud/crud/common/sharding/init.lua:148:
      in function 'func'\n\t...Development/github/tarantool/crud/crud/common/schema.lua:93:
      in function <...Development/github/tarantool/crud/crud/common/schema.lua:88>\n\t[C]:
      in function 'pcall'\n\tbuiltin/box/console.lua:415: in function 'eval'\n\tbuiltin/box/console.lua:754:
      in function 'repl'\n\tbuiltin/box/console.lua:805: in function 'start'\n\t./doc/playground.lua:153:
      in main chunk"
    operation_data: ['asd', 1550]
    str: 'CallError: Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned
      an error: Tuple field 1 (id) type does not match one required by operation:
      expected unsigned, got string'
...

Operation data for ['asd'] is ['asd', 1550].

tarantool> crud.insert_object_many('customers', {{id = 'asd'}})
2023-11-10 17:17:50.891 [1571729] main/103/playground.lua/crud.common.schema schema.lua:108 W> Number of attempts to reload schema has been ended: 1
---
- null
- - line: 314
    class_name: InsertManyError
    err: 'Failed to flatten object: FlattenError: Object is specified in bad format:
      FlattenError: Field "name" isn''t nullable (set skip_nullability_check_on_flatten
      option to true to skip check)'
    file: '...y/Development/github/tarantool/crud/crud/insert_many.lua'
    operation_data:
      id: asd
    str: 'InsertManyError: Failed to flatten object: FlattenError: Object is specified
      in bad format: FlattenError: Field "name" isn''t nullable (set skip_nullability_check_on_flatten
      option to true to skip check)'
...

Operation data for {id = 'asd'} is {id = 'asd'}.

tarantool> crud.insert_object_many('customers', {{id = 'asd', name = 'asd', age = 2}})
2023-11-10 17:20:02.788 [1571729] main/103/playground.lua/crud.common.schema schema.lua:108 W> Number of attempts to reload schema has been ended: 1
---
- null
- - line: 58
    class_name: CallError
    space_schema_hash: 247485528
    err: 'Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned an error:
      Tuple field 1 (id) type does not match one required by operation: expected unsigned,
      got string'
    file: '.../crud/crud/common/map_call_cases/batch_postprocessor.lua'
    stack: "stack traceback:\n\t.../crud/crud/common/map_call_cases/batch_postprocessor.lua:58:
      in function 'collect'\n\t...y/Development/github/tarantool/crud/crud/common/call.lua:132:
      in function 'map'\n\t...y/Development/github/tarantool/crud/crud/insert_many.lua:179:
      in function 'method'\n\t...ment/github/tarantool/crud/crud/common/sharding/init.lua:148:
      in function 'func'\n\t...Development/github/tarantool/crud/crud/common/schema.lua:93:
      in function 'wrap_func_reload'\n\t...y/Development/github/tarantool/crud/crud/insert_many.lua:331:
      in function <...y/Development/github/tarantool/crud/crud/insert_many.lua:280>\n\t[C]:
      in function 'pcall'\n\tbuiltin/box/console.lua:415: in function 'eval'\n\tbuiltin/box/console.lua:754:
      in function 'repl'\n\tbuiltin/box/console.lua:805: in function 'start'\n\t./doc/playground.lua:153:
      in main chunk"
    operation_data: ['asd', 1550, 'asd', 2]
    str: 'CallError: Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned
      an error: Tuple field 1 (id) type does not match one required by operation:
      expected unsigned, got string'
...

Operation data for {id = 'asd', name = 'asd', age = 2} is ['asd', 1550, 'asd', 2].

tarantool> crud.upsert_many('customers', { { {'asd'}, {{'=', 'age', 2}} } })
---
- null
- - line: 58
    class_name: CallError
    err: 'Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned an error:
      Tuple field 1 (id) type does not match one required by operation: expected unsigned,
      got string'
    file: '.../crud/crud/common/map_call_cases/batch_postprocessor.lua'
    stack: "stack traceback:\n\t.../crud/crud/common/map_call_cases/batch_postprocessor.lua:58:
      in function 'collect'\n\t...y/Development/github/tarantool/crud/crud/common/call.lua:132:
      in function 'map'\n\t...y/Development/github/tarantool/crud/crud/upsert_many.lua:193:
      in function 'method'\n\t...ment/github/tarantool/crud/crud/common/sharding/init.lua:148:
      in function 'func'\n\t...Development/github/tarantool/crud/crud/common/schema.lua:93:
      in function <...Development/github/tarantool/crud/crud/common/schema.lua:88>\n\t[C]:
      in function 'pcall'\n\tbuiltin/box/console.lua:415: in function 'eval'\n\tbuiltin/box/console.lua:754:
      in function 'repl'\n\tbuiltin/box/console.lua:805: in function 'start'\n\t./doc/playground.lua:153:
      in main chunk"
    operation_data: ['asd', 1550]
    str: 'CallError: Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned
      an error: Tuple field 1 (id) type does not match one required by operation:
      expected unsigned, got string'
...

Operation data for { {'asd'}, {{'=', 'age', 2}} } is ['asd', 1550].

tarantool> crud.upsert_many('customers', { { {1, box.NULL, 'name', 2}, {{'=!', 'age', 2}} } })
---
- null
- - line: 58
    class_name: CallError
    err: 'Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned an error:
      Unknown UPDATE operation #1: "=!"'
    file: '.../crud/crud/common/map_call_cases/batch_postprocessor.lua'
    stack: "stack traceback:\n\t.../crud/crud/common/map_call_cases/batch_postprocessor.lua:58:
      in function 'collect'\n\t...y/Development/github/tarantool/crud/crud/common/call.lua:132:
      in function 'map'\n\t...y/Development/github/tarantool/crud/crud/upsert_many.lua:193:
      in function 'method'\n\t...ment/github/tarantool/crud/crud/common/sharding/init.lua:148:
      in function 'func'\n\t...Development/github/tarantool/crud/crud/common/schema.lua:93:
      in function <...Development/github/tarantool/crud/crud/common/schema.lua:88>\n\t[C]:
      in function 'pcall'\n\tbuiltin/box/console.lua:415: in function 'eval'\n\tbuiltin/box/console.lua:754:
      in function 'repl'\n\tbuiltin/box/console.lua:805: in function 'start'\n\t./doc/playground.lua:153:
      in main chunk"
    operation_data: [1, 477, 'name', 2]
    str: 'CallError: Failed for 00e44528-9a64-40e1-b180-fb2b14e369f0: Function returned
      an error: Unknown UPDATE operation #1: "=!"'
...

Operation data for { {1, box.NULL, 'name', 2}, {{'=!', 'age', 2}} } is [1, 477, 'name', 2], even though it's update operation that has been failed.

It seems that we need to properly consider the potential usability of operation_data and rework current implementation to satisfy these potentials.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the examples in doc/playground.lua, then inspect operation_data handling in crud/common/map_call_cases/batch_postprocessor.lua, insert_many.lua, and upsert_many.lua. Compare the reported forms across *_many operations and define a consistent result contract that identifies the failed record and supports retries. Done means the operation_data behavior is consistent for the demonstrated cases and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.