colinmarc / colinmarc/impala-ruby

value of string column that contains tabs may cause errors

Open
#26 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Thrift
Stars
34
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Hi all:
I met an error recently, my sql is: select * from my_table where ymd >= 20181128 and ymd <= 20181129, and the error is:
```
ArgumentError: no time information in "xxxxxx"
from /Users/yy/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/time.rb:327:in `parse'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:135:in `convert_raw_value'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:109:in `block in parse_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:108:in `each'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:108:in `parse_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `block in fetch_batch'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `map'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `fetch_batch'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:82:in `fetch_more'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:40:in `fetch_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:26:in `each'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:51:in `to_a'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:51:in `fetch_all'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/connection.rb:69:in `query'
```
here is my table info:

|name | type |
------------ | -------------
| ymd | int |
| content | string |
| create_time | timestamp |

Finally I found out that this error was caused by the value of content column: the value includes a TAB ! so the content value was split into two parts, and the "convert_raw_value" method try to parse the second part to the type of next column -- timestamp, and caused error.

Thanks !

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in impala/cursor.rb around convert_raw_value and parse_row, as shown in the stack trace, and reproduce the query with a string column containing a tab before a timestamp column. Done means tab-containing string values remain intact, subsequent columns are parsed with their declared types, and the query no longer raises the reported ArgumentError.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.