colinmarc / colinmarc/impala-ruby
value of string column that contains tabs may cause errors
- Dominant language
- Thrift
- Stars
- 34
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
Hi all:
I met an error recently, my sql is: select * from my_table where ymd >= 20181128 and ymd <= 20181129, and the error is:
```
ArgumentError: no time information in "xxxxxx"
from /Users/yy/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/time.rb:327:in `parse'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:135:in `convert_raw_value'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:109:in `block in parse_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:108:in `each'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:108:in `parse_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `block in fetch_batch'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `map'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:95:in `fetch_batch'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:82:in `fetch_more'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:40:in `fetch_row'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:26:in `each'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:51:in `to_a'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/cursor.rb:51:in `fetch_all'
from /Users/yy/.rvm/gems/ruby-2.1.5@my_project/gems/impala-0.5.1/lib/impala/connection.rb:69:in `query'
```
here is my table info:
|name | type |
------------ | -------------
| ymd | int |
| content | string |
| create_time | timestamp |
Finally I found out that this error was caused by the value of content column: the value includes a TAB ! so the content value was split into two parts, and the "convert_raw_value" method try to parse the second part to the type of next column -- timestamp, and caused error.
Thanks !
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in impala/cursor.rb around convert_raw_value and parse_row, as shown in the stack trace, and reproduce the query with a string column containing a tab before a timestamp column. Done means tab-containing string values remain intact, subsequent columns are parsed with their declared types, and the query no longer raises the reported ArgumentError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100