elastic / elastic/logstash

JDBC / Oracle / Timestamp Formating

Open
#5,615 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

I am using the record_last_value configration but the Timestamp formatting is using timezone format and that does not return any rows from the Oracle database. If you correct the format manually then I get valid results.

**Format used by Logstash from debug logs that returns no rows**
SELECT SAMPLE_TIME, EVENT, TIME_WAITED/1000 FROM V$ACTIVE_SESSION_HISTORY WHERE EVENT='log file parallel write'
and SAMPLE_TIME > TIMESTAMP '2016-07-11 07:20:00.225000 **+00:00**';

**Correct format that works and returns valid rows**
SELECT SAMPLE_TIME, EVENT, TIME_WAITED/1000 FROM V$ACTIVE_SESSION_HISTORY WHERE EVENT='log file parallel write'
and SAMPLE_TIME > TIMESTAMP '2016-07-11 07:20:00.225000000'

**Configuration I am using**
input {
jdbc {
jdbc_validate_connection => true
jdbc_connection_string => "jdbc:oracle:oci:@*******_"
jdbc_user => "**_***_"
jdbc_password => "**_**_"
jdbc_driver_library => "C:\Oracle\product\11.2.0\client_1\jdbc\lib\ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement => "SELECT SAMPLE_TIME, EVENT, TIME_WAITED/1000 FROM V$ACTIVE_SESSION_HISTORY WHERE EVENT='log file parallel write' and SAMPLE_TIME > :sql_last_value"
last_run_metadata_path => "C:\logstash-2.3.4\logstash-oradb.lastrun"
record_last_run => true
schedule => "_/2 \* \* \* *"
}
}
filter {
# Set the timestamp to that of the ASH sample, not current time.
mutate { convert => [ "sample_time" , "string" ]}
date { match => ["sample_time", "ISO8601"]}
}
output {
stdout { codec => rubydebug }
#elasticsearch {}
}

Logstash version 2.3.4

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the shown JDBC configuration, Oracle connection, and :sql_last_value query predicate. Start by tracing how record_last_value is converted for the JDBC statement; done means the generated timestamp format is accepted by Oracle and the query returns the expected rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.