Graylog2 / Graylog2/graylog2-server

Date functions do not work when using Pipeline Simulator or Pipeline Rule Simulator

Open
#20,274 2 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Some pipeline functions, such as working with dates, require a lot of trial and error and testing to understand what the function expects as input and what the output will look like.

Typically, the Pipeline Simulator or Pipeline Rule Simulator can serve this need.

However, date functions do not work. For example, if i have a gelf message (copied from an existing Graylog message using 'copy message' so we can be sure its valid) and try to test against a pipeline rule that requires DateTime types, the functions will return an error.

Example:

`to_date($message.timestamp)` always return empty/null. Using `debug()` does not output anything. Note that a real message processed in real time does not have this issue and works as exepcted.

is_date($message.timestamp) always returns false

## Expected Behavior

Pipeline date functions, specifically `to_date`, work the same during realtime processing as well as pipeline simulation

## Current Behavior

Its impossible to test date functions via pipelines.

## Possible Solution

## Steps to Reproduce (for bugs)

1. Create a pipeline rule

```
rule "TEST for GH issue"
when
has_field("timestamp")
then
let date_string = to_date($message.timestamp);
// will be empty in simulation
debug(date_string);
// will be false in sumulation
debug(is_date(date_string));
end
```

2. if you do not already have a test/debug pipeline
* Create a new stream (we don't need a stream but this is required for the simulator to work)
* Create a new pipeline and connect to above stream
* attach rule above to pipeline
2. test with pipeline simulator:
* Find an existing graylog message and click "copy message"
* navigate to System / Pipelines / Simulator
* Paste gelf message, set message codec to gelf
3. Observe output of `debug()` via `server.conf`

## Context

Difficulty working with DateTime to get the desired outcome. Lack of helpful feedback when testing/debugging. Requires manipulating live messages to test.

## Your Environment

* Graylog Version: 6.0.5
* Java Version: Bunlded
* OpenSearch Version: 2.x
* MongoDB Version: 6.x
* Operating System: Ubuntu Server 22.04 LTS
* Browser version: Google Chrome Version 127.0.6533.120 (Official Build) (arm64)

Please let me know if there are any questions

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.