influxdata / influxdata/telegraf

Redis plugin: cannot use EVAL using custom commands

Open
#9,103 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/redis feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

Feature Request

The following PR #8196 allows the user to specify commands that are run in redis capturing their output as a metric.

The problem with it is that we cannot use EVAL to run some lua code that gathers those metrics. The usage of EVAL would be really convenient because you can actually script the metrics gathering logic. For example if you want to gather some metrics dynamically (let's say run a KEYS something and then for each key run other commands to gather some specific metrics) then your best option is to script it with Lua that is run through EVAL in redis.
This use case was also discussed in the issue #1027 that lead to the PR above.

Specifically the problem is that the required second argument for EVAL is an integer with the number of arguments. This is an example of the simplest EVAL command you could run

EVAL 'return 1' 0

If you put it in the commands section something like:

[[inputs.redis.commands]]
  command = ["EVAL", "return 1", 0]

You're going to receive the error (because you can't mix types in array in toml)

2021-04-08T09:39:41Z E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.conf: error parsing redis, line 67: (redis.RedisCommand.Command) array can't contain multiple types

if you try to fix it by using a string as last element like this:

[[inputs.redis.commands]]
  command = ["EVAL", "return 1", "0"]

you'll have the following error (because EVAL expects a number and not a string as second argument)

2021-04-08T09:42:02Z E! [inputs.redis] Error in plugin: redis: unexpected type=int64 for String
Proposal:

I don't think there is a strong reason to maintain it as array of single statements that compose a command. It looks like it has been done mainly because that's the input expected by redis client, but IMO it doesn't feel natural from a configuration point of view.
My proposal is to deprecate the command array, create another parameter where you can actually specify the whole command in one string.

Something like this:

[[inputs.redis.commands]]
  commands = ["EVAL 'return 1' 0"]

This would overcome the "cannot contain multiple types" and allow the consumer to write the script in one script as it would normally be expected.

NOTE: also, we would need to be able to configure to interpret the output not as a metric, but as influx lines. This will close the loop since the lua output could be directly the influx lines (that's the use case when you run a script dynamically gathering N metrics)

Current behavior:

We're not able to run EVAL commands in redis plugin custom commands due to the limitation explained above

Desired behavior:

We should be able to run EVAL commands in redis plugin custom commands

Use case:

Gather custom metrics with Lua logic run in redis through redis plugin custom commands.

Workaround:

Do not use redis plugin and go with [[inputs.exec]] executing a custom bash script using redis-cli EVAL 'your-lua-code' 0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Redis plugin custom-command handling in plugins/inputs/redis/redis.go and review how the command array is parsed and executed. Define the configuration and output behavior needed for EVAL, including the proposed influx-line handling; done means Redis Lua commands can run without TOML type errors and their output is handled as configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, lua, redis
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.