apache / apache/pulsar

Pulsar Functions Leak Database Credentials to Logs

Open
#9,462 2 comments 0 reactions 0 assignees View on GitHub
help wanted type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Is your enhancement request related to a problem? Please describe.**
When I look at logs of function workers, I can see where database passwords are revealed in plain text.

```
21:51:55.889 [main] INFO org.apache.pulsar.functions.worker.FunctionAssignmentTailer - Received assignment update: instance {
functionMetaData {
functionDetails {
tenant: "market_data"
namespace: "deribit"
name: "skew_iv_v2"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
processingGuarantees: EFFECTIVELY_ONCE
autoAck: true
parallelism: 1
source {
subscriptionType: FAILOVER
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
subscriptionName: "clickhouse-sink"
inputSpecs {
key: "market_data/deribit/skew_iv_v2"
value {
}
}
cleanupSubscription: true
}
sink {
className: "org.apache.pulsar.io.jdbc.ClickHouseJdbcAutoSchemaSink"
configs: "{\"userName\":\"USERNAME\",\"password\":\"PASSWORD\",\"jdbcUrl\":\"jdbc:clickhouse://ADDRESS:PORT/DATABASE\",\"tableName\":\"TABLE_NAME\",\"timeoutMs\":60000.0,\"batchSize\":100000.0}"
typeClassName: "org.apache.pulsar.client.api.schema.GenericRecord"
builtin: "jdbc-clickhouse"
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SINK
}
packageLocation {
packagePath: "market_data/deribit/skew_iv_v2/08b8f87d-6db5-4907-a85a-4f1fac9c5d5d-pulsar-io-jdbc-clickhouse-2.6.1.nar"
originalFileName: "pulsar-io-jdbc-clickhouse-2.6.1.nar"
}
version: 7
createTime: 1612193701790
instanceStates {
key: 0
value: RUNNING
}
functionAuthSpec {
data: "viv6c"
}
}
instanceId: -1
}
workerId: "pulsar-function-0"
```

This is an example snippet of logs I see. I've replaced some information with placeholders in all-caps, like `USERNAME` and `PASSWORD`. Most sensitive information is seen under `configs`.

Note that this is also the ClickHouse JDBC sink.

**Describe the solution you'd like**
For database credentials (at least) to not be exposed in the logs, or some option to disable it. I can already restrict access to who can administrate the cluster, but I cannot as easily restrict what logs someone is able to see in our log monitoring solution.

Contributor guide

Open the contributing guide

Research direction

Start at the org.apache.pulsar.functions.worker.FunctionAssignmentTailer entry point shown in the log and trace how received assignments and sink configs are rendered. Done means database credentials such as usernames and passwords no longer appear in function-worker logs, with any supported log-control behavior covered by the project’s existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.