apache / apache/seatunnel

[Improve][Zeta] Runtime log-level management: invalid levels report SUCCESS, no way to revert, undocumented, only on the legacy REST plane

Open
#11,981 1 comment 0 reactions 1 assignee Claimed by @SEZ9 View on GitHub
improve
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issue.

### Description

#### What exists today

Runtime log-level management is **already implemented** — this issue is about
making it usable, not about adding it:

- `Log4j2HttpGetCommandProcessor` serves `HttpCommandProcessor.URI_LOG_LEVEL` and
returns every configured logger with its level.
- `Log4j2HttpPostCommandProcessor` applies a level via `Configurator.setLevel()` /
`setRootLevel()`.
- Both are registered in `NodeExtension#createTextCommandService()`.

#### Problems

1. **An invalid level is reported as SUCCESS (bug).** `Level.getLevel(level)`
returns `null` for an unknown name; that `null` is passed straight to
`Configurator.setLevel(...)` and the handler then responds
`{"status":"SUCCESS"}` unconditionally. `POST ... logger&DEBUGG` tells the
caller it worked while the level it asked for was never applied.

2. **A change cannot be reverted at runtime.** `URI_LOG_LEVEL_RESET` is
hard-coded to reject with 500 (`"Reset logger level endpoint disabled!"`).
Combined with (4), an operator cannot even find out what the original level
was.

3. **Reaching it requires enabling the legacy Hazelcast REST plane.** The shipped
`config/hazelcast.yaml` has `network.rest-api.enabled: false`, and enabling it
also activates Hazelcast's own endpoint groups (`CLUSTER_WRITE`, `DATA`).
Turning that on merely to adjust a log level is a poor trade in a production
cluster. The endpoint is also outside the servlet-based v2 REST plane, so it
does not go through the filters wired in `JettyService`
(`ExceptionHandlingFilter`, `BasicAuthFilter`); on the legacy plane the GET
path performs no authentication, while POST authenticates via credentials
**inside the request body** (`user&pass&logger&level`, `application/text`).

4. **No way to tell configured from overridden.** The GET response reports
effective levels with no indication of whether a level came from
`log4j2.properties` or from a runtime override.

5. **No audit trail.** A DEBUG level left on by someone is untraceable.

6. **Node-local only.** No cluster scope, though the pattern exists: `LogService`
already fans out with `nodeEngine.getClusterService().getMembers()` +
`sendOperationToMemberNode`.

7. **Undocumented.** `log-level` does not appear anywhere under `docs/`.

#### Why not rely on `monitorInterval` (already 60s in our config)?

| Dimension | File hot-reload (existing) | Runtime override |
|---|---|---|
| Latency | up to 60s | immediate |
| Scope | per-node file edit; 10 nodes = 10 edits | one call, optional cluster fan-out |
| Kubernetes | ConfigMap propagation is slow; `subPath` mounts never update mtime, so hot-reload is silently dead | filesystem-independent |
| Persistence | permanent | reverts on restart — desirable for temporary diagnostics |
| Audit | engine-invisible | audit log |

They are complementary: file reload is configuration management, runtime override
is incident intervention. This issue does not change or remove either.

#### Proposal

**(a) Fix the validation bug** (small, independently mergeable): reject unknown
levels with a 4xx and a message listing valid levels; only respond SUCCESS when a
level was actually applied.

**(b) Expose the capability on the v2 servlet REST API**, so it inherits the
existing auth/exception filters and does not require enabling the legacy plane:

- `GET /loggers` — configured loggers, effective level, and `origin`
(`file` | `runtime-override`).
- `GET /loggers/{name}` — single logger.
- `POST /loggers/{name}`, body `{"level":"DEBUG"}` — validated; 400 on garbage.
- `DELETE /loggers/{name}` — revert to the file-configured level (fixing problem 2
without re-enabling Hazelcast's reset semantics).
- Optional `?scope=cluster` — fan out over cluster members using the existing
`LogService` mechanism; per-member success/failure in the response; partial
failure is not rolled back (levels are independent per node and reset on
restart).

**(c) Audit**: one INFO line per change — caller, logger, old → new level, scope.

**(d) Document** in `rest-api-v2.md`, and cross-reference from `logging.md`
explaining when to use file reload vs runtime override.

Interaction with hot-reload (documented, not merged): if `log4j2.properties`
changes after an override, the next reload rebuilds the configuration and wins.
This proposal does not attempt merge logic; the `origin` field keeps the current
source of truth visible.

The legacy endpoint keeps working unchanged — no behaviour is removed.

#### Non-goals

- No persistence of overrides back to `log4j2.properties`.
- No TTL / auto-revert (possible follow-up).
- No Web UI page; the API is shaped so one can be added later.

#### Validation

- Unit: level validation (valid, invalid, lowercase, `root`); `origin` reporting;
behaviour after a file reload following an override.
- E2E: set DEBUG on a package via REST, assert log output changes without restart;
`DELETE` restores the file level; restart restores the file level.
- Docs updated as above.

#### Compatibility

Additive endpoints; existing logging behaviour unchanged when unused.

### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.