Graylog2 / Graylog2/graylog2-server
http notification to service behind traefik fails
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
I was trying to `Execute Test Notification` a http notification to a [n8n](https://n8n.io) instance behind a [traefik](https://traefik.io/traefik/) reverse proxy.
(url is https enabled with let's encrypt by traefik, looks something like: https://n8n.some.example.com/webhook-test/4534)
## Expected Behavior
The test notification should succeed.
(Works if tried with webhook.site)
## Current Behavior
The test notification always failed with: `Error: Unknown error, please check your Graylog server logs.`
Related Error Log
```
2021-01-09 13:12:27,706 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An exception has been thrown from an exception mapper class org.graylog2.shared.rest.exceptionmappers.WebApplicationExceptionMapper.
java.lang.NullPointerException: Null message
at org.graylog2.plugin.rest.AutoValue_ApiError.(AutoValue_ApiError.java:14) ~[graylog.jar:?]
at org.graylog2.plugin.rest.ApiError.create(ApiError.java:28) ~[graylog.jar:?]
at org.graylog2.shared.rest.exceptionmappers.WebApplicationExceptionMapper.toResponse(WebApplicationExceptionMapper.java:35) ~[graylog.jar:?]
at org.graylog2.shared.rest.exceptionmappers.WebApplicationExceptionMapper.toResponse(WebApplicationExceptionMapper.java:27) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.java:601) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:476) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:334) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
2021-01-09 13:12:27,707 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An exception was not mapped due to exception mapper failure. The HTTP 500 response will be returned.
javax.ws.rs.InternalServerErrorException: null
at org.graylog.events.notifications.NotificationResourceHandler.test(NotificationResourceHandler.java:186) ~[graylog.jar:?]
at org.graylog.events.rest.EventNotificationsResource.test(EventNotificationsResource.java:196) ~[graylog.jar:?]
at sun.reflect.GeneratedMethodAccessor616.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
Caused by: org.graylog.events.notifications.PermanentEventNotificationException
at org.graylog.events.notifications.types.HTTPEventNotification.execute(HTTPEventNotification.java:118) ~[graylog.jar:?]
at org.graylog.events.notifications.NotificationResourceHandler.test(NotificationResourceHandler.java:181) ~[graylog.jar:?]
... 27 more
2021-01-09 14:31:36,220 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.util.NoSuchElementException: null
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:75) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
at org.graylog.events.notifications.types.HTTPEventNotification.execute(HTTPEventNotification.java:112) ~[graylog.jar:?]
at org.graylog.events.notifications.NotificationResourceHandler.test(NotificationResourceHandler.java:181) ~[graylog.jar:?]
at org.graylog.events.rest.EventNotificationsResource.test(EventNotificationsResource.java:196) ~[graylog.jar:?]
at sun.reflect.GeneratedMethodAccessor616.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
```
## Possible Solution
I think there is some problem with the used okhttp3 version and traefik http2.
Possibly related:
https://github.com/square/okhttp/issues/4875
https://github.com/square/okhttp/issues/3308
### Solution:
**Maybe upgrade to the newest okhttp3 version?**
## Steps to Reproduce (for bugs)
1. In n8n create workflow with **Webhook Trigger**
2. Set Webhook Trigger to `POST`
3. Get Webhook Testing URL from n8n GUI
4. Start Node/Webhook in n8n
5. Paste n8n webhook url in graylog notification form
6. Press `Execute Test Notification`; wait few seconds and fails
## Context
I am 99% sure, that it is a problem with the okhttp3 library and some SSL/TLS stuff from traefik.
When i use `http` instead of `https` to address the url of the webhook it works.
Also it works with https on webhook.site.
## Your Environment
Graylog, Elasticsearch, MongoDB are all running in independent Docker Containers and use official images.
(I also expose graylog with same traefik to the web.)
* Graylog Version: Graylog 3.3.8+e223f85 on 44107018ddbc (Oracle Corporation 1.8.0_265 on Linux 5.4.0-52-generic)
* Java Version: openjdk version "1.8.0_265"
* Elasticsearch Version: elasticsearch:6.8.9
* MongoDB Version: mongo:4.2.6-bionic
* Operating System: Host of Docker is Ubuntu 20.04.1
* Browser version: Safari Version 14.0.2
* traefik: traefik:v2.3.2
* n8n: n8nio/n8n:0.101.0
DNS Setting for traefik at the domain provider website is:
| Name | Type | Content |
|---------------------|------|-----------------|
|*.some.example.com | A | 123.123.123.123 |
Graylog docker-compose.yml
```yaml
version: '3'
services:
# MongoDB: https://hub.docker.com/_/mongo/
sense-snow-graylog-db:
image: mongo:4.2.6-bionic
container_name: sense-snow-graylog-db
restart: unless-stopped
volumes:
- sense-snow_graylog_db_mongo_data:/data/db
networks:
- graylog-sense-snow-net
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
sense-snow-graylog-elasticsearch:
# image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.5
image: elasticsearch:6.8.9
container_name: sense-snow-graylog-elasticsearch
restart: unless-stopped
volumes:
- sense-snow_graylog_es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
networks:
- graylog-sense-snow-net
# Graylog: https://hub.docker.com/r/graylog/graylog/
sense-snow-graylog:
image: graylog/graylog:3.3
container_name: sense-snow-graylog
restart: unless-stopped
env_file: config.env
networks:
- graylog-sense-snow-net
- proxy
depends_on:
- sense-snow-graylog-db
- sense-snow-graylog-elasticsearch
volumes:
- sense-snow_graylog_journal:/opt/graylog/data
labels:
traefik.enable: 'true'
traefik.http.routers.log-sense-snow.entrypoints: 'http'
traefik.http.routers.log-sense-snow.rule: 'Host(`log.some.example.com`)'
traefik.http.middlewares.log-sense-snow-https-redirect.redirectscheme.scheme: 'https'
traefik.http.middlewares.log-sense-snow-header.headers.accesscontrolalloworigin: '*'
traefik.http.middlewares.log-sense-snow-header.headers.accesscontrolallowheaders: 'Cache-Control'
traefik.http.routers.log-sense-snow.middlewares: 'log-sense-snow-https-redirect'
traefik.http.routers.log-sense-snow-secure.entrypoints: 'https'
traefik.http.routers.log-sense-snow-secure.rule: 'Host(`log.some.example.com`)'
traefik.http.routers.log-sense-snow-secure.tls: 'true'
traefik.http.routers.log-sense-snow-secure.tls.certresolver: 'http'
traefik.http.routers.log-sense-snow-secure.service: 'log-sense-snow'
traefik.http.services.log-sense-snow.loadbalancer.server.port: '9000'
traefik.docker.network: 'proxy'
networks:
graylog-sense-snow-net:
external: true
proxy:
external: true
volumes:
sense-snow_graylog_db_mongo_data:
driver: local
sense-snow_graylog_es_data:
driver: local
sense-snow_graylog_journal:
driver: local
```
### config.env
```env
# Graylog Configuration
GRAYLOG_HTTP_EXTERNAL_URI=https://log.some.example.com/
# password pepper
GRAYLOG_PASSWORD_SECRET=some
# echo -n yourpassword | shasum -a 256
GRAYLOG_ROOT_PASSWORD_SHA2=xyz
GRAYLOG_ROOT_USERNAME=myname
GRAYLOG_MONGODB_URI=mongodb://sense-snow-graylog-db/graylog
GRAYLOG_ELASTICSEARCH_HOSTS=http://sense-snow-graylog-elasticsearch:9200
```
n8n docker-compose.yml
```yaml
version: '3.7'
services:
n8n:
image: n8nio/n8n:0.101.0
container_name: n8n
restart: unless-stopped
environment:
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER=myUsername
- N8N_BASIC_AUTH_PASSWORD=myPassword
- WEBHOOK_TUNNEL_URL=https://n8n.some.example.com/
volumes:
- n8n-data:/root/.n8n
networks:
- proxy
labels:
traefik.enable: 'true'
traefik.http.routers.n8n.entrypoints: 'http'
traefik.http.routers.n8n.rule: 'Host(`n8n.some.example.com`)'
traefik.http.routers.n8n-secure.entrypoints: 'https'
traefik.http.routers.n8n-secure.rule: 'Host(`n8n.some.example.com`)'
traefik.http.routers.n8n-secure.tls: 'true'
traefik.http.routers.n8n-secure.tls.certresolver: 'http'
traefik.http.routers.n8n-secure.service: 'n8n'
traefik.http.services.n8n.loadbalancer.server.port: '5678'
traefik.docker.network: 'proxy'
volumes:
n8n-data:
networks:
proxy:
external: true
```
Contributor guide
Assessment
This issue has not been assessed yet.