elastic / elastic/logstash

logstash -f <url> fails on https with useless error message

Open
#8,188 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

- Version: docker.elastic.co/logstash/logstash:5.3.3
- Operating System: docker on OS X 10.12.6, but docker is based on Centos 7
- Config File (if you have sensitive info, please remove it): config doesn't matter, just host it on an https-enabled server (I'm using S3)
- Sample Data:
- Steps to Reproduce:

Create a pre-signed URL on S3, which always generates an https URL. Pass that url to the docker image with a command such as the following:

docker run -it docker.elastic.co/logstash/logstash:5.3.3 logstash -f 'https://...'

you'll get an error message like this:

ERROR logstash.agent - failed to fetch pipeline configuration {:message=>"Unable to fetch config from: https://...\nReason: End of file reached"}

Enabling debug logging wasn't any more useful. There were no lower level log messages related to fetching the config file. It would be really useful to have debug output that provides the content it received - at one point, after I started fetching with http instead of https, my URL expired and S3 started serving up error messages (which also implies that the code wasn't checking the HTTP response code, since it wasn't a 2xx response). That resulted in complaints about invalid config but it wasn't possible to see what it was.

tldr; - I just wasted about 2 hours trying to figure out how to get docker to load a config without including it in the image. Some instruction on how to specify the url for the config file in a docker command would be nice. I had to dig into your github repo to see how your docker image was constructed and what the entrypoint looked like in order to figure out what command to run and how to specify it. Then I had to figure out that https doesn't work. Then I had to figure out that status code's don't matter. Not exactly a user-friendly process.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the Logstash Docker command and an HTTPS pre-signed S3 URL, then trace how `logstash -f` fetches and reports remote configuration. Check the behavior for expired URLs and non-2xx responses. Done means HTTPS configuration loading and response failures provide actionable diagnostics, with the Docker invocation documented if documentation is included.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.