Clear and Compact User Error Messages

Is your feature request related to a problem? Please describe.

When Data Prepper fails due to a user error, Data Prepper outputs a stack trace. The user of Data Prepper must make sense of this stack trace to know how to resolve the error. I am including a sample stack trace as additional context at the bottom of the issue.

Describe the solution you'd like

For user errors, return a cleaner message which provides more descriptive context.

FAILURE

Where:
Failed to parse the configuration file pipelines.yaml. Unable to parse pipeline, simple-test-pipeline.

Why:
Invalid configuration, at least one sink is required

For more help, visit the project page: https://github.com/opensearch-project/data-prepper

Scope

Unexpected failures in Data Prepper Core and in plugins should still result in stack trace output. When these unexpected scenarios occur, they are likely bugs and stack traces will help track them down.

Proposed Approach

Update Data Prepper API to include specific exceptions which indicate that a user errors has occurred. In Data Prepper Core, catch these types of exceptions and then wrap them in an outer exception which has additional context.

An example is plugin configuration errors. Plugins can throw a standardized exception - perhaps InvalidPluginConfigurationException. The pipeline parsing code can then catch that specific exception and wrap it into an exception which includes additional context and indicates that the exception is user-related. Finally, that exception is caught in DataPrepperExecute and used to print out the improved user message.

Help Page

The example output above refers users to a specific web page for help. It may also be valuable to have a single page dedicated to helping users. It might point to the documentation, suggest the forums, and include a link to submit a bug report.

For example:

For more help, visit the project page: https://github.com/opensearch-project/data-prepper/docs/help.md

Additional context

Current output:

Exception in thread "main" com.amazon.dataprepper.parser.ParseException: Failed to parse the configuration file pipelines.yaml
	at com.amazon.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:75)
	at com.amazon.dataprepper.DataPrepper.execute(DataPrepper.java:129)
	at com.amazon.dataprepper.DataPrepperExecute.main(DataPrepperExecute.java:33)
Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `com.amazon.dataprepper.parser.model.PipelineConfiguration`, problem: Invalid configuration, at least one sink is required
 at [Source: (File); line: 6, column: 1] (through reference chain: java.util.LinkedHashMap["simple-test-pipeline"])
	at com.fasterxml.jackson.databind.exc.ValueInstantiationException.from(ValueInstantiationException.java:47)
	at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:2047)
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:587)
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:610)
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:293)
	at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:288)
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:202)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:518)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:351)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:184)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:609)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:437)
	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4675)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3515)
	at com.amazon.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:58)
	... 2 more
Caused by: java.lang.IllegalArgumentException: Invalid configuration, at least one sink is required
	at com.amazon.dataprepper.parser.model.PipelineConfiguration.getSinksFromConfiguration(PipelineConfiguration.java:107)
	at com.amazon.dataprepper.parser.model.PipelineConfiguration.<init>(PipelineConfiguration.java:45)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:128)
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:291)
	... 15 more

Outstanding Questions

What exceptions should result in these clear errors?

I tend to think that Data Prepper should catch JSR-380 exceptions and also have a custom exception that plugins can throw when manual validation is necessary.

• InvalidPluginConfigurationException - from Data Prepper API
• ConstraintViolationException - from JSR-380
• ValidationException - from JSR-380

Where should the errors go?

Should Data Prepper log these errors to the logs? Or perhaps stdout or stderr? Perhaps both? Or should it be configurable?

Tasks

• Create a standard exception for invalid plugin configurations - InvalidPluginConfigurationException
• Catch validation errors and extract the relevant error
• Catch YAML parsing error and extract the relevant error
• Output the error in a clean format

Start by reading PipelineParser.java, DataPrepper.java, DataPrepperExecute.java, and PipelineConfiguration.java, which appear in the example stack trace. Review the Data Prepper API and plugin validation paths before deciding which exceptions and output destinations are in scope. Done means the listed validation and YAML errors produce a clean contextual message while unexpected failures retain stack traces.