Is there a way for the log gathering to be smarter about lines that should be together?

The base assumption is one line = one log record.

This works in most cases. In some cases, like Rails logs, this may not be the case. If there is a stack dump, for instance, there will be a bunch of lines that are all part of a single log record.

The agent should have a way to be smarter about this. Maybe the solution is to provide a set of log preprocessors, so when a log is defined in the config file, the definition can name a preprocessor that can handle these special cases.