Map features to subsystems to help with detecting issues in logs

Who is this for and what problem do they have today?

I had mistakenly included a typo while configuring shadow indexing on a new Redpanda install. The issue was that I was using google's us-west2 region rather than amazon's us-west-2. While trying to debug, I enabled trace-level logging for two subsystems that I thought would be the most related to my shadow indexing issue: cloud_storage and archival. But the only message that appeared in the log related to the issue was an archival error:

Oct 24 14:31:46 ip-172-31-12-76 rpk[4166]: WARN  2022-10-24 14:31:46,009 [shard 1] archival - [fiber1] - service.cc:284 - Manifest download failed
Oct 24 14:31:46 ip-172-31-12-76 rpk[4166]: ERROR 2022-10-24 14:31:46,009 [shard 1] archival - [fiber1] - service.cc:379 - Failed to create archivers

This wasn't enough details to point me in the right direction. It wasn't until I enabled trace-level logging on http that I found the S3 error that pointed me in the right direction.

What are the success criteria?

If I have issues with shadow indexing, then it makes sense that I could enable more detailed logging on the shadow indexing subsystem and see related issues.

Why is solving this problem impactful?

It's a little difficult at the moment to know just which subsystem out of all of the available subsystems to look into to track down an issue. Subsystems aren't mapped to features anywhere.

Additional notes

This could be resolved with better documentation around how to troubleshoot specific issues, but maybe a better approach would be to include related subsystems in the main documentation page for each feature. For instance, for shadow indexing, a new section could be added called Related subsystems which lists the following:

• archival
• cloud_storage
• http
• s3

Then in some general debugging page we could mention that each feature docs page has this section, and that the user could enable more detailed logging for those specific subsystems when dealing with issues related to that feature.

But it is still very difficult to know what the differences are between archival, cloud_storage, and s3 subsystems... and that a completely different subsystem http is where the error is that will give you enough details to track down the issue.

JIRA Link: CORE-1056

Start with the shadow indexing feature documentation and the general debugging documentation. Document the related archival, cloud_storage, http, and s3 subsystems, explain their differences where possible, and describe how to enable detailed logging for troubleshooting. Done means users can identify which subsystem logs to inspect for shadow indexing issues.