Introduce logging and log management for all available processes

Start by reviewing the BusyBox init behavior and the existing /etc/init.d/SXX scripts, especially S01syslog, to determine what logging and rotation support is available. Compare how startup processes and daemons currently handle stdout and stderr, then document the proposed paths, startup logging structure, rotation, purging, and plaintext conventions in a Markdown document. Done means the approach is documented and the feasibility limits of the existing init system are clear.

Spike Type

Functional

Goal

Desired attributes for NER logging:

• A standardized, documented location for logging using tmpfs (probably /var/log/) such that sd card wear is avoided
• Each startup process NER wrote writes to file in the subfolder of the above directory using start-stop-daemon piping in the /etc/init.d/SXX file. The structure and implentation of logging and the wider startup shell scripting should be documented in a markdown document for addition to confluence or github. Note: busybox's implementation does not match openrc or even debians implentation!
• Each running daemon uses specific internal functions to write to logs in the root or seperate sub directory of the tmpfs, usually configured via config file.
• log rotation is configured such that the logs are written only periodically, and rotated given sane size and time maximums. Additionally a log purging setting should be made. This is the hardest part of this feature request as we don't want logs to affect RAM but should reduce wear. Programs for this exist, and may be already running by default in busybox init system.
• All paths are documented in a centralized location so logs can be fetched easily. all logs should be stored in plaintext form.

Reason for Spike

Currently debugging program failures that only occur in the startup environment is nearly impossible (see below for current status). Also, logs are not persisted at all, so retracing history post power loss is completely impossible if it was not emitted to a network peer.

Why could boot failures occur?
Programs may fail only at boot if the init script has a bug or a resource or process used by the startup daemon is unavailable or not yet configured. Scenarios include bringing up network processes without wlan establishment or hardware being brought up after their userland companion processes.

Additional notes

Current situation with logging:
Currently, startup process (those in /etc/init.d) stdout/stderr is logged to the serial console, however background processes (like those configured by start-stop-daemon) pipe stdout and stderr to oblivion. Each process usually can log individually using its internal functionality. Furthermore, the serial output described aboce is not routed to /var/log/messages on syslog implementations (see S01syslog). Also, all logs are purged on shutdown due to the nature of tmpfs.

Important note: It is very possible the desired goals of the logging are out of the scope of existing buysbox init system capabilities. If that becomes evident, comparisons with other init systems should be made.