Define Expected Bee Node Exit Scenarios and Improve Exit Logging

[aranyia]

Summary

When operating nodes, especially on a larger scale, employing orchestration tools like Kubernetes with Docker containers, the Bee node status can be difficult to gauge or can be ambiguous based on response from the Bee API status endpoints and log contents.

Motivation

Clear and predictable points when a Bee node process is supposed to stop could help detect operational issues earlier.

For example when running in a Docker container, the Bee node process stopping would cause the container to stop, which could be a clear signal to dependent systems that the Bee node instance is not operational and cannot serve traffic.

Logging could be also reorganized around this, always indicating a clear reason for stopping the Bee node before exiting the process. This would be useful not only for easier log analysis but open up the potential to perform automated steps based on last log events.

Implementation

There could be number of scenarios where the Bee node process is expected to reach an exit state. In addition, clear logs should precede the exit event.

Drawbacks / Alternatives

If a similar outcome can be achieved by other probes like the readiness endpoint, alternatives can be suggested. But in that case it should be clear and evident and well documented how those probes should be utilized to achieve the level of consistent behavior (predictable node status, readiness for traffic) described above.

Some other issues that can be relevant here

• Expose whether Bee is in warmup period in a status endpoint · Issue #5022 · ethersphere/bee
• Change warmup period to detect readiness instead of waiting a fixed period · Issue #5045 · ethersphere/bee
• Print log of all non-sensitive bee config values upon startup for log levels: info, debug or trace · Issue #4756 · ethersphere/bee
• /readiness endpoint clarification · Issue #4619 · ethersphere/bee
• Readiness endpoint available very late · Issue #4746 · ethersphere/bee
• bootnodes postage check point · Issue #5000 · ethersphere/bee

[aranyia]

@nikipapadatou