Add update_container_documentation script and monitoring profile

[bwbroersma]

• Fixes Options to only create main part of internet.nl Batch API (Django) #1588

[mxsasha]

The number of profiles seems to be rapidly expanding. We've had a dev misconfiguration for them already. I think we need one place in the deployment docs where they are all listed, along with defaults and recommendations. Particularly as misconfiguration may not be immediately obvious.

[bwbroersma]

Agree, also wonder if all services with the monitoring profile should also have the alertmanager profile?

It seems after this addition there are these profiles defined in the compose files:

• alertmanager
• build
• connectiontest
• cron
• monitoring
• routinator
• run-tests

I see batch used in 2 .env files:

Internet.nl/docker/build.env

Line 5 in 0e5a6d1

COMPOSE_PROFILES=connectiontest,cron,routinator,batch,build

Internet.nl/docker/batch-test.env

Line 11 in 0e5a6d1

COMPOSE_PROFILES=batch,cron

However, its unused in all compose files.

[bwbroersma]

@aequitas I think we discusses this also, and some compose profiles (batch) are indeed no longer used and can be deleted right? Is there still an use case to keep run-tests?

[bwbroersma]

Idea:

• add more documentation to the compose file
• generate container list (and which are crucial) from this
• include in which profiles containers are

[bwbroersma]

$ yq -r '"container|profiles|description","-|-|-",.services|to_entries|map([.key,(.value.profiles //[]|join(", "),(.key|head_comment))]|join("|"))[]' compose.yaml|sed 's/$/|/'|column -ts\| -o' | '|sed 's/^/| /;s/ $//;2{s/ /-/g}'

using yq, sed and column (maybe also add -W3 -c120 to column call).
Alternatively markdown-table could be used:

$ yq -r '"container|profiles|description",.services|to_entries|map([.key,(.value.profiles //[]|join(", "),(.key|head_comment))]|join("|"))[]' compose.yaml|markdown-table -s\|

container	profiles	description
webserver		nginx proxy container
app		django container
db-migrate		django DB migrations
worker		worker for queue
worker-nassl		worker for queue with potential memory leak
worker-slow		worker for slow and long running tasks that could require a lot of memory (eg: hof update)
beat		celery task queue
redis		redis cache / state
rabbitmq		rabbitmq message-broker
postgres		PostgreSQL database
routinator	routinator	Routinator for RPKI
unbound	connectiontest	unbound DNS server used for connection test
resolver-permissive		unbound resolver used for all DNS queries by app, worker, etc that needs to ignore DNSSEC errors and pass them on to the client
resolver-validating		unbound resolver used for ldns-dane that require DNSSEC validation
cron	cron	cron with periodic tasks
cron-docker	cron	cron daemon with access to Docker socket but no networking
grafana	monitoring	grafana to visualize the monitoring
prometheus	monitoring	prometheus for monitoring
alertmanager	alertmanager	alertmanager for sending monitoring alerts (also requires monitoring profile)
postgresql-exporter	monitoring	provide PostgreSQL metrics for monitoring
redis-exporter	monitoring	provide redis metrics for monitoring
statsd-exporter	monitoring	provide statsd metrics for monitoring
celery-exporter	monitoring	provide celery metrics for monitoring
node-exporter	monitoring	provide node metrics for monitoring
docker_stats_exporter	monitoring	provide docker container metrics for monitoring
nginx_logs_exporter	monitoring	provide nginx metrics for monitoring

Questions:

• Where/when to write this table? Manual make command
• Add table with profile description? No, table view is good enough
• If so, where to make this ideally single profile comment? No

Todo:

• Remove 'basic' descriptions, add stuff about beat (HoF and batch), align on breakable chars
• Add make command
• Rebase on Cleanup profiles #1655

[bwbroersma]

I've also added an auto-make workflow for PRs in:

• Auto make update_container_documentation workflow for PRs #1664

[bwbroersma]

Internet.nl/documentation/Docker-deployment.md

Lines 387 to 393 in 583c017

	### Alerting emails/alertmanager
	A Prometheus Alertmanager service is available but disabled by default. Enabling this will allow you to configure alert emails to be sent whenever the periodic tests fail to complete in a reasonable time, indicating an issue with the application.
	To enable and configure the Alertmanager add the following lines to `docker/local.env` and adjust the values to be applicable for your environment:
	COMPOSE_PROFILES=routinator,alertmanager

Should be updated too.