fix(prometheus): prometheus cannot connect to sig's metrics #531
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sobeston
reviewed
Feb 3, 2025
InKryption
approved these changes
Feb 3, 2025
dadepo
pushed a commit
that referenced
this pull request
Feb 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, our prometheus server does not work. It has some issue with the way it responds to http requests. curl is able to get data from it, but prometheus itself is unable to connect to the server. I've tested this on my macbook and three linux boxes. Prometheus always has an error connecting to sig, likewise grafana doesn't get any data.
This is a big problem because it makes us blind to sig's internals while it's running. Manual end-to-end tests of things like gossip and the shred network are not practical until this is fixed, which means other tasks are being blocked by this bug.
This problem was introduced in #520 (remove httpz), so I've reverted the portion of that change which is relevant to prometheus. I haven't made any tweaks to the code, it's just a simple
git revert.After reverting this change, prometheus connects as expected, and I can observe the data in grafana.
I'm in favor of minimizing dependencies, but we just need a working prometheus server ASAP. I'd be happy to see another PR come in after this one to remove httpz again, replacing it with an http server that is compatible with prometheus.