Unable to recover from KMS signer disconnect

[mdyring]

When using a remote signer (KMS), it seems the Tendermint version used does not properly re-establish connection in case of an error.

May 03 15:21:08 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:08.349] CONSENSUS FAILURE!!!                         module=consensus err=EOF stack="goroutine 398 [running]:\nruntime/debug.Stack(0xc0065a0760, 0x3ff0000000000000, 0x2)\n\t/usr/lib/go-1.11/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc001302a80, 0x11df880)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:584 +0x136\npanic(0xf191e0, 0xc000092030)\n\t/usr/lib/go-1.11/src/runtime/panic.go:513 +0x1b9\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/privval.(*RemoteSignerClient).GetAddress(0xc000aca520, 0x0, 0x0, 0x0)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/privval/remote_signer.go:39 +0x91\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterPropose(0xc001302a80, 0xc6887, 0x0)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:847 +0x9b5\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterNewRound(0xc001302a80, 0xc6887, 0x0)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:796 +0x76e\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleTimeout(0xc001302a80, 0x12498ebb0, 0xc6887, 0x0, 0x1, 0xc6887, 0x0, 0x1, 0x137eb18a, 0xed45e5364, ...)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:707 +0x526\ngithub.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc001302a80, 0x0)\n\t/root/gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:629 +0x3f4\ncreated by github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart\n\t/root/
May 03 15:21:08 i-059e9c680130ef5b1 iris[19142]: gopath/src/github.com/irisnet/irishub/vendor/github.com/tendermint/tendermint/consensus/state.go:306 +0x132\n"
May 03 15:21:08 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:08.530] Ping                                         module=privval err="remote signer timed out"
May 03 15:21:10 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:10.530] Ping                                         module=privval err="remote signer timed out"
May 03 15:21:12 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:12.530] Ping                                         module=privval err="remote signer timed out"
May 03 15:21:14 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:14.530] Ping                                         module=privval err="remote signer timed out"
May 03 15:21:16 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:16.530] Ping                                         module=privval err="remote signer timed out"
May 03 15:21:18 i-059e9c680130ef5b1 iris[19142]: E[3056-05-03|15:21:18.530] Ping                                         module=privval err="remote signer timed out"

This is handled gracefully in more recent Tendermint versions, would it be able to roll a new release that include these changes?

[kwunyeung]

This is related cosmos/cosmos-sdk#3190. Tendermint not sending sign request after the KMS restart.

[haifengxi]

Replaced by: #1429, #1430

[shilei-wang]

verifies the problem is fixed