fix(consumer): add recovery from no leader partitions

[liutao365]

When some topic partitions have no leader due to Kafka broker failures, the Sarama consumer group should be able to continue consuming partitions that have leaders and resume consuming the partitions that previously had no leader once they return to normal. This pull request addresses this issue.

[puellanivis]

👍 nothing significant I can see that should be improved. While there is a suggestion here, I don’t think it’s necessarily any better than the existing approach.

[liutao365]

Add test case:

1. create a more than 6 partitions and 1 replication-factor (simplify the test case) topic and produce some millions messages .
2. shut down one of the kafka brokers, which will cause some of the topic partitions lose leader.
3. run the expamples/consumergroup/main.go and add a configuration
   
4. restore the broker previously shut down.
5. describe the consumergroup and check all the partittions have no consumer lag

[dnwe]

@liutao365 thanks for proposing this change, the approach looks good to me – I added a commit to fixup the client locking (the FV was failing for the race condition) and another commit to slightly refactor the consume partition code into a single named func to cover both paths. Can you take a look and confirm you're happy? Also @puellanivis if you wouldn't mind re-reviewing that would be great too.

We should probably add a unittest to cover this scenario, but I'm happy for us to do that under a follow-up PR as it is probably worth landing this fix sooner rather than later

[liutao365]

@liutao365 thanks for proposing this change, the approach looks good to me – I added a commit to fixup the client locking (the FV was failing for the race condition) and another commit to slightly refactor the consume partition code into a single named func to cover both paths. Can you take a look and confirm you're happy? Also @puellanivis if you wouldn't mind re-reviewing that would be great too.

We should probably add a unittest to cover this scenario, but I'm happy for us to do that under a follow-up PR as it is probably worth landing this fix sooner rather than later

Hi, @dnwe , I've red the changes and its better than my original commit, I'm happy with this improvements.

[prestona]

LGTM!