Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[K8s] Problems with tables creation and "Timeout expired while fetching topic metadata" #132

Open
pulnara opened this issue May 7, 2021 · 5 comments

Comments

@pulnara
Copy link

pulnara commented May 7, 2021

Im running this application in Kubernetes (GKE cluster). When trying to create a restaurant in Swagger UI, I'm getting the following error:
image

When I log in to the ftgo-mysql service, I can see that the table message doesn't exist indeed.
image

So I've added it (and some others lacking) manually, using the contents of ./mysql/template:
image

After that, adding a restaurant works:
image

But the Order Service isn't able to see the new restaurant. After looking into the ftgo-cdc-service logs, it turned out that we're still lacking a table, this time eventuate.cdc_monitoring. So once again, I created it manually using the contents of ./mysql/common-schema.sql.

After that, I've looked into the ftgo-cdc-service logs once again, and got information about successful reconnection to database. However, now I'm getting a problem about fetching information from the Kafka topic:

10:21:44.485 [Timer-1484] INFO  io.eventuate.local.common.DaoUtils - Reconnected to database
Fri May 07 10:21:50 UTC 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
10:22:17.747 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
10:22:57.847 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
10:23:37.947 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

It seems that there's some problem with Kafka configuration. I also guess the automatic creation of all the tables should be fixed.

@eedizy
Copy link

eedizy commented May 7, 2021

I have the exact same issue

@pulnara
Copy link
Author

pulnara commented May 12, 2021

@cer could you please let us know how you've overcome these issues in your deployment?

@pulnara
Copy link
Author

pulnara commented May 12, 2021

Okay, it seems I've fixed this problem by changing this port: https://github.com/microservices-patterns/ftgo-application/blob/master/deployment/kubernetes/cdc-service/ftgo-cdc-service.yml#L40 to 9092. Now the error about Kafka doesn't appear in the ftgo-cdc-service. However, the CDC Service still seems to not post any data into the topic.

@pulnara
Copy link
Author

pulnara commented May 13, 2021

I've provided the fix for this problem in #131. Looks like the problem laid in database misconfigurations. The application appears to work correctly now, I am able to create a customer, restaurant and order.

@cer
Copy link
Contributor

cer commented May 25, 2021

@pulnara It's great that you fixed this. Sorry for not responding - I had a crazy busy couple of weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants