Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check/Monitor the kcl #25

Open
sagarsk1 opened this issue Aug 27, 2018 · 2 comments
Open

Check/Monitor the kcl #25

sagarsk1 opened this issue Aug 27, 2018 · 2 comments
Labels

Comments

@sagarsk1
Copy link

I am planning to use the kcl in a container to read data from a kinesis stream maintained by a different organization/group.
I do not have access to the streams aws account and cloud watch. Also, will not have write access to the stream.
Is there a way to check/monitor if the kcl is up and working?
Please let me know if you have any ideas on this.
Thanks.

@pfifer
Copy link
Contributor

pfifer commented Aug 27, 2018

In this case you will need to configure the Kinesis client to use the credentials that have access rights to the stream (STS or provided). You will want to configure the DynamoDB and CloudWatch clients to use your own credentials.

If configured the KCL will publish metrics to CloudWatch about the processing work it's doing. The KCL also uses a DynamoDB table to store leases and checkpoints. You can monitor that table to ensure that your application is making progress.

If you're using STS credentials you will need to include an additional jar file, I would recommend using this method

@tomsfernandez
Copy link

tomsfernandez commented Sep 27, 2018

@sagarsk1 Is logging an option?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants