Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POD restarts when applying wasm deploy #104

Open
sriramcms opened this issue Apr 9, 2020 · 4 comments · May be fixed by #95
Open

POD restarts when applying wasm deploy #104

sriramcms opened this issue Apr 9, 2020 · 4 comments · May be fixed by #95

Comments

@sriramcms
Copy link

Hi:

I went through the sample tutorial to add a wasm filter to the bookinfo sample in istio service mesh 1.5.1.
https://docs.solo.io/web-assembly-hub/latest/tutorial_code/

Does the wasm deploy force a pod restart while applying the filter. Is this observation correct ? I see the old pods terminating and new ones getting created -- for example if i am applying this on the details pod in the bookinfo example.
@yuval-k
Copy link
Member

yuval-k commented Apr 9, 2020 •
edited
Loading

it is for now as we need to apply some annotations to the pod, to instruct istio to add some volumes to it.
once #95 is merged, this will no longer be the case. no annotations will be needed, and consequently no pod restarts.

@sriramcms
Copy link
Author

Super. Thanks for the prompt response. Will wait for the new release.

@sriramcms
Copy link
Author

sriramcms commented Apr 19, 2020 •
edited
Loading

Just a FYI on what i observed -- istio 1.5.1 , minikube and kubernetes 1.17.3

A few more observation on constant POD restarts due to constantly applying newer versions of the wasm filters

  1. The kubernetes API server stops responding and to fix it I end up doing a minikube stop and start to get a working cluster again
  2. If for example I am applying this on the petstore app

First time when i apply wasme deploy to petstore - hangs and does not succeed

INFO[0003] added image to cache config... cache="{wasme-cache wasme}" image="webassemblyhub.io/sriramcm/demo-add-header:v0.7"
INFO[0003] waiting for event with timeout 1m0s
WARN[0004] event err: expected 1 image-ready events for image webassemblyhub.io/sriramcm/demo-add-header:v0.7, only found 0
WARN[0005] event err: expected 1 image-ready events for image webassemblyhub.io/sriramcm/demo-add-header:v0.7, only found 0

petstore istio - proxy logs looks like below

s Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2020-04-18T19:11:02.222499Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2020-04-18T19:11:04.223060Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2020-04-18T19:11:06.222974Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1
I sometimes noticed a invalid wasm cache location like Invalid Path : /var/lib/cache/3456yshh.....

i need to kill it and then run wasme deploy again to succeed
Also a few wasme undeploys and deploys to get a successful POD with filter --

Basically i found sometimes it was erratic. Overall a few minutes for me to get a filter applied from a successful push to webassemblyhub.io to a pull on the kubernetes cluster followed by deploy.

@solo-changelog-bot solo-changelog-bot bot linked a pull request Apr 28, 2020 that will close this issue
Draft
@Sodman
Copy link
Member

Sodman commented Oct 29, 2020

Related to #67

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix cache race status solo-io/wasm
3 participants
@yuval-k @Sodman @sriramcms