Implement stack overflow protection for webassembly

[markshannon]

Implementing stack overflow protection for webassembly is tricky, as there are two stacks:

1. The hidden webassembly stack
2. The stack used for C stack objects that can have their address taken

We need to avoid overflowing either. It generally seems that the first stack is the one most vulnerable to overflow, so perhaps a simple counter would work.

@brettcannon
@hoodmane

[hoodmane]

If the spilled stack ever overflows, we can just recompile and make it bigger whereas the wasm stack size is determined by the runtime and can't easily be made bigger. At least for Emscripten run in the browser. Perhaps with wasmtime we can easily ask the runtime to make the stack bigger.

[markshannon]

This is about runtime detection of stack limits, rather than increasing the stack size.

Is it possible to get the size of the wasm stack from the browser or wasmtime?
Also is it possible to get the current stack depth?

Alternatively, is it possible to catch a stack overflow in JS? We could maybe write a JS helper function to probe the stack.

[hoodmane]

It's possible to catch a js stack overflow. I've used the probe approach before but unfortunately the ratio of the number of js stack frames it takes to overflow vs the number of wasm frames is inconsistent between V8 and spider monkey. They have similar apparent wasm stack sizes but spider monkey can handle many more nested js frames than V8 iirc.

[hoodmane]

I think in node there are ways to query the stack size and depth but it is not part of the web standard and I think browsers don't include any such APIs unless you start the browser in a debug mode.