Threading support #4559

aykevl · 2024-10-28T15:23:16Z

Work in progress. Many necessary parts are not yet implemented, but this gives context for PRs I want to make in the future.

See #2446 for motivation (in particular, why this focuses on Linux first) and for a list of things that need to be implemented to make multithreading work in TinyGo.

github-actions · 2024-10-28T15:40:13Z

Size difference with the dev branch:

Binary size difference

not the same command!
    tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/adafruit4650
    go: downloading tinygo.org/x/tinyfont v0.3.0
not the same command!
    tinygo build -size short -o ./build/test.hex -target=pico ./examples/tmc5160/main.go
    go: downloading golang.org/x/exp v0.0.0-20241204233417-43b7b7cde48d
not the same command!
    tinygo build -size short -o ./build/test.hex -target=nano-rp2040 -stack-size 8kb ./examples/net/websocket/dial/
    go: downloading golang.org/x/net v0.33.0
not the same command!
    tinygo build -size short -o ./build/test.hex -target=nano-rp2040 -stack-size 8kb ./examples/net/mqttclient/natiu/
    go: downloading github.com/soypat/natiu-mqtt v0.5.1
not the same command!
    tinygo build -size short -o ./build/test.hex -target=wioterminal -stack-size 8kb ./examples/net/mqttclient/paho/
    go: downloading github.com/eclipse/paho.mqtt.golang v1.2.0
 flash                          ram
 before   after   diff          before   after   diff
 287772  287548   -224  -0.08%   19564   19572      8   0.04% tinygo build -size short -o ./build/test.hex -target=wioterminal -stack-size 8kb ./examples/net/webserver/
 339364  339148   -216  -0.06%   16512   16528     16   0.10% tinygo build -size short -o ./build/test.hex -target=matrixportal-m4 -stack-size 8kb ./examples/net/webstatic/
 291220  291012   -208  -0.07%   20872   20872      0   0.00% tinygo build -size short -o ./build/test.hex -target=wioterminal -stack-size 8kb ./examples/net/mqttclient/paho/
  16760   16760      0   0.00%    4180    4180      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/adafruit4650
  62232   62232      0   0.00%    6188    6188      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adt7410/main.go
   9604    9604      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/adxl345/main.go
  13716   13716      0   0.00%    6796    6796      0   0.00% tinygo build -size short -o ./build/test.hex -target=pybadge ./examples/amg88xx
   9104    9104      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/main.go
  11848   11848      0   0.00%    6580    6580      0   0.00% tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/apds9960/proximity/main.go
   9920    9920      0   0.00%    4760    4760      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/apa102/itsybitsy-m0/main.go
   8376    8376      0   0.00%    2320    2320      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/at24cx/main.go
   8164    8164      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bh1750/main.go
   7468    7468      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/blinkm/main.go
  71436   71436      0   0.00%    3656    3656      0   0.00% tinygo build -size short -o ./build/test.hex -target=pinetime     ./examples/bma42x/main.go
  65092   65092      0   0.00%    6204    6204      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmi160/main.go
  27804   27804      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp180/main.go
  64784   64784      0   0.00%    6228    6228      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/bmp280/main.go
  12248   12248      0   0.00%    4812    4812      0   0.00% tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/bmp388/main.go
   8364    8364      0   0.00%    3352    3352      0   0.00% tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/sram/main.go
  22324   22324      0   0.00%    3556    3556      0   0.00% tinygo build -size short -o ./build/test.hex -target=bluepill ./examples/ds1307/time/main.go
  70480   70480      0   0.00%    6376    6376      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/ds3231/main.go
   4628    4628      0   0.00%    2280    2280      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/easystepper/main.go
  70748   70748      0   0.00%    6980    6980      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/flash/console/spi
  67500   67500      0   0.00%    9020    9020      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/flash/console/qspi
   7308    7308      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/gc9a01/main.go
  67696   67696      0   0.00%    6360    6360      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/gps/i2c/main.go
  68248   68248      0   0.00%    6504    6504      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/gps/uart/main.go
   8012    8012      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/hcsr04/main.go
   5840    5840      0   0.00%    2280    2280      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hd44780/customchar/main.go
   5792    5792      0   0.00%    2280    2280      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hd44780/text/main.go
  10568   10568      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/hd44780i2c/main.go
  14748   14748      0   0.00%    6580    6580      0   0.00% tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/hts221/main.go
  16216   16216      0   0.00%    2364    2364      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/hub75/main.go
  10276   10276      0   0.00%    6924    6924      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/basic
  11180   11180      0   0.00%    4884    4884      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/ili9341/basic
  29644   29644      0   0.00%   38084   38084      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/pyportal_boing
  10300   10300      0   0.00%    6916    6916      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/scroll
  11264   11264      0   0.00%    4876    4876      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/ili9341/scroll
 264116  264116      0   0.00%   46752   46752      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/ili9341/slideshow
  11772   11772      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/lis3dh/main.go
  14112   14112      0   0.00%    6580    6580      0   0.00% tinygo build -size short -o ./build/test.hex -target=nano-33-ble ./examples/lps22hb/main.go
  26500   26500      0   0.00%    2328    2328      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/lsm303agr/main.go
  12484   12484      0   0.00%    4788    4788      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/lsm6ds3/main.go
  10760   10760      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mag3110/main.go
   9940    9940      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp23017/main.go
  10380   10380      0   0.00%    4788    4788      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp23017-multiple/main.go
  10276   10276      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp3008/main.go
  69720   69720      0   0.00%    6204    6204      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mcp2515/main.go
  27180   27180      0   0.00%    3640    3640      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/microbitmatrix/main.go
  27044   27044      0   0.00%    5620    5620      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit-v2 ./examples/microbitmatrix/main.go
   8276    8276      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mma8653/main.go
   8184    8184      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/mpu6050/main.go
  76476   76476      0   0.00%    7452    7452      0   0.00% tinygo build -size short -o ./build/test.hex -target=p1am-100 ./examples/p1am/main.go
  12352   12352      0   0.00%    3360    3360      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/pca9685/main.go
   6268    6268      0   0.00%    3292    3292      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/pcd8544/setbuffer/main.go
   5296    5296      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/pcd8544/setpixel/main.go
  10640   10640      0   0.00%    3336    3336      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/seesaw
   2841    2841      0   0.00%     558     558      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino ./examples/servo
  13888   13888      0   0.00%    3408    3408      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico     ./examples/sgp30
   8212    8212      0   0.00%    6788    6788      0   0.00% tinygo build -size short -o ./build/test.hex -target=pybadge ./examples/shifter/main.go
  58328   58328      0   0.00%    3688    3688      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/sht3x/main.go
  58296   58296      0   0.00%    3696    3696      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/sht4x/main.go
  58296   58296      0   0.00%    3688    3688      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/shtc3/main.go
   6704    6704      0   0.00%    2288    2288      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1306/i2c_128x32/main.go
   6216    6216      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1306/spi_128x64/main.go
   5912    5912      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ssd1331/main.go
   6848    6848      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/st7735/main.go
   6776    6776      0   0.00%    2284    2284      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/st7789/main.go
  17544   17544      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/thermistor/main.go
  10712   10712      0   0.00%    4540    4540      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-bluefruit ./examples/tone
  10208   10208      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/tm1637/main.go
  10916   10916      0   0.00%    3348    3348      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/touch/capacitive
   9648    9648      0   0.00%    6788    6788      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/touch/resistive/fourwire/main.go
  12548   12548      0   0.00%    6984    6984      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal ./examples/touch/resistive/pyportal_touchpaint/main.go
  15372   15372      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/vl53l1x/main.go
  13852   13852      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=itsybitsy-m0 ./examples/vl6180x/main.go
  24796   24796      0   0.00%   13728   13728      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-nrf52840-sense ./examples/waveshare-epd/epd1in54/main.go
   6520    6520      0   0.00%    2324    2324      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd2in13/main.go
   6188    6188      0   0.00%    2316    2316      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd2in13x/main.go
   6440    6440      0   0.00%    2324    2324      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/waveshare-epd/epd4in2/main.go
  26216   26216      0   0.00%   16420   16420      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/waveshare-epd/epd2in66b/main.go
   6988    6988      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/ws2812
   5768    5768      0   0.00%    9522    9522      0   0.00% '-xesppie' is not a recognized feature for this target (ignoring feature)
  63604   63604      0   0.00%    5948    5948      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-nrf52840 ./examples/is31fl3731/main.go
   1581    1581      0   0.00%     598     598      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino   ./examples/ws2812
   1056    1056      0   0.00%     180     180      0   0.00% tinygo build -size short -o ./build/test.hex -target=digispark ./examples/ws2812
  32256   32256      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/bme280/main.go
  17104   17104      0   0.00%    4732    4732      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/microphone/main.go
  11640   11640      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/buzzer/main.go
  12820   12820      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=trinket-m0 ./examples/veml6070/main.go
   6904    6904      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l293x/simple/main.go
   8804    8804      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l293x/speed/main.go
   6872    6872      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l9110x/simple/main.go
   9316    9316      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/l9110x/speed/main.go
   7404    7404      0   0.00%    3324    3324      0   0.00% tinygo build -size short -o ./build/test.hex -target=nucleo-f103rb ./examples/shiftregister/main.go
   6996    6996      0   0.00%    2272    2272      0   0.00% '-xesppie' is not a recognized feature for this target (ignoring feature)
  13372   13372      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=circuitplay-express ./examples/lis2mdl/main.go
   9920    9920      0   0.00%    4764    4764      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/max72xx/main.go
  77828   77828      0   0.00%    6344    6344      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/dht/main.go
  36620   36620      0   0.00%    3988    3988      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/pcf8523/
  72100   72100      0   0.00%    6352    6352      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/alarm/
   7316    7316      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/clkout/
  71620   71620      0   0.00%    6348    6348      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/time/
  72004   72004      0   0.00%    6352    6352      0   0.00% tinygo build -size short -o ./build/test.hex -target=xiao ./examples/pcf8563/timer/
  12296   12296      0   0.00%    3312    3312      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/qmi8658c/main.go
  10916   10916      0   0.00%    3296    3296      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-rp2040 ./examples/pcf8591/
   8788    8788      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/ina260/main.go
  13228   13228      0   0.00%    4780    4780      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m0 ./examples/ina219/main.go
   9392    9392      0   0.00%    5248    5248      0   0.00% tinygo build -size short -o ./build/test.hex -target=nucleo-l432kc ./examples/aht20/main.go
  74524   74524      0   0.00%   10764   10764      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m4 ./examples/sdcard/console/
  62580   62580      0   0.00%    8236    8236      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-m4 ./examples/i2csoft/adt7410/
  10416   10416      0   0.00%    6788    6788      0   0.00% tinygo build -size short -o ./build/test.elf -target=wioterminal ./examples/axp192/m5stack-core2-blinky/
   9124    9124      0   0.00%    3284    3284      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/xpt2046/main.go
  13260   13260      0   0.00%    4936    4936      0   0.00% tinygo build -size short -o ./build/test.hex -target=nucleo-wl55jc ./examples/sx126x/lora_rxtx/
  31392   31392      0   0.00%    4552    4552      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/ssd1289/main.go
  11304   11304      0   0.00%    4260    4260      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/irremote/main.go
  12188   12188      0   0.00%    3332    3332      0   0.00% tinygo build -size short -o ./build/test.hex -target=badger2040 ./examples/uc8151/main.go
  10540   10540      0   0.00%    3364    3364      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/scd4x/main.go
   8700    8700      0   0.00%    4748    4748      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=circuitplay-express ./examples/makeybutton/main.go
   9708    9708      0   0.00%    4764    4764      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ds18b20/main.go
  83348   83348      0   0.00%    6592    6592      0   0.00% tinygo build -size short -o ./build/test.hex -target=nucleo-wl55jc ./examples/lora/lorawan/atcmd/
  15928   15928      0   0.00%    4876    4876      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/as560x/main.go
   9984    9984      0   0.00%    3304    3304      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/mpu6886/main.go
   7948    7948      0   0.00%    4740    4740      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ttp229/main.go
  67656   67656      0   0.00%    4816    4816      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/ndir/main_ndir.go
  63024   63024      0   0.00%    3784    3784      0   0.00% tinygo build -size short -o ./build/test.hex -target=microbit ./examples/ndir/main_ndir.go
  66092   66092      0   0.00%    6260    6260      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/ndir/main_ndir.go
   9416    9416      0   0.00%    3296    3296      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/mpu9150/main.go
  11676   11676      0   0.00%    3332    3332      0   0.00% tinygo build -size short -o ./build/test.hex -target=macropad-rp2040 ./examples/sh1106/macropad_spi
   8532    8532      0   0.00%    3760    3760      0   0.00% tinygo build -size short -o ./build/test.hex -target=macropad-rp2040 ./examples/encoders/quadrature-interrupt
  67080   67080      0   0.00%    4784    4784      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/mcp9808/main.go
  30408   30408      0   0.00%    3680    3680      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=pico ./examples/tmc2209/main.go
  13376   13376      0   0.00%    3292    3292      0   0.00% tinygo build -size short -o ./build/test.hex -target=pico ./examples/tmc5160/main.go
  12500   12500      0   0.00%    4552    4552      0   0.00% tinygo build -size short -o ./build/test.uf2 -target=nicenano ./examples/sharpmem/main.go
  61140   61140      0   0.00%    5968    5968      0   0.00% tinygo build -size short -o ./build/test.hex -target=feather-nrf52840 ./examples/max6675/main.go
  87020   87020      0   0.00%    5148    5148      0   0.00% tinygo build -size short -o ./build/test.hex -target=challenger-rp2040 ./examples/net/ntpclient/
 298784  298784      0   0.00%   15908   15908      0   0.00% tinygo build -size short -o ./build/test.hex -target=pyportal -stack-size 8kb ./examples/net/http-get/
 119828  119828      0   0.00%    7848    7848      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-nano33 -stack-size 8kb ./examples/net/tcpclient/
 103404  103404      0   0.00%    9816    9816      0   0.00% tinygo build -size short -o ./build/test.hex -target=metro-m4-airlift -stack-size 8kb ./examples/net/socket/
 111820  111820      0   0.00%    7784    7784      0   0.00% tinygo build -size short -o ./build/test.hex -target=arduino-mkrwifi1010 -stack-size 8kb ./examples/net/tlsclient/
 154624  154624      0   0.00%    6548    6548      0   0.00% tinygo build -size short -o ./build/test.hex -target=nano-rp2040 -stack-size 8kb ./examples/net/mqttclient/natiu/
 116892  116892      0   0.00%   13124   13124      0   0.00% tinygo build -size short -o ./build/test.hex -target=wioterminal -stack-size 8kb ./examples/net/webclient/
 118744  118744      0   0.00%    9504    9504      0   0.00% tinygo build -size short -o ./build/test.hex -target=elecrow-rp2040 -stack-size 8kb ./examples/net/tlsclient/
 118636  118636      0   0.00%    9564    9564      0   0.00% tinygo build -size short -o ./build/test.hex -target=elecrow-rp2350 -stack-size 8kb ./examples/net/ntpclient/
 251596  251756    160   0.06%   12868   12868      0   0.00% tinygo build -size short -o ./build/test.hex -target=nano-rp2040 -stack-size 8kb ./examples/net/websocket/dial/
5740806 5740318   -488  -0.00%  857586  857610     24   0.00%

aykevl · 2024-11-02T10:22:36Z

Updated after some refactoring (moved internal/llsync into internal/task) and added GC support (stop-the-world scanning by pausing all threads and scanning their stacks).
Main issues remaining are:

Some problem with bits/fcntl.h that I need to investigate (see CI failures).
Timer support.
Fixing tests that assume a deterministic order in which goroutines are run.
Fixing signal support.

src/internal/task/futex_linux.c

Previously the assembler was reordering this code: jal tinygo_scanstack move $a0, $sp Into this: jal tinygo_scanstack nop move $a0, $sp So it was "helpfully" inserting a branch delay slot, even though this was already being taken care of. Somehow this didn't break, but it does break in the WIP threading branch (#4559) where this bug leads to a crash.

aykevl · 2024-12-01T13:26:21Z

Whoa, looks like all tests are passing!

jclab-joseph · 2024-12-02T08:27:40Z

@aykevl Awesome! Thanks for your hard work

aykevl · 2025-01-13T16:11:06Z

I don't think there's a whole lot more I can extract and make separate PRs for, so let's mark this as ready for review!

eliasnaur

Very much look forward to this work!

src/internal/task/task_threads.c

eliasnaur · 2025-01-15T08:57:20Z

src/internal/task/task_threads.go

+	}
+	t.state.QueueNext = activeTasks
+	activeTasks = t
+	activeTaskLock.Unlock()


Nit: defer the unlock?

I avoid defer in places like these since:

defer can cause heap allocations in some cases (mainly in loops, so not really relevant here, but it's not very visible).

Using defer currently makes every call in the function slower and larger, since it needs to be able to recover.

We don't need the run-defer-on-panic behavior here: if we panic, something is very wrong and it doesn't help to unlock.

Admittedly it could be used here, but in many other places (e.g. GCScan) it can lead to unexpected side effects if it allocates.

So in short, defer as it is currently implemented is inefficient and can have unexpected side effects so I avoid its use in the runtime.

(Same goes for all the other comments regarding defer).

eliasnaur · 2025-01-15T08:59:09Z

src/internal/task/task_threads.go

+	}
+
+	// Allow goroutines to start and exit again.
+	activeTaskLock.Unlock()


Nit: defer?

eliasnaur · 2025-01-15T09:05:47Z

src/internal/task/task_threads.go

+// GC scan phase. Because we need to stop the world while scanning, this kinda
+// needs to be done in the tasks package.


FWIW, I find this structure backwards. I would espect the package runtime to have GCScan that then calls into this package task. Especially in light of runtime.gcScanGlobals and runtime.scanStack.

I agree with you, and I first tried to put the equivalent of GCScan in the runtime. But GCScan also needs access to various private symbols like activeTaskLock, activeTasks, scanDoneFutex, etc so to avoid exporting many private details of the tasks package I put it there.

I can however make scanCurrentStack and gcScanGlobals callbacks, so they don't directly call back to the runtime. But then I don't know what to do with scanCurrentStack in tinygo_task_gc_pause. In the end, the GC and the scheduler do need some tight integration, I can't see a way around that to be honest.

eliasnaur · 2025-01-15T09:12:32Z

src/runtime/gc_blocks.go

 	runGC()
+	gcLock.Unlock()


Nit: defer?

eliasnaur · 2025-01-15T09:13:22Z

src/runtime/gc_blocks.go

@@ -733,6 +743,7 @@ func ReadMemStats(m *MemStats) {
 	m.Sys = uint64(heapEnd - heapStart)
 	m.HeapAlloc = (gcTotalBlocks - gcFreedBlocks) * uint64(bytesPerBlock)
 	m.Alloc = m.HeapAlloc
+	gcLock.Unlock()


Nit: defer?

b0ch3nski · 2025-01-15T11:59:45Z

src/internal/task/task_threads.go

+	QueueNext *Task
+
+	// Semaphore to pause/resume the thread atomically.
+	pauseSem Semaphore


Shouldn't Semaphore be used with pointer receiver?

No. The state struct (or actually the Task struct where it is embedded) will always be accessed via a pointer, like *Task. So essentially when calling a function like pauseSem.Post(), what happens is something like SemaphorePost(&t.state.pauseSem).

Also, I think you mean to say that pauseSem should have the type *Semaphore? Since the methods on the Semaphore type do in fact have a pointer receiver:

func (s *Semaphore) Post() { ... } func (s *Semaphore) Wait() { ... }

My point was about a pointer (pun intended 😄) on *Semaphore. While Semaphore methods do have pointer receiver, the instances of pauseSem and gcSem members would get copied when state struct instance is ever passed around.

I've created playground snippet which uses stdlib sync.Mutex as an example to even trigger go vet warnings: https://go.dev/play/p/cV7h16_JS1R

While Semaphore methods do have pointer receiver, the instances of pauseSem and gcSem members would get copied when state struct instance is ever passed around.

That's why it should never be copied around!
Because it's an internal API, we can make sure this is the case. (And it is).

src/runtime/scheduler.go

src/sync/mutex.go

aykevl · 2025-01-21T09:49:52Z

Updated the PR after some review comments. And rebased on dev to incorporate #4435.

Using a global lock may be slow, but it is certainly simple and safe. If this global lock becomes a bottleneck, we can of course look into making the GC truly support multithreading.

Move common functions to scheduler.go. They will be used both from the cooperative and from the threads scheduler.

Somewhat surprisingly, this results in smaller code than the old code with the cooperative (tasks) scheduler. Probably because the new RWMutex is also simpler.

This is not a scheduler in the runtime, instead every goroutine is mapped to a single OS thread - meaning 1:1 scheduling. While this may not perform well (or at all) for large numbers of threads, it greatly simplifies many things in the runtime. For example, blocking syscalls can be called directly instead of having to use epoll or similar. Also, we don't need to do anything special to call C code - the default stack is all we need.

aykevl · 2025-03-29T10:13:35Z

Rebased (which was a bit of work after #4385 got merged), let's see whether CI is happy with this.

This was referenced Oct 28, 2024

Prepare leaking GC for multithreading #4560

Merged

runtime: move scheduler code around #4549

Merged

Multicore support #2446

Open

aykevl mentioned this pull request Oct 28, 2024

sync: don't use volatile in Mutex #4558

Merged

aykevl force-pushed the threading branch 2 times, most recently from bcee341 to 28f51ca Compare November 2, 2024 10:09

aykevl force-pushed the threading branch 3 times, most recently from 0a39dc9 to d328dd4 Compare November 4, 2024 09:18

aykevl mentioned this pull request Nov 4, 2024

net: enable native golang linux networking #4498

Open

leongross reviewed Nov 6, 2024

View reviewed changes

src/internal/task/futex_linux.c Outdated Show resolved Hide resolved

src/internal/task/futex_linux.c Outdated Show resolved Hide resolved

src/internal/task/futex_linux.c Outdated Show resolved Hide resolved

aykevl mentioned this pull request Nov 7, 2024

Implement race-free signals using futexes #4586

Merged

aykevl force-pushed the threading branch from d328dd4 to 79c8f8b Compare November 30, 2024 08:03

This was referenced Nov 30, 2024

runtime: make signals parallelism-safe #4641

Merged

Prepare channels for parallelism #4642

Merged

aykevl force-pushed the threading branch from 79c8f8b to 8d67ed4 Compare November 30, 2024 09:51

aykevl mentioned this pull request Dec 1, 2024

mips: fix a bug when scanning the stack #4644

Merged

aykevl force-pushed the threading branch from 8d67ed4 to 9413785 Compare December 1, 2024 08:39

aykevl mentioned this pull request Dec 1, 2024

runtime: lock output in print/println #4645

Merged

aykevl force-pushed the threading branch 3 times, most recently from d8cf15f to 0ca0db4 Compare December 1, 2024 12:30

aykevl force-pushed the threading branch from 0ca0db4 to 6894f62 Compare December 6, 2024 13:57

aykevl mentioned this pull request Dec 6, 2024

sync: implement futex-based Mutex #4650

Closed

aykevl force-pushed the threading branch 2 times, most recently from 3d77025 to 5823015 Compare January 13, 2025 16:04

aykevl marked this pull request as ready for review January 13, 2025 16:09

aykevl changed the title ~~WIP: threading support~~ Threading support Jan 13, 2025

aykevl force-pushed the threading branch from 5823015 to 2e15613 Compare January 14, 2025 10:44

jclab-joseph mentioned this pull request Jan 15, 2025

Draft: threads scheduler #4047

Closed

8 tasks

eliasnaur reviewed Jan 15, 2025

View reviewed changes

b0ch3nski reviewed Jan 15, 2025

View reviewed changes

leongross reviewed Jan 15, 2025

View reviewed changes

src/runtime/scheduler.go Outdated Show resolved Hide resolved

src/sync/mutex.go Show resolved Hide resolved

aykevl force-pushed the threading branch from 2e15613 to 3767dec Compare January 21, 2025 09:49

aykevl mentioned this pull request Jan 28, 2025

Add support for threading on MacOS #4720

Draft

aykevl mentioned this pull request Mar 18, 2025

stdlib/sync: Match Mutex size with upstream Go #4815

Open

aykevl added 5 commits March 29, 2025 11:11

internal/task: implement atomic primitives for preemptive scheduling

b64375a

runtime: make conservative and precise GC MT-safe

b0f90fe

Using a global lock may be slow, but it is certainly simple and safe. If this global lock becomes a bottleneck, we can of course look into making the GC truly support multithreading.

runtime: refactor timerQueue

7ddfef2

Move common functions to scheduler.go. They will be used both from the cooperative and from the threads scheduler.

sync: implement RWMutex using futexes

8d68d01

Somewhat surprisingly, this results in smaller code than the old code with the cooperative (tasks) scheduler. Probably because the new RWMutex is also simpler.

aykevl force-pushed the threading branch from 3767dec to c55a11b Compare March 29, 2025 10:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Threading support #4559

Threading support #4559

aykevl commented Oct 28, 2024

github-actions bot commented Oct 28, 2024 •

edited

Loading

aykevl commented Nov 2, 2024 •

edited

Loading

aykevl commented Dec 1, 2024

jclab-joseph commented Dec 2, 2024

aykevl commented Jan 13, 2025

eliasnaur left a comment •

edited

Loading

eliasnaur Jan 15, 2025

aykevl Jan 21, 2025

eliasnaur Jan 15, 2025

eliasnaur Jan 15, 2025

aykevl Jan 21, 2025

eliasnaur Jan 15, 2025

eliasnaur Jan 15, 2025

b0ch3nski Jan 15, 2025

aykevl Jan 21, 2025

b0ch3nski Jan 21, 2025

aykevl Jan 30, 2025

aykevl commented Jan 21, 2025

aykevl commented Mar 29, 2025

		// GC scan phase. Because we need to stop the world while scanning, this kinda
		// needs to be done in the tasks package.

Threading support #4559

Are you sure you want to change the base?

Threading support #4559

Conversation

aykevl commented Oct 28, 2024

github-actions bot commented Oct 28, 2024 • edited Loading

aykevl commented Nov 2, 2024 • edited Loading

aykevl commented Dec 1, 2024

jclab-joseph commented Dec 2, 2024

aykevl commented Jan 13, 2025

eliasnaur left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aykevl commented Jan 21, 2025

aykevl commented Mar 29, 2025

github-actions bot commented Oct 28, 2024 •

edited

Loading

aykevl commented Nov 2, 2024 •

edited

Loading

eliasnaur left a comment •

edited

Loading