(support) remote connection with jetbrains gateway doesn't work, vscode does work

[danmur]

I'm self-hosting on k8s and using a k8s template. Gateway connects to my coder instance successfully, it can download the IDE I select, but when I hit "Start IDE and connect" I get the "Connecting to coder workspace..." modal for a few seconds then the dialog disappears, no user-visible errors.

I get this in the console from Gateway:

❯ gateway
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.
2025-03-03 10:32:41,212 [  28256]   WARN - #c.i.r.d.CodeWithMeClientDownloader - Running client process failed after specified number of attempts
2025-03-03 10:32:42,217 [  29261] SEVERE - CoderRemoteConnectionHandle - Failed to connect (will not retry)
java.lang.Exception: IU-243.25659.39 client was closed
	at com.coder.gateway.CoderRemoteConnectionHandle$doConnect$3$2.invoke(CoderRemoteConnectionHandle.kt:313)
	at com.coder.gateway.CoderRemoteConnectionHandle$doConnect$3$2.invoke(CoderRemoteConnectionHandle.kt:309)
	at com.jetbrains.rd.util.reactive.Signal.fire(Signal.kt:32)
	at com.jetbrains.gateway.thinClientLink.ThinClientHandle.close(ThinClientHandle.kt:263)
	at com.jetbrains.gateway.thinClientLink.ThinClientHandle.onProcessTermination(ThinClientHandle.kt:187)
	at com.jetbrains.gateway.thinClientLink.ThinClientHandle._init_$lambda$3(ThinClientHandle.kt:102)
	at com.jetbrains.rd.util.reactive.Signal.fire(Signal.kt:32)
	at com.jetbrains.rd.util.reactive.InterfacesKt.fire(Interfaces.kt:163)
	at com.jetbrains.gateway.thinClientLink.SingleClientLifetimeTracker.registerClientProcessLifetime$lambda$0(SingleClientLifetimeTracker.kt:22)
	at com.intellij.remoteDev.util.LifetimeExKt.onTerminationOrNow$lambda$0(LifetimeEx.kt:10)
	at com.jetbrains.rd.util.lifetime.LifetimeDefinition.destruct(RLifetime.kt:501)
	at com.jetbrains.rd.util.lifetime.LifetimeDefinition.terminate(RLifetime.kt:480)
	at com.jetbrains.rd.util.lifetime.LifetimeDefinition.terminate$default(RLifetime.kt:447)
	at com.intellij.remoteDev.downloader.CodeWithMeClientDownloader$runJetBrainsClientProcess$doRunProcess$listener$1.processTerminated$lambda$2(CodeWithMeClientDownloader.kt:874)
	at com.intellij.openapi.application.TransactionGuardImpl$1.run(TransactionGuardImpl.java:201)
	at com.intellij.openapi.application.impl.AnyThreadWriteThreadingSupport.runIntendedWriteActionOnCurrentThread$lambda$2(AnyThreadWriteThreadingSupport.kt:217)
	at com.intellij.openapi.application.impl.AnyThreadWriteThreadingSupport.runWriteIntentReadAction(AnyThreadWriteThreadingSupport.kt:128)
	at com.intellij.openapi.application.impl.AnyThreadWriteThreadingSupport.runIntendedWriteActionOnCurrentThread(AnyThreadWriteThreadingSupport.kt:216)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:843)
	at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:421)
	at com.intellij.util.concurrency.ChildContext$runInChildContext$1.invoke(propagation.kt:103)
	at com.intellij.util.concurrency.ChildContext$runInChildContext$1.invoke(propagation.kt:103)
	at com.intellij.util.concurrency.ChildContext.runInChildContext(propagation.kt:109)
	at com.intellij.util.concurrency.ChildContext.runInChildContext(propagation.kt:103)
	at com.intellij.util.concurrency.ContextRunnable.run(ContextRunnable.java:27)
	at com.intellij.openapi.application.impl.FlushQueue.runNextEvent(FlushQueue.java:117)
	at com.intellij.openapi.application.impl.FlushQueue.flushNow(FlushQueue.java:43)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:781)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:728)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:722)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:87)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:750)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:675)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:573)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$18$lambda$17$lambda$16$lambda$15(IdeEventQueue.kt:355)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:857)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$18$lambda$17$lambda$16(IdeEventQueue.kt:354)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$2$lambda$1(IdeEventQueue.kt:1045)
	at com.intellij.openapi.application.WriteIntentReadAction.lambda$run$0(WriteIntentReadAction.java:24)
	at com.intellij.openapi.application.impl.AnyThreadWriteThreadingSupport.runWriteIntentReadAction(AnyThreadWriteThreadingSupport.kt:128)
	at com.intellij.openapi.application.impl.ApplicationImpl.runWriteIntentReadAction(ApplicationImpl.java:917)
	at com.intellij.openapi.application.WriteIntentReadAction.compute(WriteIntentReadAction.java:55)
	at com.intellij.openapi.application.WriteIntentReadAction.run(WriteIntentReadAction.java:23)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$2(IdeEventQueue.kt:1045)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$3(IdeEventQueue.kt:1054)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:109)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:1054)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$18(IdeEventQueue.kt:349)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:395)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

and on the workspace side the logs from .cache/JetBrains/RemoteDev/dist/IU-243.25659.39/bin/remote-dev-server:

Current pid: 245016
2025-03-03 02:38:11,505 [ 962414]   WARN - #c.j.r.z.j - RD links are already set, and would be overridden

> Link help:
> bin/remote-dev-server.sh run <path/to/proj> --ssh-link-host <host> --ssh-link-user <user> --ssh-link-port <port>

Hostname is not provided, using: 'coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb'. You can specify it manually with '--ssh-link-host' option
Username is not provided, using: 'coder'. You can specify it manually with '--ssh-link-user' option
Ssh port is not provided, using: '22'. You can specify it manually with '--ssh-link-port' option


*********************************************************

Join link: tcp://127.0.0.1:5990#jt=7491ce95-c854-428b-aa59-8aa3c2acfcc2&p=IU&fp=FB0D786D7D6A2FB69ACD551303B7C0C1FA96D3BB43A23873C356190D0A49EF7C&cb=243.25659.39&newUi=true&jb=21.0.6b631.39

Http link: https://code-with-me.jetbrains.com/remoteDev#idePath=%2Fhome%2Fcoder%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2FIU-243.25659.39&projectPath=%2Fhome&host=coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb&port=22&user=coder&type=ssh&deploy=false&newUi=true

Gateway link: jetbrains-gateway://connect#idePath=%2Fhome%2Fcoder%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2FIU-243.25659.39&projectPath=%2Fhome&host=coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb&port=22&user=coder&type=ssh&deploy=false&newUi=true

*********************************************************

2025-03-03 02:38:11,590 [ 962499]   WARN - #c.j.r.u.UnattendedHostStatusStarter - Command cwmHostStatus is depreated. Use remoteDevStatus instead
STATUS:
{
  "appPid": 245016,
  "appVersion": "IU-243.25659.39",
  "runtimeVersion": "21.0.6b631.39",
  "unattendedMode": true,
  "backendUnresponsive": false,
  "modalDialogIsOpened": true,
  "idePath": "/home/coder/.cache/JetBrains/RemoteDev/dist/IU-243.25659.39",
  "ideIdentityString": "/home/coder/.config/JetBrains/IntelliJIdea2024.3",
  "joinLink": "tcp://127.0.0.1:5990#jt=7491ce95-c854-428b-aa59-8aa3c2acfcc2&p=IU&fp=FB0D786D7D6A2FB69ACD551303B7C0C1FA96D3BB43A23873C356190D0A49EF7C&cb=243.25659.39&newUi=true&jb=21.0.6b631.39",
  "httpLink": "https://code-with-me.jetbrains.com/remoteDev#idePath=%2Fhome%2Fcoder%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2FIU-243.25659.39&projectPath=%2Fhome&host=coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb&port=22&user=coder&type=ssh&deploy=false&newUi=true",
  "gatewayLink": "jetbrains-gateway://connect#idePath=%2Fhome%2Fcoder%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2FIU-243.25659.39&projectPath=%2Fhome&host=coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb&port=22&user=coder&type=ssh&deploy=false&newUi=true"
}
-^C2025-03-03 02:46:26,833 [1457742]   WARN - #c.i.o.v.i.l.NativeFileWatcherImpl - Watcher terminated with exit code 130

I'm wondering if the host should be set somewhere there? I have a CODER_ACCESS_URL set and that's what I'm connecting to but the host parameter in those URLs doesn't seem to include it.

Any ideas, or other logs I can provide?

Thank you :)

[danmur]

Versions:

$ gateway --version
JetBrains Gateway 2024.3.3
Build #GW-243.24978.56

coder@coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb:~$ coder --version
Coder v2.19.0+2f32b11 Tue Feb  4 07:53:51 UTC 2025
https://github.com/coder/coder/commit/2f32b11831a236ed550f35b2296b8f948eb64f4e

Slim build of Coder, does not support the  server  subcommand.

and using the helm chart, 2.19.0

[matifali]

Hi @danmur can you try running coder config-ssh and then use ssh to connect to the coder.workspacw-name using the SSH method.
This will let us isolate the issue.

[danmur]

Hey! Thanks for the prompt reply :). Here's the ssh config it added (I didn't check it before so I don't know if the CODER JETBRAINS stuff was there already, maybe I should remove it and try it all again):

❯ cat /home/daniel/.ssh/config
# --- START CODER JETBRAINS cd.sl.darkdisco.net
Host coder-jetbrains-cd.sl.darkdisco.net--*
  ProxyCommand /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/coder-linux-amd64 --global-config /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/config --url https://cd.sl.darkdisco.net ssh --stdio --usage-app=jetbrains --ssh-host-prefix coder-jetbrains-cd.sl.darkdisco.net-- %h
  ConnectTimeout 0
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  LogLevel ERROR
  SetEnv CODER_SSH_SESSION_TYPE=JetBrains

Host coder-jetbrains-cd.sl.darkdisco.net-bg--*
  ProxyCommand /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/coder-linux-amd64 --global-config /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/config --url https://cd.sl.darkdisco.net ssh --stdio --usage-app=disable --ssh-host-prefix coder-jetbrains-cd.sl.darkdisco.net-bg-- %h
  ConnectTimeout 0
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  LogLevel ERROR
  SetEnv CODER_SSH_SESSION_TYPE=JetBrains
# --- END CODER JETBRAINS cd.sl.darkdisco.net

# ------------START-CODER-----------
# This section is managed by coder. DO NOT EDIT.
#
# You should not hand-edit this section unless you are removing it, all
# changes will be lost when running "coder config-ssh".
#
Host coder.test
	HostName coder.test
	ConnectTimeout=0
	StrictHostKeyChecking=no
	UserKnownHostsFile=/dev/null
	LogLevel ERROR
	ProxyCommand /nix/store/4kw9ly4430yyz8dgc7fbrfbvvai0yymk-coder-2.18.5/bin/.coder-wrapped --global-config /home/daniel/.config/coderv2 ssh --stdio test
Host coder.test.main
	HostName coder.test.main
	ConnectTimeout=0
	StrictHostKeyChecking=no
	UserKnownHostsFile=/dev/null
	LogLevel ERROR
	ProxyCommand /nix/store/4kw9ly4430yyz8dgc7fbrfbvvai0yymk-coder-2.18.5/bin/.coder-wrapped --global-config /home/daniel/.config/coderv2 ssh --stdio test.main
Host coder.tomato-tuna-88
	HostName coder.tomato-tuna-88
	ConnectTimeout=0
	StrictHostKeyChecking=no
	UserKnownHostsFile=/dev/null
	LogLevel ERROR
	ProxyCommand /nix/store/4kw9ly4430yyz8dgc7fbrfbvvai0yymk-coder-2.18.5/bin/.coder-wrapped --global-config /home/daniel/.config/coderv2 ssh --stdio tomato-tuna-88
Host coder.tomato-tuna-88.main
	HostName coder.tomato-tuna-88.main
	ConnectTimeout=0
	StrictHostKeyChecking=no
	UserKnownHostsFile=/dev/null
	LogLevel ERROR
	ProxyCommand /nix/store/4kw9ly4430yyz8dgc7fbrfbvvai0yymk-coder-2.18.5/bin/.coder-wrapped --global-config /home/daniel/.config/coderv2 ssh --stdio tomato-tuna-88.main
# ------------END-CODER------------

❯ ssh coder.tomato-tuna-88.main
version mismatch: client v2.18.5+41bfccc, server v2.19.0+2f32b11
download the server version with: 'curl -L https://coder.com/install.sh | sh -s -- --version 2.19.0'
coder@coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb:~$ ls
code  lost+found
coder@coder-2bcd0fbe-258d-4d02-aeeb-4b1f7ffd40ff-6b9db88486-x9vvb:~$

[danmur]

I tried changing the ProxyCommand to the one I have installed through the system, that didn't help

EDIT: never mind, it seems to have overwritten it

[danmur]

Trying the ProxyCommand manually it does this, not sure if this indicates it's working or not (I don't get a prompt or anything):

❯ /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/coder-linux-amd64 --global-config /home/daniel/.local/share/coder-gateway/cd.sl.darkdisco.net/config --url https://cd.sl.darkdisco.net ssh --stdio --usage-app=disable --ssh-host-prefix coder-jetbrains-cd.sl.darkdisco.net-bg-- tomato-tuna-88.main
SSH-2.0-Go




^C⏎