Would like to extend basekernel to talk to an IPython Kernel

[phpmaps]

@SpencerPark Cool project. Is there anyway to use your project to communicate to an existing IPython Kernel passing in python code for execution, then access the results?

ie. I can get the connection_file from my IPython Kernel which looks like this.

{
  "control_port": 50160,
  "shell_port": 57503,
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "stdin_port": 52597,
  "hb_port": 42540,
  "ip": "127.0.0.1",
  "iopub_port": 40885,
  "key": "a0436f6c-1916-498b-8eb9-e81ab9368e84"
}

I too am having success in creating a MyKernel class which extends your BaseKernel but then I am sort of lost in how to execute python code strings. I've tried eval() and new ExecuteRequest() but not clear on the path forward. On one hand it seems like this project is both a skeleton for 1) creating new Kernels and 2) a messaging app for talking to kernels. I am looking to leverage the messaging pieces, if possible.

Another thing I would be interested in knowing is can I use this project to talk to an IPython Kernel Gateway running in a cloud. I think the answer is no. From research it seem like that would entail some web socket communication and I did not see any type of web socket calls in this project.

[SpencerPark]

Hello @phpmaps, this was a use case I didn't consider but makes perfect sense for inclusion in this project even if just for use in testing. The current machinery only considers that the jvm is running the server side of the jupyter connection which is why you aren't having any luck getting it to work as a controller.

The functionality you are describing is the job of the jupyter_client, specifically the KernelClient (although I am more inclined to write something blocking, any opinions welcome). The notebook talks to the jupyter server which is running a jupyter client which uses zmq to talk to a kernel. The idea of the gateway (from what I can see) is to provide a nice API for controlling a kernel via the same interface that the notebook uses. I don't think it would be too much extra work to implement the gateway in java but I don't think it would be necessary as it is really just a server.

All of the protocol messages are already nicely wrapped because we need to receive them so extending some functionality to create a KernelDriver or RemoteKernel (any other suggestions for a name are appreciated) shouldn't be too difficult. You would then connect one of these kernel interfaces via the connection properties you have specified and be able to send code to your running ipython kernel for execution.

If you would like to take a stab at it, I would be happy to review a PR. Otherwise I've got a midterm on Thursday after which I can start on this.

[SpencerPark]

Alright I've got something that is essentially working. It was bigger than I expected and so it turned into it's own project. https://github.com/SpencerPark/jupyter-jvm-client.

The documentation is severely lacking at the moment but hopefully I can take care of that soon as well as getting something published. If you are feeling adventurous the gist of it is that you need to create a JupyterClientConnection, call connect on that, and then doing the same with a RemoteJupyterKernel. There is some additional functionality for discovering installed kernels and spawning them but from your original issue that is not what you are looking to do.

[phpmaps]

@SpencerPark - very cool. something interesting has come up. want to email me [email protected]?

[SpencerPark]

@phpmaps just sent an email and posting here to keep track but of course the discussion can continue via email.

[aiv-code]

Hi I Just wonder do you have any working example which shown connecting to remote kernel and executing code.

[SpencerPark]

@jits023 I'm in the middle of moving things around (see #18) which includes bringing the client in here and also writing some documentation for how to use everything include what everything does. That restructure is going to bring a major version bump but mostly because of package changes. I'm also hoping to get the client published along with some api packages.

The client project currently supports connecting directly via the zmq channels but I'm also hoping to transparently support the gateway protocol as well. If that is what you're looking for it will have to wait.

To connect to a kernel that you have a connection file for (kernel-....json) like the one from the following:

$ jupyter kernel --kernel=python3
[KernelApp] Starting kernel 'python3'
[KernelApp] Connection file: C:\Users\Spencer\AppData\Roaming\jupyter\runtime\kernel-275c7c13-a8f6-4af7-9d8c-b270bd79ebe1.json
[KernelApp] To connect a client: --existing kernel-275c7c13-a8f6-4af7-9d8c-b270bd79ebe1.json

The following should work (if using the current build on the restructure branch):

import io.github.spencerpark.jupyter.api.KernelConnectionProperties;
import io.github.spencerpark.jupyter.api.display.mime.MIMEType;
import io.github.spencerpark.jupyter.client.channels.JupyterClientConnection;
import io.github.spencerpark.jupyter.comm.DefaultCommManager;

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

//...

Path path = Paths.get("C:\\Users\\Spencer\\AppData\\Roaming\\jupyter\\runtime\\kernel-275c7c13-a8f6-4af7-9d8c-b270bd79ebe1.json");
KernelConnectionProperties props = KernelConnectionProperties.parse(new String(Files.readAllBytes(path), StandardCharsets.UTF_8));

JupyterClientConnection connection = new JupyterClientConnection(props);
connection.connect();

RemoteJupyterKernel kernel = new RemoteJupyterKernel(new DefaultCommManager());
kernel.connect(this.connection);

try {
    ExecutionResult result = this.kernel.eval("print('Hello from the python kernel!')\n10", IOProcider.STD);
    if (result.hasValue())
        System.out.printf("Out [%d]: %s%n", result.getExecutionCount(), result.getValue().getData(MIMEType.TEXT_PLAIN));
} catch (JupyterPublishedException e) {
    // Code executed threw this exception
    System.out.printf("%s: %s%n", e.getPublishedError().getErrorName(), e.getPublishedError().getErrorMessage());
} catch (JupyterReplyException e) {
    // Kernel internal error caused this exception
    System.out.printf("%s %s: %s%n", e.getReply().getStatus(), e.getReply().getErrorName(), e.getReply().getErrorMessage());
}

I would appreciate any feedback on the API if you get a chance to use it :)