Define a json protocol with session support

[CodesInChaos]

Currently there are two ways to access EdgeDB via EdgeQL:

1. The binary protocol
   Implementing it is complex and a significant amount of work. This limits the number of clients you can develop and makes it unlikely that outsiders implement their own client for their favourite language.

2. The http json protocol
   This doesn't support sessions, authentication, transactions, etc. These limitations make it unusable outside demonstration contexts.

   Enhancing it with session support is difficult, since the stateless nature of http makes it difficult to detect

   Implementing clients for this is easy, in a language supporting json. I had written a proof-of-concept in C# in a couple of hours, but gave up when I realized its limitations.

I propose defining a json based protocol based on top of persistent tcp/tls connections.

I'd take inspiration from the json-lines format (one json value per line, prohibiting raw linebreaks inside the value), defining a line as a message. I'd keep the type of the message outside of the json object, so the type the json should be de-serialized to is known before parsing it.

For queries the request/response payload would look like the json objects the http API uses, so implementing it shouldn't be too much effort.

[1st1]

This is an interesting idea. My worry is that migrating from a client library implementing the proposed JSON protocol to a library that implements binary isn't going to be straightforward, because scalar types can't always be decoded in proper host language types.

In a scenario when, say, C# has a popular edgedb library built with JSON protocol, there will be some of the features available in say edgedb-js wouldn't be possible to implement in the C# one.

Since you and @tailhook are into Rust, I have a question: is it possible to design a sans-io Rust library implementing the protocol parser/flow and use it to create drivers for c++, .net, etc? The client library will need then to only implement basic scalar codecs and add an IO layer.

[CodesInChaos]

scalar types can't always be decoded in proper host language types.

Not sure what you mean here. Are you talking about edgedb supporting a larger numeric range than Float64 but some serializers being limited to Float64? If that's an issue, the common workaround is encoding numbers as strings.

[CodesInChaos]

it possible to design a sans-io Rust library implementing the protocol parser/flow and use it to create drivers for c++, .net, etc? The client library will need then to only implement basic scalar codecs and add an IO layer.

Rust can expose a C compatible API which can be consumed from most languages. But I don't fully understand what this API should do/look like.

[1st1]

Not sure what you mean here. Are you talking about edgedb supporting a larger numeric range than Float64 but some serializers being limited to Float64? If that's an issue, the common workaround is encoding numbers as strings.

I mean cal::local_data represented as string in JSON, but as a proper type when using a binary protocol. JSON also lacks knowledge of what's object and what's a named tuple, etc.

Rust can expose a C compatible API which can be consumed from most languages. But I don't fully understand what this API should do/look like.

It could implement the protocol parser and state machine. Basically it needs to expose an object with some protocol state that you can feed data to. Then it can implement pretty much the entire protocol, minus support of host language rich types. It would also make it possible to share code between sync and async clients.

[tailhook]

Rust can expose a C compatible API which can be consumed from most languages. But I don't fully understand what this API should do/look like.

It could implement the protocol parser and state machine. Basically it needs to expose an object with some protocol state that you can feed data to. Then it can implement pretty much the entire protocol, minus support of host language rich types. It would also make it possible to share code between sync and async clients.

1. For any language, parser is just a tiny part of the story: native types support will probably take more code. And then converting from C API like datetime_get_seconds_from_epoch is not much superior to using raw bytes which are microseconds.
2. Java and C# have quite slow native binary support
3. Ruby is only known to me that has C support as good as Pythonic. But even for it I'm not sure if wrapping C library would be smaller code size than parsing raw value. And also adding Rust to the pipeline is not super-straightforward I think.
4. Even when Rust makes sense, it probably also makes sense not to use C API but use rutie (Ruby), cython/pyo3 (Python), neon (node.js) or similar crate to make native objects in Rust.
5. Haskell, Erlang, Ocaml users usually are great at parsers and not great with C (e.g. in erlang distribution of C binary is much harder)
6. And when it comes to C/C++ themselves (warning: a very subjective thing follows) I think most people are into a shiny new think like EdgeDB are already into Rust (and they need to integrate Rust into pipeline, so they could make more code in it anyway).

So I'm still not sure which language this would help.

[1st1]

So I'm still not sure which language this would help.

Alright, let's scratch the idea.

[tailhook]

I propose defining a json based protocol based on top of persistent tcp/tls connections.

I'd take inspiration from the json-lines format (one json value per line, prohibiting raw linebreaks inside the value), defining a line as a message. I'd keep the type of the message outside of the json object, so the type the json should be de-serialized to is known before parsing it.

If we got this route I think we should make a JSON protocol on top of WebSockets, so we can also use it from the browser. And I think it's possible to find WebSockets library for any language nowadays.

JSON is not great because of types, but we can probably allow switching protocol into binary on per-query basis, so one can start with JSON protocol and then gradually upgrade protocol implementation and application to use better protocol.

Still not sure if it worth it. What language(s) do you have in mind?

[1st1]

JSON is not great because of types, but we can probably allow switching protocol into binary on per-query basis, so one can start with JSON protocol and then gradually upgrade protocol implementation

It's really only usable during the transition period while we don't have libraries for major languages. I think once we stabilize the protocol in the next few weeks we should hire people to help us with libraries for Go, Java, and .NET.

[1st1]

That said, we can also extend our HTTP API to have sessions once we implement websockets.