Non-streaming completion API

[louisgv]

Support a non-stream version of the API. Without streaming, it's... more tricky (?). Since we need to load the model and do all kind of jazz before we can send them back a response.

Lack of streaming would likely requires:

• Timeout configuration on the client's side
• Some way to keep-alive the connection and send a completion body

Would need to experiment and see... but low priority because I personally don't use non-stream API :p... (any taker?)

[JNeuvonen]

This is a very cool project @louisgv. I did a server implementation of the nonstreaming API, which didn't break the streaming version of the API and seems to work as expected for nonstreaming as well. Disclaimer: this is my first time writing anything beyond Hello World in Rust..

The implementation approach:

• Use the stream flag sent in a request body to decide if server events should be sent on every token
• If stream flag is false, then collect tokens to string buffer and send the string buffer once completion is built
• If stream flag is true then proceed as usual and send server events to the request sender.

If you want to quickly test the implementation, here's a request body for the nonstreaming API:

{"sampler":"top-p-top-k","prompt":"AI: Greeting! I am a friendly AI assistant. Feel free to ask me anything.\nHuman: Hello world\nAI: ","max_tokens":200,"temperature":1,"seed":147,"frequency_penalty":0.6,"presence_penalty":0,"top_k":42,"top_p":1,"stop":["AI: ","Human: "],"stream":false}

If the implementation seems good enough on the server side, I could proceed and create support for it on the client side as well & make a PR.

[louisgv]

@JNeuvonen awesome :D - feel free to open a PR! (it makes reviewing it a bit easier for me :P) I will take a deeper look in a bit