ci: enables ollama integration tests #23
Conversation
|
Lemme know if we are brave enough to run this in CI. One reason not to is that it could be flakey. |
|
always brave enough for CI. |
codefromthecrypt
changed the title
|
Made tests conditional on Note: docker/testcontainers is not a viable option as the models are slow to start (especially in docker which disallows GPU access), and also eat a lot of memory. I think if we want CI, we should just start ollama in github action. If sounds good, 👍 and I'll give it a go. |
|
oops sorry about red I'll sort it later. |
|
🤞 on CI, I am only running it in latest python, after normal unit tests. |
|
last try on CI today 🤞 |
|
notes for tomorrow:
|
|
put back to draft until I sort out what's going on in CI. Feel free to edit this PR if someone wants to finish before I respond again |
|
@k33g has some interesting options in https://github.com/parakeet-nest/awesome-slms trying again with testcontainers may not be a terrible idea. I'm out of time for the moment so anyone can give a go if they like. |
codefromthecrypt
force-pushed
the
ollama-integration
branch
2 times, most recently
from
58b3bcb
to
a6e043c
Compare
codefromthecrypt
changed the title
Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
force-pushed
the
ollama-integration
branch
from
e175077
to
d599f66
Compare
|
@anuraaga if you want to do another drive-by review, more than welcome! |
| key: ollama-${{ hashFiles('./src/exchange/providers/ollama.py') }} | ||
|
|
||
| - name: Install Ollama | ||
| run: curl -fsSL https://ollama.com/install.sh | sh |
There was a problem hiding this comment.
confusing part is the log here ends up saying "hey I'm running" when well it isn't ;) I used act locally to figure it out.
act pull_request -P ubuntu-latest=ghcr.io/catthehacker/ubuntu:act-latest --container-architecture linux/amd64
|
been pretty consistently 5m all in
|
| # this prior to running tests. This also reduces the chance of flakiness. | ||
| - name: Pull and Test Ollama model | ||
| run: | # get the OLLAMA_MODEL from ./src/exchange/providers/ollama.py | ||
| OLLAMA_MODEL=$(uv run python -c "from src.exchange.providers.ollama import OLLAMA_MODEL; print(OLLAMA_MODEL)") |
Co-authored-by: Anuraag (Rag) Agrawal <[email protected]>
Signed-off-by: Adrian Cole <[email protected]>
|
thanks again for the feedback @anuraaga this is better |
|
LGTM (seems possible for repo to disable grey check drive by approvals, just learned that 😀) |
This adds CI for the existing ollama tests.
The model we use, currently mistral-nemo is somewhat large, so not helpful to run in docker for a couple reasons. Instead, we nohup ollama directly.
To reduce problems, pulling and smoke testing the model are done in a separate step, logging on failure. This gives us a high confidence that when the tests actually run, failures are related to tests, not slow first-time model startup hitting some timeout. All that said, the tests themselves do take time as it is using real inference now.