Tutorial how to build a docker image with rules_conda and bazel

[deichbrise]

Hi,

thank you for providing this nice rules! I am a newbie with bazel and would like to ask for help building a py3_image instead of a py_binary.

Thanks you!

[spietras]

I need to research further how to use py3_image, but if you (or anyone else reading this) want to use Bazel and rules_conda with Docker directly then you can kinda do it as described below.

Assuming that:

• you want to use Bazel inside Docker during image building
• you can use BuildKit
• you are inside example directory
• you changed rules_conda rule in WORKSPACE from local_repository to http_archive (currently commented out)

then you can use the following Dockerfile:

# Using debian, because I can't get it to work on Alpine, probably because of lack of glibc
FROM debian:bullseye-slim

# Install required packages
RUN apt-get update && \
    apt-get install -y ca-certificates openssl gcc git python

# Get certificate from "releases.bazel.build"
RUN mkdir /usr/local/share/ca-certificates/extra && \
    openssl s_client -showcerts -connect releases.bazel.build:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > /usr/local/share/ca-certificates/extra/releases.bazel.crt && \
    update-ca-certificates

WORKDIR /app/

COPY ./ ./

# A little trick here, mount cache from your machine and then copy it to some container directory
RUN --mount=type=cache,target=/root/.cache \
    ./bazelw build app && \
    mkdir ./.cache/ && \
    mv /root/.cache/bazel/ ./.cache/bazel/ && \
    mv /root/.cache/bazelisk/ ./.cache/bazelisk/

# And then copy back to container cache
RUN mv ./.cache/bazel/ /root/.cache/bazel/ && \
    mv ./.cache/bazelisk/ /root/.cache/bazelisk/

# Run directly from bazel-bin (not with bazel run) to get rid of Bazel messages
ENTRYPOINT [ "./bazel-bin/app/main" ]

and the following .dockerignore:

Dockerfile
.dockerignore
README.md
bazel-*/
bazelw.bat
tools/bazelisk/
!tools/bazelisk/bazelisk-linux-*

And then run:

DOCKER_BUILDKIT=1 docker build -t rules_conda/example . && docker run -it rules_conda/example

This way you can benefit from both Docker and Bazel. Bazel cache will persist between container runs (locally on one machine). If you need, you can tweak it further and use multi-stage builds to reduce image size (e.g. you don't really need bazel and bazelisk binaries to run the app), but that will require some dirty work with the right files and directories.

[spietras]

I think I figured it out with py_image. I'm not 100% sure it works correctly though. I'm also using py3_image instead of py_image

Let's assume you are in example directory.

1. Add empty BUILD file in the current directory. Not sure why it's necessary but see the third point here.

2. Add this at the end of the WORKSPACE file. This needs to be after register_toolchain (see here):

http_archive(
    name = "io_bazel_rules_docker",
    sha256 = "27d53c1d646fc9537a70427ad7b034734d08a9c38924cc6357cc973fed300820",
    strip_prefix = "rules_docker-0.24.0",
    url = "https://github.com/bazelbuild/rules_docker/releases/download/v0.24.0/rules_docker-v0.24.0.tar.gz",
)

load(
    "@io_bazel_rules_docker//repositories:repositories.bzl",
    container_repositories = "repositories",
)

container_repositories()

load(
    "@io_bazel_rules_docker//python3:image.bzl",
    _py3_image_repos = "repositories",
)

_py3_image_repos()

3. Change app/BUILD to:

load("@io_bazel_rules_docker//python3:image.bzl", "py3_image")

alias(
    name = "app",
    actual = "main",
)

py3_image(
    name = "main",
    srcs = glob(["**/*.py"]),
)

4. Build the image and run the container with:

./bazelw run --spawn_strategy=local app

If you don't want to pass --spawn_strategy=local every time you can add it to .bazelrc

5. The image is now in your local registry. You can run it directly with Docker if you want:

docker run bazel/app:main

[yngtdd]

Thank you for putting together rules_conda! This has really been helping me out with Python dependencies that have compiled components.

And thank you for taking the time to look into how rules_conda can be used with rules_docker. I am glad that I was able to find this thread. Following your advice above, making sure that I am registering the toolchain before including rules_docker, I am hitting a bit of a snag.

It seems like a Docker ImageLayer is having trouble with getting the conda environement bin and is returning an I/O error:

➜ > bazel build //...
INFO: Analyzed 8 targets (93 packages loaded, 43364 targets configured).
INFO: Found 8 targets...
WARNING: cleared convenience symlink(s) bazel-bin, bazel-testlogs because their destinations would be ambiguous
ERROR: /Users/yngtdd/work/src/graviton/graviton/throughput/BUILD:22:10: ImageLayer graviton/throughput/throughput_image-layer.tar failed: I/O exception during sandboxed execution: /private/var/tmp/_bazel_yngtdd/967a3ed34237f1c3d96e5ff422989f42/sandbox/darwin-sandbox/514/execroot/__main__/external/py3_env/py3_env/bin (File exists)
INFO: Elapsed time: 5.984s, Critical Path: 3.42s
INFO: 54 processes: 48 internal, 6 darwin-sandbox.
FAILED: Build did NOT complete successfully

I was hoping that the register_toolchain would sort this out, but I am not having luck just yet. Any thoughts on what I could try to track down that I/O error? I realize that this is most likely an issue with rules_docker. That project is currently in minimal maintenance mode, so I am a bit hesitant to rely on it, but it would be nice if I could have Bazel handle our dependencies for us within Docker containers rather than having devs need to maintain Dockerfiles.

If this I/O issue turns out to be a snag on the rules_docker side, I may then go to your suggestion of using BuildKit with a Dockerfile.

Thanks again!

Here is my setup:

Tail of my Workspace.bazel:

# ...

# RULES_CONDA
# ---

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_conda",
    sha256 = "ee187c4902f5f8da85fcba9943064fd38b2a75f80ed0a2844b34cf9e27cb5990",
    url = "https://github.com/spietras/rules_conda/releases/download/0.2.0/rules_conda-0.2.0.zip"
)

load("@rules_conda//:defs.bzl", "conda_create", "load_conda", "register_toolchain")

load_conda(
    conda_version = "23.3.1",  # version of conda to download, default is 4.10.3
    installer = "miniforge",  # which conda installer to download, either miniconda or miniforge, default is miniconda
    install_mamba = True,  # whether to install mamba, which is a faster drop-in replacement for conda, default is False
    mamba_version = "1.4.2",  # version of mamba to install, default is 0.17.0
    quiet = False,  # True if conda output should be hidden, default is True
    timeout = 600,  # how many seconds each execute action can take, default is 3600
)

conda_create(
    name = "py3_env",
    environment = "@//third_party/conda:environment.yml",
    quiet = False,
)

register_toolchain(env = "py3_env")

# RULES_DOCKER
# ---

http_archive(
     name = "io_bazel_rules_docker",
     sha256 = "b1e80761a8a8243d03ebca8845e9cc1ba6c82ce7c5179ce2b295cd36f7e394bf",
     urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.25.0/rules_docker-v0.25.0.tar.gz"],
 )

load(
     "@io_bazel_rules_docker//repositories:repositories.bzl",
     container_repositories = "repositories",
)

container_repositories()

load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")

container_deps()

load(
    "@io_bazel_rules_docker//container:container.bzl",
    "container_pull",
)

container_pull(
    name = "python_base_image",
    registry = "registry.hub.docker.com",
    repository = "library/python",
    # 'tag' is also supported, but digest is encouraged for reproducibility.
    #tag = "3.10-alpine3.17",
    tag = "3.10.11",
    #digest = "sha256:deadbeef",
)

load(
    "@io_bazel_rules_docker//python3:image.bzl",
    _py_image_repos = "repositories",
)

_py_image_repos()

Build for my Python project in the monorepo:

load("@rules_python//python:defs.bzl", "py_binary", "py_library")
load("@io_bazel_rules_docker//python3:image.bzl", "py3_image")

py_library(
    name = "throughput",
    srcs =glob(["src/*.py"]),
    visibility = ["//visibility:public"]
)

py_binary(
    name = "microservice",
    srcs = ["api/microservice.py"],
    deps = ["//graviton/throughput:throughput"]
)

py_test(
    name = "test_throughput",
    srcs = ["tests/test_throughput.py"],
    deps = ["//graviton/throughput:throughput"]
)

py3_image(
    name = "throughput_image",
    srcs = ["api/microservice.py"],
    deps = ["//graviton/throughput:throughput"],
    main = "api/microservice.py",
    # TODO: Add a custom base image
    #base = "@python_base_image//image"
)

[spietras]

I guess the culprit is that you want to use a custom base image. I tried to do this with a more straightforward example (adding Ubuntu as a base image to my example above) but failed. Maybe this issue can give you some insight: bazelbuild/rules_docker#1247. Unfortunately, I don't know how to make it work.

But I would start by asking if you really need a custom base image. If you need some system tools to be present then it's necessary, but if it's only a bare Python code, then maybe you don't need that at all. I see that you tried to use a Python image, maybe to provide a specific Python version, but you can do the same by simply editing environment.yml. Also, some non-python tools can be installed by conda, because they are available as conda packages.

I don't have much experience in this situation, because I have never tried to build Docker images using Bazel.

I feel like it is possible to combine custom Python toolchains and custom Docker base images, but this needs further research. But it would be nice to know, so maybe someone will find this and reply in the future.