Is INTO a good idea?

[AlonKellner]

While using dockerfiles I often found myself making multiple dockerfiles that duplicate each other (or FROMing each other with static tags) like so:
docker build . -f production.Dockerfile -t production

FROM python
RUN pip install requests

docker build -f development.Dockerfile -t development

FROM python
RUN pip install requests
RUN pip install pytest

That way I can have a slim image for production and a thicker image for development.

However, this can get annoying quickly, every change in my production dockerfile must be reflected in my development dockerfile. In special case even more dockerfiles are needed per project, and keeping track of everything gets tedious.

I thought maybe a new dockerfile stage which reflects the multistage-build power of FROM could come to the rescue, when the INTO stage runs it tags that layer with the provided name.

With our previous example it could be used like so:
docker build . -f combined.Dockerfile

FROM python
RUN pip install requests
INTO production
RUN pip install pytest
INTO development

It can be viewed as a more powerful static version of the -t flag. For more flexibility build-arg may be used in conjuction:
docker build . -f combined.Dockerfile --build-arg DEV=development --build-arg PROD=production

FROM python
RUN pip install requests
ARG PROD
INTO $PROD
RUN pip install pytest
ARG DEV
INTO $DEV

I want to start working on PR, but I need to know if something like this is even a desired feature for buildkit or the dockerfile syntax in general, is it?

Also, if you have ideas about how you would've wanted this feature implemented in buildkit, let me know.
Thank you :)

[jedevc]

EDIT: woops, I misread the question.

This functionality actually already exists :) https://docs.docker.com/develop/develop-images/multistage-build/#stop-at-a-specific-build-stage

TL;DR:

FROM python AS base
# do "common" stuff here

FROM base AS production
RUN pip install requests

FROM base AS development
RUN pip install pytest

Then you can just docker build --target development or docker build --target production.

I'd be curious to see some more complex usecases - currently, your example could be done as follows:

COPY installer.sh /installer.sh
RUN /installer.sh && rm /installer.sh

Then both development and production dockerfiles could build completely independently - I suppose it gets more complex if you'd want to have other docker commands that aren't RUN, but I'm having trouble imagining a use case where that would be helpful.

I think this would probably be reasonably doable to implement, but it seems a little dangerous to ship as part of the default Dockerfile frontend - especially with allowing custom build-args, it gets a little too "C-macro"y for my personal liking.

[AlonKellner]

@jedevc That's a really cool way to use the --target flag, with buildkit's dag based resolution it's quite powerful and does solve many of the problems that INTO could have solved.

There are a few other issues I have encountered that I think INTO might come handy with:

1. Having to run multiple build commands when making a single change that affects multiple images. As you have shown a single dockerfile can be used but multiple commands are still needed.
2. Error prone manual image naming according to some consistent pattern. This happens a lot, most of the time the only thing that changes is the tag, with INTO this can be exploited:

FROM python
RUN pip install requests
ARG TAG
INTO prod-img:$TAG

3. Tagging intermediate layers without commenting many lines (for debugging purposes). I think this:

FROM python
RUN pip install requests
# RUN pip install numpy
# RUN pip install pandas
# RUN pip install torch

Is worse than this:

FROM python
RUN pip install requests
INTO debugging
RUN pip install numpy
RUN pip install pandas
RUN pip install torch

4. Unexpressive dockerfiles. Many dockerfiles will be much more expressive if the image name was explicitly written in the dockerfile. Another unexpressive example is the way --output gets used, which is mystified at best for newcomers:

FROM alpine as base
# some commands that create the /result
FROM scratch
COPY --from=base /result /result

In the imaginary world where INTO exists it might be more expressive by taking flags like so:

FROM alpine as base
# some commands that create the /result
FROM scratch
COPY --from=base /result /result
INTO --output=type=local,dest=result

I see why these benefits are not tremendous, but are they welcome?
What do you think?

[tonistiigi]

Dockerfiles (and all buildkit frontends) have a security guarantee that building them will not start to create new files on user's system, or create and push images without user requesting it in the command line.

Buildx has a higher level command called bake that allows to manage build configurations into targets inside files.

[AlonKellner]

@tonistiigi I see, that makes sense, I think there might be a way to implement INTO without violating that guarantee, but nevertheless it does go against the spirit of that gurrantee, I get it.
Thanks :)