Skip to content

feat: automatically configure vendor mode #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

feat: automatically configure vendor mode #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ The two templates are equivalent with `golang-http` using a structured request/r

You can manage dependencies in one of the following ways:

- To use Go modules without vendoring, the default already is set `GO111MODULE=on` but you also can make that explicit by adding `--build-arg GO111MODULE=on` to `faas-cli up`, you can also use `--build-arg GOPROXY=https://` if you want to use your own mirror for the modules
- You can also Go modules with vendoring, run `go mod vendor` in your function folder and add `--build-arg GO111MODULE=off --build-arg GOFLAGS='-mod=vendor'` to `faas-cli up`
- To use Go modules without vendoring, the default already is set `GO111MODULE=on` but you also can make that explicit by adding `--build-arg GO111MODULE=on` to `faas-cli up`, you can also use `--build-arg GOPROXY=https://` if you want to use your own mirror for the modules.
- You can also Go modules with vendoring, run `go mod vendor`, the build process will automatically detect the vendor folder and configure the Go environment correctly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should still mention that GO111MODULE will be forced to off.

auto-magic can end up being more confusing than setting explicit values.

- If you have a private module dependency, we recommend using the vendoring technique from above.

## Adding static files to your image
Expand Down
12 changes: 9 additions & 3 deletions template/golang-http/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,26 @@ ARG GO111MODULE="on"
ARG GOPROXY=""
ARG GOFLAGS=""
ARG CGO_ENABLED=0
ARG DEBUG=0

ENV DEBUG=${DEBUG}
ENV CGO_ENABLED=${CGO_ENABLED}
ENV GOOS=${TARGETOS}
ENV GOARCH=${TARGETARCH}
ENV GO=/go/src/handler/go.sh

RUN chmod +x ${GO}

# Run a gofmt and exclude all vendored code.
RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }

WORKDIR /go/src/handler/function
RUN mkdir -p /go/src/handler/function/static

RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
RUN ${GO} test ./... -cover

WORKDIR /go/src/handler
RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
RUN ${GO} build --ldflags "-s -w" -a -installsuffix cgo -o handler .

FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.15
# Add non root user and certs
Expand Down
30 changes: 30 additions & 0 deletions template/golang-http/go.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/sh
# go.sh is a wrapper for the Go command that will
# automatically set the required module related flags
# when a vendor folder is detected.
#
# Currently, in Go 1.18, Go Workspaces are incompatible
Copy link
Contributor

@mrwormhole mrwormhole Jun 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will be a dumb question but if this was 1.17 would we need this tweak?

I am kinda thinking how all sh scripts can be avoided but I thought vendor files were compatible with workspaces, it is intriguing. I think http templates are super simple compared to past sh scripts before 1.18 and this amount seems OK, I will chase this incompatibility issue. Looks good to me in this way tho, it is more clear than old shell scripts

I will be kinda confused again but I thought -mod=vendor was the default flag when Go detects vendor folder since 1.13, when did this actually change so we had to specify it explicitly? I am very bad vendor user(not using at all since 1.13) If you have a look at the footnotes, it says -mod=vendor is the default here https://www.ardanlabs.com/blog/2020/04/modules-06-vendoring.html

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually -mod readonly also reads the vendor as far as I understand from this thread golang/go#30404, so I think shell script may no longer needed? the end user needs to do -mod=readonly as you pointed out here Lucas https://github.com/LucasRoesler/openfaas-golang-template-workspace

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrwormhole i don't 100% follow. Were you able to get a workspaces project to build with the vendor by using -mod=vendor -mod=readonly?

The main reason for the script, is that when using workspaces, go build ignores the vendor folder and attempts to download the modules again. When a vendor folder is detected, disabling modules forces Go to use the GOPATH, which then, correctly, detects and uses the vendor folder as expected.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-mod=readonly should use vendor if vendor exists, I also don't know why this config needs to specify -mod=vendor to get vendor files, it is automatic when not specified, default is -mod=readonly, -mod=mod is the one who only looks up for go modules and ignores vendor completely

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand what you are saying, but

  1. this code does not use or require -mod=vendor
  2. you are 100% right, we normally expect go build to automatically detect the vendor folder, however, the current behavior of workspaces seems to ignore this. I tried it (I had exactly the same hope that you did) but it was not building in the correct way. Even with a vendor folder present it was still downloading the modules.

Perhaps you can share a working example repo with the changes you would like to see?

When I tried to use vendor with the go.work, the only time I could get it to detect the vendor folder was by disabling modules. This is what the script does. It automates detecting this situation instead of requiring the developer to pass an additional build arg.

Copy link
Contributor

@mrwormhole mrwormhole Jun 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I actually created the same repository but without -mod vendor flag, I vendored my dependencies (which is the user responsibility) and deleted -mod=vendor and made go modules ON, it actually detected and built with vendor folder but the key here was I deleted the repository of my module, I do think -mod vendor is completely unneeded and shouldn't be used.

https://github.com/MrWormHole/openfaas-golang-template-workspace

the end-user should be able to know as a Go dev, what could be the use case for enabling go modules and doing -mod vendor. Vendor folder should exist only for disaster recovery, else it will download all deps with go.mod and not use vendor folder. Also go mod vendor should be called explicitly by developers and stored in the repository, else it is a problem

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can move forward without using additional bash and if statements, I'd be interested to see that.

Copy link
Contributor

@mrwormhole mrwormhole Jun 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Lucas is right @LucasRoesler depending on your Go version it gets set to default to -mod=readonly, I would keep it and resolve these comments. It will enhance the dev usage of these templates even though it sets it internally.

https://go.dev/ref/mod
image

I am on version 1.17.11 and -mod=mod always runs when I don't specify -mod=vendor with go modules "OFF" or "ON", sorry for the confusion I had earlier. One theory from my side is that alpine images(Dockerfile this uses) of 1.18 sets it to -mod=mod, I didn't confirm yet today but if you fork it, you will see this one builds https://github.com/MrWormHole/openfaas-golang-template-workspace without on or off specification because -mod flag figures it out

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrwormhole can you clarify your recommendations in a few bullets or steps, or are you retracting what you said?

# with vendored modules. As a result, we must disable
# Go modules and use GOPATH mode.

# We use this bash script to wrap Go commands because
# there is no clear way to set an env variable from the
# a script in a Dockerfile.
# It is possible to use `go env -w` but, but env varaibles
# have precedence and if it is set as an arg/env variable,
# then it will be ignored by the `go env -w`.

# if the function/vendor folder exists
# then we set the env variables for
# GO111MODULE=off
if [ -d "/go/src/handler/function/vendor" ]; then
echo "Setting vendor mode env variables"
export GO111MODULE=off
fi

# if DEBUG env is 1, print the go env
if [ "${DEBUG:-0}" = "1" ]; then
go env
fi

go "$@"
13 changes: 10 additions & 3 deletions template/golang-middleware/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,26 @@ ARG GO111MODULE="on"
ARG GOPROXY=""
ARG GOFLAGS=""
ARG CGO_ENABLED=0
ARG DEBUG=0

ENV DEBUG=${DEBUG}
ENV CGO_ENABLED=${CGO_ENABLED}
ENV GOOS=${TARGETOS}
ENV GOARCH=${TARGETARCH}
ENV GO=/go/src/handler/go.sh

RUN chmod +x ${GO}

# Run a gofmt and exclude all vendored code.
RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }

WORKDIR /go/src/handler/function
RUN mkdir -p /go/src/handler/function/static

RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
RUN ${GO} test ./... -cover

WORKDIR /go/src/handler
RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
RUN ${GO} build --ldflags "-s -w" -a -installsuffix cgo -o handler .

FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.15 as ship

Expand Down
30 changes: 30 additions & 0 deletions template/golang-middleware/go.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/sh
# go.sh is a wrapper for the Go command that will
# automatically set the required module related flags
# when a vendor folder is detected.
#
# Currently, in Go 1.18, Go Workspaces are incompatible
# with vendored modules. As a result, we must disable
# Go modules and use GOPATH mode.

# We use this bash script to wrap Go commands because
# there is no clear way to set an env variable from the
# a script in a Dockerfile.
# It is possible to use `go env -w` but, but env varaibles
# have precedence and if it is set as an arg/env variable,
# then it will be ignored by the `go env -w`.

# if the function/vendor folder exists
# then we set the env variables for
# GO111MODULE=off
if [ -d "/go/src/handler/function/vendor" ]; then
echo "Setting vendor mode env variables"
export GO111MODULE=off
fi

# if DEBUG env is 1, print the go env
if [ "${DEBUG:-0}" = "1" ]; then
go env
fi

go "$@"