Dealing with SSH stuff

[mmenozzi]

It would be great to have a way to copy SSH stuff (public/private keys and host key verification) into the container. This way we could copy the same SSH keys used by the real pipeline.

What do you think?

[ktomk]

Sounds useful in case SSH is in use.

Do you have a good idea on how to configure that on a project level? Command line arguments are always an option nevertheless, but this should have some comfort.

And can you describe (outline) a minimal SSH use-case, this would be helpful for orientation.

[mmenozzi]

For example one use case could be if you have composer dependencies that must be downloaded from a private GIT repository. In that case you should generate a new key pair inside the container and add the generated public key to the private GIT repo.

An idea (I don't know if it's good) could be to have a way (for example with a command line switch) to copy the host SSH keys into the container. Something like:

docker cp $HOME/.ssh/id_rsa* <container_id>:/root/.ssh/

(but watchout to this issue under Docker for Mac: docker/for-mac#1814)

BTW this assume that the current user's keys have the same access of the real pipeline's keys (which is true in most cases in my opinion).

[ktomk]

Composer sounds like a good use-case.

Spotted the Private repositories / SSH agent section in the read-me of the composer docker image that has a different approach by making the SSH Agent usable:

$ eval $(ssh-agent); \
  docker run --rm --interactive --tty \
  --volume $PWD:/app \
  --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
  --volume /etc/passwd:/etc/passwd:ro \
  --volume /etc/group:/etc/group:ro \
  --env SSH_AUTH_SOCK=/ssh-auth.sock \
  --user $(id -u):$(id -g) \
  composer install

This looks interesting to me for two parts:

1. SSH Agent support which might be much more straight forward then to configure files to copy on the command-line.
2. Running as the current user (--user $(id -u):$(id -g)) is also something I had the idea about as a feature.

[mmenozzi]

Yes I think is much better than copying ssh keys!

[ktomk]

@mmenozzi Pipelines 0.0.42 is released with a --ssh option for the ssh auth socket mounted into the container (ssh agent forwarding). the container needs an ssh client:

$ <<'PIPELINE' bin/pipelines --file - --verbatim --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE

...

�+ ssh -T [email protected] || test $? -eq 1
Warning: Permanently added the RSA host key for IP address '140.82.118.3' to the list of known hosts.
Hi ktomk! You've successfully authenticated, but GitHub does not provide shell access.

Known hosts is not well integrated and I'm open for suggestions there.

[ktomk]

@mmenozzi not sure if there were open questions left or where you have been heading to since we had this nice conversation, a comment or two how you would like to proceed would be great so that this issue is not hanging so much in the air. but no stress, just asking. I hope you're well.

[mmenozzi]

Ehi @ktomk I didn’t give any feedback because I wasn’t able to test this feature in my pipelines. I think you can close if it works properly.

[ktomk]

Ah, well, if you could lend me a hand as I don't have mac os at hand and if you have and if you also have some github ssh access to test the "one-liner" example in #6 (comment) ? That would be great @mmenozzi . Just for feedback the --ssh switch does work or not on macos.

[mmenozzi]

I tried the one-line command but it hangs with no output. Moreover I can't kill the docker container with docker kill. Also that command hangs with no output. I have to restart Docker to kill it.

[ktomk]

Thanks a lot. Hmm, I wonder where it hangs. If you could try more verbose:

$ <<'PIPELINE' bin/pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE

and paste the output (please redact any personal details).

[mmenozzi]

The problem was that I was running the command from my home directory so it was trying to copy my whole home directory into the container...
Now I run it into an empty directory and this is the output:

➜ <<'PIPELINE' pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE
info: project directory is '/private/tmp/test'
info: reading pipelines from stdin
info: running pipeline 'default'
+++ step #1

    name...........: (unnamed)
    effective-image: ktomk/pipelines:ssh
    container......: pipelines-1.no-name.default.test
docker ps --no-trunc -qa --filter 'name=^/\Qpipelines-1.no-name.default.test\E$'
exit status: 0
docker inspect pipelines-1.no-name.default.test
exit status: 1
docker run -i --name pipelines-1.no-name.default.test -l 'pipelines.prefix=' -l 'pipelines.role=step' -l 'pipelines.project.name=test' -l 'pipelines.project.path=/private/tmp/test' -e 'BITBUCKET_BUILD_NUMBER=0' -e 'BITBUCKET_COMMIT=0000000000000000000000000000000000000000' -e 'BITBUCKET_REPO_OWNER=manuele' -e 'BITBUCKET_REPO_SLUG=test' -e 'BITBUCKET_STEP_RUN_NUMBER=1' -e 'CI=true' -e 'PIPELINES_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_ID=default' -e 'PIPELINES_IDS=c21f969b5f03d33d43e04f8f136e7682' -e 'PIPELINES_PIP_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_PROJECT_PATH=/private/tmp/test' -e 'BITBUCKET_CLONE_DIR=/app' -v '/var/run/docker.sock:/var/run/docker.sock' -v '/private/var/folders/y6/dk1m_mc54yb_j7lkhlcn8mdr0000gn/T/ssh-ea1bAHFZwdkx/agent.3061:/var/run/ssh-auth.sock:ro' -e 'SSH_AUTH_SOCK=/var/run/ssh-auth.sock' --workdir /app --detach '--entrypoint=/bin/sh' 'ktomk/pipelines:ssh'
exit status: 0
    container-id...: fc13df430a64

+++ copying files into container...
cd /private/var/folders/y6/dk1m_mc54yb_j7lkhlcn8mdr0000gn/T/pipelines-cp.xMD3gn/. && echo 'app' | tar c -h -f - --no-recursion ./app | docker  cp - 'fc13df430a643354013c8a4dc815125014fb06a9831cfcc9862a01f6daee90b9:/.'
exit status: 0
cd /private/tmp/test/. && tar c -f - . | docker  cp - 'fc13df430a643354013c8a4dc815125014fb06a9831cfcc9862a01f6daee90b9:/app'
exit status: 0

<<'SCRIPT' docker exec -i pipelines-1.no-name.default.test /bin/sh
# this /bin/sh script is generated from a pipeline script
set -e
printf '\n'
printf '\035+ %s\n' 'mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts'
mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
printf '\n'
printf '\035+ %s\n' 'ssh -T [email protected] || test $? -eq 1'
ssh -T [email protected] || test $? -eq 1
SCRIPT


+ mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96

+ ssh -T [email protected] || test $? -eq 1
Warning: Permanently added the RSA host key for IP address '140.82.121.3' to the list of known hosts.
[email protected]: Permission denied (publickey).
exit status: 1
script non-zero exit status: 1
docker kill fc13df430a643354013c8a4dc815125014fb06a9831cfcc9862a01f6daee90b9
exit status: 0
docker rm fc13df430a643354013c8a4dc815125014fb06a9831cfcc9862a01f6daee90b9
exit status: 0

I suppose it's not working... If I run the ssh connection to github outside the container it works of course:

➜ ssh -T [email protected]
Hi mmenozzi! You've successfully authenticated, but GitHub does not provide shell access.

[ktomk]

My bad, the test suggestion was incomplete. Sorry for the inconvenience. This needs some more thought on resource protection on pipelines end.

Thanks for handling this. And yes the result "[email protected]: Permission denied (pulbickey)." shows it is not working.

Docker is started however with the appropriate options (third and second last lines):

docker run -i --name pipelines-1.no-name.default.test \
    -l 'pipelines.prefix=' -l 'pipelines.role=step' -l 'pipelines.project.name=test' -l 'pipelines.project.path=/private/tmp/test' 
    -e 'BITBUCKET_BUILD_NUMBER=0' -e 'BITBUCKET_COMMIT=0000000000000000000000000000000000000000' -e 'BITBUCKET_REPO_OWNER=gulianna' -e 'BITBUCKET_REPO_SLUG=test' -e 'BITBUCKET_STEP_RUN_NUMBER=1' -e 'CI=true' -e 'PIPELINES_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_ID=default' -e 'PIPELINES_IDS=c21f969b5f03d33d43e04f8f136e7682' -e 'PIPELINES_PIP_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_PROJECT_PATH=/private/tmp/test' -e 'BITBUCKET_CLONE_DIR=/app' \
    -v '/var/run/docker.sock:/var/run/docker.sock' \
    -v '/private/var/folders/y6/dk1m_mcyb54_j7lkhl8mcndr0000gn/T/ssh-1beaFZAHwdkx/agent.3061:/var/run/ssh-auth.sock:ro' \
    -e 'SSH_AUTH_SOCK=/var/run/ssh-auth.sock' \
    --workdir /app --detach '--entrypoint=/bin/sh' 'ktomk/pipelines:ssh'

So the test failed :). Have to chew a bit on this one and check if and how the ssh sock mounting is actually supported with the macos local docker machine. Thanks for your elaborated testing again!

[ktomk]

From a quick first look it is perhaps that the ssh auth socket is on a different path: /var/run/ssh-auth.sock.

It looks possible for another test to make pipelines use that exact path by overriding the SSH_AUTH_SOCK environment parameter (change in front of the pipelines command on the first line right after <<'PIPELINE' ):

$ <<'PIPELINE' SSH_AUTH_SOCK=/var/run/ssh-auth.sock bin/pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE

Maybe if I may ask for another test for this, @mmenozzi ?

[mmenozzi]

Not working:

➜ <<'PIPELINE' SSH_AUTH_SOCK=/var/run/ssh-auth.sock pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE
info: project directory is '/private/tmp/test'
info: reading pipelines from stdin
info: running pipeline 'default'
+++ step #1

    name...........: (unnamed)
    effective-image: ktomk/pipelines:ssh
    container......: pipelines-1.no-name.default.test
docker ps --no-trunc -qa --filter 'name=^/\Qpipelines-1.no-name.default.test\E$'
exit status: 0
docker inspect pipelines-1.no-name.default.test
exit status: 1
docker run -i --name pipelines-1.no-name.default.test -l 'pipelines.prefix=' -l 'pipelines.role=step' -l 'pipelines.project.name=test' -l 'pipelines.project.path=/private/tmp/test' -e 'BITBUCKET_BUILD_NUMBER=0' -e 'BITBUCKET_COMMIT=0000000000000000000000000000000000000000' -e 'BITBUCKET_REPO_OWNER=manuele' -e 'BITBUCKET_REPO_SLUG=test' -e 'BITBUCKET_STEP_RUN_NUMBER=1' -e 'CI=true' -e 'PIPELINES_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_ID=default' -e 'PIPELINES_IDS=c21f969b5f03d33d43e04f8f136e7682' -e 'PIPELINES_PIP_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_PROJECT_PATH=/private/tmp/test' -e 'BITBUCKET_CLONE_DIR=/app' -v '/var/run/docker.sock:/var/run/docker.sock' --workdir /app --detach '--entrypoint=/bin/sh' 'ktomk/pipelines:ssh'
exit status: 0
    container-id...: 4e07696088f3

+++ copying files into container...
cd /private/var/folders/y6/dk1m_mc54yb_j7lkhlcn8mdr0000gn/T/pipelines-cp.wWL1Wq/. && echo 'app' | tar c -h -f - --no-recursion ./app | docker  cp - '4e07696088f32667f2b30711d9c468df8151a5f03f0efe8fefbf66faaae8d957:/.'
exit status: 0
cd /private/tmp/test/. && tar c -f - . | docker  cp - '4e07696088f32667f2b30711d9c468df8151a5f03f0efe8fefbf66faaae8d957:/app'
exit status: 0

<<'SCRIPT' docker exec -i pipelines-1.no-name.default.test /bin/sh
# this /bin/sh script is generated from a pipeline script
set -e
printf '\n'
printf '\035+ %s\n' 'mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts'
mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
printf '\n'
printf '\035+ %s\n' 'ssh -T [email protected] || test $? -eq 1'
ssh -T [email protected] || test $? -eq 1
SCRIPT


+ mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96

+ ssh -T [email protected] || test $? -eq 1
Warning: Permanently added the RSA host key for IP address '140.82.121.3' to the list of known hosts.
[email protected]: Permission denied (publickey).
exit status: 1
script non-zero exit status: 1
docker kill 4e07696088f32667f2b30711d9c468df8151a5f03f0efe8fefbf66faaae8d957
exit status: 0
docker rm 4e07696088f32667f2b30711d9c468df8151a5f03f0efe8fefbf66faaae8d957
exit status: 0

Here is my SSH_AUTH_SOCK:

➜ echo $SSH_AUTH_SOCK
/private/var/folders/y6/dk1m_mc54yb_j7lkhlcn8mdr0000gn/T/ssh-ea1bAHFZwdkx/agent.3061

[ktomk]

Thanks, that path to the ssh agent socket might have been wrong then. The difference to SSH_AUTH_SOCK was intended though, by default pipelines --ssh takes the system one. I might need to do something like

If I may ask another time to try with /run/host-services/ssh-auth.sock? Sorry for the trial and error, this is my last shot in the blue regarding a fixed-name one, I'll consult some docs afterwards if this ain't it. This path if I have not mistaken it is from launchd.

$ <<'PIPELINE' SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE

And if you could share your macos and docker (for mac) version would be nice.

[mmenozzi]

Not working:

➜ <<'PIPELINE' SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock pipelines --file - --verbose --debug --ssh
pipelines:
  default:
    - step:
        image: ktomk/pipelines:ssh
        script:
          - mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
          - ssh -T [email protected] || test $? -eq 1
PIPELINE
info: project directory is '/private/tmp/test'
info: reading pipelines from stdin
info: running pipeline 'default'
+++ step #1

    name...........: (unnamed)
    effective-image: ktomk/pipelines:ssh
    container......: pipelines-1.no-name.default.test
docker ps --no-trunc -qa --filter 'name=^/\Qpipelines-1.no-name.default.test\E$'
exit status: 0
docker inspect pipelines-1.no-name.default.test
exit status: 1
docker run -i --name pipelines-1.no-name.default.test -l 'pipelines.prefix=' -l 'pipelines.role=step' -l 'pipelines.project.name=test' -l 'pipelines.project.path=/private/tmp/test' -e 'BITBUCKET_BUILD_NUMBER=0' -e 'BITBUCKET_COMMIT=0000000000000000000000000000000000000000' -e 'BITBUCKET_REPO_OWNER=manuele' -e 'BITBUCKET_REPO_SLUG=test' -e 'BITBUCKET_STEP_RUN_NUMBER=1' -e 'CI=true' -e 'PIPELINES_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_ID=default' -e 'PIPELINES_IDS=c21f969b5f03d33d43e04f8f136e7682' -e 'PIPELINES_PIP_CONTAINER_NAME=pipelines-1.no-name.default.test' -e 'PIPELINES_PROJECT_PATH=/private/tmp/test' -e 'BITBUCKET_CLONE_DIR=/app' -v '/var/run/docker.sock:/var/run/docker.sock' --workdir /app --detach '--entrypoint=/bin/sh' 'ktomk/pipelines:ssh'
exit status: 0
    container-id...: fccb74debe85

+++ copying files into container...
cd /private/var/folders/y6/dk1m_mc54yb_j7lkhlcn8mdr0000gn/T/pipelines-cp.XrFlGF/. && echo 'app' | tar c -h -f - --no-recursion ./app | docker  cp - 'fccb74debe855447e82185c3a6cfbec99c9bb3e532cffc457a60fec21f728ed4:/.'
exit status: 0
cd /private/tmp/test/. && tar c -f - . | docker  cp - 'fccb74debe855447e82185c3a6cfbec99c9bb3e532cffc457a60fec21f728ed4:/app'
exit status: 0

<<'SCRIPT' docker exec -i pipelines-1.no-name.default.test /bin/sh
# this /bin/sh script is generated from a pipeline script
set -e
printf '\n'
printf '\035+ %s\n' 'mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts'
mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
printf '\n'
printf '\035+ %s\n' 'ssh -T [email protected] || test $? -eq 1'
ssh -T [email protected] || test $? -eq 1
SCRIPT


+ mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96
# github.com:22 SSH-2.0-babeld-43595b96

+ ssh -T [email protected] || test $? -eq 1
Warning: Permanently added the RSA host key for IP address '140.82.121.4' to the list of known hosts.
[email protected]: Permission denied (publickey).
exit status: 1
script non-zero exit status: 1
docker kill fccb74debe855447e82185c3a6cfbec99c9bb3e532cffc457a60fec21f728ed4
exit status: 0
docker rm fccb74debe855447e82185c3a6cfbec99c9bb3e532cffc457a60fec21f728ed4
exit status: 0

I have Docker Desktop for Mac Edge 3.0.1 (50773) and macOS BigSur 11.0.1 (20B50).