Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parent dependency check on samples #195

Closed
wants to merge 14 commits into from
Closed

Add parent dependency check on samples #195

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
aa0e1dd
ignore cached remote samples
michael-valdron Jul 14, 2023
81113bf
nightly script for building sample parent stack list
michael-valdron Jul 14, 2023
1abda83
init check parent stacks script
michael-valdron Jul 14, 2023
1321736
convert nightly parents build to on the fly parents build.
michael-valdron Jul 18, 2023
8d71e6c
multi version support for build parent stacks
michael-valdron Jul 19, 2023
9c86d26
ignore any local parents file.
michael-valdron Jul 25, 2023
76b1f4e
rename check_parent_stacks.sh to build_parents_file.sh, build_parents…
michael-valdron Jul 25, 2023
563d791
validate devfile schemas test suite parameters now changeable
michael-valdron Jul 25, 2023
88722d5
validate samples workflow
michael-valdron Jul 25, 2023
a5da72e
add check for if there are samples to be checked under validate sampl…
michael-valdron Jul 26, 2023
3464e43
add README instruction on dependency check changes
michael-valdron Jul 26, 2023
959de4d
yq 4.x installation guide link added to testing prerequisites list.
michael-valdron Jul 31, 2023
1b375fb
add missing link to test suite
michael-valdron Jul 31, 2023
7c0d08c
validate_devfile_schemas script accepts relative or absolute paths
michael-valdron Jul 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .github/workflows/validate-samples.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#
# Copyright 2023 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Validate child samples

on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: 0 5 * * *

concurrency:
group: ${{ github.workflow }}-${{ github.event.number || github.ref }}
cancel-in-progress: true

env:
MINIKUBE_VERSION: "v1.29.0"
KUBERNETES_VERSION: "v1.25.2"
TEST_DELTA: false

jobs:
validate-devfile-schema:
name: validate devfile schemas
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Install Go
uses: actions/setup-go@v3
with:
go-version: "1.19"

- name: Install Ginkgo
run: go install -mod=mod github.com/onsi/ginkgo/v2/[email protected]

- name: Test delta if on a pull request
if: ${{ github.event_name == 'pull_request' }}
run: echo "TEST_DELTA=true" >> $GITHUB_ENV

- name: Build parents file and get child samples
run: echo "STACKS=$(bash tests/build_parents_file.sh)" >> $GITHUB_ENV

- name: Validate samples
if: ${{ env.STACKS != '' }}
run: STACKS_DIR=$(pwd)/samples/.cache bash tests/validate_devfile_schemas.sh
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ index-generator
registry-support/
.idea/
devfile-web/
samples/.cache
parents.yaml
24 changes: 24 additions & 0 deletions tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,29 @@
# Devfile Registry Testing

## Dependency check

### Prerequisites

- Ensure `yq 4.3x` is installed

### Running the build script

- This script performs three actions
- Clones samples from provided `extraDevfileEntries.yaml` under `samples/.cache`
- Creates a `parents.yaml` which contains the dependency tree for parent stacks
- Outputs the child sample paths of parent stacks, `TEST_DELTA=true` will result in only outputting child samples which have changed parent stacks
- The build script takes one optional argument and works off of the current working directory
- `bash tests/build_parents_file.sh`, default samples file is `extraDevfileEntries.yaml`
- `bash tests/build_parents_file.sh <path_to_extraDevfileEntries>`

### Use with testing

- One can test the child samples using the [validate_devfile_schemas]() test suite by performing the following:
```sh
export STACKS=$(bash tests/build_parents_file.sh)
STACKS_DIR=samples/.cache bash tests/validate_devfile_schemas.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried to test the script locally following the instructions:

  • Upon root dir of project I had this error:
$ STACKS_DIR=samples/.cache bash tests/validate_devfile_schemas.sh

+ stacksDir=samples/.cache
+ stackDirs='nodejs-basic code-with-quarkus go-basic java-springboot-basic python-basic'
+ ginkgo run --procs 2 tests/validate_devfile_schemas -- -stacksPath samples/.cache -stackDirs 'nodejs-basic code-with-quarkus go-basic java-springboot-basic python-basic'
Failed to compile validate_devfile_schemas:

go: cannot find main module, but found .git/config in <mypath>/michael_valdron/devfile-registry
        to create a module there, run:
        go mod init

Ginkgo ran 1 suite in 12.799434ms
Test Suite Failed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thepetk What is your output for go version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes 7c0d08c so STACKS_DIR can be either a relative or absolute path when executing validate_devfile_schemas.sh.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thepetk What is your output for go version?

@michael-valdron output is
go version go1.19.1 linux/amd64

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thepetk What is your output for go version?

@michael-valdron output is go version go1.19.1 linux/amd64

@thepetk try running it again with a go 1.18.x runtime, go 1.19.1 could be having issues running it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally got it run successfully :)

One thing I had to do other than those steps, was to add a go.work file in my repo's root dir.

It seems that is working fine :)

One test I tried to make is to update the go-basic sample inside the extraDevfileEntries. I've tried to include multiple sample versions:

  - name: go-basic
    displayName: Basic Go
    description: Go 1.16 application
    icon: https://go.dev/blog/go-brand/Go-Logo/SVG/Go-Logo_Blue.svg
    tags:
      - Go
    projectType: Go
    language: Go
    provider: Red Hat
    versions:
      - version: 1.0.2
        schemaVersion: 2.1.0
        git:
          remotes:
            origin: https://github.com/devfile-samples/devfile-sample-go-basic.git
      - version: 2.1.0
        schemaVersion: 2.2.0
        default: true
        git:
          checkoutFrom:
            revision: update_devfile
          remotes:
            origin: https://github.com/thepetk/devfile-sample-go-basic.git

In the parent file I had:

  - name: go
    children:
      - name: go-basic
        version: 1.0.2
  - name: go
    version: 2.1.0
    children:
      - name: go-basic
        version: 2.1.0

(Which is ok)

During testing I was expecting to see the test fail for one of the golang children (go-basic version 2.1.0). Although I had an issue with the quarkus. The trace of the test was:

• [FAILED] [1.984 seconds]
validate stacks follow the schema [It] stack: code-with-quarkus
/home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:50

  Timeline >>
  {/home/tpetkos/github/michael_valdron/devfile-registry/samples/.cache/go-basic/2.1.0/devfile.yaml  [] <nil> <nil> []  <nil> <nil> map[] <nil>}
  [FAILED] in [It] - /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812
  << Timeline

  [FAILED] Expected
      <*errors.withStack | 0xc0004dde60>: {
          error: <*errors.withMessage | 0xc000693e40>{
              cause: <*multierror.Error | 0xc000693e20>{
                  Errors: [
                      <*errors.errorString | 0xc000450cd0>{
                          s: "Some Commands are already defined in parent: build-image, deploy, deployk8s. If you want to override them, you should do it in the parent scope.",
                      },
                  ],
                  ErrorFormat: nil,
              },
              msg: "failed to populateAndParseDevfile",
          },
          stack: [0x14db668, 0x14e2e38, 0x156ad58, 0x1530e1b, 0x1543958, 0x46d841],
      }
  to be nil
  In [It] at: /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812

Copy link
Member Author

@michael-valdron michael-valdron Aug 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During testing I was expecting to see the test fail for one of the golang children (go-basic version 2.1.0).
Although I had an issue with the quarkus. The trace of the test was:

• [FAILED] [1.984 seconds]
validate stacks follow the schema [It] stack: code-with-quarkus
/home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:50

  Timeline >>
  {/home/tpetkos/github/michael_valdron/devfile-registry/samples/.cache/go-basic/2.1.0/devfile.yaml  [] <nil> <nil> []  <nil> <nil> map[] <nil>}
  [FAILED] in [It] - /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812
  << Timeline

  [FAILED] Expected
      <*errors.withStack | 0xc0004dde60>: {
          error: <*errors.withMessage | 0xc000693e40>{
              cause: <*multierror.Error | 0xc000693e20>{
                  Errors: [
                      <*errors.errorString | 0xc000450cd0>{
                          s: "Some Commands are already defined in parent: build-image, deploy, deployk8s. If you want to override them, you should do it in the parent scope.",
                      },
                  ],
                  ErrorFormat: nil,
              },
              msg: "failed to populateAndParseDevfile",
          },
          stack: [0x14db668, 0x14e2e38, 0x156ad58, 0x1530e1b, 0x1543958, 0x46d841],
      }
  to be nil
  In [It] at: /home/tpetkos/github/michael_valdron/devfile-registry/tests/validate_devfile_schemas/validate_devfile_schemas_test.go:61 @ 08/01/23 17:18:55.812

This is an interesting result, it is reporting the stack as being code-with-quarkus but the error in the test seems to be the expected one with go-basic 2.1.0, I'll need to check if there is a problem with the inputs or if there is a bug within this test suite which is causing the wrong stack name to be reported.

```

## Validating non-terminating images

### Prerequisites
Expand Down
240 changes: 240 additions & 0 deletions tests/build_parents_file.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
#!/bin/bash

# default samples file path
samples_file=$(pwd)/extraDevfileEntries.yaml
# Cached remote samples directory
samples_dir=$(pwd)/samples/.cache
# default stacks directory
stacks_dir=${STACKS:-"$(pwd)/stacks"}
parents_file=$(pwd)/parents.yaml

# YAML query cmd path
YQ_PATH=${YQ_PATH:-yq}

# Read samples file as first argument
# if unset use default samples file path
if [ ! -z "${1}" ]; then
samples_file=${1}
fi

# Clones remote samples into cache directory
clone_samples() {
samples_len=$($YQ_PATH eval '.samples | length' ${samples_file})

# Removes old cached samples directory
if [ -d ${samples_dir} ]; then
rm -rf ${samples_dir}
fi

for ((s_idx=0;s_idx<${samples_len};s_idx++)); do
name=$($YQ_PATH eval .samples.${s_idx}.name ${samples_file})
versions=($($YQ_PATH eval .samples.${s_idx}.versions.[].version ${samples_file}))

# Iterate through sample versions if sample has multi version support
if [ ${#versions[@]} -ne 0 ]; then
for ((v_idx=0;v_idx<${#versions[@]};v_idx++)); do
remote_url=$($YQ_PATH eval .samples.${s_idx}.versions.${v_idx}.git.remotes.origin ${samples_file})

git clone --depth=1 ${remote_url} ${samples_dir}/${name}/${versions[$v_idx]}
done
else
remote_url=$($YQ_PATH eval .samples.${s_idx}.git.remotes.origin ${samples_file})

git clone --depth=1 ${remote_url} ${samples_dir}/${name}
fi
done
}

get_parent_version() {
devfile=$1
name=$2
version=$($YQ_PATH eval .parent.version ${devfile})

if [ "${version}" == "null" ] && [ -f "${stacks_dir}/${name}/stack.yaml" ]; then
version=$($YQ_PATH eval '.versions | filter(.default) | .[0].version' ${stacks_dir}/${name}/stack.yaml)
fi

echo ${version}
}

# Get parent index if exists, else returns -1
parent_index() {
name=$1
version=$2

if [ -z "${version}" ]; then
result=$($YQ_PATH eval ".parents | to_entries | filter(.value.name == \"${name}\") | .[0].key" ${parents_file})
else
result=$($YQ_PATH eval ".parents | to_entries | filter(.value.name == \"${name}\" and .value.version == \"${version}\") | .[0].key" ${parents_file})
fi

if [ "${result}" == "null" ]; then
echo "-1"
else
echo ${result}
fi
}

# Get child index if exists, else returns -1
child_index() {
parent_idx=$1
name=$2
version=$3

if [ -z "${version}" ]; then
result=$($YQ_PATH eval ".parents.[${parent_idx}].children | to_entries | filter(.value.name == \"${name}\") | .[0].key" ${parents_file})
else
result=$($YQ_PATH eval ".parents.[${parent_idx}].children | to_entries | filter(.value.name == \"${name}\" and .value.version == \"${version}\") | .[0].key" ${parents_file})
fi

if [ "${result}" == "null" ]; then
echo "-1"
else
echo ${result}
fi
}

# Builds sample parents
build_parents() {
parent_name=$1
parent_version=$2

if [ "${parent_version}" == "null" ]; then
parent_version=""
fi

if [ "${parent_name}" != "null" ]; then
if [ ! -f ${parents_file} ]; then
$YQ_PATH eval -n ".parents[0].name = \"${parent_name}\"" > ${parents_file}
if [ "${parent_version}" != "" ]; then
$YQ_PATH eval ".parents[0].version = \"${parent_version}\"" -i ${parents_file}
fi

return
fi

if [ "$($YQ_PATH eval .parents ${parents_file})" == "null" ]; then
$YQ_PATH eval ".parents[0].name = \"${parent_name}\"" -i ${parents_file}
if [ "${parent_version}" != "" ]; then
$YQ_PATH eval ".parents[0].version = \"${parent_version}\"" -i ${parents_file}
fi

return
fi

parent_idx=$(parent_index ${parent_name} ${parent_version})
if [ "${parent_idx}" == "-1" ]; then
next_idx=$($YQ_PATH eval ".parents | length" ${parents_file})
$YQ_PATH eval ".parents[${next_idx}].name = \"${parent_name}\"" -i ${parents_file}
if [ "${parent_version}" != "" ]; then
$YQ_PATH eval ".parents[${next_idx}].version = \"${parent_version}\"" -i ${parents_file}
fi
fi
else
return 1
fi
}

# Builds children of parent stacks
build_children() {
parent_name=$1
parent_version=$2
sample_name=$3
sample_version=$4

parent_idx=$(parent_index ${parent_name} ${parent_version})

if [ "$($YQ_PATH eval .parents[${parent_idx}].children ${parents_file})" == "null" ]; then
$YQ_PATH eval ".parents[${parent_idx}].children[0].name = \"${sample_name}\"" -i ${parents_file}
if [ "${sample_version}" != "" ]; then
$YQ_PATH eval ".parents[${parent_idx}].children[0].version = \"${sample_version}\"" -i ${parents_file}
fi

return
fi

child_idx=$(child_index ${parent_idx} ${sample_name} ${sample_version})
if [ "${child_idx}" == "-1" ]; then
next_idx=$($YQ_PATH eval ".parents[${parent_idx}].children | length" ${parents_file})
$YQ_PATH eval ".parents[${parent_idx}].children[${next_idx}].name = \"${sample_name}\"" -i ${parents_file}
if [ "${sample_version}" != "" ]; then
$YQ_PATH eval ".parents[${parent_idx}].children[${next_idx}].version = \"${sample_version}\"" -i ${parents_file}
fi
fi
}

build_parents_file() {
samples_len=$($YQ_PATH eval '.samples | length' ${samples_file})

for ((s_idx=0;s_idx<${samples_len};s_idx++)); do
sample_name=$($YQ_PATH eval .samples.${s_idx}.name ${samples_file})
sample_versions=($($YQ_PATH eval .samples.${s_idx}.versions.[].version ${samples_file}))

# Iterate through sample versions if sample has multi version support
if [ ${#sample_versions[@]} -ne 0 ]; then
for ((v_idx=0;v_idx<${#sample_versions[@]};v_idx++)); do
devfile=${samples_dir}/${sample_name}/${sample_versions[$v_idx]}/devfile.yaml
parent_name=$($YQ_PATH eval .parent.id ${devfile})
parent_version=$(get_parent_version ${devfile} ${parent_name})
build_parents ${parent_name} ${parent_version}

if [ $? -eq 0 ]; then
build_children "${parent_name}" "${parent_version}" "${sample_name}" "${sample_versions[$v_idx]}"
fi
done
else
devfile=${samples_dir}/${sample_name}/devfile.yaml
parent_name=$($YQ_PATH eval .parent.id ${devfile})
parent_version=$(get_parent_version ${devfile} ${parent_name})
build_parents ${parent_name} ${parent_version}

if [ $? -eq 0 ]; then
build_children "${parent_name}" "${parent_version}" "${sample_name}" ""
fi
fi
done
}

# Gets the children sample paths of parents.
# When TEST_DELTA is set to true, only children of parents
# with changes are returned.
get_children_of_parents() {
stack_dirs=$(bash $(pwd)/tests/get_stacks.sh)
children=()

for stack_dir in $stack_dirs; do
if [ "$(basename $(dirname $stack_dir))" == "." ]; then
stack_name=$(basename $stack_dir)

names=($($YQ_PATH eval ".parents | filter(.name == \"${stack_name}\") | .[0].children.[].name" ${parents_file}))
versions=($($YQ_PATH eval ".parents | filter(.name == \"${stack_name}\") | .[0].children.[].version" ${parents_file}))
else
stack_name=$(basename $(dirname $stack_dir))
stack_version=$(basename $stack_dir)

names=($($YQ_PATH eval ".parents | filter(.name == \"${stack_name}\" and .version == \"${stack_version}\") | .[0].children.[].name" ${parents_file}))
versions=($($YQ_PATH eval ".parents | filter(.name == \"${stack_name}\" and .version == \"${stack_version}\") | .[0].children.[].version" ${parents_file}))
fi


for ((c_idx=0;c_idx<${#names[@]};c_idx++)); do
if [ "${versions[$c_idx]}" == "null" ]; then
children+=("${names[$c_idx]}")
else
children+=("${names[$c_idx]}/${versions[$c_idx]}")
fi
done
done

echo ${children[@]}
}

clone_samples

if [ -f ${parents_file} ]; then
rm ${parents_file}
fi

build_parents_file

echo $(get_children_of_parents)
5 changes: 3 additions & 2 deletions tests/validate_devfile_schemas.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

set -x

stackDirs=$(bash "$(pwd)/tests/get_stacks.sh")
stacksDir=${STACKS_DIR:-"$(pwd)/stacks"}
stackDirs=${STACKS:-"$(bash "$(pwd)/tests/get_stacks.sh")"}

ginkgo run --procs 2 \
tests/validate_devfile_schemas -- -stacksPath "$(pwd)"/stacks -stackDirs "$stackDirs"
tests/validate_devfile_schemas -- -stacksPath ${stacksDir} -stackDirs "$stackDirs"