feat: generate showcase using docker image #3568

diegomarquezp · 2025-01-14T22:48:30Z

Fixes #2470

Approach

In `hermetic_build/`

Introduce a new configuration entry in the library model where we can specify the folder name when in a monorepo (i.e. allows creation of sdk-platform/showcase instead of sdk-platform/java-showcase)
In gapic_inputs, allow nested parentheses, allow parsing additional protos from java_gapic_library (this is necessary since the original showcase repo doesn't specify additional protos and we use our own custom BUILD.bazel with the additional protos specified in the java library rule).

In `showcase/`

Revamp the generation and verification scripts used by -Penable-golden-tests and -Pupdate to build and use the Hermetic Build Docker image

Effects

New samples/ folder added in showcase
Reformatted all of grpc, proto and gapic libraries in showcase/

sonarqubecloud · 2025-01-15T17:41:32Z

Quality Gate failed for 'gapic-generator-java-root'

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
27.8% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-01-15T17:45:12Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

blakeli0

Thanks Diego! This approach looks good in general, however, have we ever considered treating showcase a dedicate repo instead of a library in sdk-platform-java? Meaning that we create a separate generation_config.yaml in showcase module?

blakeli0 · 2025-01-16T22:12:09Z

.github/scripts/hermetic_library_generation.sh

@@ -95,6 +95,14 @@ changed_libraries=$(python hermetic_build/common/cli/get_changed_libraries.py cr
  --current-generation-config-path="${generation_config}")
 echo "Changed libraries are: ${changed_libraries:-"No changed library"}."

+# do not generate showcase automatically


We probably do want to generate showcase automatically? For example, if we update protoc version in the docker image, we do want the showcase module to be updated.

blakeli0 · 2025-01-16T22:13:44Z

generation_config.yaml

 libraries:
+- api_shortname: showcase
+  folder_name: showcase # prevents java-showcase


I feel it is probably too much effort to introduce a new argument folder_name, just to prevent showcase to be renamed to java-showcase. I think it is OK to rename it.

blakeli0 · 2025-01-16T22:14:03Z

generation_config.yaml

-  - proto_path: google/rpc/context
-  - proto_path: google/shopping/type
-  - proto_path: google/type
+    - proto_path: google/api


Is this change intended?

blakeli0 · 2025-01-16T22:21:39Z

showcase/scripts/resources/BUILD.partial.bazel

+    # fixed.
+    rest_numeric_enums = False,
+    service_yaml = "showcase_v1beta1.yaml",
+    test_deps = [


Can we make this file minimum? For example, I don't think we need any test_deps since we only parses rest_numeric_enums, transport and include_samples.

blakeli0 · 2025-01-16T22:23:56Z

versions.txt

@@ -17,3 +17,6 @@ proto-google-iam-v2:1.45.1:1.45.1
 grpc-google-iam-v2:1.45.1:1.45.1
 google-cloud-core:2.49.1:2.49.1
 google-cloud-shared-dependencies:3.41.1:3.41.1
+gapic-showcase:0.0.0:0.0.1-SNAPSHOT


This is OK as long as release-please does not start bumping versions for showcase. Can you confirm it or maybe @suztomo can confirm?

blakeli0 · 2025-01-16T22:28:54Z

showcase/scripts/generate_showcase.sh

+# download api definitions from googleapis repository
+googleapis_commitish=$(grep googleapis_commitish generation_config.yaml | cut -d ":" -f 2 | xargs)
+api_def_dir=$(mktemp -d)
+git clone https://github.com/googleapis/googleapis.git "${api_def_dir}"


Maybe I missed it but I don't see these repos being cleaned up after the generation is completed.

blakeli0 · 2025-01-16T22:34:23Z

showcase/scripts/generate_showcase.sh

+# looks at sdk-platform-java/showcase/gapic-showcase/pom.xml to extract the
+# version of gapic-showcase
+# see https://github.com/googleapis/gapic-showcase/releases
+showcase_version=$(get_version_from_pom \


Is this used? We should still use it to checkout showcase repo with a release tag.

blakeli0 · 2025-01-16T22:35:34Z

showcase/scripts/generate_showcase.sh

-  --transport "${transport}" 
+echo "generating showcase"
+workspace_name="/workspace"
+docker run \


Since this is using docker now, I don't think we can run them from local Mac or Windows anymore, can you please update the README of the showcase module to indicate it?

blakeli0 · 2025-01-16T22:42:37Z

hermetic_build/common/model/gapic_inputs.py

+}
+# This is a special case for showcase, where the additional protos
+# are not declared in the gapic library instead
+pattern_gapic_library_to_additional_protos = {


Do you mind elaborating on why we need this special case?

diegomarquezp added 17 commits January 10, 2025 02:32

feat: generate showcase with hermetic build image

f61c630

add showcase generation files

99c711f

exclude bom generation

7c0ed51

fix owlbot-cli config

0328d96

fix paths in owlbot cli

c1d1696

add historical bazel file

bb28ae6

fix owlbot yaml

85201ee

parse additional protos also including gapic targets

cad57f6

fix service yaml and gapic config

bfc9bba

adapt to docker image generation

b4aa910

update proto, gapic and grpc

779d011

add samples

d29ce8c

adapt showcase readme

334384a

explain new input in generation config

66e7366

uncomment image building

f0a7d53

ignore full generation when only generating showcase

072ee70

add showcase to versions.txt

0214666

product-auto-label bot added the size: m Pull request size is medium. label Jan 14, 2025

diegomarquezp self-assigned this Jan 14, 2025

diegomarquezp marked this pull request as draft January 14, 2025 23:36

diegomarquezp added 3 commits January 15, 2025 01:03

do not generate showcase automatically on generation config changes

2f47daa

update gapic showcase files

3063dab

fix build rule patterns

e1c270a

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Jan 15, 2025

diegomarquezp added 3 commits January 15, 2025 17:02

lint

1514dd7

skip generation on no target libraries

6476131

exit early when no libraries need to be generated

4880451

diegomarquezp marked this pull request as ready for review January 15, 2025 18:25

diegomarquezp requested a review from blakeli0 January 15, 2025 18:34

diegomarquezp mentioned this pull request Jan 16, 2025

showcase samples should be part of golden tests #2470

Open

blakeli0 reviewed Jan 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: generate showcase using docker image #3568

feat: generate showcase using docker image #3568

diegomarquezp commented Jan 14, 2025

sonarqubecloud bot commented Jan 15, 2025

sonarqubecloud bot commented Jan 15, 2025

blakeli0 left a comment

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

blakeli0 Jan 16, 2025

feat: generate showcase using docker image #3568

Are you sure you want to change the base?

feat: generate showcase using docker image #3568

Conversation

diegomarquezp commented Jan 14, 2025

Approach

In hermetic_build/

In showcase/

Effects

sonarqubecloud bot commented Jan 15, 2025

Quality Gate failed for 'gapic-generator-java-root'

sonarqubecloud bot commented Jan 15, 2025

Quality Gate passed for 'java_showcase_integration_tests'

blakeli0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

In `hermetic_build/`

In `showcase/`