Merge pull request #14 from SoftwareAG/bugfix/Move-internal-bugfixes-to-external-repo

Authored-by: Venkatachalapathy Praveen Raj <[email protected]>

Author: vallabsag

README.md

@@ -5,6 +5,13 @@
 ![API Control Plane Version 11.0.5](https://img.shields.io/badge/API_Control_Plane-11.0.5-purple)
 ![Amazon API Gateway](https://img.shields.io/badge/Amazon-API_Gateway-blue)
 <br>
+![Amazon Lambda Functions](https://img.shields.io/badge/Amazon-Lambda_Functions-purple)
+![Amazon SDK](https://img.shields.io/badge/Amazon-SDK-red)
+![Amazon CloudWatch](https://img.shields.io/badge/Amazon-CloudWatch-green)
+![Amazon CloudTrail](https://img.shields.io/badge/Amazon-CloudTrail-yellow)
+![Amazon EventBridge Scheduler](https://img.shields.io/badge/Amazon-EventBridge_Scheduler-blue)
+<br>
+
 ![Java 17](https://img.shields.io/badge/Java-17-orange?style=for-the-badge&logo=java&logoColor=white)
 ![Gradle 7.4.2](https://img.shields.io/badge/Gradle-7.4.2-DD0031?style=for-the-badge&logo=java&logoColor=white)
 ![Spring](https://img.shields.io/badge/Spring-6DB33F?style=for-the-badge&logo=spring&logoColor=white)
@@ -18,25 +25,27 @@ This repository holds an agent implementation Java project for connecting Amazon
 5. Retrieving metrics from Amazon API Gateway to API Control Plane.
 
 This project is developed using **Java 17** and **Gradle 7.4.2**<br>
-If you plan to upgrade *Gradle*, ensure that you also upgrade the supported *Java version* accordingly. For details about the compatibility between *Java* and *Gradle* versions, see [Compatibility Matrix](https://docs.gradle.org/current/userguide/compatibility.html).
+If you use different versions of *Java* or *Gradle*, ensure they are compatible with each other. For details about the compatibility between Java and Gradle versions, see [Compatibility Matrix](https://docs.gradle.org/current/userguide/compatibility.html).
 
 ## Table of Contents
 - [Implementation Overview](#implementation-overview)
   - [Core Implementation Logic](docs/core-logic.md)
   - [Customize the Core Implementation Logic](common/)
+  - [Differences between Spring Boot Application and Functions (AWS lambda) Flavour](docs/differences.md)
 - [How is this Repository Structured?](docs/repo-structure.md)
 - [Co-relation Between Amazon API Gateway and API Control Plane Terminologies](docs/corelation.md)
 - [How to Build the Gradle Project?](devops/)
 - [How to Deploy and Run the AWS Agent as a Spring Boot application in Docker?](application/)
   - [Authentication](docs/authentication.md)
   - [How to Create the Runtime Type in API Control Plane?](docs/runtime_service_mgmt_api.md)
 - [How to Deploy and Run the AWS Agent in AWS Lambda?](functions/)
+  - [Lambda Functions-Environment variables](docs/lambda-properties.md) 
 - [Best Practices](docs/best-practices.md)
 
 
 ## Implementation Overview
 
-The implementation utilizes the **Manual** approach of Agent SDK and provides options for deploying the AWS agent as a stand-alone application. For details about the approaches and deployment modes, see [Agent SDK](https://docs.webmethods.io/apicontrolplane/agent_sdk/chapter2wco/#gsc.tab=0) documentation.
+The implementation utilizes the **Manual** approach of Agent SDK and provides options for deploying the AWS agent as a stand-alone application. For details about the approaches and deployment modes, see **Agent** section in [API Control Plane](https://docs.webmethods.io/#api-management-saas) documentation.
 The implementation leverages the following Amazon services:
 - **AWS SDK** for connection management and authentication. 
 - **Amazon CloudWatch** service for retrieving API metrics.
@@ -49,6 +58,8 @@ The AWS agent can be deployed in the following ways:
 - **Spring Boot application**
 - **AWS Lambda**<br>
 
+To understand the differences between Spring Boot application and AWS Lambda deployment modes, see [Differences](docs/differences.md).
+
 For a detailed understanding of how the agent for Amazon API Gateway is implemented, see [Core Implementation Logic](docs/core-logic.md).
 
 The AWS agent developer can utilize this repository in the following ways:

application/build.gradle

@@ -11,7 +11,6 @@ buildscript {
 plugins {
     id 'java'
     id 'org.springframework.boot' version '3.1.6'
-    id 'io.spring.dependency-management' version '1.1.4'
     id "io.freefair.lombok" version "6.4.3"
 }
 

application/readme.md

@@ -1,5 +1,6 @@
 package com.softwareag.controlplane.agent.aws.configuration;
 
+import jakarta.validation.constraints.Min;
 import lombok.Getter;
 import lombok.Setter;
 import org.springframework.boot.context.properties.ConfigurationProperties;
@@ -20,6 +21,7 @@ public class AWSProperties {
     private String stage;
     @NotNull
     private String metricsByDataOrStatistics;
+    @Min(600)
     private int metricsSyncBufferIntervalSeconds;
     private int assetsSyncBufferIntervalSeconds;
 }

application/src/main/java/com/softwareag/controlplane/agent/aws/configuration/AWSProperties.java

@@ -2,7 +2,7 @@ spring.main.web-application-type=NONE
 aws.region=
 aws.stage=
 aws.metrics-by-data-or-statistics=statistics
-aws.metrics-sync-buffer-interval-seconds=60
+aws.metrics-sync-buffer-interval-seconds=600
 aws.assets-sync-buffer-interval-seconds=300
 apicp.publish-assets=true
 apicp.sync-assets=true

application/src/main/resources/application.properties

@@ -1,23 +1,21 @@
 ext {
-    jacksonVersion = '2.15.2'
-    micrometerVersion = '1.11.3'
-    log4jVersion = '2.22.0'
-    nettyVersion = '4.1.97.Final'
-    nettyCodecVersion = '4.1.100.Final'
-    nettyHandlerVersion = '4.1.101.Final'
-    springVersion = '6.0.14'
-    springBootVersion = '3.1.3'
-    springSecurityVersion = '6.1.3'
-    junitPlatformVersion = '1.9.3'
-    junitJupiterVersion = '5.9.3'
-    junitMockitoVersion = '5.3.1'
-    bytebuddyVersion = '1.14.6'
-    apacheCommonsLang3Version = '3.12.0'
-    apacheLog4jTemplateVersion = '2.20.0'
+    jacksonVersion = '2.17.1'
+    micrometerVersion = '1.13.1'
+    log4jVersion = '2.23.1'
+    nettyVersion = '4.1.111.Final'
+    springVersion = '6.1.10'
+    springSecurityVersion = '6.3.1'
+    springBootVersion = '3.3.1'
+    junitPlatformVersion = '1.10.2'
+    junitJupiterVersion = '5.10.2'
+    junitMockitoVersion = '5.11.0'
+    bytebuddyVersion = '1.14.17'
+    apacheCommonsLang3Version = '3.14.0'
+    apacheLog4jTemplateVersion = '2.23.1'
     awsAuthVersion = '2.17.267'
     awsSdkVersion = '2.23.18'
-    tomcatVersion = '10.1.16'
-    elasticsearchVersion = '7.17.15'
+    tomcatVersion = '10.1.25'
+    elasticsearchVersion = '8.13.4'
 
     junit = [
             "org.junit.platform:junit-platform-commons:${junitPlatformVersion}",
@@ -38,12 +36,12 @@ ext {
 
     apache = [
             "org.apache.commons:commons-lang3:$apacheCommonsLang3Version",
-            'commons-codec:commons-codec:1.11',
+            'commons-codec:commons-codec:1.16.1',
             "org.apache.logging.log4j:log4j-layout-template-json:$apacheLog4jTemplateVersion",
     ]
 
     json = [
-            'org.json:json:20231013'
+            'org.json:json:20240303'
     ]
 
     awsApiGateway = [
@@ -92,12 +90,14 @@ ext {
             "org.springframework.boot:spring-boot-actuator:$springBootVersion",
             "org.springframework.boot:spring-boot:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-tomcat:$springBootVersion",
-            "org.springframework.session:spring-session-core:3.0.1",
+            "org.springframework.session:spring-session-core:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-data-elasticsearch:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-validation:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-security:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-webflux:$springBootVersion",
             "org.springframework.boot:spring-boot-starter-reactor-netty:$springBootVersion",
+            "org.elasticsearch.client:elasticsearch-rest-client:$elasticsearchVersion",
+            "co.elastic.clients:elasticsearch-java:$elasticsearchVersion",
     ]
 
     spring = [
@@ -138,7 +138,7 @@ ext {
     ]
 
     log4j = [
-            "org.slf4j:slf4j-api:2.0.9",
+            "org.slf4j:slf4j-api:2.0.13",
             "org.apache.logging.log4j:log4j-core:$log4jVersion",
             "org.apache.logging.log4j:log4j-api:$log4jVersion",
     ]
@@ -155,10 +155,10 @@ ext {
             "io.netty:netty-buffer:$nettyVersion",
             "io.netty:netty-transport:$nettyVersion",
             "io.netty:netty-resolver:$nettyVersion",
-            "io.netty:netty-codec:$nettyCodecVersion",
-            "io.netty:netty-handler:$nettyHandlerVersion",
+            "io.netty:netty-codec:$nettyVersion",
+            "io.netty:netty-handler:$nettyVersion",
             "io.netty:netty-transport-native-unix-common:$nettyVersion",
-            "io.netty:netty-codec-http2:$nettyCodecVersion",
+            "io.netty:netty-codec-http2:$nettyVersion",
             "io.netty:netty-resolver-dns:$nettyVersion",
             "io.netty:netty-codec-dns:$nettyVersion",
             "io.netty:netty-resolver-dns-native-macos:$nettyVersion",
@@ -170,11 +170,11 @@ ext {
     ]
 
     reactor = [
-            "io.projectreactor.netty:reactor-netty:1.0.33",
-            "io.projectreactor:reactor-core:3.5.9",
+            "io.projectreactor.netty:reactor-netty:1.1.20",
+            "io.projectreactor:reactor-core:3.6.7",
             "org.reactivestreams:reactive-streams:1.0.4",
-            "io.projectreactor.netty:reactor-netty-http:1.1.13",
-            "io.projectreactor.netty:reactor-netty-core:1.1.10",
+            "io.projectreactor.netty:reactor-netty-http:1.1.20",
+            "io.projectreactor.netty:reactor-netty-core:1.1.20",
     ]
 
     lambdaFunctions = [
@@ -195,6 +195,7 @@ ext {
     ]
 
     springBootApplication = [
+            "org.apache.logging.log4j:log4j-slf4j2-impl:$log4jVersion",
             spring,
             springBoot,
             apache,
@@ -244,5 +245,3 @@ ext {
     ]
 
 }
-
-

dependencies.gradle

@@ -10,7 +10,7 @@ AWS_METRICS_BY_DATA_OR_STATISTICS=statistics
 # Time taken in seconds for the API metrics in AWS API Gateway to sync to AWS CloudWatch
 AWS_METRICS_SYNC_BUFFER_INTERVAL_SECONDS=60
 # Time taken in seconds for the API events in AWS API Gateway to sync to AWS CloudTrail
-AWS_ASSETS_SYNC_BUFFER_INTERVAL_SECONDS=300
+AWS_ASSETS_SYNC_BUFFER_INTERVAL_SECONDS=600
 
 
 # AWS Agent Configurations

devops/docker-compose/.env

@@ -14,10 +14,10 @@ services:
       - aws.metrics-sync-buffer-interval-seconds=${AWS_METRICS_SYNC_BUFFER_INTERVAL_SECONDS}
       - apicp.publish-assets=${APICP_PUBLISH_ASSETS}
       - apicp.sync-assets=${APICP_SYNC_ASSETS}
-      - apicp.sync-metrics=${APICP_SEND_METRICS}
-      - apicp.heartbeat-sync-interval-seconds=${APICP_HEARTBEAT_SEND_INTERVAL_SECONDS}
+      - apicp.send-metrics=${APICP_SEND_METRICS}
+      - apicp.heart-beat-send-interval-seconds=${APICP_HEART_BEAT_SEND_INTERVAL_SECONDS}
       - apicp.assets-sync-interval-seconds=${APICP_ASSETS_SYNC_INTERVAL_SECONDS}
-      - apicp.metrics-sync-interval-seconds=${APICP_METRICS_SEND_INTERVAL_SECONDS}
+      - apicp.metrics-send-interval-seconds=${APICP_METRICS_SEND_INTERVAL_SECONDS}
       - apicp.runtime.name=${APICP_RUNTIME_NAME}
       - apicp.runtime.description=${APICP_RUNTIME_DESCRIPTION}
       - apicp.runtime.region=${APICP_RUNTIME_REGION}

devops/docker-compose/docker-compose.yml

@@ -5,16 +5,17 @@ Let’s look at a sample scenario through which you can build a Gradle project u
 
 **Pre-requisites** 
 
-Ensure that you have:
+Ensure that you have
 
 - Cloned **webmethods-api-control-plane-agent-aws** Git Hub repository using any GIT client.
-- Downloaded the **IBM webMethods API Control Plane 3rd Party Runtime Agent SDK** zip file from [IBM Integration Community](https://community.ibm.com/community/user/integration/communities/community-home/all-news?communitykey=82b75916-ed06-4a13-8eb6-0190da9f1bfa&LibraryFolderKey=&DefaultView=&8e5b3238-cf51-4036-aa29-601a6cd3e1b3=eyJsaWJyYXJ5ZW50cnkiOiJhMDAxMzc2Ni0wMWNiLTQ3ZDUtYmQ3Yi0wMTkyNDllNDZjYzIifQ==).
-- Copy and pasted the Agent SDK (api, core, and model) Jars from the extracted folder into the [Libs](../libs) directory in the cloned (api-controlplane-agent-aws) repository.
+- Empower portal login credentials to download the Agent SDK Jars. For details, see [How to access Agent SDK Jars](https://docs.webmethods.io/apicontrolplane/agent_sdk/chapter2wco#ta-implementing_agentsdk).
+- Installed **Java 17** and Gradle **7.4.2**<br>
+  If you use different versions of *Java* or *Gradle*, ensure they are compatible with each other. For details about the compatibility between Java and Gradle versions, see [Compatibility Matrix](https://docs.gradle.org/current/userguide/compatibility.html).
 
 
 **To build the Gradle project**
 
-1.	Unzip Agent SDK folder downloaded from IBM Integration Community and place the (api, core, and model) Jars under *lib* folder in the cloned repository.
+1.	Unzip Agent SDK folder downloaded from the Empower portal and place the (api, core, and model) Jars under *lib* folder in the cloned repository.
 
 2.	Open Visual Studio Code editor.
 
@@ -28,11 +29,19 @@ Ensure that you have:
 
    *Build Successful* message appears, and the following Jars are created for the application and Lambda functions:
 
-   **Spring Boot application**: **application-<version>-SNAPSHOT** is created at *application / build / libs* <br>
+   **Spring Boot application**: **application-version-SNAPSHOT** is created at *application / build / libs* <br>
 
    **AWS Lambda**: <br>
 	- **lambda-layer.zip** is created at *functions / build / libs*, which includes all dependencies to run lambda function.<br>
 	- **send-asset.jar** is created at *functions / send–asset  / build / libs* <br>
 	- **send-heartbeat.jar** is created at *functions / send-heartbeat / build / libs* <br>
 	- **send-metrics.jar** is created at *functions / send-metrics / build / libs*
+  
+  **Troubleshooting Tips**: For permission denied issues: <br>
+     		1. Verify if the Gradle has execution permissions. <br>
+	 	2. Run ``` gradle build ``` (or) ``` ./gradlew build ```<br>
+	 	3. If the issue persists, run:
+	    	``` sh ./gradlew build ```
+	    	(or) 
+		``` bash ./gradlew build ```
 

devops/readme.md

@@ -9,8 +9,7 @@ Let’s understand the logic of how the agent for Amazon API Gateway is implemen
 2. **Retrieving Amazon API Gateway’s health status and sending it to API Control Plane**.
 
     1. The agent retrieves all REST APIs from the Amazon API Gateway.
-    2. For each API, the agent verifies if the API is associated with the specified stage (from the properties).
-    3. If the API is associated with the stage, the agent sends the health status (heartbeat) of the runtime (stage) as *active* to API Control Plane.<br><br>
+    2. The agent verifies if at least one API is deployed to the specified stage in the given region (as per the properties). If an API is deployed, the agent sends          the runtime (stage) health status as *active* to the API Control Plane.<br><br>
 
 3. **Publishing Amazon API Gateway’s assets to API Control Plane**. 
 

docs/core-logic.md

@@ -4,39 +4,6 @@
 | Amazon API Gateway | API Control Plane | Description |
 |--------------------|-------------------|-------------|
 | Stage |Runtime | A stage in Amazon API Gateway is considered as a runtime in API Control Plane. |
-| Deployment | API Versions | A deployment in Amazon API Gateway is a snapshot that tracks an API's version. An API in Amazon API Gateway can have multiple deployments, but only one can be active per stage at a time. <br><br> In the API Control Plane, each deployment is called an API version. For example, if an API has two deployments published to two different stages, it is treated as two different versions of the same API in API Control Plane.  |
-
-## How to identify a unique stage in Amazon Gateway?
-
-Every unique *stage* within each region in AWS API Gateway is considered as a separate runtime in the API Control Plane. Hence, you must deploy an agent for each unique stage.
-
-For example: Assume that you have two accounts in **AWS** and each account has **AWS API Gateway** with APIs in the following regions and stages:
-
-**Account 1**:
-- *Region 1*: us-east-1
-    - *Stage*: dev (In APICP, this stage appears as **dev-account1-ue1** runtime from account 1 in region us-east-1)
-       - *API*s: API1, API2
-    - *Stage*: qa (In APICP, this stage appears as **qa-account1-ue1** runtime from account 1 in region us-east-1)
-       - *APIs*: API1, API2
-- *Region 2*: us-east-2
-    - *Stage*: dev (In APICP, this stage appears as **dev-account1-uw2** runtime from account 1 in region us-east-2)
-       - APIs: API3
-    - *Stage*: preprod (In APICP, this stage appears as **preprod-account1-uw2** runtime from account 1 in region us-east-2)
-       - *APIs*: API3
-    - *Stage*: prod (In APICP, this stage appears as **prod-account1-uw2** runtime from account 1 in region us-east-2)
-       - *APIs*: API3
-
-**Account 2**:
-- *Region 1*: us-east-1
-    - *Stage*: dev (In APICP, this stage appears as **dev-account2-ue1** runtime from account 2 in region us-east-1)
-      - *APIs*: API4
-    - *Stage*: prod (In APICP, this stage appears as **prod-account2-ue1** runtime from account 2 in region us-east-1)
-      - *APIs*: API4
-
-
-Please see the following diagram to understand how the stages are treated as different runtimes in the API Control Plane.
-
-![](../docs/images/aws_stages.png)
-
-
+| Deployment | API Versions | A deployment in Amazon API Gateway is a snapshot that tracks an API's version. An API in Amazon API Gateway can have multiple deployments, but only one can be active per stage at a time. <br><br> In the API Control Plane, each deployment is called an API version. For example, if an API has two deployments published to two different stages, it is treated as two different versions of the same API in API Control Plane.<br>When there are multiple deployments of an API to a stage, only the active deployment is synchronized with API Control Plane.   |
 
+A *Stage* in Amazon API Gateway created within a *region* and belonging to an *AWS account* is treated as an *individual runtime* in API Control Plane. As per the current implementation of the AWS agent, it requires one agent to be deployed for every stage-runtime mapping.

docs/corelation.md

@@ -0,0 +1,13 @@
+## Differences between Spring Boot Application and Functions (AWS lambda) Flavour
+
+The following table lists the differences between the Spring Boot Application and Functions (AWS lambda) Flavour
+
+
+| **Key Points**  | **Spring Boot Application** | **Functions (AWS Lambda)** |
+|------------------|-----------------------------|----------------------------|
+| **Deployment** | Deployed like a regular server. | Deployed as Function as a Service (FaaS) on AWS Lambda|
+| **Maintenance and Availability** | **High maintenance**<br> User must manage deployment for full availability. | **Low maintenance**<br> For FaaS flavours, cloud hosting service providers such as *AWS* handles availability once deployed. |
+| **Resource Consumption and Cost** | Server runs 24x7, even with sparse utilization. | AWS provider manages and runs the code only when invoked. Charges are incurred only based on number of invocations and runtime. <br> More **cost effective** compared to regular hosted solutions.|
+| **Implementation** | A *single implementation* of spring boot application handles all three use cases of an agent: *assets synchronization*, *heartbeats synchronization*, and *metrics synchronization*. | Each Lambda function handles one use case of an agent. To achieve *assets synchronization*, *heartbeats synchronization*, and *metrics synchronization*, deploy three separate Lambda functions.  |
+| **Schedule** | The frequency of the synchronization of heartbeats, assets, and metrics from the runtime to API Control Plane is maintained by the agent application code. Each activity gets triggered based on the *configured synchronization intervals*. | The schedule of the Lambda function is maintained in AWS component, **Amazon EventBridge**. The Lambda function gets triggered based on the intervals configured in EventBridge.<br> The schedule configured in EventBridge should match with with the synchronization intervals configured in the Lambda environment properties for smooth functioning. |
+| **Suitability** | Ideal for on-prem deployment flavour. | Suitable for hosted solutions. <br>Best in terms of maintenance and cost compared to other hosted flavours. |