feat(cli): can ignore errors and return dummy value in CloudControl API context provider #211

go-to-k · 2025-03-08T19:16:27Z

CDK app with CC API Context Provider fails if the resource we want to get doesn't exist.

[Error at /LookupStack] Encountered CC API error while getting resource MyLookupTestRole. Error: ResourceNotFoundException: AWS::IAM::Role Handler returned status FAILED: The role with name MyLookupTestRole cannot be found. (Service: Iam, Status Code: 404, Request ID: xxxx-xxx-xxxx-xxxx) (HandlerErrorCode: NotFound, RequestToken: xxxx-xxx-xxxx-xxxx)

Also CC API Context Provider (CcApiContextProviderPlugin) cannot ignore errors even if ignoreErrorOnMissingContext is passed in aws-cdk-lib because the current code in aws-cdk-cli doesn't handle the property.

Sample cdk-lib code (based on my PR for IAM Role fromLookup):

    const response: {[key: string]: any}[] = ContextProvider.getValue(scope, {
      provider: cxschema.ContextProvider.CC_API_PROVIDER,
      props: {
        typeName: 'AWS::IAM::Role',
        exactIdentifier: options.roleName,
        propertiesToReturn: [
          'Arn',
        ],
      } as cxschema.CcApiContextQuery,
      dummyValue: [
        {
          // eslint-disable-next-line @cdklabs/no-literal-partition
          Arn: 'arn:aws:iam::123456789012:role/DUMMY_ARN',
        },
      ],
      ignoreErrorOnMissingContext: options.returnDummyRoleOnMissing, // added
    }).value;

However it would be good if the provider can ignore errors and return any dummy value to cdk library. This allows all resources to be used if available, or created if not.

Actually, SSM and KMS provider can ignore errors:

KMS: https://github.com/aws/aws-cdk-cli/blob/main/packages/aws-cdk/lib/context-providers/keys.ts#L43-L48

SSM: https://github.com/aws/aws-cdk-cli/blob/main/packages/aws-cdk/lib/context-providers/ssm-parameters.ts#L27-L30

For example, in cdk-lib, we can specify ignoreErrorOnMissingContext in the fromLookup. The dummyValue and ignoreErrorOnMissingContext properties can also be specified in GetContextValueOptions.

cli-integ

aws/aws-cdk-cli-testing#50

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

… provider

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

rix0rrr · 2025-03-10T19:08:45Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+      const dummyValue = this.getDummyValueIfErrorIgnored(args);
+      if (dummyValue) {
+        const propsObj = getResultObj(dummyValue, 'dummy-id', args.propertiesToReturn);
+        resultObjs.push(propsObj);
+        return resultObjs;
+      }


Should this behavior be in a try/catch in getValue() ? It seems duplicated.

Also: let's decide what a good error protocol would be here. I'm not convinced that return a single dummy object (which a caller has to supply a value for, and then has to test for that value... and it needs to be different from the default dummy object) is the best protocol.

Also; context provider return values are usually cached. Do we really want "not found" dummy values to be cached? (I.e., if they notice and create the resource after the first synth, we will still behave as if the resource doesn't exist).

Is that behavior going to match expected user behavior?

Should this behavior be in a try/catch in getValue() ? It seems duplicated.

Certainly. Decided to use try/catch in findResources so that the following errors are not ignored

if (args.exactIdentifier && args.propertyMatch) { throw new ContextProviderError(`Specify either exactIdentifier or propertyMatch, but not both. Failed to find resources using CC API for type ${args.typeName}.`); } if (!args.exactIdentifier && !args.propertyMatch) { throw new ContextProviderError(`Neither exactIdentifier nor propertyMatch is specified. Failed to find resources using CC API for type ${args.typeName}.`); }

ffc040e

Also: let's decide what a good error protocol would be here. I'm not convinced that return a single dummy object (which a caller has to supply a value for, and then has to test for that value... and it needs to be different from the default dummy object) is the best protocol.

I see. However, throwing an error will cause the CLI to exit, so it must be returned with the normal return type ({[key: string]: any} []). The normal return value is also constructed and returned based on the following function.

Would it then be better to return an array of objects such as [{ Identifier: ‘dummy-id’ }] or [{ ErrorIgnored: true }] (or an empty object...)?

export function getResultObj(jsonObject: any, identifier: string, propertiesToReturn: string[]): {[key: string]: any} { const propsObj = {}; propertiesToReturn.forEach((propName) => { Object.assign(propsObj, { [propName]: findJsonValue(jsonObject, propName) }); }); Object.assign(propsObj, { ['Identifier']: identifier }); return propsObj; }

Also; context provider return values are usually cached. Do we really want "not found" dummy values to be cached? (I.e., if they notice and create the resource after the first synth, we will still behave as if the resource doesn't exist).

I believe this is a matter on the cdk-lib side. The command for clearing the context can avoid that behaviour, but if we are concerned about the hassle, we could add a process on the cdk-lib side that also reports a missingContext and doesn't cache it if a dummy value is stored.

https://github.com/aws/aws-cdk/blob/v2.183.0/packages/aws-cdk-lib/core/lib/context-provider.ts#L108-L131

Either way, as cdk-cli, if we don't create a mechanism to ignore the error, the CDK CLI will terminate with an error when the resource is not found. This means that it will be difficult to fulfil the CDK use case of "use an existing resource if it is available, otherwise create it", so I think this PR should make sense. The existing SSM and KMS providers already support it, so it would be natural to do it in the CC API as well.

Also: let's decide what a good error protocol would be here. I'm not convinced that return a single dummy object (which a caller has to supply a value for, and then has to test for that value... and it needs to be different from the default dummy object) is the best protocol.

Alternatively, I came up with the idea of throwing an error with a message indicating something "ignored" in the CLI and not doing Annotations.addError (here) if the message is detected on the cdk-lib side. However, this may result in less clean code as it handles the error message and may be a bit confusing as it goes against the meaning of the property name ignoreErrorOnMissingContext...

I believe this is a matter on the cdk-lib side. The command for clearing the context can avoid that behaviour, but if we are concerned about the hassle, we could add a process on the cdk-lib side that also reports a missingContext if a dummy value is stored.

Alternatively, I came up with the idea of throwing an error with a message indicating something "ignored" in the CLI and not doing Annotations.addError (here) if the message is detected on the cdk-lib side. However, this may result in less clean code as it handles the error message and may be a bit confusing as it goes against the meaning of the property name ignoreErrorOnMissingContext...

I'm not sure, but if a custom class like class IgnoredContextProviderError extends ContextProviderError {} is created and it is thrown in getValue (findResources) if ignoreErrorOnMissingContext is true, and this code is changed with handling the error (if (e instanceof IgnoredContextProviderError) ...), perhaps it could be handled more naturally... (Even so, changes will still need to be made on the cdk-lib side, but...)

For example:

in packages/@aws-cdk/tmp-toolkit-helpers/src/api/toolkit-error.ts

/** * Represents an error originating from a Context Provider that will be ignored. */ export class IgnoredContextProviderError extends ContextProviderError {}

in findResources of packages/aws-cdk/lib/context-providers/cc-api-provider.ts

try { if (args.exactIdentifier) { // use getResource to get the exact indentifier return await this.getResource(cc, args.typeName, args.exactIdentifier, args.propertiesToReturn); } else { // use listResource return await this.listResources(cc, args.typeName, args.propertyMatch!, args.propertiesToReturn); } } catch (err) { if (args.ignoreErrorOnMissingContext) { throw new IgnoredContextProviderError(`Resource for type ${args.typeName} not found but ignore the error.`); } throw err; }

in provideContextValues of packages/aws-cdk/lib/context-providers/index.ts

export async function provideContextValues( // ... // ... try { // ... // ... value = await provider.getValue({ ...missingContext.props, lookupRoleArn: arns.lookupRoleArn }); } catch (e: any) { if (e instanceof IgnoredContextProviderError) { // Ignore the error if `ignoreErrorOnMissingContext` is set to true on `missingContext`. value = { [cxapi.PROVIDER_ERROR_KEY]: formatErrorMessage(e), [IGNORED_ERROR_ON_MISSING_CONTEXT_KEY]: true, [TRANSIENT_CONTEXT_KEY]: true }; } else { // Set a specially formatted provider value which will be interpreted // as a lookup failure in the toolkit. value = { [cxapi.PROVIDER_ERROR_KEY]: formatErrorMessage(e), [TRANSIENT_CONTEXT_KEY]: true }; } }

in ContextProvider.getValue in cdk-lib

const isIgnoredError = isIgnoredErrorOnMissingContext(value); if (ignoredError) { return { value: options.dummyValue }; } if (value === undefined || providerError !== undefined) { // ...

This way, the error protocol does not depend on dummy values in cdk-cli, and it may be possible to avoid caching them in the context (although it may require a few changes on the lib side, and if a new key should be added, on the cxapi side too).

What do you think?

Sorry for the volume of comments (I've given it a lot of thought).

Either way, I am willing to act once I have your opinion.

I had sort of missed how this error ignoring protocol, where we record dummyValue as the "official response" had already been implemented in aws/aws-cdk#31415 before, and is being used in 2 other context providers already.

I still can't say I'm happy about it, but there's precedent so we might as well do the same thing as these other context providers until we decide that all of their behaviors should be changed!

I've documented my understanding of these flags here: aws/aws-cdk#33875. If you think this is accurate, then we should just be doing what the documented protocol says.

If you think this is accurate, then we should just be doing what the documented protocol says.

I read and thought it is accurate. So I have changed to use ignoreFailedLookup instead of ignoreErrorOnMissingContext.

0262d51

I missed that the property called by cdk-lib is still ignoreErrorOnMissingContext.

https://github.com/aws/aws-cdk/pull/33875/files#diff-0d73bed6d4cd92ebf3292b51185a26f6e85c3e9cf6eb37c28bfc48a457600876R169-R172

So I changed the property name back to ignoreErrorOnMissingContext.

4ea2594

packages/aws-cdk/test/context-providers/cc-api-provider.test.ts

codecov-commenter · 2025-03-12T09:31:50Z

Codecov Report

Attention: Patch coverage is 97.63314% with 4 lines in your changes missing coverage. Please review.

Project coverage is 85.15%. Comparing base (762e57f) to head (52d95ab).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/aws-cdk/lib/context-providers/cc-api-provider.ts	96.19%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #211      +/-   ##
==========================================
- Coverage   85.18%   85.15%   -0.03%     
==========================================
  Files         221      221              
  Lines       36606    36696      +90     
  Branches     4437     4451      +14     
==========================================
+ Hits        31182    31250      +68     
- Misses       5323     5344      +21     
- Partials      101      102       +1

Flag	Coverage Δ
suite.unit	`85.15% <97.63%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rix0rrr · 2025-03-22T14:57:17Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+      const dummyValue = this.getDummyValueIfErrorIgnored(args);
+      if (dummyValue) {
+        return [getResultObj(dummyValue, 'dummy-id', args.propertiesToReturn)];
+      }
+      throw err;


After familiarizing myself with the procol a bit more and documenting it a bit more in this PR: aws/aws-cdk#33875 I notice that his behavior is wrong.

This would return the dummy value for any type of error (including "missing credentials", "malformed query", etc). Instead, we should only be returning the dummy value in case we don't find any resources.

Okay. Changed to check whether it was not found with err instanceof ResourceNotFoundException.

0262d51

rix0rrr · 2025-03-22T14:57:33Z

packages/@aws-cdk/cloud-assembly-schema/schema/version.json

-  "schemaHash": "ba7d47a7a023c39293e99a374af293384eaf1ccd207e515dbdc59dfb5cae4ed6",
-  "revision": 41
+  "schemaHash": "4c069fb88b4e8d69ffb38c74a978955d21d2fb20022266d298f1d9855f45ba78",
+  "revision": 43


Oops, we bumped twice?

I see. I guess I was running yarn build every time I changed JSDoc.
I fixed it by manually reverting to 41 and running yarn build only once.

0262d51

rix0rrr · 2025-03-22T15:00:35Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+      const dummyValue = this.getDummyValueIfErrorIgnored(args);
+      if (dummyValue) {
+        const propsObj = getResultObj(dummyValue, 'dummy-id', args.propertiesToReturn);
+        resultObjs.push(propsObj);
+        return resultObjs;
+      }


I had sort of missed how this error ignoring protocol, where we record dummyValue as the "official response" had already been implemented in aws/aws-cdk#31415 before, and is being used in 2 other context providers already.

I still can't say I'm happy about it, but there's precedent so we might as well do the same thing as these other context providers until we decide that all of their behaviors should be changed!

I've documented my understanding of these flags here: aws/aws-cdk#33875. If you think this is accurate, then we should just be doing what the documented protocol says.

rix0rrr · 2025-03-22T15:03:06Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+    if (!Array.isArray(args.dummyValue) || args.dummyValue.length === 0) {
+      return undefined;
+    }


If we're saying that dummyValue should be an array to be valid according to the protocol, we should error loudly if this isn't the case.

I could not find what we are saying that the dummyValue should be an array but could find the code that handles it as an array in the PR.

But in cdk-lib, dummyValue is defined as any. Should we still make an error in cdk-cli if it is not an array?

But yes, the format of dummyValue should be an array, since it must be in the same format as the normal return value.

It is also unkind not to tell the user when dummyValue is improperly formatted, even though ignoreFailedLookup is true.

Therefore, as you say, I have made an error in such a case.

0262d51

c315497

go-to-k · 2025-03-22T17:50:50Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

-  private getDummyValueIfErrorIgnored(args: CcApiContextQuery): Record<string, any> | undefined {
-    if (!args.ignoreErrorOnMissingContext) {
-      return undefined;
-    }
-    if (!Array.isArray(args.dummyValue) || args.dummyValue.length === 0) {
-      return undefined;
-    }
-    const dummyValue = args.dummyValue[0];
-    if (typeof dummyValue !== 'object' || dummyValue === null) {
-      return undefined;
+  private getDummyObject(args: CcApiContextQuery): Record<string, any> | undefined {
+    if (!Array.isArray(args.dummyValue) || args.dummyValue.length === 0 || typeof args.dummyValue[0] !== 'object' || args.dummyValue[0] === null) {
+      throw new ContextProviderError(`dummyValue must be an array with at least one object. Failed to get dummy object for type ${args.typeName}.`);
    }
-    return dummyValue;
+    return getResultObj(args.dummyValue[0], 'dummy-id', args.propertiesToReturn);


Removed the if (!args.ignoreErrorOnMissingContext) block in this since this function is called when err is an instance of ResourceNotFoundException and ignoreFailedLookup is true.

Also moved getResultObj(args.dummyValue[0], 'dummy-id', args.propertiesToReturn) into this function from getValue.

A few more changes to getDummyObject (and also changed to getDummyObjects).

c315497

8d9d94c

fix test

go-to-k · 2025-03-22T19:37:07Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+    try { 
+      const result = await this.findResources(cloudControl, args);
+      return result;
+    } catch (err) {
+      if (err instanceof ResourceNotFoundException) {
+        const dummyObjects = this.getDummyObjects(args);
+        if (dummyObjects) {
+          return dummyObjects;
+        }
+      }
+      throw err;
+    }


I moved the try/catch in getValue because I have made it so that it is determined by the type of error instance, so there is no longer a problem doing it with getValue instead of findResources.

go-to-k · 2025-03-22T19:40:15Z

packages/aws-cdk/lib/context-providers/cc-api-provider.ts

+      if (err instanceof ResourceNotFoundException && ignoreErrorOnMissingContext) {
+        throw err;
+      }


I thought that if the error is ResourceNotFoundException but ignoreFailedLookup is false, it should return the ContextProviderError that wraps this error as before, so I added && ignoreFailedLookup to the condition.

…ant to part of the contract Failing `listResources` is not possible yet, that will be added later.

rix0rrr · 2025-03-24T12:56:19Z

I've updated the PR a bit to simplify

feat(cli): can ignore errors and return dummy value in cc api context…

70d7a4c

… provider

go-to-k requested a deployment to integ-approval March 8, 2025 19:16 — with GitHub Actions Waiting

github-actions bot added the p2 label Mar 8, 2025

aws-cdk-automation enabled auto-merge March 8, 2025 19:16

aws-cdk-automation requested a review from a team March 8, 2025 19:17

method name

08b4b12

auto-merge was automatically disabled March 8, 2025 19:23
Head branch was pushed to by a user without write access

go-to-k requested a deployment to integ-approval March 8, 2025 19:23 — with GitHub Actions Waiting

go-to-k changed the title ~~feat(cli): can ignore errors and return dummy value in cc api context provider~~ feat(cli): can ignore errors and return dummy value in CloudControl API context provider Mar 8, 2025

go-to-k mentioned this pull request Mar 9, 2025

feat(cli-integ): add new test for fromLookup() with dummy aws/aws-cdk-cli-testing#50

Open

rix0rrr requested changes Mar 10, 2025

View reviewed changes

go-to-k added 4 commits March 12, 2025 12:41

wip

2090861

call getDummyValueIfErrorIgnored in findResources

ffc040e

fix docs

b69d69b

change CcApiContextQuery interface and modify tests

256059a

go-to-k requested a deployment to integ-approval March 12, 2025 04:27 — with GitHub Actions Waiting

rm unused import

b54e689

go-to-k requested a deployment to integ-approval March 12, 2025 06:43 — with GitHub Actions Waiting

go-to-k requested a review from rix0rrr March 12, 2025 09:27

Merge branch 'main' into cc-api-dummy

74e9aa0

rix0rrr requested a deployment to integ-approval March 18, 2025 10:15 — with GitHub Actions Waiting

rix0rrr reviewed Mar 22, 2025

View reviewed changes

modify by review

0262d51

go-to-k requested a deployment to integ-approval March 22, 2025 17:34 — with GitHub Actions Waiting

go-to-k commented Mar 22, 2025

View reviewed changes

modify a test

bb905c6

go-to-k requested a deployment to integ-approval March 22, 2025 18:04 — with GitHub Actions Waiting

go-to-k requested a review from rix0rrr March 22, 2025 18:05

change getDummyObject

c315497

go-to-k requested a deployment to integ-approval March 22, 2025 18:21 — with GitHub Actions Waiting

change description of dummyValue

407cdb2

go-to-k requested a deployment to integ-approval March 22, 2025 18:28 — with GitHub Actions Waiting

go-to-k requested a deployment to integ-approval March 22, 2025 18:33 — with GitHub Actions Waiting

getDummyObject to getDummyObjects

8d9d94c

fix test

go-to-k force-pushed the cc-api-dummy branch from 71b0ec5 to 8d9d94c Compare March 22, 2025 18:37

go-to-k requested a deployment to integ-approval March 22, 2025 18:37 — with GitHub Actions Waiting

rm await

1a933f0

go-to-k requested a deployment to integ-approval March 22, 2025 18:43 — with GitHub Actions Waiting

Merge branch 'main' into cc-api-dummy

3218de7

go-to-k requested a deployment to integ-approval March 22, 2025 18:50 — with GitHub Actions Waiting

revert property name to ignoreErrorOnMissingContext

4ea2594

go-to-k requested a deployment to integ-approval March 22, 2025 19:18 — with GitHub Actions Waiting

go-to-k commented Mar 22, 2025

View reviewed changes

Refactor to combine code paths a bit more, and remove tests I don't w…

52d95ab

…ant to part of the contract Failing `listResources` is not possible yet, that will be added later.

rix0rrr had a problem deploying to integ-approval March 24, 2025 12:55 — with GitHub Actions Failure

rix0rrr approved these changes Mar 24, 2025

View reviewed changes

rix0rrr mentioned this pull request Mar 24, 2025

fix(cli): no way to ensure exact one result in CcApiContextProviderPlugin #251

Open

Some eslint work

ae199f9

rix0rrr requested a deployment to integ-approval March 24, 2025 18:15 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): can ignore errors and return dummy value in CloudControl API context provider #211

feat(cli): can ignore errors and return dummy value in CloudControl API context provider #211

go-to-k commented Mar 8, 2025 •

edited

Loading

rix0rrr Mar 10, 2025

rix0rrr Mar 10, 2025

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025

rix0rrr Mar 22, 2025

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

codecov-commenter commented Mar 12, 2025 •

edited

Loading

rix0rrr Mar 22, 2025

go-to-k Mar 22, 2025

rix0rrr Mar 22, 2025

go-to-k Mar 22, 2025 •

edited

Loading

rix0rrr Mar 22, 2025

rix0rrr Mar 22, 2025

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025

go-to-k Mar 22, 2025

go-to-k Mar 22, 2025

go-to-k Mar 22, 2025

rix0rrr commented Mar 24, 2025

feat(cli): can ignore errors and return dummy value in CloudControl API context provider #211

Are you sure you want to change the base?

feat(cli): can ignore errors and return dummy value in CloudControl API context provider #211

Conversation

go-to-k commented Mar 8, 2025 • edited Loading

cli-integ

Choose a reason for hiding this comment

Choose a reason for hiding this comment

go-to-k Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

go-to-k Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

codecov-commenter commented Mar 12, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

go-to-k Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

go-to-k Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

go-to-k Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr commented Mar 24, 2025

go-to-k commented Mar 8, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 12, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

codecov-commenter commented Mar 12, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading

go-to-k Mar 22, 2025 •

edited

Loading