kms decrypt cannot read code sent by Cognito

[FTG-Company-Fabrice]

Describe the bug

KMS fails to decrypt the code sent by Cognito to a CustomEmailSender lambda and issues error:

ERROR Lambda runtime invoke{requestId="496c3ff2-8bb4-42cb-b345-7ef621146677" xrayTraceId="Root=1-6799211c-1337c4e0703ffcc60eb00b57;Parent=42ed294d8792c7d9;Sampled=0;Lineage=1:e20a1e54:0"}: "Failed to decrypt: service error"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Decrypt the code token.

Current Behavior

Here is the extract from the log:

Raw code from event:

AYADeOmFlubmeytliSHh3UPC6DEAhgACABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREFsY3pMRmFHcmg1d0loeEdDVlM3Q3RmKzhTT1hwMTk3dit3WVJaaGEvaWdBNHphcXZVM0Y2YU9XTE5XN0cxUWVlZz09AAt1c2VycG9vbC1pZAAYYXAtbm9ydGhlYXN0LTFfR085a0ZUdWh1AAEAB2F3cy1rbXMAUGFybjphd3M6a21zOmFwLW5vcnRoZWFzdC0xOjI0ODE4OTkxOTQ3NjprZXkvYWMzNGE4MDYtOTk2OS00ZDkzLWE3MWQtMGE0Y2NmZjkyN2IwALgBAgEAeDilmRKWFKPH4cndgi3anZx1SjTZWNilp6GmSIRD1GHZAS/d1uzBY2NbF/HYKwGNxroAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwbgIN+J92tdWLFEtECARCAOykOLIoB4BBesdXqKJJ7kgHcv2vpaQUmXwXj+3+CdAV4bwSIv36vzrDiPTQIhxK0aUxe76Fn1GXe0wruAgAAAAAMAAAQAAAAAAAAAAAAAAAAABjK/RZLzBVOKQW3f5tqSkn/////AAAAAQAAAAAAAAAAAAAAAQAAAAZ380AzDDfNusfhqQ13kdrl12aKw20NAGcwZQIxAJnLNmDTlp9ji/MrfdoobWtyvuRS8ocfGmujVcNOeY4zpZVvUt5qGUqdla51xoDGXAIwD9S6ZWS+cDmDNV3Kgh+og9EptYLy02XlS9m+Z79TcnYYnvfIvNZNezPF5SFV40gW

Decoded bytes length: 624

Decoded bytes:

[1, 128, 3, 120, 233, 133, 150, 230, 230, 123, 43, 101, 137, 33, 225, 221, 67, 194, 232, 49, 0, 134, 0, 2, 0, 21, 97, 119, 115, 45, 99, 114, 121, 112, 116, 111, 45, 112, 117, 98, 108, 105, 99, 45, 107, 101, 121, 0, 68, 65, 108, 99, 122, 76, 70, 97, 71, 114, 104, 53, 119, 73, 104, 120, 71, 67, 86, 83, 55, 67, 116, 102, 43, 56, 83, 79, 88, 112, 49, 57, 55, 118, 43, 119, 89, 82, 90, 104, 97, 47, 105, 103, 65, 52, 122, 97, 113, 118, 85, 51, 70, 54, 97, 79, 87, 76, 78, 87, 55, 71, 49, 81, 101, 101, 103, 61, 61, 0, 11, 117, 115, 101, 114, 112, 111, 111, 108, 45, 105, 100, 0, 24, 97, 112, 45, 110, 111, 114, 116, 104, 101, 97, 115, 116, 45, 49, 95, 71, 79, 57, 107, 70, 84, 117, 104, 117, 0, 1, 0, 7, 97, 119, 115, 45, 107, 109, 115, 0, 80, 97, 114, 110, 58, 97, 119, 115, 58, 107, 109, 115, 58, 97, 112, 45, 110, 111, 114, 116, 104, 101, 97, 115, 116, 45, 49, 58, 50, 52, 56, 49, 56, 57, 57, 49, 57, 52, 55, 54, 58, 107, 101, 121, 47, 97, 99, 51, 52, 97, 56, 48, 54, 45, 57, 57, 54, 57, 45, 52, 100, 57, 51, 45, 97, 55, 49, 100, 45, 48, 97, 52, 99, 99, 102, 102, 57, 50, 55, 98, 48, 0, 184, 1, 2, 1, 0, 120, 56, 165, 153, 18, 150, 20, 163, 199, 225, 201, 221, 130, 45, 218, 157, 156, 117, 74, 52, 217, 88, 216, 165, 167, 161, 166, 72, 132, 67, 212, 97, 217, 1, 47, 221, 214, 236, 193, 99, 99, 91, 23, 241, 216, 43, 1, 141, 198, 186, 0, 0, 0, 126, 48, 124, 6, 9, 42, 134, 72, 134, 247, 13, 1, 7, 6, 160, 111, 48, 109, 2, 1, 0, 48, 104, 6, 9, 42, 134, 72, 134, 247, 13, 1, 7, 1, 48, 30, 6, 9, 96, 134, 72, 1, 101, 3, 4, 1, 46, 48, 17, 4, 12, 27, 128, 131, 126, 39, 221, 173, 117, 98, 197, 18, 209, 2, 1, 16, 128, 59, 41, 14, 44, 138, 1, 224, 16, 94, 177, 213, 234, 40, 146, 123, 146, 1, 220, 191, 107, 233, 105, 5, 38, 95, 5, 227, 251, 127, 130, 116, 5, 120, 111, 4, 136, 191, 126, 175, 206, 176, 226, 61, 52, 8, 135, 18, 180, 105, 76, 94, 239, 161, 103, 212, 101, 222, 211, 10, 238, 2, 0, 0, 0, 0, 12, 0, 0, 16, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 24, 202, 253, 22, 75, 204, 21, 78, 41, 5, 183, 127, 155, 106, 74, 73, 255, 255, 255, 255, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 6, 119, 243, 64, 51, 12, 55, 205, 186, 199, 225, 169, 13, 119, 145, 218, 229, 215, 102, 138, 195, 109, 13, 0, 103, 48, 101, 2, 49, 0, 153, 203, 54, 96, 211, 150, 159, 99, 139, 243, 43, 125, 218, 40, 109, 107, 114, 190, 228, 82, 242, 135, 31, 26, 107, 163, 85, 195, 78, 121, 142, 51, 165, 149, 111, 82, 222, 106, 25, 74, 157, 149, 174, 117, 198, 128, 198, 92, 2, 48, 15, 212, 186, 101, 100, 190, 112, 57, 131, 53, 93, 202, 130, 31, 168, 131, 209, 41, 181, 130, 242, 211, 101, 229, 75, 217, 190, 103, 191, 83, 114, 118, 24, 158, 247, 200, 188, 214, 77, 123, 51, 197, 229, 33, 85, 227, 72, 22]

ERROR Lambda runtime invoke{requestId="496c3ff2-8bb4-42cb-b345-7ef621146677" xrayTraceId="Root=1-6799211c-1337c4e0703ffcc60eb00b57;Parent=42ed294d8792c7d9;Sampled=0;Lineage=1:e20a1e54:0"}: "Failed to decrypt: service error"

log-events-viewer-result.csv

Reproduction Steps

Create a Lambda function and Key with SAM Template:

Transform: AWS::Serverless-2016-10-31
Parameters:
  StackName:
    Type: String
  Env:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - prod
  MailgunSecretArn:
    Type: String
    Description: ARN of the Mailgun secret in Secrets Manager
Conditions:
  IsProd: !Equals
    - !Ref Env
    - prod
Resources:
  CognitoKMSKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: !Sub alias/${StackName}/cognito
      TargetKeyId: !Ref CognitoKMSKey

  CognitoKMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: KMS key for Cognito user pool encryption
      Enabled: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
            Action: kms:*
            Resource: '*'
          - Sid: Allow Cognito to use the key
            Effect: Allow
            Principal:
              Service: cognito-idp.amazonaws.com
            Action:
              - kms:Encrypt
              - kms:Decrypt
              - kms:GenerateDataKey
              - kms:CreateGrant
            Resource: '*'
      KeySpec: SYMMETRIC_DEFAULT
      KeyUsage: ENCRYPT_DECRYPT
      MultiRegion: false
      PendingWindowInDays: 7
      Tags:
        - Key: Environment
          Value: !Ref Env
        - Key: Stack
          Value: !Ref StackName

  CustomEmailSender:
    Type: AWS::Serverless::Function
    Properties:
      # bootstap file generated by cargo lambda build --release (do not use format option)
      CodeUri: ../src/cognito/email_sender_2/target/lambda/email_sender_2   # Points to dir of bootstrap
      Handler: bootstrap    # Do not change, as this is the default executable name produced by Cargo Lambda
      Runtime: provided.al2023
      Architectures:
        - arm64
      Environment:
        Variables:
          REGION: !Sub ${AWS::Region}
          MAILGUN_SECRET_ARN: !Ref MailgunSecretArn
          COGNITO_KMS_KEY_ID: !GetAtt CognitoKMSKey.Arn
          KEY_ALIAS: !Ref CognitoKMSKeyAlias
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - secretsmanager:GetSecretValue
              Resource: !Ref MailgunSecretArn
            - Effect: Allow
              Action:
                - kms:Decrypt
                - kms:DescribeKey
              Resource: '*'
  CustomEmailSenderLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/${CustomEmailSender}
      RetentionInDays: !If
        - IsProd
        - 30
        - 1
  CustomEmailSenderPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref CustomEmailSender
      Principal: cognito-idp.amazonaws.com
      SourceArn: !Sub arn:aws:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/*
  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      AdminCreateUserConfig:
        AllowAdminCreateUserOnly: false
      AutoVerifiedAttributes:
        - email
      AliasAttributes:
        - email
      UserPoolName: !Sub ${StackName}-UserPool
      DeletionProtection: INACTIVE
      UserPoolAddOns:
        AdvancedSecurityMode: 'OFF'
      Schema:
        - AttributeDataType: String
          Mutable: true
          Name: locations
        - AttributeDataType: String
          Mutable: true
          Name: account_id
        # Company name is used only during the sign-up to initialize the account
        - AttributeDataType: String
          Mutable: true
          Name: company_name
      Policies:
        PasswordPolicy:
          MinimumLength: 8
          PasswordHistorySize: 0
          RequireLowercase: true
          RequireNumbers: false
          RequireSymbols: false
          RequireUppercase: true
          TemporaryPasswordValidityDays: 60
      LambdaConfig:
        KMSKeyID: !GetAtt CognitoKMSKey.Arn
        CustomEmailSender:
          LambdaArn: !GetAtt CustomEmailSender.Arn
          LambdaVersion: V1_0
  PlatformAdminUserPoolGroup:
    Type: AWS::Cognito::UserPoolGroup
    Properties:
      Description: This is the group for all pltform admins
      GroupName: platform-admin
      UserPoolId: !Ref UserPool
      Precedence: 1
  EveyroneUserPoolGroup:
    Type: AWS::Cognito::UserPoolGroup
    Properties:
      Description: Every user must be a member of this group
      GroupName: everyone
      UserPoolId: !Ref UserPool
      Precedence: 1
  AccountHolderUserPoolGroup:
    Type: AWS::Cognito::UserPoolGroup
    Properties:
      Description: Account holders (one user per account)
      GroupName: account-holder
      UserPoolId: !Ref UserPool
      Precedence: 1
  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      AllowedOAuthFlows:
        - code
      AllowedOAuthFlowsUserPoolClient: true
      AllowedOAuthScopes:
        - email
        - openid
        - profile
      ClientName: !Sub ${StackName}-JS-Client
      CallbackURLs:
        - http://localhost:3000/auth/callback
        - https://www.f-logis.com/auth/callback
        - flogis://
      EnableTokenRevocation: true
      ExplicitAuthFlows:
        - ALLOW_REFRESH_TOKEN_AUTH
        - ALLOW_USER_SRP_AUTH
        - ALLOW_USER_PASSWORD_AUTH
      GenerateSecret: false
      PreventUserExistenceErrors: ENABLED
      SupportedIdentityProviders:
        - COGNITO
      UserPoolId: !Ref UserPool
  UserIdentityPool:
    Type: AWS::Cognito::IdentityPool
    Properties:
      AllowUnauthenticatedIdentities: true
      IdentityPoolName: !Sub ${StackName}-IdentityPool
      CognitoIdentityProviders:
        - ProviderName: !GetAtt UserPool.ProviderName
          ClientId: !Ref UserPoolClient
          ServerSideTokenCheck: true
  UserIdentityPoolAuthenticatedRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${AWS::Region}-${StackName}-AuthenticatedRole
      Policies:
        - PolicyName: !Sub ${StackName}-CognitoAllowAPI
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: cognito-identity:GetCredentialsForIdentity
                Resource: '*'
              - Effect: Allow
                Action: execute-api:Invoke
                Resource: '*'
              - Effect: Allow
                Action: appsync:GraphQL
                Resource: '*'
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:DeleteObject
                Resource: '*'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRoleWithWebIdentity
            Principal:
              Federated:
                - cognito-identity.amazonaws.com
            Condition:
              StringEquals:
                cognito-identity.amazonaws.com:aud: !Ref UserIdentityPool
              ForAnyValue:StringLike:
                cognito-identity.amazonaws.com:amr: authenticated
      Description: This role is for authenticated users of the F-LOGIS apps
      MaxSessionDuration: 43200
      Path: /service-role/
  UserIdentityPoolGuestRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${AWS::Region}-${StackName}-GuestRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRoleWithWebIdentity
            Principal:
              Federated:
                - cognito-identity.amazonaws.com
            Condition:
              StringEquals:
                cognito-identity.amazonaws.com:aud: !Ref UserIdentityPool
              ForAnyValue:StringLike:
                cognito-identity.amazonaws.com:amr: unauthenticated
      Description: This role is for guest users of the F-LOGIS apps
      MaxSessionDuration: 43200
      Path: /service-role/
  UserIdentityPoolRoleAttachment:
    Type: AWS::Cognito::IdentityPoolRoleAttachment
    Properties:
      IdentityPoolId: !Ref UserIdentityPool
      Roles:
        authenticated: !GetAtt UserIdentityPoolAuthenticatedRole.Arn
        unauthenticated: !GetAtt UserIdentityPoolGuestRole.Arn
Outputs:
  STACK: 
    Value: !Ref StackName
  UserPoolId:
    Description: The User Pool ID details are
    Value: !Ref UserPool
  UserPoolArn:
    Description: The User Pool ID details are
    Value: !GetAtt UserPool.Arn
  IdentityPoolId:
    Description: The Identity Pool
    Value: !Ref UserIdentityPool
  AuthenticatedRoleArn:
    Description: Authenticated Role ARN
    Value: !GetAtt UserIdentityPoolAuthenticatedRole.Arn
  UnauthenticatedRoleArn:
    Description: Unauthenticated Role ARN
    Value: !GetAtt UserIdentityPoolGuestRole.Arn
  UserPoolClientId:
    Description: User Pool Client ID
    Value: !Ref UserPoolClient

Use cargo lambda to create a base template and then build the release binary.

Archive.zip

Possible Solution

No response

Additional Information/Context

I have tried the rust lambda on both x86 and arm64 architectures and both fail.

I have then written a JavaScript version of the lambda which works perfectly well. There is no change in the SAM template besides changing:

• CodeUri: ../src/cognito/email_sender_nodejs # Points to dir of bootstrap
• Handler: /index.handler
• Runtime: nodejs20.x

I was wondering if there was an issue in the Rust version with the AWS Lambda runtime not getting the encrypted code from Cognito properly and so hardcoded in the JavaScript version of the Lambda a code read from the logs of the Rust Lambda and the JavaScript version decrypted the code from the rust lambda with no issue.

Version

├── aws-config v1.5.15
│   ├── aws-credential-types v1.2.1
│   │   ├── aws-smithy-async v1.2.4
│   │   ├── aws-smithy-runtime-api v1.7.3
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-types v1.2.12
│   │   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-runtime v1.5.4
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-sigv4 v1.2.7
│   │   │   ├── aws-credential-types v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.12
│   │   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-http v0.60.12 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4
│   │   │   ├── aws-credential-types v1.2.1 (*)
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-sdk-sso v1.56.0
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2
│   │   │   └── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-sdk-ssooidc v1.57.1
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-sdk-sts v1.57.0
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-xml v0.60.9
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-smithy-async v1.2.4 (*)
│   ├── aws-smithy-http v0.60.12 (*)
│   ├── aws-smithy-json v0.61.2 (*)
│   ├── aws-smithy-runtime v1.7.7 (*)
│   ├── aws-smithy-runtime-api v1.7.3 (*)
│   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-types v1.3.4 (*)
├── aws-sdk-kms v1.57.0
│   ├── aws-credential-types v1.2.1 (*)
│   ├── aws-runtime v1.5.4 (*)
│   ├── aws-smithy-async v1.2.4 (*)
│   ├── aws-smithy-http v0.60.12 (*)
│   ├── aws-smithy-json v0.61.2 (*)
│   ├── aws-smithy-runtime v1.7.7 (*)
│   ├── aws-smithy-runtime-api v1.7.3 (*)
│   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-types v1.3.4 (*)
├── aws-sdk-secretsmanager v1.60.0
│   ├── aws-credential-types v1.2.1 (*)
│   ├── aws-runtime v1.5.4 (*)
│   ├── aws-smithy-async v1.2.4 (*)
│   ├── aws-smithy-http v0.60.12 (*)
│   ├── aws-smithy-json v0.61.2 (*)
│   ├── aws-smithy-runtime v1.7.7 (*)
│   ├── aws-smithy-runtime-api v1.7.3 (*)
│   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-types v1.3.4 (*)
├── aws-smithy-types v1.2.12 (*)

Environment details (OS name and version, etc.)

Mac OS 14.6.1 (M2).

Logs

No response

[ysaito1001]

Hi, in order to try isolating the underlying issue from lambda, is it possible to reproduce it outside lambda environment? On the other hand, if the issue is specific to lambda environment, you might want to file an issue in the aws-lambda-rust-runtime repository.

[FTG-Company-Fabrice]

I am afraid I cannot allocate development time to this at the moment. I am attaching the rust code so you can try to adapt it at your end.

Archive.zip

[ysaito1001]

I do see the use of lambda_runtime in the attached archive, so I'd start filing an issue in the aws-lambda-rust-runtime repository. We can better assist once the underlying root cause is identified to be outside the lambda environment.