3.14.2 possible regression: not authorized to perform: cloudwatch:TagResource on cloudwatch

[ezrizhu]

using pcluster API 3.14.2, cluster fails on creation of HeadNodeAlarmD6381F07 Due to
Resource handler returned message: "Encountered a permissions error performing a tagging operation, please add required tag permissions. See https://repost.aws/knowledge-center/cloudformation-tagging-permission-error for how to resolve. Resource handler returned message: "Unauthorized tagging operation"" (RequestToken: 5d0428cd-9eba-7b8d-e27c-56bafd52bf6f, HandlerErrorCode: UnauthorizedTaggingOperation)

{
    "eventVersion": "1.11",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAXXXXXXXXXXXXXXXXX:pcluster-api-ParallelClusterFunction-XXXXXXXXXXXX",
        "arn": "arn:aws-us-gov:sts::XXXXXXXXXXXX:assumed-role/ParallelClusterLambdaRole-XXXXXXXX/pcluster-api-ParallelClusterFunction-XXXXXXXXXXXX",
        "accountId": "XXXXXXXXXXXX",
        "accessKeyId": "ASIAXXXXXXXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAXXXXXXXXXXXXXXXXX",
                "arn": "arn:aws-us-gov:iam::XXXXXXXXXXXX:role/ParallelClusterLambdaRole-XXXXXXXX",
                "accountId": "XXXXXXXXXXXX",
                "userName": "ParallelClusterLambdaRole-XXXXXXXX"
            },
            "attributes": {
                "creationDate": "2026-04-10T08:30:07Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "cloudformation.amazonaws.com",
        "inScopeOf": {
            "issuerType": "AWS::Lambda::Function",
            "credentialsIssuedTo": "arn:aws-us-gov:lambda:us-gov-west-1:XXXXXXXXXXXX:function:pcluster-api-ParallelClusterFunction-XXXXXXXXXXXX"
        }
    },
    "eventTime": "2026-04-10T08:34:59Z",
    "eventSource": "monitoring.amazonaws.com",
    "eventName": "PutCompositeAlarm",
    "awsRegion": "us-gov-west-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws-us-gov:sts::XXXXXXXXXXXX:assumed-role/ParallelClusterLambdaRole-XXXXXXXX/pcluster-api-ParallelClusterFunction-XXXXXXXXXXXX is not authorized to perform: cloudwatch:TagResource on resource: arn:aws-us-gov:cloudwatch:us-gov-west-1:XXXXXXXXXXXX:alarm:pcluster-XXXXX-HeadNode because no identity-based policy allows the cloudwatch:TagResource action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "eventID": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXXXXX",
    "eventCategory": "Management"
}

(untested yet) Seems to be fixed by #7296 Resolved on 3.15.0

[jaccromwell]

Hi @ezrizhu did you ever find the root cause of this issue, I am also had this issue on my own applications Friday, whilst I have fixed by adding the permission on the policy i'd like to know what the cause for this issue was.

[hgreebe]

Hello,
This has been resolved in 3.15.0.
It shouldn't cause any functional issues. Essentially what is occurring is that when CFN creates these new alarm resources, CDK propagates stack tags to them, and CFN calls cloudwatch:TagResource. If this fails, the alarm is created without the tags. So cluster creation should still succeeded and everything should still work as intended.

Did cluster creation fail, our did cloudformation just show that error but still succeeded in creating the cluster?

[ezrizhu]

Hi, cluster creation worked and everything is functional. It's just that the CFN showed errors, and thus our terraform showed errors.
Unsure why this just started happening around last Friday tho, was something done on aws's side? considering I didn't change the versions.

[jaccromwell]

Must've been some sort of regression on aws side, Their docs do recommend to attach TagResource to Alarms anyways. Just sucks that there's no documentation of change from them.