feat: restructure and add new datadog integration policies

[RoseSecurity]

Why

• Expanding permission sets as Datadog and AWS evolve

What

• Style update: change all join("", resource.kind.*.name) and resource.kind[0].name to one(resource.kind[*].name)
• Deprecate var.integrations and add var.policies in its place
  1. Update the description to indicate it is deprecated
  2. Make its default value null
• Add var.policies
• Create a local.policies which is the list of policies specified via var.integrations and var.policies combined with mappings and then de-duplicated

Note

For compatibility, map var.integrations "core" -> "core_integration" and "all" -> "full_integration" when adding to local.policies.

• Rename the "all" policy "full-integration" and update it
• Rename iam_policy_all.tf -> iam-policy-full-integration.tf and rename all the resources etc. named "all" to "full_integration", and trigger it with policy name "full-integration"
• Update the policy reference
• Update the permissions (statement.actions) from those sources:

full-integration permissions

  actions = [
    "apigateway:GET",
    "autoscaling:Describe*",
    "backup:List*",
    "budgets:ViewBudget",
    "cloudfront:GetDistributionConfig",
    "cloudfront:ListDistributions",
    "cloudtrail:DescribeTrails",
    "cloudtrail:GetTrailStatus",
    "cloudtrail:LookupEvents",
    "cloudwatch:Describe*",
    "cloudwatch:Get*",
    "cloudwatch:List*",
    "codedeploy:List*",
    "codedeploy:BatchGet*",
    "directconnect:Describe*",
    "dynamodb:List*",
    "dynamodb:Describe*",
    "ec2:Describe*",
    "ec2:GetTransitGatewayPrefixListReferences",
    "ec2:SearchTransitGatewayRoutes",
    "ecs:Describe*",
    "ecs:List*",
    "elasticache:Describe*",
    "elasticache:List*",
    "elasticfilesystem:DescribeFileSystems",
    "elasticfilesystem:DescribeTags",
    "elasticfilesystem:DescribeAccessPoints",
    "elasticloadbalancing:Describe*",
    "elasticmapreduce:List*",
    "elasticmapreduce:Describe*",
    "es:ListTags",
    "es:ListDomainNames",
    "es:DescribeElasticsearchDomains",
    "events:CreateEventBus",
    "fsx:DescribeFileSystems",
    "fsx:ListTagsForResource",
    "health:DescribeEvents",
    "health:DescribeEventDetails",
    "health:DescribeAffectedEntities",
    "kinesis:List*",
    "kinesis:Describe*",
    "lambda:GetPolicy",
    "lambda:List*",
    "logs:DeleteSubscriptionFilter",
    "logs:DescribeLogGroups",
    "logs:DescribeLogStreams",
    "logs:DescribeSubscriptionFilters",
    "logs:FilterLogEvents",
    "logs:PutSubscriptionFilter",
    "logs:TestMetricFilter",
    "oam:ListSinks",
    "oam:ListAttachedLinks",
    "organizations:Describe*",
    "organizations:List*",
    "rds:Describe*",
    "rds:List*",
    "redshift:DescribeClusters",
    "redshift:DescribeLoggingStatus",
    "route53:List*",
    "s3:GetBucketLogging",
    "s3:GetBucketLocation",
    "s3:GetBucketNotification",
    "s3:GetBucketTagging",
    "s3:ListAllMyBuckets",
    "s3:PutBucketNotification",
    "ses:Get*",
    "sns:List*",
    "sns:Publish",
    "sns:GetSubscriptionAttributes",
    "sqs:ListQueues",
    "states:ListStateMachines",
    "states:DescribeStateMachine",
    "support:DescribeTrustedAdvisor*",
    "support:RefreshTrustedAdvisorCheck",
    "tag:GetResources",
    "tag:GetTagKeys",
    "tag:GetTagValues",
    "wafv2:ListLoggingConfigurations",
    "wafv2:GetLoggingConfiguration",
    "xray:BatchGetTraces",
    "xray:GetTraceSummaries"
  ],

• Rename iam_policy_core.tf -> iam-policy-core-integration.tf and rename all the resources etc. named "core" to "core_integration", and trigger it with policy name "core-integration"
• Update the policy reference
• Update the permissions (statement.actions) by removing 'support:*'
• Create iam-policy-resource-collection.tf
• Follow the pattern of iam-policy-full-integration.tf and create iam-policy-resource-collection.tf to implement the resource-collection option, referencing and using the policy from https://docs.datadoghq.com/integrations/amazon_web_services/?tab=roledelegation#aws-resource-collection-iam-policy-1

resource-collection permissions

  actions = [
    "backup:ListRecoveryPointsByBackupVault",
    "bcm-data-exports:GetExport",
    "bcm-data-exports:ListExports",
    "cassandra:Select",
    "cur:DescribeReportDefinitions",
    "ec2:GetSnapshotBlockPublicAccessState",
    "glacier:GetVaultNotifications",
    "glue:ListRegistries",
    "lightsail:GetInstancePortStates",
    "savingsplans:DescribeSavingsPlanRates",
    "savingsplans:DescribeSavingsPlans",
    "timestream:DescribeEndpoints",
    "waf-regional:ListRuleGroups",
    "waf-regional:ListRules",
    "waf:ListRuleGroups",
    "waf:ListRules",
    "wafv2:GetIPSet",
    "wafv2:GetRegexPatternSet",
    "wafv2:GetRuleGroup"
  ],

• Create iam-policy-security-audit.tf
• Updated documentation and examples

[RoseSecurity]

I would also suggest adding the ability to dynamically merge additional IAM policies into the full integration as needed. For instance, with Cloudcraft functionality currently in preview, there is no way to incorporate additional policies into the integration since all values are hardcoded. How would you feel about introducing a variable to merge additional policies into the full integration for scenarios like this?

• Reference

[RoseSecurity]

/terratest

[RoseSecurity]

Do we want this to be included in everything?