Cloud / AWS Key Management Service(KMS) Interview questions

AWS KMS is a managed service for creating and controlling the cryptographic keys that protect your data. You call KMS APIs to encrypt, decrypt, sign, verify, and generate keys, while the key material stays inside FIPS 140-3 Level 3 validated hardware security modules (HSMs) and is never exposed in plaintext outside them.

Many AWS services (S3, EBS, RDS, DynamoDB, Lambda, Secrets Manager and others) integrate with KMS so they can encrypt data at rest on your behalf. Access is governed by key policies, IAM policies, and grants, and every API call is recorded in AWS CloudTrail.

KMS is a regional service: a key lives in the Region where you create it, unless you use Multi-Region keys.

A KMS key (previously called a customer master key, or CMK) is the main resource in AWS KMS. It is a logical representation of a cryptographic key: it holds metadata and a reference to the actual key material stored in HSMs.

The metadata you will see in DescribeKey includes:

• Key ID and ARN
• KeySpec (for example SYMMETRIC_DEFAULT or RSA_2048)
• KeyUsage (ENCRYPT_DECRYPT, SIGN_VERIFY, GENERATE_VERIFY_MAC, KEY_AGREEMENT)
• KeyState, Origin, creation date, and description

You grant access to the KMS key and reference it in API calls; you never get the raw key material itself.

KMS keys are grouped by who owns and manages them: AWS owned, AWS managed, and customer managed.

Pick customer managed keys whenever you need control over policy, rotation, sharing across accounts, or deletion.

KMS supports three cryptographic categories, selected through the key spec and key usage at creation time.

• Symmetric (SYMMETRIC_DEFAULT): a single AES-256-GCM key used for encrypt and decrypt. Most AWS services use these.
• Asymmetric: RSA (2048, 3072, 4096) and ECC (NIST P-256, P-384, P-521, secp256k1) key pairs for encryption (RSA), signing and verification, or key agreement (ECC). ML-DSA post-quantum signing keys and SM2 (China Regions) are also available.
• HMAC (HMAC_224 to HMAC_512): used with GenerateMac and VerifyMac for message authentication codes.

For asymmetric keys, the private key never leaves KMS; the public key can be downloaded with GetPublicKey.

Envelope encryption means encrypting your data with a data key, then encrypting that data key with a stronger, centrally managed key, which in AWS is a KMS key. The encrypted data key is stored next to the ciphertext, like a sealed key inside the envelope.

1. Ask KMS for a data key (GenerateDataKey).
2. Encrypt the data locally with the plaintext data key.
3. Delete the plaintext data key from memory.
4. Store the encrypted data key together with the encrypted data.

To read the data back, you send the encrypted data key to KMS, receive the plaintext key, and decrypt locally. Only the small key crosses the network, never the data.

A data key is a symmetric key (or a key pair) that KMS generates for you to use outside KMS, typically to encrypt large amounts of data. The GenerateDataKey call returns two copies: one in plaintext and one encrypted under the KMS key you named.

KMS does not store, track, or manage data keys. Your application is responsible for keeping the encrypted copy and for wiping the plaintext copy from memory once it has finished encrypting. Data keys can be AES_256 or AES_128, and GenerateDataKeyPair returns an asymmetric pair with the private key encrypted.

A key policy is the resource-based policy attached to a KMS key. Every KMS key has exactly one, and it is the primary way access is controlled: unless the key policy allows access, directly or by enabling IAM policies, nobody can use the key.

The default policy delegates control to IAM:

{
  "Sid": "Enable IAM User Permissions",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
  "Action": "kms:*",
  "Resource": "*"
}

In a key policy, "Resource": "*" means only the key the policy is attached to. The maximum key policy size is 32 KB.

An alias is a friendly name of the form alias/<name> that points to a KMS key. Applications can reference the alias instead of a key ID, so you can repoint it to a different key with UpdateAlias without changing code.

aws kms create-alias --alias-name alias/app-prod --target-key-id 1234abcd-12ab-34cd-56ef-1234567890ab

Alias names are unique per account and Region, and cannot start with alias/aws/ because that prefix is reserved for AWS managed keys. One key can have several aliases, but each alias points to one key. When repointing, the new key must be in the same account and Region and have the same key usage and type.

An encryption context is an optional set of non-secret key-value pairs that KMS treats as additional authenticated data (AAD). It is cryptographically bound to the ciphertext, so you must supply the identical context to decrypt it, or the call fails.

aws kms encrypt --key-id alias/app-prod --plaintext fileb://invoice.bin \
  --encryption-context tenant=acme,purpose=invoice

The context appears in plain text in CloudTrail logs, so never put secrets in it. It is supported for symmetric KMS key operations and is also usable in policy conditions.

A grant is a policy instrument that lets an AWS principal use a KMS key for specific operations, optionally limited by constraints. Grants are created with CreateGrant and suit temporary, programmatic delegation, which is why AWS services like EBS and RDS use them on your behalf.

A grant names a grantee principal, the allowed operations (for example Decrypt or GenerateDataKey), and optional constraints such as an encryption context. Grants are eventually consistent; pass the returned grant token to use one immediately. End a grant with RetireGrant or RevokeGrant.

Automatic rotation makes KMS generate new cryptographic material for a customer managed symmetric key on a schedule. The default period is 365 days and you can choose between 90 and 2,560 days.

The key ID, ARN, alias, and policies stay the same. New encryption uses the latest material; older material is kept so existing ciphertext still decrypts. You can also trigger RotateKeyOnDemand.

Automatic rotation is not available for asymmetric keys, HMAC keys, keys with imported material, or keys in custom key stores. AWS managed keys rotate automatically every year and you cannot change that.

A KMS key's state controls whether it can be used. You can read it with DescribeKey.

Operations on a key that is not Enabled fail with a state error such as DisabledException or KMSInvalidStateException. DescribeKey still works, which is how you diagnose a stuck key. Replicas of a Multi-Region key have their own state, so disabling the primary does not disable them.

GenerateDataKey returns a fresh symmetric data key in two forms in a single call: a plaintext copy for immediate use and a copy encrypted under your KMS key for storage. It is the starting point of envelope encryption.

aws kms generate-data-key --key-id alias/app-prod --key-spec AES_256 \
  --encryption-context tenant=acme

You choose the length with KeySpec (AES_256 or AES_128) or NumberOfBytes. The plaintext is returned base64 encoded; use it to encrypt locally, then discard it.

Pair it with an encryption context so the encrypted copy can only be unwrapped by callers who present the same context. The caller needs kms:GenerateDataKey now, and a separate kms:Decrypt permission later to unwrap the encrypted copy.

Call ScheduleKeyDeletion with a PendingWindowInDays between 7 and 30; the default is 30. The key moves to PendingDeletion, cannot be used, and is destroyed when the window ends.

aws kms schedule-key-deletion --key-id 1234abcd-12ab-34cd-56ef-1234567890ab \
  --pending-window-in-days 30

Within the window you can call CancelKeyDeletion; the key returns as Disabled, and you must enable it again. After deletion, data encrypted under the key is permanently unrecoverable. A primary Multi-Region key cannot be scheduled for deletion while replicas exist.

Before scheduling, search CloudTrail for recent Decrypt calls on the key and check which services, grants, and IAM policies still reference it. Anything that depends on it stops working once the window ends.

For symmetric keys, Encrypt accepts up to 4,096 bytes of plaintext. That is meant for small secrets such as passwords or data keys, not files.

For RSA keys the limit depends on key size and the OAEP hash. With RSA_2048 it is 214 bytes using SHA-1 and 190 bytes using SHA-256. Anything larger should use envelope encryption: generate a data key with KMS and encrypt the bulk data locally.

In practice the limit rarely matters, because AWS services and the Encryption SDK handle envelope encryption for you. It matters when you call KMS directly, for example to protect a database password or wrap a small configuration value.

Set the bucket's default encryption to SSE-KMS and choose either aws/s3 or a customer managed key. S3 then requests a data key from KMS for each object (or bucket key) and stores the encrypted data key in the object's metadata.

aws s3api put-bucket-encryption --bucket my-bucket --server-side-encryption-configuration '{
 "Rules":[{"ApplyServerSideEncryptionByDefault":
   {"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"alias/app-prod"},
   "BucketKeyEnabled":true}]}'

Callers need S3 permissions plus kms:GenerateDataKey to upload and kms:Decrypt to download. Enable Bucket Keys to reduce KMS request volume.

Default encryption applies only to new uploads, so existing objects stay as they were until you copy them over themselves or use S3 Batch Operations. A bucket policy that denies uploads without SSE-KMS enforces the standard.

The application talks to KMS only for the small data key. The bulk data is encrypted and decrypted locally with AES-GCM, which is why the pattern scales.

sequenceDiagram
  participant App
  participant KMS
  participant Store as Storage
  App->>KMS: GenerateDataKey(KeyId, EncryptionContext)
  KMS-->>App: Plaintext data key + encrypted data key
  App->>App: Encrypt data locally, wipe plaintext key
  App->>Store: Save ciphertext + encrypted data key
  Store-->>App: Later: ciphertext + encrypted data key
  App->>KMS: Decrypt(encrypted data key, EncryptionContext)
  KMS-->>App: Plaintext data key
  App->>App: Decrypt data locally

The AWS Encryption SDK wraps this flow, packaging the ciphertext, encrypted data key, and encryption context into one message so you do not hand-roll it.

Direct encryption through KMS is limited to 4 KB per call, so large objects simply do not fit. Envelope encryption removes that limit and brings other benefits:

• Performance: only a 32-byte key travels to KMS, not gigabytes of data.
• Cost and throttling: one KMS call can protect an entire file or batch.
• Blast radius: each object can have its own data key, so one leaked key exposes one object.
• Cheap re-wrapping: moving data to a new KMS key means re-encrypting small data keys, not the data.

The plaintext data never leaves your environment, which also helps with data-residency rules.

Both can allow KMS actions, but the key policy is mandatory and the IAM policy is only effective if the key policy allows it.

This is different from most AWS resources, where an IAM allow alone is enough within an account.

A common surprise: a role has kms:* in IAM but still gets AccessDenied. Check the key policy first; it is the usual cause.

The principal arn:aws:iam::<account>:root in a key policy means the account itself, not the root user. That statement hands authorization over to IAM, so administrators can then control the key with ordinary IAM policies.

If you remove it and the only named principals are later deleted, nobody in the account can manage the key and you must contact AWS Support. The trade-off is that anyone with broad IAM permissions can reach the key. Security-sensitive teams replace the root statement with named admin and user roles and keep a break-glass role.

Auditors often flag this statement, but removing it is rarely the right fix. Scoping IAM permissions tightly, or replacing it with explicit role principals plus a break-glass role, gives control without risking a lockout.

AWS managed keys are created by AWS in your account the first time a service needs one. Customer managed keys are created and governed by you.

Choose AWS managed keys for quick wins where you do not need cross-account access or custom policies, such as default encryption in a single-account dev bucket. Move to customer managed keys for production data that needs audit separation, key-level deny, or sharing. An AWS managed key cannot be converted, so migrating means re-encrypting the data, for example by copying S3 objects or snapshots.

Use asymmetric keys when someone outside AWS needs to encrypt to you or verify your signatures with a public key.

Performance differs too: asymmetric operations are slower and have lower request quotas than symmetric ones, so avoid them for high-volume data encryption. A common hybrid is to let outside parties encrypt a data key to your RSA public key while symmetric keys protect the bulk data.

Each rotation adds a new key material version to the same KMS key. KMS keeps all previous versions, and the ciphertext of a symmetric key embeds metadata that identifies which version encrypted it.

When you call Decrypt, KMS reads that metadata, picks the matching material, and decrypts. You do not pass a version and your code does not change. New Encrypt and GenerateDataKey calls use the latest material.

Rotation does not re-encrypt existing data. Old material is removed only when the key itself is deleted. You can inspect history with ListKeyRotations and GetKeyRotationStatus.

Services and the Encryption SDK need no change after rotation. If you want old data under newer material, re-encrypt it with ReEncrypt, which never exposes plaintext to your application.

Both stop the key from being used, but only one is reversible at any time.

A safe habit is to disable first, watch CloudTrail and alarms for failures, and schedule deletion only when nothing breaks.

Disabling also helps in incident response: if a key may be compromised, disabling it immediately stops further use while you investigate and plan re-encryption. When you later schedule deletion, use the full 30 days and keep a CloudTrail alarm on, since any call during that window shows something still depends on the key.

Use grants for dynamic, short-lived access, such as a service decrypting a volume while it is attached. Use the key policy for long-term, human-designed permissions.

EBS is a good example: when you attach an encrypted volume, it creates a grant so the instance can decrypt the data key, and retires it when it is no longer needed. Neither the key policy nor your IAM policies had to change. Grants also support a retiring principal, so a designated role can clean them up.

Because the context is bound to the ciphertext as AAD, it prevents a valid ciphertext from being decrypted in the wrong place. A ciphertext created for tenant A will fail if the request claims tenant B, which blocks ciphertext swapping attacks.

You can also require it in policy, so only requests carrying the right context are allowed:

"Condition": {
  "StringEquals": {"kms:EncryptionContext:tenant": "acme"}
}

Finally, CloudTrail records the context, which makes audits readable: you can see which tenant or resource each Decrypt call was for.

Using it costs nothing and works with any symmetric operation, so adopt it as a default. Choose stable, non-sensitive values such as a tenant ID, table name, or object ARN.

The second call suits architectures where one component prepares keys and another, with decrypt permission, uses them later. EBS follows this pattern for new volumes.

For example, a provisioning service can call the WithoutPlaintext variant to pre-create thousands of volume keys during onboarding. Nobody holds a plaintext key at that point, so leaked provisioning logs expose nothing usable. Later, the workload calls Decrypt with the stored encrypted key and the same encryption context. Both calls accept the same KeySpec options and both appear in CloudTrail.

You need permission on both sides, and the key must be a customer managed key.

1. In the owner account, update the key policy to allow the other account (for example its root principal) the needed actions such as kms:Decrypt and kms:CreateGrant.
2. In the other account, add an IAM policy allowing its role to use the key by its full key ARN; aliases do not work cross-account.
3. Share the resource itself, such as an EBS snapshot or an S3 bucket.

AWS managed keys cannot be shared because their policy is not editable. For encrypted EBS snapshots the recipient typically copies the snapshot and re-encrypts it with its own key.

Pick SSE-KMS when you need audit trails and access separation. SSE-C requires HTTPS and your own key handling on every call.

Cost and limits differ too: SSE-KMS adds per-request KMS charges and is subject to KMS quotas, while SSE-S3 has neither. S3 applies SSE-S3 to new objects by default, so SSE-KMS is an explicit upgrade for audit and access separation. Switching existing objects means copying them.

With a Bucket Key, S3 asks KMS once for a short-lived bucket-level key and uses it to derive data keys for new objects locally. This cuts KMS requests by up to 99%, lowering cost and throttling risk.

Two things to know: it only affects new objects, and the encryption context in CloudTrail becomes the bucket ARN instead of the object ARN. Any key policy or IAM condition that matches on the object ARN must be updated.

To turn it on for a bucket, update the default encryption configuration. You can also set it per request with the bucket-key header on a PUT. Review any policy that matches on the object ARN in the encryption context before enabling it.

A Multi-Region key (MRK) is a set of KMS keys in different Regions that share the same key ID (prefixed mrk-) and the same key material. Ciphertext produced in one Region can be decrypted by the replica in another without re-encryption.

flowchart LR
  A["Primary key us-east-1"] -- ReplicateKey --> B["Replica eu-west-1"]
  A -- rotation synced --> B

Each replica keeps its own key policy, aliases, grants, and deletion schedule. Use MRKs for disaster recovery, active-active apps, and client-side encryption where data moves between Regions. If you do not need that portability, ordinary per-Region keys are simpler.

Key policies are not replicated, so create policies and grants in each Region. A missing policy in the replica Region is a common cause of failures during failover.

1. Create a KMS key with Origin=EXTERNAL; it enters PendingImport.
2. Call GetParametersForImport to receive a public wrapping key and an import token, valid for 24 hours.
3. Wrap your key material with the public key (for example RSAES_OAEP_SHA_256).
4. Call ImportKeyMaterial with the wrapped material, optionally setting an expiry.

Keep a secure copy of the original material. If it expires or is deleted, you can re-import the same material to restore the key. Imported keys do not support automatic rotation.

Wrapping options include RSAES_OAEP_SHA_256 and RSA_AES_KEY_WRAP_SHA_256, the latter designed for material too large for plain RSA wrapping. If you set an expiry, KMS deletes the material when it lapses and the key returns to PendingImport until you re-import it.

Choose KMS by default. Choose CloudHSM when a regulation or contract needs dedicated hardware you control.

Many teams combine them: a KMS custom key store backs KMS keys with CloudHSM, giving dedicated hardware while keeping KMS APIs and service integrations. The cost is lower throughput and more operational work, because you must keep the cluster healthy and connected.

Every KMS API call is logged by CloudTrail as a management event, including Encrypt, Decrypt, and GenerateDataKey. Each record shows the caller identity, key ARN, source IP, user agent, error code, and the encryption context in plain text.

This answers practical questions: who decrypted a given secret, which service used a key through kms:ViaService, and whether a key is still in use before you delete it. Send trails to S3 or CloudWatch Logs and add EventBridge rules for risky actions.

To find which principals used a key in the last 30 days, query CloudTrail Lake or Athena and filter on eventSource = kms.amazonaws.com. Do this before any lifecycle change such as disabling or deleting the key.

Once the waiting period ends, KMS destroys the key material and the key. Everything encrypted under it becomes permanently unrecoverable: S3 SSE-KMS objects, EBS volumes, RDS storage, secrets, and any envelope-encrypted files whose data keys were wrapped by it. AWS cannot restore it.

Before that, during PendingDeletion, services fail with KMSInvalidStateException; afterwards callers see NotFoundException. These errors are your early warning.

To reduce risk:

• Disable the key first and watch CloudTrail for failed calls.
• Use the full 30-day window.
• Alarm on any Decrypt attempt against a key in PendingDeletion via EventBridge.
• Deny kms:ScheduleKeyDeletion with an SCP except for a break-glass role.

For imported key material you can delete just the material and re-import it later; deleting the key itself cannot be undone.

Read the error text first; it usually names the principal, the action, and the key. Then walk through the layers in order:

1. Key policy: does it allow the principal, or enable IAM via the account root?
2. IAM policy: does the identity allow the action on the key ARN?
3. Guardrails: SCPs, permissions boundaries, and session policies must not deny it.
4. Conditions: kms:ViaService, encryption context, aws:SourceVpce, or tag conditions may not match.
5. VPC endpoint policy: does it allow the key and principal?
6. Cross-account: both the key policy and the caller's IAM policy must allow it.

aws kms describe-key --key-id alias/app-prod
aws kms get-key-policy --key-id alias/app-prod --policy-name default
aws kms list-grants --key-id alias/app-prod

Check CloudTrail for the matching event and its errorMessage. Note that DisabledException or KMSInvalidStateException signal key state problems, not permission ones.

KMS enforces request-rate quotas per account and Region, shared across keys for the same operation type. Spreading traffic over more keys does not raise the quota, so the fix is to make fewer calls or ask for more capacity.

• Retry with exponential backoff and jitter; AWS SDKs do this by default, so tune max attempts.
• Reduce calls with data key caching (AWS Encryption SDK) or one data key per file or batch.
• Enable S3 Bucket Keys for SSE-KMS buckets.
• Cache decrypted secrets in long-lived processes, such as Lambda global scope.
• Find the noisy caller in CloudTrail, then request a quota increase through Service Quotas if load is legitimate.

Custom key stores have much lower quotas than standard keys, so check which you are using.

In both cases, if the store is disconnected the keys become Unavailable and everything encrypted under them is unreadable until reconnected. Standard KMS keys avoid that operational burden.

In an interview, say which you would pick. A CloudHSM-backed store satisfies dedicated-hardware mandates while staying inside AWS. XKS fits sovereignty rules where AWS must never hold the key material, at the cost of an extra network dependency that you must size and monitor.

flowchart TD
  A["KMS request"] --> B{Explicit deny in key policy, IAM, SCP or boundary?}
  B -- Yes --> X[Denied]
  B -- No --> C{Key policy names the principal?}
  C -- Yes --> OK[Allowed]
  C -- No --> D{Key policy enables IAM and IAM policy allows?}
  D -- Yes --> OK
  D -- No --> E{Active grant for the principal?}
  E -- Yes --> OK
  E -- No --> X

An explicit deny always wins. After that, any one of three paths can allow: the key policy directly, IAM (only when the key policy enables it), or a grant. Cross-account requests need an allow in the key's account and in the caller's account. Guardrails like SCPs can still block an otherwise allowed call.

Use the kms:ViaService condition so the key can be used only when the request comes through a named service endpoint, and pair it with kms:CallerAccount to limit it to your account.

"Condition": {
  "StringEquals": {
    "kms:ViaService": "s3.us-east-1.amazonaws.com",
    "kms:CallerAccount": "111122223333"
  }
}

With this in place, a developer cannot call Decrypt directly with the key; it works only when S3 acts on their behalf. Remember to add each service that legitimately needs the key, such as EC2 for EBS.

Know the limits: ViaService does not restrict what the service itself does, so also scope the service through its own resource policies. Combine it with kms:GrantIsForAWSResource on kms:CreateGrant so only AWS services, not people, can create grants. Test in a non-production account, because a missing service in the allow list shows up as a confusing AccessDenied during volume attach or object upload.

Pair either model with attribute-based access control: allow kms:Decrypt only when kms:ResourceTag/tenant equals aws:PrincipalTag/tenant, or when the encryption context holds the caller's tenant ID.

Regulated or high-value tenants usually get dedicated keys; small tenants share. Automate key creation, tagging, and alias naming in your onboarding pipeline.

Watch the operational side: per-tenant keys need lifecycle automation, quota monitoring, and a policy template, otherwise thousands of hand-made keys become hard to audit. Decide up front how to offboard a tenant, since deleting the key is irreversible.

Choose imported material only when a requirement truly demands it:

• A policy or regulator needs you to control the origin of the key.
• You must hold a copy of the material outside AWS.
• You want to expire or delete the material while keeping the key ARN, for crypto-shredding.

The costs are real. You own durability of the original material, automatic rotation is not available, and you must manage expiry and re-import. For most workloads KMS-generated keys are safer because AWS handles generation, redundancy, and rotation.

If you adopt it, define a key ceremony: generate material on a trusted system, store the original in a secured offline vault, document who may re-import it, and alarm on upcoming expiry. Without that discipline the imported option is riskier than the default.

Asymmetric, HMAC, and some imported or custom-store keys need manual rotation, which means moving to a new key:

1. Create a new KMS key with the same spec and usage.
2. Point your alias to the new key with UpdateAlias so new operations use it.
3. Keep the old key enabled so existing ciphertext, signatures, or MACs still work.
4. Optionally re-encrypt data with ReEncrypt or reissue signatures, then disable and retire the old key.

Because the key ID changes, track which key protects which data. For asymmetric keys you must always pass the KeyId, and for signing you should publish the new public key so verifiers trust it.

Put the cadence in a runbook and automate it, for example with a scheduled Lambda. Manual rotation that relies on someone's calendar reminder tends to be forgotten.

sequenceDiagram
  participant You
  participant EC2 as EC2/EBS
  participant KMS
  You->>EC2: CreateVolume(Encrypted, KmsKeyId)
  EC2->>KMS: GenerateDataKeyWithoutPlaintext
  KMS-->>EC2: Encrypted data key (stored with volume)
  You->>EC2: AttachVolume
  EC2->>KMS: Decrypt encrypted data key (via grant)
  KMS-->>EC2: Plaintext key held in host memory
  EC2->>EC2: Encrypt data at host before it reaches storage

EBS gets a unique data key per volume, stores only its encrypted form, and decrypts it on attach. The plaintext lives in the Nitro host memory only. EBS uses grants created through your kms:CreateGrant permission, so that permission is required for the caller.

Snapshots use the same key. When copying across Regions or accounts, EBS re-encrypts with the destination key you choose.

From the user's side, a missing permission shows up as a volume stuck in an error state or an instance that fails to launch with a KMS-related message. Granting the launching role the needed key permissions fixes it.

Combine event rules, metrics, and managed checks:

• EventBridge rules on CloudTrail events: ScheduleKeyDeletion, DisableKey, PutKeyPolicy, CreateGrant, ImportKeyMaterial.
• KMS-emitted events for rotation, imported-material expiry, and key deletion.
• CloudWatch metrics such as SecondsUntilKeyMaterialExpiration with an alarm.
• AWS Config rules to check rotation is enabled on customer managed keys.
• IAM Access Analyzer to find keys shared with external accounts.

Route alerts to SNS or your SIEM, and write runbooks for the first two: a surprise deletion or a changed key policy needs a fast human response.

Prioritize alerts: a PutKeyPolicy or ScheduleKeyDeletion from an unusual principal deserves a page, while rotation events are informational. Test your rules by simulating the call in a sandbox account so you know the alert fires.

Create a key with KeyUsage=SIGN_VERIFY and an RSA or ECC spec. The private key never leaves KMS; you send a message or digest to Sign and get a signature back.

aws kms sign --key-id alias/jwt-signer --message fileb://payload.bin \
  --message-type RAW --signing-algorithm RSASSA_PSS_SHA_256

• Use MessageType=RAW for messages up to 4,096 bytes, or DIGEST for larger content you hash yourself.
• Verify with the Verify API, or download the public key via GetPublicKey and verify offline.

Common uses are JWT signing, code signing, and document signatures. Rotation is manual, so plan key versioning in advance.

Pick algorithms deliberately: ECDSA signatures are smaller than RSA ones, while RSA offers broader compatibility with legacy verifiers. Confirm what your consumers can verify first, because the key spec cannot be changed later.

Caching trades security for speed: set limits on cache age, number of messages, and bytes encrypted per key so exposure stays bounded. Also use backoff with jitter and review CloudTrail or Cost Explorer to find which service drives the request charges.

Measure before tuning: track KMS request counts per key and per caller, set a budget alarm, and tune cache limits against real traffic rather than guesses.

Create an interface endpoint (com.amazonaws.<region>.kms) when workloads in private subnets must reach KMS without going over the internet or a NAT gateway. Enable private DNS so the standard KMS hostname resolves to the endpoint.

Use the endpoint policy to restrict which principals and keys can be used through it. You can also restrict the key itself to your VPC:

"Condition": {"StringEquals": {"aws:SourceVpce": "vpce-0abc123"}}

Be careful: services such as EBS and S3 call KMS from AWS infrastructure, not from your VPC, so a blanket aws:SourceVpce deny can break them. Add kms:ViaService exceptions for those.

Verify by testing from an instance in the private subnet, confirming that DNS resolves to private endpoint addresses, and checking the vpcEndpointId field on CloudTrail KMS events to prove traffic used the endpoint.

A KMS request is authenticated (SigV4), authorized against the key policy, IAM, and grants, and then forwarded to a fleet of HSMs that perform the cryptographic operation. The response returns only the result, such as ciphertext or a data key.

flowchart LR
  C["Client SigV4 request"] --> A["KMS endpoint: authenticate"]
  A --> Z["Authorize: policy, IAM, grants"]
  Z --> H["HSM performs operation"]
  H --> R["Result returned, logged to CloudTrail"]

Key material is generated inside the HSMs, and it is stored only in encrypted form outside them. The plaintext appears only in volatile HSM memory during operations. The HSMs are FIPS 140-3 Level 3 validated and operated with controls that mean no AWS employee can view or export plaintext key material. Keys in a Region share protection through HSM domain keys, which is how any HSM in the Region can serve your key.

There are two main patterns, and the right one depends on where the ciphertext lives.

Many services let you pick a key per replica Region, so MRKs are not mandatory. Whatever you choose, pre-create the destination keys and policies, grant the roles in the DR Region, and test failover.

With MRKs, if the primary Region fails, promote a replica with UpdatePrimaryRegion so you can manage rotation and policy from the surviving Region. Include key availability in your DR runbook, not just data.