Cloud / Amazon S3 (Simple Storage Service) Interview questions

Amazon S3, or Simple Storage Service, is AWS's object storage service. It stores data as objects inside containers called buckets, rather than as files in a traditional folder hierarchy or as blocks on a disk.

Each object consists of the data itself, a unique key that identifies it within the bucket, and metadata describing it. S3 is accessed over HTTP/HTTPS using a REST API, the AWS CLI, or an SDK, and it scales automatically without you provisioning capacity in advance.

It is commonly used for backups, static website hosting, data lakes, application assets, and as the durable storage layer behind many other AWS services.

A bucket is the top-level container in S3 that holds objects. Every object you store must live inside exactly one bucket, and bucket names must be globally unique across all of AWS, not just within your account.

Buckets are created in a specific AWS Region, and that choice affects latency, cost, and compliance since data does not automatically move between regions unless you configure replication.

Buckets also serve as the anchor point for configuration: versioning, lifecycle rules, encryption defaults, logging, and access policies are all set at the bucket level, though some settings can be overridden per object.

An object is the fundamental unit of storage in S3. It is made up of three parts: the data (the actual bytes of the file), a key (the unique identifier for that object within its bucket), and metadata (name-value pairs describing the object, such as content type or custom tags).

Objects can range in size from 0 bytes up to 5 terabytes, and there is no limit on how many objects a bucket can hold.

Unlike a file in a traditional filesystem, an object has no real concept of folders internally; the folder-like appearance in the S3 console comes from key names that include "/" characters, which S3 displays visually but does not treat as an actual directory structure.

Storage classes let you match how you store data to how often you actually access it, so you are not paying premium prices for data you rarely touch.

Each class trades off price, retrieval speed, minimum storage duration, and retrieval cost differently. Frequently accessed data goes into a class optimized for fast, cheap retrieval even though storage itself costs more; rarely accessed archival data goes into a class where storage is very cheap but retrieval is slower or costs more.

Because the same S3 API works across all of them, you can change an object's storage-class assignment via lifecycle rules without changing how your application reads or writes that object.

S3 currently offers several storage classes, grouped roughly by how predictable access patterns are and how long data needs to be retained.

Choosing between them comes down to how often you expect to read the data and how quickly you need it back when you do.

You can create a bucket through the AWS Console, CLI, SDK, or infrastructure-as-code tools like CloudFormation or Terraform. Using the CLI, the basic command is:

aws s3api create-bucket --bucket my-example-bucket-2026 --region us-east-1

For any region other than us-east-1, you must also pass a CreateBucketConfiguration specifying the LocationConstraint, since us-east-1 is treated as a special case.

The bucket name must be globally unique, 3-63 characters long, lowercase, and follow DNS-compliant naming rules, meaning no underscores and no uppercase letters. After creation, you would typically configure versioning, encryption, and public access block settings before uploading anything.

The simplest way is the AWS CLI's cp or sync commands, or the put_object/upload_file calls in an SDK like boto3.

aws s3 cp ./report.pdf s3://my-example-bucket-2026/reports/report.pdf

In Python with boto3:

import boto3
s3 = boto3.client("s3")
s3.upload_file("report.pdf", "my-example-bucket-2026", "reports/report.pdf")

For files larger than about 100 MB, the SDK's transfer manager, or the CLI itself, automatically splits the upload into parts using multipart upload, which improves throughput and lets a failed part be retried without restarting the whole transfer.

A bucket policy is a resource-based, JSON-formatted permission document attached directly to a bucket that controls who can perform which actions on that bucket and the objects in it.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowReadForAccount",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::123456789012:root"},
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-example-bucket-2026/*"
  }]
}

Because it is attached to the resource rather than to a user or role, a bucket policy can grant access to principals outside your own AWS account, which makes it the standard mechanism for cross-account or public sharing scenarios. Only one bucket policy can exist per bucket, though it can contain many statements.

Versioning keeps multiple variants of an object in the same bucket whenever it is overwritten or deleted, instead of letting the previous copy disappear.

Once enabled, every PUT to the same key creates a new version with its own version ID, and the previous versions remain retrievable. Deleting an object does not erase data either; it inserts a delete marker as the new current version, and the prior versions stay intact until you explicitly delete that specific version ID.

Versioning is a strong safeguard against accidental overwrites and deletions, but storage costs grow with every version kept, so it is usually paired with lifecycle rules that expire old versions after a set period.

Access to a bucket or its objects can be granted through several mechanisms, and most real deployments combine more than one:

• IAM policies attached to users, groups, or roles, granting permissions on S3 actions.
• Bucket policies, a resource-based policy attached directly to the bucket.
• Access Control Lists (ACLs), a legacy, coarse-grained mechanism now discouraged for new buckets.
• S3 Access Points, named network endpoints with their own policy, useful for simplifying access at scale.
• Presigned URLs, time-limited URLs generated by a trusted principal to grant temporary access to a specific object.

AWS recommends IAM policies and bucket policies as the primary tools today, keeping ACLs disabled and Block Public Access enabled unless there is a specific, deliberate reason to allow public access.

The key is the unique string that identifies an object within a bucket, essentially the object's full path and filename combined into one string, since S3 has no real nested directories.

A key can be up to 1,024 bytes of UTF-8 characters, and including "/" characters is a common convention that makes the console display objects as if they were organized in folders, purely for visual grouping.

Because the key is what uniquely identifies an object, uploading a new object with an existing key overwrites it, or creates a new version if versioning is enabled, rather than creating a duplicate.

A single S3 object can be up to 5 terabytes in size, and there is no practical limit on the total number of objects or total data you can store in a bucket.

However, a single PUT operation, a non-multipart upload, is capped at 5 GB. To upload anything larger than that, you must use multipart upload, which splits the object into parts of up to 5 GB each, uploads them independently, and has S3 assemble them into the final object once every part succeeds.

AWS recommends using multipart upload for any object over about 100 MB, even though it is not strictly required until you cross the 5 GB single-PUT ceiling, because it improves throughput and resilience to network failures.

S3 Standard is the default, general-purpose storage class, built for data that is accessed frequently and needs low latency and high throughput.

It stores data redundantly across a minimum of three Availability Zones, giving it 99.999999999% (eleven nines) durability and 99.99% availability, with no retrieval fees and no minimum storage duration; you pay only for what you store and can delete it anytime without penalty.

Typical use cases include actively used application data, content distribution origins, mobile and web application assets, and any workload where access patterns are frequent or unpredictable enough that discounts on infrequent-access classes would not offset their retrieval costs.

Intelligent-Tiering automatically moves objects between access tiers based on observed usage, without any performance impact or retrieval fees, making it well suited to data whose access pattern you cannot predict in advance.

It monitors each object and, after 30 consecutive days without access, moves it to a lower-cost infrequent-access tier; further tiers exist for archive-level inactivity of 90+ days and deep archive of 180+ days, which you can opt into. If an object in a colder tier is accessed again, it is automatically moved back to the frequent-access tier.

There is a small monthly monitoring and automation fee per object, so it is most cost-effective for objects larger than about 128 KB, where savings from automatic tiering outweigh that fee.

The Glacier storage classes are built for archival data, information you must retain, often for compliance or long-term backup, but rarely or never expect to read.

S3 Glacier Instant Retrieval offers millisecond access at archive-level pricing for data accessed roughly once a quarter. S3 Glacier Flexible Retrieval is cheaper still but expects retrieval requests, which can take minutes to hours depending on the retrieval tier chosen: expedited, standard, or bulk. S3 Glacier Deep Archive is the cheapest class AWS offers, intended for data accessed once or twice a year at most, with standard retrieval taking up to 12 hours.

All Glacier classes carry minimum storage durations of 90 to 180 days and retrieval fees, so moving data out early or reading it often can be more expensive than a warmer class.

Lifecycle rules are bucket-level policies that automatically transition objects between storage classes, or expire (delete) them, based on age or other conditions, so you do not have to manage that manually.

A rule is scoped by a prefix or tag filter and can include one or more transition actions, for example moving to Standard-IA after 30 days and then Glacier after 90 days, plus an optional expiration action such as deleting after 365 days. Separate rules can also target noncurrent object versions and incomplete multipart uploads, which otherwise silently accumulate storage cost.

{
  "Rules": [{
    "ID": "ArchiveOldLogs",
    "Filter": {"Prefix": "logs/"},
    "Status": "Enabled",
    "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"},
                     {"Days": 90, "StorageClass": "GLACIER"}],
    "Expiration": {"Days": 365}
  }]
}

Versioning is enabled per bucket, either through the console's Properties tab or via CLI/SDK. Using the CLI:

aws s3api put-bucket-versioning \
  --bucket my-example-bucket-2026 \
  --versioning-configuration Status=Enabled

Once turned on, versioning can be suspended but never fully disabled again for that bucket; the alternative Suspended state stops creating new versions for future writes while keeping existing versions intact.

It is good practice to enable versioning before a bucket goes into production use, since any objects uploaded prior to enabling it will not retroactively get version IDs; they are only versioned from that point forward.

You attach a JSON policy document to the bucket, either through the console's Permissions tab, the CLI, or infrastructure-as-code. To restrict access to a specific IP range, for example:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "RestrictToOfficeIP",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"],
    "Condition": {"NotIpAddress": {"aws:SourceIp": "203.0.113.0/24"}}
  }]
}

Apply it with the command aws s3api put-bucket-policy --bucket my-bucket --policy file://policy.json. Because Deny statements always override Allow statements from any source, IAM or bucket policy, an explicit Deny like this is the reliable way to lock a bucket down regardless of what other permissions exist.

CRR automatically and asynchronously copies objects from a bucket in one AWS Region to a bucket in a different Region, based on a replication rule you configure.

It requires versioning enabled on both the source and destination buckets, and it is commonly used for disaster recovery, reducing latency for geographically distributed users, or meeting compliance requirements that mandate data residency in a second region.

Replication is not retroactive by default; only objects created after the rule is set up are replicated, unless you run a one-time batch replication job for existing objects. It also does not replicate lifecycle-triggered deletions unless delete marker replication is explicitly enabled.

S3 and EBS solve different storage problems and are not interchangeable for most workloads.

In short, EBS behaves like a virtual hard drive for one instance, while S3 is a durable, shared object store accessible from anywhere over the network.

Eleven nines, 99.999999999%, refers to the statistical likelihood that a given object will not be lost over a year, and it comes from how S3 stores data: every object is redundantly stored across a minimum of three physically separate Availability Zones for Standard, Standard-IA, and Intelligent-Tiering, each further protected with erasure coding and checksums.

Concretely, if you stored 10,000,000 objects in S3, you would statistically expect to lose one object roughly once every 10,000 years. That figure describes durability, protection against permanent data loss, not availability, which measures how often the data is reachable and is a separate, lower percentage, typically 99.9% to 99.99% depending on class.

This durability model is why S3 does not require you to build your own cross-AZ replication for basic resilience; it is already built into the service.

Presigned URLs let you grant temporary, time-limited access to a specific S3 object without making the bucket or object public, and without giving the requester any AWS credentials.

A trusted party with existing credentials, such as your backend server, generates the URL, signing it with their credentials and an expiration time. Anyone holding that URL can then perform the specified action, typically GET or PUT, until it expires. This is commonly used to let a browser upload directly to S3, or let a user download a private file, without routing the actual bytes through your application server.

url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "invoice.pdf"},
    ExpiresIn=3600
)

MFA Delete adds a second authentication factor requirement before two particularly destructive actions can succeed: permanently deleting an object version, and changing the bucket's versioning state from Enabled to Suspended.

Without it, anyone with sufficient IAM permissions, or a compromised credential with those permissions, can silently purge version history or turn off the safety net versioning provides, defeating the point of keeping past versions around. MFA Delete forces a valid one-time code from a registered MFA device on top of normal credentials for those specific calls.

It is mainly a protection against compromised credentials or insider mistakes for buckets holding critical, hard-to-recreate data. It can only be enabled by the bucket owner's root account via the CLI, and it adds friction, so it is typically reserved for genuinely sensitive buckets rather than applied universally.

S3 was designed as a distributed, highly available object store, not a POSIX-compliant filesystem, so it deliberately skips features like file locks, byte-range writes, or atomic renames that assume a single machine's local disk semantics.

Each PUT to a key is an atomic, whole-object write; an object is either fully written or the write fails, and there is no partial in-place modification of an existing object. That design avoids the coordination overhead that true locking across a distributed, multi-AZ system would require, and it is part of what lets S3 scale to effectively unlimited concurrent clients.

Applications that need locking semantics on top of S3-backed data typically implement it themselves, using a separate coordination layer like DynamoDB conditional writes, or use S3 Object Lock, which enforces write-once-read-many retention rather than traditional mutual-exclusion locking.

S3 stores redundant copies of each object's data across multiple physically separate Availability Zones within a region, a minimum of three for most storage classes, so the loss of an entire AZ, whether from power, network, or facility failure, does not make the data unreachable.

Requests are served through AWS's internal network from whichever healthy copies are available, and this is handled transparently; the client never specifies which AZ to read from. Behind that redundancy sits AWS's broader infrastructure, including automated health checking, self-healing storage nodes, and continuous background verification of data integrity via checksums.

The exception is S3 One Zone-IA, which intentionally stores data in only a single AZ to lower cost, trading away that cross-AZ resilience for data that is easily re-creatable if lost.

S3 supports server-side encryption for data at rest, and since 2023 it is applied by default to every new object using SSE-S3, even if you do not configure anything.

Three server-side options exist: SSE-S3, where AWS manages the encryption keys entirely; SSE-KMS, where keys are managed in AWS KMS, giving you audit trails and control over key policies and rotation; and SSE-C, where you supply your own encryption key with each request and AWS never stores it. Client-side encryption is also possible, where data is encrypted before it ever leaves your application.

aws s3api put-object --bucket my-bucket --key file.txt \
  --body file.txt --server-side-encryption aws:kms \
  --ssekms-key-id alias/my-key

SSE-KMS is generally preferred when you need fine-grained access control and logging on key usage, at the cost of KMS API request charges on high-volume workloads.

Transfer Acceleration speeds up uploads and downloads over long geographic distances by routing traffic through AWS's global network of CloudFront edge locations instead of over the public internet the whole way to the bucket's region.

It is most useful when clients are geographically far from the bucket's region, for example users uploading large files from Asia to a bucket in the US, and it particularly helps with large object transfers where network variability adds up over time.

It is not useful, and can even be slightly slower, when the client is already close to the bucket's region, or for very small objects where connection setup overhead outweighs any routing benefit. AWS provides a speed comparison tool to test before enabling it, since it adds a small additional cost per GB transferred.

Choose One Zone-IA when the data is infrequently accessed and can be easily and cheaply re-created or re-downloaded if lost, for example secondary backup copies, thumbnails you can regenerate, or replicated data where the primary copy already lives safely elsewhere.

One Zone-IA costs about 20% less than Standard-IA because it stores data in a single Availability Zone instead of a minimum of three, meaning the loss of that one AZ, a rare but real event, could result in permanent data loss for that object.

It is a poor fit for primary copies of irreplaceable data, or for any workload where a single-AZ outage would be unacceptable, since the cost savings come directly at the expense of that redundancy.

Rather than removing data, S3 inserts a new delete marker as the current version of that key. The object effectively disappears from a normal GET or a bucket listing, since S3 treats the delete marker as the latest version and returns a 404, but every prior version of the object remains stored and retrievable by its specific version ID.

To truly and permanently remove the data, you must delete that specific version ID directly, a permanent delete, which erases that version's bytes for good, or delete the delete marker itself if you want to undelete and restore the object to visibility.

This two-step behavior is precisely why versioning is such a strong safeguard against accidental deletion: a normal delete is always reversible unless someone explicitly targets a version ID.

Both are JSON documents that grant or deny permissions, but they differ in where they are attached and who they can apply to.

In practice they are combined: IAM defines what your own principals can do across many resources, while bucket policies fine-tune or extend access at the resource level, including to outside accounts. When they conflict, an explicit Deny in either one wins.

For genuinely unpredictable access patterns, Intelligent-Tiering is generally the better choice, because it automatically shifts each object to a cheaper tier once it goes cold, without you having to guess correctly up front or write lifecycle rules based on assumptions that might not hold.

S3 Standard would work correctly too, since it is designed for frequent access, but you would pay Standard rates even for objects that turn out to sit untouched for months, since Standard never automatically re-tiers anything on its own.

The trade-off is Intelligent-Tiering's small per-object monitoring fee, which makes it less attractive for very small objects or for data you are confident will always be accessed frequently; in that narrower case, plain Standard, without the monitoring overhead, is the more cost-efficient pick.

Cost optimization in S3 usually comes from a combination of the following levers, applied based on actual access patterns rather than guesswork:

• Move infrequently accessed data to Standard-IA, One Zone-IA, or Glacier tiers via lifecycle rules.
• Use Intelligent-Tiering when access patterns are unpredictable, to automate that tiering.
• Expire or delete data you no longer need, including old object versions and incomplete multipart uploads.
• Use S3 Storage Lens or S3 Inventory to identify cold or unused data at scale before deciding on rules.
• Compress objects before upload and avoid unnecessary duplication across buckets.

The common mistake is applying aggressive lifecycle transitions without first analyzing actual access patterns, which can backfire: moving frequently read data into Glacier, for instance, introduces retrieval fees and delays that end up costing more than leaving it in Standard.

A 403 on S3 almost always traces back to a permissions gap somewhere in the chain, so troubleshooting means checking each layer in order rather than guessing:

1. Check the IAM policy attached to the calling user or role; confirm the action, such as s3:GetObject, and the exact resource ARN, including the trailing /* for object-level actions, are both covered.
2. Check the bucket policy for an explicit Deny, which overrides any Allow from IAM.
3. Check S3 Block Public Access settings at the account and bucket level, which can silently override otherwise-correct policies.
4. If the object was uploaded by a different AWS account, check the object's ACL and confirm bucket-owner-full-control was set at upload time.
5. For SSE-KMS encrypted objects, confirm the caller also has kms:Decrypt permission on the specific key.

AWS CloudTrail data events, if enabled for S3, can pinpoint exactly which policy evaluation caused the denial, which is faster than manually re-deriving policy logic in most real cases.

All three encrypt data at rest, but they differ in who manages the encryption key and what visibility or control you get over its use.

Most teams default to SSE-S3 or the now-standard automatic encryption, and move to SSE-KMS specifically when they need per-key access control, rotation policies, or an audit log of exactly who decrypted what and when.

Object Lock enforces write-once-read-many protection at the object-version level, preventing a version from being deleted or overwritten for a period you define, even by an account administrator, once it is set.

It offers two modes: Governance mode, where users with a special permission, s3:BypassGovernanceRetention, can still override the lock, useful for internal policies with an emergency escape hatch; and Compliance mode, where no one, including the root account, can shorten or remove the retention period once set, intended for strict regulatory requirements like SEC 17a-4.

Object Lock also supports an independent legal hold, which blocks deletion indefinitely until explicitly removed, regardless of any retention period, and it requires versioning to be enabled on the bucket since it operates on specific object versions.

S3 Block Public Access is a set of four independent settings that can be applied at the account level or per bucket, and they override any bucket policy or ACL that would otherwise grant public access.

aws s3api put-public-access-block \
  --bucket my-bucket \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

The four settings separately control whether new public ACLs can be added, whether existing public ACLs are ignored, whether policies granting public access can be applied at all, and whether public access via policy is blocked even if one exists. Since 2023, all four are enabled by default on every newly created bucket.

For genuinely public use cases like static website hosting, you would deliberately disable the relevant settings and pair that with a narrowly scoped bucket policy, rather than leaving Block Public Access off broadly.

S3 Select lets you run a simple SQL-like query against a single object in CSV, JSON, or Parquet format and retrieve only the matching rows or columns, instead of downloading and parsing the entire object client-side.

SELECT s.name, s.amount FROM s3object s WHERE s.amount > 1000

Because filtering happens on the S3 side before the data leaves the service, it reduces both the data transferred over the network and the client-side compute needed to parse and filter a large file, which can meaningfully cut cost and latency for workloads that only need a small subset of a large object.

It is best suited to targeted, single-object queries. For broader analytics across many objects, Amazon Athena, built on the same underlying idea but scaling across entire S3 datasets with a full SQL engine, is the more appropriate tool.

S3 and CloudFront are complementary, not competing; they are usually used together, not as alternatives.

A typical pattern is an S3 bucket as the origin holding the actual files, with CloudFront in front of it caching and delivering content closer to end users worldwide, often while keeping the bucket itself private and reachable only through CloudFront via an Origin Access Control.

An object's storage-class lifecycle is driven by lifecycle configuration rules attached to the bucket, evaluated once daily, and it typically moves from warmer to colder tiers as it ages, unless Intelligent-Tiering handles that automatically based on access instead of a fixed schedule.

flowchart LR
  A["Object uploaded to S3 Standard"] -->|Day 30, no access| B["S3 Standard-IA"]
  B -->|Day 90| C["S3 Glacier Flexible Retrieval"]
  C -->|Day 180| D["S3 Glacier Deep Archive"]
  D -->|Day 365, Expiration rule| E["Object permanently deleted"]

Each transition is a background, asynchronous S3-managed operation; the object's key and metadata stay the same, only its storage class and underlying physical placement change, so applications reading the object never need to know which class it currently sits in, aside from potential retrieval latency and cost differences for archive tiers.

Transitions have minimum object-age and minimum size constraints. For example, S3 will not transition an object to Standard-IA before 30 days, since the class itself has that minimum storage duration, and lifecycle rules will reject a configuration that violates those constraints.

Multipart upload breaks a large object into independent parts, uploads them, potentially in parallel and out of order, and has S3 reassemble them once every part is confirmed, which improves throughput and lets a failure affect only one part instead of the whole transfer.

sequenceDiagram
  participant App as Client/App
  participant S3 as Amazon S3
  App->>S3: CreateMultipartUpload (returns UploadId)
  loop For each part
    App->>S3: UploadPart (part number, data)
    S3-->>App: ETag for that part
  end
  App->>S3: CompleteMultipartUpload (list of part numbers + ETags)
  S3-->>App: Final object with combined ETag

The client must track each part's returned ETag and pass the full list back in the CompleteMultipartUpload call, in the correct part-number order. S3 uses that list to validate every part arrived intact before assembling the final object. If the process is abandoned partway, an AbortMultipartUpload call, or a lifecycle rule, is needed to clean up the orphaned parts, since incomplete uploads otherwise continue to incur storage charges indefinitely.

Event notifications let S3 automatically publish a message whenever a specified event happens to an object, such as ObjectCreated, ObjectRemoved, or a restore completing from Glacier, to one of three destinations: SNS, SQS, or Lambda.

sequenceDiagram
  participant Client as Client
  participant S3 as Amazon S3
  participant Dest as SNS / SQS / Lambda
  Client->>S3: PUT object
  S3->>S3: Detect matching event (ObjectCreated)
  S3->>Dest: Publish event notification (async, at-least-once)
  Dest-->>Dest: Trigger downstream processing (e.g. Lambda function runs)

Delivery is asynchronous and at-least-once, meaning a given event could theoretically be delivered more than once, so downstream consumers like a Lambda function should be written to be idempotent rather than assuming exactly-once processing.

Notification rules are configured per bucket with an optional prefix/suffix filter, and a common pattern is chaining them: an upload triggers a Lambda that processes the file, for example generating a thumbnail, and writes the result back to another prefix or bucket, which can itself trigger a further notification.

S3 resolves concurrent writes to the same key using last-writer-wins, based on the order S3 itself receives and processes the requests, not necessarily the order the clients initiated them in, since network timing can reorder concurrent requests in flight.

Without versioning enabled, whichever PUT S3 processes last simply becomes the current object; the earlier write's data is gone with no recoverable trace. With versioning enabled, both writes succeed and both become distinct versions with distinct version IDs, so no data is silently lost, though the object returned by a plain GET is still whichever version happened to land last.

S3 does not offer built-in optimistic concurrency control, such as a conditional write-only-if-unchanged, for most operations, though conditional writes using If-Match/If-None-Match on PutObject were added to help specific compare-and-swap style use cases avoid silently clobbering concurrent writers.

The ETag is essentially a fingerprint of an object's content, returned in response headers and usable for cache validation and basic integrity checks.

For an object uploaded in a single PUT without server-side encryption using KMS, the ETag is simply the MD5 hash of the object's contents, in hexadecimal. For objects uploaded via multipart upload, however, the ETag is not a simple MD5 of the full content; it is a hash of the concatenated per-part MD5 hashes, followed by a dash and the number of parts, for example "a1b2c3...-4", which is why comparing ETags across a multipart-uploaded object and a locally computed MD5 of the same file usually will not match.

Because of that inconsistency, ETag should be treated as a good tool for detecting whether an object has changed at all, such as in an HTTP If-None-Match cache check, rather than relied on as a universal, comparable content checksum across all upload methods.

1. Confirm whether large objects are using multipart upload/download with multiple concurrent parts; a single-threaded transfer of a large object under-uses available bandwidth.
2. Check the request rate against a single prefix. While S3 scales automatically, a sudden burst of requests against one narrow key prefix can see brief throttling before it scales, so spreading load across varied prefixes, or simply ramping up gradually, helps.
3. Verify the client's network path. Cross-region or cross-continent transfers benefit from S3 Transfer Acceleration, while same-region transfers from EC2 in the same region should not need it.
4. Check whether client-side CPU is the actual bottleneck, such as encryption or compression happening client-side, rather than the network or S3 itself.
5. For downloads, use byte-range GETs to fetch a large object in parallel chunks, similar in spirit to multipart upload.

AWS's s5cmd tool or the CLI's aws s3 sync with tuned max_concurrent_requests are common ways to get meaningfully better throughput than a naive single-threaded transfer script, without changing infrastructure.

Both control access, but they solve different problems, especially at scale with many applications sharing a bucket.

Access Points are especially useful when many different applications or teams need distinct, simpler permission sets against the same shared bucket, since managing that as one increasingly complex bucket policy with many conditional statements becomes hard to reason about and audit as it grows.

Batch Operations lets you run a single operation across a massive list of objects, potentially billions, without writing custom code to loop through and call the S3 API for each one individually.

You supply a manifest, typically a CSV file or an S3 Inventory report, listing the target objects, choose an operation such as copy, tag, set ACL, restore from Glacier, invoke a Lambda function per object, or apply Object Lock retention, and S3 manages running it at scale, tracking progress, retrying failures, and producing a completion report.

It is commonly used for large-scale bulk restores from Glacier ahead of a planned analysis, mass-tagging objects for a new lifecycle policy, or re-encrypting an entire bucket's worth of existing objects after a change to the encryption strategy, tasks that would otherwise require a custom script carefully handling pagination, retries, and rate limiting.

The standard approach combines a bucket policy on the resource-owning account with an IAM policy on the accessing account's side, since access must be granted from both directions for cross-account requests to succeed.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::222222222222:root"},
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::shared-bucket", "arn:aws:s3:::shared-bucket/*"]
  }]
}

The bucket-owning account attaches that bucket policy naming the other account's root as principal. Separately, an administrator in the accessing account must also attach an IAM policy to the specific user or role that will make the calls, granting the same or a subset of those actions; a bucket policy alone does not automatically grant permission to every identity in the trusted account.

An alternative pattern uses IAM role assumption, where the accessing account assumes a role defined in the bucket-owning account via sts:AssumeRole, which can be cleaner to audit than maintaining matching policies in two accounts, especially as the number of cross-account relationships grows.

Since December 2020, S3 provides strong read-after-write consistency for all operations, for both new object PUTs and overwrite PUTs, as well as consistency for LIST and DELETE, a change from the earlier eventual-consistency model S3 originally shipped with.

sequenceDiagram
  participant App as Client
  participant S3 as Amazon S3 (distributed backend)
  App->>S3: PUT object (new or overwrite)
  S3-->>S3: Write confirmed durable across AZs
  S3-->>App: 200 OK
  App->>S3: GET same key (immediately after)
  S3-->>App: Latest version, guaranteed

This means that as soon as a successful response is returned for a PUT, DELETE, or LIST, every subsequent read is guaranteed to reflect that change; there is no window where a different client, or the same client, could read stale data, which removed a class of application-level workarounds, like artificial delays or retry loops, that were previously considered best practice.

The underlying mechanism relies on S3 internally coordinating metadata updates across its replicated storage nodes before acknowledging the write to the client, rather than acknowledging optimistically and letting metadata propagate afterward, which is the general technique eventually-consistent systems use instead.

Requester Pays shifts data transfer and request costs from the bucket owner to whoever is making the request, which changes who is economically responsible for a bucket's traffic without changing anything about how objects are stored.

It is commonly used for publicly shared, large datasets, such as research data or open datasets, where the owner wants to make data available without absorbing potentially unbounded download costs if the data becomes popular. Requesters must include a specific header, x-amz-request-payer: requester, acknowledging they accept the charges, or the request is rejected.

An S3 Select query runs entirely within S3's infrastructure against a single object, streaming only the matching, filtered data back to the client instead of the object's full contents.

flowchart LR
  A["Client sends SelectObjectContent request with SQL expression"] --> B["S3 locates the target object"]
  B --> C["S3 parses object as CSV/JSON/Parquet"]
  C --> D["S3 applies WHERE/SELECT filtering server-side"]
  D --> E["Matching records streamed back to client as a series of events"]

The response arrives as a stream of event messages, Records, Stats, and End, rather than one flat payload, letting the client start processing matching rows before the entire result set has finished streaming, which is particularly useful when only a small fraction of a large object actually matches the query.

Because the filtering logic runs inside S3 rather than after a full download, both the network bytes transferred and the client-side CPU spent parsing and discarding non-matching rows are avoided, which is the entire performance and cost rationale for using S3 Select over a plain GET plus local filtering.