Storage Architecture

TL;DR

• Object (S3), block (EBS), and file (EFS/FSx) storage solve different access-pattern problems — pick based on how data is actually read/written, not by default habit.
• EBS attaches to a single instance at a time (Multi-Attach is a narrow clustered-DB exception); anything needing genuine concurrent multi-instance access needs EFS or FSx, not EBS.
• S3 does not auto-tier data to cheaper storage classes unless Lifecycle policies or Intelligent-Tiering are explicitly configured — untouched Standard-tier data is a common, easily fixed cost leak.
• S3 Cross-Region/Same-Region Replication only applies to objects written after replication is enabled; existing objects require an explicit S3 Batch Replication job to backfill.
• Glacier tiers trade retrieval latency for cost — picking the wrong tier only surfaces as a problem during an actual retrieval event.
• Storage Gateway bridges on-prem NFS/SMB applications to S3-backed storage without requiring an application rewrite.

Core Concepts

S3 (object storage)

Durable (11 nines design target), replicated across multiple AZs within a Region by default, effectively unlimited scale, and priced per GB-month plus request/transfer costs. The natural fit for anything that doesn’t need a POSIX filesystem: static assets, backups, data lake storage, build/application artifacts, log archives.

Lifecycle policies are the most commonly missed cost lever. S3 never moves aging data to a cheaper storage class on its own — that requires either:

• Explicit lifecycle rules (transition to Standard-IA after N days, then Glacier, then Glacier Deep Archive), or
• S3 Intelligent-Tiering, which monitors access patterns per object and moves data between tiers automatically for a small monitoring fee, with no retrieval fee.

Buckets holding years of untouched Standard-tier data (build artifacts, old logs) are one of the highest-leverage, lowest-effort cost optimizations available in a typical AWS account.

EBS (block storage)

A network-attached virtual disk for a single EC2 instance. Backed by replicated storage within an AZ (not cross-AZ) for durability, and must be in the same AZ as the instance it’s attached to. Used for boot volumes and database data volumes — anything that expects normal block-device/filesystem semantics from a single host.

Multi-Attach allows a Provisioned IOPS SSD (io1/io2) volume to attach to multiple instances simultaneously, but it is not a general-purpose shared-storage solution — it requires cluster-aware filesystems/applications (e.g., certain clustered database engines) that manage concurrent writes themselves. It is not a substitute for EFS.

EFS (elastic file storage)

A managed NFS filesystem that scales elastically and supports concurrent read/write access from many EC2 instances (and Lambda, and on-prem via Direct Connect/VPN) simultaneously. The right tool whenever multiple compute nodes need to share the same live files: a fleet of web servers serving uploaded user content, shared configuration across an Auto Scaling group, a CMS’s media library. Offers Standard and Infrequent Access storage classes, plus lifecycle management to move files between them automatically.

FSx (family of managed file systems)

Not one service — the variant chosen must match the actual protocol/performance requirement:

Storage Gateway

Bridges on-prem infrastructure to AWS storage without requiring application rewrites:

• File Gateway — presents an NFS/SMB interface backed by S3, with local caching of frequently accessed data.
• Volume Gateway — presents iSCSI block volumes, either cached (primary data in S3, local cache for hot data) or stored (full copy on-prem, async backup to S3).
• Tape Gateway — presents a virtual tape library interface for existing backup software, backed by S3/Glacier.

S3 Glacier tiers

For data retained for compliance or archival but rarely or never actively accessed. The tier choice is a retrieval-latency/cost trade-off:

• Glacier Instant Retrieval — millisecond access, cheaper storage than S3 Standard-IA, for rarely-accessed data needed immediately when requested.
• Glacier Flexible Retrieval — retrieval in minutes to hours (expedited/standard/bulk tiers), cheaper storage than Instant Retrieval.
• Glacier Deep Archive — cheapest tier, retrieval measured in hours (standard 12h, bulk up to 48h) — for data accessed rarely, if ever, such as long-term compliance archives.

S3 Replication

Cross-Region Replication (CRR) and Same-Region Replication (SRR) asynchronously copy objects to a destination bucket based on rules (prefix, tags, or entire bucket). The critical gotcha: replication only applies going forward — objects that existed in the bucket before replication was configured are not backfilled automatically. Backfilling requires explicitly running S3 Batch Replication. A DR plan that assumes existing data is already replicated, without ever running that batch job, has a gap that only surfaces during an actual failover.

EBS vs EFS vs S3 vs FSx

S3 Storage Class Comparison

Command / Configuration Reference

# Create an S3 lifecycle rule transitioning objects to IA then Glacier
aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json

# Enable S3 Intelligent-Tiering on a bucket via lifecycle config (archive access tiers)
aws s3api put-bucket-intelligent-tiering-configuration --bucket my-bucket --id archive-config --intelligent-tiering-configuration file://it-config.json

# Enable versioning (required before configuring replication)
aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled

# Configure cross-region replication
aws s3api put-bucket-replication --bucket my-bucket --replication-configuration file://replication.json

# Backfill existing objects into a replication destination (does NOT happen automatically)
aws s3control create-job --account-id 123456789012 --operation '{"S3ReplicateObject":{}}' --manifest file://manifest.json --report file://report.json --priority 10 --role-arn arn:aws:iam::123456789012:role/BatchReplicationRole

# Create an EFS filesystem
aws efs create-file-system --performance-mode generalPurpose --throughput-mode bursting

# Create an EBS volume with Multi-Attach enabled (io1/io2 only)
aws ec2 create-volume --availability-zone us-east-1a --size 100 --volume-type io2 --iops 10000 --multi-attach-enabled

# Initiate a Glacier Deep Archive restore
aws s3api restore-object --bucket my-archive-bucket --key file.zip --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}'

Common Pitfalls

• Pitfall: S3 buckets accumulate years of Standard-tier data with no cost tiering. Why: S3 never auto-tiers data unless lifecycle rules or Intelligent-Tiering are explicitly configured. Fix: Audit buckets for lifecycle policy coverage; enable Intelligent-Tiering by default for buckets with unpredictable access patterns.
• Pitfall: EBS Multi-Attach used as a general shared-storage workaround. Why: Multi-Attach requires cluster-aware applications to manage concurrent writes themselves; it is not a generic multi-instance filesystem. Fix: Use EFS (or FSx) for genuine concurrent multi-instance shared file access.
• Pitfall: A DR plan assumes S3 Replication covers all historical data. Why: Replication only applies to objects written after it was enabled; pre-existing objects are never backfilled automatically. Fix: Run S3 Batch Replication explicitly to backfill existing objects, and verify it during a DR drill.
• Pitfall: Wrong FSx variant chosen for the workload. Why: FSx is a family with distinct protocol/performance characteristics per variant, not interchangeable options. Fix: Match FSx for Windows File Server / Lustre / ONTAP / OpenZFS to the actual protocol and performance requirement before committing.
• Pitfall: A Glacier tier is selected purely on storage cost. Why: Cheaper tiers carry longer retrieval times, which may not meet a real compliance or operational deadline. Fix: Confirm retrieval time requirements against the actual tier’s restore SLA before archiving data, and test a retrieval at least once.

Interview Questions

• “A cost review finds 40TB of S3 data untouched for 3 years, still in Standard tier. Walk through the remediation plan.” — tests whether lifecycle policy design and Intelligent-Tiering are the default instinct, not just deletion.
• “Design shared storage for a fleet of web servers that all need to read and write the same uploaded user content.” — tests whether EFS is reached for instead of an EBS Multi-Attach workaround.
• “A DR plan assumed S3 Replication had all historical data. A drill just proved it didn’t. What happened, and how is the plan fixed?” — tests whether the replication-scope gap (new objects only, no automatic backfill) is understood.
• “When would FSx for Lustre be chosen over EFS?” — tests whether the distinction between general-purpose shared storage and HPC/ML-throughput-optimized storage is understood.
• “Explain the cost/retrieval trade-off across the three Glacier tiers.” — tests whether retrieval SLA is treated as a first-class design constraint, not an afterthought.