Amazon S3 @amazon-simple-storage-service :: Object storage built to store and retrieve any amount of data, designed for 11 nines of durability > Amazon S3 stores data as objects (data plus metadata, identified by a key) in buckets. You can store an unlimited number of objects, and S3 provides strong read-after-write consistency for PUT and DELETE requests in all Regions. > S3 storage classes are designed for 99.999999999% (11 nines) durability. Most keep data across at least three Availability Zones, and you choose a class to balance cost against access speed. > S3 suits data lakes, websites, mobile applications, backup and restore, archive and analytics. It is object storage, so workloads that need a mounted file system or a block device should use EFS, FSx or EBS. + Amazon S3 User Guide + Security best practices for Amazon S3 + Amazon S3 pricing + Amazon S3 FAQs Storage classes > Each object has a storage class that sets its storage price, retrieval fees and how fast you can read it. The current classes are all designed for 11 nines of durability, but they differ in availability, Availability Zone count, minimum storage duration and minimum billable object size. > Pick the class from how often you read the data, and use lifecycle rules or S3 Intelligent-Tiering to move objects as access fades. A common pitfall is putting short-lived or tiny objects in an IA or Glacier class, where minimum duration and size charges apply. + Understanding S3 storage classes + Amazon S3 storage classes + Amazon S3 Storage Class Analysis S3 Standard :: Frequently accessed data > The default class, with millisecond access, no retrieval fee and no minimum storage duration. It is designed for 99.99% availability across at least three Availability Zones. S3 Intelligent-Tiering :: Moves objects between access tiers automatically > Moves each object between access tiers based on access patterns: Frequent Access, then Infrequent Access after 30 days and Archive Instant Access after 90 days without access. Optional Archive Access and Deep Archive Access tiers need a restore before reading. > There are no retrieval fees, but a monthly per-object monitoring charge applies, and objects smaller than 128 KB are not auto-tiered. Choose it when access patterns are unknown or change. + Managing storage costs with S3 Intelligent-Tiering S3 Express One Zone :: Single-digit millisecond access in one Availability Zone > Built for your most latency-sensitive applications, with consistent single-digit millisecond access. Data is stored in a single Availability Zone you choose, in a directory bucket, so it is not resilient to the loss of that zone. It is designed for 99.95% availability. + S3 Express One Zone S3 Standard-IA :: Infrequent access with millisecond retrieval > For long-lived data accessed about once a month. It stores data across at least three Availability Zones, charges a per-GB retrieval fee, and has a 30-day minimum storage duration and a 128 KB minimum billable object size. S3 One Zone-IA :: Infrequent access for data you can re-create > Same pricing model as Standard-IA but in one Availability Zone and designed for 99.5% availability. If that zone is lost, the data can be lost, so use it for re-creatable data or secondary copies. S3 Glacier Instant Retrieval @amazon-simple-storage-service-glacier :: Archive with millisecond retrieval > For archive data read about once a quarter that must still be available instantly. It has a 90-day minimum storage duration, a 128 KB minimum billable object size and a retrieval fee. S3 Glacier Flexible Retrieval @amazon-simple-storage-service-glacier :: Archive with retrieval in minutes to hours > Objects are archived and must be restored with RestoreObject before reading. Expedited typically takes 1 to 5 minutes, Standard 3 to 5 hours and Bulk 5 to 12 hours, and the minimum storage duration is 90 days. + Understanding archive retrieval options S3 Glacier Deep Archive @amazon-simple-storage-service-glacier :: Lowest-cost archive; Standard retrieval within 12 hours > The lowest-cost class, for data read less than once a year. Standard restores typically finish within 12 hours, Bulk within 48 hours, and Expedited is not available. The minimum storage duration is 180 days. + Understanding S3 Glacier storage classes Data management > These features control how long objects live, how many versions exist, where copies go and whether data can be changed. Versioning is the foundation: replication and Object Lock both require it. > Most S3 data protection and cost questions come down to combining these four: version, expire old versions with lifecycle, replicate for resilience and lock for compliance. + Retaining multiple versions of objects with S3 Versioning Versioning :: Keep and restore every version of an object > When versioning is on, a delete adds a delete marker and an overwrite keeps the old version, so you can restore earlier data. Versioning can be suspended but never removed from a bucket, and every stored version is billed. Lifecycle rules :: Transition or expire objects on a schedule > Rules transition objects to other storage classes or expire them. Expiration and noncurrent-version rules matter once versioning is on, and an AbortIncompleteMultipartUpload rule removes abandoned upload parts. Replication :: Cross-Region and same-Region replication > Live replication copies new and updated objects asynchronously to a destination bucket in the same or another Region, in the same or a different account. Versioning must be enabled on both buckets, and objects that existed before you set up replication are copied with S3 Batch Replication. Object Lock :: WORM retention in governance or compliance mode > Object Lock stops an object version from being deleted or overwritten for a retention period. In governance mode, users with special permission can override it, and in compliance mode no one can, including the root user. Legal holds protect an object with no end date. Batch Operations :: Run one action on billions of objects > Runs one operation, such as Copy, Invoke AWS Lambda function or Restore, across millions or billions of objects with a single request. Storage Lens :: Organization-wide usage and activity metrics > Provides 60+ usage and activity metrics and dashboards that aggregate data for an organization, accounts, Regions, buckets or prefixes. Security > S3 is private by default and uses layered controls: IAM and bucket policies for permissions, Block Public Access as a safety net, and encryption for data at rest and in transit. > Current defaults for new buckets are Block Public Access on, ACLs disabled through Object Ownership and SSE-S3 encryption of every new object. + Security best practices for Amazon S3 + Protecting data with encryption Block Public Access :: On by default for new buckets > Block Public Access settings override bucket policies and ACLs that would grant public access. They are on by default for new buckets and can be applied to an account or a bucket. Bucket policies :: Resource-based access control > A resource-based IAM policy attached to a bucket that allows or denies requests based on the requester, the S3 action and conditions such as source IP address or source VPC endpoint. Bucket policies are limited to 20 KB. Object Ownership :: ACLs turned off by default; the bucket owner owns every object > The default setting, bucket owner enforced, disables ACLs so the bucket owner owns every object and access is managed only through policies. Encryption @aws-key-management-service :: SSE-S3 by default; SSE-KMS and DSSE-KMS optional > Since January 5, 2023, all new objects are encrypted at rest with SSE-S3 at no additional cost and with no impact on performance. You can instead specify SSE-KMS, DSSE-KMS (dual-layer) or SSE-C per request, and set SSE-KMS or DSSE-KMS as a bucket default. If you use SSE-KMS, check the AWS KMS request quotas for your request rates. + Default encryption FAQ Access points :: Named endpoints with their own policies for shared datasets > An access point is a named network endpoint attached to a bucket, each with its own policy and optional VPC restriction. It simplifies managing access when many applications share one bucket. Amazon Macie @amazon-macie :: Find sensitive data in buckets > Macie discovers sensitive data in S3 objects by using machine learning and pattern matching, and evaluates general purpose buckets for security and access control, for example a bucket that becomes publicly accessible. Performance > S3 scales request rates automatically, and you improve throughput mainly by spreading requests across prefixes and using parallel connections. Rates grow gradually, and you may see 503 Slow Down responses while S3 scales. > Latency-sensitive paths can add CloudFront or ElastiCache in front, or use S3 Express One Zone. + Best practices design patterns: optimizing Amazon S3 performance + Performance guidelines for Amazon S3 Request rates :: At least 3,500 writes and 5,500 reads per second per prefix > Applications can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix, and there is no limit on the number of prefixes in a bucket. AWS gives the example that 10 prefixes can scale reads to about 55,000 requests per second. Multipart upload :: Best practice from 100 MB; a single PUT is limited to 5 GB > Multipart upload splits an object into parts that upload in parallel and can be retried independently. AWS recommends it for objects of 100 MB or larger, and a single PUT supports objects only up to 5 GB. > A multipart upload can have up to 10,000 parts of 5 MiB to 5 GiB each. Uploaded parts are billed until you complete or stop the upload, so add a lifecycle rule to abort incomplete uploads. Transfer Acceleration :: Faster uploads over long distances through edge locations > Uses the globally distributed CloudFront edge locations: data arrives at an edge location and is routed to S3 over an optimized network path. It suits uploads from clients far from the bucket, and additional data transfer charges might apply. Byte-range fetches :: Download parts of an object in parallel > A GET request with a Range header returns only part of an object. Issuing several range requests in parallel is one way to apply the parallelization that AWS recommends for higher read throughput. Newer bucket types > S3 supports four bucket types: general purpose, directory, table and vector buckets. The newer three are built for specific workloads and use their own API namespaces, and Block Public Access settings are always enabled for table and vector buckets. S3 Tables :: Managed Apache Iceberg tables for analytics > Server access logging for table buckets records requests to all tables and namespaces and delivers the logs to a general purpose bucket in the same account and Region, in the same format as S3 server access logs. + S3 Tables server access logging > Table buckets store tables in Apache Iceberg format and automatically run compaction, snapshot management and unreferenced file removal. You query them with Amazon Athena, Amazon Redshift, Apache Spark and other Iceberg engines, and can integrate them with AWS analytics services through the AWS Glue Data Catalog. By default you can create up to 10 table buckets per account per Region. + S3 Tables maintenance + Integrating S3 Tables with AWS analytics services S3 Vectors :: Vector buckets and indexes for embeddings > Vector buckets hold vector indexes, and you store vectors with optional metadata and run similarity queries through dedicated APIs. AWS describes subsecond latency for infrequent queries and as low as 100 milliseconds for frequent ones, and integrations with Amazon Bedrock Knowledge Bases and Amazon OpenSearch Service. + S3 Vectors limitations and restrictions Directory buckets :: For S3 Express One Zone > Directory buckets organize objects into directories instead of a flat structure and have no prefix limits. By default you can create up to 100 per account. Block Public Access is on and ACLs are disabled, and neither can be changed. Directory buckets in an Availability Zone support S3 Express One Zone.

Amazon S3 mindmap