aws s3Copy, sync and list files in Amazon S3
aws s3 is the file-style interface to Amazon S3: ls, cp, mv, rm, sync, mb and presign. It handles multipart uploads, parallel transfers and recursion for you, which makes it the everyday tool for artifacts, backups, static site deploys and log downloads.
aws s3 ls|cp|mv|rm|sync|mb|rb|presign SOURCE [DEST] [--recursive] [--exclude/--include] [--delete] [--dryrun]You need to know which release tarballs are in the artifacts bucket and how much space they take.
aws s3High-level S3 commands. (aws s3api exposes the raw API calls one to one.)
S3 is a key-value store over HTTPS, and aws s3 builds file-like behaviour on its REST API. ls calls ListObjectsV2 with delimiter / so keys sharing a prefix show up as PRE logs/, which looks like a folder but isn't one. cp of a small file is one PutObject; above 8 MB the transfer manager switches to multipart upload: CreateMultipartUpload, parts uploaded in parallel (10 threads by default), then CompleteMultipartUpload, so an object's ETag is no longer its MD5. Downloads use ranged GETs in parallel. sync lists both sides and transfers a file when its size differs or the source is newer; with --delete it removes extras. Every call is signed with SigV4 and authorised by IAM policies plus the bucket policy, and since 2023 new buckets block public access and encrypt everything by default (SSE-S3).
ls [s3://bucket/prefix/]List buckets, or objects and prefixes under a prefix--recursiveWith ls/cp/rm: every key under the prefix (ls: add --human-readable --summarize for sizes and totals)cp SRC DSTCopy local to S3, S3 to local, or S3 to S3 (server side)sync SRC DSTCopy only new/changed files; --delete removes extrasmv / rmMove or delete objects (--recursive for a whole prefix)--exclude / --includeFilter files with globs; later filters win--dryrunShow what would happen without doing it--storage-class CLASSCheaper tiers for rarely read data: STANDARD_IA, GLACIER_IR...--sse aws:kmsEncrypt with a KMS key (pick it with --sse-kms-key-id) instead of the S3-managed default--cache-control 'max-age=300'Set HTTP headers on uploaded objects (static sites)presign URIMake a time-limited download URL (--expires-in seconds)mb / rb s3://bucketMake or remove a bucket (rb --force empties it first)aws s3 sync . s3://bucket/ --delete from the wrong directorysync mirrors exactly what you point it at: run from your home directory and it uploads everything, or with --delete wipes objects that aren't there.
Always run with --dryrun first and use an explicit source path (./dist)Making a bucket public to share one fileExposes everything in it, now and in future, and Block Public Access exists precisely to stop this.
aws s3 presign with a short --expires-in, or CloudFront with origin access controlExpecting sync to detect changed content in same-size filessync compares size and modification time, not checksums; a regenerated file with identical size and an older mtime is skipped.
Build fresh in CI, use content-hashed file names, or cp the changed files explicitly--exclude '*' placed after --includeLater filters take precedence, so the trailing exclude removes everything you included.
--exclude '*' --include 'pattern' (exclude first)Try these on a Linux machine of your own. Each task has a hint and a solution.
Download only the .gz files under s3://devops-artifacts/logs/ into ~/logs/s3, then check what arrived.
Practise the sync idea locally: mirror ~/app/src into /tmp/src-mirror with rsync, deleting extra files, first as a dry run.
Deploy ~/web/dist to s3://shop-web-static/site/ as a mirror, previewing first. Which old file gets deleted?
Create a bucket, upload a file, and share it for 10 minutes without making it public. (Common names like test are taken: bucket names are global.)
aws s3Copy, sync and list files in Amazon S3
aws s3 is the file-style interface to Amazon S3: ls, cp, mv, rm, sync, mb and presign. It handles multipart uploads, parallel transfers and recursion for you, which makes it the everyday tool for artifacts, backups, static site deploys and log downloads.
aws s3 ls|cp|mv|rm|sync|mb|rb|presign SOURCE [DEST] [--recursive] [--exclude/--include] [--delete] [--dryrun]You need to know which release tarballs are in the artifacts bucket and how much space they take.
aws s3High-level S3 commands. (aws s3api exposes the raw API calls one to one.)
S3 is a key-value store over HTTPS, and aws s3 builds file-like behaviour on its REST API. ls calls ListObjectsV2 with delimiter / so keys sharing a prefix show up as PRE logs/, which looks like a folder but isn't one. cp of a small file is one PutObject; above 8 MB the transfer manager switches to multipart upload: CreateMultipartUpload, parts uploaded in parallel (10 threads by default), then CompleteMultipartUpload, so an object's ETag is no longer its MD5. Downloads use ranged GETs in parallel. sync lists both sides and transfers a file when its size differs or the source is newer; with --delete it removes extras. Every call is signed with SigV4 and authorised by IAM policies plus the bucket policy, and since 2023 new buckets block public access and encrypt everything by default (SSE-S3).
ls [s3://bucket/prefix/]List buckets, or objects and prefixes under a prefix--recursiveWith ls/cp/rm: every key under the prefix (ls: add --human-readable --summarize for sizes and totals)cp SRC DSTCopy local to S3, S3 to local, or S3 to S3 (server side)sync SRC DSTCopy only new/changed files; --delete removes extrasmv / rmMove or delete objects (--recursive for a whole prefix)--exclude / --includeFilter files with globs; later filters win--dryrunShow what would happen without doing it--storage-class CLASSCheaper tiers for rarely read data: STANDARD_IA, GLACIER_IR...--sse aws:kmsEncrypt with a KMS key (pick it with --sse-kms-key-id) instead of the S3-managed default--cache-control 'max-age=300'Set HTTP headers on uploaded objects (static sites)presign URIMake a time-limited download URL (--expires-in seconds)mb / rb s3://bucketMake or remove a bucket (rb --force empties it first)aws s3 sync . s3://bucket/ --delete from the wrong directorysync mirrors exactly what you point it at: run from your home directory and it uploads everything, or with --delete wipes objects that aren't there.
Always run with --dryrun first and use an explicit source path (./dist)Making a bucket public to share one fileExposes everything in it, now and in future, and Block Public Access exists precisely to stop this.
aws s3 presign with a short --expires-in, or CloudFront with origin access controlExpecting sync to detect changed content in same-size filessync compares size and modification time, not checksums; a regenerated file with identical size and an older mtime is skipped.
Build fresh in CI, use content-hashed file names, or cp the changed files explicitly--exclude '*' placed after --includeLater filters take precedence, so the trailing exclude removes everything you included.
--exclude '*' --include 'pattern' (exclude first)Try these on a Linux machine of your own. Each task has a hint and a solution.
Download only the .gz files under s3://devops-artifacts/logs/ into ~/logs/s3, then check what arrived.
Practise the sync idea locally: mirror ~/app/src into /tmp/src-mirror with rsync, deleting extra files, first as a dry run.
Deploy ~/web/dist to s3://shop-web-static/site/ as a mirror, previewing first. Which old file gets deleted?
Create a bucket, upload a file, and share it for 10 minutes without making it public. (Common names like test are taken: bucket names are global.)