CrackOps
Interview PrepTracksLinuxAWS LabSoonInterview RoomSoon
Sign inStart free
CrackOps

Interview prep for the people who keep production up.

[email protected]
ProductInterview PrepTracksDashboardInterview Room
RolesDevOpsSREPlatformCloud
CompanyAboutContactPrivacyTerms
© CrackOps
Linux / Kubernetes / kubectl logs

kubectl logs

Read and follow container logs

Beginnerkuberneteslogstroubleshooting

kubectl logs prints what a container wrote to stdout and stderr. Follow it live with -f, read the run that crashed with --previous, pick a container with -c, or read every pod behind a label at once.

Syntaxkubectl logs (POD | TYPE/NAME | -l SELECTOR) [-c CONTAINER] [-f] [-p] [--since=10m] [--tail=N] [--timestamps]

Watch it run

Check that the api pods are serving requests normally after this morning's deploy.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
kubectl logs api-7f9c5d4b6-2xkqp -n prod --tail=4

Exactly what the process printed to stdout/stderr, here JSON lines. Without --tail you get the whole file since the container started, which can be enormous.

Anatomy of a command

$
commandkubectl

The Kubernetes CLI.

How it works

Containers don't send logs to the API server. The container runtime (containerd) captures each container's stdout and stderr and writes them to files on the node under /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log (symlinked from /var/log/containers). kubectl logs calls GET /api/v1/namespaces/prod/pods/NAME/log?container=worker&previous=true; the API server proxies that to the kubelet on the pod's node (port 10250), and the kubelet reads the file and streams it back. -f keeps the HTTP response open. Only the current and the one previous container instance are kept, files are rotated at 10Mi by default, and everything is deleted with the pod. That is why clusters ship logs to CloudWatch, Loki or OpenSearch with a DaemonSet such as Fluent Bit. Logs written to files inside the container are invisible to kubectl logs.

Options you'll actually use

-f, --followKeep streaming new lines (Ctrl+C to stop)
-p, --previousLogs from the previous, crashed container instance
-c NAMEContainer to read in a multi-container pod
--all-containersEvery container in the pod, init containers included
-l SELECTORLogs from all pods matching a label (default 10 lines each)
--prefixPrefix each line with [pod/NAME/container]
--tail=NOnly the last N lines
--since=10m / --since-timeOnly lines newer than a duration or a timestamp
--timestampsPrepend the runtime's RFC3339 timestamp to each line
--max-log-requests=NConcurrent streams with -l -f (default 5)

Common mistakes

kubectl logs on a crash-looping pod and seeing only startup lines

You read the fresh instance; the one that crashed is the previous one.

kubectl logs POD --previous
kubectl logs deploy/api to investigate an intermittent error

It reads one pod only; the failing request may have hit another replica.

kubectl logs -l app=api --prefix --since=15m | grep -i error
Writing app logs to /var/log/app.log inside the container

kubectl logs only sees stdout and stderr; files inside the container are lost with the pod.

Log to stdout/stderr and let a node agent (Fluent Bit) ship them
Relying on kubectl logs for last week's incident

Logs live on the node, rotate at about 10Mi and vanish when the pod is deleted or rescheduled.

Query your central log store (CloudWatch Logs, Loki, OpenSearch)

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Write the command to follow the logs of the container named nginx in pod web-5c7b9d6f4-q2x7z in namespace shop, starting from the last 20 lines.

  2. 2

    Write a command that shows only error lines from all pods labelled app=checkout in the last 30 minutes, with the pod name on each line.

  3. 3

    The worker pod in prod keeps restarting. Show the last 5 lines its previous container wrote before it died, then say why it stopped.

  4. 4

    For real practice, open AWS Lab > EKS > your cluster > Run kubectl (lab), deploy the sample app, and read the last 10 lines of one of its pods.

Related commands

kubectl describeExplain an object's state, conditions and eventskubectl execRun a command or shell inside a running containerjournalctlQuery systemd's journal: service, boot and kernel logstailShow the last lines of a file and follow it live
Previouskubectl describeNextkubectl exec
Linux / Kubernetes / kubectl logs

kubectl logs

Read and follow container logs

Beginnerkuberneteslogstroubleshooting

kubectl logs prints what a container wrote to stdout and stderr. Follow it live with -f, read the run that crashed with --previous, pick a container with -c, or read every pod behind a label at once.

Syntaxkubectl logs (POD | TYPE/NAME | -l SELECTOR) [-c CONTAINER] [-f] [-p] [--since=10m] [--tail=N] [--timestamps]

Watch it run

Check that the api pods are serving requests normally after this morning's deploy.

devops@devops-lab: ~ — zsh — 96×28
➜ ~
Step 1 of 2
kubectl logs api-7f9c5d4b6-2xkqp -n prod --tail=4

Exactly what the process printed to stdout/stderr, here JSON lines. Without --tail you get the whole file since the container started, which can be enormous.

Anatomy of a command

$
commandkubectl

The Kubernetes CLI.

How it works

Containers don't send logs to the API server. The container runtime (containerd) captures each container's stdout and stderr and writes them to files on the node under /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log (symlinked from /var/log/containers). kubectl logs calls GET /api/v1/namespaces/prod/pods/NAME/log?container=worker&previous=true; the API server proxies that to the kubelet on the pod's node (port 10250), and the kubelet reads the file and streams it back. -f keeps the HTTP response open. Only the current and the one previous container instance are kept, files are rotated at 10Mi by default, and everything is deleted with the pod. That is why clusters ship logs to CloudWatch, Loki or OpenSearch with a DaemonSet such as Fluent Bit. Logs written to files inside the container are invisible to kubectl logs.

Options you'll actually use

-f, --followKeep streaming new lines (Ctrl+C to stop)
-p, --previousLogs from the previous, crashed container instance
-c NAMEContainer to read in a multi-container pod
--all-containersEvery container in the pod, init containers included
-l SELECTORLogs from all pods matching a label (default 10 lines each)
--prefixPrefix each line with [pod/NAME/container]
--tail=NOnly the last N lines
--since=10m / --since-timeOnly lines newer than a duration or a timestamp
--timestampsPrepend the runtime's RFC3339 timestamp to each line
--max-log-requests=NConcurrent streams with -l -f (default 5)

Common mistakes

kubectl logs on a crash-looping pod and seeing only startup lines

You read the fresh instance; the one that crashed is the previous one.

kubectl logs POD --previous
kubectl logs deploy/api to investigate an intermittent error

It reads one pod only; the failing request may have hit another replica.

kubectl logs -l app=api --prefix --since=15m | grep -i error
Writing app logs to /var/log/app.log inside the container

kubectl logs only sees stdout and stderr; files inside the container are lost with the pod.

Log to stdout/stderr and let a node agent (Fluent Bit) ship them
Relying on kubectl logs for last week's incident

Logs live on the node, rotate at about 10Mi and vanish when the pod is deleted or rescheduled.

Query your central log store (CloudWatch Logs, Loki, OpenSearch)

Interview questions

Practice

Try these on a Linux machine of your own. Each task has a hint and a solution.

  1. 1

    Write the command to follow the logs of the container named nginx in pod web-5c7b9d6f4-q2x7z in namespace shop, starting from the last 20 lines.

  2. 2

    Write a command that shows only error lines from all pods labelled app=checkout in the last 30 minutes, with the pod name on each line.

  3. 3

    The worker pod in prod keeps restarting. Show the last 5 lines its previous container wrote before it died, then say why it stopped.

  4. 4

    For real practice, open AWS Lab > EKS > your cluster > Run kubectl (lab), deploy the sample app, and read the last 10 lines of one of its pods.

Related commands

kubectl describeExplain an object's state, conditions and eventskubectl execRun a command or shell inside a running containerjournalctlQuery systemd's journal: service, boot and kernel logstailShow the last lines of a file and follow it live
Previouskubectl describeNextkubectl exec