Memory hook: Observe the action, identify the actor, preserve the evidence.
Must remember
Kubernetes audit policy selects events, stages and detail levels. Metadata-level records capture request context without full bodies; request/response bodies can contain sensitive information and increase volume. Protect audit destinations, retention and access. An application log, node log and API audit event answer different questions.
Audit rules use the first matching rule. Levels are None, Metadata, Request and RequestResponse; place specific exceptions before broad rules. For a static API-server Pod, the audit-policy file and log destination must be reachable inside the container through correct mounts as well as flags. Inspect actual records for user, verb, object, namespace and response status: a configured file alone does not prove the event reached the intended destination.
Runtime tools such as Falco can detect suspicious behavior from supported event sources: unexpected shells, writes to sensitive paths, privilege changes or unusual network activity. A detection rule is a hypothesis requiring context, not automatic proof of compromise. Tune against legitimate workloads while preserving visibility into important deviations.
Correlate Pod, container, node, namespace, service account, image digest and deployment history. Capture volatile evidence where the response process requires it before restarting/removing the workload. Containment may involve isolating traffic, revoking credentials or stopping a compromised workload; follow authorized incident procedures and preserve a record of actions.
Immutable runtime design uses read-only filesystems, controlled writable volumes and replacement from trusted artifacts instead of manual patching inside a live container. A deleted malicious Pod can simply return if the controller template or source image remains compromised. Fix the source, rotate exposed secrets and verify trusted recovery.
Timed drill: explain how to distinguish a legitimate diagnostic shell from an unauthorized shell using audit identity, deployment context, runtime events and change records. Then identify which evidence a blind restart would lose.
Choose under exam pressure
| Requirement | Choice and reason |
|---|---|
| Who changed a Kubernetes resource? | API audit evidence at the configured level. |
| Unexpected process inside a container | Runtime detection plus workload/identity context. |
| Malicious Pod repeatedly returns | Investigate the controller template, image and credentials. |
Traps
- Audit logging does not capture events retrospectively if it was never enabled.
- Removing one Pod does not repair its desired-state controller.
Active recall
1. What can request-body audit logging expose?
Sensitive data submitted through the API, including secrets.
2. Why correlate image digest?
It identifies the exact artifact involved rather than a mutable tag.
3. Why preserve evidence before restart?
Volatile state and prior container evidence may be lost.
4. What supports container immutability?
Read-only runtime paths and replacement from controlled artifacts.
5. Why rotate exposed credentials?
An attacker may retain access after the compromised workload is removed.