Falco¶

Who this page is for: in plain English, Falco watches the system calls your containers make and alerts when one of them does something it shouldn't - spawning a shell, reading /etc/shadow, opening an unexpected outbound connection. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track.

Falco is a runtime security tool that detects unexpected behavior in containers and Kubernetes workloads by observing Linux system calls. Where admission controllers and policy engines prevent misconfiguration at deploy time, Falco catches active threats during runtime - a compromised container spawning a shell, a process reading credential files, or a container escaping to the host.

The core insight: most attacks, regardless of how they start, ultimately make recognizable system calls. A cryptominer executes binaries. An attacker exfiltrating data opens network connections. Privilege escalation touches /proc or suid binaries. Falco's detection model sits at this syscall layer.

Architecture¶

flowchart TD
    Kernel[Linux kernel\nsyscalls] --> Driver{Driver}
    Driver --> |kernel module| KernelModule[Falco kernel module]
    Driver --> |eBPF| EBPF[Modern eBPF probe\nCO-RE]
    KernelModule --> Falco[falco process]
    EBPF --> Falco
    Falco --> |evaluate| Rules[Rules engine]
    Rules --> |alert| Output[Alert outputs]
    Output --> Stdout[stdout / syslog]
    Output --> Webhook[HTTP webhook]
    Output --> Sidekick[Falcosidekick]
    Sidekick --> Slack[Slack]
    Sidekick --> SIEM[Splunk / Elastic]
    Sidekick --> PD[PagerDuty]
    Sidekick --> Lambda[AWS Lambda]

Driver options (as of the Falco 0.44 release line there are only two - see note below):

• Kernel module (falco-driver, kmod): compiled at install time against the running kernel's headers via DKMS-style build, then loaded with insmod. Highest event fidelity and broadest historical compatibility, but it's a piece of code running in kernel space with full access - a bug in the driver is a bug in the kernel, and it must be rebuilt for every kernel version the node runs. This is the highest-risk option from a security-posture standpoint (largest attack surface, hardest to audit) even though it's also the most battle-tested.
• Modern eBPF (CO-RE - Compile Once, Run Everywhere): uses BTF (BPF Type Format) metadata embedded in the kernel to relocate a single precompiled BPF object across kernel versions, so there's no per-kernel build or download step at all. Requires kernel 5.8+ with BTF enabled (true by default on virtually all current managed-Kubernetes node images). This has been the default driver since Falco 0.38.0 and the one to reach for unless you have a specific reason not to: no DKMS build step, no driver-loader network dependency, smallest footprint, and the strong safety guarantees of the eBPF verifier (which rejects programs that could crash or destabilize the kernel - a meaningfully smaller trust boundary than a kernel module).

The legacy/"classic" eBPF probe has been removed. It shipped for years as an alternative to modern eBPF (useful on pre-5.8 kernels that lacked BTF), but Falco 0.44.0 dropped it entirely - engine.kind=ebpf and its config block no longer exist. If you're on a kernel too old for modern eBPF's BTF requirement, the kernel module is now the only fallback. (That same 0.44.0 release also dropped gVisor engine support and the gRPC output/server - see Falcosidekick or HTTP output for event forwarding instead of gRPC.)

The driver choice matters operationally, not just philosophically: the kernel module depends on falco-driver-loader fetching or building an artifact matched to the exact running kernel, which breaks silently on node OS upgrades if the matching driver isn't available yet - a common cause of Falco pods stuck in Init or crash-looping right after a node image bump. Modern eBPF sidesteps this class of failure entirely, which is why it's the default choice for managed Kubernetes (EKS, GKE, AKS) where you don't control node kernel cadence. One more upgrade wrinkle worth knowing: the driver ABI itself is versioned and occasionally bumped between Falco releases (0.44.0 bumped it, making 0.43.0-era driver builds incompatible with 0.44.0 userspace) - when you upgrade Falco across a driver-ABI boundary, redeploy matching drivers rather than assuming the old ones still attach.

# Inspect / select the driver explicitly
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true

# falco-driver-loader forces a specific driver on an existing node (kernel module only --
# modern eBPF ships as a precompiled CO-RE object bundled in the Falco binary and needs no loader step)
falco-driver-loader kmod

Rules language¶

A Falco rule has five parts: condition, output, priority, name, and optional list/macro references.

- rule: Shell in Container
  desc: A shell was spawned in a container that should not have one.
  condition: >
    spawned_process
    and container
    and not container.image.repository in (allowed_shell_containers)
    and proc.name in (shell_binaries)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name image=%container.image.repository
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, attack]

Macros¶

Macros are reusable condition fragments:

- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- macro: container
  condition: container.id != host

- macro: open_write
  condition: evt.type in (open, openat, openat2) and evt.is_open_write = true and fd.typechar = 'f'

Note that shell_binaries is deliberately not a macro here - upstream Falco ships it as a list (see below). A macro is a reusable condition; a list is a reusable set of values. Defining the same name as both in one ruleset is a redefinition conflict.

Lists¶

Lists are reusable sets of values:

- list: allowed_shell_containers
  items:
    - "my-debug-toolbox"
    - "ci-runner"

- list: shell_binaries
  items: [bash, sh, zsh, dash, fish, csh, ksh, tcsh]

Rule anatomy and field syntax¶

Every rule resolves down to a condition evaluated against a stream of kernel events. The condition language is a boolean expression over fields, each rooted in a namespace that tells you where the data comes from:

evt.type = execve and evt.dir = < (the spawned_process macro above) is a good one to internalize: syscalls fire twice, once on enter (>) and once on exit (<), and most rules care about the outcome - the exit event, where return values and resolved arguments are populated - which is why evt.dir = < is what the spawned_process macro above and the shipped ruleset both use.

Operators available in conditions: =/!=, in (...)/not in (...) for set membership against a list or macro, contains/startswith/endswith/glob for string matching on fields like fd.name or proc.cmdline, and and/or/not for boolean composition. Parentheses group precedence exactly as you'd expect. The output: string interpolates any field with %field.name - Falco resolves these at alert time from the same event context the condition matched against.

Falco 0.44.0 added three comparison modifiers - oneof, anyof, allof - for cleaner list-based conditions, e.g. proc.name = anyof (sshd, sudo, su) instead of a chain of proc.name = sshd or proc.name = sudo or proc.name = su. They also work inside single-field exception blocks, not just top-level conditions.

Default rule categories¶

Falco ships with rules covering:

Writing custom rules¶

Detect unexpected outbound connections¶

- rule: Unexpected Outbound Connection
  desc: Container established an outbound connection to an unexpected destination.
  condition: >
    outbound
    and container
    and not fd.sip in (allowed_outbound_ips)
    and not fd.sport in (allowed_outbound_ports)
    and container.image.repository != "legitimate-egress-service"
  output: >
    Unexpected outbound connection
    (user=%user.name container=%container.name image=%container.image.repository
    connection=%fd.name)
  priority: WARNING
  tags: [network, container]

Detect reading of Kubernetes secrets from disk¶

- rule: Container Reading K8s Secret File
  desc: A process in a container read a mounted Kubernetes secret file.
  condition: >
    open_read
    and container
    and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/
    and not proc.name in (allowed_sa_readers)
  output: >
    Process reading service account token
    (user=%user.name container=%container.name proc=%proc.name file=%fd.name)
  priority: WARNING
  tags: [k8s, credentials]

Detect privilege escalation via SUID binary¶

- rule: SUID/SGID Binary Execution
  desc: An SUID or SGID binary was executed in a container.
  condition: >
    spawned_process
    and container
    and proc.is_suid_exe = true
  output: >
    SUID binary executed in container
    (user=%user.name proc=%proc.name exe=%proc.exe container=%container.name)
  priority: CRITICAL
  tags: [container, privilege-escalation]

Rule tuning and noise reduction¶

Out of the box, Falco is noisy. Production deployment requires tuning. The workflow:

1. Start with alerts going to stdout or a log index only. Do not route to paging or chat yet.
2. Let it run for 24-48 hours. Collect all rule hits.
3. For each noisy rule, either tighten the condition or add exceptions via lists.
4. Promote to alerting only rules with a clear signal-to-noise ratio.

# Suppress a known-noisy rule for a specific image
- rule: Read sensitive file untrusted
  append: true       # extends the existing rule condition
  condition: >
    and not container.image.repository = "my-legacy-app"

Never edit the shipped rules file¶

Falco loads rules files in the order given by -r flags (or the Helm customRules/falco.rules_file list), and later files can append: true onto rules defined earlier by name. The shipped falco_rules.yaml (installed by the package or baked into the image) is exactly that - a base file. It gets overwritten on every Falco upgrade, and package managers may also flag or revert local modifications to it as a matter of course. Any exception, override, or custom rule you hand-edit into it directly disappears the next time you bump the Falco version, silently, usually discovered days later when an alert that was supposed to be suppressed pages someone at 3am.

The correct pattern is a second file loaded after the base one - conventionally falco_rules.local.yaml - that only contains your append: true exceptions, list additions, and net-new custom rules:

# falco.yaml (main config)
rules_file:
  - /etc/falco/falco_rules.yaml        # shipped base rules, untouched
  - /etc/falco/falco_rules.local.yaml  # your overrides, loaded second
  - /etc/falco/rules.d                 # optional: a directory of team-owned rule files

# falco_rules.local.yaml
- list: allowed_shell_containers
  items:
    - "my-debug-toolbox"
    - "ci-runner"

- rule: Terminal shell in container
  append: true
  condition: and not container.image.repository in (allowed_shell_containers)

In a Kubernetes install, this maps to a separate ConfigMap (the Helm chart's customRules value) mounted alongside the base rules ConfigMap that ships with the chart - so a helm upgrade replaces the base rules ConfigMap cleanly while your local overrides, tracked in your own Git repo, are untouched. Diff falco_rules.local.yaml in code review like any other security control; it's the actual definition of what your cluster considers "abnormal."

Falcosidekick¶

Falcosidekick is a fan-out router for Falco alerts. Falco sends JSON events to Falcosidekick's webhook; Falcosidekick routes them to 50+ destinations simultaneously.

# Falcosidekick values for Helm install
config:
  slack:
    webhookurl: "https://hooks.slack.com/services/..."
    minimumpriority: WARNING

  pagerduty:
    routingKey: "your-routing-key"
    minimumpriority: CRITICAL

  elasticsearch:
    hostport: http://elastic:9200
    index: falco
    minimumpriority: DEBUG   # send everything to SIEM

  aws:
    lambda:
      functionname: auto-respond-falco
      minimumpriority: CRITICAL
    sns:
      topicarn: arn:aws:sns:us-east-1:123456789:falco-alerts
      minimumpriority: WARNING

The Lambda integration is particularly powerful: trigger automated response (quarantine a pod, revoke credentials, snapshot a volume for forensics) on critical Falco events.

The routing model is genuinely fan-out, not a chain: a single Falco alert is POSTed once to Falcosidekick, and Falcosidekick evaluates every configured output independently against its own minimumpriority - a CRITICAL alert in the config above lands in Slack, PagerDuty, Elasticsearch, and triggers the Lambda, all from that one event, with each output free to have a different severity floor. This is what makes Falcosidekick more than a webhook forwarder: it decouples "what counts as alert-worthy for a human vs. worth archiving for later" per destination, without touching Falco's own rule definitions.

Falcosidekick-UI¶

Falcosidekick-UI is a companion deployment (enabled via falcosidekick.webui.enabled=true above) that gives Falcosidekick its own persistent store (Redis by default) and a web console for browsing historical alerts - filterable by rule, priority, namespace, and time range - independent of whatever downstream SIEM or chat tool you've wired up. It's useful as a low-effort first destination: point Falcosidekick at the UI alone while you're still tuning rules (see below), before you route anything to Slack or PagerDuty, so you can eyeball real alert volume without paging anyone.

Falco Talon - automated response¶

Falco Talon is a purpose-built response engine for Falco events. Define what to do when a rule fires:

# rules.yaml for Falco Talon
- action: Kill Pod
  match:
    rules:
      - "Shell in Container"
      - "Execution from /tmp"
  parameters:
    graceful_period_seconds: 5

- action: Label Pod
  match:
    rules:
      - "Unexpected Outbound Connection"
  parameters:
    labels:
      quarantine: "true"
      quarantine-reason: "unexpected-outbound"

Response actions: kill pod, label pod, network policy (isolate), Kubernetes delete, AWS Lambda invocation. Falco Talon is a separate project from core Falco (part of the broader Falcosecurity ecosystem, not bundled into the main Helm chart) - it consumes Falco's alert stream over gRPC or HTTP the same way Falcosidekick does, so the two commonly run side by side: Falcosidekick fans alerts out to humans and archives, Talon takes the subset of rules worth an automated reaction.

Treat automated response as its own risk surface. A Kill Pod action tied to a rule with any false-positive rate becomes a self-inflicted outage generator - match Talon actions only to rules you've already tuned to near-zero noise (see Rule tuning and noise reduction) and prefer reversible actions (label, isolate via NetworkPolicy) over destructive ones (kill, delete) until you trust the signal. Node-level actions like cordoning follow the same logic through Talon's Kubernetes-native action set, but a cordoned node is a blunter instrument than a killed pod - reserve it for host-level rule categories (container escape, host mount abuse) rather than per-pod anomalies.

Kubernetes audit events¶

In addition to syscalls, Falco can ingest Kubernetes audit log events via the k8saudit plugin:

# Detect kubectl exec
- rule: Attach/Exec Pod
  desc: An exec or attach was performed on a pod.
  source: k8saudit
  condition: >
    ka.target.resource = pods
    and ka.verb in (exec, attach)
    and not ka.user.name in (allowed_exec_users)
  output: >
    Exec/Attach to pod
    (user=%ka.user.name pod=%ka.target.name ns=%ka.target.namespace
    container=%ka.req.pod.containers.image)
  priority: WARNING

This requires configuring the Kubernetes API server to send audit events to Falco's webhook - typically via a dynamic AuditSink/webhook backend pointed at the falco-k8saudit-webhook Service, since Falco itself doesn't poll the API server for audit data. Without that wiring, k8saudit-sourced rules load successfully and simply never fire, which is a common source of "why didn't this alert" confusion since nothing in Falco's own logs flags the missing pipeline.

Troubleshooting¶

Driver fails to load. Check the falco pod's init/startup logs first:

kubectl logs -n falco ds/falco -c falco --previous

For the kernel module driver, a failed load usually means falco-driver-loader couldn't find or build a matching artifact for the running kernel - common right after a node image/kernel upgrade, before the corresponding driver build has been published, or on a hardened/minimal node image missing kernel headers entirely. Modern eBPF sidesteps most of this class of failure since it doesn't need a kernel-specific build, which is exactly why it's worth switching to if you see this repeatedly across node pools (and, as of Falco 0.44.0, it's also the only eBPF option left - the legacy eBPF probe was removed).

Confirm the driver is actually attached and producing events, not just that the pod is Running:

kubectl exec -n falco ds/falco -- falco --version
kubectl logs -n falco ds/falco | grep -i "starting internal webserver\|driver loaded\|falco_engine version"

A healthy startup logs the loaded ruleset count and driver type. If the pod is up but you're seeing zero events for known-good test triggers (e.g. kubectl exec into a pod and run cat /etc/shadow), check whether the driver actually attached - some CO-RE failures on unusual kernels (custom-patched, very new mainline before BTF support lands) fail silently to a no-op state rather than crash-looping.

Rule syntax errors surface at startup, not at eval time - Falco refuses to start (or drops the offending file, depending on version/config) and logs the specific line and reason:

Wed Aug 21 2026 rule_loader.cpp:311: Error: Rules content is not valid (...)

Validate a rules file before rolling it out cluster-wide:

falco --validate /etc/falco/falco_rules.local.yaml

Run this in CI against every change to falco_rules.local.yaml - it catches macro/list reference typos (a condition referencing a macro that doesn't exist fails to load) before they hit a running DaemonSet.

Debugging a specific false positive: don't guess from the alert text alone - widen the output: string temporarily to include the full ancestry and context (proc.cmdline, proc.pname, proc.aname[2], container.image.repository, k8s.pod.name) and let the rule fire a few more times in the log-only tier. The parent process chain is usually what distinguishes a legitimate action from a malicious one for process-spawn rules - a shell spawned by kubectl exec/sh in a CI runner image looks identical to a shell spawned by an exploited web app at the single-event level, and it's the ancestry that tells them apart. That's what an exception list (scoped to the CI runner's image or parent process) should encode, rather than disabling the rule outright.

Operational patterns¶

Priority mapping: Falco priorities map to NIST and ATT&CK severity. Use CRITICAL only for immediate containment scenarios (active shell in prod, crypto miner detected). WARNING for investigation-required events. INFORMATIONAL for baseline behavior logging to SIEM.

Metrics: Falco exposes Prometheus metrics at :8765/metrics - falco_events_total by rule and priority. Alert on sudden spikes in a rule's event count.

Rules versioning: manage rules as ConfigMaps in Helm, committed to Git. Never edit rules directly on a running Falco instance.

Kernel upgrades: the kernel module must be rebuilt for every kernel version. In managed Kubernetes (EKS, GKE, AKS), use the modern eBPF driver (CO-RE) to avoid rebuilds on node OS upgrades.

Performance and overhead at scale¶

Falco intercepts a syscall stream, which means its overhead scales with how syscall-heavy your workloads are, not with pod count alone. A handful of idle sidecars costs nothing; a fleet of high-throughput network proxies or build agents making millions of read/write/connect calls a second is a different story.

Levers that actually matter in practice:

• Driver choice: modern eBPF has materially lower per-event overhead than the kernel module in most benchmarks, because CO-RE probes are leaner and avoid some of the copy/context-switch cost of the kernel-module path. If CPU headroom is tight, this is the first thing to check, not the rules.
• Syscall set, not rule count: Falco's cost is dominated by which syscalls it captures at the kernel boundary, not how many rules it evaluates against each captured event - the rules engine itself is cheap per event. Recent Falco versions support restricting the captured syscall set to only what the loaded ruleset actually needs (rather than the historical "capture everything, filter in userspace" default), which is the single biggest lever for reducing overhead on syscall-dense nodes.
• Buffer sizing: syscall_buf_size_preset (or the equivalent eBPF ring buffer size) trades memory for drop resistance. Under sustained high event rates, an undersized buffer causes Falco to drop events rather than fall behind - check falco_events_total{...,source="syscall"} drop-related metrics and the driver's own n_drops counter; silent drops mean silent blind spots, not just noisy logs.
• Rule coverage vs. noise/CPU cost is a real tradeoff, not just a tuning nuisance. Every additional broad rule (e.g., matching all open_read events cluster-wide to catch credential-file access) adds userspace evaluation cost across every one of those events, and broad conditions are exactly the ones most likely to generate noise. Prefer narrow, well-scoped conditions (specific paths, specific process contexts) over "catch everything, filter later" rules - it's cheaper and higher signal.
• Per-node vs. per-pod cost: Falco runs as a DaemonSet, one instance per node, observing every container on that node - its resource requests should scale with node syscall density (how busy the noisiest tenant on that node is), not with total cluster pod count.

Detection vs. prevention: where Falco fits¶

Falco is deliberately a detection layer, not a prevention layer, and that's a design choice worth being explicit about rather than a limitation to work around. Admission controllers like OPA/Gatekeeper and Kyverno enforce policy at deploy time - they can reject a Pod spec that requests privileged: true or a hostPath mount before it's ever scheduled, closing off entire classes of misconfiguration before they exist in the cluster. Falco has no opinion about what gets deployed; it watches what already-running, already-admitted workloads actually do, and catches the things admission-time policy structurally cannot: a legitimately-configured container getting exploited at runtime, a supply-chain-compromised dependency executing a reverse shell, an attacker who's already past the perimeter probing the filesystem.

Neither layer substitutes for the other. Gatekeeper/Kyverno reduce your attack surface; Falco assumes the attack surface wasn't fully closed (it never is) and watches for exploitation anyway. A mature runtime security posture runs both: tight admission policy to keep obviously-bad configurations out, and Falco to catch what got through anyway - misconfigurations you didn't anticipate, zero-days in application code, and compromised third-party images that pass every static check but misbehave once running.

Common mistakes¶

• Editing falco_rules.yaml directly instead of layering overrides in falco_rules.local.yaml - the change vanishes on the next upgrade (see Never edit the shipped rules file).
• Routing straight to PagerDuty on day one. Every cluster is noisy out of the box; skipping the stdout/log-only tuning period burns on-call trust fast and teaches people to ignore Falco pages.
• Wiring Falco Talon kill/delete actions to untuned rules. An automated response engine amplifies false positives into outages instead of just noisy Slack messages - tune first, automate destructive actions last.
• Picking the kernel module driver by default "for compatibility" on a managed Kubernetes cluster where modern eBPF is fully supported - it adds unnecessary kernel-space attack surface and per-kernel-version fragility for no real benefit on 5.8+ nodes.
• Treating Falco as a substitute for admission control, or vice versa. They answer different questions ("is this allowed to run" vs. "is this doing something it shouldn't") and neither closes the gap the other covers.
• Ignoring driver drop counters. A Falco pod that's silently dropping events under load looks healthy on every naive check (pod is Running, alerts are still flowing) while actually missing a growing fraction of the syscall stream.
• Forgetting the k8s audit source requires separate API server configuration. Adding k8saudit rules with no audit webhook pointed at Falco's listener means those rules simply never fire, with no error to indicate why.

Source Links¶

• Falco documentation
• falcosecurity/falco on GitHub and its releases
• Falco release notes / changelog
• Falco rules: conditions, macros, and lists
• Supported syscall and event fields reference
• Falco drivers (kmod, modern eBPF)
• Falco default rules repository
• Falcosidekick
• Falco Talon (response engine)
• CNCF project page: Falco
• Kubernetes: auditing

Related Concepts¶

• Security Primer
• Audit and Logging
• OPA and Gatekeeper
• Kyverno
• Prometheus