Skip to content

Velero

Who this page is for: in plain English, Velero backs up your cluster - both the Kubernetes objects and the data in your volumes - and restores them, on the same cluster or a different one. This page is production depth, aimed at readers already comfortable with Pods, Services, and Deployments. New to Kubernetes? Start with the Beginner track.

Velero is the standard open-source tool for backing up and restoring Kubernetes clusters. It backs up two things a cluster is made of: the Kubernetes API objects that describe your workloads (Deployments, Services, ConfigMaps, CRDs, RBAC - everything kubectl get can see) and the persistent volume data those workloads depend on. Together, that's enough to restore a namespace, a set of applications, or an entire cluster - either back onto the same cluster after a failure, or onto a completely different one.

Velero was originally built at Heptio and has been stewarded by VMware and then Broadcom since VMware's 2019 acquisition of Heptio. Governance is worth checking before you adopt it: the project's stewardship has been in motion, so confirm its current CNCF status on the CNCF landscape and the Velero project site rather than relying on a date written here. One practical signal that the move is still in progress: the Helm chart below is still published from the vmware-tanzu GitHub org.

That second capability is what sets Velero apart from a simple etcd snapshot. An etcd snapshot restores the exact same cluster it came from. Velero backups are portable: because they store Kubernetes objects as versioned API resources and volume data as its own artifact, you can restore a backup taken on an EKS cluster into a GKE cluster, remap namespaces and storage classes along the way, and use the same mechanism for cluster migration as you do for disaster recovery.

Architecture

flowchart TD
    subgraph Cluster
        Server[Velero server\nDeployment]
        NodeAgent[node-agent\nDaemonSet]
        BackupCR[Backup / Restore / Schedule\nCRDs]
        BSL[BackupStorageLocation]
        VSL[VolumeSnapshotLocation]
    end
    subgraph ObjectStorage[Object Storage]
        S3[(S3 / GCS / Azure Blob)]
    end
    subgraph CloudSnapshots[Cloud Snapshot APIs]
        EBS[(EBS / PD / Azure Disk\nsnapshots)]
    end
    subgraph CSI
        CSISnap[CSI Snapshot\ndata mover]
    end

    BackupCR --> Server
    Server --> |object manifests\n+ metadata| BSL
    BSL --> S3
    Server --> |native snapshot| VSL
    VSL --> EBS
    NodeAgent --> |Kopia file system backup| CSISnap
    CSISnap --> |uploads chunks| S3
    Server --> |orchestrates| NodeAgent

Components:

  • Velero server - a Deployment running the controller that watches Backup, Restore, and Schedule custom resources and drives the backup/restore workflow.
  • node-agent - a DaemonSet (formerly called restic daemonset) that runs on every node and performs File System Backup (FSB), reading volume contents directly from the node's filesystem and uploading them via Kopia.
  • BackupStorageLocation (BSL) - a CRD pointing at an object storage bucket (S3, GCS, Azure Blob, or any S3-compatible store like MinIO) where API object manifests and backup metadata are stored.
  • VolumeSnapshotLocation (VSL) - a CRD pointing at a cloud provider's native snapshot API (EBS, GCE persistent disk, Azure Managed Disk snapshots), used for the native-snapshot backup path.
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm install velero vmware-tanzu/velero \
  --namespace velero --create-namespace \
  --set-file credentials.secretContents.cloud=./credentials-velero \
  --set configuration.backupStorageLocation[0].name=default \
  --set configuration.backupStorageLocation[0].provider=aws \
  --set configuration.backupStorageLocation[0].bucket=my-velero-backups \
  --set configuration.backupStorageLocation[0].config.region=us-east-1 \
  --set configuration.volumeSnapshotLocation[0].name=default \
  --set configuration.volumeSnapshotLocation[0].provider=aws \
  --set configuration.volumeSnapshotLocation[0].config.region=us-east-1 \
  --set deployNodeAgent=true

Provider plugins (AWS, GCP, Azure, and community plugins for other stores) are installed as init containers on the Velero server; they implement the object storage and snapshot APIs for each cloud.

BackupStorageLocation and VolumeSnapshotLocation

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: my-velero-backups
    prefix: prod-cluster
  config:
    region: us-east-1
  default: true

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: velero
spec:
  provider: aws
  config:
    region: us-east-1

You can define multiple BSLs - for example, one per environment or one dedicated to long-term retention in a different bucket - and select which one a given Backup uses with --storage-location.

Two ways to back up volume data

Velero has two fundamentally different approaches to getting persistent volume contents into a backup, and picking the right one matters.

Approach How it works Speed Portability Requires
Native snapshots (VolumeSnapshotLocation) Calls the cloud provider's snapshot API directly (EBS, PD, Azure Disk) Fast, near-instant Provider-specific - an AWS snapshot can't restore into GCP A VolumeSnapshotLocation with the matching cloud plugin
File System Backup / Kopia (node-agent) Reads volume contents from the node filesystem via node-agent and uploads deduplicated, compressed chunks to object storage using Kopia Slower, scans file contents Works with any storage class, any provider, any cluster node-agent DaemonSet deployed, backup annotated to opt in

Modern Velero's File System Backup can also use CSI snapshots as a data mover: instead of the node-agent reading a live, mounted volume, Velero takes a point-in-time CSI snapshot first and then uses that snapshot as the source for the Kopia upload. This gives you crash-consistent data movement without holding a long-running read against the live volume, and it works with any CSI driver that supports snapshots - not just the handful of clouds with a native Velero snapshot plugin.

When to use which:

  • Use native snapshots when you're staying on the same cloud provider, want the fastest possible backup and restore, and don't need cross-provider portability.
  • Use File System Backup / Kopia when you need to migrate across providers, your storage doesn't have a native snapshot plugin, you want deduplicated long-term retention, or you're backing up from a CSI driver without native Velero snapshot support.

Opt a pod's volumes into File System Backup with an annotation:

kubectl annotate pod my-app-7c9d8f-abcde \
  backup.velero.io/backup-volumes=data,logs

Or opt out specific volumes when using the default --default-volumes-to-fs-backup behavior:

kubectl annotate pod my-app-7c9d8f-abcde \
  backup.velero.io/backup-volumes-excludes=scratch

Creating backups

# One-off backup of a namespace
velero backup create prod-checkout-backup \
  --include-namespaces checkout \
  --storage-location default \
  --wait

# Backup filtered by label selector
velero backup create critical-apps-backup \
  --selector tier=critical \
  --include-namespaces production

# Exclude a resource type (e.g. skip Events, which are noisy and low-value)
velero backup create full-backup \
  --exclude-resources events,events.events.k8s.io

# Check status
velero backup describe prod-checkout-backup --details
velero backup logs prod-checkout-backup

Scheduled backups

Recurring backups are defined with the Schedule CRD (or velero schedule create), using a standard cron expression:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-prod-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"        # 2am daily, cron syntax
  template:
    includedNamespaces:
      - production
      - checkout
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 720h0m0s               # retain for 30 days
    hooks: {}

velero schedule create daily-prod-backup \
  --schedule="0 2 * * *" \
  --include-namespaces production,checkout \
  --ttl 720h

Each scheduled run creates a Backup object named <schedule-name>-<timestamp>. Retention is enforced per-backup via ttl; Velero garbage-collects expired backups and their storage artifacts automatically.

Backup hooks - application-consistent backups

A snapshot taken mid-write can capture an inconsistent database state. Backup hooks run a command inside a container immediately before and/or after the backup captures that pod's volumes, so you can quiesce an application first.

apiVersion: v1
kind: Pod
metadata:
  name: postgres-0
  annotations:
    pre.hook.backup.velero.io/container: postgres
    pre.hook.backup.velero.io/command: '["/bin/sh", "-c", "psql -c \"SELECT pg_backup_start(''velero'', true)\""]'
    pre.hook.backup.velero.io/timeout: 60s
    post.hook.backup.velero.io/container: postgres
    post.hook.backup.velero.io/command: '["/bin/sh", "-c", "psql -c \"SELECT pg_backup_stop()\""]'

Hooks can also be defined in the Backup/Schedule spec itself rather than pod annotations, which keeps hook definitions out of application manifests:

spec:
  hooks:
    resources:
      - name: postgres-freeze
        includedNamespaces: [production]
        labelSelector:
          matchLabels: {app: postgres}
        pre:
          - exec:
              container: postgres
              command: ["/bin/sh", "-c", "psql -c \"SELECT pg_backup_start('velero', true)\""]
              onError: Fail
              timeout: 60s
        post:
          - exec:
              container: postgres
              command: ["/bin/sh", "-c", "psql -c \"SELECT pg_backup_stop()\""]

These functions were named pg_start_backup() / pg_stop_backup() before PostgreSQL 15, which renamed them to pg_backup_start() / pg_backup_stop() and dropped the old names entirely. Use the older spelling only if you're still on PostgreSQL 14 or earlier - on 15+ the old names error out, and with onError: Fail below that fails the whole backup.

Use onError: Fail for hooks where an inconsistent backup is worse than no backup - Velero marks the backup as failed if the pre-hook doesn't succeed.

Restoring

# Restore a whole backup as-is
velero restore create --from-backup prod-checkout-backup

# Restore into a different namespace (namespace remapping)
velero restore create checkout-restore-to-staging \
  --from-backup prod-checkout-backup \
  --namespace-mappings checkout:checkout-staging

# Restore only a subset of resources
velero restore create --from-backup full-backup \
  --include-resources deployments,services,configmaps \
  --include-namespaces checkout

velero restore describe checkout-restore-to-staging --details
velero restore logs checkout-restore-to-staging

Restore gotchas

  • Namespace remapping doesn't rewrite everything for you - cross-namespace references baked into ConfigMaps or application config (a hardcoded namespace in a connection string, for instance) still point at the old namespace unless your app reads it dynamically.
  • Resource ownership and UIDs change on restore. Restored objects get new UIDs; anything that hardcodes a UID (rare, but some operators do) breaks. ownerReferences are remapped by Velero for objects it restores together, but references to objects outside the backup's scope are not.
  • PVC/PV restore ordering matters. Velero restores PVs before the PVCs and pods that reference them, and for native snapshots it recreates the volume from the cloud snapshot first. If a StorageClass or the underlying snapshot no longer exists in the target cluster, the PV restore fails and dependent pods stay Pending.
  • StorageClass mapping across clusters is handled via a ConfigMap in the Velero namespace when migrating between clusters with different StorageClass names:
apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  gp2: gp3-migrated     # old StorageClass name : new StorageClass name
  • Existing resources are skipped, not overwritten, by default. Restoring into a namespace where the resource already exists leaves the existing resource untouched unless you use --existing-resource-policy update.

Disaster recovery drills

A backup you have never restored is a hypothesis, not a disaster recovery plan. The most common way Velero deployments fail in a real incident is discovering, mid-outage, that:

  • the BackupStorageLocation credentials expired weeks ago and backups have been silently failing
  • a CRD required by a workload wasn't backed up because it lives outside the included namespaces
  • the target cluster's Kubernetes version doesn't support an API version present in the backup
  • restore takes far longer than the RTO budget allows, because nobody timed it

Run scheduled restore drills into an isolated namespace or a scratch cluster, on a cadence tied to your recovery objectives - monthly at minimum for anything covering production. Time the restore, verify application health after restore (not just that objects exist), and treat a failed drill as an incident, not a footnote. velero backup describe and velero backup logs after every scheduled backup catch silent partial failures before you need the backup for real.

Cluster migration

Because Velero backups are portable across clusters, the same workflow doubles as a migration tool: back up the source cluster (or the namespaces you're moving), point Velero on the destination cluster at the same BackupStorageLocation, and restore. This is the standard pattern for moving workloads between regions, between managed Kubernetes providers, or off a cluster being decommissioned.

# On the destination cluster, configure Velero against the same bucket
velero backup-location get

# Then restore the backup taken on the source cluster
velero restore create --from-backup source-cluster-full-backup

Combine this with the StorageClass mapping ConfigMap above when the destination cluster uses different storage classes than the source.

Common mistakes

Mistake Consequence
Never testing a restore Backups silently rot; failures only discovered during a real outage
Relying on native snapshots for cross-provider DR Cloud snapshots are provider-specific and won't restore elsewhere
Skipping backup hooks for stateful apps Restored databases can be corrupted or in an inconsistent state
Not monitoring Schedule health A broken credential or expired token silently stops all future backups
Backing up everything with no TTL Object storage costs grow unbounded; old backups also increase restore-time confusion about which one is good
Forgetting cluster-scoped resources (CRDs, ClusterRoles) Restoring a namespace whose workloads depend on a CRD that wasn't included leaves objects stuck without a controller
  • Storage - the PV/PVC and CSI layer Velero's volume backups operate on
  • Operations and Maintenance - where backup/restore fits in cluster lifecycle work
  • Troubleshooting - debugging a Backup or Restore stuck in progress
  • Operators and CRDs - the controller pattern behind Velero's Backup/Restore resources
  • Helm - installing Velero, and what a restore does to Helm release Secrets