Terraform State Backup and Recovery Runbook: Safe Workflows for Teams

Overview

Terraform state is operational data, not merely a cache. It can contain resource identifiers, dependency relationships, provider metadata, and, depending on the configuration, sensitive values. Losing it can make Terraform unable to recognize resources that already exist; restoring the wrong version can cause inaccurate plans or attempts to recreate infrastructure.

A dependable Terraform state backup and recovery process has four parts:

• Controlled storage: Keep shared state in a remote backend with access controls, encryption, and a locking mechanism where supported.
• Independent backups: Retain recoverable versions outside the normal working copy and test that they can be downloaded and inspected.
• Safe operations: Use one controlled workflow for initialization, planning, applying, state inspection, and recovery.
• Documented decisions: Record workspace names, backend locations, lock identifiers, recovery approvals, and validation results.

Do not edit a state file manually as a first response. First preserve the current evidence, identify the last known-good version, and understand whether the issue is a lock, an access problem, a damaged object, a configuration mismatch, or genuine state loss.

Remote backend comparison

Backend features and Terraform CLI behavior can change. Before a migration or incident, check the current Terraform and backend documentation for locking, version retention, encryption, and recovery commands.

Checklist by scenario

Routine backup and protection

• Use a separate state location for each environment, account, subscription, or project boundary as appropriate.
• Enable object versioning, snapshots, or state history where the backend supports it.
• Set retention rules that match the recovery window your team actually needs. Avoid deleting historical versions before confirming they are no longer required.
• Restrict state access to the identities and CI/CD roles that need it. Treat state as potentially sensitive.
• Enable encryption and document who can decrypt or restore the object.
• Record the backend configuration without committing secrets. Prefer short-lived credentials or workload identities over long-lived keys.
• Run terraform state pull from an approved workspace when you need an export, and store the export in an access-controlled recovery location.
• Test a restore in an isolated directory or non-production workspace. A backup that has never been restored is an assumption, not a verified recovery asset.

When Terraform reports a state lock

1. Stop other applies and identify the workspace, backend path, actor, and lock ID.
2. Check whether a CI job, local terminal, or automation runner is still running.
3. Inspect the job logs and process status before changing anything.
4. If the operation is genuinely abandoned, obtain approval from the owner or incident lead.
5. Use terraform force-unlock only with the exact lock ID and only after confirming that no active operation can still write state.
6. Run a fresh plan after unlocking. Do not immediately apply simply because the lock disappeared.

A lock is a safety control. Removing it while another process is active can produce competing writes or an invalid state update.

When state appears lost or corrupted

• Do not run terraform apply against an unverified or newly initialized state.
• Preserve the current object, error output, lock details, and timestamps.
• Identify the workspace and exact backend key or path.
• List available object versions, snapshots, or state history using the backend's current tooling.
• Choose a candidate version from before the incident, then copy it to a separate recovery location for inspection.
• Restore only after confirming that the configuration, provider lock file, workspace, and credentials match the state being recovered.
• Run terraform init as needed, then use terraform state list and a plan to inspect the result.
• Require human review before any production apply.

When migrating a backend

• Freeze applies and announce the migration window.
• Back up each workspace before changing backend configuration.
• Review the planned backend change and verify destination permissions.
• Use the supported Terraform migration workflow rather than copying files by hand.
• Confirm that every workspace has arrived at the expected destination.
• Keep the original backend read-only or otherwise protected until validation is complete.

What to double-check before and after recovery

Before restoration

• Identity: Are you operating in the intended account, subscription, project, region, and workspace?
• Configuration: Does the checked-out code correspond to the state version you selected?
• Providers: Is .terraform.lock.hcl available, and are provider versions compatible with the configuration?
• Secrets: Can the workflow access required secrets without placing them in shell history, logs, or a new state export?
• Impact: Have you paused scheduled pipelines and informed anyone who could run Terraform concurrently?

After restoration

• Run terraform state list and compare important resources with the cloud provider inventory.
• Use terraform show carefully; redact sensitive output before sharing it.
• Run a plan and classify every proposed create, update, and destroy action.
• Investigate unexpected replacement or deletion actions instead of approving them to make the plan “clean.”
• Validate dependencies, outputs, module addresses, and critical resource identifiers.
• Run a narrowly scoped plan or apply only after review, then take a new backup.
• Document the restored version, commands used, reviewers, remaining drift, and follow-up work.

State recovery does not automatically repair infrastructure drift. If the restored state and real resources disagree, use a controlled reconciliation process; the Terraform drift detection and reconciliation workflow provides a useful companion checklist.

Common mistakes

• Keeping shared state only on a laptop: Local files are easy to overwrite, lose, or use from the wrong workspace.
• Backing up only after an incident: Establish scheduled or event-based retention before the first failure.
• Treating versioning as a full disaster recovery plan: Version history may not protect against deletion, access loss, regional failure, or an operator who can alter every version. Test an independent recovery path.
• Running force-unlock reflexively: A stale-looking lock may belong to a slow or disconnected operation.
• Restoring state and applying immediately: Restoration must be followed by inspection and plan review.
• Mixing workspaces or backend keys: Similar names can conceal different environments. Display and verify the active workspace and backend location.
• Ignoring sensitive data: State exports should use the same access, encryption, retention, and deletion controls as other sensitive operational records.
• Changing providers during recovery: Keep recovery focused. Provider upgrades and module changes should be handled in a separate, reviewed workflow; see the Terraform provider upgrade workflow.

When to revisit

Review this runbook before a backend migration, environment split, seasonal planning cycle, major provider or Terraform upgrade, identity redesign, or change to CI/CD execution. Revisit it after a failed apply, unexplained lock, accidental deletion, access incident, or state restoration.

At least twice a year, run a recovery drill in an isolated environment. Confirm that backups exist, the team can locate the correct workspace, permissions still work, historical versions can be restored, sensitive output is handled safely, and a plan can be produced without touching production. Update the commands and capability notes whenever Terraform or the backend changes.

Action checklist: name an owner, record the backend and retention policy, create an independent backup, test a restore, and schedule the next drill. For broader delivery controls, pair this process with your CI/CD pipeline workflow so that state access and approval gates are explicit rather than accidental.