Terraform Environments Done Right: Workspaces Are a Trap

Terraform workspaces look like the easy answer for environments. Then one day someone runs terraform destroy on the workspace that controls the prod state, and you understand the real cost. Here is the structure that scales.

Never share state files between environments

Workspaces share a backend. Environments share risk. A mistake in environment code can reach production's state. The pattern that holds: one state file per environment, isolated backends.

infra/
├── backend/
│   ├── s3/            # sets up the S3 + DynamoDB lock
│   └── main.tf
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── app/
├── environments/
│   ├── dev/
│   │   ├── backend.tf
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   │   ├── backend.tf
│   │   ├── main.tf
│   │   └── terraform.tfvars
│   └── prod/
│       ├── backend.tf
│       ├── main.tf
│       └── terraform.tfvars

Each environment points at its own S3 key. Production is detached from dev by construction, not by discipline.

The module contract

Modules are where you get value. Three rules:

Small and opinionated. A module with 14 optional variables and 9 count flags is a config generator, not a module.
Versioned, not copied. Publish modules to a registry and pin source = "git::...?ref=v1.2.0". Copy-pasting modules across environments recreates drift by hand.
Tested at plan time. terraform validate in CI plus terraform plan against each env's state catches 90% of accidents before they land.

The drift problem

Terraform is a source of truth only until someone clicks a console button. Catch drift with:

terraform plan -detailed-exitcode
# 0 → no changes, 1 → plan succeeded with changes, 2 → error

Run it nightly, alert on unexpected changes, and treat a modified resource outside your code as a bug to fix — not to ignore.

Promoting through environments

The promotion question is: does prod run exactly what staging ran? Close to it:

Same module versions, same source revision
Environment-specific differences confined to variables and remote state
No locals that secretly branch on env name — it hides real divergence

CI/CD for infra

Apply only from CI, never a laptop (unless the laptop is the person on-call fixing an outage). Plan in PRs as a pull request comment:

terraform plan -out=tfplan
terraform show -json tfplan  # comment the summary on the PR

Summary

Isolated state per environment, versioned modules, drift alerts, and plans gated by CI. That's the whole trick — the rest is avoiding the temptation to make it clever.