DevOps Engineer interview questions and answers

For me DevOps is mainly a culture, supported by tools. In a traditional setup, developers write code and hand it over to an operations team who deploy and run it, so there is a wall between them and slow, risky releases. DevOps removes that wall: the same team shares ownership of building, shipping and running the service. We automate everything we can, such as builds, tests, security scans, deployments and infrastructure through code, so releases become small, frequent and boring. Fast feedback from monitoring and users tells us quickly if something is wrong. We measure delivery with DORA metrics like deployment frequency, lead time, change failure rate and time to restore. When things fail, we run blameless reviews and fix the system, not the person.

The pipeline starts when someone opens a pull request or pushes a commit. First it checks out code, restores the dependency cache, then runs linting and unit tests so developers get feedback in a few minutes. Next it builds a versioned artifact, usually a container image tagged with the commit SHA, and runs security checks such as dependency and image scanning. After merging to main, the same image is pushed to the registry and deployed to a staging environment, where integration and smoke tests run. Production deployment uses the exact same image, either after a manual approval or automatically with a canary or rolling update. Health checks and metrics decide whether to continue, and if something goes wrong the pipeline can roll back to the previous version quickly.

A virtual machine runs a full guest operating system on top of a hypervisor, so each VM has its own kernel. A container is just an isolated process on the host: Linux namespaces give it its own view of processes, network and filesystem, and cgroups limit its CPU and memory. Because containers share the host kernel, they are much smaller, start in seconds or less, and you can pack many more on one machine. VMs give stronger isolation, which matters for untrusted multi-tenant workloads, or when you need a different kernel or operating system such as Windows on a Linux host. In practice we often use both: containers for applications, running inside VMs or managed nodes in the cloud for an extra security boundary.

Infrastructure as Code means we describe servers, networks, databases and permissions in files, for example Terraform or OpenTofu code, and a tool creates them for us. Because the files live in Git, every change goes through a pull request, gets reviewed, and has a history we can audit or revert. It makes environments repeatable: staging and production can be built from the same modules with different variables, so they do not slowly become different. Before applying, terraform plan shows exactly what will change, which reduces surprises. We can also detect drift when someone changes something by hand. Clicking in the console is fine for learning, but it is slow, error-prone, hard to reproduce after a disaster, and nobody knows why a setting was changed.

The Deployment describes the desired state of the app: which image to run, how many replicas, and how to roll out updates. It creates a ReplicaSet, which keeps the right number of Pods running. Pods come and go and their IPs change, so a Service sits in front of them. It selects Pods by labels and gives them one stable cluster IP and DNS name, load balancing across Pods that pass their readiness probe. To reach the app from outside, an Ingress (or the newer Gateway API HTTPRoute) defines rules such as host shop.example.com and path /api go to this Service. An Ingress or Gateway controller, like NGINX, Envoy or a cloud load balancer, reads these rules, terminates TLS and forwards traffic to the Service.

Trunk-based development means everyone works on small, short-lived branches and merges into main at least once a day, so integration happens continuously and merge conflicts stay small. Unfinished work is merged behind feature flags. GitFlow adds long-lived develop and release branches plus hotfix branches, which gives structure for products that ship numbered releases on a schedule, but it delays integration and makes merges bigger. For a team deploying several times a day I would choose trunk-based development. Main is protected: changes come through pull requests with review and required CI checks, and every merge to main produces a deployable artifact. Feature flags separate deploying code from releasing a feature, so we can turn things off without a rollback. If we support old versions for customers, I would add short-lived release branches cut from main only for those versions.

The Docker CLI sends the request to the daemon, which checks whether nginx:1.27 is in the local cache. If not, it resolves the tag to a manifest in the registry and pulls each layer, skipping any it already has. The daemon, through containerd and runc, then creates a container: a thin writable layer on top of the read-only image layers, Linux namespaces for process, network, mount and hostname isolation, and cgroups for CPU and memory limits. It sets up a virtual network interface on the default bridge and adds NAT rules so traffic to host port 8080 reaches container port 80. Finally it starts the image entrypoint, here the nginx master process, as PID 1 inside the container. The container keeps running only while that process runs; when it exits, the container stops, and its writable layer is lost when removed.

Monitoring is watching for problems you already know about: you define dashboards and alerts for things like error rate or disk space. Observability is a property of the system: how well you can understand its internal state from the data it emits, including problems you never predicted. In practice you need both. For a web service I would collect metrics for the four golden signals: latency, traffic, errors and saturation, mostly with Prometheus. I would ship structured logs with request IDs to a central store, and add distributed traces with OpenTelemetry so I can follow one request across services. The key is correlation: the same service, version and trace ID labels on all three signals, so I can jump from an alert on a metric to example traces and then to the logs for exactly those requests.

A load balancer sends periodic requests, for example HTTP GET /healthz every 10 seconds, to each target. After a set number of failures it stops sending user traffic there, and after a number of successes it adds the target back. These thresholds prevent flapping on a single slow response. A good endpoint is fast and cheap, needs no authentication, and answers whether this instance can serve requests right now, such as having finished startup and loaded config. I avoid making it check every downstream dependency: if a shared database blips and every instance fails its check, the load balancer removes all of them and a partial outage becomes a full one. In Kubernetes I separate readiness, which controls traffic, from liveness, which restarts the container, and keep liveness very simple so it only catches a truly stuck process.

Terraform is a declarative provisioning tool. I describe cloud resources such as VPCs, databases and clusters, and it keeps a state file so it can plan the exact create, change and destroy steps before applying them. It is strongest at the lifecycle of infrastructure objects through cloud APIs. Ansible is mainly a configuration management tool: it connects to existing machines over SSH without an agent and runs idempotent tasks to install packages, write config files and restart services. It has no state file; it checks the current state of each host when it runs. So I would use Terraform to create the servers, network and managed services, and Ansible to configure long-lived VMs or network devices, or to bake machine images with Packer. In container-based platforms the Ansible part often shrinks, because configuration moves into images and Kubernetes manifests.

My first rule is that secrets never go into Git, Docker images or logs. I keep them in a central store such as HashiCorp Vault or AWS Secrets Manager. In CI, I avoid long-lived keys by using OIDC federation, so each pipeline job gets short-lived cloud credentials for a role scoped to that repo and branch. In Kubernetes, I use the External Secrets Operator or the Secrets Store CSI driver to pull values from the secret manager, and workload identity so Pods authenticate without static keys. Kubernetes Secrets are only base64 encoded, so I enable encryption at rest for etcd and restrict access with RBAC. Every secret has an owner, the narrowest possible permissions, automatic rotation where possible, and access is audit logged. I also run secret scanning on repos to catch mistakes early.

CrashLoopBackOff means the container starts, exits, and Kubernetes keeps restarting it with growing delays. If users are affected, I first roll back with kubectl rollout undo, then investigate. I run kubectl describe pod to see events, the last state, the exit code and the reason. Exit code 137 with OOMKilled points to memory limits; exit code 1 usually means an application error. Then kubectl logs with --previous shows the output of the crashed container, which often reveals a missing environment variable, a bad config file, or a database connection failure. I also check if a liveness probe is too aggressive and killing a slow-starting app. Finally I compare what changed in this release, such as the image, ConfigMaps or Secrets, and fix it through the pipeline.

A rolling update replaces instances a few at a time. It needs no extra environment and is the Kubernetes default, but old and new versions run together and rollback means rolling again. Blue-green runs a full copy of the new version next to the old one and switches all traffic at once, so rollback is instant, but it costs double capacity during the release and all users move together. Canary sends a small percentage of real traffic to the new version, compares error rate and latency with the stable version, and increases traffic step by step, often automated with Argo Rollouts or Flagger. I choose rolling for low-risk services, blue-green when I need fast full rollback, and canary for high-traffic, high-risk services with good metrics. In all cases the database and APIs must stay backward compatible.

I would start by measuring, not guessing. I look at timings per job and step over the last few weeks to find the biggest costs. Usually the quick wins are caching: dependency caches keyed by the lockfile, and Docker layer caching with BuildKit and a registry cache, plus a better Dockerfile order. Next I run independent jobs such as lint, unit tests and security scans in parallel, and shard large test suites across several runners. In a monorepo, tools like Nx, Turborepo or Bazel can build and test only the affected projects. I also check runner size, since a bigger machine can be cheaper than developer waiting time. Finally I find and fix or quarantine flaky tests, because retries waste a lot of time. I track pipeline duration as a metric so it does not slowly grow again.

I put shared building blocks such as the network, the Kubernetes cluster and the database into reusable modules, versioned with Git tags or a private registry. Each environment then has a small root configuration that calls those modules with its own variables, for example instance sizes or replica counts. Every environment has its own remote state with locking, and I often split state by component, like network and apps, so one bad apply cannot break everything; this limits the blast radius. Provider and module versions are pinned. In CI, every pull request runs fmt, validate, a linter and terraform plan, and posts the plan for review. Apply to production happens from the pipeline after approval, not from laptops. Tools like Terragrunt or Terraform Stacks can reduce repetition when there are many environments.

I would start with data: export the last month of pages and tag each one as actionable, duplicate or noise. Alerts that never needed human action get deleted or turned into tickets or dashboard panels. For paging I would move to symptom-based alerts on user impact, mainly multi-window SLO burn-rate alerts for availability and latency, instead of CPU or single-host checks. Every remaining page must have a clear owner, a severity, and a link to a runbook and dashboard. I would group related alerts in Alertmanager so one incident causes one page, and add inhibition so downstream alerts are muted when an upstream one fires. On the process side, a weekly on-call review looks at page counts, sleep-hour pages and recurring causes, and turns them into backlog items with real priority. We also make sure rotations are fair and that time is reserved after a heavy shift.

Helm is a package manager: a chart has Go templates and a values file, it is versioned and published to a repository, and Helm tracks release history for upgrades and rollbacks. It is the right fit when you distribute an application to many consumers who need many knobs, like most third-party software. Kustomize works on plain YAML: you have a base and per-environment overlays that patch it, with no templating language. That makes it very readable for your own services where environments differ only in a few fields such as replicas, image tags or resources. Heavy Helm templating can become hard to read, while Kustomize becomes awkward when you need real logic. They combine well: Kustomize can inflate a Helm chart and then apply patches, and Argo CD supports both. In practice I often install vendor tools with Helm and manage in-house services with Kustomize overlays in a GitOps repo.

First I confirm and size the problem: which endpoints are slow, from what percentile, since when, and for which users, using latency histograms and the SLO dashboard. Next I look at what changed around that time: deploys, config or feature flag changes, node upgrades, autoscaling events. If a deploy lines up and impact is serious, I roll back first and investigate after. To locate the slowness I open traces for slow requests and see which span grows, for example a database query, an external API or time spent in the service itself. Then I check the Pods: CPU throttling, memory near limits, restarts, garbage collection, and whether they are spread across healthy nodes. I also check dependencies such as database load and connection pools, DNS latency and network errors. Once service is stable, I write up the cause and add an alert or dashboard that would have caught it sooner.

CI builds the image once, tags it with the commit SHA, signs it and pushes it. The same digest moves through every environment, so what we tested is what we ship. The GitOps repo has a base and an overlay or values file per environment, and Argo CD watches each environment path. Dev is updated automatically by CI or Argo CD Image Updater. Promotion to staging and production is a Git change: a pull request that bumps the image digest in that environment overlay. The staging promotion can be automatic once dev smoke tests pass; production needs the staging checks green plus an approval from the service owner. Argo CD then syncs, ideally through Argo Rollouts for a canary. Rollback is simply reverting the commit, and Git history shows who promoted what and when. For many services, a tool like Kargo can manage these promotion stages for us.

Migrations live in the repo as versioned files, applied by a tool such as Flyway, Liquibase or the framework migrator, and every environment records which ones ran. The main rule is that during a rolling deploy both the old and new app versions run at once, so every schema change must work with both. I use expand and contract: first add new columns or tables in a backward-compatible way, deploy code that writes to both or reads the new one, backfill in batches, and only drop the old structure in a later release. In the pipeline the migration runs as its own step, for example a Kubernetes Job or Argo CD PreSync hook, before the new pods start, and the deploy stops if it fails. I test migrations against a copy of production-sized data to catch long locks, and use online index creation. Rollback means rolling back the app, not running destructive down migrations.

I would treat the platform as a product, with the product teams as customers. First I would interview teams to understand their pain. Then I would build golden paths: a service template, for example through Backstage, that creates a repo with a Dockerfile, a CI pipeline, Helm or Kustomize manifests, dashboards and alerts already wired. Deployments go through GitOps with Argo CD, so teams self-serve by merging pull requests. For multi-tenancy, each team gets namespaces with RBAC, resource quotas, limit ranges and default-deny network policies. Instead of manual reviews, guardrails are policy as code with Kyverno or OPA Gatekeeper, for example requiring resource requests, non-root containers and signed images. Shared concerns like ingress, certificates, secrets and observability are provided centrally. I would measure adoption, lead time and developer satisfaction, and keep good documentation.

I think of each step as something an attacker could tamper with. For source, I use branch protection, required reviews, CODEOWNERS and secret scanning. Builds run on ephemeral, isolated runners with least-privilege tokens, and third-party actions and base images are pinned by commit SHA or digest. Dependencies are scanned, and tools like Renovate keep them updated in small steps. The pipeline generates an SBOM and build provenance following SLSA, and signs the image with Sigstore cosign, ideally keyless with OIDC identity. The image is pushed by digest to a registry with tag immutability. At deploy time, an admission controller such as Kyverno or Sigstore policy-controller only allows images that are signed by our pipeline, have provenance, and pass vulnerability policy. In production, runtime security and regular rescanning catch new CVEs in old images.

The hardest part is data. I would work with developers to decide which data needs strong consistency and which can be eventually consistent, and pick a database that supports multi-region writes or route writes for each record to a home region to avoid conflicts. For traffic, a global load balancer or latency-based DNS with health checks sends users to the nearest healthy region. Both regions are built from the same IaC modules so they stay identical. Releases go region by region, with a bake time and automatic rollback, so one bad change cannot take down both. Each region must have enough capacity, or fast autoscaling, to take all traffic if the other fails. Most importantly, we practise failover with regular game days, because untested failover usually fails when you need it. Cost and cross-region data transfer also need tracking.

First I would look at the data: which changes failed, in which services, and why, for example config errors, missing tests, schema changes or dependency issues. Usually large batches are a big cause, so I would push for smaller pull requests, trunk-based development and frequent deploys, which are easier to test and roll back. I would close test gaps found in the analysis, adding contract tests between services and checks for config and migrations in the pipeline. To limit impact, I would use canary releases with automated analysis and feature flags to separate deploy from release. For recovery time, rollback must be one click or automatic, with dashboards and alerts that point to the change that caused trouble. Blameless postmortems turn each failure into a pipeline or platform improvement. Speed and stability improve together this way.

I start with the release notes and scan all manifests and Helm charts for deprecated or removed APIs using tools like Pluto or kubent, then fix them before upgrading. I check that add-ons such as the CNI, ingress controller, CSI drivers, service mesh and operators support the new version. Kubernetes supports upgrading one minor version at a time and has version skew rules, so the control plane goes first, then node pools. I upgrade dev and staging first and run tests. For nodes, I create a new node pool on the new version, cordon and drain old nodes gradually, and rely on PodDisruptionBudgets, multiple replicas and good readiness probes so services stay available. I watch error rates and cluster health during the process. For very critical systems, I prefer a blue-green cluster approach, moving traffic to a new cluster so rollback is simple.

I start with visibility, because you cannot cut what you cannot attribute. Using a tool like OpenCost or Kubecost plus cloud tags, I break spend down by team, namespace and service, and share it. The biggest lever is usually over-requested Pods: compare requests with real usage, apply VPA recommendations, and since the scheduler packs by requests, this frees whole nodes. Then I improve node efficiency with Karpenter or cluster autoscaler, consolidating underused nodes and picking cheaper instance types, including ARM where images support it. Stateless, replicated workloads move to diversified spot capacity, and the steady baseline is covered by savings plans or reserved capacity. Next is waste: dev environments that run overnight, orphaned volumes and snapshots, NAT and cross-zone traffic, and log ingestion. Throughout, I watch SLOs and error budgets so savings never come from reliability, and I make cost a monthly review metric with owners.

First I declare the incident and its severity, open a dedicated channel and video bridge, and page the right owners. I assign roles: I stay incident commander and coordinate rather than debug; an operations lead drives the technical work; and a communications lead posts updates to the status page and internal stakeholders on a fixed schedule, such as every 30 minutes. Our first goal is to stop customer impact, not to find the root cause, so we look at recent changes and use the quickest safe mitigation: roll back, fail over to another region, disable a feature flag or shed load. I keep the team focused, avoid many people changing production at once, and make sure every action is announced and logged in a timeline. After recovery we confirm metrics are healthy. Within a few days we hold a blameless postmortem covering impact, timeline, contributing factors and detection gaps, with action items that have owners and due dates.

The core idea is to never build the whole repo for a normal change. I would adopt a build tool that understands the dependency graph, such as Bazel, Nx, Turborepo or Pants, so CI computes the affected targets from the diff and only builds and tests those and their dependants. A shared remote cache means unchanged targets are fetched instead of rebuilt, across CI and developer laptops. Work runs in parallel on autoscaled ephemeral runners. For merging, branch protection requires one aggregate status check, so path-skipped jobs never block a PR, and a merge queue tests changes against the latest main to keep it green. Each service still deploys independently: merging to main builds its image, and GitOps promotes it per environment, with CODEOWNERS routing reviews. I would track p50 and p95 pipeline time, cache hit rate and flaky test rate on a dashboard, quarantine flaky tests quickly, and run a full build nightly as a safety net.

I start by mapping systems to their real RTO and RPO, because not everything needs the same tier. For a 1 hour RTO and 5 minute RPO on region loss, backup and restore alone is usually too slow, so I would choose warm standby. Databases replicate continuously to the second region, for example cross-region read replicas or a global database, keeping lag well under five minutes, and object storage uses cross-region replication. The standby region runs a small Kubernetes cluster built from the same Terraform code, with Argo CD syncing the same manifests, so scaling it up is fast. Images and secrets are replicated too. Failover is driven by a runbook: promote the replica, scale out, then switch traffic through DNS or a global load balancer. Separate backups, immutable and in another account, protect against corruption that replication would copy. Finally, we run DR drills at least twice a year and measure actual recovery times against the targets.

Each team gets its own namespaces, created from a template that sets RBAC, quotas and policies, so isolation is consistent. Access comes from SSO groups bound to namespace-scoped roles; nobody but the platform team has cluster-admin, and CI deploys with narrow service accounts. Every namespace starts with default-deny NetworkPolicies, and teams add explicit allow rules, enforced by a CNI such as Cilium or Calico. ResourceQuotas and LimitRanges stop one team from starving others. Pod Security Admission at the restricted level blocks privileged containers, host mounts and root users, and an admission controller like Kyverno enforces extra rules such as signed images from approved registries and required labels. Workloads use workload identity for cloud access, and secrets come from a secret manager via the External Secrets Operator. I enable API audit logs and runtime detection with a tool like Falco. For tenants that need strong isolation, such as untrusted code or strict compliance, I would use separate clusters or virtual clusters instead.