SOP: The first response when a web service goes unreachable

Scope and safety

This SOP is an educational starting point for an authorized operator investigating a web service that appears unreachable. It assumes you have permission to inspect the service, know whom to contact and have a safe recovery channel. It is not a replacement for your organization’s incident policy or the cloud provider’s documented recovery procedures.

The first priority is to understand impact, preserve enough evidence and keep the right people informed. An unavailable app could be caused by DNS, edge configuration, TLS, a failed origin, storage, a credential or an upstream dependency. Restarting everything before identifying the failing layer can destroy useful evidence and make recovery slower.

Use the interactive checklist below to work through a sample investigation. Its progress is temporary and stored only in the current page; it does not send a command or mark a real incident resolved.

1. Establish impact and an incident record

Capture the hostname, exact failing URL, first known failure time and timezone, reported user impact, and whether the problem happens everywhere or only from certain networks. If multiple people are responding, designate the person coordinating and the person communicating status updates.

The Google SRE incident-response workbook emphasizes roles, communication and coordination alongside technical mitigation. For a small team one person may handle several roles, but an incident record should still make decisions and observations visible.

• Record a timestamp in UTC and the affected public URL.
• Check provider status and recent scheduled changes or deploys.
• Identify affected users and services without disclosing personal data.
• Choose an incident channel and a person responsible for updates.

2. Test the public path before opening a shell

Use a workstation or a separate network to confirm the symptom. Inspect DNS answers, HTTP status and TLS behavior. A 502 or 504 from a proxy or CDN is a signal to trace the edge–origin path, not proof that an application container needs restarting. Cloudflare’s 502/504 documentation distinguishes responses originating at Cloudflare from responses returned by an upstream server.

An HTTP HEAD request may not match an actual GET workflow. Use the method that reproduces the issue, and record the result before making changes.

dig +short app.example.comcurl -I --max-time 12 https://app.example.com/# Record the status, redirect chain, date and relevant request ID.# If the symptom differs by network, compare with another authorized client.

3. Isolate the broken boundary

If DNS and TLS appear correct, check the reverse proxy and origin health from an authorized management path. Compare what the edge sees with the service’s own logs and basic host capacity. A healthy homepage may mask a failing database-backed API; check a representative permitted operation if safe.

Avoid outputting passwords, bearer tokens or customer data to a shared incident channel. Redact sensitive information from logs and screenshots. When a third-party provider is involved, gather the error code, timestamps and request identifier required for a useful support request.

uptimefree -hdf -hsudo ss -tulpnsystemctl --failed# Inspect only the services and logs you are authorized to access.

4. Consider mitigation before irreversible change

A known bad release may justify rolling back to an approved compatible artifact, while a capacity incident may call for a reviewed workload change. These are different actions. Preserve the current service version and compatibility constraints, identify the responsible approver, and choose the smallest reversible mitigation that addresses the observed failure.

Do not assume that an old application build can run against a database schema that has already been migrated. When the recovery path is uncertain, escalate rather than attempting an untested fix.

5. Verify recovery as a user would experience it

After an authorized mitigation, test the affected public hostname and a representative permitted workflow. Confirm that error rates return to normal, data is correct, and the change has not broken a related dependency. Include another network if the original symptom varied by location.

Document when the incident was detected, the contributing factors found, what changed, who authorized the change, when functionality returned and what needs follow-up. Mark recovery only after checking observable service behavior rather than relying exclusively on a green deployment job.

• Compare public endpoint behavior with the recorded baseline.
• Check a representative read and authorized write flow where appropriate.
• Confirm monitoring and logs support the recovery assessment.
• Communicate status and create a follow-up for the underlying cause.

6. Close the incident without losing the lesson

Once the urgent impact is addressed, write a small review: what users experienced, which signals were useful, which checks were missing, and how the runbook should change. This is not an exercise in blame. Good documentation helps the next responder understand the system faster and avoid repeating a risky guess.

A runbook should have an owner, scope, revision date and maintenance process. Review it after significant architecture, routing or cloud-provider changes and practice it during safe simulation exercises.