ObservabilityGENERALFRAMEWORK-SPECIFICCLOUD-SPECIFIC
Not Leaking Your Internals
Stack traces, SQL fragments, internal hostnames and library versions in an error response are free reconnaissance for an attacker.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
What is safe to put in an error response, and what am I giving away without noticing?
The requirement
A security review flagged that our 500 responses include a stack trace. Engineering says they need it to debug. Both are right, and the current design forces a choice.
The obvious build
Return the exception message and stack in the response body. Developers can debug straight from the client, support can paste it into a ticket, and nobody has to go log-diving.
Why it breaks
The stack trace names your framework and its version, which is a direct index into published vulnerabilities for that version (Dependency Security).
How it breaks in production
- The stack trace names your framework and its version, which is a direct index into published vulnerabilities for that version (Dependency Security).
- A database error message contains the failing SQL fragment, the table name and often the column names — a schema map handed over for free, and a strong hint about what an injection payload should target (SQL Injection).
- Connection errors contain internal hostnames and ports:
connect ECONNREFUSED orders-db-primary.internal:5432. That is your private topology, described by you (SSRF — When the Backend Fetches a URL). - File-not-found errors contain absolute paths, which reveal the deployment layout, the runtime user and sometimes the container image structure (Path Traversal).
- It also breaks debugging in the long run: engineers stop building usable logs because the response is doing the job, so when a caller reports a failure without pasting the body, there is nothing to look at.
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
- Error messages are written by library authors for developers reading a terminal. They assume a trusted audience and interpolate whatever is in scope — queries, paths, hostnames, parameter values.
- A response body crosses the trust boundary. Anything in it is public to the caller and, in practice, to anyone the caller shares it with, including logs the caller keeps and error trackers the caller runs (The Trust Boundary).
- The information an attacker wants at the reconnaissance stage is exactly this: what stack, what version, what database, what internal names, what is behind the edge. Errors are the cheapest source because they require no exploitation at all.
- The debugging need is real, but it is a need for correlation, not for detail-in-the-response. If the caller can quote an identifier that finds the full trace server-side, everyone gets what they need.
- Leaks also occur outside the body:
ServerandX-Powered-Byheaders, verbose 404 pages from the framework, an unhandled crash rendering a debug page, and stack traces echoed into a webhook you send outward.
Read your own 500 as an attacker would
The exercise that convinces people is to take a real error response and annotate each fragment with what it discloses. A single body typically gives away the runtime, the framework and its major version, the ORM, the database engine, a table name, a column name, an internal hostname, a port and a filesystem path.
None of that is an exploit. All of it removes guesswork, and guesswork is most of the cost of attacking an unfamiliar system.
| Fragment in the response | What it discloses | What it enables |
|---|---|---|
at OrderService.create (/srv/app/dist/order.js:88) | Language, build layout, deployment path, source structure | Targeted path traversal; confirms a Node build with source maps possibly served |
Error: connect ECONNREFUSED orders-db-primary.internal:5432 | Internal DNS name, port, Postgres, primary/replica topology | SSRF target selection; lateral movement planning (SSRF — When the Backend Fetches a URL) |
duplicate key value violates unique constraint "users_email_uniq" | Table name, column, index naming convention | Schema reconstruction; injection targeting; user enumeration |
X-Powered-By: Express / Server: gunicorn/20.1.0 | Framework and exact version | Direct lookup of published CVEs for that version (Vulnerability Management by Exposure) |
ENOENT: no such file or directory, open '/app/config/secrets.yaml' | Config layout and file naming | Confirms a target for path traversal or a misconfigured static route |
Invalid token: eyJhbGciOi... echoed back | The submitted credential, now in the caller's logs and yours | Credential capture via any intermediary that stores responses (Secrets in Logs) |
The correlation id is what makes the safe response acceptable
Engineers resist generic errors for a good reason: the alternative on offer is usually "go grep the logs with a timestamp and a rough guess". That is genuinely worse than a stack trace, so the argument is lost before it starts.
The trade only works if the response carries an identifier that resolves, in seconds, to the full server-side detail. Then the caller-facing body can be minimal without costing anyone anything.
One boundary, two audiences
1app.use((err: unknown, req: Request, res: Response, _next: NextFunction) => {2 const e = toAppError(err) // unknown -> internal, always3 const correlationId = req.correlationId // set at the edge, see [[correlation-ids]]4 5 // Server-side: everything. Cause chain, stack, query, host, tenant.6 logger.error({7 correlationId,8 route: req.route?.path,9 tenantId: req.auth?.tenantId,10 err: serializeError(e), // includes cause chain + stack11 }, 'request failed')12 13 // Caller-side: a fixed shape, nothing derived from the exception.14 res.status(STATUS[e.category]).json({15 error: {16 code: e.code, // stable, documented, part of the contract17 message: e.safeMessage, // written by you, never err.message18 correlationId, // the whole point19 },20 })21})e.safeMessage is a distinct field from e.message on purpose. If the two were the same field, forgetting once would leak, and forgetting once is inevitable.
Where the leak comes back
Teams fix this at the application boundary and it reappears somewhere nobody was looking. The pattern is always the same: a component that produces errors and was never part of the review.
Check these by name. Each has leaked internals in a real production system more than once.
Leaks that survive a careful error handler
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Proxy cannot reach the app | 502 body names the upstream address and port | The gateway generates its own error page, upstream of your handler | Template the gateway's error responses; treat it as part of the application surface |
| Request never reaches a route | Framework 404 page shows a route table or version banner | Default not-found handler was never replaced | Register explicit 404 and 405 handlers (Route Precedence) |
| A crash outside the request context | Process-level handler prints a trace that a supervisor surfaces in a health endpoint or status page | Uncaught exception path bypasses the request boundary entirely | Handle process-level errors separately, log and exit; never render them (Graceful Shutdown) |
| Outbound webhook fails to serialize | Your internal error text is delivered to a customer endpoint | The outbound path has no sanitizing boundary at all | Apply the same fixed-shape rule to every outward direction (Outbound Webhooks) |
| Error tracker installed with defaults | Headers and bodies with tokens land in a third-party SaaS | Capture defaults are permissive; scrubbing is opt-in | Configure an allowlist of fields before the SDK goes to production (Secrets in Logs) |
How to build it
Most important first.
- Return a fixed shape: status, stable code, a human message written by you, and a correlation id. Nothing derived from an exception (The Error Model: Structure Over Apology).
- Generate the correlation id at the edge and put it in both the response and every log line for that request, so "quote the id" replaces "paste the stack" (Correlation Ids That Survive Every Hop).
- Make the safe path the default in code.
AppErrorcarries amessageyou wrote and acausethat is never serialized — leaking then requires effort rather than being what happens if you forget. - Never make verbosity a runtime environment flag alone.
DEBUG=truein production is one misconfiguration away, and that misconfiguration has happened to almost everyone (Configuration: Separating Code From Environment). - Strip identifying headers at the edge, and replace framework default error pages with your own for 404, 405 and 500.
- Treat outbound directions too: what you put in a webhook payload, an email to a customer, or a response to an internal service that will forward it.
What can go wrong
Failure modes
- Validation libraries that echo the received value back in the message — helpful for a form field, disclosure when the value was a token or a password typed in the wrong box.
- An error tracker configured to attach full request bodies, sending credentials to a third party. The leak is to your vendor rather than to an attacker, but it is still a leak (Secrets in Logs).
- The mitigation failing: a sanitizer that regex-strips known patterns. New library, new message format, leak returns. Allowlist a shape; do not denylist strings.
- A gateway or sidecar generating its own error pages that you never templated, so a 502 from the proxy leaks the upstream address even though the application is careful.
- Timing and status differences that leak existence even when the body is generic — the message says nothing and the 404-versus-403 split says everything (Object-Level Authorization).
Security
- Assume anything in an error response is read by an attacker. The question is not "is this sensitive" but "what does this let someone learn that they did not already know".
- Version disclosure converts a generic scan into a targeted exploit. Removing it does not make you secure; it removes the cheapest step of the attack (Attack Surface).
- Internal hostnames and ports are the input to SSRF and to lateral movement. A service that names its database host in an error has told an attacker where to point a request-forgery payload.
- Error text derived from user input can carry an injection into whatever renders it — a message reflected into an HTML error page is reflected XSS (Cross-Site Scripting (XSS)).
- Never differentiate error text on authentication paths. "Invalid password" versus "no such account" is user enumeration regardless of how careful the rest of the system is.
Misreads
- "It is only an internal API." Internal APIs get exposed by a misconfigured ingress, an SSRF, or a compromised service. Internal is a deployment fact, not a security property.
- "We strip stack traces, so we are done." Messages leak more than traces do.
relation "billing_accounts" does not existhas no stack and tells you plenty. - "Nobody reads error bodies." Scanners read every error body, systematically, and that is the whole point of running one.
- "Debug mode off in production is enough." Framework defaults, proxy error pages and third-party middleware each have their own verbosity, and none of them read your flag.
Operating it
How you see it in production
- Alert on your own responses: a check that scans outbound 5xx bodies in staging for
atstack markers,SELECT,.internal,/var/, and known framework banners. - Log the full exception with cause chain and the correlation id at error level. The detail should be abundant server-side and absent client-side.
- Count how often a correlation id from a customer report actually finds the request. If it does not, your logging retention or your propagation is broken, and engineers will go back to demanding stack traces.
- Watch for 500s whose bodies vary in length. A generic error is fixed-size; variation means something is being interpolated.
What changes at 10x and 100x
- Nothing about the volume changes the rule, but scale changes the surface: more services means more edges, and each new service arrives with its framework defaults switched on.
- At many services, the leak usually appears at a proxy, sidecar or gateway that no application team owns. Sanitizing has to be a platform default rather than a per-service discipline (The Gateway as Policy Boundary).
- The correlation-id path has to keep working across services, or the pressure to put detail back in the response returns immediately (Tracing From the Backend's Side).
What this costs
- Debugging genuinely gets slower for the first week. Engineers who could read the answer off the response now have to look it up, and that only works if lookup is fast.
- Generic messages frustrate legitimate API consumers. The mitigation is stable error codes and good documentation, which is more work than echoing an exception.
- Stripping detail from internal service-to-service calls costs debuggability for little gain if that network is genuinely trusted — but "internal" is a claim about your network that is worth checking before you rely on it (Defence in Depth).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALApplies to any protocol that returns a message to a caller — HTTP bodies, gRPC status details, GraphQL error arrays, and error payloads on a queue.
- FRAMEWORK-SPECIFICDefaults differ sharply: Django and Flask render a full interactive traceback when DEBUG is on, Express prints stack traces in its default error handler unless NODE_ENV is production, Spring Boot's error attributes are configurable and include the exception message by default. The rule is identical; the switch you must find is not.
- CLOUD-SPECIFICManaged load balancers and API gateways generate their own error bodies for upstream failures that never reach your application. Whether those can be templated, and what they include by default, differs by provider and product.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.