Deploying and Hardening a Public Site on AWS

Standing up a public WordPress site on EC2 — the architecture, the controls, the constraints that shaped it, how each control was verified, and the things that broke.

Summary

A single-instance deployment on AWS EC2 running a public WordPress site behind a reverse proxy, with no SSH daemon running at all, and backups pushed off-site over Tailscale to a receiving server that has no public listener.

The interesting parts aren't the WordPress install — that build is covered separately. They're the constraints: what happens when you can't use a WAF, when your DNS lives somewhere that rules out the obvious tooling, and when the box is small enough that memory pressure is a real design input.

The other half of the work is account-level. Instance hardening with a permissive AWS account underneath it is a locked door in an open frame.

The DNS constraint

The domain's DNS is hosted at the registrar, alongside Google Workspace MX, SPF, DKIM, and DMARC records. That single fact eliminated most of the standard options.

Cloudflare Tunnel — the usual answer for exposing a self-hosted service, and the pattern already in use elsewhere in this environment. A tunnel requires a CNAME to <id>.cfargotunnel.com, which only resolves if the zone is on Cloudflare. That means a nameserver migration, and moving a zone carrying live mail records is a real risk: a missed SPF or DKIM record means mail starts failing authentication and landing in spam, often without an obvious signal.

Tailscale Funnel — publishes a service to the public internet with TLS and no port forwarding. It fails for a subtler reason: Tailscale issues certificates for *.ts.net names only. A CNAME from a custom domain sends that hostname in SNI, Tailscale has no matching certificate, and the TLS handshake dies before anything reaches the server. Custom domain support is an open feature request.

Port forwarding to a home server — no third party, no monthly cost. Rejected: it publishes a home IP address and opens an inbound hole for the single most-scanned application on the internet, in an environment otherwise built entirely outbound-only.

What's left is renting a public IP. EC2 was chosen over a cheaper VPS because the surrounding services — VPC, security groups, IAM, CloudWatch, Elastic IP — are the things worth having hands-on experience with.

The takeaway: infrastructure decisions are frequently made by constraints nobody lists in the architecture diagram. DNS ownership determined the hosting platform here.

Instance

Arm rather than x86 for cost, and because everything in the stack has arm64 images. The AMI architecture has to be selected before the instance type, or Graviton types aren't offered at all.

Cost runs about 1/month, of which the public IPv4 address is $3.65 — more than a third of the total, and more than a t4g.nano instance would cost. AWS bills for public IPv4 regardless of whether an Elastic IP is attached, so there's no saving in going without one.

Network

Security group

Port 22 existed during the initial build, scoped to a single source address, and was removed once Tailscale SSH was verified. The OpenSSH daemon itself was subsequently stopped and masked. There is no SSH listener on this host and no SSH port reachable from the internet.

Port 80 stays open because ACME HTTP-01 validation needs it, and because redirecting to HTTPS is better than timing out.

Container networking

Three containers, one of which publishes ports:

backend is declared internal: true — it has no route off the host. A compromised plugin cannot exfiltrate directly from the database container or pull down a second stage. WordPress bridges both networks; the database sits on one.

This is cheap to implement and rules out an entire category of post-exploitation movement.

Docker and the host firewall

Docker publishes ports by writing its own iptables rules in the DOCKER chain, which the kernel traverses before the chains ufw manages. This produces a well-known failure mode: ufw reports a port as denied while Docker has it open to the world.

The security group is a stateful firewall enforced in the VPC, outside the instance entirely. Packets to any port other than 80 and 443 never reach the kernel's netfilter stack, so there is nothing for a host firewall to filter. Running one would mean maintaining DOCKER-USER rules to make it actually govern container traffic, duplicating a policy already enforced upstream, at the one layer Docker can subvert — while carrying a real risk that a bad rule severs the only administrative path into the box.

The Docker chain was inspected directly to confirm what it actually permits: exactly two ACCEPT rules, both to Caddy's container address on 80 and 443, with everything else dropped.

That calculus changes if this host ever runs services that shouldn't be reachable from the whole VPC, or moves somewhere without a provider-level firewall. The decision is conditional, and the condition is written down.

Tailscale as the administrative plane

The most consequential architectural decision in this deployment is that administration doesn't happen over the public internet at all.

Replacing exposed SSH entirely

The conventional hardening advice for SSH is a progression: disable password authentication, disable root login, move off port 22, scope the firewall rule to a source address, add fail2ban. Every step reduces exposure. None of them removes it — there is still a daemon on a public address, answering unauthenticated connections from anyone who finds it, and its security depends on the ongoing absence of vulnerabilities in the pre-authentication code path of a service listening to the entire internet.

Tailscale removes the listener instead of hardening it. The host joins a WireGuard mesh as a node; connections arrive over that mesh or not at all. Port 22 was removed from the security group once mesh access was confirmed, and ssh.socket and ssh.service were then stopped and masked with systemctl mask, so no package upgrade or dependency can silently restart them.

Both halves were verified: ss -tlnp on the host shows nothing bound to 22, and an external scan shows probes to it dropped at the VPC edge.

Device authentication rather than key files

Tailscale SSH moves authentication from "possession of a private key file" to "this device is an authenticated member of the tailnet, and tailnet policy permits it to reach that host as that user." The distinction matters in three practical ways.

There is no authorized_keys file to manage. Access is granted and revoked in tailnet policy, centrally, taking effect immediately. Revoking a key file means editing it on every host that holds it.

A stolen laptop is revoked once. Deauthorising the device in the Tailscale admin console cuts its access to every node simultaneously. A stolen SSH key requires knowing every host that trusts it.

Connections are attributable to an identity, not a key. The logs record which user on which device connected, rather than which key fingerprint was presented.

Why this survives a changing home IP

The alternative to mesh access is a security group rule scoped to a source address. That works exactly as long as the source address doesn't change — and a residential connection's address changes without notice, on the ISP's schedule. When it does, the rule silently locks out the operator, and recovery means reaching the AWS console from somewhere, finding the new address, and editing the rule.

Worse is what happens next. The usual reaction to being locked out twice is to widen the rule to a CIDR block, or to 0.0.0.0/0 "temporarily." The control degrades under operational pressure, which is how a great many firewall rules end up permissive.

Tailscale has no such dependency. The node authenticates outbound to the coordination server and is reachable at a stable tailnet address regardless of what the underlying network does. Access survives an IP change, a move to a different network, and tethering to a phone.

The mesh as backup and monitoring transport

The same property is what makes the rest of the architecture possible.

Backups. Borg archives push from this host to a backup server at a different physical location. Conventionally that means the receiving server exposes a public SSH port — precisely the exposure this deployment removed on the sending side. Over Tailscale the receiving server has no public listener at all: not port-forwarded, no public DNS record, unfindable by scanning. An attacker who fully compromises this host learns the backup server exists; an attacker scanning the internet never finds it.

Monitoring. The Wazuh agent reaches its manager over the same mesh, so the SIEM's agent and enrolment ports are never exposed publicly either.

One transport, two controls, no additional attack surface for either.

The one place it complicated things

The backup platform builds its Borg SSH connection string server-side from a global hostname setting, and its per-client host override is ignored by the job builder even though it saves and reads back correctly through the API. The fix was setting the global hostname to the backup server's Tailscale address, which works only because every client is already on the tailnet. See "Five identical failures" below — this cost several hours of misdiagnosis.

TLS

Caddy handles certificate issuance and renewal automatically via Let's Encrypt.

Two operational notes worth recording. The certificate state directory must be backed up — losing it means re-issuing on every rebuild, and Let's Encrypt rate limits are real. And because TLS terminates at the proxy, PHP sees plain HTTP on the back end; without explicitly honouring the forwarded-protocol header, WordPress builds http:// URLs and the admin interface redirect-loops.

Hardening

Controls are layered so that no single misconfiguration removes all of them.

Edge

HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy. Server and X-Powered-By headers stripped. XML-RPC returns 403 before it reaches PHP.

Web server

• No PHP execution under the uploads directory. The standard path from

arbitrary file upload to remote code execution is dropping a .php into uploads and requesting it. This closes it.

• XML-RPC denied a second time at this layer.
• Config files, dotfiles, VCS metadata, and editor droppings return 403.
• Directory indexing off.
• Real client IP recovered from the forwarded header, so logging and per-IP

blocking work at all.

PHP

exec, passthru, shell_exec, system, proc_open, popen, and pcntl_exec are disabled. Most PHP webshells stop functioning. expose_php off.

Application

• File editor disabled. If an admin account is taken, this is the difference

between a defaced site and a persistent web shell.

• User enumeration closed on both vectors — the anonymous REST users endpoint

and the ?author=1 redirect. Usernames are half a credential.

• Generic login failure messages.
• Version disclosure removed from headers, generator tag, and asset URLs.
• Minor and security core updates apply automatically; major versions stay

manual.

Application filters live in a must-use plugin rather than wp-config.php — see "Calling the plugin API too early" below.

Why XML-RPC gets blocked twice

system.multicall allows hundreds of password attempts inside a single HTTP request, which defeats request-count-based rate limiting outright. Pingback is a live SSRF primitive. Nothing modern needs the endpoint. Blocking it at the proxy means the request never reaches PHP; blocking it at the web server means it still fails if the proxy config is ever changed.

What replaces the WAF

The DNS constraint rules out a CDN or managed WAF, so credential-stuffing volume has to be absorbed on the box.

fail2ban watches Caddy's access log rather than Apache's, because the proxy sees real client addresses while the application behind it sees only the proxy's container IP. Five failed login POSTs in ten minutes produces a one-hour ban.

Two details that matter more than the jail itself:

It was tested, not assumed. Six deliberate failed logins produced a ban at five. A fail2ban jail that silently stops matching after a log format change is worse than no jail, because it produces false confidence.

The test banned the wrong address. Because the requests left the host and came back via the public address, Caddy logged the source as the instance itself. Harmless in a test, but it demonstrates exactly the failure mode that matters: a filter that bans the wrong thing during a real incident. The ignore list now covers loopback, the instance's own address, and the Tailscale CGNAT range 100.64.0.0/10, so administrative access can't be locked out by its own defences.

File integrity monitoring and SIEM integration

The controls above are preventive. They assume the attacker is stopped at the edge, at the web server, or in PHP. A hardening design that stops there has no answer to the case where one of them fails.

This host runs a Wazuh agent reporting to an existing Wazuh manager on a separate security server, reached over Tailscale. It becomes an additional monitored host in a deployment that already covers the rest of the environment — which means a public-facing server now feeds the same SIEM as everything else, rather than being the one box nobody watches.

Why the agent runs on the host, not in a container

The webroot is a bind mount. Watching it from the host means a compromised container cannot tamper with the agent watching it, and file events are observed at the filesystem layer rather than through the runtime being monitored.

Scoping, and why scoping is the whole game

FIM that alerts on everything gets ignored, which is functionally identical to no FIM at all. Directories are monitored according to what a change there would actually mean:

Image files under uploads are excluded, along with logs and cache directories. A .php appearing in uploads is not excluded — that is the entire point.

Uploads deserves particular attention because it is where two controls meet: PHP execution there is already blocked at the web server, and FIM is what reports that someone tried anyway. Prevention without detection tells you nothing about whether you're being attacked.

Configuration lives on the manager

The FIM policy ships as a shared configuration attached to an agent group, not as a local file on the host. Group membership grants the policy; a second WordPress host would inherit it by joining, with no risk of someone forgetting to copy a config. Group configs also merge rather than replace, so baseline monitoring of /etc, /bin, /usr/bin and similar remains in place alongside the WordPress-specific rules.

Tested, not assumed

The same standard applied to the fail2ban jail. A .php file was created in the uploads directory, then deleted. Both generated alerts — rule 554 for the addition, 550 for the checksum change, and a deletion alert — confirming all three event types are covered and that real-time monitoring is genuinely live rather than merely configured.

Worth noting: the resulting alerts carry compliance mappings automatically — PCI DSS 11.5, NIST 800-53 SI.7, HIPAA 164.312, GDPR. "File integrity monitoring on the webroot" and "PCI DSS 11.5 control evidence" are the same thing described to two different audiences.

Operations

Automatic updates, at three layers

• OS security packages — unattended-upgrades.
• WordPress core — minor and security releases apply automatically; major

versions stay manual.

• Container images — a scheduled nightly pull and recreate.

The third one is the layer most deployments forget. OS and application updates being automatic while container images sit pinned at whatever was current on build day is the most common form of silent drift in a Docker deployment.

The image update script

A cron job rather than a watchtower-style container, deliberately: no additional daemon holding the Docker socket (which is root-equivalent on the host), no memory cost on a 1 GiB box, and a script that can be read and reasoned about when it misbehaves.

Design decisions worth stating:

It runs at 05:00, after both backup jobs. Local archive at 03:15, off-site Borg push at 04:00. A bad image is therefore never the state that got backed up, and there is always a pre-update archive to fall back to.

It backs up compose.yaml before touching anything. The same habit that made recovery from the wp-config.php incident a one-minute job.

It verifies the site afterwards. A pull-and-recreate that leaves the site returning 500 at 05:00 is worse than a stale image. The script curls the site through loopback with the correct SNI and exits non-zero on anything but a 200 — so cron mails the failure rather than logging it quietly.

It does not auto-roll-back. A script guessing at recovery during a failure is how small problems become outages. It fails loudly and leaves the rollback to a human, with the previous two image generations retained on disk to make that possible.

It keeps the current image plus two previous generations, pruning older ones. Generation-based rather than time-based, because update cadence varies and rollback depth shouldn't.

Images are pinned to major tags — wordpress:6-php8.3-apache, mariadb:11.4, caddy:2-alpine — so a pull collects patch and security releases without ever jumping a major version. That pinning is what makes unattended image updates defensible rather than reckless.

Memory as a design constraint

1 GiB total. Baseline usage with Caddy, PHP, MariaDB, the backup agent, and the Wazuh agent all running leaves roughly 425 MiB available.

Four mitigations:

• A 2 GiB swapfile with swappiness at 10. Swap here is not a performance

feature — it's an OOM guard. The failure being prevented is a PHP spike getting the database process killed.

• MariaDB buffer pool reduced to 128 MB.
• Per-container memory limits.
• FIM scoped so only four directories are watched in real time, with core

WordPress directories on a scheduled scan instead. Real-time monitoring holds inotify watches and costs memory proportional to what it covers.

The Wazuh agent costs roughly 50 MiB resident and pushed some idle pages to swap when it was added. That is a real cost on a box this size, accepted deliberately — and precisely the scenario the swapfile was provisioned for.

Right-sizing on a small instance isn't optimisation, it's availability engineering. The OOM killer does not choose gracefully.

Backups

Two layers, deliberately independent.

Local, nightly. Consistent database dump plus a stack archive, 14-day retention. Covers accidental deletion, a bad plugin, a botched update.

Off-site, daily. A Borg agent pushes to a backup server at a different physical location over Tailscale — no public service, no forwarded port, no published DNS record on the receiving end. The SSH account is restricted to append-only Borg operations: the client can write archives but cannot delete or prune them. Retention is enforced server-side, where an attacker who has compromised the client isn't.

That restriction is the point. The client is the thing most likely to be compromised. Ransomware that encrypts the host and then reaches for its backups finds a credential that can only add data.

The local archive deliberately excludes the live database directory — the dump is the consistent copy, and archiving live InnoDB files produces a torn snapshot that restores to a corrupt database.

AWS account-level hardening

Instance hardening is only half the problem. An attacker who obtains AWS credentials doesn't need to exploit anything on the host — they can snapshot the volume, launch a new instance from it, and read the database at leisure. The account is a control plane sitting above every host-level control.

Identity

Root is not used for daily work. Root cannot be scoped by policy, restricted, or revoked; a compromised root session can close the account and delete everything, and no guardrail exists above it. An IAM administrator can have permissions narrowed, credentials rotated, and access killed by another identity.

An IAM user was created with console access and MFA, receiving AdministratorAccess through a group rather than a directly attached policy. The group indirection costs nothing now and matters later: permissions attached to individuals drift, and revocation becomes removing a membership rather than auditing attachments.

Verified on the root account: MFA present, and no access keys — long-lived root programmatic credentials are the worst artifact an AWS account can contain, because a leaked key grants unrestricted, unrevocable access.

IAM access to billing was activated, so cost data is visible to the administrator identity rather than requiring a root login. A billing alarm nobody can see without root is a billing alarm nobody checks.

At organisational scale the correct answer is IAM Identity Center with federated short-lived credentials rather than IAM users at all. That's heavier than a single-account deployment warrants, and naming the tradeoff is more useful than pretending the small answer is the general one.

Instance metadata

IMDSv2 is enforced, verified by behaviour rather than by reading a setting: a token-less request to the metadata endpoint returns 401, a token-based request returns 200.

This matters because IMDSv1 turns any server-side request forgery in the web application into credential theft. The chain is: SSRF → request the metadata endpoint → retrieve temporary IAM credentials → use them against the AWS API. IMDSv2's session-token requirement breaks it, because the initial PUT needed to obtain a token is not something a naive SSRF primitive can issue.

This deployment breaks that chain twice. No IAM instance profile is attached — the metadata endpoint returns 404 for iam/info. There are no credentials at the end of the chain even if the request succeeded.

That isn't an accident of configuration, it's a consequence of the architecture: backups go to a self-hosted Borg server over Tailscale rather than S3, and certificates come from Let's Encrypt rather than ACM. Nothing on this host needs to call the AWS API, so nothing was granted. Least privilege arrived by way of the design rather than by way of a policy document.

VPC

The deployment uses the default VPC, which AWS pre-builds with a permissive posture.

The default security group's inbound rule was removed. Stock configuration allows all traffic between any members of the group. Nothing currently uses it — the instance has its own group — so this closes no active exposure. It closes a future one: AWS assigns the default group to any instance launched without an explicit choice, and an empty group means such an instance fails closed rather than inheriting lateral connectivity.

The default network ACL was reviewed and deliberately left in place. NACLs are stateless, so restricting inbound to 80/443 requires a matching outbound rule permitting the ephemeral port range for return traffic, and vice versa — producing a rule set that permits nearly everything in practice while appearing restrictive. The security group already enforces the same policy statefully and correctly, verified externally. Adding a NACL layer would duplicate a working control at a layer with worse failure modes, one of which is severing the only administrative path.

NACLs earn their place for coarse subnet-wide blocks that shouldn't depend on per-instance configuration — a hostile CIDR blocked across everything in a subnet, or a tier that must be private regardless of what any instance's group says. One instance in one subnet is not that.

Data protection

• EBS default encryption enabled for the region using the AWS-managed key,

converting a per-launch decision into a default. A customer-managed KMS key would add a controllable key policy and scoped audit trail at /month plus the operational weight of never losing it — correct for compliance or cross-account sharing, unnecessary here.

• Block public access for EBS snapshots enabled, in "block all" mode rather

than "block new." A publicly shared snapshot is a complete copy of a disk. This was closed before any snapshot existed, which is the cheapest time to do it.

• Block public access for AMIs verified already enabled.

Account hygiene

• Alternate contacts set for Billing, Operations, and Security, on a mail

provider and domain separate from the root account email. The root address is Google Workspace on the same domain whose DNS carries this site's records; a DNS or Workspace failure would otherwise take out every AWS notification path at once — including the ones AWS sends when it detects abuse or exposed credentials.

• Resource inventory confirmed account-wide. One instance and one volume

across all 33 regions, one Elastic IP and it is attached. No orphaned volumes, no unattached Elastic IPs, no NAT gateways. Unattached Elastic IPs and orphaned volumes bill silently, and resources in a region nobody looks at are invisible by default.

• Opt-in regions confirmed disabled. A disabled region cannot host resources,

which both shrinks where a compromised credential could operate and reduces the surface that has to be audited.

• Billing alarm at 5/month via AWS Budgets. Budgets evaluates actual and

forecasted spend and doesn't depend on billing metrics published only in us-east-1, which is the usual reason CloudWatch billing alarms silently fail.

What would be added at organisational scale

Every enabled region ships with a default VPC and an internet gateway attached, used or not. At scale the right move is deleting default VPCs in unused regions and applying a Service Control Policy restricting which regions can be used at all — removing the ready-made path to the internet that a compromised credential would otherwise find waiting. Disproportionate for a single account, but it's the answer to "what would you do differently with an organisation behind you."

Verification

Every control above was checked rather than assumed. This section exists because "configured" and "working" are different claims, and a writeup that only lists configuration is asserting the weaker one.

What is actually listening

ss -tlnp on the host, with process attribution:

• 80 and 443 bound by docker-proxy — Caddy's published ports, as declared.
• Nothing binds 3306. MariaDB has no host port and no route off the internal

network.

• Nothing binds 22. The daemon is stopped and masked.
• High ports bound by tailscaled are the tailnet peerapi — mesh-only.
• systemd-resolved on loopback, not reachable off-host.

External confirmation

Host-side checks prove what the host believes. They don't prove what the internet can reach. The public address was scanned from off-network using two independent scanners on different networks:

The distinction between filtered and closed is the interesting part. Closed means packets reached the host and were actively refused with a TCP RST — the firewall rule isn't there. Filtered means they were silently dropped and the scanner cannot determine whether anything is behind them. The security group produces the second, which is the correct posture: it gives an attacker no information at all.

Break-glass access

Removing SSH entirely means a Tailscale failure — a bad upgrade, an expired key, a policy mistake — would leave no way in. Before disabling the daemon, an out-of-band path was established and tested end to end: EC2 Serial Console access enabled at the account level, a password set on the host account (the serial console is a raw TTY and cannot accept SSH keys), and a successful login performed through it.

This is deliberately not EC2 Instance Connect, which still requires port 22 reachable in the security group and is therefore not a fallback for this design.

The order is the point: the fallback was proven working before the primary path was removed, not after.

Audit trail

CloudTrail's Event history — free, enabled by default, 90-day retention — was confirmed to contain the full build and hardening sequence, each event attributed to either root or the IAM administrator. That attribution is the concrete return on creating an IAM user rather than continuing to use root.

Its limit is worth stating: CloudTrail records AWS API calls, not host activity. The serial console session appears as an API event, but nothing typed inside it is captured. Everything happening on the instance is invisible to CloudTrail — which is exactly why the Wazuh agent is not redundant with it. The two cover different planes, and the deployment needs both.

Services evaluated and not enabled

A writeup that lists only what was turned on tells you nothing about judgement. Most of the AWS security catalogue is priced per unit of activity, so the interesting question is never "is this a good service" — they all are — but "does this deployment generate enough signal to justify the spend, and what would have to change for the answer to flip."

Prices are US East list rates as of August 2026, sized against this deployment: one t4g.micro (2 vCPU), one 20 GiB volume, one region, one account.

Amazon GuardDuty

Managed threat detection over CloudTrail events, VPC flow logs, and DNS query logs. Foundational detection runs $4.00 per million CloudTrail management events and roughly .00/GB for flow log analysis — plausibly under /month here, with a 30-day free trial. The cost trap is the optional plans: Runtime Monitoring at roughly $0.008 per vCPU-hour is ~1.68/month for 2 vCPU, more than the instance itself.

It is the highest-value paid security service for most small AWS estates — no agents, no tuning, no rule authorship. The reason it's off is not cost. With no IAM role attached to the instance, the largest class of GuardDuty findings — anomalous credential use — has nothing to fire on. The moment this host needs S3, SES, or any AWS API, that changes and GuardDuty becomes the obvious next purchase.

AWS Config

Records resource configuration over time and evaluates it against rules. $0.003 per configuration item continuously, $0.001 per rule evaluation. Low single digits at this size — which is also the problem. With one operator making deliberate changes already recorded in CloudTrail and a written changelog, Config would answer a question that is already answered. Trigger to revisit: a second operator, or infrastructure-as-code with a pipeline that can drift.

AWS Security Hub

Continuous best-practice checks against standards like the CIS AWS Foundations Benchmark. AWS's own published pricing example is almost exactly this deployment — one account, one region, Ohio, 250 checks — and totals $0.25/month. The catch is that its checks are built on configuration items recorded by AWS Config, a hard dependency billed separately. The $0.25 is the visible price; Config is the real one.

Posture management is a problem acquired at scale. For a single account reviewed by hand, the CIS benchmark would largely restate findings already closed above. Trigger: a second account, or a compliance framework requiring documented benchmark evidence.

VPC Flow Logs

IP traffic metadata per interface. $0.25/GB as a vended log to CloudWatch, or straight to S3 at ~$0.023/GB-month, bypassing the vended ingestion fee and queryable with Athena.

The security group's behaviour was already proven correct, host-side and externally, and there is no analysis pipeline waiting to consume flow data. Logs nobody reads are a cost, not a control. Trigger: enabling GuardDuty, which consumes them, or an investigation where reconstructing outbound connections matters.

Amazon Inspector

Continuous CVE and network-exposure scanning for EC2, roughly $0.00174 per instance-hour — ~.27/month here. Genuine overlap with what is already running: unattended-upgrades for OS packages, WordPress core self-updates, scheduled container image pulls, and a Wazuh agent performing its own vulnerability detection. Trigger: a fleet too large to reason about individually.

AWS Backup

Policy-driven backup orchestration, $0.05/GB-month warm EBS storage. This isn't a cost decision, it's an architectural one. At the volume-snapshot level, AWS Backup produces backups whose deletion is authorised by the same AWS credentials an attacker with account access would already hold. Vault Lock in compliance mode closes that — nobody, including root, can delete before retention expires — and is the correct answer when AWS is the whole environment.

Here it would be a second, weaker copy of a guarantee already held off-platform: append-only Borg on infrastructure outside the blast radius of an AWS account compromise. Trigger: adding RDS or any managed service whose data doesn't live on the instance filesystem.

CloudTrail beyond the free tier

A configured trail is what's needed for retention past 90 days, S3 delivery, or data events. The first copy of management events to a trail is free; S3 storage is cents at this volume; data events are billed per 100,000 and are the line item that surprises people. Trigger: a compliance retention floor, or a second account to centralise trails from.

(AWS closed CloudTrail Lake to new customers on 31 May 2026, directing new users to CloudWatch.)

Summary

None of these are expensive at this scale. The reason to leave them off isn't the money — it's that a control producing signal nobody consumes is worse than no control, because it manufactures the impression of coverage. Every one has a stated condition that would make it worth enabling, and those conditions are specific rather than aspirational.

Two failures worth recording

Calling the plugin API too early

A hardening filter added to wp-config.php took the entire site down with a 500. The plugin API doesn't exist at the point that file is evaluated, so add_filter() is an undefined function and the fatal error happens before anything renders.

The correct location is a must-use plugin, which loads after core and needs no activation. The fix is trivial; the lesson is that "it's a config file, config goes in config files" is an assumption worth checking against load order.

Recovery took under a minute because a backup of the config file existed before editing. That habit is worth more than the knowledge it substituted for — and it is now built into the image update script.

Five identical failures, and reading the clock wrong

A backup job failed five consecutive times, each run lasting 2 minutes 15 seconds. Consistent duration reads like a timeout, so the investigation went after timeouts: SSH keepalives, server-side client-alive intervals, relay behaviour on the mesh, memory pressure and OOM kills.

All of it was wrong. 135 seconds is simply the TCP connect timeout to an unroutable address. The job was being handed a LAN address that the cloud instance could never reach.

The consistency was the actual signal, and it was misread. Networks fail at random times. Code fails at exactly the same one. A duration that repeats to the second is a configured limit, not a flaky link — and the next question should have been "what is it failing to connect to" rather than "what is closing this connection."

What made it hard to see: the backup platform builds the connection string server-side, and its per-client host override field saves correctly and is returned correctly by its own API, but is ignored when the job is generated. The setting looked right everywhere except the one place that mattered. Reading the actual generated command — rather than the configuration that supposedly produced it — was what solved it.

Current state

What this deployment demonstrates

Three things, in order of how much they mattered.

Constraints drive architecture. DNS ownership determined the hosting platform, which determined the absence of a WAF, which determined that filtering had to happen on the instance and that fail2ban had to do work a CDN would normally absorb. None of that appears in an architecture diagram.

Verification is a separate discipline from configuration. Every control here was checked by a method independent of the thing that configured it — external scans rather than reading the security group, behavioural probes rather than reading the IMDS setting, a deliberately triggered file event rather than trusting that FIM was running. Two of the most useful findings in this build came from tests that were expected to be formalities.

Deciding not to do something is engineering work too. Seven AWS services were priced and assessed against this specific deployment and left off, each with a written condition that would change the answer. Knowing the catalogue well enough to justify a decision in either direction is the actual skill.