Notice the failure.
Know the next step.

Start with the service people use

Pick one important application and describe a working outcome. “The process exists” is weaker than “a client can reach the application and retrieve the expected page.” For a document service, a controlled read of a harmless test document may tell you more than the host answering a ping.

Choose a check that is useful without creating unnecessary side effects. Repeatedly sending real email, opening support tickets, or changing household devices is a poor default for a health check. Use documented health endpoints, read-only requests, or a dedicated synthetic test account with limited permissions.

Write down what a successful check proves. An HTTP success code from a login page does not prove that sign-in or the database works. Content checks or a carefully designed application transaction can give a stronger signal, but add complexity and credentials that must be maintained.

Separate outside-in checks from internal measurements

Service checks

Exercise a route a client depends on: DNS resolution, TLS, HTTP, or another application protocol. These help identify the user-visible symptom.

Host and application metrics

Track available storage, memory pressure, errors, latency, queue depth, or another resource linked to the workload. These help explain the symptom and anticipate some failures.

Logs

Provide event detail around an incident. They are useful for diagnosis, but retaining more logs is not the same as configuring an alert.

An uptime checker can cover basic availability. A metrics system can help with history, trends, and alert rules. A dashboard visualises the data it receives; it does not necessarily collect that data or notify you on its own. Choose tools after deciding which checks and actions you need.

Place the checker where the result means what you think it means. A check inside the same container bypasses parts of the client path. A LAN check does not test your external access route. If the checker runs on the monitored host, losing that host can stop both the service and the alarm.

Monitor the resources that can stop useful work

Storage needs more than a percentage-used number. Track the filesystem the application actually writes to, its growth rate, and how much time you have to react. A large disk at 85% may have months of room; a small log filesystem at 60% could fill that afternoon.

On filesystems with a finite inode pool, many tiny files can exhaust inodes while byte capacity remains. Some applications also have quotas, database-specific limits, or separate temporary storage. Check those boundaries rather than relying only on the host’s main filesystem.

df -h
df -i
free -h
uptime

These commands report current information; they do not configure ongoing monitoring. Availability and output vary by platform. Read the relevant filesystem and memory fields: Linux uses otherwise idle memory for caches, so low “free” memory alone does not prove a problem. Load averages also need interpretation in the context of the workload and available processors.

Begin with a measured normal range and a sustained warning that gives time to respond. Treat suggested thresholds as examples to tune, not universal limits. For critical data, capacity alerts should arrive before writes start failing.

Alert on the age of usable evidence

A scheduled backup job can exist without ever succeeding. Monitor when the most recent successful capture completed, whether it contains the required scope, and whether the destination still has space. An absence of error notifications is not a positive success record.

Relate the allowed age to the application’s recovery needs and the job’s schedule and duration. If a daily backup can take several hours, a threshold of exactly 24 hours may alert during a normal run. Conversely, setting a very generous threshold can hide missed backups. Write down why the threshold is acceptable.

Track restoration separately. A recent backup and a successful repository check do not prove that the application can use the recovered data. Keep the date and result of the most recent restore drill alongside the capture status, and repeat the drill after meaningful changes.

A host that cannot reach its monitoring service may fail to report a failed backup. A heartbeat or missing-success check from outside that host can help detect silence, provided that monitoring path is itself reliable.

Every alert needs an action

Page or notify on conditions that need attention, not every normal fluctuation. A short processor spike during a backup may be expected. A document service that remains unavailable or storage that is likely to fill before the next maintenance window deserves a different response.

Use a suitable delay or repeated failures to avoid transient noise, balanced against how much interruption the service can tolerate. Group related alerts where possible: one host outage should not necessarily generate a separate urgent notification for every dependent service.

Service: household documents
Condition: client-facing check failed repeatedly
Started: include the observed timestamp
Impact: document access may be unavailable
First checks: host, DNS, proxy, application logs
Runbook: reference the recovery instructions

Include the affected service, condition, start time, and the first diagnostic step. Send notifications through a channel you will notice, and record who is responsible. Keep secrets and private application content out of notifications and public status pages.

Distinguish an acknowledgement from a repair. Acknowledging an alert tells others it is being handled; it should not silently erase the underlying condition. A recovery notification can confirm the check has returned to its expected state.

Test the alarm, not just the dashboard

Use a disposable test service or a planned maintenance window to simulate one known failure. Confirm that the condition is detected, the intended notification arrives, the linked instructions are accessible, and a recovery message follows when the check succeeds again.

Also consider the monitoring system itself. If your email service, DNS resolver, checker, and monitored application all live on one host, a single outage may remove every signal. A separate checker or external heartbeat receiver can help, but only if its dependencies and notification route are actually independent enough for the failure you care about.

Bound the monitoring system’s own storage use. Set log and metric retention according to diagnostic needs and available capacity. Too many labels or unbounded values in metric names can create an unexpectedly large dataset. Keep user identifiers, tokens, and personal content out of metrics unless there is a specific, reviewed need.

After an incident, ask whether the alert arrived in time and whether it helped. Remove noisy checks, improve unclear instructions, and add coverage for a missed symptom rather than collecting more data without a purpose.

A small monitoring plan

Checks worth keeping

• Outcome: Each important service has a check tied to a real user action or dependency.
• Location: You know which parts of the network and application path the check covers.
• Capacity: Storage, relevant quotas, and other limiting resources have useful warning thresholds.
• Backups: Missing or stale successful captures are detected; restore-drill results are tracked separately.
• Action: Alerts identify the service, impact, start time, and a sensible first step.
• Delivery: A controlled failure has produced a notification through the intended channel.
• Independence: Monitoring can detect the loss of its monitored host, or that limitation is explicit.
• Retention: Metrics and logs have bounded storage and do not expose unnecessary sensitive data.