What monitoring and alerting would you set up for a production environment?

Monitor at layers, because a symptom at one level usually has a cause at another:

• Infrastructure — CPU, memory, disk space and I/O, network throughput and errors, and hardware health.
• Service — is the process running, is the port listening, is the certificate about to expire, is the backup job succeeding.
• Application — request rate, error rate, and response time percentiles.
• End-user experience — synthetic checks that perform a real transaction from outside your network. This catches what internal monitoring misses entirely.
• Logs — centralised, so evidence survives the failure of the host that produced it.

Alerting is where most teams go wrong. The principles:

• Alert on symptoms, not causes. Alert that the site is returning errors, not that CPU is at 80%. High CPU with everything working is not an incident.
• Every alert must be actionable and have a documented response. An alert nobody acts on trains people to ignore all alerts, which is worse than no monitoring.
• Use severity tiers. Page for something that needs waking someone; email or ticket for what can wait until morning.
• Use trends for capacity. Alerting when a disk will be full in five days beats alerting when it is full.

Note: Naming alert fatigue as the main failure mode, and reviewing alerts periodically to delete noisy ones, is exactly what experienced operations people say.