IT operations Interview Questions

Note: Operations interviews reward specifics about scale and consequence. "I supported the infrastructure" says nothing; "I was on call for 40 servers supporting 1,200 users, with a four-hour recovery objective" says everything.

Cover:

• The estate. How many servers, users, sites, and applications. On-premises, cloud, or hybrid.
• Your scope. Whether you owned the systems or supported someone who did. Be honest — this is easy to test with follow-up questions.
• The service commitments. Uptime targets, support hours, and recovery objectives. Working to a defined SLA is a different discipline from best-effort support.
• What you improved. Strong candidates: automating a manual build process, introducing monitoring where there was none, cutting ticket volume by fixing a root cause, or completing a migration without downtime.

Numbers matter here more than in most interviews — ticket volume, uptime achieved, time saved.

Tell it as a timeline, and separate the technical work from the coordination work — the second is what distinguishes senior operations people.

• Detection and initial assessment. How you found out, and how quickly you established scope: who is affected, how badly, and is it getting worse.
• Communication, started early. Notifying stakeholders and setting an update cadence before you have answers. Silence during an outage causes more damage than the outage.
• Restore service before finding the cause. Failing over, rolling back, restarting, or activating a workaround. Root cause analysis is for afterwards.
• The diagnosis and fix, and how you verified recovery rather than assuming it.
• The post-incident review. What the root cause was, what the contributing factors were, and the specific actions taken so it cannot recur.

Note: Naming an incident command structure — someone owning the technical fix, someone owning communication — is a strong signal. So is describing a blameless review, because it shows a culture where people report problems early rather than hiding them.

Give a framework, since triage under pressure is the core operational skill.

Prioritise on impact and urgency together:

• Scope of impact. An entire site down beats one user with a broken printer, regardless of who is shouting loudest.
• Business criticality. The system that processes revenue outranks the internal wiki. Knowing which systems are critical before an incident is what makes this fast.
• Is it getting worse? A disk filling up outranks a stable failure, because the window to act is closing.
• Is there a workaround? A severe problem with a viable workaround can wait behind a moderate one with none.
• Security incidents jump the queue, because the cost of delay compounds.

The management part: delegate what others can handle, communicate expected timelines to everyone waiting so they stop chasing, and escalate early rather than when it is already too late.

Note: Mentioning that you look for the common cause when several tickets arrive together is a good detail. Five unrelated-seeming issues at once are usually one underlying failure, and recognising that saves hours.

The skill is de-escalation without over-promising.

• Acknowledge the impact first, before explaining anything. People need to know you understand what it is costing them. Leading with technical detail reads as deflection.
• Never be defensive, even when the cause is on their side. Assigning blame during an incident helps nobody.
• Give a realistic timeline, or say honestly that you do not have one yet and commit to an update time. "I do not know the cause yet, I will update you at 3pm" is far better than an optimistic estimate you will miss.
• Offer a workaround where one exists. Partial function restored buys enormous goodwill.
• Follow up after resolution to confirm it is genuinely fixed and explain briefly what happened.

Note: The strongest version of this answer includes recognising when frustration is legitimate — the same problem recurring, or a known issue never fixed. In that case the right response is to escalate for a permanent fix rather than to manage the person's feelings. Interviewers value someone who advocates for users rather than only placating them.

Documentation is where operations teams most commonly fail, so a considered answer stands out.

• Write it for the person on call at 3am, not for an audit. Runbooks with exact commands, expected output, and what to do when it does not match. Prose explaining the architecture is separate and less urgent.
• Document at the moment you solve something. A fix written up while fresh takes ten minutes; reconstructed six months later it takes an hour or never happens.
• Keep it where people look. A wiki nobody opens is worse than nothing because it creates false confidence. Linking runbooks directly from alerts is the single most effective practice.
• Test the documentation. Have someone unfamiliar follow the runbook. Gaps surface immediately.
• Prune aggressively. Out-of-date documentation actively causes incidents.

On knowledge sharing: rotate who handles which type of ticket so knowledge does not concentrate in one person, review incidents as a team, and treat any system only one person understands as a risk to be scheduled away.

Note: Saying you would rather automate a procedure than document it — because automation cannot go stale unnoticed — is a strong closing point.

This is one of the most revealing operations questions. The interviewer is checking honesty, calm under pressure and process improvement. Around 70–80 per cent of outages follow a change, so an experienced candidate who claims this has never happened sounds either inexperienced or evasive.

Structure your answer with STAR, spending most of the time on Action and Result:

• Situation — the change and its context. “During a Saturday maintenance window I applied a firewall rule update to our DMZ.”
• Task — what you owned: the change itself, and therefore the recovery.
• Action — the sequence that matters:
  • Noticed the impact quickly — synthetic checks for the payment gateway went red within five minutes.
  • Linked the symptom to your change immediately rather than hunting elsewhere; the most recent change is the first suspect.
  • Rolled back first, diagnosed second — you executed the rollback plan from the change record, and service returned in 12 minutes.
  • Informed the on-call manager and the service desk, and updated the change record honestly.
• Result and improvement — the root cause (the rule order put a deny above an allow), and what changed: a peer review of rule order for every firewall change, a pre-change export of the ruleset, and a post-change validation checklist that includes testing the payment path.

What interviewers want to hear:

• You owned it without blaming the vendor, the CAB or a colleague.
• Your change had a tested rollback plan, and you used it without hesitation.
• The fix was a process improvement, not just “I will be more careful”.

Note: End with the metric that proves the improvement stuck — for example, “no firewall-related incidents in the following 18 months”. It turns an admission of failure into evidence of maturity.

This question tests on-call discipline — whether you stay methodical when tired, alone and uncertain, and whether you know when to escalate. There is no single right technical answer; the interviewer wants a repeatable approach.

A strong structure for your answer:

1. Acknowledge the page promptly in PagerDuty, Opsgenie or your tool, so it does not auto-escalate and others know someone is on it.
2. Assess user impact first — is the service actually down or degraded? Check the dashboard, synthetic checks and error rates. An alert with no user impact can be handled more calmly than a customer-facing outage.
3. Find the runbook linked to the alert. If there is none, read the alert definition to understand what it measures and its threshold.
4. Check recent changes — deployments, configuration changes and scheduled jobs around the time the alert fired.
5. Gather evidence before acting — logs, metrics and systemctl status output — and note the times of everything you do in the incident channel.
6. Escalate early if impact is high or you are stuck — a common rule is 15–20 minutes without progress on a customer-facing issue. Waking the secondary on-call or the service owner is part of the job, not a failure.
7. Stabilise, then hand over — apply a safe mitigation such as a restart, failover or rollback, and leave a clear written summary for the morning team.

Afterwards: raise a ticket to write or improve the runbook, tune the alert if it was noisy, or open a problem record if it will recur.

Example close: “The alert was for replication lag on a read replica I had never worked on. The customer impact was stale search results, not an outage. I followed the generic database runbook, escalated to the DBA at 20 minutes, and the next day I wrote a dedicated runbook for that alert.”

Note: Avoid heroics. Interviewers are wary of candidates who describe making risky production changes alone at 3 a.m. without telling anyone. Calm, documented and escalated is the winning pattern.

Operations teams value engineers who reduce toil — manual, repetitive work that scales with the size of the estate. This question checks whether you spot automation opportunities and deliver them safely.

Structure with STAR:

• Situation — quantify the toil. “Every month the team spent about two days creating user accounts, mailboxes and group memberships for 60 new joiners by hand, and roughly one in ten had a mistake.”
• Task — what you set out to do, and why it was worth the time.
• Action — show engineering discipline, not just scripting:
  • Documented the manual steps first, which exposed inconsistencies.
  • Wrote a PowerShell script (or Bash or Ansible) driven by the HR export CSV.
  • Made it idempotent — re-running it does not create duplicates — and added a dry-run mode and logging.
  • Tested it against a test OU before production, and got a peer review.
  • Stored it in Git and wrote a short README so the team could run and maintain it.
• Result — time saved, errors reduced and knock-on benefits. “Onboarding dropped from two days to 20 minutes a month, errors went to near zero, and new joiners had access on day one, which cut first-day tickets.”

Points that impress interviewers:

• You chose what to automate by frequency multiplied by effort and error risk.
• You considered what happens when the automation fails, with alerts and clear error messages.
• You made it a team asset rather than a personal tool only you understand.

Note: Numbers make this answer. Hours saved per month, error rate before and after, or tickets reduced turn a nice script into a business result. If you have no work example yet, a well-documented home-lab automation on GitHub is still worth describing.

This tests whether you can protect service stability without being obstructive. Operations people are often pressured by senior stakeholders to “just do it now”; the interviewer wants to see firmness, tact and a helpful alternative.

Structure your answer:

• Situation — who asked, what, and why it was risky. “A business head asked me to restart the ERP application server at 11 a.m. on month-end to apply a vendor hotfix, bypassing the change process.”
• Why you pushed back — frame it in business terms, not rules for their own sake: finance was running month-end closing, the restart would disconnect around 200 users, there was no tested rollback, and it would breach the 99.9 per cent availability SLA with a key client.
• How you pushed back:
  • Acknowledged the urgency and asked what problem the hotfix solved.
  • Explained the risk calmly, with facts.
  • Offered options: an emergency change with quick CAB approval that evening, a temporary workaround for the bug in the meantime, or testing the hotfix in the staging environment that afternoon.
  • Escalated transparently to your manager so the decision sat at the right level, and put it in writing.
• Result — “We applied it as an emergency change at 8 p.m. after testing it in staging. The staging test revealed a missing dependency that would have caused a failed restart in production. The business head later thanked the team.”

What to avoid: sounding rigid (“policy says no”), or a story where you gave in and it went fine — that teaches the wrong lesson.

Note: The key phrase is “not no, but here is how we can do it safely”. Showing that you know the emergency change route exists, and helped the stakeholder use it, demonstrates you understand ITIL as an enabler rather than bureaucracy.

Anyone can restart a service; this question looks for engineers who stop the same incident happening again. It tests analytical skill, persistence and the ability to get fixes delivered across teams.

Structure with STAR:

• Situation — show the pattern and its cost. “Our file server became unresponsive roughly twice a month, usually on a Monday morning, generating 40–50 tickets each time. The standard fix was a reboot.”
• Task — you raised or owned the problem record, separate from the individual incidents.
• Action:
  • Correlated the incidents — pulled ticket timestamps and matched them against monitoring data, finding the hangs always followed the weekend full backup.
  • Gathered evidence — performance counters showed non-paged pool memory exhaustion during backup, and a filter driver from the backup agent was growing steadily.
  • Used a structured method — the 5 Whys led from “server hangs” to “memory leak” to “agent driver version” to “the agent was never included in patching”.
  • Documented a workaround in the known error database so the service desk could act quickly in the meantime.
  • Delivered the permanent fix through change management — upgraded the agent after the vendor confirmed the bug, and added backup agents to the patching scope.
• Result — “Zero recurrences in the following year, around 1,000 fewer tickets annually, and backup agents are now on the patch schedule for the whole estate.”

Points worth emphasising:

• You looked for patterns across tickets rather than treating each one in isolation.
• You separated the workaround from the permanent fix.
• You fixed the underlying process gap, not just the single server.

Note: Close by naming the process lesson — here, that an unmanaged software category existed. Interviewers remember candidates who fix classes of problems, not just instances.

ITIL is a framework of best practices for IT service management, organising operations around defined processes with clear ownership and measurement.

The three terms are distinct and routinely confused:

• Incident — an unplanned interruption or reduction in service quality. The goal is to restore service as quickly as possible, by any means, including a workaround. Fixing the underlying cause is explicitly not the objective. "The email server is down."
• Problem — the underlying cause of one or more incidents. The goal is to find and eliminate the root cause, which is investigative work done on a longer timescale. "Email goes down every Monday because the backup job exhausts memory."
• Change — any addition, modification, or removal that could affect services. The goal is to make changes with minimum risk, through assessment, approval, scheduling, and a rollback plan.

Why the distinction matters practically: incidents are measured on speed of restoration, problems on preventing recurrence. A team that only does incident management fixes the same thing forever — high ticket volume with no improvement. Recognising that pattern is what problem management exists for.

Note: Change types are worth knowing: standard (pre-approved, low risk, such as a routine password reset), normal (assessed and approved by a change advisory board), and emergency (expedited to fix an incident, reviewed afterwards).

RTO (Recovery Time Objective) — how long you can afford to be down. It drives your recovery infrastructure.

RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time. It drives your backup frequency. An RPO of one hour means backing up at least hourly.

Both are business decisions, not technical ones. The cost rises steeply as either approaches zero, so the conversation is about what an hour of downtime or an hour of lost data actually costs.

Designing the strategy:

• The 3-2-1 rule — three copies of the data, on two different media types, with one off-site. It survives hardware failure, site loss, and most ransomware.
• Match backup type to RPO. Full backups are simple to restore but slow and large; incremental are fast to take but slower to restore; differential sit between. A common pattern is weekly full plus daily incremental.
• Immutable or air-gapped copies. This is now essential — ransomware specifically targets backups, and a backup an attacker can delete is not a backup.
• Retention aligned to legal and regulatory requirements, not to disk convenience.

Note: The point to emphasise above all: an untested backup is not a backup. Restore drills on a schedule, timed against the RTO, are what turn a backup policy into a recovery capability. A great many organisations discover their backups were unusable only when they need them.

Work methodically from the symptom outwards rather than guessing.

First, define the problem precisely: who is affected, since when, is it one application or everything, one site or all, and what changed recently. "Everything is slow" and "one application is slow for one office" have entirely different causes.

Then check the resource layers on the server:

• CPU — top or htop. High load average relative to core count, and whether one process dominates.
• Memory — free -h. The critical thing is swap usage; a server swapping is catastrophically slow, and this is often the real answer.
• Disk — iostat and df -h. High I/O wait, or a full filesystem. A full /var from unrotated logs is a classic.
• Network — netstat or ss for connection counts, and interface error counters.

For network problems specifically: ping for reachability and latency, traceroute or mtr to find where latency appears, and DNS resolution time — slow DNS presents exactly like a slow network and is a very common cause.

Note: Two points that impress. First, check monitoring history to see whether this is new or normal, rather than judging from a single snapshot. Second, always ask what changed — a recent deployment, a patch, or a configuration edit explains far more incidents than genuine capacity exhaustion.

Active Directory Domain Services is Microsoft's directory service — a central database of users, computers, groups, and other objects, providing authentication and authorisation across a Windows network. It means one identity instead of a separate account on every machine.

The structure:

• Forest — the top-level security boundary, containing one or more domains.
• Domain — an administrative boundary with its own policies and its own directory partition.
• Organisational Unit (OU) — a container for organising objects, and the unit that Group Policy and delegated administration are applied to.
• Domain Controller — a server hosting the directory and handling authentication, using Kerberos.

Group Policy Objects (GPOs) apply configuration centrally to users and computers: security settings such as password policy and account lockout, software deployment, mapped drives, firewall rules, and desktop restrictions. They are what makes managing thousands of machines feasible.

How GPOs apply — remembered as LSDOU: Local, then Site, then Domain, then OU, with later ones overriding earlier. gpupdate /force forces a refresh and gpresult /r shows what actually applied, which is the first troubleshooting step.

Note: Design OUs around how you administer and apply policy, not around the company org chart. Mirroring the org chart is a common mistake that makes policy application awkward as the organisation reshuffles.

DHCP assigns IP configuration — address, subnet mask, gateway, and DNS servers — to devices automatically through a lease. Without it every device needs manual configuration.

DNS resolves names to addresses. A client asks its configured resolver, which either answers from cache or walks the hierarchy from root to top-level domain to authoritative server.

The record types worth knowing:

• A — name to IPv4 address. AAAA — to IPv6.
• CNAME — an alias pointing to another name.
• MX — mail servers, with priority values.
• TXT — arbitrary text, used for SPF, DKIM, and domain verification.
• NS — the authoritative name servers for a zone.
• PTR — reverse lookup, address to name.

Troubleshooting name resolution, in order:

• Can you reach the IP directly? If yes, it is definitively DNS, not connectivity.
• nslookup or dig against the configured resolver, then against a public one such as 8.8.8.8. A difference means an internal DNS problem.
• Check the client's configured DNS servers — ipconfig /all or resolv.conf. Pointing at a public resolver instead of the domain controller breaks internal name resolution and is a very common misconfiguration.
• Flush the cache — ipconfig /flushdns — since stale entries persist after a record changes.
• Check TTL when a record was recently changed; propagation is really cache expiry.

Patch management balances security against stability, and needs a defined process rather than ad hoc updating.

• Maintain an inventory. You cannot patch what you do not know you have, and unknown systems are where breaches start.
• Categorise by risk. Critical and actively exploited vulnerabilities get an expedited path; routine updates follow the normal cycle.
• Test in rings. A pilot group, then a wider group, then production. This catches the patch that breaks a line-of-business application before it breaks everyone's.
• Schedule a maintenance window with communication and a rollback plan — including a snapshot or backup taken immediately before.
• Verify and report. Compliance reporting shows what remains unpatched and why.

Hardening reduces the attack surface:

• Remove what is not needed — unused services, default applications, and open ports. The most reliable security measure is absence.
• Change every default credential, and disable default accounts.
• Least privilege — users are not local administrators, and service accounts get only the rights they need.
• Enable host firewall and disk encryption, enforce secure protocols, and disable legacy ones such as SMBv1 and TLS 1.0.
• Centralise logging so evidence survives the compromise of the host.

Note: Cite a baseline standard such as CIS Benchmarks. Working to a published benchmark rather than personal preference is what makes hardening auditable.

Virtualisation runs multiple isolated systems on one physical host through a hypervisor, which allocates CPU, memory, storage, and network to each guest. Type 1 hypervisors — ESXi, Hyper-V, KVM — run directly on hardware; Type 2 run on a host operating system.

The operational benefits are consolidation (better hardware use), isolation, snapshots for safe rollback, and live migration allowing hardware maintenance with no downtime.

Virtual machines versus containers:

• A VM virtualises hardware. Each has its own complete guest operating system, so it is heavy — gigabytes, and tens of seconds to boot — but strongly isolated and able to run any OS.
• A container virtualises the operating system. Containers share the host kernel and package only the application and its dependencies, so they are megabytes and start in under a second. Isolation is process-level, which is weaker than a hypervisor boundary.

How to choose: VMs for running different operating systems, for legacy applications, and where strong isolation is a security requirement — such as multi-tenant workloads. Containers for application deployment, microservices, and anywhere consistency between development and production matters.

Note: They are complementary rather than competing. In practice containers usually run on VMs — a Kubernetes cluster is typically a set of virtual machines — combining the isolation of one with the density and portability of the other.

Monitor at layers, because a symptom at one level usually has a cause at another:

• Infrastructure — CPU, memory, disk space and I/O, network throughput and errors, and hardware health.
• Service — is the process running, is the port listening, is the certificate about to expire, is the backup job succeeding.
• Application — request rate, error rate, and response time percentiles.
• End-user experience — synthetic checks that perform a real transaction from outside your network. This catches what internal monitoring misses entirely.
• Logs — centralised, so evidence survives the failure of the host that produced it.

Alerting is where most teams go wrong. The principles:

• Alert on symptoms, not causes. Alert that the site is returning errors, not that CPU is at 80%. High CPU with everything working is not an incident.
• Every alert must be actionable and have a documented response. An alert nobody acts on trains people to ignore all alerts, which is worse than no monitoring.
• Use severity tiers. Page for something that needs waking someone; email or ticket for what can wait until morning.
• Use trends for capacity. Alerting when a disk will be full in five days beats alerting when it is full.

Note: Naming alert fatigue as the main failure mode, and reviewing alerts periodically to delete noisy ones, is exactly what experienced operations people say.

Least privilege means every user, account, and process gets the minimum access needed to do its job, and no more. It limits both accidental damage and the blast radius of a compromised account.

How to implement it:

• Role-based access control. Define roles matching job functions, assign permissions to roles, and users to roles. Never assign permissions to individuals — it becomes unmanageable and unauditable within months.
• Separate administrative accounts. Administrators get a normal account for daily work and a separate privileged account used only for administrative tasks. Browsing email as a domain administrator is how domains fall.
• Just-in-time elevation. Privileged access granted temporarily and automatically revoked, rather than standing rights.
• Service accounts with only the specific rights required, non-interactive logon, and managed or rotated credentials.

The lifecycle is where this usually fails:

• Joiners — provisioned from a role template, not by copying an existing user's access. Copying is how permissions accumulate across an organisation.
• Movers — old access removed when someone changes role. Privilege creep from people who have moved teams three times is a real and widespread risk.
• Leavers — disabled immediately, ideally automatically from the HR system.
• Periodic access reviews where managers confirm their team's access is still appropriate.

RAID combines physical disks for redundancy, performance, or both.

• RAID 0 (striping) — data split across disks. Fast, full capacity, and no redundancy at all: one disk failing loses everything. Only for scratch data you can recreate.
• RAID 1 (mirroring) — identical copies on two disks. Survives one failure, gives fast reads, but costs 50% of capacity.
• RAID 5 (striping with distributed parity) — survives one disk failure, with the capacity of n-1 disks. The concern is rebuild time: on large modern drives a rebuild takes many hours under full load, and a second failure during it loses the array.
• RAID 6 — dual parity, surviving two simultaneous failures. The sensible choice for large arrays precisely because of the rebuild risk.
• RAID 10 — mirrored pairs, striped. Best performance and fast rebuilds, at 50% capacity cost. Standard for databases.

Capacity planning: baseline current usage, measure the growth rate over several months rather than guessing, project forward, and add headroom — filesystems degrade well before 100%, and 80% is a common trigger point. Plan procurement lead time into the threshold, so the alert fires early enough to actually order hardware.

Note: The essential point: RAID is availability, not backup. It protects against disk failure only — not deletion, corruption, or ransomware, all of which replicate instantly across every disk in the array.

These three terms describe service reliability at different levels, from the raw measurement up to the contract.

• SLI (Service Level Indicator) — the actual measurement. For example, the percentage of HTTP requests that succeeded, the proportion served in under 300 ms, or the ticket first-response time. Good SLIs reflect what users actually experience.
• SLO (Service Level Objective) — the internal target for an SLI over a time window. “99.9 per cent of requests succeed over a rolling 30 days.” It is set by the service owner and engineering team.
• SLA (Service Level Agreement) — the external contract with a customer, with consequences such as service credits or penalties if it is missed. SLAs are deliberately set looser than SLOs — for example SLO 99.9 per cent, SLA 99.5 per cent — so the team gets warning before breaching the contract.

In ITSM, SLAs also cover support: “P1 incidents responded to within 15 minutes and resolved within 4 hours.” Internal agreements between teams are called OLAs (Operational Level Agreements), and contracts with vendors are underpinning contracts.

Error budget — the allowed unreliability implied by the SLO: 100 per cent minus the SLO.

How it is used: while budget remains, the team can ship changes and take calculated risks. If the budget is exhausted, the policy shifts to reliability work — a feature freeze, extra testing or slower rollouts — until it recovers. It turns the endless argument between “move fast” and “keep it stable” into an objective, data-driven decision.

Note: Aiming for 100 per cent is a mistake — it is unachievable, extremely expensive, and users cannot tell the difference beyond a point because their own networks are less reliable. Choosing the right SLO is a business decision.

Change management (called change enablement in ITIL 4) exists to let the organisation make changes quickly while minimising the risk of disruption. A typical normal change follows these steps:

1. Raise the RFC (Request for Change) — in ServiceNow, Jira Service Management or similar. It records what is changing, why, which configuration items and services are affected, the planned window and the owner.
2. Risk and impact assessment — how many users and services are affected, whether it touches a critical system, what happens if it fails, and whether it has been done before. Many tools score risk from a short questionnaire.
3. Plan the implementation — step-by-step procedure, pre-checks, a tested rollback (back-out) plan with a clear trigger point (for example, “if validation fails by 23:30, roll back”), and post-change validation steps.
4. Approval — low-risk changes by a team lead or change manager; significant ones by the CAB (Change Advisory Board), which brings together service owners, operations, security and the service desk to spot conflicts and risks. The forward schedule of changes prevents two risky changes on the same night.
5. Communicate and schedule — inform the service desk and affected users; respect freeze periods such as month-end or festive sales.
6. Implement — within the approved window, following the plan and logging deviations.
7. Validate — confirm the service works using the agreed checks, not just “the command succeeded”.
8. Review and close — record the outcome. A Post-Implementation Review (PIR) is required for failed changes, emergency changes and major changes.

Change types:

• Standard — low-risk, repeatable and pre-approved, following a documented procedure.
• Normal — assessed and approved as above.
• Emergency — needed urgently to resolve or prevent a major incident; approved by a smaller ECAB and fully documented afterwards.

Key metrics: change success rate, number of changes causing incidents, emergency change percentage and unauthorised changes detected.

Note: A mature view is that the goal is safe speed, not bureaucracy. Moving well-understood changes into the standard category and automating their checks frees the CAB to focus on the genuinely risky ones.

Incident — an unplanned interruption to a service or a reduction in its quality, or the failure of a component that has not yet affected service. “Outlook will not open.” “The VPN keeps disconnecting.”

Service request — a formal request from a user for something standard and pre-defined that is not a failure. “I need access to the finance shared drive.” “Please install Visio.” “New joiner laptop.” Requests are usually fulfilled from a service catalogue with standard workflows, approvals and fulfilment times.

Why keep them separate: they have different SLAs, different workflows and different metrics. Mixing them distorts reporting — a spike of access requests can look like an instability problem. Requests are good candidates for automation and self-service portals.

Setting priority: impact multiplied by urgency.

• Impact — how widespread and how serious: one user, a department, the whole company, or customer-facing revenue.
• Urgency — how quickly the business needs it resolved: can work continue with a workaround, or is a deadline at risk?

Example targets: P1 — respond in 15 minutes, resolve in 4 hours, often triggering the major incident process; P4 — respond in 8 business hours, resolve in 5 days.

Common pitfalls:

• Priority driven by who is shouting rather than by business impact. A VIP may justify higher urgency, but that should be an explicit, documented rule.
• Not re-prioritising when impact grows — one user’s problem may turn out to affect everyone.

Note: Mention the other two ITIL record types for completeness: a problem is the underlying cause of incidents, and a change is a modification to the environment. Knowing all four and how they link shows a solid ITSM foundation.

Root cause analysis (RCA) aims to find why an incident really happened, so that the fix prevents recurrence instead of treating the symptom. Several techniques are used, often together.

1. Timeline reconstruction — always the first step. Build a minute-by-minute sequence from monitoring, logs, change records and chat messages: when the fault began, when it was detected, what was done, and when service recovered. Many false assumptions disappear once the timeline is accurate.

2. The 5 Whys — keep asking “why?” until you reach a cause you can act on.

• Why did the website go down? The web servers ran out of disk.
• Why? Application logs filled /var/log.
• Why? Debug logging had been left on after a release.
• Why? The release checklist does not include resetting log levels.
• Why? Log configuration is manual and not part of deployment automation.

The fix is automation and log rotation — not just deleting logs. Its weakness is that it follows a single chain, and real incidents often have several contributing causes.

3. Fishbone (Ishikawa) diagram — brainstorm causes by category so nothing is missed. Common IT categories: People, Process, Technology, Environment, Vendors, Monitoring. It suits complex incidents with several contributing factors.

4. Change and fault correlation — ask “what changed?” Compare deployments, patches, configuration and traffic patterns against the incident start time.

5. Fault tree analysis — work top-down from the failure through AND/OR logic gates to basic events. Used for high-risk systems.

6. Kepner-Tregoe “is / is not” analysis — compare where the problem occurs with where it does not: one data centre but not the other, one browser but not another. The differences point to the cause.

Good RCA principles:

• Blameless — “human error” is a starting point, not a root cause. Ask why the system allowed the error.
• Evidence-based — every conclusion supported by data.
• Every cause leads to an owned, dated corrective action.

Note: Many teams now prefer the phrase “contributing factors” over a single root cause, because complex outages rarely have just one. Saying so shows you have done real RCA rather than read about it.

A postmortem (also called an incident review or post-incident report) captures what happened during a significant incident and what will change as a result. It should be blameless — focused on systems and processes, not individuals — so that people share the full truth.

Typical sections:

• Summary — two or three sentences a senior manager can read: what broke, for how long and who was affected.
• Impact — quantified: duration, users or customers affected, failed transactions, revenue, SLA breaches, and the error budget consumed.
• Timeline — timestamped events, including when the issue started, when it was detected and how, key decisions, mitigation and full recovery. The gap between start and detection is often the most important finding.
• Root cause and contributing factors — the technical cause plus the conditions that allowed it, such as missing tests, alert gaps or an unclear runbook.
• Detection and response — what went well, what went badly and where we got lucky.
• Action items — each with a single owner, a due date, a priority and a tracking ticket, classified as prevent, detect faster or mitigate faster.
• Supporting data — graphs, log excerpts and links to the incident channel.

Making sure actions are completed:

• Create every action as a ticket in the team’s normal backlog, not just as text in a document.
• Prefer a few high-value, specific actions over a long wish list — “add a disk-usage alert at 80 per cent on all web servers” rather than “improve monitoring”.
• Review open postmortem actions in a weekly operations meeting, and report overdue items to management.
• Link the actions to the problem record so the problem closes only when they are done.
• Track repeat incidents — a recurrence with open actions is a clear escalation trigger.

Timing: draft the postmortem within a few days while memories are fresh, and hold a short review meeting with everyone involved.

Note: Sharing postmortems widely — even across teams — spreads the learning. A well-written postmortem is one of the most valuable documents an operations team produces.

These “mean time” metrics measure how often things fail and how fast the team responds. Together they show where to invest: preventing failures, detecting them sooner or fixing them faster.

• MTBF (Mean Time Between Failures) — the average operating time between one failure and the next for a repairable system. Higher is better. If a server ran 2,000 hours with four failures, MTBF is 500 hours. It measures reliability.
• MTTF (Mean Time To Failure) — the equivalent for non-repairable components such as disks, which are replaced rather than fixed.
• MTTD (Mean Time To Detect) — from the start of an issue until it is detected, ideally by monitoring rather than by a customer phoning in. It measures observability.
• MTTA (Mean Time To Acknowledge) — from the alert firing until a person acknowledges it. It measures on-call responsiveness and alert routing.
• MTTR — usually Mean Time To Restore or Recover: from the start of the incident until service is restored. Sometimes it means Repair or Resolve, so always define which one you use.

How the timeline fits together:

failure starts --[MTTD]--> alert --[MTTA]--> engineer engaged --> diagnosed --> restored
|----------------------------------- MTTR ------------------------------------|

Availability relates them: availability equals MTBF divided by (MTBF plus MTTR). Halving MTTR improves availability just as effectively as doubling MTBF — and is usually cheaper to achieve through better runbooks, automation and quick rollback.

Using them well:

• Track trends per service and per priority, not one blended number.
• Averages hide outliers, so also look at the median and 90th percentile.
• A falling MTTD usually comes from better monitoring and synthetic checks; a falling MTTR from runbooks, automation and practice.
• Do not let targets encourage bad behaviour, such as closing tickets before the fix is confirmed.

Note: The DORA metrics used in DevOps — deployment frequency, lead time for changes, change failure rate and time to restore service — overlap with these. Mentioning that link shows you understand both the ITSM and the DevOps view of operations.

systemd is the init system and service manager on almost all modern Linux distributions — RHEL, Rocky, Ubuntu and Debian. It starts services in parallel, tracks dependencies, restarts failed processes and collects logs in the journal.

Everyday commands:

systemctl status nginx          # state, main PID, recent log lines
systemctl start|stop|restart nginx
systemctl reload nginx          # re-read config without dropping connections
systemctl enable --now nginx    # start at boot AND start now
systemctl disable nginx
systemctl is-enabled nginx
systemctl list-units --failed   # everything that has failed
systemctl daemon-reload         # after editing unit files

Important distinction: start runs the service now; enable only creates the links that start it at boot. Forgetting to enable a service is a classic cause of “it worked until the reboot”.

Reading logs with journalctl:

journalctl -u nginx -e             # this unit, jump to the end
journalctl -u nginx -f             # follow live
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b               # errors since the last boot
journalctl -b -1                   # the previous boot (needs a persistent journal)

A unit file lives in /etc/systemd/system/app.service:

[Unit]
Description=My App
After=network-online.target

[Service]
User=app
ExecStart=/opt/app/bin/start
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Use systemctl edit app to create a drop-in override instead of editing vendor unit files.

Troubleshooting a service that will not start:

1. systemctl status — read the exit code and the last log lines.
2. journalctl -u app -e for the full error.
3. Validate the configuration, such as nginx -t or apachectl configtest.
4. Check that the port is free (ss -tlnp), file permissions and ownership, SELinux denials (ausearch -m avc) and missing environment variables.
5. Run the ExecStart command manually as the service user to see the error directly.

Note: If a service hits its start limit after repeated failures, it stays failed until you run systemctl reset-failed. Knowing that small detail tells an interviewer you have actually fought with systemd in production.

Logs are the first evidence in almost every investigation. A methodical approach is to know where logs live, narrow down by time and keyword, and correlate across sources.

Where to look:

• /var/log/messages (RHEL family) or /var/log/syslog (Debian and Ubuntu) — general system messages.
• /var/log/secure or /var/log/auth.log — logins, sudo and SSH activity.
• /var/log/dmesg or dmesg -T — kernel messages: disk errors, OOM kills, driver problems.
• Application logs — /var/log/nginx/, /var/log/httpd/, /var/log/mysql/ or an application directory under /opt.
• The systemd journal — journalctl, covering everything logged by services.

Useful commands:

tail -f /var/log/nginx/error.log
grep -i "error" /var/log/syslog | tail -50
grep -c " 500 " /var/log/nginx/access.log                  # count HTTP 500s
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head   # top client IPs
zgrep "Failed password" /var/log/auth.log*                 # includes rotated .gz files
dmesg -T | grep -i -E "oom|error|fail"
journalctl --since "2025-06-01 10:00" --until "2025-06-01 10:30"

Technique: start from the time the problem began, work outward, and compare several sources — an application timeout at 10:02 plus a kernel disk error at 10:01 tells a clear story. Make sure clocks are synchronised with NTP, or correlation is unreliable.

Log rotation with logrotate stops logs filling the disk. It runs daily from cron or a systemd timer, using /etc/logrotate.conf and files in /etc/logrotate.d/:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

copytruncate suits applications that cannot reopen their log file; otherwise use a postrotate script that signals the application to reopen it. Test with logrotate -d /etc/logrotate.d/myapp for a dry run, and force a rotation with -f.

Note: At scale, logs are shipped to a central platform such as ELK or OpenSearch, Graylog, Splunk or Loki. Saying that grep is for a single server, and centralised search is for correlation across fifty, shows the right operational perspective.

A full disk can stop databases, block logins and crash applications, so the aim is to free space quickly without deleting anything important, then fix the cause.

1. Confirm which filesystem is full:

df -h           # space per filesystem
df -i           # inode usage — "no space left" can mean inodes are exhausted

2. Find what is using the space, staying on one filesystem with -x:

du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head
find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null
ncdu -x /       # interactive, if installed

3. Check for deleted files still held open. If df shows the disk full but du totals far less, a process is still writing to a file that was deleted — the space is not released until the process closes it.

lsof +L1                   # open files with zero links
lsof | grep deleted

Restart or reload the process, or truncate through its file descriptor: : > /proc/PID/fd/N.

Common culprits and safe fixes:

• Huge log files — truncate rather than delete while the application is running: truncate -s 0 app.log. Then fix logrotate.
• Old rotated logs — compress or remove old .gz files once you have confirmed retention requirements.
• The systemd journal — journalctl --vacuum-size=500M.
• Package caches — dnf clean all or apt clean.
• Old kernels, core dumps, forgotten backups and Docker images — docker system df and docker system prune, with care.
• Inode exhaustion from millions of tiny session or cache files — find the directory with a very high file count and clean it.

4. Prevent recurrence: alert at 80 and 90 per cent, apply log rotation and retention, put /var or application data on separate filesystems, and extend the volume with LVM if growth is genuine.

Note: Never delete files in /var/lib/mysql, database transaction logs or anything you do not understand to win space quickly — you can turn a disk-full incident into a data-loss incident. Remember that 5 per cent of an ext4 filesystem is reserved for root by default.

Every file and directory has an owner, a group and three sets of permissions — for the user (owner), the group and others.

$ ls -l deploy.sh
-rwxr-x---  1 appuser devops  2048 Jun 1 10:00 deploy.sh

Here the owner can read, write and execute; the group can read and execute; others have no access.

What r, w and x mean differs for directories:

Changing permissions and ownership:

chmod 750 deploy.sh        # rwx r-x ---
chmod u+x,g-w,o-r file     # symbolic form
chmod -R g+rX /srv/app     # X = execute only on directories
chown appuser:devops file
chown -R www-data:www-data /var/www/site

Common octal values: 644 for normal files, 755 for scripts and directories, 600 for private keys, 700 for private directories. Avoid chmod 777 — it is almost never the right fix.

umask sets default permissions for new files by removing bits. With the typical umask 022, new files get 644 and new directories 755; with 027, others get nothing.

Special bits:

• setuid (4xxx) on an executable — it runs with the owner’s privileges. /usr/bin/passwd uses it to update the shadow file. Unexpected setuid binaries are a security red flag: find / -perm -4000 -type f.
• setgid (2xxx) on a directory — new files inherit the directory’s group, which is ideal for shared team folders.
• Sticky bit (1xxx) on a directory — users can delete only their own files, as on /tmp (mode 1777).

Beyond basic permissions: ACLs (setfacl and getfacl) grant access to extra users or groups, and SELinux or AppArmor can block access even when the permissions look correct.

Note: When “permission denied” makes no sense, check every parent directory for the x bit (namei -l /path/to/file shows the whole chain) and check SELinux contexts with ls -Z. Those two checks solve most puzzling cases.

cron runs commands on a schedule. Each user has a crontab, edited with crontab -e and listed with crontab -l. System jobs live in /etc/crontab and /etc/cron.d/, which include an extra user field, and there are drop-in folders such as /etc/cron.daily/.

The five time fields:

# minute  hour  day-of-month  month  day-of-week   command
  30      2     *             *      *             /opt/scripts/backup.sh
  */15    *     *             *      *             /opt/scripts/healthcheck.sh
  0       9     *             *      1-5           /opt/scripts/report.sh
  0       0     1             *      *             /opt/scripts/monthly.sh

These run at 02:30 daily, every 15 minutes, at 09:00 on weekdays and at midnight on the 1st of each month. Day-of-week 0 or 7 is Sunday. Shortcuts such as @daily and @reboot also exist. Check expressions with an online cron-expression tool before deploying.

Troubleshooting a job that does not run — work through these in order:

1. Is cron running? systemctl status cron (Debian/Ubuntu) or crond (RHEL).
2. Did it trigger? Check grep CRON /var/log/syslog or /var/log/cron, or journalctl -u cron.
3. Environment — the most common cause. Cron runs with a minimal PATH (often just /usr/bin:/bin) and no login profile, so commands that work in your shell fail. Use absolute paths for every command and file, or set PATH at the top of the crontab.
4. Capture output — cron discards errors unless mail is configured. Redirect them: /opt/scripts/backup.sh >> /var/log/backup.log 2>&1.
5. Permissions — is the script executable, does the user own or can it read the files, is the user blocked by /etc/cron.deny?
6. Syntax — a % in the command must be escaped as \%; files in /etc/cron.d must not contain dots in their names and need a trailing newline.
7. Time zone — the server may be on UTC while you expected IST.

Good practice: use flock to stop overlapping runs (flock -n /tmp/backup.lock /opt/scripts/backup.sh), log start and end times, and alert on failure — or use a “dead man’s switch” check that alerts if a job has not reported success.

Note: systemd timers are the modern alternative, with built-in logging in the journal, dependency handling and catch-up of missed runs through Persistent=true. Mentioning both shows current knowledge.

Load average is the average number of tasks that are either running or waiting to run on the CPU, plus those in uninterruptible sleep (state D, usually waiting for disk or network storage I/O). Linux reports it over 1, 5 and 15 minutes:

$ uptime
 10:42:01 up 12 days,  load average: 7.85, 4.10, 2.02
$ nproc
4

Interpreting it: compare the load with the number of CPU cores. On a 4-core server a load of 4 means the CPUs are, on average, fully busy; 8 means roughly as many tasks waiting as running. The three numbers show the trend — above, 7.85 against 2.02 means load is rising.

How it differs from CPU utilisation:

• CPU utilisation is the percentage of time the CPUs were busy — a measure of how hard they are working, capped at 100 per cent.
• Load average is a queue length — a measure of demand, with no upper limit.
• Because Linux counts D-state tasks, you can see high load with low CPU usage. That almost always means processes stuck waiting for I/O — a slow disk, a saturated SAN or a hung NFS mount — not a CPU problem.

Commands to break it down:

top            # %us user, %sy system, %wa iowait, %st steal
vmstat 1       # r = run queue, b = blocked on I/O
mpstat -P ALL 1   # per-core usage — spot one pegged core
iostat -xz 1   # disk utilisation and await times
ps -eo state,pid,cmd | grep "^D"   # processes stuck in D state

What each pattern suggests:

• High load, high %us — application CPU demand; find the process with top and consider scaling.
• High load, high %wa, many D-state tasks — storage bottleneck.
• High %st (steal) on a VM — the hypervisor is giving your CPU time to other guests.
• High %sy — kernel overhead, such as heavy context switching or interrupts.

Note: Never alert on load average alone with a fixed number across servers of different sizes. Normalise it by core count, and pair it with CPU, iowait and latency metrics so the alert points to a cause rather than just a symptom.

SSH is the main door into Linux servers, so it is also a prime attack target. Internet-facing servers see constant automated password-guessing. Key-based authentication plus a hardened configuration closes most of that risk.

Setting up key-based login:

ssh-keygen -t ed25519 -C "ravi@laptop"      # creates private and public key
ssh-copy-id ravi@server01                   # appends the public key to authorized_keys
ssh ravi@server01                           # test BEFORE disabling passwords

The public key goes into ~/.ssh/authorized_keys on the server. Permissions matter — SSH refuses keys if they are too open: chmod 700 ~/.ssh and chmod 600 ~/.ssh/authorized_keys. Protect the private key with a passphrase and use ssh-agent.

Hardening /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups ssh-users
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
ClientAliveInterval 300

Validate and apply safely: sshd -t to check syntax, then systemctl reload sshd — and keep your existing session open while you test a new login in a second window, so a mistake cannot lock you out.

Further layers:

• Restrict by network — firewall SSH to the VPN or office ranges only, or put servers behind a bastion (jump) host: ssh -J bastion server01.
• fail2ban — bans IP addresses after repeated failures.
• MFA for SSH through PAM, or short-lived SSH certificates from a central CA (for example HashiCorp Vault or Teleport) instead of long-lived keys.
• Cloud options — AWS Systems Manager Session Manager or Azure Bastion remove the need for open SSH ports altogether.
• Key lifecycle — remove keys when people leave, and audit authorized_keys files regularly.
• Auditing — review /var/log/secure or auth.log, and forward logs to the SIEM.

Note: Changing the SSH port from 22 reduces log noise but is not real security. Interviewers are more impressed by bastion hosts, disabled passwords and key lifecycle management than by a non-standard port.

Most Active Directory operations are multi-master — any domain controller can accept changes. A few operations would conflict if two DCs did them at once, so they are assigned to a single DC as FSMO (Flexible Single Master Operations) roles.

Checking and moving roles:

netdom query fsmo

# PowerShell — transfer (graceful, holder is online)
Move-ADDirectoryServerOperationMasterRole -Identity DC02 -OperationMasterRole PDCEmulator,RIDMaster

# Seize (holder permanently dead) — add -Force
Move-ADDirectoryServerOperationMasterRole -Identity DC02 -OperationMasterRole PDCEmulator -Force

Transfer versus seize: transfer when the current holder is online, as during planned maintenance or decommissioning. Seize only when the holder is permanently lost — and that DC must never be brought back online afterwards; it must be rebuilt, or you risk duplicate RIDs and corruption.

Best practice: keep the PDC Emulator on a well-connected, reliable DC configured to sync time from a trusted external NTP source, and document which DC holds each role.

Note: For a short outage of a FSMO holder you usually do nothing — most roles are not needed minute to minute. Seizing roles in a panic during a brief reboot is a classic mistake that turns a small incident into a large one.

Active Directory is multi-master: a change made on any domain controller must replicate to all the others. Replication failures cause confusing symptoms — a new user can log in at one office but not another, a password reset “does not work”, or Group Policy differs between sites.

How it works:

• Intra-site replication is fast and change-notified, usually within seconds.
• Inter-site replication follows site links and schedules — by default every 180 minutes — and is compressed to save WAN bandwidth.
• The KCC (Knowledge Consistency Checker) builds the replication topology automatically from sites, subnets and site links defined in AD Sites and Services.
• Changes are tracked by USNs (update sequence numbers), and conflicts are resolved by version number and timestamp.
• SYSVOL (Group Policy files and logon scripts) replicates separately through DFSR.

Troubleshooting commands:

repadmin /replsummary              # overview: failures and largest deltas per DC
repadmin /showrepl DC01            # inbound partners, last success, errors
repadmin /queue                    # pending inbound replication
repadmin /syncall /AdeP            # force sync of all partitions, all partners
dcdiag /v /c /e                    # comprehensive health tests on all DCs
dcdiag /test:dns /v                # DNS health specifically

Common causes and what to check:

• DNS — by far the most common. DCs must use AD-integrated DNS servers and resolve each other’s names and SRV records. Check DNS client settings and run ipconfig /registerdns and restart the Netlogon service to re-register records.
• Network and firewalls — required ports: 135 plus the dynamic RPC range, 389, 88, 445 and 53. Test with Test-NetConnection DC02 -Port 135.
• Time skew — more than five minutes breaks Kerberos authentication between DCs.
• Error 1722 (RPC server unavailable) — connectivity or DNS. Error 8606 or event 1988 (lingering objects) — a DC was offline longer than the tombstone lifetime (180 days by default).
• Journal wrap or DFSR errors for SYSVOL — check the DFS Replication event log.

Note: A DC that has been offline longer than the tombstone lifetime should not simply be reconnected. The safe fix is usually to demote or remove it with metadata cleanup and rebuild it, rather than forcing replication and reintroducing deleted objects.

Repeated lockouts are one of the most common and frustrating AD tickets. Unlocking the account treats the symptom; the real task is finding which device is sending the bad password. Usually it is an old credential cached somewhere after a password change.

Step 1: Find where the lockouts come from. Lockouts are processed on the PDC Emulator, so its Security log records event 4740 (“A user account was locked out”) with a Caller Computer Name field.

# Find the PDC Emulator
Get-ADDomain | Select-Object PDCEmulator

# Query lockout events for a user
Get-WinEvent -ComputerName DC01 -FilterHashtable @{LogName='Security'; Id=4740} |
  Where-Object { $_.Properties[0].Value -eq 'rsharma' } |
  Select-Object TimeCreated, @{n='Source';e={$_.Properties[1].Value}}

# Current lockout state and bad password count
Get-ADUser rsharma -Properties LockedOut, BadLogonCount, LastBadPasswordAttempt

Also check event 4625 (failed logon) and 4771 (Kerberos pre-authentication failed) on DCs for the source IP address. Microsoft’s free Account Lockout Status tool (LockoutStatus.exe) shows the bad-password count per DC.

Step 2: Check the usual culprits on the source device:

• Mobile phones syncing mail through Exchange ActiveSync with the old password — the single most common cause. The caller is often an Exchange or mail gateway server.
• Stored credentials in Windows Credential Manager (rundll32 keymgr.dll,KRShowKeyMgr).
• Mapped drives that were connected with explicit credentials.
• Scheduled tasks or services running under the user’s account.
• Disconnected RDP sessions on other servers still using the old credentials.
• Wi-Fi or VPN profiles with saved passwords, and old laptops the user forgot about.

Step 3: Fix and verify: remove or update the stale credential, unlock the account (Unlock-ADAccount rsharma) and watch for new 4740 events.

If the source is external or unknown — many failures from unfamiliar IP addresses against many accounts — treat it as a possible password-spraying attack and involve security.

Note: Service accounts that lock out can take down applications, so the long-term fixes are Group Managed Service Accounts (gMSA) with automatic password rotation, and a lockout threshold sensible enough — commonly around 10 attempts — to stop attacks without constant user lockouts.

PowerShell turns work that would take hours in graphical consoles into repeatable, auditable commands. It is object-based — commands pass structured objects, not text, through the pipeline — which makes filtering and reporting reliable.

Active Directory tasks (the ActiveDirectory module, part of RSAT):

# Users inactive for 90+ days
Search-ADAccount -AccountInactive -TimeSpan 90.00:00:00 -UsersOnly |
  Select-Object Name, LastLogonDate | Export-Csv inactive.csv -NoTypeInformation

Import-Csv newjoiners.csv | ForEach-Object {     # bulk-create users from a CSV
  New-ADUser -Name $_.Name -SamAccountName $_.Sam -Department $_.Dept `
    -Path 'OU=Staff,DC=corp,DC=local' -Enabled $true `
    -AccountPassword (ConvertTo-SecureString $_.TempPwd -AsPlainText -Force)
}

Add-ADGroupMember -Identity 'Finance-Share-RW' -Members rsharma, akumar
Get-ADGroupMember 'Domain Admins' -Recursive     # audit privileged groups

Remote administration of many servers:

# Run a command on many servers in parallel
Invoke-Command -ComputerName (Get-Content servers.txt) -ScriptBlock {
  Get-Service -Name Spooler | Select-Object Status
}

Enter-PSSession -ComputerName FS01               # interactive remote session

Get-CimInstance Win32_LogicalDisk -ComputerName FS01 -Filter "DriveType=3" |
  Select-Object DeviceID, @{n='FreeGB';e={[math]::Round($_.FreeSpace/1GB,1)}}

This relies on WinRM (PowerShell Remoting), enabled with Enable-PSRemoting and usually managed by Group Policy.

Everyday Windows Server tasks: Get-EventLog or Get-WinEvent for logs, Install-WindowsFeature for roles, Get-HotFix for patches, Test-NetConnection for connectivity, and Restart-Service.

Good practices:

• Use -WhatIf and -Confirm to preview destructive changes.
• Filter on the server side with -Filter rather than retrieving everything and using Where-Object — much faster in large directories.
• Use Start-Transcript for an audit trail, and keep scripts in Git.
• Never hard-code passwords; use Get-Credential or a secrets vault.
• Use JEA (Just Enough Administration) to let helpdesk staff run specific commands without full admin rights.

Note: Know the difference between Windows PowerShell 5.1, which ships with Windows, and PowerShell 7, which is cross-platform. Most AD administration works in both, but some older modules still need 5.1.

DHCP (Dynamic Host Configuration Protocol) hands out IP addresses and network settings — subnet mask, default gateway, DNS servers and lease time — automatically. It uses UDP ports 67 (server) and 68 (client).

The DORA exchange:

1. Discover — the client, which has no address yet, broadcasts “is there a DHCP server?”
2. Offer — a server offers an available address from its scope.
3. Request — the client broadcasts that it accepts that offer, which also tells other servers their offers were declined.
4. Acknowledge — the server confirms the lease and its duration.

At 50 per cent of the lease the client tries to renew directly with the same server; at 87.5 per cent it tries any server (rebinding).

Why relays matter: DHCP Discover is a broadcast, and routers do not forward broadcasts. Clients on a different VLAN from the DHCP server need a DHCP relay — the ip helper-address command on Cisco devices — on their gateway interface.

Troubleshooting a client with no address (often showing 169.254.x.x):

1. Is it just one client or many? Many clients on one VLAN points to the server, scope or relay; one client points to that device or its switch port.
2. Physical and link layer — cable, switch port status, correct VLAN, and whether port security or 802.1X is blocking the device.
3. Client side — ipconfig /release and ipconfig /renew on Windows; dhclient -v eth0 or nmcli con up eth0 on Linux.
4. Scope exhaustion — the classic cause. Check the scope statistics in the DHCP console, or run Get-DhcpServerv4ScopeStatistics. Fixes: shorten the lease time on busy guest Wi-Fi, reclaim stale leases, or enlarge the scope.
5. DHCP server health — is the service running, is the Windows DHCP server authorised in AD, is the scope active?
6. Relay configuration — is the ip helper-address present and correct on the VLAN interface after a recent network change?
7. Rogue DHCP servers — a home router plugged into the network handing out wrong addresses; defend with DHCP snooping on switches.
8. Packet capture — tcpdump -i eth0 port 67 or port 68 shows exactly which DORA step fails.

Note: For resilience, use DHCP failover between two Windows servers (load-balance or hot-standby mode) or split scopes, and reserve addresses for printers and servers rather than giving them static IPs inside the dynamic range.

Good network troubleshooting works up the layers, using the right tool to answer one question at a time.

Is the interface up and configured?

ip addr show ; ip route              # Linux: addresses and routing table
ipconfig /all                        # Windows: IP, gateway, DNS, DHCP lease

Can I reach the host? ping uses ICMP echo to test reachability and round-trip time. Packet loss or rising latency is significant. A failed ping does not prove the host is down — ICMP is often blocked by firewalls.

Where along the path does it fail? traceroute (Linux) or tracert (Windows) lists each router hop. mtr or pathping combine this with ongoing loss statistics per hop — ideal for intermittent problems and for evidence to give an ISP.

Does the name resolve correctly?

dig app.corp.local +short
dig @8.8.8.8 example.com            # ask a specific resolver
nslookup app.corp.local 10.0.0.10
Resolve-DnsName app.corp.local      # PowerShell

Is the port open and the service listening?

ss -tlnp                            # Linux: listening TCP ports and their processes
netstat -ano                        # Windows: connections with PID
nc -zv db01 3306                    # can I reach this port?
Test-NetConnection db01 -Port 3306  # PowerShell equivalent

A timeout usually means a firewall is dropping traffic; “connection refused” means the host answered but nothing is listening on that port.

Does the application respond properly? curl -v https://app.example.com shows DNS, TCP connection, TLS and HTTP stages, and curl -w "%{time_connect} %{time_total}\n" -o /dev/null -s URL times them.

What is actually on the wire? tcpdump -i eth0 host 10.0.0.25 and port 443, or Wireshark, shows packets directly — retransmissions, resets and handshakes that never complete. Useful when the other tools disagree.

Other useful checks: arp -a or ip neigh for duplicate IP issues, ethtool eth0 for link speed and duplex mismatches, and iperf3 to measure bandwidth between two hosts.

Note: Always test from the right place — the user’s machine, the application server and the database server can each see a different network. Many “network problems” disappear once you run the same test from both ends.

A disaster recovery (DR) plan describes how IT services are restored after a major disruption — a data-centre fire, flood, regional cloud outage, ransomware or a prolonged power failure. It is part of the wider business continuity plan (BCP), which also covers people, premises and manual workarounds.

Key contents of a DR plan:

• Business impact analysis — which services matter most, and the agreed RTO and RPO for each, grouped into recovery tiers.
• Scope and dependencies — a dependency map, because the ERP is useless without AD, DNS, the database and network links. Recovery order follows these dependencies.
• Invocation criteria — who can declare a disaster and on what basis, since failing over is itself risky and costly.
• Roles and contacts — DR coordinator, technical teams, vendors and communications, with contact details stored outside the systems that may be down.
• Step-by-step recovery runbooks per service, including DNS changes, failover commands and validation checks.
• Failback procedure — returning to the primary site, which is often harder than failing over.
• Testing schedule and results.

DR site options:

Cloud patterns map onto the same idea: backup and restore (cold), pilot light (core database replicated, rest built on demand), warm standby (scaled-down copy running) and multi-site active-active (hot).

High availability versus DR: HA protects against component failure within a site — clustering, load balancing, redundant power — with automatic, near-instant failover. DR protects against losing the whole site or data set. You need both, and replication alone is not DR, because corruption and ransomware replicate too.

Testing: tabletop walk-throughs, partial failover of individual services, and a full DR drill at least annually.

Note: The most common DR failures are not technical: out-of-date runbooks, missing credentials at the DR site, and no one having practised. A plan that has never been tested should be treated as a hypothesis, not a capability.

In a VMware environment, ESXi hosts run the virtual machines and vCenter Server manages them centrally. Hosts are grouped into clusters with shared storage, which enables three core features.

vMotion — live migration

• Moves a running VM from one host to another with no downtime; users typically notice nothing.
• Memory is copied iteratively while the VM runs, then a final brief switchover moves execution.
• Needs shared storage (or shared-nothing vMotion, which moves the disks too), compatible CPUs — EVC mode masks CPU feature differences — and a dedicated vMotion network.
• Storage vMotion moves a running VM’s disks between datastores.
• Use case: planned maintenance — evacuate a host with maintenance mode, patch it and bring it back.

vSphere HA — High Availability

• Protects against host failure. If a host crashes, HA restarts its VMs on surviving hosts.
• This is a restart, not a live move — the VMs experience an outage of a few minutes while they reboot.
• Hosts are monitored by network heartbeats and datastore heartbeats; admission control reserves spare capacity so there is room to restart VMs (for example, tolerate one host failure).
• VM monitoring can also restart a guest OS that has hung.
• Fault Tolerance (FT) goes further, keeping a live shadow VM on another host for zero downtime, at a significant resource cost.

DRS — Distributed Resource Scheduler

• Balances load across the cluster by using vMotion to move VMs away from overloaded hosts, and chooses the best host when VMs power on.
• Automation levels: manual, partially automated or fully automated.
• Affinity and anti-affinity rules keep VMs together or apart — for example, never run both domain controllers on the same host.
• Works with maintenance mode to evacuate hosts automatically.

Note: Equivalent features exist elsewhere — Hyper-V has Live Migration and Failover Clustering, and Proxmox and KVM have live migration and HA groups — so explain the concepts, not just VMware product names. None of them replaces backups: HA restarts a corrupted VM just as happily as a healthy one.

Ansible is an open-source automation and configuration management tool. It is agentless — it connects to Linux hosts over SSH and to Windows hosts over WinRM, so nothing needs installing on the managed machines apart from Python on Linux. Configuration is written in human-readable YAML.

Core concepts:

• Control node — where Ansible runs, such as your workstation or a CI runner.
• Inventory — the list of managed hosts, organised into groups, either static (INI or YAML) or dynamic (generated from AWS, Azure or VMware).
• Modules — units of work: dnf or apt, copy, template, service, user, lineinfile, win_updates and hundreds more.
• Playbooks — YAML files mapping groups of hosts to ordered tasks.
• Roles — reusable, structured bundles of tasks, templates, variables and handlers; shared through Ansible Galaxy.
• Handlers — tasks that run only when notified by a change, such as restarting nginx only if its configuration changed.

# inventory.ini
[web]
web01 ansible_host=10.0.1.11
web02 ansible_host=10.0.1.12

# site.yml
- hosts: web
  become: true
  tasks:
    - name: Install nginx
      ansible.builtin.package: { name: nginx, state: present }
    - name: Deploy config
      ansible.builtin.template: { src: nginx.conf.j2, dest: /etc/nginx/nginx.conf }
      notify: restart nginx
  handlers:
    - name: restart nginx
      ansible.builtin.service: { name: nginx, state: restarted }

ansible web -i inventory.ini -m ping                         # ad hoc connectivity test
ansible-playbook -i inventory.ini site.yml --check --diff    # dry run showing changes
ansible-playbook -i inventory.ini site.yml --limit web01

Idempotency — the most important concept. Tasks describe a desired state (“nginx is present”), not an action (“install nginx”). Running a playbook once or ten times gives the same result; on the second run it reports ok rather than changed. That makes it safe to re-run and useful for detecting configuration drift. Raw shell and command tasks break idempotency unless you add creates or changed_when conditions.

Secrets are protected with Ansible Vault: ansible-vault encrypt group_vars/prod/vault.yml.

Note: Position Ansible correctly against other tools: Terraform provisions infrastructure (create the VM), Ansible configures it (install and configure software), and Puppet or Chef are agent-based alternatives for continuous configuration enforcement.

A quick script that works once on your laptop is very different from one that runs unattended every night across fifty servers. Production scripts should fail loudly, be safe to re-run and leave an audit trail.

Fail fast and predictably:

#!/usr/bin/env bash
set -euo pipefail          # exit on error, on unset variables, and on failures inside pipes
IFS=$'\n\t'
trap 'echo "ERROR at line $LINENO" >&2' ERR

In PowerShell, use $ErrorActionPreference = 'Stop' with try/catch blocks, and Set-StrictMode -Version Latest.

Key practices:

• Idempotency — check state before acting, so a second run does no harm: create a directory only if missing (mkdir -p), add a user only if absent, append a config line only if not already present.
• Dry-run mode — a --dry-run flag, or -WhatIf in PowerShell, that prints what would change.
• Validate inputs — check arguments, confirm that paths exist and refuse dangerous values. A classic disaster is rm -rf "$DIR/"* when $DIR is empty; set -u and explicit checks prevent it.
• Quote variables — "$file", not $file, so spaces in names do not split arguments. Run ShellCheck on every Bash script.
• Logging — timestamped start, end and key steps, sent to a file or syslog with logger, and meaningful exit codes that monitoring can act on.
• Locking — use flock so overlapping cron runs do not collide.
• Timeouts — wrap network calls with timeout 30 or curl’s --max-time so a hung dependency does not hang the job forever.
• No hard-coded secrets — read from a vault, environment variables or a root-only file, never from the script in Git.
• Least privilege — run as a service account, and use sudo only for the specific commands that need it.
• Clean up — use mktemp for temporary files and a trap on EXIT to remove them.

Process around the script: keep it in version control, get a peer review, test it on a non-production host, document its usage in a header comment, and alert when it fails — or when it does not run at all.

Knowing when to stop scripting: once a script grows past a couple of hundred lines or manages the state of many hosts, a configuration management tool such as Ansible or a proper programming language such as Python is usually more maintainable.

Note: Interviewers often ask you to spot the bug in a short script. Unquoted variables, missing error handling and relative paths that break under cron account for most of the answers.

Capacity planning ensures there is enough infrastructure to meet demand at acceptable performance — without either running out during a peak or wasting money on idle hardware.

1. Measure the baseline. Collect at least several weeks, ideally a year, of utilisation data from monitoring such as Prometheus, Zabbix or CloudWatch:

• CPU — utilisation, run queue and steal time on VMs.
• Memory — used versus available, page cache, swap activity and OOM events.
• Network — bandwidth on interfaces and WAN links, packet drops, connection counts.
• Service metrics — requests per second, latency percentiles and queue depth.

Plan on peaks and percentiles (95th percentile), not averages. An average of 30 per cent CPU can hide a daily peak at 95 per cent.

2. Understand the demand drivers. Link resource usage to business metrics — users, orders, transactions. Then business forecasts (“50 per cent more customers next year”, festive sales, month-end processing, a new branch opening) translate into infrastructure numbers.

3. Forecast and set headroom. Trend the growth — linear regression in a spreadsheet is often enough — and work out when each resource crosses a threshold. Typical targets: sustained CPU below 70 per cent, memory below 80 per cent, and links below 70 per cent at peak. Headroom absorbs spikes and the loss of a node (N+1). Account for lead time: procurement of physical hardware can take weeks.

4. Validate with load testing using tools such as JMeter, k6 or Locust, to find the real breaking point and the resource that saturates first.

Scale up (vertical) versus scale out (horizontal):

Note: In the cloud, capacity planning shifts toward auto-scaling policies, right-sizing over-provisioned instances and cost management (FinOps) — but you still need to plan for quotas, database limits and reserved capacity. Review capacity at least quarterly, not only when something breaks.

A CMDB (Configuration Management Database) is a repository of the components that make up IT services, called configuration items (CIs), together with their attributes and — crucially — the relationships between them.

Examples of CIs: servers, VMs, network devices, databases, applications, business services, cloud resources, software licences, certificates and even key documents or contracts. Typical attributes include owner, environment, location, OS version, IP address, support group and criticality.

Relationships are what make a CMDB valuable:

Business service: Online Payments
  └─ depends on ─ Application: payment-api
        ├─ runs on ─ VM: pay-app-01, pay-app-02
        │     └─ hosted on ─ ESXi host: esx-07
        └─ uses ─ Database: pay-db-cluster
              └─ stored on ─ SAN: san-02

Why it matters in daily operations:

• Incident management — when esx-07 fails, you know immediately that Online Payments is affected and who to notify, and the service desk can link fifty tickets to one cause.
• Change management — impact analysis: before patching san-02, you can see every service that depends on it and schedule accordingly. The CAB relies on this.
• Problem management — spotting that recurring incidents share a CI, such as one firmware version.
• Security and vulnerability management — “which servers run the vulnerable OpenSSL version?” becomes a query instead of a week of emails.
• Audit, licensing and compliance — knowing what you own and what is out of support.
• DR planning — recovery order follows the dependency map.

Keeping it accurate — the hard part:

• Automated discovery — ServiceNow Discovery, agents, or cloud APIs populating CIs, rather than manual spreadsheets.
• Tie updates to change management — a change is not closed until affected CIs are updated.
• Regular reconciliation and audits; flag CIs with no owner or no recent discovery.
• Scope sensibly — model what you actually use for decisions instead of trying to record everything.

Common tools: ServiceNow CMDB, BMC Helix, Jira Service Management Assets, and open-source options such as iTop and GLPI.

Note: An inaccurate CMDB is worse than none, because people trust it and make wrong decisions. Interviewers value candidates who stress data quality and automation over the tool itself. Also note the difference from an asset register, which tracks financial ownership rather than service relationships.

Knowing the boot sequence tells you where a failure is happening, which determines the fix.

The boot stages:

1. Firmware (BIOS or UEFI) — runs the power-on self-test, initialises hardware and finds a boot device. UEFI reads the EFI System Partition; legacy BIOS reads the MBR.
2. Boot loader (GRUB2) — shows the boot menu and loads the selected kernel and initramfs into memory, passing kernel parameters. Configuration is generated into /boot/grub2/grub.cfg or /boot/grub/grub.cfg.
3. Kernel — initialises CPU, memory and drivers, then mounts the initramfs as a temporary root.
4. initramfs — a small early userspace with the drivers and tools needed to find the real root filesystem: storage drivers, LVM, RAID, encryption. It then switches to the real root.
5. systemd (PID 1) — mounts filesystems from /etc/fstab, starts services in dependency order and reaches the default target — multi-user.target for servers, graphical.target for desktops.

Diagnosing by symptom:

• No boot device found — firmware boot order, a failed disk or controller, or a missing EFI entry.
• GRUB rescue prompt — the boot loader cannot find its files; reinstall GRUB from rescue media.
• Kernel panic or “unable to mount root” — often a bad kernel update or an initramfs missing drivers. Choose the previous kernel in the GRUB menu, then rebuild with dracut -f (RHEL) or update-initramfs -u (Debian and Ubuntu).
• Drops to emergency mode — very commonly a bad /etc/fstab entry, such as a removed disk or a typo’d UUID. Log in, run journalctl -xb, fix fstab (use blkid for UUIDs; add nofail for non-essential mounts), then test with mount -a before rebooting.
• Filesystem errors — run fsck or xfs_repair on the unmounted filesystem.
• Forgotten root password — edit the kernel line in GRUB and add rd.break (RHEL) or init=/bin/bash, remount root read-write and reset it. On SELinux systems, touch /.autorelabel afterwards.

Access when the system is not reachable over SSH: the hypervisor console, iLO, iDRAC or IPMI for physical servers, or the serial console in the cloud, such as the EC2 Serial Console or Azure Serial Console.

Note: Prevention matters: always run mount -a after editing fstab, keep at least one older kernel installed, and snapshot VMs before kernel or boot-loader changes. Most boot failures in production follow exactly those three kinds of change.

LVM (Logical Volume Manager) adds a flexible layer between physical disks and filesystems. Instead of fixed partitions, you pool storage and carve out volumes that can be resized, extended across disks and snapshotted.

The three layers:

• Physical Volume (PV) — a disk or partition initialised for LVM, such as /dev/sdb.
• Volume Group (VG) — a pool made from one or more PVs, such as vg_data.
• Logical Volume (LV) — a slice of the VG that holds a filesystem, such as /dev/vg_data/lv_app, mounted at /app.

Inspecting the current layout:

lsblk
pvs ; vgs ; lvs            # summaries; vgs shows VFree (free space in the pool)
df -hT /app                # filesystem type matters for the resize command

Case 1: the volume group already has free space.

lvextend -r -L +20G /dev/vg_data/lv_app     # -r also grows the filesystem
lvextend -r -l +100%FREE /dev/vg_data/lv_app  # or use all the free space

Case 2: add a new disk first — for example, a new virtual disk from the hypervisor:

for h in /sys/class/scsi_host/host*; do echo "- - -" > "$h/scan"; done   # detect the new disk; confirm with lsblk
pvcreate /dev/sdc
vgextend vg_data /dev/sdc
lvextend -r -L +50G /dev/vg_data/lv_app

Case 3: the existing virtual disk was enlarged — make the kernel see the new size with echo 1 > /sys/class/block/sda/device/rescan, grow the partition with growpart /dev/sda 3 if there is one, then pvresize /dev/sda3, then lvextend as above.

If you did not use -r, grow the filesystem separately: xfs_growfs /app for XFS (it takes the mount point) or resize2fs /dev/vg_data/lv_app for ext4. Both work online, while the filesystem is mounted and in use.

Important caveats:

• XFS cannot be shrunk — only grown. ext4 can be shrunk, but only while unmounted, and it carries real risk. So extend in sensible increments rather than over-allocating.
• Take a backup or snapshot before any storage change, and record it as a change.
• LVM snapshots (lvcreate -s) are useful for short-term rollback before upgrades, but they degrade write performance and fail if they fill up — they are not backups.

Note: In cloud VMs the same pattern applies: enlarge the EBS or managed disk in the console, then growpart, pvresize and lvextend -r inside the OS. Being able to recite that sequence confidently is a very common practical interview test.

Expired certificates cause sudden, total outages — browsers show security errors, API clients refuse to connect and integrations fail — and they are entirely preventable. They still catch out major companies because certificates are scattered across load balancers, web servers, appliances and applications, with renewal dates a year or more apart.

1. Build an inventory. You cannot renew what you do not know exists.

• Scan networks and known endpoints for certificates on ports such as 443, 8443, 636 (LDAPS) and 993.
• Pull certificates from load balancers, CDNs, cloud certificate managers and Kubernetes secrets.
• Record the common name and SANs, issuer, expiry, where each is installed, and the owner.
• Use Certificate Transparency logs (for example crt.sh) to discover public certificates issued for your domains.

2. Monitor expiry and alert early.

echo | openssl s_client -connect app.example.com:443 -servername app.example.com 2>/dev/null \
  | openssl x509 -noout -subject -enddate

openssl x509 -in cert.pem -noout -checkend 2592000 && echo "valid for 30+ days"

Add expiry checks to monitoring — Prometheus blackbox_exporter exposes the expiry as a metric, and Zabbix, Nagios and Uptime Kuma all have certificate checks. Alert at 30, 14 and 7 days, with escalation, routed to a team queue rather than one person’s inbox.

3. Automate renewal.

• ACME with Let’s Encrypt or another ACME-capable CA — certbot, acme.sh, or cert-manager in Kubernetes — renews automatically, typically 30 days before expiry.
• Use cloud-managed certificates (AWS Certificate Manager, Azure Key Vault) that renew and deploy themselves where possible.
• Use an internal PKI with auto-enrolment for internal services, such as AD Certificate Services with Group Policy auto-enrolment.

4. Validate the deployment. Renewal is not complete until the new certificate is served. Reload services after renewal, confirm the full chain including intermediates is installed, and check from outside.

5. Govern it. Each certificate needs an owner. Include certificates in the CMDB, and treat an expiry incident as a postmortem-worthy failure of process.

Note: Maximum certificate lifetimes for public certificates are being cut in stages, heading towards 47 days by 2029, which makes manual renewal impractical. Saying that automation is now mandatory rather than optional shows you are up to date.

Clock drift sounds trivial but causes some of the most confusing failures in an environment, because the symptoms appear far from the cause.

What breaks when clocks are wrong:

• Kerberos and Active Directory — by default Kerberos rejects authentication if the client and DC clocks differ by more than five minutes. Users cannot log in and domain trust errors appear.
• TLS — a clock set far ahead or behind makes valid certificates look expired or not yet valid.
• Log correlation and forensics — incident timelines become unreliable when servers disagree by minutes; security investigations and audits depend on accurate timestamps.
• Distributed systems — database replication, clusters, one-time passwords (TOTP), scheduled jobs and backups all assume consistent time.

How NTP works: the Network Time Protocol (UDP port 123) synchronises clocks against a hierarchy of strata — stratum 0 are reference clocks such as GPS or atomic clocks, stratum 1 servers are attached directly to them, and so on. Clients gradually slew the clock rather than jumping it, to avoid disrupting applications.

Recommended design: two or three internal NTP servers sync to reliable external sources (for example in.pool.ntp.org or a cloud provider’s time service), and every other system syncs to the internal ones. In Active Directory, the PDC Emulator of the forest root domain should sync externally, and all other domain members follow the domain hierarchy automatically.

Linux — chrony (the modern default):

# /etc/chrony.conf
server ntp1.corp.local iburst
server ntp2.corp.local iburst

chronyc tracking       # current offset, stratum, reference source
chronyc sources -v     # configured sources and their status
timedatectl            # system clock, time zone, sync status

Windows:

w32tm /query /status
w32tm /query /source
w32tm /config /manualpeerlist:"time.google.com,0x8" /syncfromflags:manual /update   # on the PDC Emulator
w32tm /resync

Troubleshooting checklist: is UDP 123 allowed through the firewall, are the sources reachable, is the time zone correct (a correct UTC time with the wrong zone looks like drift), and on VMs, is the hypervisor’s time sync fighting with NTP inside the guest?

Note: Standardise servers on UTC and convert only for display. Mixed time zones across servers make log correlation during an incident painful, particularly for teams working across India and other regions.

Tiered support routes each issue to the lowest level that can resolve it, so experts are not consumed by routine tickets and users get fast answers to common problems.

Making escalation work well:

• Clear escalation criteria — for example, escalate if unresolved after a set number of minutes, if it needs admin rights, or if more than a few users are affected.
• Good handover notes — what was checked, what was tried and the error messages, so the next tier does not repeat everything or call the user again.
• Functional versus hierarchical escalation — functional moves the ticket to a team with more expertise; hierarchical informs management because of impact or an SLA risk.
• Measure first contact resolution (FCR), escalation rate, reassignment count and customer satisfaction.

Shift-left means moving resolution towards the lower tiers and earlier in the process, where it is faster and cheaper — a password reset handled by self-service costs a fraction of one handled by an engineer.

• L2 and L3 write knowledge articles for issues they solve, so L1 can handle them next time — the practice called Knowledge-Centred Service (KCS).
• Give L1 safe, scripted tools and limited permissions, such as unlocking accounts or restarting print spoolers.
• Automate common requests through the self-service portal.
• Review escalated tickets regularly for patterns that could be shifted left.
• In development, shift-left also means testing and security earlier in the lifecycle.

Note: In interviews for L1 or L2 roles, show that you know when to escalate as well as how to troubleshoot. Holding on to a ticket too long to avoid escalating hurts the SLA more than a well-documented early escalation.