Story · Manufacturing · Out of hours
Their server died at half past eleven at night. Nobody at the company noticed.
The main production server failed at about half past eleven at night, well outside on-call hours. Two people saw the alert anyway. One of them admits he was up too late playing video games.
The situation
A production server going down overnight at a manufacturer is not a quiet failure. It is a morning where nothing runs, orders do not move and a shift stands around.
The client had their own internal IT person, and he was asleep, which is where a reasonable person is at half past eleven at night.
There was a disaster recovery server sitting ready, which existed because somebody had argued for it during a budget conversation some time earlier.
What we did
The two of them brought the disaster recovery server up together, that night, without being asked and without anybody escalating it to them.
It ran a little slower than the primary, which is what a disaster recovery host is supposed to do. It ran.
The following morning the client's IT person woke up to a business that was already working and did not know anything had happened until he was told. The hardware replacement got coordinated the next day, in daylight, at a sensible pace, because the emergency had already been handled.
"It was outside our business hours. We still get notified, and we still have to look, because we care what happens at eight the next morning."
How the engineering team puts itWhat changed
ResolvedThe business opened on time and the failure cost them nothing except a scheduled hardware swap.
Worth being honest about what this story shows. It was not protocol that saved that morning. It was two people choosing to look at an alert when nobody would have known if they had not. Everybody here is empowered to do the right thing, and occasionally the right thing happens at midnight.
Order the internet circuit the week the lease is signed. Carriers take six to twelve weeks. Nothing else on a move fails as often.
The relocation checklist Did you knowA backup that has never been restored is a hope, not a plan. Ask for the date of the last tested restore.
What to actually testQuestions people ask about this
What is a disaster recovery server actually for?
Running the business at reduced speed while the real one is fixed. It is the difference between a slow day and a lost week, and it only helps if somebody has tested bringing it up.
Do you monitor outside business hours?
The monitoring never stops. The emergency line reaches an on-call technician. And, as this story shows, the alerts go to people who tend to look anyway.
How often does a server just die?
Rarely, and almost always with warning signs beforehand that monitoring picks up. This one was a component failure, which is the kind that does not announce itself.
"Prompt call return. Tyler saved the day! Even managed to calm my nerves after the panic of potential data loss. You guys are awesome!"
Client, 2024 surveyMeet Tyler- A technician who knows your setup takes every request. Not a dispatcher, not a queue.
- Every closed ticket is surveyed and a partner reads every response.
- 99.5% of surveys come back positive.
- Family-owned since 2005. 21 years in the same corner of Wisconsin, not going anywhere.
"Thanks for the hard work. I was kept informed with regular updates until the concern was resolved."
Jeff, insurance brokerageWhat happens at your business at midnight?
If the answer is nobody knows, that is the thing worth fixing first.
Talk to a partnerTalk to a partner, not a sales script.
Tell us what is going on. A partner will call you back the same business day, look at what you have, and tell you honestly whether we are the right fit. No pricing pressure, no pitch deck.