openmic.social is an uncensored community. You may encounter strong language, controversial opinions, and mature or NSFW material. You must be 18+ to browse. Illegal content is prohibited and removed on sight — please report it. By continuing, you accept that you may see content you personally disagree with.
I’m a system administrator for a relatively big SaaS for a relatively big company.
I was attempting to configure a new inbound email protocol in the TEST environment. I needed to reference how it was configured in PROD so I could copy certain settings into TEST, so I had both up on my screens (you can see where this is going). But first I needed to clear out the old cloned configurations in TEST.
…And not even half a second after I deleted those config files did I realize what I’d done: deleted inbound email configuration on PRODUCTION. In the middle of the day. Killed inbound email for the system for the whole company.
The whole reason I had PROD open was because I didn’t have good documentation of how it was configured (what was cloned into TEST was old and not correct). So now I was seriously fucked. I had to redo the config from scratch live in PROD. Even tried backing out the deletion but that failed.
Had to advise my boss of the fuckup. He was new. Had only been with the company for like 2 weeks. I had to assure him that I am NOT actually a total fuckhead and I usually don’t do stupid shit like this. I felt terrible, because I knew he’d have to give the post-failure analysis review on this to his new bosses, and that would definitely be awkward.
Anyway, he was cool with it. Assured me that he’d done something similar at a previous job. He let me do my thing to fix it, which I did. Inbound email was only down for …well, a long time (3-4 hours), because once I was able to get the config working, I had to re-register all of the inbound email addresses to use it -which had its own sordid story. That took some time.
After the fact, we modified the TEST environment to have a different “theme” (header colors) so it wasn’t identical to PROD on-screen, and I documented the email configuration thoroughly.
I worked in software ops where part of my day was handling incidents and bring anxiety to the room when I came running to a dev team with an incident caused by their recent release. A few times it was caused by someone accidentally running a test scenario in prod. At one point I got enough and said “add a bright orange ‘test’ banner on your UI for dev environments.” Apparently that was a foreign concept to absolutely everybody, and the more I look around the less I see people actually trying to make an effort to distinguish the two to prevent such catastrophic failures. Especially when you’re handling tens of gigawatts of energy on a daily basis.