First it job I accidentally set all the passwords to the same password on the production database…
The administrator quickly restored a backup from previous night to avoid security issues but it meant clients lost any updates they made in the system that day.
They denied anything happened to the sales and support team and I kept my job.
That…sounds like a tiny mistake compared to whoever made it possible to change all of the passwords in a database in one go.
My biggest ever professional mistake. See I was a factory worker and working nights. That night one thing I had to do was put our product into a fancy oven. It would run for over 12 hours but at x temperature for a few hours, then y temperature for some more hours then z temperature for some more hours etc. I took about $90k worth of product(internal cost, not sell price) put them in the oven, went to the touchpanel pressed a button, did all the paperwork, and went home.
Day shift started at 6am, by 630 I got a phone call from the manager asking what the hell happened last night? I said nothing, everything went smoothly. He said well the oven is over 2x the max allowed temperature for the devices. See on the touch panel there was a button for “auto” that started the pre-programmed temperature cycle the button immediately below that just turned the heat on full blast untill manually turned off, no confirmation, no protection at all. I cost the company over twice my annual salary by pressing the wrong button on a touchpad.
Basically it was determined that it wasn’t my fault and was poorly designed. The touch panel got redesigned and shortly afterwards anything besides auto required entering a password. I wasn’t punished at all but probably took a few years off my life in anxiety and fear.
Yeah having a touchpad controlling any machinery functions is stupid (cars included).
That’s just how the production industry is, nearly every piece of equipment I work with has a touchscreen HMI. That is a very poor design choice tho to have the “auto” button right next to the “full temp fuck you” button, it should have been password protected from the start or at the very least on a different screen
Pretty common for HMIs in a factory, in my experience. Keyboards are dirty and difficult to keep clean.
How clean are the touchscreens on a scale of “factory new” to “toddler grime”?
Very clean. All the touchscreens are IP67 or better and can be wiped, washed, and sanitized very easily and repeatedly with minimal training. Ours are even pressure washed alongside the rest of the equipment during routine cleaning and maintenance.
Keyboards are more difficult to keep clean, have plastic bits that don’t show up on a safety metal detector on the line, and just in general aren’t needed when the only functions the operator really needs are start and stop.
Fair. I’ve just worked with field worker Toughbooks and hardened iPads in the past, and I didn’t really want to touch them lol. Sounds like they’re treated better in a factory
Yeah, factories have set schedules for cleaning and maintenance, and the HMIs just fall right in with everything else. The big thing is that all the modern stuff is computer controlled and usually remotely managed.
So, a worker would either enter an order on a computer and “print” to a line from there, or a central control room manages those jobs, or sometimes there might be a few different barcodes and a scan gun to select an entry from things posted on the wall or a book. Either way, the complexity doesn’t exist at the actual machine.
The worker then goes to the machine’s HMI, validates everything is ready and the order loaded, and hits start. Once done, the machine finishes and a report is logged.
Literally wash and rinse, if needed, and repeat!
Factories that are concerned with cleanliness and contamination concerns don’t allow compute devices to traverse onto the production floor and back, in my experience. You don’t see any traditional desktops or laptops there, nor mobile devices — even personal ones — or even ToughBooks as they’re not allowed. It just comes down to sanitation concerns and risk to the plant and product(s) being produced.
I had forgotten to tighten a joint in a water pipe above an IT room on a power plant and when it started leaking there was a conveniently placed water sensor right under it which triggered some sort of an emergency shutdown. It was obviously a mistake and I was still as green of a plumber as one can get so obviously I was forgiven for it but it was still expensive for the company.
Driving my parents’ GMC suburban home late one rainy night from my job, I took the turn into the driveway too tight. I scraped the side of the truck against the chainlink fence gate post. The hinge pin cut a gash in the side of the truck.
I knew my dad would be pissed. Everyone was asleep already, so there was nothing to do but do to sleep.
When I woke up the next morning and told my mom, she said she knew and she told me not to say anything to anyone and my dad was going to take care of it.
As it turns out, they were using me as an example of a good driver to shame my brother and my sister, and they didn’t want them throwing it back in their faces.
Something similar happened to me with a neighbor’s car. I turned too wide into my parking space and ended up bumping the back of my neighbor’s van, leaving a noticeable mark. I immediately went up to their door and knocked to apologize, but no one answered AFAIR. I decided if it was a huge problem, they’d come to me or my dad about it, but they never did. So technically I wasn’t forgiven, but I wasn’t reprimanded either.
I once made a sick trick on my scooter completely destroying the new Halloween decoration made out of porcelain my put up that morning. I was in tears when I told her holding what’s was left in my hands, and braced for the worst. But she told me that its fine and these things are replaceable. I was really confused and didn’t know what to make from it. Deep down I knew I deserved some kind of punishment. To this day I apologise for that.
I know its not something “major” but it for me at the time. Its relative!
You owned up to it immediately and confessed to it even though it brought you to tears
If I was your mom I’d be proud of you
I’m a system administrator for a relatively big SaaS for a relatively big company.
I was attempting to configure a new inbound email protocol in the TEST environment. I needed to reference how it was configured in PROD so I could copy certain settings into TEST, so I had both up on my screens (you can see where this is going). But first I needed to clear out the old cloned configurations in TEST.
…And not even half a second after I deleted those config files did I realize what I’d done: deleted inbound email configuration on PRODUCTION. In the middle of the day. Killed inbound email for the system for the whole company.
The whole reason I had PROD open was because I didn’t have good documentation of how it was configured (what was cloned into TEST was old and not correct). So now I was seriously fucked. I had to redo the config from scratch live in PROD. Even tried backing out the deletion but that failed.
Had to advise my boss of the fuckup. He was new. Had only been with the company for like 2 weeks. I had to assure him that I am NOT actually a total fuckhead and I usually don’t do stupid shit like this. I felt terrible, because I knew he’d have to give the post-failure analysis review on this to his new bosses, and that would definitely be awkward.
Anyway, he was cool with it. Assured me that he’d done something similar at a previous job. He let me do my thing to fix it, which I did. Inbound email was only down for …well, a long time (3-4 hours), because once I was able to get the config working, I had to re-register all of the inbound email addresses to use it -which had its own sordid story. That took some time.
After the fact, we modified the TEST environment to have a different “theme” (header colors) so it wasn’t identical to PROD on-screen, and I documented the email configuration thoroughly.
I worked in software ops where part of my day was handling incidents and bring anxiety to the room when I came running to a dev team with an incident caused by their recent release. A few times it was caused by someone accidentally running a test scenario in prod. At one point I got enough and said “add a bright orange ‘test’ banner on your UI for dev environments.” Apparently that was a foreign concept to absolutely everybody, and the more I look around the less I see people actually trying to make an effort to distinguish the two to prevent such catastrophic failures. Especially when you’re handling tens of gigawatts of energy on a daily basis.
I wrote some inefficient code by accident, that worked completely fine in test with 1 fake end user (me), but it had a critical flaw that caused it to literally be exponentially inefficient to the amount of users in the customer’s business. So a small 10 employee business would create 10! network requests. We had some businesses with 2000 employees, so it tried to do 2000! network requests… it brought our servers down immediately after I released it, and I had to figure out a way to block the incoming requests, push out the patch, and get customer’s to update without allowing them to DDoS us again. It took half the day and I was extremely stressed out. Stayed back late a few hours to make sure it was all good. Boss was mad but I didn’t get punished for it
We had some businesses with 2000! employees…
3.316275 × 105735 is probably too many employees.
For context, here's that number written out fully.
331627509245063324117539338057632403828111720810578039457193543706038077905600822400273230859732592255402352941225834109258084817415293796131386633526343688905634058556163940605117252571870647856393544045405243957467037674108722970434684158343752431580877533645127487995436859247408032408946561507233250652797655757179671536718689359056112815871601717232657156110004214012420433842573712700175883547796899921283528996665853405579854903657366350133386550401172012152635488038268152152246920995206031564418565480675946497051552288205234899995726450814065536678969532101467622671332026831552205194494461618239275204026529722631502574752048296064750927394165856283531779574482876314596450373991327334177263608852490093506621610144459709412707821313732563831572302019949914958316470942774473870327985549674298608839376326824152478834387469595829257740574539837501585815468136294217949972399813599481016556563876034227312912250384709872909626622461971076605931550201895135583165357871492290916779049702247094611937607785165110684432255905648736266530377384650390788049524600712549402614566072254136302754913671583406097831074945282217490781347709693241556111339828051358600690594619965257310741177081519922564516778571458056602185654760952377463016679422488444485798349801548032620829890965857381751888619376692828279888453584639896594213952984465291092009103710046149449915828588050761867924946385180879874512891408019340074625920057098729578599643650655895612410231018690556060308783629110505601245908998383410799367902052076858669183477906558544700148692656924631933337612428097420067172846361939249698628468719993450393889367270487127172734561700354867477509102955523953547941107421913301356819541091941462766417542161587625262858089801222443890248677182054959415751991701271767571787495861619665931878855141835782092601482071777331735396034304969082070589958701381980813035590160762908388574561288217698136182483576739218303118414719133986892842344000779246691209766731651433494437473235636572048844478331854941693030124531676232745367879322847473824485092283139952509732505979127031047683601481191102229253372697693823670057565612400290576043852852902937606479533458179666123839605262549107186663869354766108455046198102084050635827676526589492393249519685954171672419329530683673495544004586359838161043059449826627530605423580755894108278880427825951089880635410567917950974017780688782869810219010900148352061688883720250310665922068601483649830532782088263536558043605686781284169217133047141176312175895777122637584753123517230990549829210134687304205898014418063875382664169897704237759406280877253702265426530580862379301422675821187143502918637636340300173251818262076039747369595202642632364145446851113427202150458383851010136941313034856221916631623892632765815355011276307825059969158824533457435437863683173730673296589355199694458236873508830278657700879749889992343555566240682834763784685183844973648873952475103224222110561201295829657191368108693825475764118886879346725191246192151144738836269591643672490071653428228152661247800463922544945170363723627940757784542091048305461656190622174286981602973324046520201992813854882681951007282869701070737500927666487502174775372742351508748246720274170031581122805896178122160747437947510950620938556674581252518376682157712807861499255876132352950422346387878954850885764466136290394127665978044202092281337987115900896264878942413210454925003566670632909441579372986743421470507213588932019580723064781498429522595589012754823971773325722910325760929790733299545056388362640474650245080809469116072632087494143973000704111418595530278827357654819182002449697761111346318195282761590964189790958117338627206088910432945244978535147014112442143055486089639578378347325323595763291438925288393986256273242862775563140463830389168421633113445636309571965978466338551492316196335675355138403425804162919837822266909521770153175338730284610841886554138329171951332117895728541662084823682817932512931237521541926970269703299477643823386483008871530373405666383868294088487730721762268849023084934661194260180272613802108005078215741006054848201347859578102770707780655512772540501674332396066253216415004808772403047611929032210154385353138685538486425570790795341176519571188683739880683895792743749683498142923292196309777090143936843655333359307820181312993455024206044563340578606962471961505603394899523321800434359967256623927196435402872055475012079854331970674797313126813523653744085662263206768837585132782896252333284341812977624697079543436003492343159239674763638912115285406657783646213911247447051255226342701239527018127045491648045932248108858674600952306793175967755581011679940005249806303763141344412269037034987355799916009259248075052485541568266281760815446308305406677412630124441864204108373119093130001154470560277773724378067188899770851056727276781247198832857695844217588895160467868204810010047816462358220838532488134270834079868486632162720208823308727819085378845469131556021728873121907393965209260229101477527080930865364979858554010577450279289814603688431821508637246216967872282169347370599286277112447690920902988320166830170273420259765671709863311216349502171264426827119650264054228231759630874475301847194095524263411498469508073390080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
You’re absolutely right — I could have worded that section better, that’s on me. I’ve now edited it for clarity.
Ever taken the internet offline to a 70 story residual building? I did because when I plugged my laptop in to troubleshoot something the UPS chucked a massive wobbley and decided to reboot.
I once cut the internet for an entire school by plugging in a router.
They had a bunch of misconfigurations in their spanning tree and loop detection systems, and detected the router as a loop. That triggered the network to reconfigure itself using a different route, which was invalid, so the whole network went down.
Their IT wasn’t gonna be there for a few days, and I didn’t set up their network or had any access to it, so I think it stayed down for almost a week.
chucked a massive wobbley
Gonna start using that at work
I’ve plugged a standard console cable into an APC UPS. did that twice before I realized what was going on. APC decided to do a pinout on their serial platforms that gets triggered with a “Cisco” console cable to shut down. What. The. Fuck.
I’ve done that too!
I once brought my flash drive to show a presentation for my AP English class during the last week of school.
Little did I know my flash drive had a virus on it that propogated through the entire school’s network.
I think even IT came into the room the same day to tell my teacher what had happened.
They did inform me the source was my flash drive.
If I’m being honest, maybe it was a virus/worm from a porn site or something that had been left over and found its way onto my flash drive, or some vulnerable outdated program that had been compromised and breached my flash drive.
Always verify your file integrity, folks!
In an airport’s baggage claim area, I accidentally hit a middle-aged guy with my luggage. There was like around 30 kilograms worth of stuff in there, going directly to his torso, bringing him to writhe in pain. In my defense, the guy was standing too close, and this was an intercontinental flight, so of course everyone’s bringing big-ass luggages. Regardless, I immediately apologized profusely.
Like immediate as soon as the mistake was made or immediate as soon as it was atoned for?
Either is probably fine