It’s the late 90s and I was working in IT for a small (200 employees) company.
One of my tasks was performing daily data backups, which ran for about four hours and started after everyone else had left for the day.
We had an intercom system in the building, where if you picked up your phone and dialed a specific extension, you could page someone, or make announcements so everyone could hear.
One night I was in the process of running Ethernet cable from switches on the second floor down to individual terminals on the first floor, so there was a lot walking around the building.
I like to have music when I’m working, but I didn’t have an MP3 player at the time, so I thought… “I have music on my PC and no one is around. Why don’t I just…”, and I went around and cranked the volume knob on every ceiling intercom speaker in the whole building. Then I dialed into the intercom from the phone in my office, put the handset up as close as I could to my PC speakers, and boom, music throughout the building.
Cut to a few hours later, it’s late, I’m tired, the backup is finished, so I close down the system, hit the lights, lock the doors, and go home.
The next day, I come in, not thinking about anything from the night before, and get to work. About 11:00am, the owner’s secretary got on the intercom to page someone… and blew everyone out of their seats. There was an almost collective, “What the fuck!!!” from everyone, and an “Ohhh, shiiit” from me, because of course I immediately knew what happened.
I ran around like a madman, turning the volume down on all of the speakers that I had forgotten the night before, while saying, “No idea… Yeah it’s crazy… No no, it’s fixed now…”
No one ever found out exactly why the volume was so loud that morning.
Accidental reply-to-all set on mailman list, of 8,000 users.
Only minor at about 16 minutes and 11 replies shared.
Still had a little bit of cold-sweats, and dolly-zoomed my surroundings.
I don’t have any non-destructive ones. I have plenty of destructive ones if you are fascinated, however.
I am pre-fascinated
So I work IT and I am damn good at it, relatively speaking.
Recently, I had a caller contact me about an issue they were having with their laptop of it not allowing them to use some tool or something? I forget the specifics. But I checked the laptop and saw it had an uptime of like a month and a half. So I thought, oh just give it a reboot.
I sent a command “RESET COMPUTER” and as it turns out that was a fucking factory reset! So she then lost absolutely everything off her laptop. I got off with no real issue as all my boss did was give me a stern talking to, but no write up or pay dock or demotion or whatever, but holy shit that coworker? She was pissed at me.
I made it up to her by taking her out for drinks and letting her spend all my money, sobbing.
Not your problem. She could have lost everything to an unfortunate drop of her laptop on her way to a meeting room five minutes later anyways. There should be backups of the device. Or all work needs to be saved/synced to a network drive regularly and it should be made clear to everyone that one doesn’t save important data locally.
It was a surprise test of her business continuation planning.
That’s also on the IT department for not implementing a backup solution for users (or upper management not listening to IT that a solution is needed).
If a work laptop dies and the work files are lost, that sounds like a work problem.
I used to work at a big cable company whose name rhymes with bombast. One day I (M) found myself in a meeting with a bunch of female vice-presidents (Bombast has thousands of vice-presidents). The presenter couldn’t get her laptop to connect to the big TV so I pulled out one of my gadgets and got her set up. She said “wow, you’re pretty handy to have around.” My brain decided that it would be a clever thing indeed to say “well, I’m pretty well-endowed in the dongle department” but when I started to say it my brain realized how completely insane a comment it was. So my brain cut me off and I ended up just saying “well, I’m pretty well-endowed” which was not exactly any better.
To their credit everybody ignored this except for one woman who actually guffawed briefly. I wasn’t fired immediately so there’s that.
To be fair telling someone “wow, you’re pretty handy to have around” is opening the door for all kinds of inappropriate comments…
not me, someone else.
someone deployed a change to the development AWS account at work before a holiday that caused a lambda to execute over 60 million times in a week.
not bad until you find out that the lambda was responsible for creating EC2 instances.
And then?
not only was the dev summarily dismissed, the monitoring team that was responsible for escalation was sternly talked to by the CFO. evidently there were alarms going off that entire week but nobody did anything about it because “it’s in dev, it’s always noisy.”
they actually worked it out with AWS and was able to get the bill reduced but it was still more than $500k (at least that’s all my boss would let me know).
For me, it’s occasionally making a code update to a website that has evident and immediate visual glitches. But, I always make these updates on a staging platform before pushing them live, so it ends up being a funny thing in the end. I actually have a folder of my “bloopers” that I keep for my amusement.
Not anything destructive (to the Company) per se, but seeing a meeting with a senior and a member of HR suddenly pop up on my calendar after weeks of waiting in limbo to be placed on a new project was definitely an OH SHIT moment.
Yep, I was made redundant 🙃
Not physically destructive but someone wrote a script that was supposed to clean up unused phone extensions and decided to run it in production without testing it. It wiped out all the phones.
We probably spent a week or two fixing and testing thousands of phones and the person who ran the unvetted script was fired, but nothing was physically destroyed.
Your post reminded me of a video I saw once where someone did something like this with the fire alarms throughout the building, except there was no fallout. It was just cool to see the speakers in the alarms getting repurposed to play music.
Fortunately not as serious as other comments. For context, I have autism, and I frequently stim by making random noises.
One day I was at a global online meeting with very important people. I thought I was muted, so I said “MEOW”.
Suddenly everybody goes quiet and somebody says “did anybody say ‘meow’?”
I quickly turned off my microphone, and fortunately nobody mentioned that again.
Got creative at work, i was managing my crews .profiles so we could share aliases and such. I got to clever and did a symlink so we all had the same exact one. Then i tested a change that made logins fail.
Luckly it was just the admins during downtime but we had to have the account team make a temp account lol
Added a bug that would delete data from a form but only when they reopened it and didn’t paginate all the way through to the end of the form. Took ages to discover and a big job to review all the potentially missing data.
Disabled the incorrect password so that it would allow login no matter what password you use. (Original system wasn’t mine and I completely agree with “don’t roll your own security”).
Haha that’s great! I cant think of one off the top of my head but I’m sure I have plenty.
Reading through the comments, I remembered my first day on my second ever job, where we were instructed to turn off computers and power strips too, at the end of the day. When I turned mine off at the end of the day, I found out other power strips were using mine, so I turned off other people’s machines. Unfortunately, the guy that gave us the instructions was first to leave in the evening and hadn’t have his machine turned off that way.
Also remembered that I once broke production server (maybe it didn’t get to production, can’t remember) by adding a single line of log, I didn’t notice that what I was logging was computationally intense, so it halted everything eventually, but it was barely reviewed because it’s just one log line
Had a content filter server I had built. Dansguardian.
Installed some new packages, which I had wgot (joking) at / as root. Fuck, I was a dumbass kid.
After I installed the packages, I created the folder /installs and then: holy shit, mv is naturally recursive?!
mv * /installs
Well, let’s ls to see if the root directory is clean.
ls not found.
Um… ok, this is a call center, seems to still be working as everything is still in RAM, but as it is working as a proxy for content filtering, you could lose the company MILLIONS, so think fast!
I rebuilt another server, swapped it with ninja speed, fixed the other server, and swapped it back with ninja speed.
And everyone asked me if I was ok. “Nothing. I mean, yeah, good.”
edit: mv is recursive by default, in contrast to cp, which requires -R. Scary lesson to learn on the job.
Tooks out search on Wikipedia. I exposed a path that can perform an unconstrained Powerset Construction concurrently on 7 servers. It filled up memory and died. User retried. While cluster OOMing.
ELI-non-IT this “oh shit” moment?
Wikipedia has a search feature. Not many people use it, but Wikipedia is huge, so enough people do. Back then, a couple dozen full text searches a second. And maybe 100 quick prefix searches.
Anyway, one thing I made for it was a way to search the wiki text with regular expressions. It’s useful for maintainers.
Wikitext is the funny markup language Wikipedia uses. If you know html or md it’s like that, but it can kind of “call” other pages. Whatever. Doesn’t matter. It’s text. And people have strong opinions about it.
You can see it by clicking “edit” (sometimes looks like a pencil) and then clicking “edit without logging in” then “source editing”. Those lovely boxes at the top for a person are made with something like
{{Infobox person | name = Douglas Adams | birth_place = [[Cambridge]], England }}Imagine you have 1000 articles that use
birthplaceinstead ofbirth_place. Both work, but it’s driving you and all your editor friends crazy there are two ways to do it. This is believable. Trust me.You want to find every page that has an
{{Infobox personfollowed bybirthplace. Then fix them. The nerds who came before us invented a language to ask that question called “regular expressions”. The one for this looks like\{\{Infobox\wperson.+birthplace. Maybe. Nerds will know regular expressions are bad for this. But nerds will also know that regular expressions being bad has never stopped anyone from using them anyway.One way to run these regular expressions is to convert them into an https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton . You build these in memory and they are not big. But you can’t really run them directly. They are pretty and fairly easy to read once you get used to them. But you can’t easily ask “does this match this text”. At least, not with the tools I had.
I could only run https://en.wikipedia.org/wiki/Deterministic_finite_automaton . It’s deterministic! Much nicer. And you can go from a nondeterministic one to a deterministic one. Easy. Our forenerds solved the problem. You use a https://en.wikipedia.org/wiki/Powerset_construction .
The trouble is, it can make very very very big deterministic finite automata. Like, if there are 3 states in the nfa you can get and 8 state dfa. 4 is 16. 5 is 32. In the worst case. Usually you get much better. But a fairly big but not super frightening regular expression can turn into a big nfa. And the dfa would need more states then there are grains of sand on earth. Too big for computer.
So, user asked for this regular expression. The search servers tried to convert it, and filled up their memory and died. They ran Out Of Memory. OOM. The usual thing is you copy the full memory so you can look later and start the server again.
But that can take time. And sometimes people are confused and haven’t set up the restart to be automatic. And they can’t find you. So search stays broken for a while.
