What's a guy to blog about on Haloween when it's a programming blog?
War stories, what else?
That Time They Thought I Was An Intruder
First day at a new company several years ago. I was a twenty-something upstart in a hoodie with a backpack on, and walked through the doors.
I started wandering around, looking through each cubicle hallway, seeing what was going on. Oddly, there wasn't really any onboarding. No one to come and usher me in, give me something to read. I just started wandering around.
That is, until someone stopped me and asked who I was. This person was trained to look for people in the office who didn't belong, to make sure an intruder didn't come into the office and steal intellectual property. I explained who I was and they relaxed, smiling and introducing themselves. Funny in my head, even if in no one else's, because:
- Someone should have been there to show me around and introduce me, for security reasons if not for politeness' sake.
- To them, I was this rando who was out to steal their data. Me? Scruffy upstart? Oh, well.
That Time My Department Took A Whole School Down
At my first technical, real, sys-admin student job, I had a boss who was given 4 VLANs by his superiors. He was told that's all he was allowed to use.
He didn't like this. So he made his own VLANs in vSphere (heavy use of VMware at the time in the department) and used a network bridge to bridge them. VMware networking had a bug in it that would broadcast packets the wrong way over layer two if the networks were bridged. This caused a broadcast storm that took down the School of Business on campus at the time.
After that, he set me on a project that used ProxyARPing to make a transparent router so that he could have his VLANs without alerting the office of IT or causing a broadcast storm.
Scary Stuff.
That Time I Made a Very Expensive Mistake
This is the scariest one.
I like to commit my dot files to a public repository. During our tale, though, I committed my AWS secret and access keys to a public git repository. GitGuardian had my back, told me about it, but I foolishly took them out by making a new commit with the credentials removed, rather than removing all credentials from all commits and cycling my credentials.
A hacker waited until the weekend with my credentials. Then they spun up $180k of VMs in AWS that all mined Monero coin all weekend and into Monday morning. When it was found out what happened, I was honest and said it was my key.
An investigation was launched. It was ruled I was reasonable for having a dotfiles repository, but that I did the wrong thing. They didn't fire me, though, because I was honest. It would have sent the wrong message. If they did, other engineers would see that hiding their mistakes is better than bringing up problems when they happened. Instead they formally reprimanded me, saying if it happened again I would have to be let go.
I was so scared after this. I closed my personal AWS account and took down my dotfiles. If it were my own personal AWS key, I would have gone bankrupt and been fighting to save my house. I have a wife and kids. This stuff was super freaky. It was a long time before I was able to make myself re-open a personal AWS account. Digital Ocean is nice, they let you pre-pay and not give them a credit card (at least at the time).
Now I am extra careful to put stuff like that to a separate file that is not my ~/.profile, but more importantly, AWS creds aren't written at all. I either use environment variables or SSO. Much nicer, much more breathe-easy, better on my nerves.