- cross-posted to:
- fediverse@lemmy.world
- cross-posted to:
- fediverse@lemmy.world
Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn’t use postgres). From what we can tell, the attacker did not do anything with that access, so we don’t believe any user data was accessed, but we can’t be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it’s as smooth as we think it will be, but we can’t be sure!
As stated earlier, our writefreely instance is still up and running as it wasn’t impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don’t plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won’t be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn’t seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
Edit - Piefed is back now!
You must log in or # to comment.
Sounds like a real mess. It must be a lot of work running infrastructure like this, so you should know we appreciate all the work you guys do.
Thank you for bringing my favorite source of memes back online. You two are much appreciated!
o7 Admiring your tenacity, welcome back.
People wishing to manage their lemmy account should use the Lemmy UI (web) frontend.
Thank you so much kaity and ada for everything you do and your moral integrity. I don’t envy any of this.
Thank you both for working so hard to deal with this! I’ve changed my password, hopefully nothing more will come of it. I hope other instances are also on the lookout for this hack.
Thank you 🫂 I’ve been through some disastrous code deployments, but I know those experiences could never truly compare to something like this- stress, fear, accountability, and just feeling violated. You all must have put in sooo much effort and had to make some difficult decisions. Thank you for all of your time and knowledge to creating and supporting this space for us 🩷
The feeling when a small hobby non-profit project gets hacked and the owners quickly respond to the users and say “hey, we got hacked but don’t worry, your passwords are safe because they were encrypted!!”
But a damn multi-billion company gets hacked, takes months to tell the users and their answer is: “so… a few months ago, we got hacked, but it wasn’t that bad so we didn’t think about telling you until someone found our database for sale in a forum. Also, change your passwords, email, physical adress, bank account, credit cards and if you sent it to us, your SSN, because we didn’t think it was important so it was all stored in a plain wordpad file without any encryption”.
I know this must have been awful for you guys, but damn if it feels good to know that even if the fucker got access to your database, they couldn’t do shit because you were competent and took measures to prevent your users in a way a multi-billion company doesn’t.
I get your sentiment, but the difference is the mega-company has to worry about what they say for when they inevitably get sued.
No one is going to sue blahaj, and their currency is trust and communication, so it helps to be open.
No, the difference is that blahaj encrypts user passwords while a multi-billion dollar company stores them in a fucking plaintext file (alongside the credit card numbers and other sensitive data).
Also, under GDPR, a company must inform of a databreach ASAP, and they only do when they get caught.
I’ve been on Lemmy for a while and the sheer naive idealism on here boggles me sometimes. You guys think “megacorp is evil and stoopid”, when realistically the person who discovered the breach freaks out because he may lose his job and is the sole breadwinner of the family, then takes it to his manager, who isn’t in a much better position. It bounces through a few low level admins and teams trying to figure out the extent, since there are hundreds of applications and systems, all while the clock is ticking by what you want to see. Only once they have things does it get bubbled up because they want to make sure they can answer to the VP in hopefully a way they won’t be scapegoated and lose their jobs. Finally it may get to the people who do have control of information but by now it’s been a while and past your idealistic timeframe. All because a common working man doesn’t want to become homeless.
But sure, blame the corporation, because it’s a single monolithic entity that you can focus your ire on instead of showing some sympathy for the working people, who are also here on Lemmy, fearing for their jobs.
he may lose his job and is the sole breadwinner of the family
Corporations are responsible for him being in that position. Hope that clears it up for you.
Uh, what entity do you think it’s responsible for the environment you just described?
Thanks for sharing all the technical details!
Did you have ssh keys configured between your machines, or is all of blahaj on a single server? (Wondering how they got from postgres -> root -> other servers)
We run our instances across multiple servers, but the postgres databases are all hosted together on a single server, though technically not a single server, as, at the time of the attack, we also hosted a backup database server, which was spec’d to backup our instances, but not serve them. Their access was limited to the main postgres server, but that server holds the databases for all of our instances. It looks like the script they used in the postgres exploit to give them local access interfered with the cleanup/backup process, so WOL files would get written, but not deleted, which filled up the disk on the main machine, and ultimately, caused it to fail over to the backup machine.
In theory, they could have used the same script/exploit on the backup machine, but because it wasn’t spec’d to serve all of our instances at once, everything fell over at this point. That is what alerted us to the issue, and also limited the attackers available time in the system.
Thank you all for everything you’re doing to keep users safe and the servers functional ~
Oof! I’m really sorry that happened, to our blaj kindred. Hopefully everyone and everything successfully mitigates damage and restored to the fullest extent.
I’ve waded through my share of critical incidents and systems recoveries. The work can be deeply stressful and infuriating as you gradually uncover inevitable missteps, find the footprints of malicious actors and dream up countless hindsight mitigations that would have prevented all this.
Bless you, kind friends. I know how hard this is. Your work and diligence has value, and this entire community appreciates it.
Getting hacked is never an if, it’s a when.
So sorry that you’ve had to shoulder all this. I really do hope you took breaks and didn’t overwhelm yourselves. I understand remediating the hack itself quickly was important, but I hope you took a break and got good rest before you brought everything back online. Even in such a serious situation, I want to know my admins are still caring for themselves, too. It’s hard to do this stuff on such a small scale when we have literal nation state actors doing hacks, it’s a literal 24/7 threat.
Anyway, please be kind to yourselves. Thanks so much for all the hard work and bringing a beautiful community together.
This is a reminder from someone in IT with an interest in security, use discrete, unique passwords for accounts you are concerned about. Finance, health, banking, etc. use different passwords. For places you don’t care about use a throw away only for those sites that don’t have PII or HIPPA.
This, very much this!
Use a password manager. Keepass if you’re paranoid, Bitwarden if you’re slightly less paranoid.
Idk how the built-in OS and browser ones are these days. Browser seems a little loose.
+1 for Keepass. Specifically, KeepassXC.
Browser ones aren’t to be trusted. But yeah a password manager is pretty much a requirement at this point. Just make the master password hard to crack.
Were your services containerized? Just curious. Systems architect here. Find me on LinkedIn. Curious if you need or want a hand. - Opal Wild
Mostly no. Our smaller ones were.
Your passion project is community. You’re doing well and good. Thank you for protecting me and protecting us. If you need, I’m free as in beer and sex. 💖🫶
Slightly off topic but what is the exact meaning of the ‘free as in sex’ phrase? I know ‘free as in speech’ (which is ironic because speech is not that free nowadays) and ‘free as in beer’ but it is first time coming across the sex version.
I suspect it is a reference to something Linus Torvalds said, something asking the lines of “software is like sex; it is better when it’s free”.
I’m a whore.
If that were true, the sex would most definitely not be free.
Fuuuuck.
Glad everything could be straightened out, but dog damnit that sounds like a shitload of work just because someone decided to be an asshole >:-(
