Fail of Your Day


#1404

My younger sister has now turned into my mother because I was asked “what is a podcast, how to I make a podcast, how do I subscribe to a podcast.”


#1405

A customer got whacked with ransomware and we need to help save their bacon now.


#1406

As long as they compensate you for your efforts.


#1407

That’s what the support contract with our company is for, I guess.


#1408

The first time I had to deal with ransomware. it was an old friend’s business. It was a super primitive ransomware and the key to decrypt it was inside the client. In practice it meant that one fix file later and the filesystem was decrypted.

This was maybe 2007, double check it’s not that, ended up being so so painless.


#1409

This is possible, but unlikely. Nowadays even the most basic ransomwares tend to encrypt all the storage and will only deliver the decryption keys from a remote server after some kind of bitcoin is transferred or some shit.


#1410

Yeah they’ve gotten better at software. Most of the time the only option these days is restore from backups.


#1411

Backups solve just about everything. Have them.


#1412

They are also really really easy to do these days, provided you can afford like $100 a year

Though if ya wanna do them right by my definition, they’re tricky. I want my backups encrypted when they get to carbonite’s servers.


#1413

When you restore from backups there can be other secondary concerns depending on how far back you have to go. I’ve seen at least one situation where someone blindly went to backups on something… which just left them exposed to the exact same attack again…


#1414

That’s just bad recovery procedures. Here is a summary of a good recovery procedure:

Create new system identical to production with best available backup data. This system should be entirely offline.

Correct all problems with this system that led to failure that required backup recovery.

Run thorough tests against new replacement production system.

Swap new production in place of the old one. Usually by moving IP addresses around. DNS change also ok, but not as good.

When confidence in new production environment is achieved, delete old production system.


#1415

The problem with this ransomware case is that it attacked the backups as well.

I basically work on a NAS that you use to write your backups to (oversimplification here, but it’s good enough for the sake of discussion). Somehow the ransomware found the NAS shares containing the backups and nuked them as well.

On the bright side, because of the nature of how our product works, while the backup files themselves are gone (i.e. they don’t show up in a directory listing), the backup data itself appears to be mostly intact. The trick is to find a way to create new directory listings and connect them to the intact data.


#1416

I’m going to guess someone had them mounted as network disks in Windows. Which is why you don’t leave your backup drive mounted.


#1417

I agree with that assessment. Although, I guess there is still a window where the ransomware attacks the mounted backup drive while you’re doing your backups.


#1418

So apparently the ransomware in question was Ryuk, which is particularly nasty as it’s explicitly designed to not just encrypt your data, but to go after your backups and backup apps too.


#1419

It’s also a hack for a journal that lets you kill people.


#1420

On Facebook, mostly. I really need to make the forum part of my daily routine again.


#1421

So apparently the malware in question was the Ryuk ransomware. Fortunately, due to a quirk on how our product works combined with good timing on the part of customers and support, it appears as if we were able to salvage most of the customer’s data so far.


#1422

I remembered why I don’t like and stopped going to conventions.


#1423

I made it 4 days cold turkey.