10 Lessons Learn from an MSP Buffalo Jump

And we are live. Welcome everybody. Episode 84, cyber Call. We've got Gary Ryan and Wes in the house, special guest Eric Woodard joining us. And shortly, Robert Chaffey, who is sailing the Caribbean seas. Um, I'll, uh, uh, Jack Sparrow. So, um, real quick, Gary, I only have 46 announcements today, so, uh, we'll have five minutes for the call. How does that sound? That sounds awesome. All right, now, so real quick, uh, just one announcement. Actually, we're getting a really good turnout.

I think we're getting up towards 200 for the vulnerability management. Best practice session. Tell your friends, family, clergy, everybody. Um, the call to action is right below, uh, you can click on that green button. Uh, it's gonna have, we, uh, wes, we're gonna have Carl Bmore, who's fantastic at positioning vulnerability management in both assessments and his stack. So where he's gonna pul take it from the sales side. Um, and then Dennis Hanick from Watts sec. Watts sec is a true MSSP.

They only focus on vulnerability management, um, both at the co-managed mid-market and upper, uh, markets as well as they, they work with MSPs, so they, they get it very well. And Dennis has been at vulnerability management for literally north of 20 years. He's gonna take the op side. And Gary, we're gonna talk about people and process, Oh, sorry, I lost my train of thought you were going on there. Alright, so yeah, I think the net, you said click right click, click below. Actually click it.

Was that the upshot of it? Yeah, bud. Alright, so Gary, three, like a little in the, uh, I didn't Promote that one. You know what, Andrew? I might've missed it before, but I wish I, I've been able to promote that to my folks. Okay. Well, I, I can give you the url. Is that your, uh, AC CDC, uh, music going there? My, my ringtone. Sorry. Your Phone. Yeah. Uh, so I put in Gary a two and a half minute snippet from your session with Robert and Eric, which was just phenomenal.

Um, and I put that in there for everybody to see. Um, setting the stage, this is our top 12, or I shouldn't say our Eric's selfless top 12 lessons learned from an rm m com compromise. Um, we're gonna bring, bringing on Robert, who was also involved with Juan momentarily.

Um, but in setting the stage, Gary, I think one of the most important things at the write of boom was this session and how you were able to get into not just the tactical what was happening, but the literal emotional rollercoaster of this. Because you can tabletop something all you want, but when Eric sat there and said, and it's in this video, 50 calls a minute, that's a whole different ballgame of stress.

Um, so there Was a couple of moments, you know, I'd spent some time just looking at the crowd and there was a handful of times when people were just completely locked on. Yeah. Uh, it was a amazing, and, and I wanna say this before I get into some questions, Andrew, Eric, to you and Robert, I feel like that session was a central part of making it feel like a community, uh, at, at, uh, at Right. A boom. And it did feel like a community.

So, um, yeah, I want to thank you for that, but kind of like, that was almost a time when I felt everybody kind of pulling, uh, you know, pulling together. I don't know, Wes, I see you shaking your head. No, Totally agree with that. And, um, I, I love that we're in this stage now in the channel where we, we real, like, we don't just say we're in this together, but we act like it too. I mean, we're not, none of us are perfect at this. Right?

And there, there are some sore thumbs out there, but, um, the fact that we, we, we lean in and we listen when, when folks like Eric share the wisdom that only comes from battlefield knowledge. I mean, they're real veterans, right? And so, um, we're, we're past this stage of five years ago where we hide and ignore and act like it didn't happen. Now we embrace and learn, and that's, that's encouraging to me. I wanna see more of that. Yeah, absolutely. Really good point.

Last thing, I'll just, I'm gonna just make one comment about the poll. Please fill out the poll right now. Uh, Gary, look at this poll. I'm and Ryan, this is horrifying. Yeah. Horrifying guys. But I'm glad everybody's being honest. I, I'm not, I agree about being honest. I'm just like, let's go. We have got to get this going and be honest, because if you're not doing this and haven't done it yet, um, we have got to, it is just telling us we've gotta do more enablement around this. Hands down.

Can we get to chat real quick? I want to ask, I wanna know why the, the nos if you guys will give us a no. Like, is it because you don't know how? Is it because you don't have time? Yeah. Is it because you don't have executive buy-in? Is it something else? If you guys will give us a, a reason why that would be immensely helpful for us. Because if it's, if it's primarily we don't know how, then this, yeah, we're gonna do this again, right? We can help that. Absolutely. Russ.

Um, so fair, fair enough. Um, okay. So please fill that out. Um, yeah. So, um, time, Yeah, it's time, capacity, resources, but that's not really when, when MSP says that when, when a, when an SMB says that or a mid-market enterprise with an IT department, it means they don't have enough resources. Right? When an MSP says it, it means I have not baked this process and roll. I've not baked it into my pricing and I can't afford To do it.

So I have a, I have a personal mantra I live by, and I'm gonna share it here with this charge more what? Charge more? Uh, yeah. And that's your personal mantra, right? I've never seen that. Oh, yeah. I heard that you have a tramp stamp that says charge more. Uh, Gary is what I've heard, but I, I can't authenticate that. Uh, but mine is a little bit different. Um, so, so mine is this, like, when we say we don't have time for something, what we're actually saying is it's not a priority for us.

And that that's okay. I'm not hating on anybody. We have, we have, we all have 24 hours in the day. That's it. It's a universal standard. What we choose to do with those 24 hours is our choice. And so if you say, I don't have time for it, what you're actually saying is it's not been a priority for me. And that's okay. There's only so many things we have priorities for.

But I do want you to go back and think, are there other things that we could prioritize less so that we could do something like that? Just some food for thought. Uh, Eric, to you, as you see 80, almost 90% of MSPs not table topping and RMM, you've lived it. I'd just like your comments. Well, I mean, in full transparency, we didn't tabletop this before it happened to us because Yeah, yeah. No, I get it. I'm just saying I, but you were early, you were one of the earliest out there.

Yeah, We, we got hit before it was cool to get hit. Right. Um, so it scares me. Um, and, and you know, I'll pick on my local competitors. I know, you know, I'm pretty certain that they haven't done that and I'm competing against these guys, right. Um, to your point, Gary, you know, like, it, it, the competition is not fair.

You know, they're claiming these things that they're doing these things, but, you know, I I, from what I see their quotes best practices, they're, I, I would highly, highly doubt that they are. And so, you know, it's, it's tri not if it's when, maybe it's not your RMM, maybe it's some other tool you have, right? Like, you know what AVS have been hit. So it's, it doesn't have to be just RMM. Think of all the freaking agents we have out there across all of our clients. Pick one, it could get hit.

And so, yeah. I mean, if nothing more out of today that if we cut that number in half that this will be a huge success because it's not if it's when, um, and, and we can we'll dive into that further, but yeah, I mean, it, unfortunately, it doesn't surprise me, but it also, um, you know, I hope today's a good wake up call. Yeah. People go through what we went through, or if they do, it's a lot less painful. Very true. So I'm, I'm gonna hand this over to Gary.

Gary, here's the, can you guys see the top 12? Yep. Okay. So for use out there, take, please take your screenshot so I can pull it down and pull in. Robert cfi. So I can do intros. Does everybody, I'm gonna count 5, 4, 3, 2, 1, and everybody got it? Screen it. Good. Okay. So lemme get Robert and Gary handing it over to you. Starting this off. Okay. So I love your first lesson because it literally everything we've been talking about, right?

About, um, about write a boom and about a assumed breach. This kind of speaks to it. Your first lesson is it's gonna happen. Yeah. So, um, to paraphrase John Strand a little bit from the event, um, it's gonna happen. It's gonna suck and you're not going to enjoy it. So this is going from a, a top level, uh, security person. He said that his, he's gonna get hit. And so I do think we have to pull our head outta this sand a little bit and just say it's not if it's when.

And, um, as I kind of tell you some of the things we had in place, um, I think some of these things might give us a little bit of false positive that are a, a false, uh, uh, comfort that, oh, I've got this in place. This can't happen to me when, um, we'll go through our, my list. But it, it, it can happen and, and it most likely will at some point. Yeah, absolutely. Assume breach mentality. Uh, and then we move on to the next thing. Build your incident response team.

So on this one, like, who's on the team from your perspective, and how does this team fit into a tabletop? Sure. Uh, maybe it makes sense, Gary, if it's all right, I'll, I'll kind of go over our incident a little bit, and I think it'll kind of help shape these a little bit, these questions a little bit more Yeah. My answer, if that's okay. Yep. Um, so, uh, as I mentioned, we got hit before it was cool to get hit.

So on February 4th, um, 2019, uh, we had a vulnerability in our, uh, RMM server that our PSA vendor created. And, uh, they're able to come in and bypass two factor, get to SQL unauthenticated, turn off two factor, hop into our RMM, and we deployed ransomware to 1700 endpoints in about a half an hour's time. Okay? So that's the, the big event. Um, and just a few hours, we encrypted about 75 million files. Okay? So a huge amount of files, uh, very quickly. The, the ENCRYPTER was amazing.

Um, we'll talk about the DECRYPTER in a little bit, but that was the event. So that was kind of phase one, phase two of our event were 75 million. That was with an M, right? With an m, um, 75 million. And that's a, a guesstimate because keep in mind, RPSA, our RMM is down, so we can't see exact at this moment, right? We're blind, we're completely blind, um, to the real numbers, if you will.

So that was phase one, phase two, and I think this is another huge lesson learned today, is our backup vendor completely epically failed us, right? So, um, I called 'em up, I'm like, Hey, I have a situation. I need you to spin up two to 300 servers. There was a long pause and there was a, well, I think we can maybe do a dozen or something. And this is a company I've been paying, you know, thousands of dollars to every month for eight years. We had certified engineers.

Um, we had tested one client here and there. We had done disaster recovery, we had all green checks, you know, like, you log in in the morning, do we have our backups? Good? But unfortunately they didn't have a DR plan. So you're saying you could have restored any one of those 200 servers without a problem? Y yeah, as long as there's just one. We could have done a, well, actually, I take that back. Their cloud crashed as well.

And so we couldn't even log into their cloud for almost a whole day, but after the fact, it was scheduled maintenance. Um, so that was the situation. It took them five days to get us our first bare metal restore drive. So that's where I, I, I just said I lost complete faith in their cloud.

Gimme the drive, you know, our customer's data on drives, it's a local company, and took them five days to produce the first server, and then they, they came clean that their system could only produce one terabyte out of it in 1 24 hour period. So, to one Gary's favorite comments, and How many terabytes did you need? That's one terabyte a day. One terabyte a day. We had 88 terabytes in their cloud. So it's Gary. So That's only 88 days. Yeah.

Gary, just quick, so Eric, how many servers Approximately? Um, it was about between two 50 and 300. Ryan, what was your question? Last week? Yeah, my question was, um, the question was, could you, well, there were a couple questions. One was, could you restore a couple hundred servers to a cloud in eight hours and have it operational?

If you went from physical to virtual, the scenario was, do you think you could recover meeting your maximum tolerable downtime for 500 servers if you have a 98% success rate of a single server? I actually, uh, this weekend I went back and did some math because I realized there was a missing component. I was, I was like, I wonder how long it takes to download a terabyte, assuming 10% kind of TCP inefficiency for various different internet circuits.

And I did like your standard like cable modem t one line two days to download one terabyte off of T one, Gary, what do you call that? Sipping, That's a math problem. And, and, and sipping peanut Butter, or trying to suck peanut butter through a straw, right? And at that point, your only option is hard drives. Like you just, you can't download from the cloud fast enough unless you can virtualize in that cloud.

But yeah, again, this is why I was like, you're proving the point of the whole talk, which is you, you have to test at scale. Yeah. You know, if I take every lesson, I'm, I'm gonna go over 12 lessons today, honestly, if, if nothing else you take home, you have to test at scale. I don't care what the product is, I don't care what the technology is, you have to test at scale. That was the biggest failure of our part, is we tested recovering this client.

We tested recovering three servers and a couple laptops. That was the biggest thing is the scale problem. So Ryan, I'm actually going to hop on. You know, I'll, I'll talk about your session here in a little bit. Um, and, and how important that is to do that as well. So, So again, we start, so Eric, I was gonna say we started on this and make sure we get back. I want to touch on the incident response team. Sure.

So, um, yeah, after we found out the, the, we couldn't recover our, our data, um, and we, it was days before our, our first server Okay. Uh, re recovered. So if we come back to, uh, lesson two, um, that is the response team. Um, this is another thing, and I, again, I'm being very open. I'm telling you some of our successes and a lot of our failures 'cause I don't want them to be repeated, right?

And so your instant response team, you need to have your insurance agent on speed, dial your hotline for your cyber coverage on speed dial. You need to have a legal team. And I mean a cyber legal firm, not just Joe Attorney down the street. Um, you need to have an IR company. I mean, thank goodness I had Chris Laer on speed dial. Um, I mean, it, it, it's amazing. He wasn't approved on my insurance policy, but I called him first and my insurance is like, you know, what?

If he's, if he's already spun up, go. And so I'm not saying that's always gonna be the case, but that was critical to have these people on speed dial. And, um, you know, you, you don't wanna build a relationship in the middle of an incident, right? Like, okay, what are your rates? Okay. You know, uh, is your tone of voice? Are you always like this? Like whatever you, this is not the time to start that.

So you need to have your team, um, we'll talk about the players of the team a little later, but, um, that, that team is critical and you need to engage them, even if it's a false, you know, you have a customer with the email business email compromise, you might need to engage them to say, Hey, where should I go? What, what should I offer? Should I tell them they need to go get their own ir? Should I tell 'em to call their insurance? Where should we go with this one?

Because each scenario has its own little twist. And, and you need to have that expert guidance on that. And then what about your, on the response team from your standpoint internally now? Sure. So, um, so we had a few key players and, and you know, ready to, you know, our senior techs, right? The guys that you always rely on. We had, we had things spelled out. We had a small IR plan. It was a two pager for ProTech, right? A couple little things.

We had no IR plan for all of our clients all at once. And so it didn't fit super well. Um, one thing that I will warn you is I have, you know, a phone to each ear like this for three days, 15 hours a day. So your, some of your key players won't be available because they're on with attorneys and clients and IR firm and all that stuff.

So you need to make sure that your team out there, and this is a, a lesson a little later distributed decision making and that, you know, the way you present yourself as the leader is gonna be the, if you have success coming out of this thing. Yeah. That's interesting to think about it. Like you're communicating with all those people you just mentioned, trying to figure out, you have customers you need to communicate with, and then you, someone has to be running the team and making decisions.

You probably can't do all three at one time. You can't. I had a line outside my door for the first few hours 'cause people, should I do this? Should I do that? And I fi we finally had to pull everyone in and say, okay, this person, you are in charge of how we're going to automate this. You are in charge of, um, determining how many files are out there. You are in charge of how to match up the decryptor with the, uh, the, the serial number from the encryption with the decryptor.

And so we kind of had to lay things out so that we could keep things moving forward. Because I was the bottleneck for the first few hours. I mean, I was in shock and I was, I didn't know where to go. Like, this was my lack of planning from, from scale that, that really hurt us at, at the beginning. Yeah. Could we just get Robert's commentary real quick on Yeah, absolutely. What's happening? 'cause he's, he's awesome. Robert, you got, you're on. Yeah. Perfect. Yep. I'm unmuted. So, exactly.

I mean, um, you know, Eric is hitting it right on the head. Um, you will, you know, uh, peeling away the emotional impact of everything that's going on and people barking and yelling at you. Um, just even having, uh, everybody, uh, with a plan of exactly what to do, like, you know, I'll just add in as an example. We had our tools administrator, you know, just making, uh, USB sticks for guys to run out in the field.

You know, we ended up having a lot of boots on the ground, uh, come in from different, um, cities to help us, but we needed to equip them with things. So just having one of my really like, important technical resources, who's brilliant, peeling him off recovery to give him a job of, you know, just create tool sets and checklists for everybody to follow. Really good. And then, um, that kind of leads to number three, which is communications. You.

So specifically you have your external IR team, you have your internal team. We talked a little bit about that. And then you have your customers. So can you talk a little about communications? Yeah. So the first thing that I'll mention is, man, stuff changes quick. What we think happened in 15 minutes later, we find out, oh, that's not what happened. And all this communication I had prepared is all out the door. And now it's a different thing.

And so one of the things I, I really liked about, and I'm gonna say I had a awesome attorney, and I don't put awesome in an attorney in the same senate very often, but I had an attorney usually That's an oxymoron, right? Right. They pulled in a crisis communication company, like within that first hour. And I actually went and pulled it up and I have this three page document, what I could say to my clients and what my people could say to our clients.

And if you listen, I'll, I'll read a couple points on here. It's like every other one out there, right? You don't use the B word. Um, so it says, ProTech was a victim of a cyber attack on, on our network on February 4th. Um, we found that our, some of our servers contain, uh, ransomware message as soon as we discovered the attack. We took immediate and precautionary actions to deny outside access to the network. Our customer employees are our top priority, and we are taking responsibility.

We take the responsibility as your service provider seriously. And we are deeply sorry for the interruption of your services. Yeah, yeah, yeah. And it just keeps going. But notice you're, you're not totally something blamed, but you're not saying we're not gonna, that we're skirting it. Yep. They choose words really carefully. And this was what I, you know, we hand we print this out and hand it to the text and say, Hey, every question they ask, it has to be one of these answers.

I don't care what the question is, how's the weather? It has to be one of these answers, right? And so communication is, is vital. The other thing that I've gotten feedback from after meeting with a ton of our clients after the fact is they're like, I was so appreciative at six o'clock every day you send out communication. And it was absolutely brutal.

I have to start that communication at two o'clock on whatever my information I have at two o'clock, I have to run through the crisis communication company through the attorneys and the, at six o'clock I could send out a message saying, this is where it's at. And I would give them a little bit of meat and then they'd strip it out, strip it out, and then at the end of the day, it would be, we're working around the clock to restore your files. Thanks for your patience.

And, um, you know, we're doing everything we can because that's really what can be said in instance like this. So it it, And we'll have you up in 88 days This case. Yeah, exactly. Um, it, it, it's, uh, after the fact is when I appreciated this more than during, during, I just felt like tongue tied. I was frustrated. I didn't realize how, you know, the legalities of it and what could come up after the fact and how people would twist words. And so it is a super important part.

And you, you can google this stuff. I mean, it's not rocket science and you need to have it ready for your team because, you know, we're techs, we wanna help people. We wanna tell 'em what's wrong. That's in our DNA and we just have to be really vague and, and choose our words really carefully on this one. Yeah.

I mean, some of it I obviously is legal, but the other one is just, you know, you say something and something changes and now you have all these different clients taking it different ways. Now you're in a word, wrestling match wasting time when you need to be focused on helping them and other customers. So it's, you know, definitely full transparency is in this case, is not, is not the best thing for your customer. Yeah. So this, my incident hit Twitter the night it happened. Okay?

So one of my customers could you not went into a Microsoft store and said, my MSP was hit. I need to buy a bunch of surfaces. And the guy at Microsoft store looked up who the partner of record was and tweeted about this event. Now he was classy enough to not put the name of the company, but man, first night I was getting stolen. Oh, stupid sales guy doesn't, must not have had backups, had a super VPN to all this client. I mean, it was horrible, right?

And then I had clients mostly call center type stuff where they were taking pictures of my email, you know, explain what was going on and tweeting those. And so when you talk about communication, like it's gonna come from everywhere. And then I also had this guy, I'll never forget him, um, I think he had 159 followers and he was just calling me out every day. And he was unemployed by the way. Um, but I remember that was getting in my head because I was so beaten down.

So I guess what I'm trying to say, Gary, is you're gonna have communication from so many different medias. Like, I literally went dark on social, my personal, my business, I just went dark for almost a year. I just did not want to hear it, didn't want feedback. Um, so yeah, communication's a a tricky situation because it comes from so avenues these days. Robert, similar for you. Uh, yeah, I mean, I'll echo everything that Eric, uh, said.

I mean, he went through it, uh, worse than we did in that regard. But one thing I, I will, I mean, two things actually. I'll underscore uh, awesome attorneys. Yes, they're out there. Um, and so don't, don't be, uh, don't be a negative Nelly on attorneys 'cause they're really, really important. The other thing is, if you don't have anything to say, it's important to say that. And that was a mistake that we made early was just to say, we have no updates right now, but please stand by.

Like, we're still working on it. You know, even even plain things like that let people know that we just didn't go into a black hole Really good. I remember some MSPs closing their doors and just put up a webpage thing. Go seek other help, we're done. Yep. I've seen Eric, I've seen that more than once. Yeah. They just walked away from it. Um, so your number four is manage the process. Yeah. So this is a crazy one.

So I'm, you know, we hopefully don't have any, uh, PSA vendors online, but a PSA is not a tool for this. When you're getting a hundred tickets an hour, like how do you manage the thousands of tickets? It's not, it's, this is not how it's gonna work. So you have to be nimble and you have to get really agile really quick. So I had someone here that made like EO or Airtable or something, and it just had a bubble of each of our clients.

And under the bubble, it had a bubble of each of the PCs that were hit and the color. We had a color coding system, and we could just, as we got bodies and we could throw bodies at this, we say, go grab all the blues. And that was the one where you just need to go babysit the computer and make sure that decrypter is running and take off multiple decryptor and so on. And so we had the system and it was so cr I mean, it looked like kindergartner through this thing, right?

But it had meaning and it was super agile, and as soon it was done, we would drag it into a different board, right? You can't do that in a PSA, like, it just, it's just not very practical. And so, and then also the last thing in the world you're gonna do is wanna sit around and take notes. So grab someone, grab a marketing person, grab someone that's not technical and have them help take notes and walk through things with you.

Because guess what, in a week when the insurance says what happened, this happened or that happened first. And that determines if you're gonna get a $200,000 payout or not, right? So you had to take notes through the whole situation. Um, I remember Chris Laer sent, uh, someone up here and I was a little, you know, butt hurt at, at first, like, why is he sending someone up here?

We got this, you know, and it was critical 'cause this person was the one taking notes and communicating back to the insurance and things like that. So the manage the process, there's a lot of different aspects of it, but if you are not organized enough that external help can help you, then you just have a bunch of people knocking at your door, Hey, I'm ready to help, what can I do? And you can't use them if you're not organized enough.

So we had a way of creating, uh, accounts in a different RMM and we had instructions and how did they do this? And so that we could take all these external resources, plug them in to start helping us with the, the, you know, undoing the, the aftermath of the, of the ransom. So it's a, there's no magic to this. It's just you don't forget about that while you're going through it. Yeah. And Robert, on stage, you talked a lot about that, that basically, uh, your job, right?

Was to be patent, I think is the word you used. Uh, yeah, I did turn into a sort of a tyrant, but it was necessary. And I think my team really wanted that. But, uh, if I can echo what, uh, Eric said about, um, your PSA is not the tool to be managing this type of male mass scale kind of recovery efforts. We did something a little bit more low tech.

We had colored post-it notes and a blank wall in a empty office that we used to just play this dynamic and fluid chess game of who's going where and who's doing what and when. And, um, as crazy and as stupid as it sounds, it worked. And I have 'em all, by the way, they're my company company lawyer. They're in a ga glass case someplace. Uh, I I've seen 'em. It's about that thick when you have 'em all like stacked together. Yeah, it's crazy.

It's a kaleidoscope of like, all these just different, uh, you know, post-it notes and, uh, but, but you know, Eric's a hundred percent right? Like, you know, you don't realize these things until you're in the midst of it that you just have to do things completely different. Yeah. Gary, um, to that point, Eric, what, what you and Robert just said. So we had a company called Exigence that write a boom. Chris LA's pla Chris Laer manages everything in it.

And they built, uh, temp some templates specifically on BEC and ransomware for the attendees that they can tabletop. Um, and what that platform does is it pulls in all the third party communication channels, it's built to do incident readiness, incident response. So if you guys are interested, I will ask the CEO if he'll extend it to you all. It was for right of boom attendees. It's a lot of, you know, stuff that he, uh, set up for them. Um, so just gimme a yes or no.

And if you're interested, um, uh, send me an email at andrew@thecybernation.com. Um, I'm, I'm gonna hand over to Ryan before I do, I wanna make sure at some point I want you guys to mention the fact that you had to get reimbursed by insurance. We talked about this on stage and that just that concept, while you're doing all this, you also have cash outlays, uh, that you have to do and you have to have the cash flow through. Where do you wanna address that now?

Uh, if you want, You know, I, I think probably worth, uh, just addressing now. And so, uh, when we found out that our backup provider failed us, um, we were negotiating, uh, with the, uh, threat actors on, uh, a payout and, uh, they came back and said, I, it is funny. I, as I looked at it the other day, it was like $86,000 and you'll get all the, and so we had to crack our piggy bank wire that to someone that luckily Chris knew.

'cause there's a lot of rules around large sums of Bitcoin going wherever and, um, send that off to the attackers. The attackers then said, oh, this is a lot of keys. We have to wait for our senior programmer to get back into town so that he can write a script to generate all these keys. Um, but yeah, I mean, if we didn't have that cash in hand, I'm not sure what we would've done. Right?

Because when you, when the backups are bad, that went now we're, you know, decryption is our only method really. Um, unless someone can wait 88 days for their server to come back up. Um, so yeah, it was cash in hand. So managing those finances and, you know, to, to echo what Gary says, you got charge, like there has to be a healthy enough company to have money in the bank. Um, yeah, Well, you know what, you know, and one part, you're lucky at that time. Like today, 86,000 seems like a bargain.

Yeah, Right? It's, and so that, like you had an option today. Yeah. They're smart enough to look through all your customers and, and know what each of them is worth. So that number could be, you know, 500,000, a million or, you know, and then north, you know, north of that where it's not even an option for you. Absolutely. I mean, I think, uh, of first ours was 8 million and so that just kinda puts in perspective, but I don't think they really understood what they had. Right.

Um, our tools were down, so, you know, we had no choice but to pay. So Robert says 5 million. Did your wife have that in cash, Robert? The 5 million? Yeah, he got a bigger trunk. Uh, we keep that only in, uh, jewelry and precious stones and other, um, precious metals. Not cash. He's Italian from New York. I mean, do math. Hey, that's a stereotype Which Are, which are created for a reason. But I digress, Which you can listen to more on the prior recording, Robert. Yeah.

Alright, I'm gonna, I'm gonna hand things over to Ryan. Alright, thanks. So earlier in the chat there was some conversations around, um, community, and that's full of, that's kind of takeaway number five for you. Can you talk to us about what the role community played, um, both kind of your SMB community of your customers, your local peers, your maybe larger peer groups, your vendors? Like what, how did the community kind of help you through this?

Yeah, so community is, I mean, I am, I feel like we are very fortunate in this industry to have community. I wish our clients had a community like what we have, right? So I've been part of HCG for six or seven years and you know, one of the, the companies in my group sent two senior checks out for two weeks. I mean, sacrifice, right? They wrote PowerShell to match up the serial numbers with the decryptor so that we could send that out.

I mean, can you imagine trying to hand match all these things? And we had no PowerShell programming expertise. Um, and then, you know, the aftermath of going to groups and you know, like this isn't something, you talk about how your company is worth x say a million dollars one day and then the next day it's worth negative x, right? Talking through that process. Um, I have a friendly competitor in town, and this is probably one, this is still tugs of the heartstrings a little bit.

Um, he heard about things because our clients were calling him saying, Hey, our, we got hit, we need a new MSB. And he called me, he is like, what's going on? I kind of told him, he is like, what, what can I do? And I'm like, well, our backups are all toast and we can't reload 'em 'cause we don't know about forensics yet. Uh, I need BDRs. And he pulled up the next morning, I don't know how he did it, he pulled up and the next morning with three servers in his trunk, BDR.

So he's like, okay, here they are. They've got 10 terabytes each, here's the passwords. Where do I need to take 'em? Right? And he was plain clothed, not in his logo or anything. And so, um, you know, he sent a tech here that worked out of our office for a while. And so it's hard to underestimate how important that is and when you need to draw on that.

Um, and you know, even now, like as a whole, as a community, I think we're doing way better about not shaming and working together, as you mentioned earlier at the beginning of this, working together as a community to, to make us all stronger, you know? Um, yeah, if you're not part of a peer group, you need to be part of a peer group. Talk to Gary, it's true methods or, um, true peer. Um, it, there, there's a few good ones out there, but I think it's vital.

Robert, can you share your part Of that? Yeah, before you go on, I just wanna mention that everyone heard not just being part of a peer group, but how important it is to have a relationship with your local competitors when whenever possible. Robert, I want you to share your part y your, your story on this is pretty profound. Yeah, totally.

Um, I, I, you know, only because of internet connection quality here, I'll try to make it quick, but, um, you know, my situation was very different from Eric. Uh, we had to recover 250 servers and about, you know, 2,500 endpoints total. Uh, meaning we had to new 'em all and just rebuild from scratch. Um, we had no ability to get decryption keys, so I needed cheer manpower. And it was our, you know, Eric and I are both, uh, part of the evolve peer group.

Uh, and I'll echo, I don't care what group, get into a group and make lots of friends. Um, and we had technicians, you know, one, one of our members just, you know, had the wherewithal, uh, to look past our shock internally here and say, you know, you guys need help. And, uh, so three business owners along with, uh, approximately 20 techs from as far away as California, Florida, Texas, Kansas, Minnesota, Massachusetts. I know I'm missing a few states, uh, Iowa.

Uh, I had one guy show up, I didn't even know him. He was a friend of a friend. He said, I heard through the grapevine. I'm like, how do I know you? Like, what's the connection? He said, yeah, your friend up in Minneapolis told me about it. And I'm from Iowa, so I just got on a plane and I came here. Um, it was pretty effing amazing, um, to have some people show up for, you know, 8, 10, 15 days at a time.

Um, and really just to take money out of their own pocket in terms of meals, hotel flights, everything. You know, eventually in pandemic, you know, In A, in a pandemic. Yeah, in a pandemic. And, you know, we're gonna make them whole, but they fronted all of that expense themselves. And why? Because we are brothers and sisters.

One, one thing that we're gonna need to solve for in the community, uh, and I think this needs to be a grassroots effort led by you guys, is, um, and this is something Jason's label mentioned a long time ago. He says, you know, the the challenge we're gonna run into is in these future events when someone comes in to help what the liability there is in terms of like, you know, you just came in and helped and you saw a bunch of like internal, you know, confidential information.

You don't have NDAs to protect all this kind of stuff. And he's like, I'm not comfortable. He's like, I would love someone to come help when something hits a fan for us, but I'm not comfortable with that at this point, um, without agreements in place.

And so I do think there's a, I'd love to see the peer groups or some charity intermediary of some kind, some nonprofit broker that in, in, in that way, if you're part of this group, you've already signed into the, you know, all the liability waivers and all that kind of stuff. It's being worked on. It's being worked on. Wes, I can promise you, I didn't, I didn't really reveal a lot of information yet, but Eric has been in, uh, I got, I got involved in something. I instantly pulled Eric in.

'cause I can't imagine navigating it without him. I like hearing that. And many Of you will get the call too. Trust me, it's coming Robert, when that, when that comes forth and you're ready for the details, we'd love to have you back on the cyber call to talk about that. Yeah, Absolutely. And Robert, anything i I can do to piggyback on that with our groups, you Just let me know. You all will. You all will. And we can't do it without you. So honestly, that's the answer.

Seriously, I, you know, I can't do this alone. Eric can't do it alone. We can't even do it together, right? Even the, the six of us, you know, our, our silly faces right here can't do it. It's gotta be bigger and broader than all of us. I I was hoping Eric was on till Eric, uh, till, did you just comment like that point that Wes just bought up, that it sounds, is it fair to say from a legality standpoint, there, there needs to be things in place?

Um, let, we will let Eric kind of give his 2 cents in chat, but okay, Ryan, please continue. Yeah. So distributed decision making, I have to imagine that 50 calls a minute and thousands of systems to recover. There's a few decisions to make. How do you decide who makes what decisions? So, um, I, I already talked about this somewhat, you know, the, the, some players are just not available to make decisions.

And, and I would echo like in a tabletop exercise, you need to do tabletops with, with a couple of the key people sitting there mute on the side that they can't be a crutch, they can't be part of it because that's gonna be closer to reality. Um, one of the most difficult things I've ever had to do my whole life is decide who gets on the lifeboat. Okay? So that was our, our term. Um, even very mature MSPs that thought that they had table topped everything.

When, uh, Chris Laer gave me this feedback that he, you know, went and did one for a large Ms. P and said, how have you decide who's gonna come up first? And you know, just blank stares, like which customer you have a finite level of resources, which ones are gonna come up in which order? Is it your favorite? Is it, or biggest revenue? Is that the ones that you know are most litigious? Is that the ones that are hurting the most? What does that look like?

Um, what we've done now, so again, the benefit of hindsight is based on their security package. We have three levels, and I make it very clear that if you're in the top level, you are the first people coming up, then it'll go to the next. So the people that invest the most in security, because it's gonna be the easiest for us, they have A-A-A-B-I-A business impact analysis. We know which systems need to come up, we know in which order and which ones hurt their revenue the most.

Uh, they have IR plans, they have things like that in place, right? And so the people that invest the most, we can bring up faster than the people that you know, oh, this will never happen. I really need AV type people. Um, so it, it's a, that was so incredibly hard. I had a, a trucking company that had a hundred trucks across the us They couldn't tell where the trucks were at and they couldn't tell how to pay these guys.

They went to the bank and got out $40,000 in a hundred dollars bills and they were just handing out hundreds as a paycheck and have 'em sign their name, right? Well, that was one of our first ones we wanna bring up, right? We hear the story, but then another one over here is having another thing. And so having that plan of who's gonna come up and then the BIA, I mean, that shouldn't be covered under managed services. That's a billable project. Which system needs to come up first?

What needs to come up second? And what can you live without for two weeks, right? The problem is if you get chaos coming in the door, oh, if you get my wifi up, I'm good for a week and you can let me off. No, actually I need my QuickBooks up. No, wait, we had this one order. I need my email. And we had all these crazy conflicting things where if we would've had a BIA in place with, if our customers had a BIA man, how easy would it have been to kind of step through that process?

Andrew, there's a clip in there. If you go back, uh, there's a little clip when he's talking about making those decisions will be awesome. So your seventh is company culture. Talk to me about how your culture kind of helped you through and also failed you in some way through this. And if you had, you know, if you could rewind time and reset your company culture to something that really makes it, uh, kinda stronger to withstand this type of incident, what would you do? Sure.

So I mean, I actually have a few of my people on, they can, they can put some chats, some comments in there. Um, when the s**t hit the fan, everyone stepped up. No one quit, right? And so that's one thing that I take is the, one of the best compliments. And when we told our clients, no one quit, they were so, they were, you know, shocked. We had, um, uh, the best person we could have had in service dispatch. She had a nice sweet, soothing voice and she was our human shield.

She took angry call after angry call for hours on end, for days on end, for weeks on end. And she stayed positive and she worked through that whole thing. Um, you can't start company culture when you're going through an event, right? Like we had invested in it. We have been, you know, there's, we have our, our missions and our values, we did activities together. We had proven we had each other's backs, but this really tested it. I had people working a hun I had people sleeping over.

I had people, um, you know, that didn't leave here for two or three days and, and we had people bringing in, or we had clients bring in dinners and lunches. Um, we had people, you know, a hundred hours a week was kind of the average. Um, a little dark secret about this is hourly people insurance will cover, you know, the extra overtime. They will not cover anything for salary people.

So I had all these salary people doing a hundred hours a week and insurance says, oh, that's an expense you would've had out regardless, no additional, right? So these are people that live through that. Now, on the flip side, and I'm not trying to pat myself on the back, you know, six months later when we were get back on our feet, we made right with the sly people that worked, all those extra showers, right? And so that was just kind of part of our culture that we take care of each other.

Um, I don't have a great example of failure other than, you know, in hindsight, I, we probably should have done more post traumatic things, uh, for healing and talking it through and kind of dealing with that. Um, I did my own personal thing, but, um, you know, that in hindsight, I, I, I think that would've benefited the group more. So, uh, you know, we've been through hell and back. A lot of 'em are still here.

You know, we still talk about it and kid about it and, uh, it was a bonding experience. Uh, though I would not recommend bonding that way. Yeah, I always tell my team when we talk about these things, the one thing I want, the one sentence I want you to repeat to yourself is, this is not happening to me, right? Like, you are okay. You are healthy, you are safe. This is not happening to you. This is a situation that you need to help deal with, but it's not happening to you.

And we, we talk a lot about mental health 'cause actually something that I worry about a lot, you know, just the, the rate of the kind of the simmering six with stress and security responders and people that work in the field on a day to day basis, it's high already. Throw 'em into a situation like that without good mental health support. It's, it's, you know, people, people are gonna be left with some scars for sure.

But I, I love that you, yeah, I think that's a good takeaway in terms of what you would do from a culture perspective or, or really just in terms of taking care of people in the future. So, um, so last one, uh, never whitelist. So this one's a little more tactical, so this will appeal to the geeks in the room a little more, but do not whitelist anything ever for any reason. Um, so our, uh, AB vendor that was kind of managed ab took it upon themselves to take, uh, our RMMs, uh, best practices.

That is the whitelist, the, the folder your RM dropped down onto the desktop. So then our notice, I'm not saying vendors, I'm saying our next gen AV welcomed in the ransomware with open arms, because that was whitelisted by the way. When it's whitelisted, it doesn't even log what it does. It's just like, ah, I pay attention to you. Go ahead and do whatever you want. Right? So as we looked at different vendors and different things going forward, first question is, I will not whitelist anything.

Are you, you know, is will your product still work? Because I won't whitelist. Like I, this is a non-negotiable for us. Um, and so that was a, a big, a big lesson for us because man, if, if that wasn't happening, I mean there, that would've been an easy stop and this would've been a non-event if that was a white list, right?

So just, I, I would call, I would highly recommend for anyone to go look at your thing and do not white list, work through whatever you need to work through to allow like one executable through or one path through. But don't give a folder, especially an RMM folder. The access is just to run away with things, even If it's inconvenient. Even if it's inconvenient. So yeah, that's, again, that's a little down the weeds, but I mean, that was a huge, like, wow, that never considered this scenario.

Okay. Um, so Wes, over to you. If I can unmute myself. Um, man, that's good. Um, okay, so this next one is loaded. Uh, oh boy, is it loaded and I'm just gonna come out firing and I'm gonna let you guys fire back. One thing that vendors are not so well known for is, um, handling any amount of liability default. TOS. And this is not just in the channel, right?

I've dealt with a lot of vendor contracts, um, in my days as a CIO, and I also know my own bank and the contracts that they have with their own clients. We absolve ourselves of every piece of liability that's humanly imaginable. So your, uh, your next one is review contracts. Yikes. How do we navigate this? Can you first talk about your experiences with this, Eric? And then how the heck do we navigate this together? I mean, do we start some huge MSP union with, you know, collective bargaining?

Like how do we get through this? Because every one of our vendors are basically writing by default into, we have no liability for anything no matter what period, end of story. How do we handle this? So, uh, I have one word answer and that's redline. Um, uh, I think that, I don't know if Mike Lee's on the call, but he, I've, I've had a conversation early on and he took it upon himself to go and redo all his vendor contracts.

And he's told me on multiple occasions how it just changed everything for him as they were growing through iconic and, and getting larger that he had that leverage. The problem is, is, is one MSP, sometimes we don't have the leverage. I mean, I've had vendors say, you know what, unless you're gonna do a petite, a petite a petabyte of data with us, we're not gonna redline. And then that, that's feedback for me.

I have to go see, seek a one, uh, vendor that will, and so redlining is whatever the agreement is, the attorneys, hopefully the attorneys strike it through, it goes back to the vendor, their attorney strikes it back, and you, you, you do this, this, this battle back and forth. I can tell you that all of every vendor contract I sign now goes through legal and it will cost me from 300 to $3,000 on every product. So talk about tool cost, right? We have 41 things that go into ConnectWise.

It sounds like a lot, but if you, if I was to pull it up there, I think that probably 75% of the people on this call would have almost a lot of those. They just don't have it. One ab Absolutely. You're right. Yeah, it's crazy. And so you think of all those vendors, right? And the, so the first one, when we see a new backup provider for some reason we need a new provider. The first question was, you have to be able to produce a hundred terabytes within 48 hours. First thing.

And I went to a couple big companies and I, I applaud them. They came back and said it would be 37 days. Thank you for at least being honest and doing the math right? Um, I found one that they had the, the, the capacity to do that, and that was why I went with that company. Um, so redlining is, is vital. I'll, I'll put, uh, myself out there as a, uh, a young entrepreneurial, uh, didn't know better. Uh, the backup vendor I signed eight and a half years ago, and it had, it was airtight.

It was, we can screw up in any way, shape or form, and the maximum you can ever come back for us is one month refund. Okay? And the epic fail that they had, that contract was still airtight that we, the insurance company, I, I didn't have a say in this, said, you know what, it's not worth the battle to go fight 'em, even though it was a hundred percent the wrong and you have recordings of all the thousands of ways they screwed up is still not worth going over because that contract was so tight.

Okay? So, um, may again, peer group, maybe you work as a peer group and you get a dozen companies and to do business with a b, c vendor, you're all gonna say, okay, we're all gonna buy 500 agents of this, but we need this red lined out. Or you just need to share in some of the liability here. Some of the vendors, and again, I I apologize for the vendors on the call, it, this says like, we can be negligent and you still can hold us harmless.

Like we, there's nothing you can do to, to touch us, and those aren't the vendors we need or want. And I had to leave the MSP space for some vendors because I couldn't get what I wanted from a contractual standpoint. So I don't think there is an easy answer, Wes, I wish there was. Um, I do think it's super important, and again, the more of us that push back, the the better this will go.

Um, but I mean, we've redlined with Microsoft, so don't, don't assume that the size of the company means that they won't play ball, okay? So that you, you don't know unless you ask. So Yeah, the, we should think about it is whatever the terms are of the agreement is the initial, like, that's the position I ideally want. It's the best for me. And then if you're willing to sign that, great.

But I also know and expect that you're gonna come back to me and say, well, I will, I think this is more fair. And most vendors have like, you know, both, both being a vendor and negotiating these all the time. Most vendors are like, well, my position would be, I would like to only give you, you know, one week's notice of a potential breach, but I'm really willing to go up to 48, but I'm not gonna give you 48 unless you ask for it. Yeah. Right?

And so, like, there's a lot of these terms in these contracts that vendors expect you to red line and send back and negotiate. But the opening position is just accept whatever's there because most people don't, don't negotiate. And so, um, yeah, I've, I, I've, I've been on both sides of it, right? I've gotten legal calls with customers, MSPs trying to negotiate security language and contracts, and I'm like, that's reasonable. Yeah, we should do that. Let's switch it.

Let's, let's make that concession, right? Um, and then on the other side, both as a, as a, as a procurer of technology, like I've walked away from technology contracts too, where it was actually the better fit, but it was dangerous for our company given the terms. Uh, yeah. So definitely important to have a lawyer on your side for sure, when you're looking at tech stack. So th this is a good lesson learned for everybody.

And you may ha we talk about tech debt, you may have legal debt as well, right? You should go back and look. Do I have an 8-year-old contract l laying around? It's super scary. Gotta deal with it. All right, here's my problem, Eric. I only got, uh, three minutes left for two. Okay, so why don't we pick one, the last two are people and protect your BDR. Pick one, and then I'm gonna ask you a question on it. Um, so I, I think I could do people in one sentence. Okay.

Um, people, you're gonna, it is, these type of events will bring out the best and the worst in people. So my best clients became the most unre unreasonable monsters. And some of my clients that I didn't think I had much relationship with were bringing us pizza. So just know and be prepared that emotions are not rational. They're gonna be all over the place. So be prepared for that. Okay. That's, That's, do you want me to hop on the last one or we can wrap it up.

Let, let's jump on the last one then, because I think it's important. We talked about this one. So the last one's protect your BDR. Jump into that one for us. So BDRs, uh, this is super technical. Don't use the same RMM on your BDRs as you do the rest of your stuff. If, if it's a Windows, um, try to get on different BLANs and only allow the necessary traffic through require two FA on all the BDRs, and also have it notify when anyone logs in, right?

This is for the horse stories of people getting their backup through, um, white list. Even within Microsoft, you can actually say only these type of files can go in the full in this folder. Um, let's see. Uh, don't put on the same domain for heaven's sakes. I, I can't believe I still hear people that have their, where if you have a credentials pop, you could get across to the, the BDR as well. Consider putting a firewall in between 'em and put some acls in place. Um, have a mutable storage.

Um, you know, again, super tactical, but things that I never considered an RMM situation. And if I would've had a different RMM on my BDRs, then my BDRs would've been in a good place. So, Yeah. So this goes back to what Ryan often talks about with threat modeling, understanding, okay, so if the BDR is my last source of, uh, restoration, how do I protect all of that? How do I, how do I assume a threat actor has access to my network?

And how can I, um, limit and cord off and prevent access in certain ways? Um, that, so I think that's solid advice, um, and probably something that's worthy of a much deeper discussion on how we can tactically go about that. Um, this has been fantastic, Andrew. I'm gonna turn it back over to you and for, for, uh, time's sake here. Um, but man, Eric, these 12 are solid gold, my friend, and really, really appreciate you, um, coming in and sharing these things on our behalf. Uh, thank you guys.

I, I still hate talking about this event, but it, it, I think, you know, people can benefit from it. So I'll, I'll keep talking. So, uh, Eric, one of the questions was, would, can we share your 12, uh, can I put it in Cyber Nation? Obviously that's putting you on the spot. Uh, yeah, no, it's, it's fine. I don't, I mean, I, I try not to call out vendors, uh, you know, because I didn't want people doing the same to us. Uh, but the, yeah, Okay, so I'll put it in Cyber Nation guys.

I'll, uh, I'll, um, if you're, if you are just join Cyber Nation, it'll be there. I'll, I'll post it momentarily. Um, Robert, hey, you're on vacation and you are also just incredibly selfless with your time and helping everybody. So thank you as well for all you are doing for the community. Um, really, really appreciate it. Get a martini on us, my friend. I'll buy, I, I have the unlimited drink package, Wes, so that's why I'm buying. Oh, they definitely have one on us.

All right, well, on behalf of everybody, um, make it a fantastic day. We look forward to seeing you all next week. I hope we'll see you all tomorrow for the vulnerability Management best practices session. Make it a great day. Thanks again, everybody. Okay, thanks everyone. Thanks Robert. Thanks Eric.