Critical Veeam Vulnerability Exploited to Spread Ransomware

All right. Welcome back everybody. Happy Monday to you all. I'll give it a few minutes here, um, and make sure you all can hear us. I'll put something here in chat. Welcome everyone. Uh, let's see. Can you all just gimme a a yay, nay, can you audio coming through, chat, coming through all those good things, if you guys could just let us know, be much appreciated. And we'll get kicked off here in just a moment or two. Let as, as folks trickle on in. Hey, ed, good morning. Can you hear us okay?

It, ed. Hey, Andrew. Joel, see you Allall. Good. Now they're all coming through. All right, fantastic. I hope you all have an awesome weekend and, uh, all right. All right, so I'll set the stage here, by the way. Um, um, let's see. Gosh. Alright, so, um, just quick announcement. I'm gonna put in a URL for tomorrow. We have a, uh, a webinar with, um, with Eric TILs. And I think, Bob, I don't know if, did you, were you at MSSP alert and got to hear Eric do a little bit of this Yep. Uh, session.

Yeah, it was a great session. Yeah. Yeah. Um, really great. So this is called Cover Your Breaches. I just put the URL, it's tomorrow at 1:00 PM uh, Eastern, I'm pretty sure. 1:00 PM not 2:00 PM but I'm almost, it's one, 1:00 PM Um, just check out the link, you'll see it. And, um, uh, it's gonna be Eric Tills and, um, uh, Reed from Fifth Wall.

And this is the case, if you guys remember back the MSP in California who, um, had a lawyer, a legal, sorry, had a legal firm that got ransomware, right, Bob? Yeah. And, um, and then, you know, basically based on, you know, what was in MSA, what was said, how it went all about, found themselves being sued for about a million bucks. And, um, Eric's gonna unpack, uh, kind of what went wrong and kind of how to insulate, I shouldn't say insulate, but be better prepared, right?

Because we all know we live in an era, an era of inevitability, where we're gonna have, we're gonna be dealing with incidences in one way, shape, or form, hopefully more, uh, less, uh, uh, Ben or hopefully benign relative to something like this. Alright? So that's, uh, on the announcement side. Um, so setting the stage. So we've been talking for the past few weeks. We've had, you know, the, basically the, we're, we're kind of retro right back to Dr.

Right who, who, you know, obviously Dr is important, we know it is, but, you know, never really thinking these days so much about, you know, the physical threats, right, Bob, that we've had these natural disasters, although with me in Florida, and you and Louisiana, you can relate. And, um, they've, they've, uh, certainly, uh, you know, as, as they say here in Florida, we are done, you know, we wanna, we wanna unsubscribe for the hurricane of the month club, right? Um, um, yeah, right.

But we did some really good stuff on the positive side. We were able to raise a, a fair amount of money for Kemper and his staff. And, um, that's being put to good use. So again, thank you for all that. So this week we're kind of back at it on the digital side of things.

And you know, as I was saying to Chris and Bob offline, you know, most vulnerabilities, you know, if you talk to folks like, um, Steve Carter, who's the CEO of Nucleus security, um, who ran a, a big portion of the DODs, uh, vulnerability management before starting his own organization. Mo Yeah, ed, by the way, ed, yes. It will be recorded mo most, um, vulnerabilities these days, Bob, you know, you have to be aware, but it's not like, you know, fud, Hey, the sky's falling, et cetera. Yeah.

But occasionally you get one that the threat actors really glom onto. And this deem vulnerability, which has been out now for several weeks, um, is really gaining some momentum by threat actors, ransomware, gangs, et cetera. And so I really wanted to get Chris on here because this is something, you know, it's one thing when it's, you know, a a a platform that isn't used a lot by MSPs. Um, and I don't wanna say we don't care, but this one, this one's in a lot of people's stacks. Fair, Bob. Yeah.

It's like SolarWinds, right? Reminds me of the SolarWinds thing. You had a lot of people using it. Absolutely. Yeah, yeah, yeah, yeah. So you have CISO ringing the bell out with articles. You have, um, many of the main publications talking about the ransomware groups. And then in speaking with Chris, um, he's personally, you know, they're, they're personally dealing with it, you know, when I say personally, their firm right? Is actively working cases of this.

So I thought, you know, again, let's dig into what might be going on here, the why behind it. Um, Chris, I think everybody knows you, but quick intro for if maybe one or two people on here that may not know you. Quick intro. Yeah. The two people have been in the cave. Yeah, it's Chris. I'm with, we're based in Austin. We're an response firm.

Uh, we used to be an MSP, uh, when we were acquired, uh, a few years back by a, uh, London-based insurance carrier called CFC, uh, that MSP got rolled off and is now part of Integris. So, um, so lived in the MSP space and the security space for a while now. Just, um, uh, live in the MSSP incident response space, but we do a lot of work with MSPs, like Bob, I happen to be on here today, so, um, yep. Happy to be here and talk about this. Awesome.

And Bob, thanks for sitting in, because Gary has, he's been in, um, uh, gosh, meetings. He's got troop here this week, and Datacom the following week. Phyllis has had travel all month, so we'll have the crew back together in November, just in time, I think, for IT Nation. So then everybody out again. But I appreciate you coming in co-hosting. So for those that may not know you, Bob, thanks. Tell us a little about yourself. Yeah, I'm, so, my name is Bob Miller.

I'm the Chief Operating Officer of Global Data Systems. We are an M-S-P-M-S-S-P and Lafayette, Louisiana. Um, and as Chris said, we're also partners with quite a few companies, but we're partners with Solis, um, from a MDR standpoint. So that's the connection between Chris and I. We, you know, we know a lot of the same people. I'll, I'll put it that way. Yeah. All right.

And according to Ed, your, your photo or Chris's photos in every post office, and I don't know if that's a good thing or a bad thing. All right. Um, alright, Chris, so let, let's just level set about this vulnerability, you know, cv uh, CVE 20 24, 4 0 7 7 1, you know, a 9.8. Um, talk to us a little about, get everybody on the fairway. It's easy to Google, but what, what, what is it about this Veeam vulnerability that makes it, you know, so critical in terms of its rating?

Yeah, so it, it basically allows, uh, anybody, any Veeam system has that vulnerability. It allows, uh, anybody, and it, it doesn't appear that they need to have a great bit of knowledge to start launching commands at will with, with admin rights. So, I mean, it's the worst of the worst. I mean, the vulnerability's there, the exploit information's out there. So once somebody finds this, it's easy for them to take advantage of and do what, do what they want.

And, and it just to, to say is what we, what we are seeing is that they're using it to delete backups. Yeah. So that's the, that's one of the hard, you know, over the years it's, we've done a better job. We collectively as MSPs and, and IT people have done better jobs of securing backups. But here's the, this is the one little piece that, you know, people thought that they had immutable backups, but threat actor gets in, they're gone, gone forever. Yeah, absolutely.

Uh, any, any thoughts of, are they using it for extortion to Chris, do you think there as well? Yeah, yeah. So what we're seeing is it's the ransomware groups that are taking advantage of this. So obviously the first thing they wanna do is delete backups. So they're deleting backups and then encrypting, encrypting data and exfil trading data. All, all those same, same things.

And so, um, you know, when they, when they delete those backups and they encrypt your data, you're kinda, you're screwed. Yeah, that's right. Yeah. Yeah. You're, yeah. Yeah. So, um, let, let's talk a little about, we, we were hoping to get, um, you know, Sophos had some really, they were quoted quite a bit. They've got a really good IR team. Um, we're hoping to get Linda Smith on who, Chris, you did a panel two years ago at write a Boom with, she's awesome.

But she had a class she couldn't attend, but, you know, they were quoted 'cause they're, they're seeing a lot, uh, quite a few cases. So talk to us a little bit about initial access here, Chris. We're seeing VPN as, you know, surprisingly. Yeah, yeah. Um, as one of the main ways in which the threat actors are, are any, any thoughts on, is it just easy pickings? And can you unpack that a little bit about why you think Sophos is seeing that as the kind of the initial access point? Yeah.

Well, I think number one, it is easy. Pickens, I mean, they're finding either firewall vulnerabilities or VPN credentials that are, makes it easy for them to get in. And the ones that we're seeing is the firewalls weren't updated. And the, and the issue that they have, the vulnerability they have allows you to bypass MFA. So even in the case where they had BPN with MFA, the threat actors have figured out that they can use this vulnerability to bypass it. So thus they're into the network.

And then once they're in the network, and this is what we've always talked about about patching, is you gotta be patched. It's usually not the patches that allow the initial access, but it's the lack of patches that allow the threat actor to kind of do whatever they want once they're in the network. And this is a, a perfect example of this situation where they found a, you know, easy firewalls to get around and they just weren't updated like they should have been.

This wasn't a case where it's a zero day and there wasn't any type of mitigation that you could taken. And they've gone in there and then they found this vulnerability and they said, well, I don't need to crack or do anything with the beam password. I'm just gonna use that vulnerability to get into the system and delete backups.

And then they can just go off their, they, and they can do the rest of the stuff that they normally do, uh, and the rest of the environment to gain access and, and do the exfiltration and the encryption at that point. Chris, we talk a lot over the years about, um, access brokers, um, ms the, the, the ransomware gangs have been able to move very rapidly on this.

Um, so I'm curious, how much is the ecosystem, you know, the threat actor ecosystem at play here, where, you know, we hear about a vulnerability like this. Is it, you know, do you as your hypothesis that, hey, like let's go to the access brokers, who's got VPN creds to, you know, get into set amount, you know, companies that we feel have, uh, Veeam, you know, what, what, what's your thoughts on how this is working maybe from an ecosystem, if you had to guess?

Yeah, I think in, in, in, in, in what we're seeing in these situations is that I think, uh, e either a ransomware group or an access broker discovered maybe, um, that there was a consistent way of getting into a, a number of clients, several clients that belong to one MSP. So we have seen cases, especially in the last 12 months, where, um, how that initial access broker gain that information, we don't know, but they've gained in information around an MSP in their clients.

For example, we've seen it where there's the, the same admin username and admin passwords used across client. So an access broker has figured that out and then turned around and probably sold that access or whatever. And what we're seeing in this particular situation and, and that the cases we've seen seem to have a, a common point and a common thing.

So if I was to hypothesize about it, I would say this is probably a situation where the MSP was really good about standardization across their client base. So their clients all probably run the same, same SonicWall or similar SonicWall, and they all happened to not be updated. So that was how they figured out how to gain initial access across the board into these multiple clients.

And then they were able to take advantage of the Veeam exploit because probably, again, hypothesizing the MSP is standardized on beam and they haven't pa it hadn't passed or updated mitigated across their client base. And so that's probably, you know, again, this is complete speculation at this point, but this is what I think, think happened in this case.

So it was one of these situations where, I don't think this is back in the day where we saw people specifically targeting MSPs to take advantage of this. I think somebody found this information with one MSP and found out that it had a one to many effect, and then somebody hasn't since taken advantage of that. It was just, again, easy pickings. 'cause they could use the same recipe on each of these clients associated with this MSP. Yeah. Interesting. Chris.

Um, so in terms of, you know, I, I mentioned this earlier a little bit, but in terms of really attracting ransomware actors, is is ransomware gangs? Mm-hmm. Is the fact that, you know, we're talking about backups, um, and that we can do double and triple extortion, you know, anytime we're talking about data. Is that, is that a big reason? Do you feel, or a reason why the ransomware gangs are like, you know, spun up pretty heavily around this one?

Yeah, I mean, in the, the, if you can exfiltrate data, I mean, that's, that's their strongest leverage they have, right? But, but then again, if they can make it to where they have to pay them to decrypt the data, that's even better. Right? I mean, the thing about it is, there's really nothing dissuading these, these ransomware operators from doing their work. I mean, we're seeing even more so today where people are like, Hey, look, I'm not gonna pay these criminals.

Uh, I mean, they got my data. I get it. But, but these ransomware operators are, are, are, they're going to different links to really just put the pressure on these victims. I mean, they're, they're creating websites instead of just putting stuff on dark websites. We're seeing 'em creating public facing websites now that have that data on there. They're reaching out to their clients, you know, if it's healthcare, they're reaching out to the patients.

I mean, they, they're, and they're not, it used to be we could see 'em maybe try this for maybe two to four weeks and give up, but they're not giving up and they're just, they're just going and going and going and doing everything they can to, to force somebody to pay. And a lot of times people are like, is this guy ever gonna quit bothering harassing me? I'm just, I'm gonna entertain paying him just so he stops harassing me. And the threat actors have figured this out.

I mean, they're, there's, there's no risk to these guys getting caught or, or anything. So they just, they just stay after it. It's, it's, it's the worst I've seen in a long time, especially with the harassment side of things. So if they can get in, destroy backups, take data, and have you basically, you know, in their, in their fist squeezing as hard as they can, it, it puts people in a really difficult situation, um, that really probably had no motivation to pay whatsoever.

And that kind of getting in a, in a stuck in a corner, it's tough. Of course, you know, I, I've been fortunate to be, you know, in your proximity when you've negotiated, obviously not knowing clients and this and that, but it's an art, which you have a very amazing skillset to do in negotiating with threat actors. It, you know, I'm just curious, has it changed?

Are they, 'cause you know, you, you've established yourself over the years, they interact with you because you are of your word that you'll get things done and, you know, there's certain code, but is it getting more ruthless, like, just based on what you're saying there? And Definitely more ruthless, definitely less organized.

Meaning like before, you know, a couple years ago before the bigger groups were either being quote, shut down or whatever you would, there was a, a little bit more uniformity in how things went and what things were said. And you kind of knew like kind of lower and upper limits and all that type of stuff.

But, um, you know, the negotiate, this is why we always say don't try to negotiate yourselves because you have to build a rapport with these people at the individual level, and you gotta feel 'em out. And you can't just use like a, a script to do it. And we've seen people try that, or we see people try to, you know, be rude or demeaning or whatever, these threat actors. And that doesn't help the situation at all.

We've seen people who are like, they don't talk to them at all, and they completely ignore them. And that, that's a bad news. But yeah, it is definitely getting tougher because there's, there's not these threat actors when they're negotiating with you, in my opinion. They don't have as many eyes watching what they're doing as we did in the past. Like, you know, these ransomware groups of the past. Mm-hmm. They would've people and seemingly monitor these conversations. And you could see it too.

You could see where maybe the conversation's not going the right way and you could tell someone else has come in because the tone has changed, the vernacular has changed, those types of things have changed, uh, but not so much anymore. And, and it's just a, it's just, yeah, it is weird. And, and with, again, with like take lock bit as an example.

So lock bit, you have, if lock bit hits you and you're negotiating through their web portal, that's, you're probably negotiating with what is Lock bit. Then you have other people that you're negoti that have attacked you, and they're negotiating through email. That's just people using lock bits, tools and methods and, and, and malware, whatever you wanna call it. That's that. So even though they both look smell and feel like lock bit that from a negotiation perspective, they're different.

So they're not, so yeah, it's, it's, it's, I would, I don think any ransom negotiator on here would say it's, it's gotten more complicated and more challenging, because again, you're dealing more at an individual level than at a group level.

I got a, I got a theory about that, honestly, is I, I think what's happened is, is that what we're seeing is, is that malware and or extortion software as a service, that that market is maturing and there, and it's going down market so that less and less experienced people can use the system. Right?

Um, and like Chris was talking about in the very beginning, they were very careful about monitoring those communications because it, the value of their framework, their malware framework was based on people actually doing what they said they were gonna do right. In the early days. And we're getting, we're quickly getting beyond that now. This has gone more down market. People are just paying so much per month to have access to the tools.

They're not as refined or as educated as the other people that used to do it. So I, to me, this is just a clear indication that, that, that, that market's maturing, the, those frameworks and the people that are using them have changed just like we would expect 'em to over time. Right. Less professionals doing it, more amateurs getting into the ball game, which is why we're gonna see a whole bunch of variation on a go forward basis. So that, that's my, that's my take on it. Yeah.

You bring, you bring up an excellent point because you think about the phishing kits out there today and how, you know, we're seeing the continuation of, um, uh, uh, att um, uh, adversary in the middle attacks with M 365. It's Yeah. You know, relatively an e low talk about low barrier entry, I mean, and kit it up and reasonably priced. I mean, um, so yeah, we Also, yeah, we also see sloppiness on the encryption side too.

So we've run into more issues decrypting data when people need to decrypt, because, you know, you know, a couple years ago you run into issues. There was a, there was a escalation path, and those ransomware operators were, they wanted to help you. They wanted to, to be successful. They didn't want the reputation of having bad encryption. I mean, lock Pit was a great example. They used to brag about how fast they decrypted and how quality it was and everything.

But we're starting to see where just the, the actual encryption process itself and the encrypter, it just corrupts files to where they're not decryptable. And so yeah, it's, it's, it's gotten a lot uglier. Yeah. Yeah. I love all the comments, right? Eric's, Eric, and both Eric and Steven were talking about the lower the barrier of entry, but commoditized crime, that's exactly what we're seeing with that. Just like Eric said, we're getting to the point where that framework's commoditized.

So I agree with him a hundred percent. Yeah, No doubt. Um, so Chris, you know, you've shared a bit about, um, you know, what your team is seeing with this vulnerability is, you know, is is this gonna be one of these, you know, ones where, you know, pick a, a hard internet facing hardware vendor where, you know, we've seen a lot of that where, you know, um, via things like Shodan. Mm-hmm. Right? It's, you know, the gift that, that keeps giving.

What do you, what do you make of something like this? Is this gonna be one that we're gonna see for the foreseeable future? Um, or, or is this one that's, you know, just, you know, we're seeing it right now. I do think we'll see it linger, and I don't know why. I mean, it's, um, again, I've been thinking about this one a lot. Like you say, Hey, look, I'm a, I'm an IT person, or I'm an MSP, and I'm looking at where this is, goes back to this kind of data flows. Where is data flow, right?

It's going to the backup system, right? How do I get to that backup system? I mean, what's, what, what's the attack surface? What's the, you know, I mean, to me, again, there's two sides of the story, but on this one I can't see. It's like, how can you not as an MSP, when you see a 9.8 and it's Veeam, how do you just not jump all over that and say, Hey guys, tonight we're gonna be patching the living hell out of ve Yeah. And then same thing with firewalls, right?

I mean, I've been talking about this for a while too. It's like, and we talk to our clients, I'm like, Hey, look, there's gonna be situations where there may be a vulnerability that something happened has to happen in the middle of the day, and we have to disrupt business to update this or to do whatever. I mean, that's just the reality of things.

And so when you see something like this with both two critical vulnerabilities, one at a firewall level, and one at a software level of two critical platforms, I mean, it just is, is it, it boggles the mind, but that's what makes me lead to believe this can happen at this point, then this is gonna be those things to linger. I mean, hey, you know, member of the proxy shell, we just had another one of those, somebody spun up an exchange server and didn't patch it.

That just spun up a brand new exchange server didn't patch it. And when the guys got in, there're like, this can't be right. This can't be, this can't be this thing happening again. It's two years ago. Yep. There it is again. So, yeah, it's crazy. But I, I, I do see we're gonna see these things linger for a while because I just think people with backup systems, I, I think at one point to, to their defense, they're sometimes afraid to touch them 'cause they want to screw 'em up.

But at the same time, they're also, maybe they don't view them as production and maybe that's why they put 'em on the back burner. I don't know. I mean, it's just crazy. So, Bob, let me ask you a question as we turn over the questions to you. Yeah. Chris mentioned something, and I gotta wonder, is it, you know, something Gary Pika talks a lot about with roles and process is, is that where things get, like Chris is sitting there going, how is it that a 9.8 slips through the cracks?

Yeah, well, look, I mean, it's insane, right? I mean, because now, you know, we're an MSP, but we picked up on that vulnerability as soon as it was advertised, right? By the way, we, we use Veeam just so you know. So we saw, it immediately came out because we run a vulnerability management program, right? And as soon as that CV was published, our entire, all of our dashboards lit up like a Christmas tree saying, Hey, this is a critical change that's gotta happen inside.

But we're the same way with our firewalls. Like Chris was talking about. We don't play games with firewalls either. I don't care if it's a critical update, we do 'em, then we don't wait to think about it too much or anything else. We, we, we do those updates right away. And you know what? We've done a risk assessment for ourselves and we know which ones are our critical systems. And so we treat 'em that way.

So when we see these that are related to those critical systems, we act like we should act on those. And we also have a very mature act of directory and roles, you know, roles and responsibilities matrix, so people can't get access to things they're not supposed to. So, I mean, it's a layered thing. What will it linger? Yeah. Because people are just not very disciplined about the layers in their security strategy. I mean, you know, we talk about phishing being the damn problem.

I mean, that's still a problem after all these years, right? That, That, that's my point is, is it, yeah. Is it because does it come down to route roles and pro, you know, again, maturity roles, process. Is that where things are falling down here? When you see, you know, like Strand will always talk about, give me, gimme an incident and I guarantee it's gonna be more than not basics. It's gonna be fundamentals.

It's not gonna be some crazy thing that couldn't have been prevented as we know from this. Yeah. I mean, you know, it's an operational maturity level thing, right? I mean, you, you know, at some point you, you have to change your focus to being very rigid on those things. And that's when you start really improving from an operational maturity level. Yes, you give up a little bit of flexibility, but in, in, you know, in response to that, you get a lot more security.

You mitigate a lot more risk, right? And, and my, that's, that's my take on this, is that you just have to be very disciplined about how you operate in MSP and it requires a certain level of attention. Um, and that means internally and externally, uh, that leads right into this next question, by the way. So I'll, I'll ask Chris this question. Go ahead. Because it leads right into this. So, 'cause I, I know the answer to this, but I would love to get Chris's take on it.

So, Chris, from an m MSSP standpoint, I know you work with 'em, you, you don't, you know, you don't do a lot with 'em directly, but you understand that ecosystem. How important is, is it for them to understand their customer's third party vendor relationships as it relates to them, right? Because we know what we are as an MSP, we know what our vendors are, but we, how important is it for us to know what our customer's third party software ecosystem is, you know, is what it has in it?

Yeah, no, it's, it's incredibly important to know what it has in it. What, I mean, we see a lot of situations where we're in a, we're in a, we're in a recovery efforts. We, the MS feed knows, knows quite a bit. Other third parties that are there for maybe a piece of software they support or a piece of hardware they support. And you can see right there, there's just like, Hey, you know, we don't have anything to do with that. And then the client's, like, they, they don't know either.

And so we're trying to figure all that mess out on who's responsibility is what. And in an incident situation, we want to try to limit the amount of people that are involved in that situation for, you know, legal reasons and confidentiality reasons. That's right. And so, you know, the, we're not asking the MSPs to substitute for those people, but we need to at least to know like, hey, you should at least know points of contact what software is being used.

You know, is that, is that third party, uh, one of these third parties that's gonna be cooperative or not cooperative? Uh, you know, a lot of times we find out during the incident that the customer hasn't been renewing the maintenance and support with the third party. So when something does happen, then we're, there's nothing we can do like the right, you know, or the software's so deprecated and the MSP learns of that during the incident as well, as opposed to knowing that ahead of time.

So it's incredibly important. Well, Chris, we saw this with, we saw this with Screen Connect. I mean, literally out of your last statement, we knew people were running deprecated versions and not paying maintenance. And, you know, kudos to Patrick Beggs for, you know, saying, we don't care and if you paid or not, we're gonna give you the update all to, to, to, that was great. Move on their part.

But you know, this, this is the, you know, this is an area where, you know, if you had Ryan Weeks on today, you know, another area of process, Bob, is about, you know, who are third party, you know, really having detailed inventory. And sorry for, lemme just digress here.

As Brian Blakely said, looking at your customer's contracts and like, who they're top cust, you know who they're doing business with because you might find out, oh wow, you know, this is, you know, how you're, you know, your top customers and this is what they're expecting, and oh, they're using these systems and or you're using this line of business app we're not supporting, and these are the critical remote control tools that are using, or these are other remote, like, those things are so critical, right?

But it comes back to, as you said Bob a minute ago, is maturity, right? To be able to do that? Yeah. Yeah. I can tell you from a practical example standpoint, when, when SolarWinds was happening, right? A few years back, we, we knew which of our customers were also using SolarWinds. Um, so the first thing we did was start notifying them about the weakness.

Even before it was really announced publicly, we were letting 'em know that about the, you know, the back channel information we were getting related to SolarWinds because we knew they used it. You know, we're, we're kind of a, we work with mid-market enterprise customers, so there's a lot of co-management that goes on for those bigger enterprise customers. And so they run their own software stacks, right?

And so anybody that we are very familiar with our customer's software stack on critical systems, whether they're using the same thing as us or not is not really relevant. But if they are, then when we see things that are vulnerabilities and we see 'em before they're actually public, we call 'em and say, Hey, by the way, we're getting a lot of scuttlebutt here. We need you to be paying attention.

You might wanna start thinking about taking some pro action, even though we're not managing that for you. We know it's in your environment, right? And it can cause some real damage. So that's, that follows on with Screen Connect that follows on with any number of, uh, RMM tools, you know, that are out there.

We know which RMM tools people use because when you get right down to it, there's about a, you know, half a dozen pieces of software that really have the run of the room in, in, in MSP and or an enterprise. And knowing which categories those are and which softwares in 'em really critical because you take on a lot of risk when you take on a customer, you're taking on their ecosystem. It's, it's like a girlfriend, right?

You, you're taking on everybody that everybody's known on both sides of the partnership when you do these kinds of things. So being very familiar with what they have in their ecosystems, hypercritical for an MSP to even protect themselves, right? Because you are, you are uptight and friendly with 'em in a lot of different ways from a technology standpoint. So that's, that's just part of, like you said, goes back to operational maturity levels.

Um, I got a whole nother rant about unsigned DL dls. I'd love to talk to Chris about one of these days about the same sort of thing about the lack of discipline about these, you know, about these systems, right? But, uh, I'd just like to take this to a whole, you know, we could have the X-rated version of the cyber call after the Oh, I could, yeah. I'm ready for it. I can tell you went the girlfriends man. I had, yeah, I pulled that back. I pulled that back. That was about to go too far.

So I Pulled back. You almost, you opened the doors wide open for Chris. No, no, I got it Outta there. I shut up. I shut up about that. So I had good sense. No, Chris, Go ahead. No, I was just gonna say on this, you know, we, we still, you know, we have, we have cases we have today where companies have multiple offices and there's a DI and they've chosen to use an MSP that's local to each office, but they're on the same network.

And so that thing makes things real interesting too, because you don't have complete visibility and, and then a lot of times you have pointing fingers and now you're trying to, now you need the co you need the cooperation of multiple MSPs when you're dealing with an event slash incident. So yeah, I mean this, um, I think this third party risk to your client is probably more of a risk to you as the MSP when it all comes down to it.

If you're just playing, if you're playing the ignorance card, um, or whatever, or if, if for some reason the client's not sharing the information with you, you really need to press forward and get that information. Quick, Quick question if I may. Chris, what about our vendors here? Should, you know, could a vendor running one of our big vendors running Veeam and they're, they're, they happen to get owned? Like what could the implications there be?

Because we're gonna have, you know, incident notification, like, walk us through that. Is that something we need to really think about here? Yeah, I mean, if you had, like, you, you could take the example from, um, black bot from a few years ago, right? Mm-hmm. Where they got in, they, they, they sent notifications out and said, Hey, no, everything's fine. That data was encrypted, meaning they encrypted the data, Blackbaud did, it's been safe.

And then they had to come back and release a new notification saying, oh, wait a second, we had all the backups and the backups weren't encrypted, so therefore your data was not as safe as we initially told you. Right? So, yeah, I mean, if, if a vendor gets, gets popped on this type of situation, it it, 'cause look, I mean, they deleted backups, but that's not what to say. Somebody couldn't take the time to exfiltrate the backup data.

And potentially, depending on how it was configured, I, I can't go down that road 'cause I'm not a Veeam expert. But I mean, there's a possibility they could have just taken the backup data itself. One of the examples I like to fall back on was, one of the earliest cases I remember having was an MSP who did backup as a service.

So they had clients that they only did backups for the threat actor got into their environment and deleted the backups because they were backups that was the only copy of that backup data for their clients. So in essence, it was production data, if you think about it, right? That was production data, the production, the data of the business being backups. So, you know, 2020 hindsight, but they should have been backing up the backup data.

So they, they should have had an immutable copy of their client's data in case that happens. And so it, it gets pretty ugly. So if a backup vendor was to get, um, or somebody that uses Veeam, 'cause there's cloud providers that do Veeam and everything like that. Yeah. It, it gets, it gets pretty nasty in this third party stuff on who's gonna do what and when and to what degree. Last comment I'll make Bob and back to you. I mean, sounds a lot like Snowflake to me, Chris. Sure. No Snowflake guys.

Yeah. Yeah. I mean, Yeah, that's a little, yeah, that's a, that's a, that's a one in itself right there. That's a difficult one because you were really, I don't know, there's, that situation could be handled a lot differently too, in hindsight. But, you know, it's like, it's like the screen, it's like the, uh, change healthcare thing and, and ConnectWise taking the hit on that, when it was even that Was, that was pretty s****y.

Even when the change healthcare guy came out and said, Hey, it wasn't screen well, I'm gonna tell you it wasn't screen connect. And so those Yeah, I mean, so it, yeah, so I think from a vendor's perspective, uh, being as transparent as they can and, and if you are using a third party for backups, I mean, this is where I tell MSPs, your client should be asking you these questions. Like, what are you doing to make sure that this stuff is secure? Yeah.

Because I mean, if you look at the exploit this thing, I mean, you were asking about vendors. If they're running Veeam, what, what could happen? Well, I mean, if we wanna be theoretical about it, this particular exploit allows them to escalate a user that automatically gets created as part of this exploit. And it's, and it's the user is point and it's got administrative capabilities. Well, if you're not the ransomware type, let's go back and look at what they did with SolarWinds.

They went and got in into development chain. They were a lot sneakier, right? So they went and duplicated a certificate server so that they could run amuck and move laterally inside of that network. And then they got in the development chain so that they could put, they could mule in their malware as part of the golden image of these updates going through. So, I mean, once you're in with elevated credentials, all bets are off.

You can do anything and everything that's ever been done on any network, regardless of whether you're a vendor or a enterprise or an MSP or anything else, once you get an au once you get an authorized account inside of the wire, all bets are off, right? I mean, unless you've got a layered defense, you're gonna get your ass handed to you. That's, that's what's Real. And we only know what we know, right? I mean, yeah, we know that companies of all sizes use Veeam.

I mean, it's not just specific to the s and b space. So I guarantee that, Bob, just what you brought out, I mean, we're talking about these ransomware guys taking advantage of it, but you damn well know nation state guys have been in there and A hundred percent. Yeah, no, tell 'em what they've been doing. They, They, they, they probably packed muled in all kinds of toolkits and laid 'em around right. For when they really want to get in there and mess things up. So yeah. That's huge.

Well, you know, and again, goes back to our, I like how these things just flow into the next question, but, you know, one of the, the rapid civic guys are basically saying that put 20% or more of their cases in 2024 related to this Veeam, this Veeam ex exploit. And I, I'm kinda like, you know, I'm kinda like Andrew on this. That seems like a hell of a big number. It does seem oh, A hell of a big number for, for that. I mean, if, if, what's your perception of that? I mean, that seems high to me.

And this is published Bob. Yeah, Yeah. It's published. Yeah. I, and it's 20% of their cases, so I don't know what that means. Right? Yeah. So it's tricky. I mean, they're, could be, they're kind of a big shop and handle, they, they have the tendency to handle bigger cases, so it's very possible that that isn't, that is it? Right? Because they're like the majority of their cases run vem. Yeah. And so maybe that's just, it makes it super easy to delete.

I would, I mean, coming from the banking world and coming from our change management process, and I mean, our backup systems were incredibly crucial. Uh, but it took a lot to update 'em. I mean, it was pretty involved back then, but it was a different deal. We're in a mixed environment. A IX windows, we had multiple, I mean, there's a lot to it.

So, uh, but I can't give that excuse for this vulnerability 'cause it just seems like this vulnerability and that it sounds like, Bob, you, you went through the steps. Yeah. It doesn't seem like it was that difficult to No, no, no, no, it's not. The fact is it's kind of a normal day at the shop. I mean, honestly, you get, you get firewall updates all the time that are hypercritical, right? That, you know, have a real exploitable situation. You gotta deal with that now.

So, I mean, it's just a maturity thing again, go ahead. But you I was gonna say, Bob, my sense is you have policy creation with your That's correct. Customers. And in that policy, there's some language that says, Hey, based on this level of vulnerability, that, and it's, we will disrupt your business. And Yeah, we let 'em know up front, right? We definitely do, especially on firewalls, because firewalls are kind of a special critter as well as VPN concentrators.

Those are both you, you know, we can, routers, switches those things. We can, we can take a little bit more of a, you know, a longer view on, but not VPN concentrators and not firewalls. Those things are, those are, those are critical gateways into any networks. So you have to, you have to jump on those quickly. Yeah, but my my point is there, there's policy in place. They know that they can't say, well, we shouldn't do this, or No, we agreed. Right? We agreed. And this is, that's correct.

You, you know, so anyway, Yeah. Our, our patching policy is actually, part of it comes when, when you get our, when you, when we do, when we establish a relationship with you and we do a scope of work on managed services, our patching policy is actually included in that scope of work about what we do and don't do, right?

Because we have a couple rules about, we don't stay on the, the absolute current release of router software because typically new versions of router software have more exploits than, you know, two back. We use the two back rule, we stay two back of the current version. Um, and that's where we're by default. And we'll keep you at that rate. You know, now, in some cases, we'll keep you a little bit further back if there's been no actual patches of substance.

So I, we could talk all day about Chris, I bet we could talk all day about patching strategy, right? And how to really accomplish that, you know, to, to maintain an operational maturity level. 'cause there's some very, there's some subtleties there, There, there are some we can kind of test on, right? As MSPs just isn't practical. Uh, but I would say what, what, and, and while I, I still hear this today, is a lot of MSPs when it comes to the patching that's automated, they do that all day.

But when it comes down to stuff that they have to manually do, they, they just find excuses to put it off. Yep. Well, we, we, we honestly, we wound up having to build our own system, um, to, you know, things like Ansible and other techniques we got, we actually built some DevOps software to help us with those auto automate the updates of the ones that are not automated on their own by the vendor for the very reason you're talking about keeping up with it takes work, right?

And you're either gonna throw people at it or you're gonna throw technology at it. So it's one of the two, but you got, it doesn't matter. You have to do it. You, we, there's not really any room to get around that anymore.

So, you know, it, this comes back to the age old question, how in the hell are we gonna get, you know, developers of these, uh, you know, developers and operators and researchers to all work better together about releasing the information as it relates to these exploits, right? I mean, I think some of this is, you know, I still don't feel like there is an organized structure way to do it.

I know that we see CVEs, but I think there's a lot of work that goes on before it even makes it to the CVE level that really we don't have good visibility in. What do you think about that, Chris? No, I, I agree. I mean, I don't, um, I mean, it, it, you know, I mean, there's just not, I, I still don't see like people thinking about this from an impact perspective, right? Right. So it's a cv blah, blah, blah.

And I don't know, on this particular one, when people are like, well, that's behind my firewall. It's not publicly facing. I don't need to worry about it. I just, there's so many, it'd be interesting to maybe put these guys in the room. Their, their, their voices changed and we darken their face and go, what the hell were you thinking? Be honest with us. Right? Right. Yeah. But you're right.

I mean, on these things, I mean, there's just not a, there is not a system or process or anything that I guess lights a fire. That's what we're trying to get at. I, I just, There's no sta you know, Aaron talks about this in his years about, and about security too, though, Bob, he talks about, you know, in it, there's standards in security, there's no standards.

So your VDP for your, for this company, vulnerability disclosure program for this manufacturer can be drastically different from another, the threat researcher, you know, the, the white hats that bug bounties and things of that nature. It, it's, it's, you know, to your point where we could spend all day on, you know, a topic, this is one where we could spend, you know, oh, uh, you know, hours, you know, debating what's the right way, the wrong way.

But, um, you know, there's this sense of do we, you know, hey, we want the glory that we found this and publicize that, you know, we, we were able to replicate the exploitation and the research and notoriety, uh, versus allowing, you know, the manufacturer to, you know, the necessary time before disclosure and should they up, you know, notify everybody. And, you know, it's, it's certainly not an easy subject For sure.

No, I, you know, I mean, I, when, um, when I first joined up with global data systems, I spent a lot of time looking for, 'cause I really expected to see that there would be a lot of work out there around, okay, what's the right sort of discipline related to patching and frequency, and why do you choose doing it this way versus other ways? And the thing I found out right away was there was none of that.

Nobody wanted to take, nobody wanted to take responsibility for putting any language in the universe that said, Hey, this is the best practices. Right? Because that's how scared everybody is of the process. And some of it's what Chris is talking about. Not everybody does it. You judiciously, right? And, and the only way if you put your name on something and say, this is how we do it, and then you don't do that, then you take on risk, right?

So I started understanding why there was not a lot of it out there, but I think we gotta get ourselves past that point. I, I really do. I think we gotta be able to call out, just like Chris, Chris is saying, we gotta be able to look each other now and say, here's where my head was. And then we go, well, that's not smart. Let's look at it. Let's look at it from these other angles and figure out a better solution.

And by the way, this isn't this, there's a lot of this that rubs on my wrong bone specifically about, you know, we're talking about software and how these vulnerabilities show up all the time, but we're also in, in a lot of our webinars and a lot of sessions like this, we talk about regulatory bodies trying to force the MSPs to be responsible.

And, and what they're saying is you need to be responsible for the global environment of non discipline as it relates to the damn software development processes that half these people are using. Right.

And that's the part that really burns my ass, is that, okay, you're telling me, is the MSP in the middle that I gotta be concerned about what third party software I use when we all know that the discipline at the software development level is just not being followed to, to the best practices standpoint. And there are a million examples of that, right? So We do have an executive order now, Bob, that's gonna fix that. Yeah. Okay. Yeah, I get it. Yeah. Yeah, exactly.

It's like the F CCC saying, your IP phones need to tell you where they are all the time. Well, they're in buildings and they can't see you, they can't see satellites. So you can make that rule all you want, but the universe is telling you no, it doesn't work that way. Right? So we running that all the time.

It's interesting that in what, what degree, I mean, I saw an article I think last week, uh, was that there's still, you know, 86,048 that are vulnerable that haven't been updated from something from months ago. I mean, it's, you know, I mean, it's, it's just, just, it's just crazy.

So, I mean, to what, and, and, and I'm not a fan, like, you know, during that, we, I brought a proxy log on and proxy shell, I mean, there were reports that the FBI was updating systems for people whether they wanted it or not. Right. Um, I'm not a fan of that either, so I don't, you know, I don't want the government in my stuff. Uh, but what are you gonna do? I mean, it's a, it's a tough one in this, in this Veeam situation.

I mean, again, kind of hypothetically though, if I'm a customer and I get popped, and I know that this was something my Ms you know, we talk about a lot of things aren't the MSVs fault, but you have a firewall and a firewall and a system that's not patched, not updated. The Ms P knew well damn about it. They gotta own that. That's absolutely right. They have to own that part. Not, not the flaw in Veeam, but the fact that they didn't respond, they gotta own Absolutely. Exactly. Right. Yeah.

Yeah. Well, we're gonna, now that we're gonna go from, you know, the aggravating to the entertaining, because I really, I, I read this article, you know, but I love Andrew on what he throws out here. But this was perfect. This is a good way to end it up because this is gonna be interesting. It's operational maturity to personnel maturity, right?

So, you know that the article came out a few months ago where a, a company, I'm not gonna name a name, uh, turns out had and, and a company who should know better. I They should know better. I Mean, they, they should know better, right? But they came out and they had actually hired an employee from North Korea, right? There was a employee, or, and I, I can't remember if they wound up it was a contractor or, you know, or employee proper.

But anyway, inside of the wire, individual, Employee proper, if I recall. Yeah, that's, that's what I thought too. So, Yeah, I mean, they shoulda known better and I'm, I'm surprised this didn't happen to them before. Yeah, Well played Chris well played. It's only they'd known about it before. Yeah, I get you. So, uh, I don't even know where to go with that particular, so, I mean, um, so this is the first one that I've seen where they'd been there quite a while too, right?

So they had the opportunity to actually do things like exfiltrate data and, you know, download whatever information they wanted to 'cause they were inside the wire. Right? Go ahead, Andrew. Well, I maybe just set the stage for everybody. There's an article out there now about, and I, and I can, I'll put it out there. Ironically, it's a secure works. You're just a, now it's being acquired by Sophos, but, uh, but I'll put it out there. Yeah, I just put it out there for you. Oh, Okay.

Thanks Chris. I hope out there. Yeah. Okay, perfect. But go ahead, Bob. That, that This, no, yeah, that's, I mean, in, in reality it's like, well, because I, I I, I thought back to my years of when I was, um, having to hire a bunch of people and the amount of background checks that we did that had to be state and federal cleared.

And in some cases we work with, um, you know, we work with law enforcement agencies, so we have to actually get like federal clearances on people to kind of do some work. But I thought this was interesting that this particular group, whatever their policy policies are right, obviously didn't cover the, the right level of background checks. Chris, have you ever heard of any other applications where that's been the case where it turned?

Well, I noticed that there have been some, I, I have seen some articles about Chinese nationals being hired into organizations. Right? There seems to have been a lot of that recently as well. Um, this one hits kind of close to home because of the company that he was a part of and what that actual company does for our ecosystem. Yeah. But have you heard of other, have you heard of other transgressions like that? No, I mean, yeah, I, I, you know, I, it's an interesting, right?

I think this is a byproduct of, as you get bigger, I think A, you get bigger and B, you're looking for cheaper talent. Right? And so, I mean, you you, this comes up in discussions all the time. Hey, hire people from here, from here, from here. They do great work and get 'em at a, at a low cost. And if you look at in this particular link, you know, it's, it's dev work, right? It's the risk. Yeah. There's some list in there and, and full stack developer.

And a lot of people don't look for communication skills as necessary as a necessity for that job. So you could have somebody that has poor English or whatever and you're like, well I don't care. As long as they can code, I'm happy. And that's one of the biggest red flags, right? I mean, so it's, uh, yeah, this, I think this is just a, this is really, um, it's, it is very similar to business email compromise when I read this, right? It's like, Hey, I'm hiring you, whatever.

Oh, hey, can you go ahead and send that laptop here instead, Right. Check for no problem. Right. And you know, it's, it is the same kind of concept, right? Instead of getting phished for, for banking details, you got phished on from sending, uh, you know, a laptop. Um, yeah. It's, uh, then you gotta wonder like what do you need to do to, to vet that better? I mean, there's background checks and all other stuff.

It just, it, it, it would be, it would be interesting to know more details on how they were able, this person was able to get that far in. Did, yeah. Was it just a human error? Did they miss some steps? Have they added some steps? I mean, it's just, or did they, 'cause they're a foreign national, they couldn't do a background check. 'cause that happens a lot too, right? There's no way to do a background check on somebody as easily overseas as there is here because the data's not available.

So yeah, there's more details to be known on this one, but I can definitely see this being an, like, the extortion tactic on this is fantastic. I mean, you're in there and you're grabbing everything and you're a developer, you could be grabbing Oh, you're gonna get the code base Yeah. Property out the wazoo. Yeah. Yeah. You grabbed the whole code base. You did. They exfil quite a bit of stuff in this particular case, right, Bob? Well, that's what got, that's what caught him.

It was the volume of file transfers. I mean that, it wasn't, it, it wasn't somebody, you know, was asking about how you liked the new 401k you'd just gotten, it wasn't anything like that, that it was the fact that he was downloading gigabytes of data or trying to onto his, you know, you know, onto his laptop and, and you know, that's, it was really, and as it usually is, it's the ones who are not the smartest ones that get caught. Right.

I mean, because if he'd had, if he'd been more judicious, he could have, he could appealed that stuff off in smaller chunks and probably, he probably could have been there to exercise his 401k before they figured out where you know who he was. Right. Well, it's just like the phone companies and that, you know, the Chinese access into those carriers, I mean that's a, like how does that stuff happen?

I mean, some of those guys have, if you look at their security teams resumes, those guys are highly credentialed and it just seems like it was too e too easy for these guys to do what they're doing. Yeah. Yeah. Knowing, knowing what I know about the communication sector, I will tell you this, is that, you know, those things have been built up and acquired and all that over time. So none of it talks smoothly together. And the, the level of integration is low.

So it doesn't, it doesn't matter if you got, you know, lipstick on a pig, it's still a pig. But behind that lipstick, and that's typically what happens in the old telecoms is 'cause they're kind of Frankenstein together, right? So I kind of understand why they get themselves into that situation, but they should spend some of that profit going back and, you know, modernizing that and, and reducing the risk, right?

So it's all part of an, again, these are mature companies, but it's all about money at some point. Yeah. Right. And just for the record for everybody, um, when Bob's talking about lipstick on a pig, he is not referring to his old girlfriends that he mentioned earlier. So I don't want them to Nope, no comment. A bunch of down thumbs on this YouTube from Bob's ex-girlfriends. So I ain't touching that. I'm done with that. Back to you, Andrew.

Alright Chris, so, um, just one more question about this, Chris, because you know, we, MSPs are going upscale, there's a lot more, um, you know, co-managed Chris upmarket co-managed. When you hear something like this, do what steps or should there be steps when we're, you know, interfacing with, you know, larger corporations where we might run into something like this or God forbid we run into something like this.

Do we wanna understand hiring policy in some of these big companies We're dealing with that, you know, we're on the networks of these organizations. Um, Yeah, I, it, the co-managed stuff is interesting because we do, I mean, we have a lot of companies that have us do security, but they have internal IT and, and we do security and, and so on and so forth. And yeah, there's a lot of stuff you need to know. I mean, you do kind of need to know how they're staffed, where people are staffed.

Um, you know how, you know, one of the things we've always done on risk assessments for people is ask about hiring practices. That's really important. You know, one of the first questions always ask is do you ask questions of when you're hiring somebody, no matter what position you ask questions about cybersecurity of them. Like, you know, how, how versed are they even no matter what, they should know someone and have some idea of how to secure stuff when they're working for a company.

So I think, um, it is important to know, but even it's there, there's just so many things that can go sideways and instantly people are gonna point the finger at you, um, as that third party and you're gonna have to defend yourself and, and, and it's, um, yeah. So the more you know, the more you can know, the better you're gonna be. And I think your, that your customers are going to respect the fact that, you know, all that type of stuff. It's just, it's just a, it's a lot to know.

I mean, um, especially if you're co-managing and maybe you, you don't know about every new hire because internal it is handling part of that and you're not or whatever. Yeah. You, you gotta figure that out. I was speci especially if they have developers, those are the people, those positions that scare you the most. Right, Right. So bonus last question, Chris, we'll do a bonus question. You're, uh, now that we're in Q4 2024, thanks as always for joining.

Most interesting case that you've had this year. Well, um, that's a good one. I'll just say we have had a couple of cases where, I don't know if I mentioned this in a prior episode where the MSP owner's phones got cloned. Mm-hmm. No, you didn't. And so their phones got cloned and they had SMS as two factor authentication to some critical platforms. Uh oh.

So when their phones got cloned, the threat actor was then able to basically gain access to these systems, change passwords, change MFA and lock them completely out. Oh, wow. And not only did they have, um, their own stuff in these platforms, but customers as well. So, um, and it's interesting, we can't tell if it could be one situation where a person was, was, was a very specifically, it was specifically targeted. Like somebody's like, Hey, I'm gonna go after that guy.

Versus the other one might have been not as targeted, but they were able to clone phone and just gain and then figure out that it was an MSP. Uh, but those, those are, those are the most interesting ones I can talk about right now. Got It. Went too much in detail, but yeah. Awesome. Well, uh, another awesome session. Chris, thanks, uh, for coming in back to back weeks.

Bob, as always, thanks for filling in as an awesome co-host and everybody out there, thanks for joining us on Monday and wishing you all a safe, happy, healthy, and productive day and week ahead. Look forward to seeing you all next Monday. Take care. Thanks.