3CX Supply Chain Attack

All right, let's Roll. Welcome back, everybody. Uh, happy Monday to you all. I hope it is a happy Monday and you're not dealing with the three CX issue, um, internally or any of your clients, but we certainly need to address it for many reasons. Um, not the least of which it is. Uh, I would probably argue our first major supply chain attack of 2023. Um, and, um, we, let's see, from an announcement perspective, before we get going, Gary, you said you had something? Yeah, I have an announcement.

I have something new for the show. Okay. Spice it up. I got the hype button. So, so how does, how's it gonna work? Uh, yeah, I can just take my level of obnoxiousness to another level. Awesome. So, Patrick, you're in for that today. Congratulations and great, great to join us. We might, we might wanna go forward, measure how many hype buttons get pressed during an episode to Right To Measure Good. How the episode went. Yeah. Awesome. Alright.

Um, by the way, Eric, um, vol, thank you for your comment about, uh, we did send five people, uh, from Write of Boom to John Strand's, uh, core skills. Um, glad you enjoyed it. Who else, um, went or sent somebody last week? Love to, uh, hear it in chat and, and I couldn't agree more. I think it's something everybody should take. Um, get right on into it. I did put a poll up, um, as always hundreds, uh, attending Gary, and we'll probably have three people answer the poll as usual.

Um, not sure what that's about. It's very difficult poll, Patrick. It's a yes or no question. Okay. Okay. Um, so, uh, anyway, so let's get on into it before we introduce our guests. So, um, as I, if you're on today, I'm sure you're well aware of the three cx, um, compromise. Um, and as I shared, arguably the first major s uh, you know, supply chain attack of 2023, CrowdStrike was again, arguably the first to report on it. Um, they did a great job. I will put that in chat momentarily.

And, uh, shortly thereafter, the folks at Total Sentinel One, um, were picked up as well. And, um, because we hadn't, uh, I, I went to, to, you know, someone called out Total's report, and I went and went and looked at it, uh, day one, and it was really well done. And Patrick's the architect behind that. Um, you can check him out on LinkedIn and I'll put his, uh, his LinkedIn as well. He has some, you know, tremendous background in this kind of stuff.

But if you think about it, right, this is an organization three CX with over 600,000 customers, 12 million, uh, end users, not insignificant, and many of which, um, we know are in the MSP ecosystem. MSPs do resell this platform, um, in some way, shape or form. Uh, uh, they are either partnered or it's a direct, um, uh, channel type play. And, uh, it, now we have attribution to the People's Republic of North Korea.

Uh, the threat actor, which I'll probably butcher the name Patrick maybe can help me with, but it's, it, is it Labyrinth? Um, Koma, Kama Labyrinth. Kalina Kalina, okay. Um, or Lazarus Group if you Or Lazarus. Okay. Um, Gary, I'll mention a question like this, but this is the, there are they also, I mean, three CX has some significant clients. We're talking the American Expresses of the world, the Coca-Colas, McDonald's, and others.

So, um, with that, Patrick, um, I'm going to see if I can get this right, but Patrick or Zekowski, did I get it? Yep. Perfect. All right. From who heads up, he's the VP of Threat and Intel at Total. Appreciate you joining us. Patrick, tell us a little about yourself, what you do there at Total, and then I'll hand the reins over to Gary and hopefully you'll make it past the, uh, buzzer. Awesome. Thanks. Thanks for having me. It's a pleasure to be here. Yeah. Um, great to have you.

Yeah, so I'll keep it super brief. I know we want to get into the content. Uh, worked in, uh, DOD in the early two thousands on the, on the blue team side, uh, transitioned to the red team side and the intelligence community. I was lucky enough to pen test cross domain solutions and satellite systems, uh, for several years, which was super fun. Um, then I went on and, uh, was one of the founding employees of Deep Watch, so I did that for seven years until last year.

Um, built out an MDR there, um, mostly serving enterprises, and then came to total mid-year last year, um, to head up and Intel for total here. And, um, you know, uh, thanks for the kudos on the blog. I do want to put out there. It was a, it was a total team effort, um, no pun intended. Uh, from, from the whole team here, detection, engineering, MXDR, the threat team, um, everyone kind of jumped in all hands that we saw in time. It's Okay. You can take all the credit here.

You don't have to thank everybody else As well as external. Take all the credit. It's fine. Gary, Gary, it was, yeah. Yeah. No Lot to learn. Yeah, You're, you're a quick learner. Yeah. I mean, you'll hear about chocolate cake by the end of this, and you'll know that that came from Gary. All right. By the way, um, I put CrowdStrike's URL report in there.

One other I'm gonna put in Gary, is, um, there's a LinkedIn post here, Thomas CIA from mc, uh, Microsoft, um, who I've sent this, uh, um, image out in, in the emails. But this is a Fanta like, if you wanna see really a concise way to look at the attack, he did a phenomenal job on it. Um, so I'm putting that in there. Great to see Bob. Hello as well. So, Gary, with that, um, as always, we start out with the most technical people first in the show. Yeah.

So we're starting, Andrew, I was thinking about this weekend, um, what's, uh, interesting about this. We focus so much on APIs, ms. P tools, all these things, but this just tells us it, you know, how many other things that we're responsible or work or sell that are, can potentially be weaponized, right? So I feel like this one is different, right? It's different than some of the other supply chains and that, that, that, that we've seen, uh, in that way.

And really, uh, before I start the questioning, when we say assumed breach, this is a great example of living in an assumed breach world. Yeah. Yeah. And, and you know, I was saying to Patrick before we got on, not that you want to, you know, there's shouldn't be adoration for a threat actor, but this is fricking pretty ingenious, right? The way they did this. Automatic updates are on Yeah. Download.

They, you know, the, you're gonna, we're gonna talk about how these were, uh, signed still by Microsoft, and I forget who, I mean, this was a, like, this was a well executed attack, thank goodness. The threat teams of all the major EDRs and security companies acted very well together. But yeah, it's, it's pretty frightening how, uh, you know, when you look at the overall supply chain, Gary, what, what, and how they're going to work and continue to pivot. Yeah.

So, well, let's start, Patrick with this, uh, as Andrew said, very sophisticated, right? Um, yeah. Threat actor. Can you take us through at a high level what they did? In other words, how they built out their infrastructure, how they, um, were able to use the updates? Just kind of take us through a high level, the, the, you know, the strategy behind this. Yeah, absolutely. And someone said in the comments, this is, this is our worst nightmare from a security perspective.

And I, I agree with that. I wholeheartedly, uh, yeah. So a high level timeline, high level of what happened, uh, somewhere in mid to late 2022, uh, this threat actor, um, Lazarus group associated with North Korea, uh, that we're thinking, um, started to set up their C two infrastructure. So the domain names that were included, um, and URLs that were included with the malware or the staging site on GitHub, which I'll get into in a second, um, started to be registered with DNS.

Um, so, you know, this goes back months of planning, um, and understanding what the selecting looked like, uh, and how they were going to infiltrate it, right?

So late 2022 in December, the GitHub site that was hosting these icons, um, pretty cool way to distribute, um, your C two, uh, uh, infrastructure, um, they actually appended, uh, encoded messages, RC four encrypted messages to the end of these icon files hosted on GitHub, um, to obfuscate the way that they were going to, uh, distribute things, right? And this suggests two things. They're very advanced, um, and they could use GitHub to update their software, right?

So the initial thought process is that it was an info dealer, um, to exfiltrate, uh, uh, authentication credentials, browsers, um, any, any browsing history, stuff like that. Um, but they could also update their own software through this infrastructure in GitHub. Um, so after, after that, um, kind of D-Day from a, a malware standpoint was March 22nd, um, that's when we at total, um, Sentinel Juan and some of the other EDRs started alerting on this update or package that came down from three C.

When was that? March 22nd of this year. Okay. So that was when we first started getting alerts, malware alerts, right? This was unknown malware, but the EDR was picking it up as shell code injection, right? So, um, you know, this happens right? As, as as it does with UPDATERS and new software. Um, sometimes we'll see false positives. Um, we, we never really assume a supply chain attack, um, right. With with our customers.

But, you know, Sentinel one others started saying this was a false positive. We were blocking it in the software itself, which allowed the software to still run. Um, but we did not assume a supply chain attack. So that was D-Day, uh, from a detection standpoint. Um, we did some heavy investigation with one of our largest partners, um, who runs three cx, uh, for a call center. Um, and, uh, kept, kept working it, um, and, and with the understanding that it, we don't know what's happening yet.

Um, on March 27th, uh, or, or the 28th CrowdStrike posted on Reddit, right? Which became the CrowdStrike blog, I think that was, uh, was linked here, um, and mentioned that this is malicious stuff that's happening, right? This updater is, is malware. Um, specifically there were two pieces of malware that were shipped with the updater and the MSI installer. Uh, one was ffm, peg dll, uh, ffm PEG is a, uh, codec open source codec, uh, library that was compiled and included.

It's included with a lot of software, right? Um, I, I had never heard of it prior, but a lot of developers use it and include it. Um, signal, slack, Nvidia drivers, you know, this, this thing's everywhere. Um, as well as, uh, D 3D compiler, which is a direct 3D video, um, accelerator, right? And these were signed files, as Andrew said, right? The FFM peg was signed from three cx.

Uh, the D 3D compiler was signed from Microsoft, and the actor used software like SIG Flip, um, pen tester software to actually keep the signature valid as it came down to folks desktops, right? So that was the indicator that, uh, you know, this is a really advanced attacker. So, uh, the 28th, um, we started live blogging, uh, our thing, right? We wanted to get it out there.

As soon as we knew something malicious was going on, uh, we started sharing information, whether it was on our blog, Twitter, um, working with other researchers. And as Andrew mentioned, um, you know, the last five days has been an example of how vendors can come together and, and really, um, uh, help their customers. It's not clear you're pulling a thread. You're trying to get to the center of really what's going on, right?

You're, you're kind of just starting to put together enough symptoms to know that there's something nefarious, but it takes some time. Yeah. Yeah. And, and, and like you said, uh, Gary, assume, assume breach, right? At this point. Um, we can't trust the supply chain. Uh, you know, we, we as vendors, we as providers need to understand that, um, software vendors can be breached, right?

Uh, no matter how big, no matter how ubiquitous, um, these things can happen, and we have to, we have to assume that the software we're downloading and installing is untrusted until it's trusted. Um, that's kind of a, a, a mindset shift. Um, and I think SolarWinds really woke us all up to, Hey, this can happen.

We need to be quicker about responding to this and, and put our differences aside and share and be transparent as much as we can with the communities, and provide the IRA signatures, update our software packages, make sure we're detecting and responding to this as quickly as possible before pro proliferates and before, you know, real serious damage is done. Yeah. Listen, and I commend everybody.

I, I I, I, we've seen in the past, right, when something happens, some vendors, their first goal maybe is to use it to help 'em, like with marketing or whatnot, rather than, uh, diving in and fixing it. So I commend seeing the community come together, like this is, uh, really awesome. Andrew mentioned some hit some big folks, right? Coca-Cola, McDonald's, um, B and w you, you don't think my BM W's hacked, do you, uh, Honda? I hope not.

Do you think that was the focus, or do you think this is a grab to get as many credentials as possible? What, what do you think from that standpoint? The motivation? I, our best guess would be, um, that this was disrupted early in the, in the attack chain. Um, I think, I think they went after this vendor because they knew, as Andrew mentioned, that they had a, uh, a multitude of major customers. Um, you know, not just MSPs, although they are a large piece of it.

Um, but major, uh, enterprises, um, you know, you can sign up for a free account. That's how we did our initial research, was we signed up for a free account and started downloading all of their updaters, right? So, you know, this, this software is ubiquitous, and I think anything with a wide software install base is a target. Um, I think that they knew that they were going to get access to these customers through that supply chain.

Much like the SolarWinds actors knew that if they infiltrated the development supply chain of solar SolarWinds, that they would access all of SolarWind's customers. Gary, can I just ask Patrick? Yeah, Yeah, go ahead. Andrew. Patrick, you made me think of something. 'cause we've seen this with a terra, you know, and others, the fact that you can go, like your research, you went out and got a free trial. Yeah.

Is this gonna make, you know, our software's gonna have to, the vendor's gonna have to rethink this kind of stuff because of now how a threat actor is working. Why, why wouldn't I go download a free trial and arm a terror like they have? Why wouldn't I try to do it on a screen connect like that has, you know, been done. Yeah. And, and we could go on and on and on. So do you think this could change how we trial software? Yeah. I mean, that's a fantastic notion.

Um, on the one hand, you know, you want to get your software in the hands of as many people as you can, right? As you're growing, as you're building a customer base, as you're building that brand loyalty, you want to give folks easy access to your software. On the other hand, you know, threat actors will use these platforms to either use, um, the features of it, right? CloudFlare CF Tunnel comes to mind.

Um, you know, that's a reverse proxy, zero trust, uh, thing that's free, um, that folks are using to tunnel out without having, you know, had to use DNS without having to ride traditional, you know, network, uh, systems. Right? So I think, I think each software vendor has to kind of come up with their own philosophy and how they're going to tackle free trials dis distribution of their software, how ubiquitous it is.

Can I, one last question, 'cause I'm, I'm gonna throw something Gary here to, uh, uh, is that, you know, he may, I'm so glad you mentioned the, uh, the, the, the, um, oh gosh, the CloudFlare tool, because, you know, I've heard, you know, people say, oh, well, you know, my firewall blocks outbound traffic to, you know, certain, you know, regions and can Yeah.

Can you just blow that stupid notion away of how a threat actor works and can work that it doesn't matter what re they're, they're smart enough to not Yeah. Plant their beaconing back in North Korea, Right? Yeah. I mean, those, the days of being able to look for source ips from Russia, um, and, and, and have an indicator that way, I think are, are bygone, right? Um, both from an in inbound and an outbound perspective.

Um, you know, threat actors will use a VPN software the same way folks use it to bypass, uh, regional restrictions on content, right? Um, at the consumer level, uh, as well as reverse proxies. Um, if you allow access to, uh, an external, an external site like CloudFlare or Google's DNS, um, they will leverage those ways to bypass security, uh, restrictions, right? Um, you know, shameless plug for total, we use a SASS e product here that will, will help with that, right?

You can, you can shape traffic, you can make sure that, you know, these, these tunnels and VPNs aren't being used by your use base from the endpoint perspective, right? I think traditional network kind of, Hey, we put a firewall in, it's, it's application layer, it's next gen, um, is, is kind of gone away, right? With, especially with remote work, right? You never know what firewall they're coming through.

You have to route traffic and, and be deliberate about how traffic is routed from your users. Yeah. Awesome. Gary, back to you. As Phyllis likes to say, right? The perimeters at the keyboard and, and, and, and we live in a different world. This is not, you know, 10 years ago. Uh, well, I would, I was gonna say on here that I think this one is actually from a net wide scale net perspective, much more effective than the SolarWinds one.

I think the SolarWinds one, there was a, a kind of a minimum customer that ran that stuff for the most part that you could argue this one, I mean, they probably, they probably got a, I mean, name the industry, they probably could have checked every box for the industry, checked every box for size, you know, it was, uh, I mean, this, this is kind of interesting.

And from a risk perspective, you might wanna look at things that way to say, Hey, am I using a piece of software like this that's got a BA much bigger target for a threat actor than maybe something that's more niche? So you, you gotta kind of start thinking about it. Yeah. I think we're not thinking about it. I, I know talking, I was talking in the past month with a couple real large MSPs. We were talking about security and, you know, um, they're segmenting, right?

They're segmenting their, their tools and that, but never once did we mention something like a VoIP client. Yeah. You know, but that never came into the conversation. So really expanding your thinking in terms of everything, like you said, that can have this kind of scope. Chris, And by the way, I just wanna clarify for Elliot. Great point.

Look, we're not saying don't, like, especially for like, things like M 365, don't use Geoblocking with combination with mail forwarding rules and, you know, look for, you know, certain behavioral anomalies. That's not what we're saying. We're simply saying, you know, the days of, oh, you know, just, I'm fine because my firewall is blocking outbound traffic to a specific geolocation. That's, you know, that's 20, that's, that's 2010, right? Thinking that's, yeah.

You're Not saying shouldn't be doing those things. You're saying understand what the effectiveness really is. Correct. Absolutely. Absolutely. Yeah. And it's a defense in depth approach, right? You, you know, you layer on those things. We're not saying you shouldn't do it. Correct. It's just understand that the threat actors, as soon as you set a bar, they're gonna jump over it, right? There's, there's, it's a never ending cat and mouse game For sure.

And, and this, and Gary, this is a nation state threat actor. They're, they're highly sophisticated. So that's, that's simply my point here. Back to you, bud. Uh, I, so I had to have, I have a, something I was thinking about, you know, and Patrick, whether you have a perspective in thinking about this attack relative to like a SolarWinds and the time that's gone by, this got uncovered and dealt pretty quickly, right? Yeah. Compared to SolarWinds.

Do you think that's a, a positive for the good guys and the community in terms of the, the learning process? We talk, always talk about the morphing and the maturity of threat actors, but is this any type of a small win for the good guys? Um, yeah, I think it is. I think it's a win from, from several standpoints, right? I think it's a win from, obviously the collaboration that happened with a lot of the different security vendors. Um, you know, it's, it's a tough threat.

Intelligence and sharing indicators is a tough line to walk when you're trying to protect your customers first, right? Um, and, and not exposed from an OPSEC perspective, things that you don't want the actors to know. Um, and then protecting the internet as a whole. I think when, when one of these supply chain attacks or a major attack happens, uh, we as a community need to come together and, and, and really share as much as we possibly can.

And CrowdStrike started that notion with their Reddit post and, and it kind of took off from there. So I think that there's a win definitely from that perspective. And it's an also a win for folks in the EDR space. Um, because this was a behavioral alert, right? This wasn't based on any signature. It wasn't based on anything that traditional AV would've cau. Um, it was based on shell code injection that was a behavioral, um, uh, detection, right?

So, uh, you know, folks who think, who might think that AV is good enough, um, and I just saw a thread on Twitter this weekend, uh, where, you know, it's a 80 20 game. AV is good enough for most. It's like, well, with an attack like this, a traditional AV approach isn't really going to help you. Um, so you need that additional layer of EDR on there. And folks who've been red teaming and pen testing have been saying this for years, right?

Um, they don't like seeing those eds on there, because no matter how you pack your malware or, or what you do, it's going to flag something. And you're just keeping your fingers crossed that the blue team or the sock on the other side ignores it or doesn't see it. Something like that. That's usually how things get passed, right? So, Andrew, when these kind of things happen, do you like me flashback to sitting on stage at Schnoz Fest and watching how few of the hands went up?

And we asked about managed, uh, E-E-D-R-M-D-R, You already even said Yeah, yeah, yeah, yeah, yeah. Yeah. Very, very, it was a very small percentage. And I don't know, maybe this is an opportunity, like if you can really understand what we're talking about today and all the great resources you're posting, this is a great way to go to your customers and explain, like assume breach in a different kind of way, right? And making it real for them and say, listen, we're a really good vendor, right?

This is what, this is what's happening. Yeah. And the only thing that will stand between potentially us being the position to protect you and recover your resilience is we need to do these three things, X, y, Z. Yeah. And, well, Gary, it's like, you understand what I'm saying? Yeah. Well, think about it. It's like, you know, we, you, you know, the analogy would be look, the neighborhood, right? We, you know, we used to live in this lovely neighborhood, and the neighborhood has changed, right?

We could leave our doors open, right? And lock, now we have to lock. And then we went, you know, a few years. A we had to start locking our doors, then we had to put alarm systems in, and now we gotta start having somebody watch the alarm systems. And we gotta have a safe in our house for our most important things. Yeah. Right? Right, right. And we're gonna start point, someone's Gonna Have start, you're gonna have to start segmenting things. Yep. Micros segmenting. Right?

And, and if, and if you have valuables, that critical, right? We need to talk, and this gets in the whole Brian Blakely conversation of how do you make money? What are the most critical systems? What systems can't be down, right? If everything else is down, what systems can't be down, you know, are reputationally in business ending. So, um, I need your fr send me that thing.

I'm, I, I'm more fired up than I'm, I'm gonna, I'm gonna pass it over to Phyllis, but I, I, I wanna close by saying, you hear me say here, the cost of doing business has already changed. Yep. It's already changed. The problem is our customers are putting that cost on us. Well, And that's not, and that's not cool. And they don't even know it. 'cause we haven't explained it to them, because sometimes we don't even know it. That's even worse. Well, we give them an out, right?

I mean, we offer that basic AV why, I mean, if it's not gonna Work offer, we don't offer this anymore because it's now been rendered obsolete in 20% of situations, which basically 5% should be zero. Yeah. Right? Yeah. Yeah. All right. Now we're all fired up. So, Phyllis, let's go. I'm sweating Phyllis. Let's go. I need a shower. I just have one comment. You know, I think it's also, um, you know, everyone's like, oh, well, how do I convince my end organization?

'cause they don't think they're a target. I think, I think, you know, you can, you can look at these kinds of things and say, you know, they aren't targeting businesses per se, right? What they're targeting, right? Is any way in which they can, at the end of the day, compromise any network, regardless of sector, regardless of size. So it's really about, um, you know, kind of the biggest bang for a criminal's buck, right?

Th this is an equal opportunity, uh, world we live in Phyllis when it comes. And that's that. I really am glad the way you said that. Like, this isn't like discriminatory. We're too small. We don't, they wouldn't care about us. They don't care about anybody. Big, small, little, Right? They, they, they, they, they, you know, the, the very de definition of supply chain is, you know, to get in everybody's supply chain, not just one sectors, right?

And so I think, you know, trying to get that message across, um, hopefully can, can help some MSPs. Um, so Patrick, we'll, we'll stop, like, you know, getting, getting hyped up over here. Um, and so, you know, it's great that you guys, and CrowdStrike, it seemed like first to report, first to do all these things. I'm curious on the back end, what is your relationship with three CX or with any potential vendor that you find a vulnerability in?

And how do you think that, um, three cx, um, responded? Yeah. So, uh, the first question's easier, most of the time we'll reach out privately, um, if we see something ahead of time, um, you know, disclosure to the, to the, to the vendor and give them a chance, right? From a, for, and I'm speaking from research in general, right? Not just total, right? If, if, if, if you see something with a vendor, um, give them a chance to respond, right?

Don't just put it out there that they're, they're compromised, things like that, because, you know, you could, you could be doing more harm than good. Um, uh, so, you know, we don't have specific relationships with vendors, but we will reach out and, and talk to 'em and talk to their security teams, um, ahead of any major announcements or blogging of anything, right? So, um, that's the first answer. The second question is, you know, it's a tough one.

It is a, it is a tough one because, you know, on the one hand, um, I, I read the full thread on, on their sites, um, and their response that initially they did indicate, they didn't come out and say the phrase false positive, but they did indicate that they deal with a lot of AV alerts on their system. Uh, they deal with a lot of alerts that security alerts that come that aren't normally, uh, you know, uh, malicious.

So when that happens, you kind of get that alert fatigue from a vendor perspective. If, if people are ping them every time an AV alerts, then they get that alert fatigue that socks get, right? It's the, it's the cry wolf aspect. Um, so, you know, I think it's a lesson for software vendors that, um, you need to take it seriously.

Uh, when, when customers come to you and say that there are security alerts firing on an, an update package that you might be, uh, releasing, or, you know, your API is exposing something. I've, I've seen those in the past where, you know, vendors respond and say, ah, we're our APIs locked down, we're fine. And then a couple weeks later, they realized that it's leaking some information. So, you know, I'm not a PR expert. Um, I think they could have handled it better.

Uh, honestly, I think they could have been out in front of it as soon as CrowdStrike released, they should have come out and said, Hey, um, we, we are investigating, we're, we're initiating our incident response process. We will release things as, as we can. Right? Um, I think that would've been a, a better approach, especially to give your customers the idea that you're on top of things, right? But you're not, you're, you're not a lawyer. Yeah.

I mean, you got the lawyers, you got the sales and marketing people that are worried like, heck, that their business is gonna go away. And then I, I'll tell you, 'cause we get in a lot of these situations when we're dealing with incident response and we're trying to talk to third parties or whatever, I'll tell you the number one thing is egos. The freaking egos is what is the biggest blocker in the filter in the world?

Because, uh, I mean, it comes all, you know, Hey, we got, we have our own security team, or I'm a security professional. I've got this handled. Thanks, but no, thanks. Get outta here, type deal. And I, I think, I think if you could somehow get those egos out of the way, um, you could, you could be much more successful when you're communicating these things. But, And think about it, like, put yourself in that spot.

You're, you're the CEO, and I don't know who their CEO is, but you're the CEO or, uh, management team of a company that has 12 million users. And this is happening in real time, very hard to make decisions, very hard to make. Yeah. I mean, I, I, I didn't want to imply that it would've been easy. I I just, you know, my own Personal in hindsight. Yeah. I mean, hindsight, you would say for sure.

Uh, yeah, But I'm with, but Gary, but this goes, And there are from the SolarWinds too, But, but Gary, this goes into crisis communication. Like if you're a software vendor today, I agree, you've got to have a, like, bullet. I mean, again, think about what's the ramifications of Uber, right? Like, what's happened to them as a result and, and, and their executives. And, and so I, I understand the challenge of it, but Yeah, there's that window.

There's that window that even if you have a plan, there's that window of tr of trying to come to grips with everything that's ver very hard, Chris, you see it all the time. Yeah. Yeah. And this one was really interesting, right? Because I, I'll just give you a little bit of my perspective is, I mean, we had people call in and they were like freaking out, you know, like, Hey, launch a full scale investigation. You're like, whoa, whoa, wait a second. You know, there's a little bit more to this.

Uh, there's probably not the need in your particular case, person calling in that you need a full scale investigation. There's some steps we can take to kind of understand that. But in the end, you know, I kind of ask myself internally, not them, is why does North Korea care about you?

And that's really kind of the kind of say, okay, look, if you're defense related or something like that, hey, you know, maybe it's a little bit more, more likely that stage two or even stage three, depending on when this thing happens occurred. But if you're not, then you know, there, there's a way to say, look, we still need to look at it. You still need to take it serious, but you don't need to start pulling the plug on on your network and telling everybody to go home at the same time. Fair.

But, but to, to, to Phyllis's point, Chris, I think we all need to be aware that oh, yeah, it doesn't care who you are, number one. And if the info stealer gets your credentials and you know, they're, they're sold, you know, there's, there's mitigating things we all need to take, period. Yeah, yeah. No, no, no doubt about that changes Passwords and, you know, right.

I just think one of the biggest, I think, not one of the biggest ones, but one big lesson here is, is like you could have had a lot of companies freak out and completely just flood with noise to where other companies couldn't get to the bottom of it, is my point. Mm-Hmm. And so you gotta figure out how to kind of balance that out and figure out, you know, what you need to do.

And, and, and I will say, you know, to Patrick's point is I think the, from the security industry and the security community, the amount of communication was very quick and, and swift, and, and people were sharing that information so people had a good understanding of what the heck was going on pretty darn quickly. And, you know, and I saw some emails that went out from different people that probably had some information that was not a hundred percent accurate.

And so, uh, but I think for the most part, people got it right. And I think that did save a lot of people's bacon. Chris, If I could just say this too, Phyllis, I know we're taking a little of your time. No, you're fine. But I think it's really important. This is for MSPs, right? To be proactive communicating to their customers, right? Here's what we know so far, right? Especially if they are using three cx, right? It is a, you know, this is, this is how the compromise is taking place.

So, you know, we, we've, you know, we've done an inventory, you know, you do, or you do not have any of the desktops running you, you know, or this is how the compromise is taking place. Um, we're available to, to talk, but, you know, but don't have it where, you know, your customer's reaching out to you first is my point. Well, I'll also pick up on what Chris said. Chris said, are you a DOD supplier? He, he did have that in the, in the thing.

So the question is for MSPs is, do you know if your customers are in the DOD supply chain? Do you know if your customers do have this high value data, even if they're a small business? Right? And so, you know, he kind of started off that also saying, you know, perhaps he's already done the work and he knows about his customers. Right? You Mean you're assuming everyone's done control three, which all every MSP has Done, right? Right. And so, have you, have you done that?

So you can actually make a bold statement that LER just said, right? Like, have you done that work? Um, and if you have, then yeah, maybe you can make those kind of statements. So anyway, I was just picked up on that. Um, and so, um, you know, in March, Biden administration, um, has said, Hey, you know, they wanna put more responsibility on the software vendor versus the end organization especi, especially when it comes to small, medium businesses.

And I think part of that really is, um, you know, you can see that in SBO m right? And, um, you know, software vendors. One day hopefully we'll be creating SBOs. Right now it's only in certain sectors, et cetera. What are your thoughts on, on this, and how is it that, um, you know, I don't know if you know much about an SBO m so I don't wanna put you on the spot, if something like that Would bill materials? Yeah, yep. Or what is it that vendors can do?

Uh, by the way, Robert Chaffey, if he's watching, has a different version of SBO m So go ahead, bill Patrick. Okay. He has several bombs. Yeah. Awesome. Love, love the hype button. That's, that's a good, good addition here. Um, yeah. So I mean, I, I think as a cybersecurity industry and as a software industry, we need to get better at understanding the supply chain. Um, SBOs are, are good. I'm not a hundred percent sure that an SBO m would've helped in this situation.

Um, you know, these were signed DLLs from vendors that were trusted, right? Um, and, and if you can't trust the ca of your vendor, who can you trust? Um, you know, and, and this, someone on Twitter posted that this, the reason why this is still a thing is because in 2013, Microsoft chose, there's a CBE from 2013 about signing and, uh, temp just reposted late last year around, um, being able to sign things and change them after they've been signed, right?

Um, so, you know, there's a whole industry of software supply chain startups happening right now. Um, you know, making sure that your, your software is, uh, is secure as you ship it. I think, um, you know, government regulation can help and has helped, I think it specifically CISA and some of the things that they're doing about, um, bubbling up risk and announcing things.

And, and I just saw, I think last week, they, they're actually going out and telling people that they've been ransomware before the ransomware kicks in, right? We need more proactive things happening. Um, and that static analysis, dynamic analysis, you know, as you run your software, you have to run dynamic analysis and use engines. Uh, the CEO of three CX came out this morning and said, you know, we use virus total to check our software, and it was clean.

And I, I can confirm the first post on our blog, we, we uploaded that stuff and it was zero vendors were detecting it. So you have to do dynamic analysis in a live environment that simulates what your customers are running, um, with EDR vendors on it, uh, with a, with a SIM package logging data, um, to, it Sounds expensive. Yeah. I mean, it, it's also expensive to, to respond to a, a, a serious chain. I'm, I'm with you. Yeah. Just haven't been involved in several small software companies.

Just think about how many competing things outside of this Yeah. That they're dealing with, trying to be competitive with software, trying to get out new features, trying to deal with a stable platform bugs. And, and now, you know, this, it's, it's, uh, yeah, it's, it's definitely a, a, a challenge. But Gary, to, to your point, I mean, you know, again, this is a pie in the sky thinking, but you know, when it comes to whether it's a, let, let's just say the medical world, right?

You know, you're dealing with a specialist, and obviously there are certain bars, right? They, they have to go through licensing and credentials and all that. Like, we don't think two seconds that it costs this much money. Same thing, you know, with a lawyer, an accountant. It's because a lot of, and I'm not suggesting don't all get on a soapbox.

I'm not suggesting regulation and license, but it is part of the reason that we're down here, albeit for, uh, some that have gone, you know, I don't care that I'm, you know, we're gonna, you know, really take our business to a whole nother level, our security to a whole nother level. But part of it's self-imposed, right? Mm-Hmm. I, I agree. Yeah. Bill, it's back to you. Go ahead. This Is a really good conversation.

So, um, so we, you know, we've already said, Hey, this is great, the collab between, you know, the vendors who detected this, do you see this as a way of the future that organizations are going to be being more cooperative as we move forward the way kind of ISACs and ISOs are? Or, um, do you think that the competition, you know, the first to report wins, right? Yeah. And so, um, you know, is that going to be a barrier? Yeah.

I, you know, this is a very nuanced, uh, right answer in question, right? Um, I think part of the reason why it was so great to see, and everything happened so quickly was because people share publicly, it was on social media. People use the avenues that they could to share, right? Uh, versus once you start putting in process and you, you start putting in ISACs, right? Um, that you need to join and you need to share with, right?

Then you, you, you kind of wall off potential people who can help, right? From, you know, newer startups, uh, to large, huge market cap vendors, right? That everyone was kind of jumping in together here. So, you know, I, I would love this to be a case study in how vendors distribute data, talk to each other, understand what each other are seeing, right? But, excuse me, I don't think we'll see, you know, a a a a AV ISAC or EDR ISAC coming down the road anytime soon. That's funny. Okay.

No, I get that. Thank you. So, um, do you think that, um, MSPs should be looking at this particular threat actor, um, and be doing some threat modeling? Or would you suggest another threat actor? So here, you know, like, we wanna be proactive. What is it that MSPs can be doing to help protect themselves? Yeah, I mean, there's a, there's a huge landscape out there, um, of, of threat actors targeting all the industries that have been mentioned in the last 45 minutes, right?

So, you know, it's, it's gonna be difficult for MSPs to track and understand all of these actors. Um, I, I would definitely leverage, uh, collaboration and response capabilities with your security vendors. Um, you know, uh, manage detection response is critical. You know, our team is looking at this stuff every day. Um, you know, we're all kind of, uh, understanding that you, you can't track every actor all the time.

And, you know, there, there are some that are, have different, um, motives, right? A lot of the crime wear stuff is, is just ransomware and EXFIL data. Um, this nation's data have eight different, um, motives for attacking a supply chain. So I think collaboration with security vendors, making sure that you understand who your customers are and where they fit. I think it's already been mentioned, uh, Phyllis. So, you know, um, is, is the best that you can do.

Uh, I wouldn't, I wouldn't specifically call out any threat actors at this point. Mm-Hmm. Um, they're all doing different things and have different motives. Just, you know, work with your security vendors to understand how you're being targeted and what your attack surface looks like. So before I pass it to Chris, I just have one quick question. What is it that MSPs can and should expect from their security vendors then? I think they should expect, uh, transparency.

Uh, first and foremost, understanding what the security vendor is capable of. Um, right. The software itself is one thing. Uh, but understanding, you know, the sock triad is something we've been talking about, and it's kind of has different legs now, uh, with different components, right? We've been talking about for 20 years, right? You have the endpoint that you look at, you have the network, you look at everything should come into a SIM and be analyzed from a behavioral analytics standpoint.

You know, we're, we're looking at a world where threat intelligence is kind of table stakes, basic IOCs, the pyramid of pain, you know, that everyone sees IP addresses, domain names, that's table stakes at this point. Um, you're, you, you should expect from your security vendors to be investing and researching into advanced attackers, into ways to detect things that aren't, you know, traditional IOC matches or traditional AV signatures.

Um, you know, we have to level up our game from a blue team perspective as well. So expect that from your vendors and expect transparency on how you do that, right? It shouldn't be a black box where you send your data over and then they flip back an alert. Uh, there Should be also, I, I, I don't wanna interrupt you, but also the expectations fields have to be based on what you bought. You understand? I, I feel right, right.

Sometimes MSPs treat security vendors the way our customers treat us as MSPs, which is, that's a good point. Well, I, I thought I have, I have av I I thought we were covered, so it's also, and we get mad at our customers when they act that way, but sometimes we, we have the same mentality with our vendors, make sure we're making the right investment and we have the right products and services that match the expectations that we have, right?

Maybe you get, you're getting what you pay for, But, but again, Gary, you point, what you just said points out about raising prices, it, it really does, because we've gotta be charging our customers enough so that we can do some checking checks and balances on our security vendors due diligence. Maybe we wanna run some threat modeling on our own detection systems, you know, did they even catch this? You know? And so again, it all comes back to I'll, you know, I'll, I'll give you kudos, Gary.

We're not charging enough. And it's, and it's because it's that threshold you hit with, uh, um, what Patrick just said. When somebody, it's not gonna be the MS P in most cases, but you have to invest, someone has to have eyes on glass and be doing research, and that's the imp that's the expense of part Andrew. Yeah. Like, that's the expensive part. And, and again, having the right technology and having a technology enabled is super important.

And that's where vendors try to get a competitive advantage. But at the end of the day, Yeah, Many MSPs, they don't, and they don't have, they don't really have a vendor that does either. And in this kind of a situation, it, it, it, it shows. Yeah. Yeah. Going back on the expectations piece, I think, you know, understanding you yourself as an MSP, as a business, understanding your customers and finding a partner who is going to want to understand the, the business aspect of it, right?

I, I sit with our MXDR team, and they're, they're constantly asking, you know, what kind of unique software do your, do your customers run? Um, what, what kind of things are your crown jewels that we need to keep an eye on, right? And maybe write detections for, right? We we're living in a world where, you know, just installing a, a piece of software and installing a SIM with some detections on top isn't enough.

You need a software vendor who, uh, and a, and a MDR vendor who understand that this is evolving, right? Uh, and, and we need to keep up with the attackers. Well, and, and as we hand it over to Chris, I mean, Gary, what you're just saying, and what you're saying Patrick, is, you know, let's think, Sunil, you, let's think the matrix right? Slide, right? What happens at the tech respond recover it's people in process period, right? And so, if, you know, if we're going to e Exactly.

And it costs money to do it. So Yeah, you, you're absolutely right. Wes, Chris, over to you, bud. Hey, so, and just a couple questions, Patrick. I mean, you know, from the, from the onset with your team, how did you guys know this was gonna be as big as it was? Was it pretty obvious just from the, the vendor and, or how'd you guys kind of calculate ole moly? This could be a big one. Yeah, so, you know, our stack is, is pretty cool.

Um, shameless plug, we have very, uh, wide visibility both on the network side and the endpoint side. So as soon as we started looking at this, we saw how many customers we had with this install base in it, right? All the way down to the DLL aspect on the end point. Um, you know, and I think vendors that have that visibility, uh, were the ones you saw jumping on top of things early on, because we saw the magnitude of, of what was happening, um, from, from a, a, a scale perspective, right?

This is, this is something we need to jump on. And, and it's huge. So, um, yeah, you know, early on, I, I think all vendors, no matter what your response was, we, we didn't call it a false positive. We let it block the shell code injection, um, and let the software run. Um, you know, we all kind of came together when it was confirmed that a nation state threat actor was part of this attack. Well, I think the other thing we haven't mentioned it really too, is Max, we're involved, right?

I mean, we, so many of these times we're only windows, windows, windows, windows, all vulnerability, windows, windows. But hey, this is Max man. I mean, they, they're in there. Yeah. Yeah. So, Yeah, yeah, the Mac, um, they initially said it was only Windows, uh, actually, and then, and then a researcher did some great stuff, um, and dug into the, uh, DMG that was shipped to Mac OS as well. Um, and that's a, that's a great point.

You know, if, if a lot of folks traditionally think, oh, I have a Mac, I'm, I don't, I don't, these things won't hit me. Um, you know, but yeah, I mean, actors attack Mac, they attack windows. They're, they're equal opportunity. Android, iOS, it doesn't really matter. Um, as long as, as long as they can get in. And what do you think about like, some, some of the secure code frameworks out there, like BS IM or even, even Google's minimal viable security aspect.

I mean, still see a lot of vendors in the MSP space not really adopting this. What, what's your take on that? And do you, you know, Yeah. Do you think The direction is, You know, our engineering team is, is much more tuned into these frameworks than I am. So, you know, obvious Grant of salt, we're doing a blog post about this and how frameworks can help, um, you know, both from a compliance side, from a government regulation side, also from an internal secure coding aspect of things, right?

Um, you know, something is better than nothing. I think I, I mentioned I, you have to do testing, right? You can have a secure coding framework, you can implement process in your CICD pipeline. You can make sure that you're, you know, you're doing static code analysis, you're running against virus total, you're pen testing your own software and your own systems and your own APIs.

Um, but until you do all of that as part of an ecosystem, I think we'll continue to see these software vendors, uh, getting targeted. Yeah. Great. That was awesome, Patrick, I'm gonna just ask Chris a few quick questions. Chris, for those that don't known Chris, Chris Laer, SVP of Solace, uh, they're the incident response team, uh, for CFC insurance. Chris, just curious, are you guys seeing incidents? Are you run managing any or MSPs involved? Yeah, I mean, ma we are seeing them.

I mean, we, we haven't had much interaction with MSPs on these deals. Mainly when, when the people have come, uh, you know, they're very concerned. I mean, that was, you know, that's kind of what I hinted at earlier. Uh, this one really has somewhat lit a fire more than the other ones.

I would say, you know, even from the, just talking to people and listening to them respond is a much different tone and sense of urgency than they had with the, with the, the exchange proxy, shell, proxy log on vulnerabilities. Uh, so that, so that's really good. So, uh, that, that's kind of it, but it really is having that conversation with them to say, okay, look, yeah, this is serious. What steps have you taken already?

You need to take the following steps and, uh, let us get in and, and just start that initial kind of assessment to understand whether or not we need to deep dig, you know, dig deeper or not. So that's kind of where we're at. I, I, I actually was thinking we would see more calls than we have. Uh, but, but maybe they've, they've got it figured out, or hopefully that's the case.

Or Maybe, again, this is, uh, to, to, we've all been talking maybe, you know, again, this is a really well coordinated, thank goodness for all the security teams, right? Total SentinelOne, CrowdStrike, et cetera. If I didn't mention you, please don't shoot me. Those are some of the ones that come to mind out at, uh, right, right.

So, um, and then Chris, Gary kind of started to mention this on the sales side, and Gary certainly chime in here, but, you know, thoughts like, should, should, and how could MSPs be using this in the sales process? Like, I'm just thinking, Gary, you're, you know how you're saying like, no one, like this is wow, um, the VoIP system. Like, could this be a tabletop exercise, Chris, should this be something that we look at and go, well, what about, you know this, right?

Would you have been ready for this? Could you have, you know, could you have managed through this crisis? Yeah, I think it's, I think it's a great scenario for a tabletop, just, just talk about that. But I also think this is from a sales pers process, this is another reason why you is, is if you have, uh, prospects or, or customers, you know, pushing back on you, you know, asking more questions or doing asset discovery or whatever the case may be, you gotta really learn about the business.

And I think this is another great example of you can learn about the business more. There's gonna be more opportunities for you, and it's gonna be much easier for you to explain how things like this can impact the business when they're not, well, maybe they're not DOD, but maybe they have a client that is DOD. You know what I mean? I mean, there's, there's a number of things that you have to understand about the business.

I mean, there, there are, there are times where I see information where to you and me, it looks benign. Like it's not important. But then you start to look at who that information belongs to, that no longer becomes benign, it becomes more important. So I think that this again, is just another example of how, from a sales perspective, it gives you the opportunity to push deeper into your clients and know more about their business, uh, so you can kind of surface those opportunities.

What's great about this one is we can take 'em right. A boom directly. Right. Right. It, it, it, it explains Right. A boom to them, uh, in a, in a very real way. And again, let's us understand the only reason your clients care how much you charge them is because you have not painted the clearest possible picture of their risk profile. Make a clip outta that one, Andrew. This, this has been, uh, Patrick, you're, you're awesome man. This has been so helpful and thorough, dude.

Yeah, this was a good one. Thank you. Um, Patrick, thank you so much for coming on today. It was a blast. Um, Gary, thanks for bringing a whole nuance, if you will, to the cyber call. Um, Yeah, I'm gonna do some research. I'm working on a whole board, But, um, you've Taken us to another level that Gary Pika, So Carl, um, Katz brought up a good thing. Just in closing everybody. Um, next week, we'll, we'll do something on questionnaires, security questionnaires for vendors.

Um, and then on the seventh, the following week, we have the head of Defer for CrowdStrike coming on Clark Harshbarger. Uh, and we're gonna be talking, Chris, this is near and dear to your part. We're gonna be talking IR and privilege, uh, so that's gonna be a good topic. The hype button seems to be polarizing. Yeah. When, when, when Clark comes on. That's, that is definitely a must see one to be at. So yeah, I mean, his background and what he does today is pretty freaking amazing. Yeah.

So that should be one. Yeah. We're excited to have. When is that? Seven, uh, two in two weeks. Uh, we changed it 'cause Phyllis couldn't be here next week, so, uh. Okay. All right. Patrick, again, thank you so much. Um, I really appreciate you coming on. It was fantastic. And thank you everybody for joining us. Make it a great week. Be safe. Take care, and we'll see you next week. Take care. Bye-Bye. Thank you. Bye-Bye.