Threat Modeling for MSPs

All right. Welcome everyone to week 64. Good to see you all. Gary, how are you? Hey. Not, not, nothing rough there, man. Nothing. No. Too hard this weekend. Yeah, I'm, nope, I'm fired up today. All right, Wes, how are you man? I'm doing good. Uh, really good. Yeah. Awesome, Ryan. How are you? Very well, thanks. Fantastic. And Darren, we'll introduce you momentarily, so stay tuned. Alright, Gary. I only have 25 announcements today. Okay, good. Good. That's, uh, down from last week. All right.

Now, so real quick, uh, only a few announcements. Number one, the, we're getting awesome feedback from folks that are, uh, on the, uh, training with John Strand right now. The MSP security analyst training. He like, literally, like, it's awesome. So several hundred over there. So we're kind of competing with ourselves today. But, um, there's gonna be more, Andrew, I don't know, like I told my members it took like, you know, we put 10 people right in, in, into that. Right. We sponsored them.

Right. And it took two emails and I got on a call with them, uh, you know, and said, why do I care more about your cybersecurity than you do? Yeah, Let's go. Yeah. Well, it's interesting, just Gary, just real quick, 'cause you've talked about this a lot, and the folks that registered and the folks that are like interacting with me, it's all the top MSPs. It, it's, it's unbelievable. It's, and, and this training is incredibly affordable and, you know, what are you gonna do?

So, alright, uh, the other thing is we released, uh, it's in the call to action. I'll put the URL, but if you click on the green little thing there, uh, Wes Ryan, Phyllis Lee, um, we went into studio. We just finished control, uh, secure configuration. It's fantastic. Highly, highly recommend. Ironically, Ryan, um, Azure, uh, why did it get compromised? Uh, the Cosmos db you know, DB thing, right? Configuration. Um, okay, Gary, I think that is it. Um, so let's get right into it.

You know, we've talked, you know, a lot recently about, you know, different ways that, you know, we have compromises. Last week it was APIs and it made me start to think, you know, we really have to figure out a way in which we can talk about threat modeling, adversary on the emulation with our community, and do it, and synthesize it down into kind of like a basic 1 0 1 level.

And lo and behold, um, I was, um, fortunate that, uh, Darren, who will tell you about himself momentarily, was, uh, open to doing it with us. So you guys are in for a treat today. Let me just kind of frame this abstract out that, you know, we did a threat modeling workshop. We could do more and hopefully do a lot more with Darren. And Darren. Maybe we can con coerce your wife to get involved with this too. Yeah. Who knows.

Um, but, um, so, you know, when you look at the definition of what threat modeling is, you know, typically around enterprise. But one of the things they talk about, Ryan, I'm gonna come to you for a little bit. This is justification specification for spend. And in talking to, you know, Darren about it, I'm like, gosh, if MSPs just got a little bit of command around this and could show their customers and or prospects their security gaps. Gaps and, you know, where they're weak in their defenses.

Wow. What, how powerful could that be? Um, so Darren, pleasure to have you with us. Can you tell us a little bit about yourself and we'll kick right on into this? Sure. Um, so I am, uh, security analyst for Black Hills Information Security. Um, I've been doing this for five years now, so not a super long time, but it's, um, getting to be, uh, quite a while in this industry. It's amazing how how many people are, uh, you know, just starting out or whatever.

But, um, so I started off, I was not in information security. I was actually an online teacher. Um, my wife was, I'm doing computer stuff and I was letting her do her thing and I was doing my thing. And then one day she came home from work after, and this was after she got her, um, master's in Computer science. And she, uh, was almost in tears just saying my, my application that I just wrote just got pen tested. And I don't even know what these comments are.

I don't know any of these vulnerabilities that it says it has. I never even heard of 'em. And she was like, I don't know how I got a master's in computer science without knowing any of this stuff. And she was just so confused. And, uh, I told her, well, you can either stay in the dark with this or you can become an expert. And she chose to become an expert. And so she went back and got a ma another master's in, um, information security and has been doing that for, uh, eight, nine years.

I can't remember how long. And I saw what she was doing and I saw what I was doing and what she was doing was more interesting to me. So I made the change, went back and got another bachelor's in Computer Information Technology. I've been working at Black Hills and I'm, I'm doing some threat emulation for him and, uh, security testing. And it's been great. I love what I do. It's fun. And um, so I've been really enjoying it. So real, you got a real tech background.

I see a folding chair there behind you. And then up on those shells, I'm guessing there's, um, wires, uh, in, in those boxes that can't be used anymore. 'cause they plug into things from many years ago. Am I correct? Yeah, Of course. Yeah. Nice. And then of course, I've gotta have a stash of wires when I, when my kids come and they say, I need this kind, I need to hook up this and I need to do this. So I've got all kinds of wires and stuff. You know how it is, you never get rid of wires. Yeah.

It must mean, you know your stuff if you have that Back. My wife is literally, Go ahead. What was that, Ryan? My my wife is literally shaking her head at me as you're talking about this. 'cause we have a, we have a box that she has named something I won't repeat, but it's full of wires from 17 years ago. She hates it. She's like, she's like this. Ugh. This is a thing. Like, Well, you know what's frustrating is we were, uh, about six months ago, we were gonna sell our house.

And so we started getting rid of things. And my wife upstairs has this drawer just full of wires that everybody does. And she got rid of most of them. And then of course, you go and set something up and you need a wire, and then you're like, I just had this and I threw it away 'cause I was moving. Then you have to buy it. And so that's why you never get rid of wires is because it could come in useful. Too funny. Um, okay, so Gary, I'm gonna kick this over to you.

Like I said, I was really intrigued about the whole side of the business side of this. You know, we're gonna kind of back our way into Yeah. You know, is there this, as west as this kind of voodoo of threat modeling, how hard is it, this is something MSPs can do, but let's take the premise that it's not as voodoo ish and dark theory as we think. So kick us off from the business side, Gary.

Well, first thing I do I want to do is, um, can you Darren, start off by like, let's start at 30,000 square feet and can you tell us how you would explain in a sim in the simplest terms what threat modeling is? Sure. So to me, uh, what threat model, you first gotta realize who it is that you're trying to model. So it sounds just like it is, what are the threats that are coming to you in your environment. Um, an MSP is kind of hard because there's so many different things that are coming at you.

And so you've, I mean, you've got a lot of different customers, but you've gotta know who's trying to, um, you know, attack your network. Um, once you find out who it is, you can then look at some of the specific ways that they go about doing that. Um, look at those specifics about the different attacks that they do. Um, and then comes the fun part is trying to detect those attacks that, that they do. And so it encompasses, for me, the threat modeling encompasses the whole thing.

It's kind of starting at who and then how, and then detecting and then trying to mitigate that and, uh, make it so that they can't actually do that kind of thing. Yeah. And you know what, the important thing, like we're trying to bridge a gap is, uh, you know, MSPs, the average msp, they, they don't have a security department, right? And everybody kind of has a role in the company.

So to, to start to do in your organization some ba some basic and regular threat modeling to do it and be able to actually use the information, you know, in your business and your customer's business, right? To, to make changes. Like what does that look like? So there's a lot of different products out there that, that do this type of thing. Um, some of them are, and we'll talk more about this later, but um, for example, one of the things that we use is Atomic Red Team. And that's free.

Um, it's got a great support system behind it. Um, the, it's very simple to use. Um, and so that getting in start getting started with it really isn't very difficult. It's just running some commands on your, on the command prompt. Um, you can use Windows or Mac Os or Linux or whatever. Um, and so it really isn't that difficult to do that.

Again, the hard part is after, you know, after you run it, um, figuring out what happened and if you're detecting it and, and you know, you're gonna need to work with the SOC or whatever it is that you have, um, helping you with those with the security side of things. But getting started really is, Oh, you're making assumptions, right? Uh, Wes, is it safe to say that not every MSP has a SOC they're working with? I'd say it's probably fairly safe. Yeah, yeah, yeah.

So that ma so that's where we are. You get, get, get the idea kind of where we are as a, as a So, so, even so, even if without the SOC that then you, you know, if you run one of these attacks and with atomic red team, it's really a, you know, a single attack at a time. Um, you can see are, are we detecting this type of an attack? Right? I mean, it doesn't, you don't have to have a sock that that really can de detect that. I mean, you should be able to do it yourself.

What kind of, uh, telemetry is happening with this? And are we seeing it? Are we able to detect, you know, this, this specific attack? Yeah, so, um, I, I was thinking how do you, can you see MSPs being able to use this like once they're doing it, like in in thought leadership, like explaining to customers and prospects, you know, what, this is why it's important to kind of separate themselves and build some value. Sure.

This is, um, really a great way to go in, um, once you have some of this set up, and it doesn't even take a lot to, to get it set up that way, um, it really isn't very difficult. But once you get it set up, you can take this to a prospective client or even a current client and say, this is, you know, some of the attacks that we're seeing, um, and run these attacks and say, okay, you're not detecting this. Um, we can come in and we can help you detect this.

We can, you know, we can do all this stuff. Um, or you can say, we have this product that can help us, or, you know, whatever it is a service, a product that can, uh, give more information on this. Um, you can also, you know, with, with seminars, whatever it is, if you're showing what the threats are, um, it really does give more value to the defense. And I'm thinking about how not, you know, so much internally, but to, to the MSP's customers.

Like whenever they get a something like this, they need to do how to monetize it. And I look at it similar to, um, you know, we're, we push a lot of people to, to build IR plans and work with their customers to do the same, and now they're struggling. 'cause it's hard to figure out like a, how to put it into your monthly fee till you have enough experience of what the average cost is.

So would you suggest maybe this is something that would be a fee-based thing that MSPs would do to their customer in the beginning?

Um, it could be, um, As meaning project Darren as opposed to, like with the MSP, they call it a stack, you know, just think E-D-R-M-F-A and these cost of good sales start adding up on your, on, in your line item so that, you know, if, if you did, you know, an a threat model exercise, is it a project from, if you're doing it at Black Hills with a client as an example? Yeah. So when we do it, um, we do, um, it is a kind of a one-time thing.

So we go in and we will, um, emulate all the, you know, a bunch of different attacks and see what comes out of it. And then we kind of give them, this is what we saw and this is what was detected. And that kind of a, an an idea. Um, so that is a project based thing. Um, with threat emulation though, it should never really stop.

Um, you know, what you detect today might not be the same thing what you detect tomorrow, because you've brought in this new appliance, this new tool, this new, um, process of something. It's easy to break some of those detections that you've thought that you've had and you were thinking that you were protected against, but maybe it got turned off or, um, something happened. And so you need to be doing this all the time.

So that's another thing is, um, it it's an ongoing process of trying to get all, you know, to do this threat emulation. Yeah. So you, I look at this Andrew from two standpoints. One is, it's easier when you think about prospects, right? Mm-Hmm. Because you're uncovering risk, you're using it and in return, you know, what you get from it is hopefully a really good prospect. And the solution is you're gonna start to manage their, you know, their, their infrastructure moving forward to customers.

It's a little bit trickier, right? 'cause they have that, well, I thought you were already doing that kind of thing. Like, isn't that exactly. Yep. So there is where I think you have to be able to assume there's gonna be a cost for us to be running these things and then just look then, you know, from A-V-C-I-O standpoint and, and a governance and compliance, have to look across your customers to say, okay, what changes upgrade stack process do we need to put in place?

And how do we communicate that if there's a cost associated with it? Um, and to them it's not from threat modeling, it's, Hey, there's new threats we're dealing with. Yeah. That's kind of how I'm thinking of it. Wes, are you thinking of it in the same way Trying to find that mute button? There it is. Yes. I am thinking about it in that way. And, and I even like kind of, I want to call out Keith's comment here.

You know, instead of s says add-on commodity sale, it should be part of your consultative, say, we can read the whole thing down there. But I, I think, um, yeah, I think what Gary, what we're we need to learn to do as MSPs is understand like, like best practice and what things do we bring in that's part of, you know, a model in which we sell. What things do we just do is sort of our consultative overlay services, adding cybersecurity on top of what we do.

Like when you think about how enterprise, a lot of this, it's just built into that normal, the operation stuff that they do. So what are some of those things you just bring in as like a added benefit as a a, a perk get by working with? And I think this is a discussion for some of the, Hey, uh, Gareth, if I could just add this though. You know, there's, as we know, and you said it, there's new threat actors all the time. There's new techniques, right? That they're using.

So, you know, I think about, like Ryan was talking about how, and I forget the, uh, I forget the threat actor, but he modeled the whole thing. And you know, if you think about him being the MSP, he's gotta go back to his board and go, Hey, you know, this, this tactic and this technique, we were whatever, 83% covered, here's some pretty big gaps. And, you know, they got a pretty big team. So I think it's all how it's postured. Um, but if, if that makes some sense. Yeah, absolutely.

So I'm, before I hand over to Ryan, I guess what last thing I'll add to this is, um, we're, we're at this point where what this is telling us, if you go back, what we've been building these past 60 some, you know, weeks when we go back to our customers now and have these conversations and prospects, um, we gotta start raising our prices in chunks, Andrew, not $5 a C, right? Like, we have to do it in chunks.

And if we're doing the kind of things that Darren's talking about and the things that, if you look at Ryan's, uh, email that he li or, or, um, uh, chat that he listed, um, we're gonna be able to show 'em the value and we're seeing it happen, right? Like you said, and it's the top MSPs that are out getting trained, but right now, right? And so they're, they figured out that it's good business. So with that, I'm gonna hand it over to Ryan. Cool.

Um, so in your definition of threat modeling, you said, uh, where it's where you model a specific threat that you have in mind. And when a lot of people, if you Google threat modeling, you'll come up with dread and stride and all these other like frameworks on how to do threat modeling. And so people think like, oh, I'll just do stride. Like I'll learn the framework and then I'll do a threat model. But you and I both know it starts with understanding the threat actor or knowing your enemy.

And to do that, you need to have threat profiles. So talk to us about how an MSP should get started building threat profiles and what resources are out there to help them do that. Um, and just maybe riff on that for a bit. Sure. Um, so the easiest way that I have seen, um, and the most complete is Mitre attack. So if you don't, if you haven't looked at Mitre attack, um, that is by far the best resource that's available and it's, um, free, it's government funded.

Um, and there's, I mean, every, every threat that's out there that and gets put on that, um, MITRE attack framework, um, and it's updated regularly. So, um, you know, I think it's updated at least twice a year. And so every, um, a PT group that's out there is on that. Um, all the tactics and techniques that they use are there.

Um, it starts with, um, the tactics about getting access even before that with reconnaissance and, um, you know, developing all of the resources that it takes to be able to start your attacks. It talks about all that stuff. And then it talks about initial access all the way till the end of your attack cycle. And it has all the techniques that go along with that.

And so you can actually just, um, look up, um, whatever, um, vertical your company or the company is, or company that you're servicing or whatever it is, you can look up that and you can see who it is that's trying to attack that specific, um, type of a corporation. And you can set up your, um, threat attacks using that, that framework. It really is a great resource. It's got so much information that's out there.

Um, it's got everything from the attacks that they use to how to mitigate and, um, and so it, again, it's a free resource and available to everybody. Andrew, before, I'm sorry, Ryan, could you, uh, throw up a poll and maybe ask how many people are using that information from Mitre on a regular basis as part of their security policies? Yeah, and, and it's, it's, you know, Wes and I did a quick video on this morning.

You know, it's, I mean, as Darren said, type in MSP, literally it'll tell you, and ironically, uh, gold Southfield comes popping right up there. Uh, They don't exist anymore. Yeah, Exactly. Uh, Incarnate though, they'll be back. Same tactics. Oh, I'm sure they will. Ryan Right back to me. Be back. Yeah. Alright. So there are a lot of different open source tools for threat emulation, right? And so my arguing argument is, before you do threat emulation know what you're trying to emulate, right?

That's 1 0 1 of threat modeling. So you go to Mitre Attack, you get a threat actor that focuses on your or your industry, you dig into their tactics, and then you download called and Atomic Red and you know, insert other free open source threat emulation tools here, and you install them. Then what? Right? How, you know, this can be difficult to do. It can be difficult to do it scale. So again, where do MP start? How do we make it, yeah.

How do we make it really simple for them, um, to, to deploy and to execute and to take the information they get and feed that back into their continuous improvement processes. So, um, I, if, if I was going to be starting, I would use Atomic Red Team because it's free and easy to use and there's a great support system. Um, after you get started with Atomic Red Team, you could go and you could change to something else if you want.

It's, it's kind of funny 'cause um, I would, every other service that I've seen paid for or not paid for, at some point they use Atomic Red Teams. Atomics are what they call the Atomics are the little scripted attacks. Um, so any other service is going to use the attacks from Atomic Red Team. Now they're going to, um, wrap it up into their own proprietary, um, attack or whatever it is.

But at some point they're going to use their own now and again, they might have their own in addition to Atomic Red Team. But, um, again, starting with a free ver a free thing is great.

Um, and what I, what I would recommend, again, the one of the first things is I would look at the most recent attacks, um, because most of the time your, um, you know, your antivirus or, you know, endpoint protections aren't going to be set up to detect those because they are new and it could be a long time before they come out with those detections. So if you have something in place, you know, you could catch it before your, um, other service, other products are able to catch it.

Um, and so with Atomic Red, as soon as these attacks come out and they're known, uh, it's, it doesn't take a long time for them to get some kind of a, you know, their own attack that looks like it set up, you know, and then you can run it on your network and say, okay, well it's going to create this kind of a, an alert. We need to be really monitoring for this alert bef, you know, so we can catch it as soon as it comes. Okay.

And, uh, non-sponsored un, you know, I'm a non-paid attorney spokesperson for Black Hills Information Security, but, um, you guys did a half day atomic red team, um, threat modeling course where you actually got hands-on in a lab environment with, with atomic red, right? So you do that periodically. So if you're an MSP and you don't know where to start, go take the half day Black Hills Atomic Red course. And I, I loved it when I saw it.

I got super excited if I was an MSP, you could literally take what you learn in that half day course in the morning and in the afternoon, you could apply it to your entire environment using your RM and you could in one day map every single computer in your environment and your customer's environment for a specific tactic or atomic test in one day just by taking that morning and learning how to use atomic grid and then correlating that back into your how, you know, you're deploying.

So I, I'd like to, I'd like to do that, but, uh, we're working on tickets right now, Gary, why do I care more about your security than you care about your security? Uh, yeah, exactly. Um, yeah. So give give one, give one of your texts one day off to go take the atomic red course and figure out how to instrument it in your RMM. And that can become, you could bill for that, right? It's one day worth of work.

And you can add a line item to the security services that you sell for threat emulation, and you could turn that into a billable revenue generating item for you with one day of work. So, uh, would you rather have your tech crunching tickets supporting existing revenue or opening a new revenue stream for you? You decide up to you. I don't run your business. I, I'm sorry, I don't mind to interrupt. Just real quick.

You get that information and now all of a sudden there's a bunch of ways to monetize it that next quarter when your VCIO goes to every client, you're bringing that information, you're updating them that you're on top of things and you're telling them the things you need to do. And sometimes that means it's additional investment also. These are the conversations that other people aren't having with prospects.

And, you know, one good MSP prospect that you land that you wouldn't have, uh, pay it forward a hundred times over. Yeah. Yeah. And it's, it's interesting. I, I've been thinking about this quite a bit. And, um, one data breach, I mean, the average, I think is over $3 million to, to fix it. Um, and if you can stop that one, which this is something that you can't really put money on, right? You can't actually go and say, yeah, we stopped at that, we stopped one attack.

Um, but if you do end up only stopping one, you've paid for how many, how many of your employees time? Uh, so I mean, again, it's, that's not something that you can really put a fingerprint on and say, yeah, we, we've saved, but it, uh, it doesn't take much to save money in this field. Well, I mean, it is part of an assurance process, right?

You have the one side of it, which is, as an MSP, can I detect things about a customer environment that I could then turn into project or, uh, you know, recurring revenue? The other side of it is control effectiveness, right? Do I know that the existing technologies I've deployed or operating world can detect real world TTPs or tactics or trade craft that an attacker is using that are observable in the wild? And doing this one thing helps you answer two questions. Again, no brainer.

The the amount of assurance and the amount of continuous improvement data you give from it is, is phenomenal, right? But again, anchored in threats. And you, you had mentioned that, you know, the atomic tests map back to specific trade craft. Um, and Mitre attack is good at like generalizing. Like there's a, there's a theme of this, uh, TTP being used.

But if you look at an individual specific news item, or, you know, like we did a whole thing on TA 5 51 in, um, uh, bizarre Loader and Cobalt Strike a couple weeks ago. Um, it just came outta Sands, uh, sands diary. Where do MSPs go to consume the latest trade craft that might not be in Mitre attack yet? Um, like for instance, hi, this new ransomware threat actor group that's out, um, black matter, relatively new, uh, ransomware group where maybe there's not a lot of TTPs known.

How do they stay up on this stuff that kind of hasn't fed its way into minor attack yet? And and less so about the groups, but more so about the evolving tactics. So, um, one Twitter is a great way, right? Who do you follow on Twitter? Um, find it's great to follow finding someone who, you know, spends the time to look at these things and do the research themselves, then you can follow 'em on Twitter. That's a great way to do it. Can We just follow you on Twitter?

I'm not super active on Twitter, to be honest. That would be great. I, I don't, I don't spend a whole lot of time on it, but, um, but another thing that you can do is, um, with this Atomic Red Team project, there's a Slack workspace. Um, so get on that Slack workspace with Atomic Red team because they're talking about things like that. And, you know, any new attack that they see they'll talk about, as you know, has anybody tried to emulate that?

And, you know, put out that information about those attacks as they come out and if they're, as they're able to, um, model the, or emulate that attack. So someone asked for where the Atomic Red training is. So I'm just gonna drop this link to, uh, mild West Hacking Fest. Yep. On that web pager, you're gonna see a very familiar face. It's not mine, it's not West's, it's not Gary's not Andrew's. Uh, and there's two dates. There's September 21st and September 22nd.

So, um, if you're really serious about wanting to learn more about how to use something like Atomic Red, um, very practically, very applied, um, go there, sign up, do that and that, does that, does that pay what you can? Or is that free? Or, Um, so that's actually with, uh, that wild web hack, hack confessed, uh, conference that's happening at the same time. So you'll get part of the conference and be able to learn a lot more about that. There's another class that we're teaching in October.

Um, and so the class itself in October is going to be a little cheaper if you only want the class. I'm not saying you shouldn't go to the train, go to the conference, 'cause there's gonna be a ton of great speakers and a ton of things to learn from. Um, but the class itself is four $95, so it's not pay as you go, but it's very affordable for, um, for this type of training. Okay.

So Andrew had mentioned that you give a crash course on MSP tools, specifically RMM solutions, um, which have godlike permissions and characteristics. Mm-Hmm. Uh, if you are an MS p, what, what TTPs what, what tradecraft might you do, whether it's manual or using something like Atomic Red to test for weaknesses around these tools? So I, with that, um, there'd be some things that I would try to do first is, um, I would be checking the login for that.

Um, and so I would set up and I, um, I'm not even sure if there's a, an atomic red team. You could use the framework for Atomic Red Team to, to set up, um, this. But, you know, are you checking to see if people are trying to log in and having all these unauthenticated login failures? Um, are they generating any alerts?

Um, and are you looking for those, um, are you also looking to see if there's a successful login after a bunch of unsuccessful logins and then all of a sudden there's no unsuccessful logins after that? Um, are you able to check to see if, um, your RMM is issuing commands without you knowing? Um, so you could try to, you know, just set up a schedule for, you know, an rm, one of these RMMs to run a test and see if you're able to just, you know, detect that it ran.

Um, it'd be good to know if, if that's running. Um, you could also set up tests should the people be logging into these at weird hours of the night. Um, maybe that's the case. If there is, if there is, then who is it? And, you know, I would be checking for things like that. Um, for specific Atomics, I don't know that there's actually specific Atomics that would go with that necessarily. Um, but those are the things that I would look for. And you can script those type of activities.

Ryan, just quick, uh, maybe tangent, but if you were an MSP, might you, from a deception side, might you put some docs, you know, that said, you know, you know things about your RMM, you know, the RMM credentials, the RMM, this, the RI see you shaking your head down. What would talk, give us your thoughts there, Like Yeah, for sure.

I mean, this, this isn't necessarily threat modeling, but, um, honey documents, honey credentials, um, I know that when I'm on a test and I get those, they're really tempting not to use, you know, if you see a username and password Yeah. Um, you're gonna try and use that because it's, it's amazing how many times they are really valid and they aren't looking at those and they're actually not honey documents. So, um, yeah, James just said canary tokens in the chat.

So, um, I would be looking at that, you know, just seeing, and again, checking to make sure that people are on there who should be on there. Yeah. Here's another steal this idea moment. Um, speaking of recent trade crap, Ray, it's, it's come out, it's been fairly well documented recently that one of the first things attackers do once they gain persistence to your environment is look for insurance documentation to determine your coverage levels to help them in the negotiations of the ransom.

So wouldn't it be really cool if you had a Canary file that was cyber insurance policy coverage and just put that in some file share? Sure. No one should be looking at it and alerting yourself if someone, uh, hits on that. Like that's, you know, that that's something that's very simple, very easy to do, very quick to set up an alert, you know, something really simple. So Yep. And then you something repeatable that you can test, right?

You could have some process that Monday at two 30 touches that file every week and just make sure that your alerting still works. So lots of cool stuff you can do there for sure. So over To we. Yeah. And he's More, he love these things more than I do, Ryan.

Well, Ryan, you, you really get me thinking, you know, like, I love those, um, like home alone kind of movies where like, if we think about, uh, Macaulay Kin had a few days he knew that an invasion of his home was impending and he set up a whole bunch of traps all around the house. Um, we really ought to think more about that in terms of like enterprise defense. We don't, and I know that there's a lot of creative ways you can do that.

And the the cool thing is a lot of 'em are great ideas that cause very little to zero damage and may cause some false alarms and some tweaking down the road, but like, I don't know why you don't spend more time on that. I'll give you an example, and this doesn't work for MSPs, but I just want you to like, I love that idea of the insurance like document. I think that makes a ton of sense.

Another example is, I know a really large bank, and one thing they do is they intentionally feed in false user accounts and credentials into like dark web, like dumps and things like that. And the reason they do that is they want to cross correlate. If they ever seen any of those hits, they know a hundred percent that those are false. And then they use that to correlate other activity that's associated. I'm like, what a brilliant idea.

Like obviously this doesn't work for MSPs, but we really should spend more time thinking about what are deceptive types of techniques we can have to design our own home security, just like Kevin did in, uh, home alone. Uh, I think it, it, it's a fun thing to explore. Yep. Yeah, when I, when I was in financial services, we did, um, active defense against, uh, directory harvesting and credential theft.

We would create a new ad user with a mailbox that was not a real person and just set it up and the mailbox would go to a shared mailbox and it would get monitored, and we would see if those emails got picked up in dumps or if those passwords got picked up in breaches because that would be an indication that a kind of, uh, point in time user and given which users were in the dumps, we could determine if, uh, well the logic was, we could determine if or when the, the dump had actually been stolen.

Um, luckily we never had a hit, but like, again, very simple, it's just a process. You create a new user every month and then you monitor for it in your dark web monitoring. Very simple to do, and it helps you narrow in your window a potential breach, right? When did boom happen? So yeah, a lot of very simple things that we could be doing that we don't. And I agree. All these things are really easy to automate across your entire environment.

Like we just have to think more about how to use them. Andrew, maybe bring back our, uh, Our honeypot buddy. Yeah, I was just thinking probably been a year. It's funny, I literally took the words outta my mouth, Gary. I was gonna say to Wes, we could have a what when Chris Sanders and Darren Roberts meet, and I was gonna just simply come to you, Gary, for this new service called, Yeah, let's see what we can work up.

I've, one of my friends is Gadi Evon, uh, from ria, they've been acquired now and, and GA's one of those big box outside like the, like I've never thought of that before. Like, I, I wonder if we can bring GA on too and talk about some of this, but there's a World here of active defense, um, you know, active countermeasures, deception. Like, there's, there's a whole, like the whole idea of creating more opportunities for you to detect a threat, your environment and to slow them down, right?

Yeah. That's what deception technology is about, right? It's, think about it as a counterintelligence, that's effectively what you're doing here, Ryan, I forget how you threw this out, but there's another course that Black Hills teaches that John actually teaches on deception. That's, yeah. We'll, we'll, we'll get it all out there before we're done here. Yeah. Just go take everything from Black Hills, just go to Wild West, I confess and take all the things. Okay.

We're gonna say, if anyone wants to read more on this concept of deception, uh, check out. This is a, and I shared this, Andrew with a group that you, you were talking to recently. S uh, NIST 800, um, dash one 60 is I feel like sort of sometimes a little redheaded stepchild in security and, but it's long. But they have a whole inside of cyber resilience, they have a whole focus on deception. And so yeah, I do think these are fun things to talk about.

I think it's, it's great for like, like these are great beer conversations to just, what if we did this? What if we did this? And we should think through, like the inputs should be highly automated, highly configurable, but let us not have to have a lot of human interaction and outputs should be, you know, as much as possible true positives. And so if we think what that criteria in mind, I think we can come up with some great ideas and I think it would be a lot of fun to do that.

And, uh, and I see Andrew's point in the chat, who has time for that? And I think that's the way we, we we help ourselves is like, let's make sure that we're not building in processes that we can't possibly maintain. We'll, we'll just have Tim Fornet create a GitHub. Tim, can you take care of all this for us and just let us know when it's ready. Yeah, just a bunch of PowerShell scripts that MSPs can import into their arm. S and just Yeah. Stuff. Tim down from new businesses we need to launch.

Yes, that's right. I'll steal that IP baby. All right, Darren, I got questions for you. So, um, I come from an MDR and, um, you know, one thing about MDRs is the whole premise of what they're supposed to do is give us some kind of visibility into something that may be happening inside our, um, premise that we might want to take action on, whether it's known, um, you know, attack activity, it's something that's odd and, and un you know, that doesn't behave well.

Can you talk a little bit more about your experience in, um, doing adversary emulation with an MDR in place? Do MDRs typically detect a lot of this stuff? Is that not even what they're designed for? Talk to us a little bit about the interplay between MDR and um, simulation. Yeah, so, um, it, it's a great question and that it comes out in every class we teach multiple times. Um, so theoretically that's what they do is they detect these, these attacks, that's what they're supposed to do.

Um, unfortunately they don't detect them a lot of the times. Um, and, and so the question at that point becomes, what is it that you're really trying to test? Are you testing that endpoint protection or are you trying to test how well you can alert and detect those types of attacks? Um, I mean, when we get on a test at Black Hills, there's sometimes when you know, or put on a, a network and they say, okay, go for it.

And then you run some attacks and of course you're getting blocked by, you know, whatever it is, you know, whatever endpoint software they have. And pretty soon it's just, you're just kind of beating your head trying to get around it, and eventually you do or you can, um, get around it, but then you've wasted, you know, a day or two or three trying to get around this solution that they have that they know is pretty effective. Um, and so the question is what is it that you're trying to test?

Um, I think it's a good idea to make sure that they're, they're detecting what they say they are. So it's a good idea to run these tests with that knowledge in, in place. Um, you know, make sure that they are, it's also good to turn that off and say, okay, if somebody is able to bypass that, if somebody can get in there and turn it off, are we still able to detect it?

So It's good to run it both with it on and with it off because you, you do wanna be able to make sure that any of these attacks, I mean, they're not good, right? Um, if, if they can run on your network, we wanna know about it. That's something that we should know, we should be alerting on, we should be detecting.

Um, and so just being, just relying on your endpoint that, I mean, that's not, um, necessarily the smartest thing because I mean, at Black Hills we get around that, um, every test we're able to get around it one way or another. Um, but it just takes time. And again, you know, what are you trying to, to test? Yeah, I I really like that approach because that's a great, an great answer. I, I agree.

And, and I think that's really the way we should think through our design considerations when it comes to security, is understanding. I want to know what gaps that I have, and I intentionally want to like trial and error and test some things and turn this control off, especially if it's a preventative control, as you said, to really get better visibility into, you know, what would this thing be doing.

Like, a great example of that is like a firewall that might run, you know, some kind of like threat intel kind of stuff. It's doing all the GOIP blocking, but then it has some kind of threat intel that's coming in that gives us some, you know, insight into, um, like IP reputation stuff. It's great to have that in place.

We should always have in place, but it's great to see if we turn those shields down and get a peek at what's happening without the, without that happening, what else would we see and where would we see it? I think those are, I know Elon Musk, uh, one time said, I was listening to a podcast he was on, and he said something along the lines of like, I like to think of things like to reverse engineering things and say, what would happen if we don't do this instead of we do this thing?

And it, it ends up giving us a lot more insight into, um, you know, how things may, may or may not work, right? So I really think that's the right approach. Um, talk to us more about like what kind of preventative measures or maybe shields down approach we might take.

Like what are some good considerations if we're doing control testing, like I mentioned a firewall, but what other kinds of things might you want to take down when you're testing to see what would work and what would, what you'd see, Um, for, I mean, for sure you'd wanna take antivirus off. Um, so that's one of the big things.

Um, any kind of in, if you have web applications that you're, you know, you're looking at take out the, the, that WAF web application firewall, um, those are the main things. Um, any, any other, anything else that's, you know, stopping you from running things, you know, do you, if that makes sense. If there's anything in place that you have security wise, um, given enough time, attackers are gonna be able to get around it.

And so yeah, The, um, the French equivalent of the cert put out a really great update when started to do, um, uh, spreading over file shares. And they actually listed out, uh, automated like 50 or 60 different services and processes that would be killed in an automated way before it ran, because those were the things that would potentially try to stop it from doing its thing. So again, really great observable list of things that ran very capable ransomware threat actors.

And I listed Wizard Spider earlier in the chat. There's actually lists of these services they try to kill. Um, and like some of them are fairly basic, like star backup, star anything backup, kill the service, kill the process, right? And so there's a lot of good stuff out there from an observable perspective just to go look up and, and, and you can, you can look in your environment and figure out what types of things they're gonna try and stop. Yeah, agreed.

And, and I think it's even important threat modeling to understand what happens in a typical or a, a scenario in which a bad guy gets, um, some pretty deep level access to the machine and can turn controls off. You know, it's like we always rely, oh, this next gen AV is the best thing at detecting whatever. Well, what happens in a situation in which it's not running or a situation in which it's not deployed or a situation in which it's controls or are not effective for whatever reason.

Like, that's why it's important to take down certain things and see how it inter interacts. You look at how a scientist operates, you know, when they're, they're thinking through their scientific procedure, they're, they're constantly testing against a control set and then testing different hypothesis to see what happens here. Okay, now let's take that off and turn this on and turn that off. And so I think it's important to have those, those perspectives. I think it's good.

Um, so Darren, another question for you. This one's off script, but it, I just, I'm very curious to know, so you guys do a lot of these assessments, right? You mentioned earlier, you know, that's a big part of what you guys do at Black Hills. Can you give me some common outputs or common gaps or common deficiencies you see when you're finished with some kind of emulation, adversary, emulation, um, engagement with a client? Like what are some of the big gaps or big things you commonly see?

That's a good question. And it's not any one specific thing, unfortunately. And, and it's, it, you know, um, as good as the customers are that we test, there's very few that actually are alerting and detecting on our attacks. Um, it really is. It's very few that are, they've got, I mean, even the ones that have great, you know, great security features and products, and they're, you know, advanced, um, they're not alerting on a lot of the attacks.

The ones that, for me, when I'm testing the ones that are the most frustrating, the test are the ones that, you know, you run, if any kind of a PowerShell script, and then right after you do, you get an email that says, we, we saw you run this PowerShell script, you know, and then you're like, okay, well now what am I gonna do? And so you try the next thing and you get an email that says, we detected it, and you're like, okay, at this point, you know, you're, it's awesome.

You're detecting, um, I'm glad you're detecting, um, I'm gonna keep going. I'm gonna pretend like you're not, even though I would've been, you know, taken out way early. Um, and so those that are detecting things like that, they're, they're miles ahead in security, then the ones that don't have that turned on. We, Wes can I just ask that, and again, maybe this is the reverse kind of question, Darren.

What, what characteristics do those organizations have where you're like, the majority of folks we can just, and you guys work with big companies, what, what do those ones look like? What's the differences? Anything you can put your finger on, one or two things that do get you, do you catch it? They're always trying to improve.

So, um, if you do, I mean, we do f tests year over year with some of these customers and you know, the first time you test them, you're, you know, you have DA within a few minutes literally. And then the next year, you know, it takes a little while longer. By the time the third year comes around, sometimes it's almost impossible to get anywhere. Wow. Those, the ones that, that are, um, that are the best, they are constantly trying to improve.

Um, they'll get on the call, a phone call with you and say, okay, you were able to do this. What can we do to get to block you next time? Um, they're always asking questions. They're trying to learn, they're trying to get better. Um, those are by far the ones that, you know, the ones that take it seriously, I guess. And when I say seriously, they really are trying to improve, not just, yeah, we take it seriously and then they get the report and nothing happens after.

It's, it's like this attitude mindset, right? And I've seen this before too, is, you know, some security leadership at some orgs and even board leadership is like, you know, we don't want, we want to clean report. We don't want anyone to find anything on us. And so if we find something in the report, we're gonna like, wait. Yeah, but it was this and this and this. Versus the ones that are like, heck yes, you found something. I don't care how remote that is.

That is awesome and we want to find more. It's exci like, it's almost like we're incentivized by you finding workarounds or gaps or deficiencies. Like it, it should be celebrated and excited, right? We should almost like think through as the ciso, how do I reward that kind of behavior of finding those things versus, you know, sweeping under the rug. And I think it's, that's going away more and more, but it's still prevalent.

And, and MSB should think about that too, is, you know, how do you treat and celebrate versus hide? And, um, try to explain away. Ryan, I see your defensive here. And the first I Was, I was thinking what Darren is saying really is what's driving what we're seeing in the MSP marketplace. We're starting to see a wider separation in terms of growth, profitability. And this is one of the things that are driving the type of organization, uh, that they are, right?

Because they're the ones that are really working and improving and moving their posture forward. They're, they're easy to figure out how to monetize it from a prospect and a customer standpoint. Yeah, I completely agree with, with Wes said, CISO's set in a very interesting spot, right? You have to incentivize your team to find all of the areas that need improvement continuously. Um, and I mean, we built a, a Datto, um, I actually fi funded the building of a WWE belt.

Um, and the pen testers have to pass the heavyweight belt around, uh, based off of who has found the next kind of most serious, most interesting vulnerability. And so it's actually like a, it's like a, a heavyweight match. And if we haven't had it switch hands, we'll actually set up a challenge that's like a really difficult challenge. And then whoever wins gets the bell and it's like this thing, we literally ship it from house to house to house now, and they put it in the background, right?

The crazy things like that. Like we've done like, um, happy birthday crossey scripting cakes before, like fudgy whale cakes and stuff like that. But on the other side of it is you have the board and they just want the clean board. They wanna feel like they're safe, they wanna feel like they're secure.

And so you have to sit between this place of really incentivizing finding the things that are just not known yet, and then figuring out how to communicate them to, in a safe way that doesn't make people feel like, um, 'cause they, they, they just don't always have context. They don't know, they don't know what to make of that information. And so it's your job to to, to help digest it and make it psychologically safe for them to consume that information.

Uh, and that's, but Ryan, if you're an msp your customers are your board. Exactly. That's exactly kind of why I was going down this pathway is your customer shouldn't fear if you find something, it's a teachable moment for them, but you know, you don't wanna be like, you know, oh man, this is really bad. You, you know, you'd be like, this is, we're gonna work through this. We're gonna figure this out. You know, everybody has things to improve on.

They, they don't have a framing for how bad the thing is that you just found. They have no context. And, um, and you're not gonna give them the context they need. They need, they just need to know they have a partner that's gonna help them through it. And so you have to figure out how do I, how do I walk both of those lines, find everything I possibly can, and then how am I gonna package this in a way that's very digestible for them to get bought into?

We, wes I know we've gone on off script a little bit. Can I ask a question just to Darren while I think about it? 'cause there's MSPs, Darren, that still complain that not, not complained, you know, the legitimate situations where they can't get customers to implement MFA. So knowing that, and you know, again, now you're an MSP and you have your background on this, what might you do from an emulation perspective to go, let me show you what your pushback of MFA has will result in.

Here's what I did, you know, kind of a thing. So, um, one of the things that I do on almost every test is, uh, something called a password spray. Um, where you get a list of users, um, and then you try one password over that list of users. Um, and then it, you wait an hour and then you try it again over that list of UA different password over the list of users. Um, and, and so this, I have yet to lock out any users doing this.

Um, and so you can keep guessing you, you know, password and then after password. And then if you ha I mean, if you have a list of common passwords, it's season and year works very good. Um, it's amazing how many times you'll get in. Um, and once you get in, if there's no MFA, I mean that username and password gets you anywhere that user can go. Um, but again, if you show that there, there's that MFA in place, even if you get the username and password, you're stuck.

You can't log into anything without that second factor. Yeah. And that is, I mean, it is a crucial finding. I mean, for us it's a high finding. Um, sometimes it's even critical depending on the, the password policy. Um, if the, if it's a weak password policy and someone, I mean, gets it, it's game over. Yeah. Great stuff. And Keith, I get your point too. I, it was just, uh, curiosity. 'cause we hear that so often, Gary, don't, don't you still hear it?

Like, Uh, yeah, it drives me crazy, uh, when I hear it because I'm saying you have a fundamental relationship issue with your customers. If you're not, if they're not able to see their risks on something so basic the same way that you are, you gotta take a hard look in the mirror. It's great. You're gonna have some tactics if that can help you. But I think at some point it just has to be your belief system and it has to be ongoing.

You, you don't have an ongoing relationship, uh, about how your customer needs to view their security posture. That's a problem right now. This far down the line. Yeah. Well, I know we're at the, pretty much at the end here. I'm gonna ask one last question to you, Darren, that came into the questions from Matt Clark.

And he says, when you're doing a penetration test, um, do you inform them, I believe he means the security team, the organ, you know, that you're gonna be doing scanning, give him a window of time, like how does that work? And we'll wrap it up there and, uh, close things out. So when we start a penetration scan, we, we always have permission. So kind of what the, what happens on their side is kind of, you know, up to them.

Sometimes they want us to not tell the rest of the security team and just see how far we can go. Um, if we start with a scan, it's kinda hard to hide a scan. I mean, that's, those are pretty noisy. So if you're not detecting the scan, you've got some serious issues. Um, and so if, if they want us to do the test without anybody else knowing kind of like the, a red team type engagement, um, we don't, we won't start with a scan.

But then, you know, if we, after, if they detect us, um, then we'll say, well, let us run, we'll run the scan now or, you know, halfway through the test we'll run a scan or something like that. But you, you know, it depends on the type of a test that we have. Excellent. Well really, uh, can't thank you enough Darren, for coming on.

Um, I know this is kind of the beginning, the beginning of something, um, that, you know, we're, you know, the, the courses you're doing, the things that we can do, um, with the MSPs as a critical component. Gary, any closing thoughts from you as I kind of loop around here? No, we're not gonna give up. Our thing when we look at MSPs is no child left behind. Fantastic. Wes, closing thoughts.

It's rather apt that Gary looks at these MSP friends as his children, uh, giving me sort of like the godfather here. I kind of like that, that analogy. So my closing thought is this is, we're going down this journey of security and testing controls is sort of a scientific approach which we should celebrate and look for in security.

We have too much of throwing things at walls and seeing what sticks and too much subjectivity in the industry, which then lends into confusion and, and you know, a need to demystify where we're at. And so I love the discussion and hopefully everyone's come away with some degree of, I think understanding what adversary emulation is all about and how we might begin to incorporate it, even if it's a baby step, is the right direction. So, um, great call today. Thank. Yeah, thank you Ryan.

You're still your thoughts? I know you're a big fan of the topic. Yeah. I mean, know your enemy, right? You've heard me say that thousands of times. You cannot have a risk aware, threatened forms cyber defense program. If you do not know your enemy, you have to know your enemy. If you have any shot at doing this, you have to know your enemy. Excellent. And threat profiling and threat emulation is the way to go. Fantastic, Darren, we will be in touch. Really appreciate you coming on. Thank you.

Yeah, it was great. Thanks for having me. Alright Everybody, take care. We'll see you soon. Alright, bye Bye-Bye.