MacEnzie Brown of MSFT DART discusses IR Competence & Capabilities

It's always like, rule. No, we welcome bad words. So you're Good. We welcome. And we're live. So, uh, yeah. Um, yeah. In fact, uh, our guest last week said, you know, the, you know, he is like, you know, s**t. He's like, oh, Andrew, can I say that? I'm like, no, you can say more. Please keep going. We are, you know, we're more PG 13 to R rated than we are, you know, G Okay, Mackenzie, you're, you're fine here. Absolutely. Um, welcome everybody. It is great to see you all.

Who is gonna be in Orlando this week at, uh, IT Nation Connect. Let's just see some yeses or nos in, in chat. Uh, we'll see, uh, what people say. Uh, there. Uh, I'll get things, get things going and rolling here. Uh, before I introduce the very awesome Mackenzie Brown from Microsoft Dart, uh, just wanted you guys to know, I told you to register for Write of Boom. If you wanted to see John Strand and John Hammond's pre-day, that is sold out.

So please, um, don't say, can I still get into John Hammond and John Strand's pre-Day? I, I told you guys it was gonna get sold out quickly. It did. Um, however, there will be some cool pre-days coming from Huntress, uh, from Black Point from Roost, uh, Gary and his team at Kaseya, uh, and a few, and who else, who else, who else it is, I'm trying to think of the other one. Well, I'm gonna, I'm plug mine, Andrew, I'm gonna plug mine Andrew.

Me, me and Chris Laer are doing a really cool tabletop with the Hunters folks. It's gonna be Awesome. Yeah. Yeah. With Eric Tilt, I think too, right? Yep. Yep. It's gonna be awesome. What's really cool about this is it's gonna be not only just a typical tables table top, I, you know, you can't all be together. Sure we can. Oh, I thought you said, I thought you said IBS I'm sorry, Gary. I gotta check the, I gotta check these ear things.

Um, but what's cool about this tabletop is we're, oh yeah, by the way, Andrew, I like, I like your new earrings. You know, I had to do it. Wes goes, let's just look. I, wes, I gotta, I gotta turn off all these dinging and doning and all this stuff. Wes' just like, put your ears in. So that's why I'm doing it. Wes will get we, Wes, let's, let's help 'em out with AV after this. Okay. Yeah. All right. So anyway, um, anyway, register for write a Boom. It is gonna sell out, um, quickly.

Um, let's see. And oh, and then we're gonna have an awesome cyber cast out with that. We put together Phyllis, uh, uh, Wess and Ryan on security and awareness education. Phyllis is literally changing the control from training to, to education, uh, per this last, uh, recording on our podcast. Alright. Enough funny games, setting the stage. As you guys know, we've been talking a lot about, um, incidents over the past three to four weeks.

Uh, we've been speaking about crisis communications, uh, and we've been speaking about things like last week, even around DDoS. But the thing that we really wanted to hit on is, as I talked to MSPs, Wes and I have spoken about this at blank, and, you know, when does in IT incident begin? Uh, sorry, end. And when does the security incident, uh, begin? And what I found is, uh, most MFPs don't have a good process or defined process around it.

Gary, in your estimation, is that fair with all the coaching you? Yes. Yep. That's why I'm happy that McKenzie's here, uh, although her name comes up on here as Mac Brown, so No, there's like a couple Mac Browns too. I mean, I, I, I probably registered as Mac Brown 'cause I was on auto. Well now you're Mac brown, so It, it's, I mean, it sounds cool, but You're the Mac brown. It does sound cool. Yeah. The Background. Yeah. Yeah. My album drops. What that Sounds like. Yeah.

So we're glad Mac is here. Yes, absolutely. So happy to Mac, thanks for joining us. Um, I have to tell you, uh, about, I don't know, three, four weeks ago, Wes came back from an event and he's like, Andrew, you've gotta have Mackenzie Brown on the cyber call, and if possible get her to write a boom. So, and you were cool enough to agree to both, not only just being here, but then leading the incident response panel at write a boom, which is really cool. So welcome.

Tell us a little bit about yourself and this small organization called Microsoft and the dart Yeah. Small organization, right. Emphasis on, on small. I've learned, I learned that the other day, um, that we have almost as many employees as the US government as a whole or something. I was talking to Andrew about this too, of the, the big wide world of Microsoft. I've been here a little over three and a half years, um, at this company. And it is in fact the big wide world.

Like, we had a guy on our team who, um, I was about to be like, I miss him so much, but he's still around. Uh, he's amazing. Russ Rogers, and he was a big mentor when I first joined the team. And, uh, he just decided, I think I'm gonna go over to Xbox. Like, that's how big Microsoft is. And I'm really kind of don't fathom it still on a day to day. It's not really like drinking from a fire hose. It's like getting waterboarded by a waterfall, probably. I'm getting the hang of it.

So, um, but yeah, I'm at Microsoft. I'm on the detection and response team, or dart or DART team 'cause we like redundancy. Um, and I'm an incident manager. Uh, I've been an incident manager new to this role. Uh, when I first joined the team, it was actually to develop tabletops, build out our tabletop service, and I call it more like a hybrid tabletop because it was a little bit different. Uh, and then I came into this incident management role.

There's only a few of us, and we are like the, we are the pulse on what's going on operationally on our team. So all of the intake, all of the, um, customer cases and incidents that are coming in, we essentially, I kind of describe it like fantasy football.

We, we pick our players, we put 'em together, we, um, do the initial triage conversations with the customers, and then we get them onto the road of investigation support, and then, and then we hand off to our other cohorts that do, uh, recovery. So as an incident manager, I just have to make sure I am, um, involved in every single conversation that's going on, uh, especially the ones with the lawyers. Uh, or I prefer to have the lawyers there sometimes to protect me.

Um, but they're, uh, yeah, that's in a nutshell, my job. Can you, can you tell us like maybe about what would, can you explain something that would be like, um, a, a comment, like one that would be common that you see most often that you're dealing with an incident? As far as incident? Well, yeah, it's, it's low hanging fruit, but, so we just released on Friday, I think, uh, last Friday was the Microsoft Digital Defense Report for 2022.

So we grabbed all of DART data and of course, um, we have data coming in from the defender research team, from mystic, our threat intelligence team, and from other groups at Microsoft. And, um, the majority of part, one of the stats I saw that we had put in for dart, the majority of our incidents, like 50% of them, at least half of them are ransomware. I mean, wow. Oh, I thought that was like the ransomware theme music that goes on, because Yep. So a lot of it is ran.

That's Gary, that's Gary's ringer. That's Gary's ringer. It's, it's his theme. It's his theme song. Whenever he hears ransomware, he's Exactly. So yeah, I'd say that's probably a, it's been a hot for a couple years, as we all know.

Um, but, and, and surprisingly though, and MD in the digital defense report, I also read that, um, we had a decrease in the amount of ransomware cases, but the concern from law enforcement and agencies are that that decrease is usually a pause as they, as they being the bad guys, the threat actors are recalibrating their strategy and their tactics. So, um, we are definitely concerned currently right now with some of the cases coming in for ransomware. Yeah.

So Mackenzie, when it comes to like SMBs, you know, the majority of them right, are managed by MSPs, right? Our, our primary, uh, audience. And so they end up being like the initial triage point, right? Uh, in mm-Hmm. In a lot of these cases. Um, not always by default. Uh, not only, not always because they're, we, we may not always be the most qualified.

So, um, what I want to know is like, what are some of the skill sets required to, you know, successfully, um, um, identify, um, incidents, like what would be some of the skill sets that an MSP would want to have to do it well, Um, so skillset wise, you know, if I'm thinking the totality of the kill chain and thinking of initial access and credential theft, a lot of the techniques that are often employed before the purpose, the actions on objective, like the actual goal for that threat actor, um, a lot of living off the land, a lot of things that don't necessarily generate noise or on the other hand, very normal things that generate a lot of noise.

That being said, I, when something comes onto our plate, that's when we could probably use the term incident, but a lot of the IR lifecycle is really in the beginning of just the validation. I think the biggest skillset to have is the ability for a team to triage and validate what they're seeing and then determining if it is in fact a true positive. Um, because a lot of false positives can create incidents or declare them incidents that are not accurate or they're not technically true. Yeah.

And, and you know, like, again, you, you can tell by the fact that, you know, Andrew named the conference, right? A boom. Uh, one of the big things we've been trying to do here is as MSPs as a community, we historically have just been so focused on protection, right? And because it's technology, we're good at the technology piece of it.

And now, like what you're describing is, it's not always easy when you're right of boom, uh, to be able to even know and understand one, to find things that are there two, like you said, deal with false positives and, uh, that can throw people off. Uh, you know, many times it's almost like when you get a lot of alerts, no one alert seems as important. Yeah, right. Yeah. Lots of noise out there.

It's, so, it's interesting too, at Microsoft, you know, we have a separate team before it escalates up to dart that does some of these smaller scope validation. Um, so a lot of, like, it's really important for people on our team and we train, you know, accounts and sales development teams all around Microsoft who work with us in any capacity how to use like that correct terminology and when to use it. Because an incident doesn't necessarily mean a breach.

A compromise doesn't necessarily mean an incident. And so we have these teams that do this validation process where they examine some artifacts, definitely smaller scope of systems, some logs and account activity to say, okay, this is, um, a level of bad usually, you know, domain admin type of level of bad that requires a full investigation, but that that triage and validation part that's directly competing with like the noise, the actual noise that, so see on a day-to-day basis.

So having the skillset to get through the noise and find what's considered high fidelity detection, um, that's, that's the skills we, we really need to be successful. Yeah. Um, so most MSPs, uh, from a soc you mentioned soc from a SOC standpoint, um, the majority of them are outsourcing that they're working with a third, you know, they're working with a third party. What do you think would be like some of the, um, what would be some of the, uh, insights on how to develop talent right, in a soc?

Um, because I feel like even to manage that relationship, they have have some of those talents. Does that make sense? Yeah, definitely. Um, uh, talents, talent development. I mean, first off, I think I feel bad for recruiters nowadays too in tech because they'll find, um, people to become analysts in their SOC and then the, uh, now they're just trying not to get those people poached because they develop their overall skillset and expertise and then after a year they go for the next big thing.

Um, but as far as developing talent goes, I always think it's easier to teach technology than it is to teach critical thinking and the complexities of what we're seeing out there, um, as far as we're gonna see a bunch of alerts pop up if we're having a sock.

Again, the validation and the triage of those alerts determining that they are bad and this has a high fidelity and we need to do some additional threat hunting, um, threat hunting in itself for our team, like that's one of the top things that we teach. The minute, uh, we're hiring an a whole ramp up of people, we put 'em through threat hunting training and they do it. I mean, you can hunt in very many different aspects. You could do it via whatever platform people are using.

Um, or you can just do it through just general data collection and then hunting through that data. But the people that do really well, ironic, I shouldn't say ironically, the people that do really well at that are ones with the network architecture, network engineering background, um, ones that have the ability to know exactly from a traditional aspect for on premises, know exactly where they need to hunt. And those skill sets are super valuable. And for our team, right?

We, we may want to hire, um, uh, a new security analyst or someone that's gonna come in and do hunting or endpoint analysts or what we call them, but really do digital forensics. Anyone that can do digital forensics, that's probably the number one skillset that we need next to hunting.

And then things like active directory, I know it sounds random, but we sometimes are in desperate need of people who just, these are like cis admins if you think about it, that know active directory and they know the general even CIS internal tools and they can actually go through that.

Like, those are the people that we strive to get on our team because that expertise is harder to find, um, experience wise than someone who's worked in a SOC previously but only has like stared at Carbon Black for, you know, 10 years. But that's all they've done. Yeah. What, what's interesting to me is, McKenzie, you just described a whole bunch of MSBs, right? Gary, like really good CIS admin capabilities, really good network engineers.

We just see a big max mass ex of MSP folks going over to Microsoft. Yeah, no. Well, uh, Wes, what I was gonna say is, you know, the hard part about this, you know, we, we teach MSPs talent in general. It's hard to find. So find people that have, uh, aptitude, right? We give 'em to wonder link tests, we test their logic and then try to give them the rest.

The problem is, when it comes to those other generic things, that's our domain knowledge is MSPs when it comes to some of the things we're talking around about around security, um, identification, threat hunting, the hard part is we're sometimes looking to that person to bring that with them. 'cause we don't have that same domain knowledge with inside a, you know, an MSP that has, you know, 10 or 15 or less employees. That's, that's the hard part.

Yeah, I mean, I really like what you said, Mackenzie, just 'cause I'm former NSA and those folks also were the best. Like if you were a, a Unix CIS admin, right? They would scoop you up. Oh, I mean, right now, if you think about critical infrastructure too, how hard is it to find?

So I have a friend who it does SCADA based engineering and, um, he probably makes really good money, but he is a very busy man because they, that is a very hard position to fill if someone who maybe knows like ladder logic or can work on a PCL or something like, so for us, just even from a security aspect, once we have a customer that's critical infrastructure or ICS type of controls, that those are the people we really need on our team. And it's a much smaller scope of people that can do that.

Gary, can I mention something real quick? Mm-Hmm. I had a fascinating conversation. I don't know if you or the folks out there have heard of a lady named Naomi Buckwalter. She's actually out of Philly, the Philly area. Um, she only has like 60,000 followers on LinkedIn. Um, and she's, um, phenomenal. And, uh, she worked in big financial, uh, initially in security. She has a, a foundation now a nonprofit called Gate Gate Breakers.

And one of the things she was stressing to me, and we're gonna have her on the cyber call, was that we are not training the next generation of security people and therefore we're in this race. Like, almost like you said Mackenzie, who can poach from who, and she's like, how are we gonna solve the security issues in our nation when it's, you know, I can pay more than you and you can pay more than me.

And it's, and we're keep taking each other's talent and what she's really focusing in now on, I'd love your perspective, is there's a whole like litany of, of, you know, folks CTOs that don't know how to ask the right questions in interviews, don't know how to craft the right job description, and these are the things she wants to help with. This is what her foundation's about. I'd love your perspective, Phyllis, maybe yours as well.

And then in chat would, would understanding what the right questions are and how to build the right, um, job descriptions be helpful to you, uh, will, will bring her on. But Mackenzie, your thoughts on that whole, whole, you know, perspective.

I mean certainly, I mean from the CTO comment you made, I, I think people maybe would think, say you're trying to fill a CISO position, are you gonna hire the CISO that's never had an incident that's never gone through chaotic type of scalability within their environment that hasn't historically been with a company a really long time and maybe seeing migration to the cloud and, and a lot of changes. Technically, no.

You're gonna wanna hire the person that has, you know, who hears the helicopters coming, coming in their head and they're like extremely worried because they've been through it before. They have a little PTSD from an IR side, if I think about like who you want to hire, we get so structured into these, you have to have X amount of experience and this degree path or this background, and you have to know all of these things that is just not realistic, right?

You want someone I think that has kind of the war stories and has been in the trenches on top of some traditional knowledge that is still needed today, foundationally to be good at your job. So, and all of that goes with asking the right questions. Um, and you know, you learn that the hard way when you get into an investigation or we're working with a customer that's never experienced an incident before or they told us that, um, but they've never experienced one before.

And so they don't really know what questions to ask that are even nuanced to outside of technology. Like, um, when to call attorney-client privilege and like making sure an attorney's actually there when you say it, it's not like, you know, bingo, but like making sure you have the right people in the room knowing how to navigate cyber insurance and, and what your insurance policy is requiring from a business perspective. Like how to actually when to start disaster recovery and containment.

Like these are the things that people wanna hire for, but we're so stuck on, stuck on this like, very structured idea of what a good person or, or what a good, um, recruitment of a person would be in that candidate. Yeah. I'm just gonna put this in Gary, and I want to hear you Phyllis. I'm gonna put in the three minutes from Gary and Ryan Eer last week in chat right now this YouTube because you, something you just said, Naomi, which is, walk me through when there's an incident.

Walk me through your process. It's like one of the greatest sales questions I've ever heard. And you know, if you can understand what's right and wrong in that process, Gary, what kind of separation can you get in a sale? In in? Yeah, absolutely. Listen, it's why, you know, top security first. MSPs are having such success right now because we have wedges to create separation in the, in, in the sales process.

But the process is exactly what McKenzie just described, which is figuring out people's real world experiences and whether it's interviewing or in our case when we're talking to a prospect, you know, we're talking to a customer a, a prospect rather. Um, we, we don't, they're not able to answer the question a, a huge majority of the time. Phyllis, you wanna say something? Sorry. Oh, I was gonna talk about the talent gap. I think it's a recognized issue, honestly.

And you know, the US government actually recognizes it. CISA has been given money DHS to actually try to close the gap. And many organizations, you know, we as a nonprofit also put money towards this. I think it's a difficult problem and I agree with Mackenzie. You know, I used to do recruiting at the NSA and you get so many resumes, um, and those resumes don't necessarily reflect the skills of someone that you would want. Like those, someone that has those analytical skills.

Um, and so, you know, what we recruit for are GPAs and school name recognition, um, for people right out of college and like the folks that maybe don't get good grades and that do run their own ISPs out of their college dorms and stuff like that. 'cause we recruited people like that when I was there. Um, and now they're super successful.

They've moved off from the NSA, but like, you know, the, the problem is like automation also has killed it a little bit because people use automation at companies to try to get call through those resumes. Um, but you know, it is a, it is a, and I'm sure Microsoft gets tens of thousands of rems and they have to do the same exact thing.

And so, you know, it's really hard for folks to get their foot in the door and, you know, explain, this is how I have analytical skills and this is what I can bring to the table. So I think we need to recruit for different things. And so I'm, I am somewhat optimistic, but the proof is in the pudding. You know, Google now has certifications, right? So Google has said, Hey, you don't need a college degree.

We're gonna put out these certifications and try to find ways that, you know, perhaps people who college is not that route is not their route and can't afford it or whatever for whatever reason is not their route. Um, I'll be interested to see who actually would hire someone who, here, for example, here we have MSPs. Would you ever hire someone that didn't have a four year degree but perhaps had a certification, right?

How open are MSPs to recruiting talent that don't have, you know, those typical, um, credentials that organizations Yeah, we just have to get, we have to get creative on this, right? Um, yeah, as an example, uh, Eastern Kentucky University, I'll throw them out there for a minute. They have an accelerated cyber program. They're literally taking people from coal mines that have been shut down in eastern Kentucky and West Virginia that don't have a job. They're training 'em in cyber.

And then the university's got like a, some like co-work space that they can work out. They're literally hitting the easy button for, for people. Wow. We just gotta think about how to get creative and there's some great ways that we can do that. Um, and I just wanna throw them out there because it's an awesome idea of taking a destitute area that's been ravaged by the coal industry falling apart. Why not cyber? It makes so much sense.

So yeah, Listen, as someone who, uh, was on academic probation, two of my four years at Penn State, I definitely would hire someone without a college education. I could have made better use of my time. Yeah, I, I mean I, I used to be really embarrassed about it 'cause I don't have a formal degree and I went the certification route and I went the networking route and, you know, just hustle and grind type of way to get into the role that I wanted.

Um, but I'm going back to college right now, uh, for cyber ops and it's so funny. I'm looking at like, okay, these are all the classes I gotta take for the undergrad part and I'm looking at them, uh, on top of, I'm sure West realized this, everyone knows e everyone here in Boise, um, it's very small security community. You could stick a hundred of us in a room, but we know, um, who's looking for a job or who's not.

But, um, I know all of the professors, like personally have hung out, drank a beer with lock picked with, gone and did consulting with, um, all of the professors teaching for these degrees. And it's just funny 'cause now I'm, you know, I used to be a lot more embarrassed of it and now it's just kind of funny for me to go back to school and try and finish up and seeing that I, I guess I could just text my teacher and see what's going on. 'cause I know them. Um, yeah, it's formal education.

Someone had comment about that, but it's, it's shifted a lot. I like to think of cyber is more of like a trade skill that we need to do, just similar to you're getting hours in if you're to be a plumber, an electrician, like how do we create that trade skill route for IT instead? And McKenzie, if I may, you, you, one of the things we talk about a lot in, in on cyber call is, you know, there are like, to be a hairstylist, you have to have more credentials to be an it, right?

And you just mentioned a trade organization. I, I like that idea. Do you see like some kind of need of like a defined minimum standard that brings you into the trade of like cybersecurity? Could we do that at some, at some point? Should we do that? What are your thoughts? We definitely should do that.

I mean, um, we're talking about right, this gap that's only going to consistently be around, but also on the other side of it, the poaching that's going on between companies who are able to offer, you know, 40 k more on top of a salary, who wouldn't go take that during this time? Um, and I, I, I think it would be better.

I have, I have a couple friends and one of them ironically is a hairstylist, um, out in Mountain Home Idaho, and she went, she finished her degree and I just emailed her a couple names around town and she was a hairstylist for her daughter just went to college for 18 years, or a little over 18 years, almost 20 years. And now she is a cyber analyst. Like, just, That's awesome, Right? Like really weird flip, but maybe we need what, what are those like aptitude tests you take in high school?

I, we need to develop, maybe start with that. Start with the aptitude test. I'm sure the NSA probably has a good one though. I was, I was gonna say Mackenzie, I love what you're saying. I mean that's literally, you know, we've had John Strand on here many, many times.

It's this thing, you know, he, you know, he literally have his job boards now and you know, they in in, in their antis siphon training and he's taken moms that are working, you know, single moms working two jobs that want to get into cyber and he's literally gotten 'em into, you know, his, you know, paid what you can classes, he's helped mentor them and now they're, you know, he's giving them their first sock job. Yeah. I mean, oh yeah. So it's possible.

And, and it's, it's really cool that we have people out there like yourself and Strand. Um, so very cool. Yeah, we definitely need to find a way to transition or evolve the grassroots efforts to something that's repeatable and happens a lot more. And I'm sure there are companies out there that are, are trying to implement this and do it, but, um, yeah, I I am a big proponent for doing what he's doing there. Yeah. Very cool.

And listen, this conversation that we're having this past, you know, 10 minutes about this, it's really important for MSPs. Like we need to be what I always teach my peer members. We need to be really, um, aware of what's happening around us right outside of our company and whether it's, um, what we're talking about here in terms of talent and what we need to think about, um, or whether it's, you know, the economy.

Uh, like there's so many things, right that affect an MSP right now that didn't really affect them three or four years ago. And so this is really good. It's, it's really good that everybody's aware. So. Well, I had one more question, but um, do we have time for me to ask? Yeah, go. Yeah, yeah, yeah. You're fine. So Ryan Weeks couldn't, was gonna be here and he could ended up not being here today.

So I'm gonna ask a question on his behalf, um, to, to Mackenzie, um, which is, uh, Mackenzie, um, why is asset management so critical in identifying security incidents? Because it's never done. Uh, yeah, no, it's, it's definitely done it'ss. So there's this thing on, um, you can find it out on GitHub. It's the hierarchy of incident response needs.

Um, and when I first joined Microsoft, um, someone on our team who's amazing and um, has since left Chris Kirk, he actually sat me down in a conference room randomly, is like, let's look at this. And the very bottom. So it's like kind of, you know, the, the hierarchy, Maslow's hierarchy of, of, of needs. Um, it's similar to that. And the very bottom one is, is asset management or rather inventory. And this is the first thing. We even talk through this in our tabletop exercises.

I would bring it up because that is the most foundational part that needs to be done. It's the most important part because we don't have, we need, in order to be good at response, in order to investigate, we need to have access to data, which relies on telemetry and visibility.

But if we're coming in and deploying a tool, there are so many times that we've identify systems and, um, systems that they thought they had retired that they thought weren't going to, that weren't in use, uh, they, they didn't even know about it that had been spun up. Um, that, that Their tools aren't deployed on availability. Yeah. Yeah. All of that impacts investigative ability and to be like really good at it at incident response.

When you're looking at a system or we're talking to a customer who's like, I don't even know what that server is. I don't know what this endpoint is like, that is not a managed device. So asset management, like what are your critical systems being at that tier zero? Like what are the keys to the kingdom first and foremost for asset management that we have to really protect?

And then on the other side of asset management is how are we actually doing remote management and how are we patching and how do we know classifying those systems is the biggest thing? Um, especially if they're all ran by the same service or multiple service accounts that like domain admin, like this is the, again, going back to like the MSP stuff, right?

Those, that's the talent you want, you want, you want the people who understand that whole architecture concept of, um, you know, basic network architecture, but also your endpoints and operating systems and patching is great, but asset management impacts everything. Yeah.

It's like most things, like I I I find like that so many times we go on to step three and four and five before we've completed step one and, you know, completing step one is always the best route, uh, to take and maintaining step one and then going on to step two.

Well, well it's like, it's like, you know, as we go into Phyllis here, this is almost like last week Phyllis and Ryan kept saying what a fanny was of CIS, uh, I know now your favorite guest, but, um, but it's fundamental, like what makes great teams, right? I mean, Gary, I mean it's fundamentals. It's always fundamentals and we miss, you know, the part A and part B interestingly, what does a threat actor do initially to gain access? They identify, right?

I mean, and that's one of the things we're gonna be looking at, right? Mackenzie, through the lens of a threat actor, that's what they do. And that's how we're gonna, by the way, the journey from left to right, a boom this year in, in the event we're gonna look through the lens of the threat actor so we understand what they're doing so we can build defenses and take things to market again, you know, for them. So I'm glad you brought that up, um, Phyllis. Yeah. Yeah. No, thank you.

And I just wanna foot stomp what Mackenzie said because, um, you know, like Andrew said, the first three controls are really know your environment, hardware, software, um, asset management, and then know where your high value data is, right? And so often across the board, regardless of the size of the organization, they're like, okay, Phyllis, I got it. But if you just have to recommend three to five things, what would those things be? And they don't wanna do it. Those first three.

Yeah, those are all, we're recommending the same things all the time. We get that a lot too on dart. Like they're expecting us to pull AI out of a hat and we're like, no, no, we would like you just to turn on MFA, let's just start there. Yeah, let's just, let's, they're all like, what's the layer defense approach? What do we gotta do? We're like, just enable m FFA and done. You can get rid of some of these legacy things, you know, and classify your data. Just do a little bit of data protection.

You know, we and immutable backups, right? We could go down the list and we're like, there's, there's a handful, just do this. But they don't really, No, it's awesome. That's awesome. It's great to hear you say that with your operational experience, so thank you for that. Um, so, you know, there are many MSPs on this call. They're on the front lines and, and you really emphasize the skills that you need really understanding, um, you know, basically expected behavior or the good from the bad.

So, um, what are your thoughts on how to document ex ex post facto, um, anything that's been done to a system, um, prior to determining that it's a security incident? So, um, you know, ideally what kind of process do you think organizations should be implementing so that um, you know, if it is an incident, you don't destroy that ev evidence and you can, um, recover and figure out what happened? Yeah, well definitely don't turn off the system.

I mean that's, you know, defer 1 0 1, they'll always tell you like, don't get rid of that to the, the memory that's gonna go bye-bye in a second. Um, I, I think what we see a lot too is isn't necessarily lack of technology, although we would like to see a lot more companies investing in EDR. Um, the other statistic that I pulled from M DDRs report, the ones that I find really interesting are like 60% of customers didn't have EDR across their systems.

And, you know, there are obviously ways to get around certain anti-malware antivirus type of things, but um, I honestly, I think as far as getting to determining from event to instant, like event is suspicious activity, something nefarious, and you need just enough to really say, yes, there's something bad going on. One, we need specific log collection that's not done. And a lot of these things aren't skill sets, right? They're just decisions to store data is often retain it.

Having longer retention when we risk a system, perhaps it's been wiped or already rebuilt before we get it, which also happens. Um, and customers won't tell us that. We're like, we need you to pull a forensic, run our forensic tool on this system and then they'll send it back to us. And the alerts that we're seeing do not match what we're seeing on the system itself. We're like, oh, someone rebuilt this before they gave it to us, but they won't admit it.

We're like, oh, it just, okay, well some data, it's, it's where can we find that data then, then we're relying upon. So it could be in a seam or some sort of storage collection. It could be, um, uh, obviously in a, a platform like an EDR service, at least there's something there. But from a chain of custody and preservation side, it's so important to know how to isolate, having some sort of ability to isolate that system, not turn it off, protect it in into that case.

Um, also if you have, if depending on the system, so asset management, if you know what that system does, perhaps it's something very important, what is your process for preservation? Getting a copy of it for evidence. Um, people need to usually forensically probably should have a lawyer look at their processes. Like those are, those are the processes people don't think about. So when we see a system that's of interest is, is what we would say, then we would ask cus Mr.

Customer, can you, uh, we see this system, can you run, what is this system? What is the importance of it? Um, it's doing this, can you run our forensic tool on it? So then we could individually analyze that system. Um, the customer's expected to say, I know what this system is, that's extremely important. And sometimes they're even like, I'm gonna get our lawyer in one second.

Um, there's so many things that go back to knowing what that system does and then being able to isolate and protect it. Because even if that system's deploying a bunch of malware across the entire environment, you don't wanna lose the evidence that's sitting on it. You want to know how it was actually done, how the threat actor was actually able to manipulate or utilize this, leverage this one system.

And if you focus more on wiping and rebuilding, that's, that's what's gonna mess you up in the long run. Um, whether in a small scope or a large scope, right? Because the threat actor's just gonna simply come back and you're probably leaving remnants of adversarial activity all over your place because you wiped that one system that had a web shell on it.

That web shell could be sitting anywhere else and now you, but you don't, you don't know that because you don't have anything to compare it to. Right, right. Can I jump in two seconds? 'cause one of the things Mackenzie said relates to the question that Tim posed in, in the question section. He says, any suggestions on how to define boundaries between IT problems and security incidents?

Um, and, and this is a really important thing, it's kind of the narrative I gave Gary back in the beginning, which is a struggle for most MSPs. And I think you kind of hit on it, it's a nuance Mackenzie, in that you said, Hey, suspicious activity seems potentially nefarious in in nature. And that's where I think it's critical as MSPs that you define those terms, make sure you review it with your customers upfront and you know, this is the best of the best out there.

I'm not gonna say who, but they tell their customers first flat out. This is where, um, our MMSA ends, this is where our, you know, everything inclusive in your agreement ends. We are going to now move into A TNM, we're going to bring in this MSSP that understands how to look at logs and forensic data, and they're gonna determine whether this is nefarious or not. If it's not wonderful, we'll move back into it. If it is, here's the process for moving forward. What are your thoughts on that?

Love your thoughts from the ops side and then Gary from the business side? Yeah, from McKenzie, from The ops Side. Oh God, honestly from the business side a little bit more. So, okay. Pulling from My experience, I used to work, um, for an agency, um, for the state of Idaho and all of the agencies, you know, and this is probably similar to most states though, uh, legally knowing what words to use are so important. So the attorney general, there's the protocol, okay? Mm-Hmm.

You have 48 hours to report, you've had a security incident once you've determined it's a security incident. So I remember my ciso, who's my original mentor, was like, whoa, whoa, whoa, let's take our time before we use any phrasing of incident. Because if we cannot validate that we have evidence and we know what's going on, then we don't have, we don't wanna set that timer of the 48 hours, right? Like we don't wanna start it.

So it's so important to really understand an IT problem in itself is gonna require a lot more analysis to determine is this, even when we use the term compromised on our team from an operational stance, we always say potentially compromised because if we haven't had hands-on keyboard, we're not looking at it, then we don't wanna say the customer told us this is compromise. We don't want to, we just trust but verify everything right? Is kind of the golden standard there. Awesome.

Gary, what are your thoughts on having that process defined and ahead of time and, you know, letting our customers know? So first off, I just think it's awesome. We get to use words like nefarious now. We should only be able to use that when we refer to like cartoon villains and now we get to use it in our everyday life. I'm a huge fan, uh, there.

So I just wanna say that, um, listen, if you are having that conversation for the first time when this comes up with a customer, it's very hard Andrew, um, to have that conversation go well. Like, yeah, hey, there's a potential, there's, there's a potential issue here. Can I talk about our billing arrangement for a few minutes?

Like, like that is not, you know, so what we keep stressing to everyone, just like, um, we had the conversation with Ryan LA last week about with a prospect, these are the conversations that you need to be having with every one of your customers, right? Uh, from a, from a, like A-A-V-C-I-O standpoint. So that, that time when, when it finally, the time comes when it happens, that conversation is not the first time you've had it and it's gonna go much, much smoother. Yeah. Very cool.

Thanks Phyllis, back to you and thanks for, uh, letting us talk about that. Yeah, sure. Um, just one quick question and then, um, I'll pass it off to Wes. So he has time. Um, you know, you talked about perhaps EDR not being on a machine, um, maybe EPP or maybe they're being bypassed.

What's the next dataset, um, that you look at and that maybe perhaps, um, organizations like on here MSP should be looking at, um, on the next step for investigating whether or not an incident occurred or validating, um, that something happened was, uh, really nefarious. I wanted to say that word too. I know. How do we validate nefarious? Um, uh, so as far as like the next data set goes, this is why we really should emphasize more on having people, having your current employees.

Um, regardless, I think even regardless of their role from an IT perspective take, um, a digital forensics class to some extent, and whether it's, you know, a, a smaller scope of one or a full on SAMS course, um, because they got a good Christmas bonus, but like taking that, taking a forensics course to understand what type of data needs to be pulled to at least to an extent validate that something was going on, especially with some type of bypass of AV or EDR, I think a lot of EDR out there.

So not to do a plug for Microsoft Defender for endpoint, but only because that's like the most recent one I'm touching on a regular basis. Have they look for things like AMZ bypass, they have things like tamper protection out there. Mm-hmm. There's a lot of alerts that come up, even if there's an attempt of it. But also we are battling threat actors, these nefarious actors that are on a, on as their day-to-day job, looking for ways to understand decompile the things that protect our endpoints.

So we're working against people who are getting 10 steps ahead of us, extremely sophisticated, who wanna understand how do I obfuscate my tool from AV and EDR. And really it's just from that point we're either looking at memory forensics, so now you all of a sudden need to have someone who's ready to go. And I, I think, um, there's a lot of EDR platforms who also are able to pull, if you can identify the system, then you can at least do some level of memory forensics.

But there's also a ton of open source tools, right? Most of Zimmerman tools out there are open source and on GitHub to get to the memory forensic side to see is there something hidden within the process or below that process that is purposely, um, kind of, uh, not going to be picked up by behavioral analysis or what we commonly eeds commonly engineered to look for or scan. And the next thing though that we do is we look for potential signals of account abuse.

So if we can't identify the system that's doing something weird, we wanna say, okay, how, let's focus then on lateral movement. If they bypass the DR now we're getting past initial detection, we're getting past the compromise of systems and getting into persistence. Well, the only way they also persist is through identity.

And so, you know, identity is the no security perimeter is more, more true now because if we can say, this account is doing weird things and also inventory your accounts, this account is doing weird things and this one is privileged, or service accounts shouldn't be acting like this, right? Tho that's at least the next indicator. That's the next data set we wanna look at is, uh, what those, what those legitimate accounts are doing. Wow, that's a great point. Yeah, Wes, that's so good.

Yeah, that, that is awesome. Um, and I think that that is something I don't think we're talking about enough in the MSP space of like really good visibility into identity management and the activity that's happening. So it's really, really good stuff. Um, Mackenzie, I'm really glad to have you back. Thanks for, thanks for, um, being willing to join. This has been really good for us. I'm just curious if I can put you on the spot a little bit.

Can you tell us a story from dart, like that would be impactful to our clients? Like those that aren't necessarily, um, IT people, they're, they're leveraging the MSPs. Any interesting stories from Dart from, I don't care even two or three years ago, if you have to, that are just fun or interesting for, for a client to hear? Oh, I know there's a whole host of them most in my, my observations that I could definitely talk about without, but not wanting to get fired.

So I'm trying to think of like the most interesting things. No, it's fine. Here we have immunity. So Yeah, I was joking with Andrew. I think I was telling Wes those two. No, it was Ryan. I was like, all right, well I, if I, if I get fired tomorrow one, one of you guys have to hire me or something, you're not pro me. Well lemme just say, lemme jump in here then Mackenzie. Yeah, well I don't, I don't want you to feel uncomfortable.

Is there another is in, So I mean, but there are some things that are public that are really, you know, we've had have a lot of really great stories and, um, I think it's also, you know, the, the, the silver lining of this one and, and many others. Um, 'cause before I was with uh, Microsoft, I was on the IR team with Optiv and, and it was right when the WannaCry event happened.

Um, but the biggest thing, you know, kind of seeing that again as helicopters, we can hear it in the air, um, type of event was certainly solar agate and it was the SolarWinds breach, and I can use the B word in this case knowing this. Um, that was probably the most interesting thing. Our team.

And, and to put it the silver lining here is to understand when things like that happen, um, even though we're like, we have a retainer with you or we have, we have a support agreement in place, the expectations immediately almost get thrown out the window of like, look, you and 500 other entities are now calling our team and wanting assistance. So during these events we were partnering up with other IR teams like Mandiant, um, to be able to perform mass investigation.

It sounds weird, but you know, when you have an event that impacts, it's the same attack path, all the same TTPs and you know what you're looking for, but you are having to look for it across the board of hundreds of customers like that. That's to me the most interesting thing. No one wants to go out and watch it, but there is a really good senate intelligence debriefing episode. It feels like an episode sometimes when you watch those, it's like three hours long.

No one wants to watch it, but it has, you know, the, the CEO, it has Kevin mania, it has, um, Brad, the president of Microsoft, it has a couple other people I'm trying to remember. And they're sitting there just explaining almost how, like how wifi works to, um, to the, to these, uh, intelligent during this intelligence debriefing.

But it's going over, um, the SolarWinds and it's actually understanding, like everyone's trying to think it's some sophisticated thing, but it's funny to watch for three hours people talking about the basics not being done. And that's how the threat actors were able to be successful.

But then in the larger scope of things, the amount of people that are customers, organizations that were impacted by that, to me that was the most interesting experience of we are doing, you know, we're killing 40 birds with one stone at a time because we're trying to keep up. And I'm, we're all hearing helicopters on my team now from those days.

'cause we're just waiting for the next one to happen that Some of those Prepared, listen, if you come to uh, write a boom, uh, and you register, um, you'll get better stories because we'll make sure we get Mackenzie to the bar and give her some true serum. Well, and I won't be on, you know, live streaming here recorded in any capacity. Hopefully not, but yes, probably.

Yeah, we'll do one of those like silhouetted, you know, things we change your voice, you know, with the modulation and all that Can tell Some great stories, but no, that, that actually was what I was hoping you would say some of those things. I think that's great. It just makes me think of another question around, it's, it's so encouraging here that like Microsoft, Mandy and others or like when the time arises and the event is big enough, everyone gets together for the common good.

You know, there's a lot of swirling discussions in the insurance circles around like cyber acts of war and what that would look like. Mm-Hmm.

Do you feel like the past few years of even like the half am attacks and, and, and what happened with Kase, like these big massive, a colonial pipeline, do you think that's positioned all of us to be, if there was a true cyber act war that was an extension of the Ukrainian conflict, you know, what do you think about Microsoft Dart and so many of the other IR firms? Like do we think, do you think we're better positioned to be able to weather a storm like that now? What are your thoughts?

I'd say yes. You know, I think we're definitely better positioned because we've um, you know, have the experience now of going through a few of those. And we also, over the years, because those events happened, that is kind of like, you know, the, the lemons and the lemonade of these massive things is we know what it takes to respond. Like we know how many people we need to hire. We know what skill sets those people need to have.

We know, um, how fast or what are the potential overall business impact to the victims and or, you know, political impact to the world. Um, I think we've, we are way better positioned and from a threat intelligence capacity to all that data that's been collected and tracked, that information is more useful for us. Now that being said, we're still obviously, you know, we're combating very sophisticated threat actor groups.

And the more concerning is thing is is that while I definitely think we're more prepared as responders, um, the basics still aren't being done. So foundational stuff isn't being done. Um, the other side, the criminal organization side, the cyber crime, the increase in what we're seeing out there of just, you know, cyber crime as a service, ransomware as a service, initial access brokerage, it's a completely different Walmart of options essentially that they are creating and hosting.

And it is a full ecosystem. And so it's tiff, while we're more prepared, it's, it's concerning because we know that those things exist and we're almost just cat and mouse, right? We're just trying to keep up. Yeah. It's weaponizing and consumerizing the entire attack, uh, lifecycle and kill chain for sure. It, for sure it is. Um, okay. Let's talk about, one of the industries that we see, uh, have really come under fire of light is education, right?

K 12 universities, all of this, we've a couple threat actors that we've been seeing that have really begun to target them. Uh, insurance even reflects the same thing, calling them a very high risk industry. It's hard to write for because typically what we've seen is a lot of risk and a lot of payouts. What are your thoughts around that? Um, any causes for that? Any, any thoughts around that? McKenzie? Yeah. Um, well I do have like one kind of experience but via a party, uh, law enforcement.

So I guess I'm allowed to say it 'cause I'm not gonna get subpoenaed. But, um, they were collectively, it was an interesting case and it was where this agency was, um, arresting after many years of investigating, um, um, an individual, an individual, one guy, um, probably working within a larger criminal kind of, um, syndicate going on, but, but just a normal guy, um, who actually was relatively close, uh, to my state.

But he, his, his one thing he did on a regular basis was by day he worked as an IT admin guy, but by, by night he was ransoming schools, like middle schools, high schools everywhere. And he was monetizing that. And so obviously, um, that was a big part of what his income looked like. Um, so we do, are able to eventually kind of investigate.

I have faith in law enforcement that goes after and tracks these individuals and being able to find them, but it's because most of the schools also from my, my brother's a network engineer and he worked for a company that specifically worked for IT management for all of the education entities around, um, around the state. And none of them, that's one IT guy for all of them essentially on a regular basis.

They don't have the funding and budgeting to do the basics, let alone be able to manage and know what's going on. And they probably all share the same local admin type of account. And that's not never going away because they don't have the, the allocation or the people, the resources to be more secure. So right away you have the most low hanging fruit of a sector who's, at least from a compensation side, you're gonna get pulled.

We see probably more so on our team, we see a lot, um, over the years of like universities and think tanks being targeted, um, that I, you know, they're definitely one of the larger percentage of the educations, um, industry or vertical because universities have a lot of research and they have a lot of valuable data. So it's, you know, one aspect from a maturity wise, you have valuable data. That's why they're gonna be targeted. And then the other one is the criminal side.

That's, you know, it's getting a payout extortion that's gonna be far more successful. 'cause what are they gonna do at that point? Hmm. Yes indeed. Yes, indeed. Um, maybe one last 45 second answer then I'll turn it over to Andrew. Let's talk about a threat informed defense for a little bit. We've been exposing the MSP channel into like what a threat informed defense is, what it means.

Can you talk to us about Microsoft's view or even your view, uh, McKinsey about what a threat informed defense should look like for an MSP who is, you know, dealing with a lot of different industries, they might have a lot of different threat actors that they're having to kind of handle and manage just given the nature of them working with so many other SMBs. Um, any thoughts around that? Any ideas, any words, wisdom?

Um, yeah, I mean we're definitely looking at as far as nation state targeting goes for, for all industries out there, it is probably one of the biggest ones. And I think in our IT bucket fits it service providers, because we're talking about supply chain attacks, really getting, um, the immediate benefit of a trusted relationship by targeting one entity that has trusted relationships with many entities.

Um, having, whether it's through, you know, an actual subscription and and feed going on with a, a company that specializes in threat intelligence or working on building out your own kind of threat intel life cycle, um, allows, I think will allow MSPs to get ahead of the game and to be able to just understand, 'cause like you said, it is, there are many threat actor groups out there and there's many that are gonna target the IT provider or service MSPs, service providers in general and looking more so at, let's just start with a handful of those threat actor groups and understand how they do it.

So what, what techniques they're all doing and then doing a risk assessment right against you and how are you guys monitoring for it or how do you res have processes around responding or do you have automation options for those things? I think, um, I think we're gonna, I, I think that's the biggest thing is we could see an increase in, um, is just starting somewhere small.

I think people think threat intelligence is a huge component and it certainly is, um, as far as the amount of work, but we have our own data. You know, I always like to think that our incident repository tends to be the most valuable of all.

Look back, who is tar, what has continuously happened and do the attribution to the best of your ability and then do some more, um, you know, research open source wise of who are the threat actor groups that target MSPs, um, and what are they doing within their attack path and how do we, how do we assess our risk because there's only so much you can get. Yeah.

You know, Wes, when, uh, I guess it was like a year ago when we, um, uh, did the session right on, uh, threat informed what I came away with. Like, hey, this is the first thing we've done. Most of the things we talk about, it just seems like more risk, more surface, more than MSPs have to deal with. That was the one that we did that said, Hey, if we're smart and understand this, there's a few things we can do to cover a lot of risk.

So to me it was good counterbalance to not having things be so overwhelming. So I think that's the power of it. Mm-Hmm. Yeah, very much agree. I, I would wrap up with this. Um, you know, as Ryan says, you know, know, know yourself, know your enemy, right? Uh, know your battlefield and you know, that's really what I think, Mackenzie, you just said, uh, and Gary kind of put an exclamation point on it. So I wanna thank you so much for your insights and time. It was an awesome hour with you.

I wish we could spend another hour and, uh, maybe have, we can write in. Um, yeah. Great. And apparently, um, the whiskey happy hour too. Oh yes. Yeah. Oh, awesome. Alright. Hashtag Mackenzie, that's Mack Brown, right? That's right, that's right. Um, everybody, I hope to see you guys at IT Nation this week if you're gonna be there. Mackenzie, again, thank you so very much and can't wait to meet you in person and write a boom in a few months.

Um, as always, uh, for our audience, thanks so much and, um, wishing you all a fantastic week. Take care everybody. See everyone. Take care.