Blackpoint Spots New “Initial Access” Tactics from Threat Actors

Episode 1 27. Here we are. And go Bird. Gary Con, congrats to your Eagles. Um, it's gonna be a big weekend coming up here. Um, what's the address? If people wanna drop in? Yeah, they can drop in. I, I, I actually will be in the city at my son's place with a bunch of his, uh, friends, so it'll be, uh, yeah, it'll be, it'll be cool. Fair enough, fair enough. Okay. Just a few quick announcements.

Um, if you scroll all the way up in chat, um, one of the things I highly encourage everybody to attend is, um, uh, John Barrows is gonna be back, and we're doing it on the Black Point Command Channel, um, February 14th at 1:00 PM. Um, he's excellent. Uh, in terms of sales acumen, prospecting, moving a Dale Advancement. He's Good. Yeah. Um, and again, knows our business. I say he's good. And I don't like most sales trainers. You don't. That's right, Gary.

And Not that I don't like them, they just stink. I don't wanna say you We're nice. We're nice people. They're Nice people. I don't wanna say what you think about all marketing people, because that's a, we, I don't think we can take that live. But, um, but, and then, um, we also, I also put in, um, CIS Control 15 with Phyllis, Wes, and Ryan on service provider management. This one's excellent. And, um, interestingly, Wes, what we're gonna be talking about certainly today is credentials.

And, um, we obviously had the LastPass episode where we talk about that. In that podcast. We talk about, um, what happened with Rackspace. And so, um, all right. Um, just, uh, also, we, uh, closed down registration for write a Boom. We are fairly over capacity right now, Gary, so I'm gonna have the fire marshals deal with you when they come. So if you could hand that, There's still like 10 seats for left of, boom. It's next door, just to the left. All right.

So if you're coming, uh, make some noise and chat. We can't wait to see you in a few weeks. It's gonna be awesome. Uh, Ann, hello from Massachusetts. Great to see you, Chris Lair. Always great to see everybody out there. So just kind of setting the stage here. Um, later last week, John and Team Dave, um, at Black Point, uh, posted something up on Twitter, got picked up right away, which was about the, um, Microsoft OneNote, um, file attachment, phishing scams.

And then, uh, that was proceeded also or followed with Evernote and, um, using, um, legitimate domains. And, and so I really wanted to come back 'cause we, we have done a touch of stuff. I think Wes on phishing, but in social engineer, but really haven't gotten into it. And the evolving, you know, talk about the, the EV evolving, um, uh, uh, put the, the, the, uh, technicality, if you will, of how these threat actors are doing what they're doing at the pace at which they're doing.

I mean, um, we we're gonna talk later about lapsis and what they did to, you know, in the past in 2022 to the biggest organizations in the world in terms of credentials. Um, so taking a step back when you think about it, uh, Microsoft recently published that, um, people are now the primary attack. Not that the, that's a huge revelation, but the greatest vulnerability to an organization. And they say that identity driven attacks now represent 61% of breaches.

So that's not, remind me, remember, that's not incidents, that's breaches. So the number on initial access via social engineering and phishing, IAK credentials is statistically that much higher. So that's what I wanted to kind of set the stage with. And, um, in doing so, welcome back, uh, John to the show and then Dave Rusher, um, from the adv adversarial pursuit group at, at Black Point.

John, starting off with you, quick intro, uh, of yourself, what you do over there at Black Point, and then Dave, and then we'll head right over to Phyllis for some questions. Awesome. First off, thanks for having me back on. It's been a while. I'm glad to chat with everyone again. So, uh, yeah, obviously a lot of folks know us. You know, we're, our kind of core offering is MDR managed detection response.

You know, really under the hood, we're a software company and we're building out a whole kind of ecosystem of products. Everything from, you know, kind your endpoint server response. But we're also doing live response to Microsoft 3 6 5 infrastructures, which is quite timely for some of this discussion. 'cause that part has gone up, uh, a ton. So yeah, that's us. We serve the MSB community and do a bit of upmarket large enterprise as well.

What's cool too, before you go, Dave, is John, you guys were highly contributory for the threat brief that we'll release it, right? A boom mm-Hmm. Um, we're not gonna say who the threat actor is 'cause we're that off or, um, for those that are coming. But, um, yeah, it's a fairly, as, you know, um, substantial piece of, of, of threat intelligence and, and, and collaboration between you, Microsoft, tres, uh, connect, YCRU, um, uh, and even the Sophos group was involved a little bit too.

So, but you guys were, and of course Me, Andrew. Well, well, you were coordinating it all. Yes. Yeah. You, you know, your knowledge of sticks and Taxi Gary, and how you were, you know, correlating the threat intelligence is just stick stones, Whatever it takes. All right. And Dave, uh, welcome. Uh, thanks for coming on and joining us a little bit about yourself and, uh, what you do there at Black Point. My pleasure. Always glad to be here. So, uh, my name's Dave Meyer.

I head up our adversary pursuit group over at Black Point. Uh, so we're responsible for the rapid development of prototypes. So when we see something happening in the security landscape, we try and come up with interesting and unique ways that we can either stop it, identify it better, um, but we're also the primary, uh, team responsible for the production of threat intelligence.

So we take information coming from the, the soc, uh, from our threat operations center, from, um, open source intelligence, as well as just general hypotheses and concepts that we think might sound interesting and might be used by bad guys. And we see whether or not we can detect them and, and turn them into something that benefits the wider security community, not just the organization.

I mean, obviously we want to provide new products and, and help, uh, the company grow, but really we also wanna just make sure people keep us safe as possible. Thanks, Dave. Can I, I'm sorry. Do you mind, can I just jump in and ask a couple quick questions before we get started? What If we say no to you, Gary, Let's say No, just to watch him do it anyway. So how, just because it's something we were talking about last week. How many people on your team?

Uh, we we're running at three at the moment. Okay. So Andrew, the point I wanna make here is we were talking about the reason, we were talking about the, the breach reports, and we were talking about the, um, ISACs and why it's so difficult like that this kind of for MSPs, and this kind of puts an exclamation point on it Mm-Hmm. Is that like you have someone with, you know, with Dave level of experience and expertise with a team, right?

To be able to sort through all this to make it like actionable. And that's why people, right. Why MSPs struggle with it. Right? Yeah, no, it's really good point. And John was gonna say maybe articulate too, this is, you know, the a PG versus your overall response team. Yeah, I guess that's the thing. There's actually a lot more folks involved. I mean, we have, you know, I don't know, well over 20, somewhere between 20 and 30 just, uh, threat hunters and responders and our threat operators.

So there's a lot of times these guys are seeing, let's say we catch something and as we're going through like, oh man, look at these two new techniques we saw them use. They are joined at the hip with a PG and a PG is like, we modeled it after kind of skunkwork at Lockheed. So their job is to kind of run an elite intelligence operation.

But at the same time, if we see a new technique, for example, like, you know, some of the ISOs that are also used, kinda like OneNote we're gonna talk about today, how can we make it easier to auto stop it or get, you know, real time telemetry coming from these new techniques I just found from our previous life in the government, um, if you try to boil the ocean, uh, and with these huge product projects, they usually fail.

But if we have an elite team of experts that can prototype, get it out in the field, see if it works, and then throw it over to productization, that's kind of the model of, of the system. And then there's a whole team of writers as well that have to write this, these reports and kind of proofread of QC so that we're pumping out good intel, right? Phyllis from our old life, we have a high standard here. Listen, I think it's so smart actually, that you mentioned that you have writers. Yeah.

Because analysts aren't necessarily the best people at communicating their job. And so I think it's so important to make it consumable. And so that's just so smart, and I'm so glad you said that. Yeah. And, and also, if I might say, uh, making sure people qa, the intel that goes in, I can't tell you how many times we've detected stuff that's complete garbage. You know, even from cisa that's like really bad false positive CDNs and things like that that never should have made it through.

And then you, you initially, you're, you think, oh no, all hell's breaking loose and it's fine. Yeah. So that too. Yeah. Yeah. And no, we, we, we 100% try and have all of our research peer reviewed by the, even the threat operations, um, center themselves. You know, we'll take artifacts that come out of them as, as John said, they're a 20 strong team.

Um, and make sure that anything that we put down matches what they were expecting and what they were seeing, but also give them an opportunity to get involved in, in our research as well, because they might have extra, uh, information and evidence that just makes it a more solid rounded writeup. Hey, so Phyllis, I just wanted to let everybody know, and there are hundreds and hundreds on this. Can you please fill out a poll? It's an, it's all of you. You go in the bottom there, you hit yes. No.

Um, but uh, um, and then thank you for Craig Taylor for those that wanna know more about FI oh two. Um, but hold off on that, clicking off that, 'cause Phyllis isn't gonna talk to Dave and John. Now an awesome Question. Hey, Andrew, to that, to that end, someone told me the other day at, uh, some of the conference I was at, they said, so I always listen, but I take this where I'm like jogging or something, so I can't go and vote. Ah. So that's where some of them come from.

I finally figured it out. A lot of people are like, I'm not in front of the screen. Fair, fair, fair, fair. Okay. Alright, miss Phyllis, thank You. Well, first I wanna say hi to John and Dave because in the green room, I forgot to say hello, so excuse my bad man. Hey Phyllis. Hey. So, hello. So, you know, as Andrew, um, talked about earlier, we see like phishing evolving, right? And so, um, you know, the threat actors are becoming more innovative.

Um, you've got like legit email domains now we've got chat, bt chat, chat, GPT, phishing lawyers, et cetera. And so, um, talk to us about, you know, the a PG, the Advers adversarial pursuit group and, um, seeing how and how they saw and how they detected the OneNote phishing campaign. Yeah, I mean, I think to kind of set the stage on, on phishing, obviously this has been the number one access vector inside networks.

And there's, I think, you know, a lot of this, you know, Andrew, when you kind of put, put this topic together to talk about on the call, you know, there's, there's kind of two lanes, right? There's, there's phishing to try and get some sort of code down or tool down that you can run to get access behind, you know, a firewall.

And then there's phishing to try and get single sign on or gen almost all the time, Microsoft 3, 6, 5 credentials or some sort of hook to get someone to put them in, um, uh, that way. And so it was actually our, um, threat operations center that caught this. Um, we've been seeing it come through a lot with attachments and then, you know, if that attachment gets through, we pick it up because we see some malicious activity going in the endpoint.

But I would say, you know, with, with OneNote specifically, if you take a step back, typical phishing campaigns that result in something, whether it's code running on a computer or trying to get you to a login link to, to trick you, it used to be always macros. And we saw zips ISOs were, are, are still kind of popular and we saw is the virtual hard kind of dis and they, and they might masquerade as other things.

What's interesting with OneNote is obviously most of the world seems to be on Microsoft 3 6 5 at this time. So anything that is delivering content that you might normally share within your company or share between companies that's kind of tied to 3, 6 5 applications becomes very, very interesting.

So we see a lot of SharePoint, you know, kind of tricks or whatever, but really at its core, OneNote is just a more, from what we've seen, a more believable way to get a user to interact with the attachment to either shift them off to some login link, Hey, I need to log into 3 6 5 to get access to OneNote. And then they're gonna, you know, kind of steal, uh, um, creds and then, you know, for MFA bypass, which David can talk about a lot more maybe, you know, session cookies and things like that.

So I, I think that's kind of what we're seeing. And then when we see this stuff, in fact, I just got a brief this morning. We've had eight in the past seven days just in our customer base using OneNote. So it's either it's one group or multiple groups have all jumped on this. You know, it's a lot of times if you catch it early, it's hard to tell what group's involved. Uh, but that's, that's kind of how, how, um, we came to see this. Wow, that's cool.

So, um, you know, what's interesting is, um, you know, Gary talked about what does it take to, to do this kind of threat hunting or whatever. It's really like, you know, your team's experience, um, understanding good good versus bad, right? Which is so hard, especially when it's like a, a little bit like living off the land, right? You're yeah. You're using what looks good. Um, and so that's why it's hard and harder to detect, which makes it even more impressive, you know? Right.

That the, that the team could detect it. So that's awesome. Mm-Hmm. Um, so when we talk about, you know, initial access and supply chain and initial access brokers, um, can you tell us, you know, why is it that threat act threat actors, um, are getting more role specific? Ooh, I think it's really simple. It's just professionalization of the craft.

I mean, if you look at any like, elite hacking operation, whether it's the bad guys or the good guys, you know, whether it's state sponsored or whatever, the roles are really defined. And, uh, and so in, in each role takes its own kind of specialty. You might, you know, someone that's great at writing kind of exploit kits or back doors or command and control. They might be not be the best social engineer type, right?

And so I think, I think what's really gone down is there's just kind of a, a supply chain in the own, in the kind of ransomware world where you have folks whose job is to gain initial access and get a wad of creds and then sell it off.

And then the next job, you know, for the guys that come in after that are really to swim around that network laterally spread, get everywhere they can steal as much day as they can encrypt, and then, and then, uh, then it's on to handling, you know, payment and data and release the data. So I think it's really, it's been very lucrative for these groups. And I think what you're just watching as a professionalization of the craft, um, you know, It's so impressive.

I mean, honestly, like how well organized, um, the criminals got in such a short time. Yeah, For sure. I think the groundwork's been laid with, you know, things like software as a service, right? I mean, you've already got a more legitimate business. Why not just make it a little bit more evil? Yeah, exactly. Exactly.

Hey, Phyllis, when, um, when we put up the, um, supply chain slide, uh, yes at SCH Fest, um, it, I think it really, everyone knows what's happening, but when they saw it in that graphic mm-Hmm. Um, it really made people think like, wow, this is this, this, you know, the bad guys are maturing faster. Right? Right. Than, than than than we are in many cases. Right? Exactly. Exactly.

Um, so Dave, Gary touched on this earlier that, you know, hey, um, you have, um, a great team and, um, how is it that, um, how is it that you, you compiled your team? What skills were necessary? Um, how do you create a good security operations team that can, you know, identify this kind of behavior? That's the tricky thing. Um, I think every member of the team brings something that's unique to them.

I think you have a core set of skills that you, you look for or that you, you ideally want, um, the ability to understand networking information, maybe a comfortable level of understanding windows logs. Or it might be even they've done some reverse engineering and they've, they've played around and tried it. I don't really care for qualifications all that much. And so when people come in with qualifications, that means very little to me.

Um, what I'm after is people that have tried to do things themselves, they may have, you know, tried to use tools like IDA or LED bug to reverse engineer. They've played around with wire shark and they can talk to me about the, the, the way network traffic looks when they've observed it. They might not necessarily know what they're looking at, but that means very little to me. You can teach people that what you can't teach is a drive for, for knowledge and a thirst for it.

So when it comes to building out the, the security operations team, when it comes to building the A PG, you're looking for people with a fundamental passion that have done things in their own time that have really gone and tried to learn because they don't think enough is enough. You know, they never wanna stop knowing why it does a certain thing or why they're trying to, uh, why something's acting in a certain way. Right. Um, really you're looking for, for, for unicorns, they're crazy. Yeah.

They have, um, you know, they're eccentric and, and possibly the best people you can, you can work with And will also give kind of practical reverse engineering exams as part of the interview process. So they actually have to come back, like reverse engineer and then explain it. Uh, so one of the things we found in this industry, I, we, we could care less about the certs, right. Except for probably OSCP. We really like that one. 'cause it, it, we want hands on skills.

This is so much more akin to a trade or like a doctor's, you know, evolutions to go to residency and then they specialize in a particular area of surgery. And so we're looking for folks with hands-on skills that, that David said that like to exercise 'em outside of work hours, because that's when you c can really, really learn. And the hands-on is, is really everything that, and the curiosity and drive. Right. That's interesting. You know, Good point.

Um, we saw a lot of, we've seen a lot of really good, um, software engineers coming through where they get given like code blocks and they're asked to answer those questions and, and provide either pseudo code or real code.

So we decided to flip it on its head and, and go, well, what if we wrote a piece of malware that doesn't do anything truly malicious, but has indicators and flags inside it, and you have to debug it properly and we give, you know, anywhere between two to three weeks and ex and, and a template to fill in Mm-Hmm. And really one that shows whether people know how to use the tools. Right. But for me, it also shows where there might be some, some small weaknesses Mm-Hmm.

And things that we can work on when we hire them, because they might have everything else and Mm-Hmm. You just need a little bit of training. So Right. We're all about, um, growing in a role. Right. You know, we don't want someone that knows everything. There's always more to learn. And if they're willing to accept that, then great. I love that.

I mean, we used to try to look for that when I was recruiting at the NSA, although there was like this grade requirements, which I could not handle, but whatever. I mean, no, I, I love all these qualifications. I love it. Like, you know, you can't be perfect coming in and you can train them. I'm just outta curiosity, um, do you kind of have internal training or is it OJT on the job training? Or what is it that you all do to kind of grow those skills? Hmm. Yeah.

So within the threat operations, uh, center there is, on the job training, we have a number of, um, and, and will, uh, Santiago, the, the, the VP of that has spent a lot of time making sure that there's a proper pathway for, for education and learning. We have some standards that we, we hold the team to. So before they can really get stuck in, they need to at least show a, a base level of competency. And there's training time allotted to, to reach that.

Um, within the a PG, it's a bit more tricky. Um, really when you are, when you're kind of at this level, there's, um, already a lot that's been done beforehand. And that's why we do the reverse engineering challenge to, to really thread it. But I'm very much of the mind if, if people need to learn something and want to learn something, they only need to ask if they hit a roadblock, we'll remove that roadblock and give them an opportunity to learn and, and, and do. Yeah.

You know, we've got some things at the moment that, uh, you know, a member of the teams come to me and said, this isn't working. Right. So we took time, we went back over it and we've now worked out a plan for growth to make sure it's not a blocker in the future. Yeah. Phyllis, one of the things we took a lot of, obviously lessons learned from our previous life in the government and Will was also in the military. So we run our, our threat ops centers, like we have senior watch officers.

It's a very military run sort of org Mm-Hmm. We even have a whole JQR process. And that's what what David was referring to. Like, you have to, even if you're tier three, you have to go through a whole kind of qualification process before you know, we kind of bless you to be OPS certified, if that makes sense. Mm-Hmm. Um, and then, and then there's, on the job training, a lot of it's hands-on, and from time to time we'll run scenarios that are kinda unannounced fire drills as well. Mm-Hmm.

That's awesome. Like this is also internal analyst qa. Yeah, yeah, for sure. Continuous qa. Yep. Um, that's great. And, and you know, it's nice to follow it up with, Hey, we're gonna give you the opportunity to train. Mm-Hmm. Um, so that's, that's, that's important as well. 'cause so often it's hard to get that time in, right? 'cause you're so busy working. It is. It is.

But if you look at, you know, this is one of the things that, that makes certain parts of the government function really well, and specifically in the military side is like a huge emphasis on training. We don't put any emphasis on certain people.

And, you know, kind of, this is something that I know happened a bit in the government where you bring someone in, you put 'em through courses, they get all these certifications, then poof, they leave to go just trade that for, you know, for a different, you know, job. We focus a lot on practical training and kind of, you know, side saddle training with our best folks. I, I personally, this, this game is so much in the weeds and the details.

Like, you have to have that kind of deep level of side saddle if you're gonna be any good. That's awesome. That's great. Um, so Dave, um, can you talk to us a little bit about IOCs or indicators of compromise just in general, just to kind of level set, um, the audience and then how do you figure out, um, perhaps maybe how you create the IOCs? Which ones should you be emphasizing and the ones that your team and clients really, you know, should be looking out for? Okay. Yeah.

Um, again, this is, this is an interesting one for me because I'm, I'm split on IOCs versus what I, I would call, uh, indicators of behavior. So, um, and IOC, it, its, its core is nothing more than an indicator that something suspicious has happened on an endpoint on a network. Um, I think calling it compromise gives, uh, a bit of a indi uh, it, it's not correct. 'cause it can't always be a compromise. It's sometimes just a, it is just an indicator. Mm-Hmm.

And that's where indicator of behavior comes in, because we're in a, a place now where it's no longer enough to say, if you see this contacting this IP address, it is bad. If you see this happening on an endpoint, it is bad. It has to be a group of, of, of, uh, indicators together. You see this, then this, then this, that is bad. And really, um, oh, so it's so tricky to, to sum this up, um, It's a little controversial topic actually. It is, yeah. You ask a loaded question.

I think, you know, the, uh, right. It's, it's kind of like the virus signature versus, you know, the behavioral, there you go. Yes. Kind of. Absolutely. You know, as soon as, as soon as the signature goes out, it's outdated, right? Yeah. So It's, so, you know, there's this whole thing with like, IOCs, let's make YA rules. And we always felt like that was a few steps behind. We almost never do a response where the C two was shared. Mm-Hmm. It was brand new. It was created for that operation.

So I even watched, you know, love csa, but they put out the report on rms and they put a bunch of that stuff in there. I'm like, this is so old. Um, already, if that makes sense. Yeah. That it's, I think, you know, when you really break it down, and this is maybe slightly controversial, uh, but this is kind of how I see it, right? You know, in a professionalized hacking operation, you tend to have those that write the tools, those that use the tools, right?

So maybe you could call 'em, you know, operators or red teamers or whatever. And then, you know, when you take it to another level and it's really targeted text, you tend to have analysts who are target experts, but also kind of know how to use the tools as well and, and move around.

If you think about asking a software engineer how to catch a hacker, I guarantee like it's almost always gonna be an EDR AV style machine because their best tuned to catch what they would make for persistence or for C two. And, but the other part is, and this is why we look for folks with a lot of deep windows domain system administration and networking skills in our threat ops center. If you talk about using the tools, the amount you can accomplish living off the land Mm-Hmm.

Is shocking, uh, that can use totally legit tools. And so what you really need, you know, to Dave's point is that that elegant blend of malware detection and behavior analysis happening kind of at the same time intertwined. And when you start doing that, you know, for example, we, we talk about a kind of standard of intelligence.

You know, when we put it out, um, we had the, that exchange zero day, the Chinese one thrown, uh, the more recent one at one of our customers two months prior to it being public. We knew it was a new zero day. Unfortunately, the forensics firm that kind of came in messed up the forensic image and deleted a bunch of the artifacts to allow us to change together. It's the only reason we didn't release it.

And so we didn't, um, obviously, you know, we talked with the records and or records of government authorities, but I think big picture, you really need a blend of those two worlds. That's why we actually find a lot of times the best threat hunters have a deep IT background. Mm-Hmm. Um, and you're seeing more and more of the threat actors leverage legitimate IT tools for, uh, for bad now because, you know, what do you do at the end of the day?

Is it, in it we're pushing files on machines, taking off accessing 'em remotely. What does the hacker wanna do? That's the same thing. I mean, So I mean, I agree. I would say some of the best analysts that we had, you know, looking at malware and detecting malware at the agency back in the day, they used to be former SOIs admins and, you know, units admins or whatever. 'cause that was, that was the OS back then people back in the olden days. Yeah.

So, um, yeah, no, and, and so I'll just kind of sum it up. I would say that yes, I mean, I think it's great to have these kind of signature type things. Just, just, yeah. I mean it's, it's like, yeah, you wanna prevent the apple biters and, and all that kind of stuff. But, um, I think what, in my opinion what I'm hearing today is kinda like that advertisement of like, why do you need a black point? Why do you need, um, like kind of a professional service?

Because not everyone, um, can do the type of analysis of like a black point. And it's like, you know, how, how, how can you consume that data? How is it that you can take action on, um, you know, data from, um, you know, someone like a, a black point? Because it's, as you can tell today, it's not easy, right? Mm-Hmm. Well, I mean, to give you a real world example for this, okay.

Um, you know, if you'd asked this maybe five years ago, you could say, well, if it's coming in from a Russian ip, it's it's gonna be bad. Mm-Hmm. Well, what about the scenarios now, for example, where you're seeing a, uh, an Azure VM based in the US logging in using legitimate credentials. There is nothing straight away about that. That would, could be an indicator of compromise, because there's nothing there that's suspicious.

But you might take a look at, say the time they logged in, what they did after they logged in, what they, you know, any of the communications from that IP address before they logged in to build up a picture of, now we've got a behavior that is malicious. Now we've got something that is suspicious and needs deeper investigation or responsive actions. So it, it, it is really tricky. I think IOCs still have a place in this world Mm-Hmm.

You know, when you gonna have the SHA 2 56 when you can have those Yara rules, but you're going to struggle when you deal with things like encrypted binaries because it's not until it's in motion, till it's actually running that you can detect that it, it's truly malware. Because really with a Yara rule at that point, you're just detecting a packer. Mm-Hmm. Right. So, um, behaviors for the win, uh, I've seen a couple of comments that agree with that, and I, uh, right.

It's nice, it's nice to see some backup from, from the audience as well. I'm glad they agree as well. Thanks. Yeah, because David, just to get a, a bit technical, I mean, I, you're right. That's where IOCs are helpful, is I can regression analysis and I'd say 0.01% of MSP should ever be in the business of regression analysis. Right. Like's no way they're gonna have the skillset or even the need to do that. Big orgs for sure.

And that's where tools like what's the Google thing, the backstory, that's where things like backstory are really awesome, but it'll, that'll never come down market because to your point, that's not where the focus should be. No. Andrew, I was just telling you this weekend that I hate, uh, encrypted binaries, But I, the, the, the real place where I see right now is, is when you have been compromised if something bad has happened, yes.

And you are in that right of boom name drop, let's go for it. Um, when you're in that scenario, IOCs are, are really valuable because that's going to give you an indication of what's happened and possibly attribution if you're lucky. So, Hey, so Phyllis, I wanna switch over to we, if we can, 'cause I make sure we get through the Wes and Gary. So, but awesome stuff so far. Thank you. Uh, Wes, you're up My friend. Yeah. So this is a great conversation.

Let's get back to the, um, a little bit of the topic of what Andrew had written on this, like OneNote stuff. This is, I covered the same thing last week when this stuff was starting to emerge. And it, it's like you said, John, I think it's natural to like, think through bad guys are just naturally gonna think through, um, what it's like water, right? Where's the path of least resistance for me?

And I, I was just as my opening question before I say it, I'll just say this, if I'm a bad guy, here's what I would do. I dropped, um, earlier today, I dropped a PowerPoint in Google Docs to a whole bunch of my folks in Slack. And I'm thinking, you know what? I guarantee everyone just clicked and opened it, right? Because it came from Wes on Slack. Just click open. A hundred percent will fall for this, right?

If I'm a bad guy and I had access to someone's inbox, I'd use the Slack magic link, I would get access to their Slack and I would deliver all day through that. And this OneNote thing is very similar, right? Because people see that, they're like, oh, this seems legitimate. Let me open it. So I, I like what you said, John, that's exactly how I see it too. Of like, we're going to continue to see leveraging trust relationships and leveraging legitimate cloud-based platforms and applications.

'cause it just, it bypasses our our level of trust. Right? So can you just talk about that a little bit more, John, high level of like how you're seeing those things evolve and why that's happening more? Yeah. You know, before actually 3 6 5 and Azure in the cloud and, and really domain, uh, authentication in the cloud, I used to have this diagram and it kind of showed the intersection of your physical security systems, you know, maybe some of your network device.

But at the end of the day, it all kind of centered around active directory. And the answer is why? Because that controls all the trust within a Windows domain. And what we've watched now is this whole push to kind of a cloud first approach. And we have a lot of hybrid environments out there, and a lot of folks manage all their, their credentials and leverage, you know, 3, 6, 5 for, for single signin.

And there's a lot of other, you know, uh, Okta Auth Zero do all those guys, uh, that are handling other products as well. And so it's really simple at the core. Once you have credentials and you can get in, you can pretend to be someone else, the amount of influence you can have over others greatly increases. And when these, uh, credentials, uh, you know, link together a whole bunch of applications, it opens the door even more.

In fact, we had in that, that, um, that event I was talking about where the Chinese, uh, attacked one of our upmarket customers, what was really interesting there is they actually went after the MS P too. Um, and you know what, they were going after network documentation tools and firewall management tools. Why? Because that gives 'em a whole picture of the playing field, right? And so as, as we continue, you know, cloud is amazing. We all should go there. We need, we need the advantages of it.

But as off keeps getting consolidated, consolidated, consolidated, if anything goes wrong, it's built on a house of cards. Um, and this is anytime you're gonna kind of standardize on something, you know, directly in the middle, it's, it's gonna be an issue. You look at software defined networking, for example, you know, there's kind of one brain of the network, it's not very decentralized, and that does open up that up for a different set of attacks. So I think that's it really simply.

And if you look at what most people do when they break into Microsoft 3, 6, 5, it's all a hook to trick accounts payable to wire money. I mean, it's simple. It allows a, a lot greater influence in, in social engineering that can lead you a lot more applications. Agreed. Well said. So, um, David, let's talk a little bit about weaponization for a minute. You know, there's chat GT's been all the rage for the past month or two, and we're all starting to get burnout on it.

Uh, but it's still, when people get in front of it, they're still like, whoa, this is really cool. Like, people are blown away by just how natural it feels. And, um, you know, I've done all kind, I did a few live streams and like getting it to at least write me some, some rudimentary code to encrypt machines. Sure. I mean, we can do these kinds of things, right? But what I think chat GPT will, where it's scary is just in how it levels the playing field of communication, right?

How it levels that ability for us to weaponize language, which typically has been one of our defenses, you know, for all these years we've been, if you see something that's written really poorly, you just know it's a phishing attack. Well, those days are closing for sure. Can you give us some commentary from Black Point's side of the house, how you guys sort of see this and this evolution? How bad guys will use this and, um, in, in in their own operations or, or things even like chat g pd.

You don't have to stick to that if you don't want to. Yeah, no. Um, we'll, we'll stick with chat. GPTI think it's the easiest one to, to explain, but it's not by any means unique to just this ai. Um, the way it levels the playing field is that it allows people with even limited language skills to create phishing emails and other capabilities that will entice users to click. The real danger is that it takes some of the, the steps away in terms of craft an email.

So for example, if I wanted to target, um, I don't know, pharmaceutical company, well, I can get specific, I can ask the AI to say, craft me an email that looks like it's come from a pharmaceutical supplier or a, you know, supplier into pharmaceutical. And it will pull information from many different places to create an email that looks legitimate. But, you know, we as a society have have been trained to look for suspicious emails to try and look out for them.

But it becomes really easy with AI to craft something that looks like it's come from another person. It doesn't have the spelling mistakes, it doesn't have the grammar. It has all the, the, the things that you expect to see in your email. And when you start running it through an SMTP relay, or heaven forbid you have, you are launching it from inside a compromised 3, 6, 5 account. So it is legitimate. That's it. You're, you're in trouble. So it's not going to replace the bad guys.

You won't ask, it's not going to be able to generate a full ransomware payload or, or a full malware piece, but it does make it quicker for the threat actors to adapt their their playbooks. Yeah, it, it does. And those are the things I think we should just be aware of, um, because it, it is going to continue to make the social side of social engineering just both more effective in the volume of attacks, I think go up. For sure. And, and John, this brings a question back for you.

Looking at Lapsis, right? That was a threat actor that really had significant, um, capability and leveraging credential based attacks through household names, right? Like Okta and Rockstar Games, it makes, um, grand theft auto and like all these, like how, like these names that every company knows, they just fell one after another. What's an MSP to do in this world, right? Should they be rethinking their controls? What's your advice for them?

Yeah, I mean there's, there's, there's, I have of kind of several pieces of advice. You know, first off, MFA really helped blunt some of this for a while, but I'm telling you, we are seeing a lot, a lot more MFA bypass attacks right now. Uh, and we've seen 'em both mostly criminal group, but we've been seeing on the nation state side as well, just very targeted. Um, it's tricky, right?

I, but our, us as a security industry is going to have to start having live eyes on exactly what we talked about early on. I mean, I think we're, we're part of the way there with our, we have a a, um, a product called cloud response that does, allows us to do live interrogation. So every time someone logs in, we run through a whole analytic system to detect is this proxy not the user agent string makes sense that even that, you know, that's really good and it's been hyper, hyper successful.

But there are some cases, as David alluded to, where we're seeing folks actually buy domains with a high reputation score, right? Uh, they do their user agent string, right? And then they just log in to, from an Azure box to Azure does not look sketchy at all on the surface.

And so even that front of the funnel kinda log in analytic, that's been so hyper successful for us, we're having to add more after seeing some of this so that we can understand what are they doing right after, are they permanently delaying files? Are we seeing new kind of off tokens created? Um, and then that's the kind of indicator of behaviors that kind of change. So, you know, there's a couple things. Uh, what do MSPs do about this is hard. It's gonna be really hard to handle all your own.

I would, you know, this might go against the, the, um, advice on kind of consolidation and, and ease of use, but be very mindful of how much of your critical applications you completely unify with a, with a single sign-on provider. I know that's kind of counter, but I'll give you a real world example. You look like Sony for example.

You know, anytime you have like your badge swipe or access control or even like me, this one was talked about I think publicly anytime that is directly tied to actor directory. If you have a ransom event or a destruction event, it smokes your physical, you can't swipe in like the turnstiles don't work. There's a lot of cascading kind of downstream consequences. So I think first off, you have to get some sort of live monitoring, uh, and an ability to at least respond decently quickly.

Um, you know, if you're going to, you know, be able be in any position to detect and stop this stuff. Second, uh, hygiene has to be huge focus. If, you know, you think of those kind of five core pillars of nist, pillar two is protect. I don't like that I changed that to harden. Uh, 'cause I think it makes more sense and this is where, you know, because shameless plug for Phyllis, but we believe it wholeheartedly is CIS has a great like domain and cloud hardening guide.

You know, so that really helps. You know, it's one of the things we do. We interrogate 3 6 5 and map it to CIS, right? To help make that easier. So you have to do the hardening in addition to kind of the detection. Then the third kind of pillar to this, it's you have to show, uh, employees not just phishing training. Ah, tricked you. You shouldn't have clicked the link, but you need to show 'em what an MFA bypass attack looks like on the receiving end.

So for example, if you're getting a lot of Microsoft off notifications popping on your phone and you hadn't just logged in, like you wanna weaponize your folks so that they know that is a warning and to, before you click anything or send it to you know, your head of it or if you have a security person, uh, to them. So I I, there's kind of three parts to that in, in my opinion. Yeah. John, can I, Wes can I just jump in a second real quick?

So John, what you just last said though, on let's air quote security awareness training, most of it's like, you know, set and forget. I'm using company A, company B, company C, it goes off at a certain time, certain cadence. Wes, you might even wanna chime in 'cause I know you're involved with this. But I guess two part question, one to John, to you, is it that as MSPs we need to, as thought leaders, you know, provide insights on this is what's going on right now.

Let me get in front of you and your team and Wes to the second point, again from the cybersecurity cyber insurance side, will they catch up and go, well, yeah, it's a checkbox, but are you doing more, are you educating in more of a real time? So you're, to you first John, you know, your thoughts on does it have to be more than set and forget, which is I think what most people do? Yeah. Honestly, I think the set and forget stuff is, it's a checkbox, right? Yeah. It's, yeah.

You know, yeah, you need to, here's the deal. In my opinion, you can trick any human once. So if you're targeted enough, you can really be hyper successful. This is why like chappy, GPT is a little bit, you know, I think it's a little bit overhyped on the hacking side, but you know, I'm gonna, uh, actually drop some completely fresh threat intel, uh, at right of boom Andrew, that isn't out there publicly, uh, when, when we're up on stage in our section.

But one of the things we saw was we unearthed a whole command and control network for this particular threat attribute. It's called a teaser, what you just did there. Oh, nice. Yeah. Um, and the, uh, the, the interesting part is one of the things that caught our attention right away is they spelled a word wrong.

And, you know, as humans are used to dialects, how, I mean even the United States, you look how North Easterners talk verse, you know, uh, Southern California, like, it's totally different. We're speaking different languages, but chat GPT and some of these AI tools allow you to break that down. Now, when you think of that with foreign, you know, with folks that aren't steeped in our culture, it, it really kind of helps. So I think, uh, um, I'm not quite sure I was going with that totally.

But it, it's kind of what we're seeing, you know, big picture. And, and I think it's just really important if you're running an MSP or you're kind of, you know, the, the lead tech person to get steeped in every form of threat until you can get, and you don't have to turn it into FUD to sell more stuff necessarily to your end customers.

But I do think the more you can, first there's tons of good security companies out there pumping a lot of threat intel out, become a student of the game, consume it, digest it, and repurpose it back to your customer base. It'll show more, more domain knowledge. I think it'll make you look like a much more ma and act like a more mature MSP getting ahead of the problem.

'cause here's the deal, what's our job at the end of the day, whether it's an MSP, you wanna focus on bd, bring new customers, having great, you know, kind of, uh, customer retention rates. And at the end customer, a lot of these folks that we serve in the MSP community, their entire livelihood is built into this particular construction practice or this small manufacturer. They need to focus on what they do well.

And so that, which means we have to up our game a whole lot because really at the end of the day, MSPs are critical infrastructure. They controlled most of the IT infrastructure in the United States. If you ask me Wes, I dunno if that, but, but back to you. But it, it's just interesting, uh, what yeah, you know, you, you talk a lot about how insurance is behind, but this is just another area which is a checkbox right now. Yeah. And I can answer this in 30 seconds. Um, that's exactly right.

They, the carriers are way, way, they're just miles behind where they need to be. They know it, they're trying to catch up, you know, they're currently doing things like external like attack surface scans that are going to be short-lived, and right now they're very poor results as a whole in terms of, you know, reducing underwriting, uh, the pre underwriting risk.

But we're going to get ready for this continuous underwriting get ready for you to have to be able to show some amount of data to the carriers to see configuration practices, you know, in place, right? And so right now they're focused on, you know, even though MFA bypass is become, as John said, is becoming a big deal, they're still like, look, the majority still hasn't even implemented MFA. That's what we want to cut off now.

Uh, we, we should not be writing, uh, claims for those kinds of events, right? So get ready for this, it's gonna happen and it'll continue to be an evolving process. Cool. Um, to that end, Phyllis, a question for you if I may. Um, so John was talking about, uh, how important it's for MSPs to understand M like MFA bypass. And John mentioned the, um, guidance that CIS has.

Are there plans for CIS controls to have some updates towards not just having MFA in place and rolled out for everybody to, but to even look at like new controls or new baselining standards now that we're seeing this evolution of ad guys leveraging, um, attacks, whether it's like cloud or API based stuff, or single sign on, those kinds of things that do bypass MFA? Is there a, where, where's CIS on that? Any thoughts?

Yeah, we definitely are looking to see, you know, when is the tipping point, right? When is it that we need to update our guidance? So, you know, it's funny when you talk about that because, you know, um, NIST based on, I believe pro and, and Microsoft, depending on who you talk to in Microsoft and whose blog you read, they basically are like, you know, just roll back to eight character passwords.

Um, as long as those passwords are truly random versus, you know, Phyllis number one, you know, you know, it, it truly like machine generated random a character password is fine and then, you know, MFA, right? And so, um, we kind of look at where is the guidance currently and where's, you know, threat currently standing and then try to update that way.

I mean, I would say right now, um, unfortunately, I don't know John and Dave, maybe the people who enlist Black Point are, you know, more forward or fast move, you know, forward thinking using MFA. What we find is so many organizations aren't even using MFA still, right? Yeah. And so it's like, okay, can we move away from MFA when so many organizations really aren't even using MFA?

And so it's like, you know, let's, you know, we are gradually trying to bring everybody, you know, up to speed and it's, it's hard. It's hard. It's like, it is striking that balance though, Wes It is tricky. Like I would say definitely not just you. Yeah.

I mean, you guys have to talk to the masses, so you gotta keep pushing MFAI think, you know, Wes one small, you know, nuance I would add that I think has been shown to be decently effective is inter introducing kinda geolocation into that, that, that that kind of multifactor off. I think if there was to be one nuance guidance Phyllis, that would be like kind of, you know, you need MFA and then let's like encourage you to, to kind of add this part kind of end customer Mm-Hmm.

Because that's the one that can, in my opinion, from what I've seen, help, help stop a lot of the tax we're respond to right now. It's interesting. Thanks. And, and there's a bit of this, um, confirmation bias, right? In the sense of like, if I care about RD enough, I'm gonna buy Black Point, and if I care about security enough, I'm also gonna have MFA.

But the vast majority of this, Gary, this goes to you and your point, right, of like the yes, that's true, but think about the thousands upon thousands upon thousands of s and Ps that don't care about security, therefore they don't have Black point, therefore they don't have MFA and, and so I see your point there, Phyllis, and, and that's where I think MSPs need to remember this in their conversations because it's still, we still have a lot of problems all, all here among the, the majority speaking Gary, Gary's point, everybody cares about, nobody cares about security, right, Gary?

Yeah. Until it, Yeah, it's not a problem until it is. Gary, I'm gonna turn over to you for time sake. Yeah, yeah. I have a few questions. I'm not, I I'm going to start with the ones I want to make, make sure I, uh, start with you. First off. You guys are awesome. We have a lot of guests on here. Um, and your ability to take, uh, all the complex and deep knowledge people can figure out you have and relate it to what we're doing is as good as any guest that we have. So kudos to both of you.

Thank you very much. Um, so let me see where I wanna start here. Um, let's start with this. We asked a question right at, we had a show of hands at Schnoz Fest. We have 500 people in the room. Andrew, what's the question that we asked? I'm trying to remember which one we were talking about Gary, It was about EDR Oh oh. Oh, you mean do do you have it right? Do all Your clients. Yeah.

And then we said how many of, and a lot of hands went up and we said, how many of you have it deployed at all of your customers? Right. Right, Right. And it was shocking. So John, people come and they, you know, they find a great vendor to do business with, but they're almost out there reselling it, and now they got 50 clients and they have it deployed on, you know, a a, a subset of them. Yeah. And I was surprised by, by, by that.

And so, and We, and we were stipulating MDR with it, right, Gary? Not just, yeah, yeah, yeah. So just to perspective John. Yes. Okay. Yeah. So this is what we run into is that the business model is holding back security. And so is it the vendor's responsibility to explain to the cu you know, to the MSP how to get their customers or, or to explain to the ms. Like, we talk about it every week here about building the most important things into your model. But do you see the same issue? Massive.

In fact, it's a huge focus of our company this year is putting out enough content to kind of give our view on how to go to market. Well, I mean, I think that's the one thing if you, if you look at the evolution, you know, in kind of the MSP world, it was break fix and getting into managed services and it was starting to get wrapped in a nice bow and it was fairly standardized.

And then security and, you know, upper level security, whether it's MDR or extra, whatever you want to call it, uh, was always an upsell game. It's an upsell game. Upsell game. And then you have certain customers willing to pay for it. Others' not. But at the end of the day, take a big step back and look where the puck is going. Right? Insurance is increasingly demanding. A you Said puck, right? Yeah, yeah. Okay. Did it come across wrong? Sorry. Yeah, yeah. I said puck.

Um, so we gotta get to a point as an industry where security's kind of baked into the offering, it's pushed upfront and it has to be benchmarked on kind of industry standards. I mean, 'cause one of the issues you can get into is where you're kind of so far ahead of the game, people can't conceptualize it. So I like those five simple pillars in this asset identification. Harden, I need to detect, I need to respond real time. That's all left to boom.

Write a boom is, if you know, s**t goes totally sideways, how do I limit the blast radius? And the point is, you need to package this up and sell that as like a main reason to come with your MSP versus your competition. There's several reasons. One, the sales process easier. Two, when you upsell, you need this great technology that's gonna save you from hackers. Let's be clear, most networks are not hacked in your typical QBR cycle.

So when you have to go show value, right after you've, you pitched that and there hasn't been any saves 'cause there's been no activity, it kind of, you know, sucks the air outta your balloon. So you really need to kind of sell on, on kind of industry best practices value. Here's the other thing. MSPs are being asked to fill out insurance applications like crazy. And when you have a different stack at all, your different customers just becomes one, a massive headache.

And two, the insurer's not gonna like you as much, you know, kind of going down the road. They want to see that you have a modern standard stack, it needs to be integrated, and your post-sale customer success needs to bring in threat intel education. This is what MFA bypass. This is why we have this ecosystem in place, because at the end of the day, it's like nine one one. You don't call it all the time, but you sure as hell want it when you need it. Um, yeah.

And so I think that's maybe where you're going with that question, but that's kind of my view of far too many folks are doing this kind of add-on upsell thing and, And they're looking at it like, what I try to teach our members is pulling their head back. If it's me, I'm looking at it and saying, okay, I gotta roll something out. I wanna roll, you know, MDR out everywhere. I think 70% of my customers will pay for it.

How do I price it so I can roll it out any, everywhere, even if only 70% initially are paying for it, right? And then I'll, I'll go work on the rest of 'em later. But are you seeing, I guess in your customer base, you have to be seeing people that are getting out there, right? 3, 3 50, 400 bucks a seat now and that's their standard and they are baking everything in, right?

Yeah, we're definitely, there's no question the most sophisticated, largest A RR MSPs with the highest gross margin consistently. That also, by the way, when you think about exiting someday, if you do, all that goes into your valuation multiple. So if you want another reason to kind of do this and have a sophisticated operation with predictable margins is because it makes you as a company more valuable.

Um, and so we are, we are seeing it, we're seeing a huge push this year where partners are saying every new customer we onboard is getting upgraded. And what they're trying to do is figure out how to get the stack super aligned so that they can roll it out to everyone else where maybe, you know, sometimes people negotiate a contract three years ago and they haven't really increased it as much as they probably should have because they wanna retain the customer. Totally understand.

So it's the job of the vendor community, the ones that really struggle, the ones that pick a whole bunch of point security solutions from all different vendors try to string it together. Their cost of goods sold is too high and they can't manage or run it well. Yep. Um, and, and so, you know, I think where the industry's going is the vendors are moving towards a more ecosystem play. So you can be like, you can save two bucks here, four bucks here, three bucks here, guess what?

You just paid for it and now your customers are, you have a defensible stack, like Matt Lee likes to say. What you really agree with is Yeah, defensible Terminology. I mean, I remember Andrew, we were talking to someone who said they started putting a model year on their offering. It's like, you know, you're on the 2022 offering. Oh, that's really interesting. Here's 2023 T actually, you can never be more than a year behind. So this year you can be on 2022 if you, you know.

Um, but so it's almost like you gotta find creative approaches and it kind of goes to my in in time. We have my last question for you, Dave, and we touched on this, but part of this is our customers don't understand the risk they have or they wouldn't care, they would pay. So how, like, based on what you do, and, and John kind of alluded to it, how, how would you recommend MSPs start to share this information with their customers in a way?

So they really, 'cause if they understood their risk, they would make the right investments. Are you with me on this? I think as, as John mentioned, right? There's a lot of these customers that their sole business is these, these one thing. If they are hip or ransomware, that's it. Their business is going under.

Um, and I'm not, I'm not encouraging scare tactics, but a lot of the content that we put out from a technical level can sometimes be a bit too in depth for a customer and end customer to understand. So it's important for the security providers and the MSPs to have a common language.

Whether that's, and, and I don't mean this condescending, we bring it down to an eighth grade level, we bring it down low, we make it easy speak that can be translated to the customer in a way that they can understand. That's the best way we can do this. Yeah. If a customer doesn't want something, they're gonna drag their feet kicking and screaming. And the best thing we can do is try and provide security for them anyway.

But if we can make it easy for them to digest, if we can help them understand the dangers, the, the, the negative impact of not accepting these security risks and dealing with them, that's, that's a win for us. Right? Yeah. And They don't want any security products That coming out that, that Yeah. They don't want any security products. Like, I don't want any security products, I just don't want security risk. Right. Yeah, That's the point is, so, all right, Andrew, I'm gonna let you wrap it up.

We're a couple minutes, a minute or so before the hour. Yeah. Well, it reminds me of your great quote, Gary, why do I care? You know, in a, in a, you did a role play, why do I care more about your security than you do, right? Yeah. If you recall that, Um, yes, I do recall that. Why do I care more about your security risk Mr. Customer than you do? Listen, I say the same thing sometimes at our peer meetings to our MSPs, why do I care more about your security risk than you guys? Yeah, yeah, yeah.

So in closing, John, Dave, this was, as Gary said, awesome, the way you guys are able to articulate it and bring it down some very complex, uh, terminology, some very complex, um, things that you guys see. Um, so thank you for doing that for everybody here. Um, as always, Phyllis, Gary West, thanks a million for, for coming on each and every week. We'll be back next week and, uh, look forward to seeing everybody at 1:00 PM next Monday. It'll, uh, until then, make it an awesome day. John, Dave.

Thanks sir. Take care. Thanks everyone. Thanks Guys. Thank you so much. Bye.