July 20th, 2020

All right. Live and recording. I think this is week 11, Kyle, am I right? Is this episode One? It is. It's week 11. I've ran out of fingers. So gonna toes Well, um, fantastic. So I see Wes is back. Welcome everybody. And uh, let me just take care of some housekeeping here. We've got Paul Scott from the Perch Threat Research Team with us. We're trying to get Wes back up. Unfortunately Gary couldn't make today. He had a last minute thing come up. There he is there. I'm the, the Astute Library.

I like that. And a retro perch shirt. Yeah. Or a new perch shirt I've never seen. Yeah, This is, uh, the new perch burn shirt. You guys will see more about this soon. It's uh, it's pretty rocking. Excellent. Okay, so just a few quick housekeeping things. I'm gonna put in chat number one tomorrow. Kyle and Gary are doing an event, uh, at one o'clock and it's on optimizing your security offering. The URL is in chat, and so I highly encourage all of you to attend that.

Uh, I am also putting in the chat, um, something that Perch, Huntress and ID agent are doing, um, in August, uh, August 11th through the 13th. Um, and it's the cyber trifecta, the nuance of the acronym CTF. So we're gonna have two things going on simultaneously, almost like we did with the Cyber Con, but not the immense amount of content because we know you guys are pretty much all content out of these virtual things. It's gonna be one session per day.

So IE the huntress day, there's gonna be one session, one hour, but simultaneously through that event will be, um, uh, a, a Capture the Flag event. And so you technical folks that might want to, uh, get involved in ethical hacking forensics, that'll be a time for them to do that. Um, even if you've never done it before, we're gonna have some courses available for you. I'm gonna be mentioning more of that. I sent an email out to that as well. Um, from EC council.

Um, I've been able to get a dirt cheap price for everybody on some fantastic content around that. Alright, so with that, uh, quick agenda for today, I'm gonna have Kyle kick things off. He, um, did a excellent piece on trade Craft Tuesday, it's around Mitre Attack framework. I'll have Kyle just give a snippet about what Mitre attack is, but it lends itself really well into what we're all gonna talk about. And specifically Paul and, and Wes.

But I certainly, Kyle, I'd be chiming in here because it's, it lends itself perfectly to what he does. And this company does is this exploit that check checkpoint team has found, uh, A-C-V-S-S score critical vulnerability in Microsoft's DNS of a 10. Of a 10. There you go. So we're gonna talk about that and the implications. We've got some polls for everybody. We'd like to know what you think. And then from there, um, we're gonna bring on Mike bag beard, uh, the of Marco.

Mike, you know, he's already thinking sales and opportunities and we were discussing some of the things that he and his team are doing. Um, he was mentioning specifically a, a significant opportunity that they're working on right now and mentioned this to this company replacing their Microsoft DNS over two years ago, uh, to an F five solution. So Mike will be talking about the monetization aspect of this whole piece.

Lastly, we'll close it out, um, with a new guest on this show, uh, uh, Peter Lowe from DNS filter, who's gonna be talking about, um, DNS encryption and, and, uh, you know, I love your guys, you know, whoever is on at that point, whether it's Wes, Kyle, um, I'm sure you guys can add your, uh, thoughts on that. So with that, let me, Kyle, let me turn it over to you.

And if you could frame out briefly, and I'll put A-A-U-R-L in of a video on a Mitre attack that people can listen to is really well done. It's short. Um, but what is it, why is it important? And tell us a little bit about execution and maybe lead us into where this whole thing's going with the DNS. Yeah, so for those of you, and I won't even say living under a rock 'cause we've all got 12,000 things to, to handle, right?

Um, but one of those things is, well, how did you keep track of what either good or bad attack or defense? How do you keep track of what's going on? And so thankfully the US government stepped in, uh, quite a while ago and said, look, we use some similar nomenclature in the DOD world as well as some of the, you know, financial sector, which talked about tactics and techniques, uh, procedures thrown in there as well.

And so they said, you know what, it would be really nice to, like, let's try to define this in a more graphical version. So if any of you have not come across and, or just posted the link there, but the idea behind it is the Mitre attack framework says, look, we're gonna try to organize this in a more simple matrice, right? So if you think about this of tactics might be like initial access or execution or my house's persistence, long-term access to main maintaining it in there.

But then there's techniques, right? Tactics have techniques. And what was beautiful was Mitre just went and revamped, they've been working on this for years now, but one of their most recent updates that they did was they revamped some of the matrice to collapse some of these sub uh, techniques and sub, uh, you know, more or less capabilities to make it a little bit easier to understand and follow. So we've mentioned it before here on the cyber call.

I would highly encourage because what's beautiful about it is it gives you at least a way to organize. It's not the only way, but when you start thinking about it a high level for compliance or even just making sure that you're doing better job protecting your house, let alone your clients, it's nice to align them with a common nomenclature that whether it's in the enterprise or SMB, we can all use the same terminology.

With that said, Andrew mentioned execution was one of the pieces that we recently did an episode on. And if you think about this, some of this way of getting execution could be like your RMM and they call that in the Mitre attack framework Software deployment tools. So this is a way to be able to give execution in your network. Other ways of execution, as you could imagine are those end users and they call that user execution, right? The people who will fall for those phishing mechanisms.

But all the way down to maybe you want somebody to set up like a time bomb right? In your network, that could be a scheduled task or Aron job that kicks off later. What's really beautiful is one of these execution techniques, those stands out for today's content and that's called exploitation.

And so one of the things that both, uh, I guess the easiest way to describe it is both Wess, and correct me if I'm wrong, but some of the, uh, conversation Paul, today, you guys are gonna be di diving into the DNS exploitation bug that was, uh, impacting, uh, what's like a 17-year-old vulnerability, uh, in Windows DNS. Is that correct? Right. That's right. So maybe as the segue onto there, this Mitre attack frame, which would you have all kinds of links to? There's a lot you could dive in.

We did a whole one hour episode on it and then had another 20 hour, uh, 20 hour, 20 minute q and a. But where I want to go with, it's a beautiful way to start having a technical conversation with your own technicians. Even if the clients never appreciate it, you can at least start saying, Hey, does this fall, you know, how are we doing a better job? Or when your clients are asking you how do you provide coverage?

You could say, well, believe it or not, we once a month focus on a new tactic and that tactic we harden ourselves. It's similar to how accountants might say, Hey, if I have federal regulations on acquisitions, I follow each new chapter of the acqui acquisition or regulation. Well, for us in the technical world and security, we follow Mitre attack framework. So Andrew, was that a good enough, uh, quick introduction to that side? That Was perfect, Kyle.

And you know, it's interesting Wes, I'm gonna say something, but then, but then hand it over to you Wes, is the, is this kind of a, a segue if you will for, you know, common language and you know, we've talked about the potential to have an ISAC or an ISO for MSPs to talk and a way for them all to communicate amongst one another. And I is, is that really the intent of the Mitre attack framework when there are threats and tactics out there? Um, at a high level? Yeah.

You know, it's it, Kyle, I know you'd agree with this, especially in your background too, coming from NSA, we need in the world of security, a unifying taxonomy when we talk about all things. And I think because security has just sort of grown and I like to call it Frankenstein out of it and IT operations, we struggle. We have really good taxonomy and unified language in the world of IT for the most part. But we struggle with that in security.

So Kyle, you and I both did you know that demystifying cybersecurity talk 'cause it's a pet passion project of both of ours and we're starting to do a good job of this in the world of security, like you just mentioned, we said, I saw you go put 10 fingers up for a 10 on the CVSS score on this DNS vulnerability. That's good. That's a valuable thing because now we can at least to some degree speak on level playing field of what that 10 might mean.

And I think Mitre attack does the same thing, doesn't it? Kyle just unifies and gives us a taxonomy we can all talk the same language for, right? Yeah. You and I both said, you know, a lot of times it's not necessarily a technology problem or you know, of all the problems we could blame, but sometimes it's just a good old fashioned language problem. Yeah.

And so, like you said, whether it's just agreeing that a 10 is a 10, uh, or being able to say, look, I'm trying to solve this one specific problem with security. It's beautiful that we now have that language in there or jargon to be able to have that, you know, standard conversation. Yeah. Hey, and, and I appreciate seeing Gary and Hart with us. So, uh, this is great one pour out one for our, for our homie Gary. Alright, so, oh, go ahead.

I was just gonna say, and last thing I'll say, you and I were talking offline and, but I thought it was really important a piece that, you know, so often, unfortunately these days we're, you know, hearing about vulnerabilities and the tools that the, the MSPs use and that, you know, if you analogize it, you know, checkpoint found this Microsoft put something out right away. There was no like, what do you mean, why would you do?

It was okay, there's a known vulnerability, let's collectively for the better of the good of everybody take care of this, make it known as quickly as responsibly as possible. And, and so I I just kind of put that out there of how, looking at how the enterprise looks at these things, um, and that, you know, we've got probably some maturing as well to do in this whole industry as, so lemme let Yeah. Yeah, we, we really do.

And I think we've got some takeaways for all of this that I want to chat about that I think we can all learn from Microsoft's, um, own journeys. Those of you that remember, I think most of you do on this call, the, the 1990s Microsoft, the early two thousands, Microsoft, they are a completely different organization in how they work through and handle and disclose security vulnerabilities I think we can all learn from and take value from. So we're gonna get into that.

And I wanna ask a question to Paul to get us started. So those of you don't know Paul, he is our director of threat research and we brought him on. And Kyle, I really want you to stay on too 'cause I I value your feedback as well. 'cause I know you've been in this as much as anybody as well, ki or, uh, let me start with, with, uh, you first. Um, Paul, can you kind of give us a walkthrough?

I think most of us are familiar with the DNS vulnerability, but can you give us a high level just in case anyone's popped on and they haven't been reading the news channels and why it's important? 'cause I think there's some more things we can extrapolate from this. Yeah, sure. Thanks Wes. Um, yeah, so it's a definitely a critical vulnerability, uh, for those that haven't heard. Um, there was a vulnerability found by Checkpoint in the Windows DNS service.

Um, so that, so this is kind of unique in that, you know, it's not necessarily, um, exploited in like a standard way. It has like a really interesting attack vector in kill chain. Um, the, the victim, um, is actually gonna be your Windows DNS server or a Windows server running, uh, the DNS service. And then the attacker, uh, or the malicious party in this case is going to be a re a, a third party authoritative DNS server for some domain, uh, that the attacker controls.

Um, so it's really kind of a vulnerability between two servers talking to each other. So it's pretty interesting there. Um, but yeah, once they, uh, trigger this vulnerability, they can get code execution on that box like Kyle was talking about, get some persistence, um, and other things that we would talk about in the, uh, MITRE attack kill chain. Yeah. And, uh, Andrew, can you get the poll question up? One of the things we wanted to do is just get a qualitative question out to you guys.

We're curious the level of concern that you have, um, you know, are you somewhat concerned, super concerned, not at all concerned about this? Uh, and, and as we get that poll question, I actually wanna direct that right at Kyle. Kyle, in your opinion, how concerned should MSPs be around this vulnerability?

So, I mean, when you hear cv, uh, you know, uh, C-V-V-S-C-V-S-S, if I could speak correctly, uh, bug, uh, you know, of 10, you immediately say, okay, that's 10 out of 10, I should be concerned. At the same time, I would say within this last like three weeks, we've had multiple 10 vulnerabilities that have come across. Whether it's been in, you know, uh, enterprise firewall hardware, whether it's been our Windows domain controllers that might also be duplicating as our DNS servers.

Um, at the end of the day, to me, I, I try to just bring this back into business terms, Florida folk, right? We don't get terrified that every time a hurricane comes around destroys things, they just, hurricanes happen to me. It's another patch, another day, another vulnerability test it, make sure it's stable, get it fixed, and move on. So I would encourage, although it is serious, there is no underestimating, they don't give a 10 for no reason. Um, tens happen. There's been multiple here.

And I think what's at least beautiful about this is a lot of it is patches available, uh, you know, near real time. They did a pretty good job at making sure it was disclosed appropriately from the checkpoint team to the Microsoft team to push to customers. So, um, yeah, I I would say at the end of the day, yes, as Wes said, tens uh, tens happen, but, you know, patch the darn thing and get back to what your customers are really excited about.

'cause none of them are excited about a DNS security vulnerability. Yeah. And I wanna go off script for a minute, Paul, I got a question for you. So, 10 Kyle, to your point, tents don't happen that often yet. We've seen a slew of them recently.

Paul, do you see a lot of tents come across and is there a point we can get to with CVSS scoring that there's almost a marketing drive behind it to go discover the high nines and the tens and we can get to a point in the security community, we get like 10 fatigue. You know, like we talk about breach fatigue. You hear so many breaches in the news, people just finally turn it off. Are, could we get to that point? Is there risk in that? Paul, what are your thoughts?

Um, I think it's a, it's, it's an evolving standard. So I think that if we got to a point where we were just saturated and everything was tens, we'd probably look for different ways to kind of, uh, analyze that risk and talk about them differently. Um, so I, I assume that like, maybe like in an updated standard, they would like maybe add some more criteria that makes not everything a 10 if, if all we're getting is tens now. Um, but yeah, there have been a lot of tens recently.

There's been a 10 in like almost every major, uh, networking product, right? Uh, recently. Um, so yeah, uh, definitely seeing a lot of tints. And Kyle, a question for you, and I actually want to kind of dive into this from what PJ's saying in chat, you know, it's mostly bad because of the news cycle and customers asking about it, right?

So, um, in your opinion, how can MSPs use this to answer in a way that, um, you know, this would normally be a Gary Pika question, I'm turn it to you to, to really show the, the, the resilience of the MSP themselves and understanding the threat vector. How should MSPs message this back to their clients when they ask, Hey, what about this thing? What do I do about it? To me, it's more important about the recovery.

Like when, when people buy security, if you really think about our end client personas, they don't, they're not buying the recipe for the sausage. They want the sausage to taste good. They're buying the end result, which is either very, very fast recovery or just to stay resilient up and running.

So to me, there you could easily let this opportunity, like, uh, you know, marketing folks would tell you, you know, don't let any disaster, uh, you know, go, uh, I guess it's, you know, go without waste, right? Or don't, don't, uh, waste any good disaster. But, uh, I would argue at this point your customers probably need to know, hey, at a high level there's some pretty big security issues that have gone on. You might hear about 'em.

'cause they might make news, especially in light of like the Twitter situation that happened that I think made even bigger news. Um, but what's beautiful about it at the end of the day is to say, look, part of the reason you didn't hear anything about us is because it's just another day we fixed it, we moved on. You are secure, you're done. But that at least gives him a heads up of, alright, if I do hear this, I know not to be freaked out. It alleviates calls late at night.

At the end of the day, you get to take a little bit of credit for something that you are gonna do anyways. So that, that's how I would suggest it get used. Yeah, that's good. So we've got about 40 votes here, uh, that came in on the polls and it's not too late to pop 'em in there. Yep. I'm seeing more come in. Most people, it looks like about 75% of those that are responding are saying they're somewhat concerned and there's a goose egg, right?

A zero and not at all concern, which I think everybody that has a rational thinking brain is probably gonna go Yeah. I mean, hey, it's a 10 for a reason. Right? But, uh, Paul, does that kind of, does that follow suit with what you're thinking here, somewhat concerned? And then a few varies. Is that kind of what you would expect to, Um, yeah, I would be, I would expect people to be concerned. Um, you know, you hear, okay, this is a vulnerable DNS service, uh, but I'm not running DNS externally.

I don't know, you know, I'm not running, I'm not like letting my windows DNS service be accessible externally. I did a show dance search before this. I only found like 2,700, uh, publicly accessible, uh, windows DNS services. Uh, I don't know if Kyle's expecting that to be higher or lower. I saw him grimace. Um, yeah, There's, I mean, there's a lot, but it's, I mean, it's DNS and should DNS be exposed this way? Should it be behind stuff?

I have a feeling there's a lot more of these boxes that are unpatched behind one of those probably network devices that we just talked about, which are probably already vulnerable as well. Yeah. So, but that's not, that's not the only way. I was talking about how interesting this vulnerability is to exploit. In the demo checkpoint gave a really cool proof of concept where they sent an email, got a user to click on a link.

That user was in your network, they make a DNS request, and that hacks your DNS server, right? The response from that, uh, request hacks your DNS server on its way back to the client. Um, but that's not, uh, that's not the only way either. There's a couple other scenarios that that, that are interesting here to consider. Um, if you, how many people are using DNS SEC in their Windows environment? Who has DNS SEC validation enabled?

Uh, Yeah, we've got a poll question open for that were curious to, yeah, I was just curious to know if that's something that's commonly rolled out. Um, I'm guessing not at all. Or maybe, maybe like you've got a couple banks or something that may require it. Right now we just have a few votes coming in at none. So, so this is the, this is my registry key of the day. If you were gonna go check out a registry key today, I'd go check out this registry key.

It tells you if DNS SEC is enabled or DNS SEC validation is enabled in your environment. If that is, this vulnerability is actually way more exploitable, um, getting, uh, any, any device in your network. If, if, if an attacker can control that thing, making a DNS request somehow, uh, it could trigger this vulnerability.

Um, for instance, um, you know, a user browsing a website with Chrome DNS prefetching turned on that could trigger this vulnerability if you had DNS SEC turned into your environment. Um, any server side request vulnerability in an application, uh, could trigger this vulnerability.

Uh, maybe even your Barracuda email filtering gateway, if it's using your Windows DNS server for DNS resolution, it may be looking up domain names on, uh, emails coming across, or even just your normal email server in the, in the normal course of validating, uh, who the sender is of an email address. Like all of those things could trigger this vulnerability if you have dnssec enabled in your environment. Um, so that's something that wasn't really out there.

Um, and the details that was released, I mean, yeah, I, I see in chat has some hilarious conversation, uh, Dustin saying, Hey, so we're all secure by having no DNS sec. Right? Let's, uh, I mean, I couldn't take an, uh, I couldn't take this, I couldn't miss an opportunity to, uh, talk smack about DNS sec. So I just had to bring up that if you had DNS SEC on it was actually made you worse off in this situation. So, Kyle, a question to you, and this also goes to what's happening in chat, right?

There's a fine line or there's a balance between not peppering your clients with so many, uh, newsworthy events that they just finally tune you out, but also being that trusted advisor, I love what R Potter says there in chat. You don't have your full name, so you're just R Potter to me right now. But they're saying, you know, Hey, look, it's great to be that trusted advisor and it's great when they come and ask you. Right? So what's that fine line?

Do you have some recommendations for MSPs on when they proactively reach out and when they may just have messaging that just is there and maybe when they choose not to, to message or share something? Any thoughts on that? Kyle? Uh, I think about the population of our MSPs, and the reality is, we, we are aging and we're usually white men. That's not always the case. But one thing I know about aging white men is we're infamous, uh, you know, infamous for terrible dad jokes.

Dad jokes can take anything, right? And make it relatable to your kids in the most cringey way. I think if we redirected some of that same experience towards maybe some analogies that our partners and clients would appreciate, right? Don't come to them about the security vulnerability of DNS. Nobody cares about that. Um, I tend to pick on accounting folks because they're always the ones that don't understand the value of money or risk, yet their job is money and risk.

But being able to take these examples and relate 'em to your audience is gonna be key to any good communication. My wife doesn't care about details of Hunter, she just doesn't care. But she loves to celebrate some of the success when we do something good for somebody. So make that that example, make that something that's not, not cringe-worthy, that your kids won't be like, ah, but something, your clients will actually appreciate it.

And I think good analogies right, are a good place to get it started. Yeah. So that, that's good. And I wanna segue just for a minute, Paul, a question for you. So, we're currently not seeing any active use of this exploit, and there's reasons for that, right? Like, uh, when you read the checkpoint re, uh, report themselves, they, they hide what they called some of the primitives, right? The requirements to make this work.

Some of that was not published both at the mi the request of Microsoft for obvious reasons, and then also Checkpoint, I think, you know, no one's wanting to publish this, um, and, and get the news out that bad guys could potentially use it. But we want to show the severity and where the weaknesses are and show how Microsoft was quick to patch, right?

But Paul, do you see, and Kyle, I want you to follow after this, but Paul, do you see, or do you foresee threat actors starting to figure this out ahead of time and those that have not patched that we might begin to see activity, uh, from threat actors around cigarette? Yeah, so, um, I do expect that we will see some proof of concepts and exploits and attempts to exploit this.

Um, based on, you know, the nature of how this is exploited, it's gonna be a little bit harder to see scanning attempts because to exploit this, if you had a publicly facing, let's say they were trying to do some scanning, right? So they would wanna try to hit these public, uh, facing DNS servers, um, they would have to set that DNS server as their resolver and then try to resolve their de malicious domain using your DNS server as a resolver.

Um, so we're not going to, uh, we won't see like, broad scanning for this type of activity. Um, like as I mentioned, there's only 2,700 of those, uh, systems out there. So we might pick up some of it, but I wouldn't expect to see like the broad scanning activity that we see with other botnets. Um, we have, even though there was, uh, limited information released by Checkpoint, they did do a good job of releasing enough information for the blue team to be able to kind of, uh, see what's going on.

And also for defenders to be able to put workarounds in place. Um, so they gave us just enough, see, this is the key, right? So you want to give people just enough information that they can, that we can help defend and understand if we're being targeted or attacked or impacted by this thing. But not so much information that you give attackers, uh, the ability to go rapidly exploit it. They have given people enough information for attackers to go find that vulnerability today.

Uh, Kyle could probably go, uh, reverse engineer checkpoints research and get us a vulnerability today, but, uh, an exploit today. Um, but, you know, uh, they haven't released enough information for the general public to just whip out an exploit. Oh, there's an exploit. Yeah, I, I had to share, I ironically, a lot of the exploits are only in proof of concept or even like, uh, you know, it'll denial a service, the computer, uh, right.

But I know about six days ago, which is about one day after, um, the internet delivered with a, you knows, I'm looking at you for the exploit for cigarette. So, um, take it with a grain of salt, right? I mean, that's why they have the, the term right? Zero days, meaning it's unpatched and unknown, but usually one day. 'cause that's about a time of any real exploit developer it takes to be able to weaponize something with the right skills. So, uh, Mike, Jed, warming, don't click that.

Yeah, I'm, I'm sure that's gonna light somebody's, uh, content filter up. So we'll go run this in the lab and, and see what we can detect. We've already gotten, um, uh, four different signatures for possible, uh, exploit based off of the details that were released. So I'm pretty sure that we would catch it. Uh, but definitely gonna execute that proof of concept, see what happens. Yeah.

So, so with, so, so with that, I think it'd be good, a really good segue 'cause we started to talk about communicating to clients. What's too much, what's too little, um, you know, what, what, what porridge is just Right, right. And the three little bears kind of thing here. So with that, Paul, thank you so much for coming on. We'd love to have you again, I'm gonna move you to the audience. I'm gonna bring Mike Ard up, who's on the front lines.

Um, you know, who is the first of all, you know, acting internal CISO of, um, Marco, but then, you know, the external one as well. And, you know, what is it like for him and, you know, his sales team that he's getting pulled into on some of his larger deals and, and how are, how are they communicating? 'cause they've got a pretty wide swath of customers that they could, uh, you know, have a good litmus test of really what's going on in the world.

So with that, Mike, thanks so much as always for coming on, uh, with us. So, uh, good to see you. Yeah, thanks. Happy to be here. Yeah, absolutely. So, okay, to segue that to you, and I'm sure Wes will have, and Kyle have some questions for you along the way, but did you kind of get the sense of what was going on, the ba the banter as far as, you know, what is the happy medium? And then, you know, what are you guys seeing really on the front lines and is it, can you turn this into money?

Because at the end of the day, that's what we, you know, we're trying to always bring a fine line here to, can we operationalize something? Is there money behind it or is it just another thing and more noise that there are threats and you know, the world's falling, right? So let you take it, Mike. Yeah, no, that, that's right on. I mean, it is a fine balance and I think that's, um, that's gonna be different for everybody depe, depending on the industries that they serve.

Uh, for sure, if you're in more of a regulated space, I think obviously the messaging is a little bit stronger. Um, if you're in just a more general industry and you serve customers of all flavors, you're probably gonna be a little bit more generic. Um, I, I do think you want to control the messaging, and that's been in line with a lot of the comments that have gone through, um, just in the chat today. But to absolutely control the messaging. I, I look, I look at it like this.

If, if attackers are gonna monetize it, we should monetize preventing it, right? And that, that's pretty straightforward. I think, uh, something like the DNS vulnerabilities that are out, um, you know, I look at, have we had conversations around how to prevent that for some organizations? And again, it depends what they're doing.

If they're hosting public DNS on Microsoft servers, you know, we've probably had conversations with 'em over time on DNS SEC and some of the other things that we've talked about. Um, yeah, this is gonna be an accelerator for doing potential projects to change that and maybe move to a hardened platform. Maybe it's a F five running a little bit more secure features around DNS or perhaps it's moving it to Azure DNS, you know, moving it to an offsite hosted DNS.

Uh, those are absolutely something that we can do. Uh, I look at this though, when I look at vulnerabilities as a whole, it, it, you know, Kyle said it, it's been a brutal couple months. Um, for vulnerability management. It's been tough on MSPs 'cause it adds a lot of burden that we're not necessarily ready for, right? We, we take these things on, and a lot of cases, customers expect it to be done. But I would say pivot that conversation.

Um, you know, I look at some of the laws out there and, and like Oregon for example, they've gotta, if you're selling iot solutions, you have to sell some preventative measures and security around that. We'll take that type of approach to it, right? If I'm selling solutions, I should be selling security as part of that right from the get go, sell those mitigation strategies. Um, the example I'd use for that is web application firewalling. You know, there's a lot of vulnerabilities.

I'd say most of 'em typically come out and they really hit us on the perimeter. We'll put better prevention technologies in place in front of that. And, you know, I know some of our MSPs out there do that, put a WAF in, um, you know, build that layer on, um, right from the get go. And then sometimes when these vulnerabilities come out, the goal is, is that when most of them come out, it's not quite the fire drill that it necessarily, uh, needs to be.

So that's what I would, I would say do first and foremost, it all comes back to vulnerability management and assessing, right? Know where your risks are. Um, one of the things I'd see is we don't want to get lost to be upfront. We're probably not gonna notify on every single, even if it's a 10, every vulnerability that comes out, how we handle it is gonna determine if we notify on that. Um, that just goes back to, you know, your notification process.

And that's part of soc compliance for anybody that's working towards that, how you handle it. 'cause what I don't wanna do is get fatigue around messaging out to clients to where then they start losing messaging as well, right? I don't want them to miss something that's actually important because we're telling 'em all the time, Hey, there's another nine or 10 out that we've taken care of for you.

Mike, was there a process though, that you guys went through knowing you have a fair number of clients that you may have messaged this to? Did you know, was it like, hey, you know, search through a certain mechanism? Like how did, because for the ones that may have still had it, like how, what, what did you guys the operational side of this? Yeah, so I mean, filtering through client, obviously clients that are in a more impactful area.

So, you know, financial clients, that kind of thing, we're, we're, they're more likely to have some of the features that are more exploitable, DN ssec, that kind of thing. We're gonna filter through that. And that's, that's a little bit of the, you know, your records are always gonna be the, the success factor there. Um, being able to identify that. Um, otherwise I think it turns into, um, in some cases, you know, clients are gonna come to us.

That's never a position we want to be in, but we have, we have bar type customers as well. So I do expect a fair amount of that. That's having that consistent message across the board. Um, you know, we'll, we'll see that as well. So I, again, I think it's gonna depend Ms P to MSP, but if you're in one of the more impactful areas, financial, um, you know, government of really any kind, um, definitely want to control and, and proactively go after those customer segments.

I, I wanted to challenge my, or you know, I more or less channel my inner Gary here for that conversation because I think about how simple that is. Like, can somebody really, when something like this happens, right, say it's the next 10 vulnerability, can somebody honestly say, here's the Venn diagram of my customers that are actually the ones that rack and stack and rank like that. That's a significant level of operational maturity to even just start there.

And I'm, I'm willing to challenge, I bet most of us, if you're talking about what can I pull away from the cyber call today, maybe you're, one piece that you pull away is when something like this does happen, who would you even notify first? Or what categories do you even have that broken down? So Mike, huge thanks though for highlighting that. 'cause I probably wouldn't have pulled that away without you saying it the way you did. Yeah.

It probably falls in line, Kyle, similarly to some of the, like the, we did recently a tabletop exercise around incident response. It's like, you know, if this really happened, who, who actually calls the shots? What's the team look like? Is there the policy in place? Have we run it before? You know, like to your point, right? So it highlights those things. Yeah. Because there's there, there's value in figuring this out and there may be multiple paths to get there.

Um, but like even in the tabletop last week we did, we spent a lot of time talking about breach notification, right? Like that's something finite that every MSP must know. Breach notification requirements. You may have some of your contracts, especially, uh, healthcare that mandate, you know, they may say within 30 days or five days or whatever the timeframe is of a breach notification to them. You've gotta understand the regulatory requirements. You've chosen to swim in that pool, right?

If you took on financial healthcare, DOD manufacturing, whatever it may be, if you took those clients on, you also take on the regulat regulation that's going to, um, have oversight over all that as well. And so knowing breach notification and knowing how I handle, that's really important. I think going back to this also knowing, hey, who are those clients that we're serving that really care and understand and need to know this and expect some kind of early notification?

And who are those that just say, like to David Powell's point, David wrote some really good stuff in the chat about, like most clients, it's like a 401k report. I just wanna see how I'm doing here and if I need the rest of it, I can go check later. I just wanna know the health checks. Right? I, I do think it's important knowing those clients, knowing the difference between them, so you know, proactively how to reach out and how to discuss. I think that's important.

I I do think there's multiple ways we can get there though. Um, but I do think for all of us having some amount of a value report that here's what we've done for you over this period of time, and including in vulnerability and, and major newsworthy occurrences are, that's really important. I know I did that at the bank, spent a lot of time, every single time we'd meet with a steering committee for information security. I'd walk through the big notable events that are happening that month.

And I would walk through why they're important, because I want to be that trusted advisor and I want them to know, Hey, you've got Wes as your key tie in to all of these things that are happening. I can distill it for you and tell you what you need to know. So I do think that's really important. So let me close it, kind of put a bow around it.

'cause you were talking about what lessons learned, what key takeaways, um, and that the last three months, as Mike has said, and Kyle said, have been kind of brutal around vulnerabilities. My question to you guys is this, will vulnerability management play a bigger role for the MSP, you know, in the upcoming years? Will they have to some way somehow figure out how to include that in their, in their cake and, and, you know, increase their cost of goods and then be able to pass that down?

Mike, starting with you, what are your thoughts? Yeah, absolutely. It's gonna, I mean, we've already seen that, you know, we've heard about CNMC on this call, uh, recently. There, there's some drivers there. You know, CJI, anybody that works with criminal justice, they had an update in June. We're, we're seeing all the frameworks change and there's a significant focus on vulnerability management going forward. Yeah. And the upfront we're seeing it.

I'm seeing it just in, uh, in the healthcare industry as well, and BAAs and business associate agreements and that type of contract as well. So that you, you're absolutely gonna see it. I think, uh, don't be surprised to see c you know, CBSS with a seven or higher targeted. We're starting to see that be a prevailing concept in a lot of industries. I think you'll see in state cyber laws as well. Yeah. Well, Mike, thanks as always for coming on.

That would be a great, I think follow up conversation with all the things you're doing personally and with some of the other MSPs in the space around vulnerability management to have you guys back on to, to talk about how do you package and price. How do you all of a sudden tell a customer, Hey, by the way, we really haven't been, haven't been doing patch man.

I mean, we've been doing patch management, there's this other thing, you know, so trying to be a little coy here, but I think there's this whole other piece that we're gonna have to start looking into, um, and that the customer's gonna assume we're handling. So, Mike, as always, thanks a million. I'll move you over to the audience. Gonna bring Peter up. So, uh, have a, so you gonna, While we bring, uh, if, if, correct me if I'm wrong, Andrew, we got another guest, right? Yeah.

I got Peter closing things out, Kyle and I, I think this is where I'll need, you know, probably your and Wes' help and some of the conversation here. 'cause this is, uh, I, I have to say a little bit above my pay grade, but getting into the ch the changes that are happening with, uh, the different encryption methods in DNS, uh, on the internet.

So, um, I got Peter coming up here and we'll close things up, but did you have a thought, Kyle, as we found You just, you asked Mike a really good question, which was, you know, do we have to eventually like, uh, you know, determine where does vulnerability management exist? I published two of the vulnerabilities just in the, the stuff that we're all familiar with in here, which were RMM ones in the last week, I'd argue. We've just always had vulnerability management.

We just tend to be really bad at articulating and taking credit for what we work our butts off to do. So, um, if you are not charging for it, and if it's not built into your margins, now you're just giving that away for free. But it's still something that we've all been responsible for, is my take on it. Yeah. Um, while you pull, uh, you said it was, is Paul coming back or you said it was Peter, Peter. I, I, you know, You said Yeah.

And while we're waiting for Peter, I mean, Kylie, I think you're exactly right. And I think one of the problems we've had inside of the channel is it's not that vulnerability management is something nobody cares about or nobody, um, wants to address. It's just, that's been one of the challenges in the industry, right? There've been some really good enterprise tools that help with vulnerability management, but they don't scale down to the channel. And that's been a challenge.

I mean, I can't tell you any conversations that I've had, and Kyle, I'm sure you're the same of the woes of triumph. Just throw a name out there, you know, like a Qualys or ESIS that's out there that are great tools, but just don't really work well for us in the channel. And it's not all about tools to solve vulnerability management. We all know there's gotta be some automation that happens behind the scenes. And I do think that's been a big challenge that MSPs have had.

And vulnerability management is not just like throwing some tool out there that reports on like active directory domain status, and you drop 500 pages on the client's desk saying, look what we could do for you that doesn't make sense to them, right?

What they really wanna understand and know is at the end of the day, how do I make sure that if I go with you, the value is passed back to me in terms of I've got, I, I don't have to, I can sleep at night, I can worry about running my company and you guys are gonna handle all those other things. And so vulnerability management is a critical piece to that. But MSPs have struggled with not really having the tool sets that they need to really operationalize it. Kyle, what do you think?

My, my lead investor on my first investment we took at Huntress was the co-founder of Tenable who wrote Nests. And he describes the industry that we serve is below the security poverty line. So I mean, if there, if there's anything somebody who's wrote a vulnerability, probably the most popular vulnerability management product in the world, yet it acknowledges that it's not available. That's, that's that, Was that Ron? Yeah, that is, that's Ron Gula. Yeah. Sounds like Ron.

Well, I'm not sure what is going on technically here with Peter. It says it's accepted and connecting. Um, we might have to have a, have a, Let's just through that DNS payload at, uh, Crowdcast, some side effects going out. I'm gonna try one last time here to see if I can get Peter on and I don't wanna hold everybody up. Um, Hey Kyle, while we're waiting for him, I had a question I was gonna ask Paul and, and then I forgot and I reremembered.

So I'm gonna ask you, there's the way that Microsoft handled this, like you can even see it on the timeline, uh, you know, at the very end of the, the sig red notification and, and released by checkpoint of how they received it, they answered it. They worked internally all the way out to the release and working with Checkpoint. Like that's the way, that's standard operational procedures for enterprise and handing vulnerabilities. You know, I always say they're like belly buttons.

We all have vulnerabilities. There's lessons we can learn in the MSP community around how to handle these things. Don't you think? Yeah, I think both, uh, you know, two forward, uh, when, when you think of the vendors getting better at communicating, uh, that, that's obviously always something there. Uh, but even companies, right? I, I gotta work with the MSP partner this weekend, unfortunately, it gotta work with 'em this weekend because it was a security incident.

And what was really beautiful is they used, they didn't use the Microsoft vulnerability, but they went and showed like, look, the response that I gave you is better than best of breed at like, it was an example of another, it might've been Adobe and how they responded to their incident, but they actually used one of these enterprise players with a multi-billion dollar company and said, I delivered you service that was above the timeline that these, you know, industry titans are delivering to their clients.

I thought it was super, super coy on their end. Uh, I wouldn't have thought to use a timeline that way, but it was pretty innovative in my end. So I, I think there's multiple sides of that that we could use in this side of the community. Is it these, uh, best practices trickle down to the SMB? Yeah, absolutely. And I, I think, um, you know, we even, we as the vendors owe ourselves to, to the same thing as well, right?

Like if somebody publishes something and says, Hey, I found something in perch, not only taking it seriously, but willing to have that open and honest conversation of here's, you know, how can we engage with you? How can we commit to a timeline of fixing? And then on top of that, how can we work you as the research to give you the props that you deserve for doing that?

Even if it's not a bug bounty, at least at the minimum, it's something like giving them that, that um, uh, that kind of, that street cred, Hey, we got Peter on. And I think that's valuable for us, Peter, as we get over to you because I, we, we should engage and encourage researchers to find the things they find. 'cause it makes us all better, right? Yeah. 110%. But we got Peter on, we got Peter On. We defeated the, uh, the Crowdcast gods. Yeah, we stayed with Thanks guys. I sorry about that.

I had a problem with Chrome for some reason. Uh, Firefox is working. Okay, so no, we now, Alright, well we did it. So Peter, thanks for coming on and, uh, where are you call, where, where are you calling into us from? I, I know you air from South Africa. Correct. And, and I've shared with you that, is that, is that correct? No, I, I'm, Oh, why, why do you say I'm not sure.

Uh, I'm, I'm English myself, but oh my god, I'm, I'm calling from Malta today, so it's about quarter to eight at the moment for me just wrapping up my day with you guys. Oh, it's though, I appreciate you, you jumping on and thanks for having for butchering the background. But, um, how about, you know, in, in, um, interest of time here, Peter, can you share a little bit about, you know, the DNS encryption, the, the Yeah, sure.

A section, uh, so people understand the, I guess in context what it is, what's changing, and then why it's important. And I'm gonna let folks, smart folks like Kyle and Wes banter a little bit with you here. Yeah, Sure. Uh, so DNS is one of the few major protocols, uh, in use on the internet at the moment that is still mostly unencrypted, uh, basically 'cause it goes on in the background, it's not as visible as, uh, things like going to a bank website and seeing an unencrypted uh, connection.

Uh, there's been a couple of solutions or a few solutions put forward. The two main ones is, uh, uh, uh, DNS over TLS and DNS over htt. PS uh, DOT does it via, uh, port 8 53 and it adds an SSL uh, TLS, uh, layer on top.

And H-G-T-P-S, uh, DOH uses reuses, the HTT PS port and the biggest practical differences that they have different ports and do it seems like a, a good solution at first 'cause it's reusing existing protocols that are already widely implemented and it's being pushed by some major vendors like Google and Sili and, uh, CloudFlare. But there's a few issues. Um, the mainly because it's being implemented at the application layer, so it's in the browser or per application.

And the problem with this is that it bypasses any system level configuration. So, uh, you're gonna circumvent any kind of DNS level, uh, protection, like DNS filter, we provide protection against phishing and malware. So we return a, a fake result for, um, any domains that we know are associated with these things. And DOH bypasses that every client is essentially acting like a, it's connected to A VPN. Um, there's ways to turn it off.

Uh, there's thing's like canary domains in Firefox, um, or registry tweaks and things like this and it's being rolled out over, you know, IOS's, uh, apple have just announced it for iOS 14 and Mac OS 11 Windows Insider preview has got it available now. Um, and yeah, it's, it's kind of tricky. The problem is, it's a bit of a moving target because, um, the protocol itself hasn't been finalized. It's still a proposal and, um, the different, everybody has different implementations.

So like, for example, the Canary domains in Firefox, that's kind of a nice way for, uh, DNS providers to preemptively disable it. They, there's a special domain that can be used, which if you return particular results for it, then DOH point be used, but it's not universal and it's not been defined. So, um, yeah. But Peter, so this where maybe I could bring in Kyle first here.

Like how, how is this gonna impact MSPs, will it, and, and Kyle, what are, what are some, some of your thoughts if you Yeah, I, I continue to meet, uh, Ms. P partners on the road, right, a couple thousand a year. And I run into more and more network admins and less and let less and less network architects or less, uh, network engineers.

And where I'm gonna go with is the architecture of your network and all the apps under the hood that depend on it as, uh, you know, Paul mentioned earlier the apps that are in your network that could be resolving things like this when you go and make broad swath changes to either enable or disable. Um, it's something that could almost, uh, cause denial of service level, uh, events. So I think the biggest concern to me overall is actually not on the security side.

Obviously we need to take, uh, some of these benefits of the protocols more serious, but at the same time, like making sure you understand not breaking everything. Because if you do break everything, a lot of the concerns will be who wants that security? It just makes everything not work when the reality is just an architecture problem. Got it. Yeah. Yeah, that's very true.

Um, that, but, and as I was saying, like it's all these, these different implementations so you can change it one place and it won't be reflected on other parts of your network as well. So it'll break for some people in some ways. And fixing it is gonna have is a huge pain in the out. Yeah.

And this is where you see, like on the enterprise side too, Kyle, this kinda goes to your point as well, like we're, we're slow to adopt some of these major changes in the enterprise and there's reasons for it. Like take TLS 1.3 as an example, I know we're getting really technical on this call, but even with something like that, it's been out for a while. There's a lot of concerns around the enterprise have. And so the, there's a slow rollout.

You're seeing very slow adoption of this internally because what are the, the security versus privacy ramification risks. And I love Tim actually pointed this out here in chat, is these are the kinds of things that are good. Like, we shouldn't shy from these things. We should be open and excited about new technology, new changes that are happening.

These are always good things We just gotta think through, realistically, as an enterprise, and I'm including MSPs in this, how do we decide between security and privacy and, and really include both? This is not an either or conversation. This is a both question. We gotta bring 'em both together. Yep. Peter, I'm curious with all the places, um, on, on, on my side of the house, I, I like Tim's comment about, uh, you know, DNS over HTP HT BS right?

Is is great for personal privacy, horrible for enterprise, uh, security me is on the attacker side of the house. I love the idea because it probably means less introspection into my malicious traffic. Yeah, I think that's what we're getting at. Um, you got any words of the whys or how, how to help the audience balance the, that, uh, you know, the delicate trade off that Wes just mentioned?

Well, I mean, DOT is the way to go from my point of view, basically because it enables you to have the same advantages of DOH. So, uh, you can get encrypted, uh, connections to your DNS server, but you don't have to go o over the same port as HGPS. It's not hidden traffic, it doesn't bypass any kind of cloud protection that you've got from DNS filtering companies.

Um, so it's, it's the same benefits, but you don't have this issue of, of uh, um, skipping around any protections that have been put in place by your very responsible network admins. We're just trying to protect you. There was, uh, talking about, um, from an hacker's point of view, do HS was responsible for a piece of malware that came out last year and it used DOH to uh, contact the CNC service. And because it was DOH and it was over H GT PS, it's completely hidden.

So there was no way to detect that that was actually happening. So, so Peter, would you maybe keep us in the loop on what's going on either via blog article or something if, if there are changes that we can keep everybody informed here on what are the changes and, and what are the implications, uh, for the MSP? Yeah, absolutely. Uh, we posted, uh, a blog, uh, something on our blog recently. It's the first of two.

It's a kind of more general overview of DS encryption and the different, uh, different options at the moment. Uh, the second one that's coming up is gonna be a lot more focused on MSPs and how it impacts them and what they can do about it. Um, it's a bit, you guys talk about education a lot on cyber core, which is, you know, it is really what you need, what needs to happen.

It's a bit different from, uh, kind more established, uh, protocols because it's still being sorted out as we're talking about it. But yeah, just try and stay on top of it and, um, we will do our best to, to do that from our plug. Thanks. Thanks for sharing that. And so, so when that comes out, please share it with me and I'll get it out to everybody. Absolutely. Thanks. Okay, thanks Peter. Take care. Thanks Guys. Thanks Peter. Thanks for joining. So, wow, it was a wrap today.

We had no Gary, we had no, Hey os we had no, I'm getting fired up, but closing thoughts from you, us on, uh, on Today. Yeah. You know, uh, closing thoughts. So we're at 1853 on the, we gotta get this to 2000, don't you say. So, uh, here's something I think everybody could really do if you just pop onto LinkedIn and be like, Hey, great cyber call today. Really enjoyed it. Here's some things we got out of it. A great discussion around X, Y, z. Pop that in there.

Make sure people have a link to the cyber call. I think they just need the url, crowdcast io slash e slash cyber call and they can register. We'd love to continue to grow this thing.

So I do want to say that second thing I want to say is I appreciate the conversations around vulnerabilities and how we as MSPs can actually use them internally to, uh, not only ensure that we're doing the right things and we're staying maybe not ahead of the curve, but with the curve, but also how we can communicate these in a rational way to our clients. It's really important. It's something that we need to have experience around because we're gonna get that more and more.

It's not going to be just the regulated clients. And so I think really gearing towards and getting towards knowing how to communicate and who to communicate with, I think it's been a good topic of conversation I've really enjoyed today and I think it's something we all should pursue and have that nailed down. Thanks Wes. Great closing comments and thanks for the, uh, um, the shout out to, to drive more people. Kyle, how about yourself? I mean, we're Gary not here.

I mean, can we, uh, coordinated ha amongst all of us to, to send us outta here? I mean, I feel like we're missing out. Well before we do that, let's, we can close with that. But again, I wanna make sure everybody knows you guys are on tomorrow at one. I'm sure it's gonna be fantastic. I know I will watch that. So how about a 1, 2, 3, and we'll carry out on a ha there. And you guys have a fantastic week. So a 1, 2, 3 ha. Hey, thanks.