Log4J Vulnerability & CIS working with MSPs

Welcome everybody. Just another Monday. Small vulnerability out there. We, this should be really quick, right, Wes? Very quick. Um, so, uh, let me just let you guys know I'm keeping an eye out for Ryan. He should be here shortly, ideally, and a few quick announcements. Okay, so, number one, um, in the green little, uh, call to action, Chris Lahr, Chris Sears, uh, tomorrow incident handling workshop, 1:00 PM Eastern Standard Time. Um, write a boom cyber summit coming up in February.

If you're interested, you can email me Andrew, at the Cyber Nation, putting it in there right now. Um, going quickly here 'cause we have a lot to cover. Wes, I'm gonna hand the floor to you briefly. Obviously there's an incredibly tragic, um, mother Nature event this weekend in Kentucky, and wanted to see if you could chat a little bit about it, what we might be able to do to help. Yeah, so, um, appreciate you giving me just a minute for a platform there.

So, um, I think many of you guys know, I I used to live in, um, Kentucky and worked in Mayfield of all places. And, uh, really, really rough things happened, uh, Friday night. Um, if you haven't seen the damage, I'm posting this into the chat right now, so you can take a peek at it. But, um, you know, they had some, some major tornadoes come through and really just leveled, um, Mayfield and, uh, very sobering to me. I have, um, many family and friends that are there in Mayfield.

Um, I ended up spending all weekend, um, you know, just checking in. Fortunately, my family, uh, direct family is all okay. Um, uh, no lost homes, anything like that, but several kids in my, um, my kids' old school have lost their homes. Um, we, I think the, the death toll is, is still climbing. Um, it's been really rough. Um, and, uh, so yep, it's really been heartbreaking to me personally. Um, I did want to say a shout out though, Andrew. Um, so I'm continually amazed by MSPs.

Um, there's, uh, some amazing folks that are out there, and one person I wanted to give a shout out to, um, Jason Richner, I, he's probably not on today, um, uh, but Jason is from Joplin, Illinois, uh, uh, Joplin, Missouri. And he, he knows, uh, what this is like, and I've been texting with Jason all afternoon. Um, he actually loaded up a whole bunch of stuff, um, really vital stuff and just drove down there.

And I was connecting with tons of people and some churches that are in the area, all this. And, um, man, it just, it, it continually amazes me to see people like that, that just, um, no accolades, just behind the scenes. Um, Jason meant a lot to me, and one of the things I wanted to do is I wanted to, to provide a link. This is Convoy of Hope.

Um, this is part of a, um, charity that Jason and others are involved with, and, and the city of Joplin themselves are very, very, uh, influential with this because they've gone through this kind of event before. Yeah. Um, so if you're interested in looking for a place to give, um, check out Convoy of Hope. Um, additionally, if you're looking for kind of a list, I would say when you start local and you go, um, from there, the impacts are probably a little bit more meaningful.

So this is a link, um, to WVXU. Um, they've listed a whole bunch of charities that are all involved in the recovery. Um, so, um, just wanted to let everyone know thoughts and, and, and prayers and deepest sympathies to my friends in Mayfield. Um, really tragic to see what's happened. Even looking at the bank I used to work at the entire second floor has just been leveled. If you look on my LinkedIn feed, um, you'll see, you can even see where my, my office was. Um, just very surreal, Andrew.

So thanks for letting me take a minute. And just wanted to say thank you specifically to Jason Ner, um, in stronghold data systems, um, for what they do and for everyone that's been involved in, in the recovery. Um, I love that this community, um, these are the things that we're all about. Yeah, it's, uh, you know, of course, I mean, look, this is what this is all about.

And, um, ironically, I reached out to, um, your very good friend, uh, and friend of the cyber call Chris Sanders, um, as well as Drew Drew's on here, Perry, who both been contributors. Chris, uh, I guess knows some folks that didn't make it, unfortunately. And it's just a tragic, tragic thing. So anyway, um, thanks for that, Wes. Appreciate it. Mr. Weeks is coming up onto the stage as we speak there where he is. Thanks for joining us.

Ryan, you probably had nothing going on, uh, in your world, uh, internally there, so, uh, I'm, I'm extremely well rested and feeling really good right now And channeling your inner Eminem. I like It. You know, wait, I was gonna say, he looks like remember the old GI Joes when they had the hats? You know, I'm really dating myself. Alright, so Phyllis with kung with kung fu grip. Yeah, of course, of course. Little Outta order.

Um, Phyllis, let, let's, uh, I think a lot of the folks out there know you, um, know the great things you guys do at CIS, but for those that don't, Phyllis, can you tell a little bit about yourself, your background, and who the heck this group CIS is, and, um, thanks again for always being, uh, willing to come on. Yeah, sure. Um, my name is Phyllis Lee. Um, I work at Center for Internet Security. I've been here, um, almost four years.

Prior to that, I spent 25 years at the National Security Agency. The bulk of my career there, um, working on defense. Um, CIS is the whom of, um, the MS iec Multi-State Information Sharing and Analysis Center that is funded through a cooperative agreement with DHS and is managed by csa. Um, underneath Ms. IAC is also EI iac, the Elections Infrastructure, um, sharing and Analysis Center, as well as, you know, other services, um, that they offer.

Um, uh, my side of the house is security best practices. We are home of the CIS benchmarks or secure configuration guides. And of course, the critical security controls, which fall under me. And most recently, um, cyber, um, nation had me on, uh, so we could talk about controls version H eight, which dropped in May.

And Phyllis is also, along with Ryan and Wes, um, a phenomenal contributor, something we do call the Cyber Cast, which is a free podcast where we, um, every few weeks put out a control, um, and we're up to vulnerability management. We're just trying to get a date to get that knocked out in studio. So, Phyllis, again, thanks for all you do.

Um, quick setting the stage, like if you haven't been living under a rock, everyone I think has heard of Log four J I'm gonna put Jan Easter Lee's, uh, quick post because I, I simply want quote her. And then there's three recommendations she has. But she says, to be clear, this vulnerability poses a severe risk. We will only minimize potential impacts through collaborative efforts between government and private sector. Again, great idea to have Phyllis on because of what they do at the MS isac.

We urge all organizations to join us, uh, in this essential effort to take action. So there's three, um, additional steps she puts in there. I know we're gonna be talking about them today. So, Ryan, let's get right on into it. Um, you know, maybe you could just from your perspective and your words obviously give, give everybody a sense, you know, protecting an organization of about 3000 and then tens of thousands of MSPs. Why the significance of this?

Yeah, I mean, I, I don't even think you need to look at it through that lens. So this is, this is a single packet server, completely compromised vulnerability. That packet can come from anywhere in the world and requires no previous authentication. So it's not like a SQL injection behind a web app that requires authentication. This is like unsolicited. I can send your server a packet and own it.

And so the, the, the single packet nature of this, like, um, one of the ways I've explained this to people is in browsers, there's something called a user agent. It, it actually provides a string that's eventually like a fingerprint of the browser that you're using. Um, you can literally just copy the attack string for this into your user agent of your browser and just start browsing the internet and exploiting servers. You don't have to. It is literally that easy.

So like, it's, this is, in my opinion, we haven't seen anything this serious since Shellshock back in 2014. And, and Ryan, should I put by the way, when, you know, as you start to answer this next question, should I put that gray noise so article in so people can see Yeah, Yeah, Yeah. How easy it is, what you're talking about. Yeah. Yeah, go for it.

I mean, it's, you know, so gray noise is a resource that a lot of people in the security community have been using to visualize the state of, uh, attacks. Um, so largely what I've been doing is I've, I've been, there's three resources I've really been using before. One is the attacks we're observing, um, better mass scanning the internet, including data's, internet leasing infrastructure, and grabbing kind of those payloads.

And doing analysis and seeing what people are trying to do that work is a little more, uh, time consuming, um, than Twitter has just been absolutely a, a phenomenal resource. Like I wish Twitter was what it is now, back when Shellshock happened, for the amount of good information that we've been able to glean out of there. And the gray noise has done a really good job of like putting together a list of all the kind of IP addresses that they've seen attempting, uh, exploit payloads.

And then what you can do is you can literally take those URLs or portions of those URLs and you can go to a resource like URL scan.io, and you can actually look at those payloads and you can see what those payloads were. You can get screenshots of them. Uh, so you can see what they were trying to do. You can see what virus totals, uh, re you know, uh, determination of them was.

And so Saturday morning when I woke up, I grabbed every single payload that we saw overnight, and, uh, you know, also verified no C two connections were made because, like, that's job one, right? Um, but then take those payloads and started doing analysis on them. And, and what I saw immediately was overnight, um, from Friday night, there was maybe some emerging points of coin miners. By Saturday morning, there were multiple coin miners and DDoS botnets using the exploit to recruit hosts.

Um, so this, this escalated very quickly. And then yesterday, Microsoft reported, um, targeted exploitation and lateral movement using the vulnerability. So not just mass scanning now targeted attacks using the vulnerability. So this escalated very quickly, and I, I should note, CloudFlare actually went back through their logs and they, they saw the first attack using this vector December 1st. Hmm. So a week before it even became known.

So this has been out there, but largely we haven't seen a lot of activity, um, until it really became known to the research community. Um, on, like December 3rd or fourth is around when we started seeing initial scan activity from researchers, but it went Friday morning, it went from researchers to malicious actors real quick. Like that bit flipped probably around one o'clock in the afternoon. Um, so if, if you haven't been responding to this, you are really behind the curve.

Ryan, can you speak to why, you know, you feel some folks out there are A, downplaying it, or B, why we're in the early innings of this? Is it, and and regards to the early innings question, is it just, again, people, like, whether it was like exchange, just ignore what's going on. I think there's a little bit of, this one's really complex. It's, it's complicated, right?

You're talking about a, like a, a, a software component that then gets, it's like a library that gets in, imported into software and then gets baked in. And so it can be, depending on how that Java application is written, um, it, it can be actually really complicated to enumerate that, the use of that component because, you know, maybe I import some Java file, just another like Java file from somewhere else, but that JAR file embedded log for J use in that JAR file.

So you're needing to go, this is like a Russian doll. Like you're needing to go layers and layers and layers deep to find where this, this log for J vulnerability is. And I don't know, like, I just, I, you know, I, I had the sinking feeling when I read this. I was like, this is shell shock again for shell shock. We were still finding out about systems that were vulnerable. Oh, you know, new systems that were popping onto the vulnerable list of week later.

So we're on the early innings of this because it's gonna take a while for people to assess, um, the, the vulnerability. I also think we're in the early innings because I think there's like the simple exploitation, which is take the texturing and put it in a header, put it in the URI, put it in a cookie, you know, whatever, and send it and see what happens. That threat actors and researchers are gonna start to look and see, well, how can I send this attack and have it evade detection?

Or is there a nuanced way that I can do this? Um, that, uh, you know, that that has, uh, other unintended side effects or, so the maturity of the exploitation is going to increase over the next day. In fact, just before this call, I was looking at another variant of an exploit kit that got put out, which claims to completely bypass all current pattern matching, uh, capability that exists.

So if you're relying on IPS signatures, WAF rules, you know, anything that does pattern matching in order to determine if you're being attacked and then respond, you should at this point be considering those methods ineffective. And that means your only option right now is to patch. Um, and that, that can be difficult, right? Certain versions of Elasticsearch have, like Elasticsearch, I think six and maybe certain versions of five, I think 5, 6, 4 and higher use Java security manager.

So Elasticsearch itself isn't largely vulnerable post 5, 6 4, but pre 5, 6 4, they don't have the Java security manager. And so those are potentially vulnerable, but like, we didn't know that until Saturday morning. And anybody that uses Elasticsearch is out there trying to determine if they're vulnerable using these scans. And so the exploits are rudimentary, the knowledge of the specific versions or the conditions. So like, this is a really fluid situation.

Like I think anybody that did their initial assessment on Friday and over the weekend and considers themselves done, um, you're looking at this the wrong way. You need to be monitoring for evolutions. Uh, and, and, and you need to be quadruple checking, um, your systems. So one thing I will, I will note here, I don't, I don't even, I haven't had a chance to catch up, but, um, we wrote, um, so Florian Roth is a really phenomenal cyber defender. Um, you can just Google his name.

He has an excellent GitHub page, his tremendous tools out there and available, um, especially for Linux. He wrote a really great, um, set of libraries for Python and for batch to do enumeration. Um, we took those rules, we packaged them into a component for Datto, RMM. So Datto, RMM partners are gonna be able to download that component, run it on any system, and also set it up, uh, continuously to run as a monitor and look for a specific file that'll drop on the server if a detection is found.

So we did that, but we also, um, we took that script and we open sourced it. So it's gonna go up on our GitHub for Windows servers, for Linux servers. We think you should just use what Florian has already put out. Um, I think this tool called Veer, F-E-N-N-I-R.

Um, so yeah, like this is, you know, and remember this software can enter your environment and at any time, um, in the blog post I put out on Saturday, part of the thing we did on Friday was reach out to all of our vendors and say, do you believe that you have exposure to this? Um, we contacted over a hundred vendors. We have two responses. Wow. And I'm sure World Is still trying to figure out the extent to which it is vulnerable.

And to that degree of continuous scanning, um, the CTO and founder of Cyber CNS just messaged me. He said, you can use cyber CNS for free for the upcoming month as an MSP. I put the URL in there he can to download a free trial. And it's, and he gave me that message of this is what it will report on. So to Ryan's point, you guys can do continuous scanning, um, and use that tool at no charge. Yeah. Um, Now I wanna, I wanna make one more comment. Yeah.

Part of the reason that we were able to respond so quickly to this was because of our inventories. Well, you took the, you just killed my next question for you. Sorry. I don't even know the questions. Right. Systems, and this is, this ties back to who are your third parties, your vendor inventories, what servers do you have? What software are they running? Right? Those three inventories. Like, I, I've been so busy, I didn't even get a chance to look at the questions. That's okay.

So, um, you know, if you don't believe me now, like you are feeling the pain right now, or worse, you have a false sense of safety in your environment because your inventories are not up to date. There's a reason this is controlled group one. Yeah. Right. Can Phyllis, can we, can we have you comment on this? 'cause this is, this is your baby on inventory, and is your dog barking or can you come off mute? Off mute. Usually Ryan's dog's going nuts. Yeah, he's, he's due in about 10 Minutes.

Am I coming Off mute? You're off mute. Okay. Can you hear me? Gotcha. Okay. Yes. Um, so definitely your software inventory is crucial here. Um, and, and keeping everything up to date. So for example, if you're on an older version of Java, if you're not up to Java eight now you're gonna update to Java eight. So it's knowing your, what software is running as well as, um, keeping that software up to date. I do wanna make a little bit of a commentary. Someone just wrote SBO m Yes.

So I wanted to comment on that, right? So Biden, in his executive order, talked about having SBO m software bill of materials. And so I think from the consumer side as well as from the supplier side, this is something that, um, we need a little bit of activism behind, right? So, um, I'll say Control 16 application software security really talks about, hey, if you're a supplier, you should need to consider, um, you know, creating an SBU software bill of material.

So there's a SWD tag, which is an ISO standard, which talks about having a manifest so that organizations know what version of, you know, where are the dependencies, what version of Java am I running, et cetera. Um, as well as on the consumer side, we really need to demand that. Ryan just said he, he did outreach to a hundred, a hundred of his suppliers, and he got a response from two, because no one knows what versions, you know, they're dependent on. It is not an easy problem.

It is non-trivial, non-trivial. However, can we as consumers, you know, demand that, can we say how many more of these incidents have to occur be before we say, yeah, you need to provide me a SWD tag? Or I think like the other one is like SPD that folks are using, which is not as adequate and I don't know would help in this situation. But, um, I think there's also some activism that we could do to help with the software, um, asset inventory. Awesome. Wes, any comments so far?

I know you've been patient, Gary, I'm gonna come to you next. Um, see what you might be hearing from your community. But Wes, anything from your side? Oh, we, we haven't made this easy for ourselves, especially not in the s and b space, let alone MSPs, right? Like, I agree that the stage one is what's, you know, what, what are your software inventories? This is why we preach this forever, but it's more like, lemme just give you an example. I just shut my map Minecraft server down this weekend.

It's web exposed. Uh, you should see how many plugins and jar files that thing it installs. Have I ever bothered to look at what is inside of each of those? Heck no. And nor will I, but just as an example, my Minecraft server has un who knows what dependencies involved. Uh, it, it, this compounds far more difficult. In fact, Slagel, I think you said this to me in a chat the Other day. Mine mind what? Uh, my, yeah, my, sorry, Gary. Uh, you're, uh, you're above age there.

I would, I would say, uh, and one thing I wanted to say that Slagel said to me earlier, and I thought this is a really good point, is like, you know, um, it's kind of easy for the stuff I build out on my, on my own, but it's monumental for things that I consume. Uh, boy, this is, um, salient point that was brought up in the, um, Biden, um, executive order, talking about all this, all those months ago. We're, we're seeing this really come full force.

And not only that, but you know, Ja, Jason, you know, he's, he's again just such a great sharer. He was telling, you know, and he's been on talking about this with I think when, when Phyllis has been here too, but just him alone, trying to look at his own internal inventory of tools that slipped through the cracks. I think he said he was up to Jason. Was it 50 some odd tools just alone? You can probably comment in there, Gary, any, what are you hearing? Uh, anything from your community?

Yeah, just the normal rumble that we hear. But I'll tell you as I'm sitting here listening to Ryan, it just feels like, again, we get, I get to see what's happening through this hour a week, a lot of right. And, and then, you know, dealing with, with members. But it, it just feels like we can't now go 30 days, right. Without something else. And this is the second or third time there's been something really, you know, significant.

Ryan, what was interesting about what you were saying, the way I interpret it, what's different about this vulnerability is, um, that it's almost like a race between trying to, I mean, outside of patching everything, you know, trying to understand what, what your, you know, exposure is and bad guys being able to morph around it. So there's work they can do, right? Uh, to be able to be more effective. So that, like you said, scans don't work and, and whatnot. And that's a dangerous situation.

And it's not, that's not the way it is with all, all of the vulnerabilities that we've talked about on here, right? No, I mean, this, the closest thing we've had this year is the exchange on-prem, half medium, uh, where there again, internet exposed, no authentication, drops of war, shell gives control of the server. Like this is the same thing, but the exposure there was exchange on-prem, which is finite.

This is any Java based web application potentially that exists on a Linux server on the internet. Like the Way greater scale. Way bigger scale. I mean, like, it's slightly less than Shellshock. 'cause Shellshock was any Apache server running on the internet, right? This is any Apache server that's feeding a Java based application. So it's a subset, but man, that's still a lot. That's, that's still huge.

Uh, and like, I think a lot of the initial focus too is, you know, and, and I, I came out and I sat on Saturday, our initial assessment of our external facing vulnerability is this, because in one, your internal vulnerability is gonna be entirely different. Um, and that's a second order priority, right? Because first, you know, your, your most risk is anything that's in internet facing.

But Microsoft, again, on Sunday said, active lateral movement, exploitation, internal using this vulnerability. So like you went from cover your external attack service on Friday to, you better have a good strategy and be executing it for internal on Sunday. And we made the decision on Friday to attack both, which normally you would, you would serialize them, it would be external fix, then internal fix.

This one was so serious, you had to do 'em in parallel, because I just knew what we've seen with half and what we've, and just the severity of this was that this was gonna evolve so fast. Um, Brian, Curious, 'cause you mentioned exchange this previously. Are you getting any MSPs asking you this time? I know you put a forward statement in your, your blog out there this weekend, but I'm just curious, are your, are your customer's MSP's asking you About what, Uh, you know, is Datto vulnerable?

Oh yeah, of course. I mean, those, honestly, those questions didn't really start to flow in until Saturday at the earliest Mm-Hmm. Um, largely because we didn't put an announcement out Friday because we were online until four in the morning, Friday night, triple checking everything. And I was not gonna have my team send it all clear until I was positive. Right? Right. Um, and then we got up and wrote the blog the next day and continued. And then same thing, Saturday, Sunday, Monday.

Um, but, um, I lost, I lost the thread of the question. No, I'm just glad to hear 'cause last time you heard nothing, you heard crickets this time. Yeah, I mean, it's, it still isn't where it needs to be, but I will caveat this with everybody is in, is just in a mode of trying to figure out what's going on. Like, how do I enumerate this? How do I scan for it? How do I know if I've been attacked? How do I know if those attacks been successful?

You know, what's the right fix for certain versions if I apply this config flag? Is that enough? Think about the number of people that have applied the config flag but not restarted their services. They think they're not vulnerable anymore. Right? Like there is so many layers of confusion to this. Some people are ripping the j uh, JNDI, uh, live, you know, functionality out of the JAR file, uh, in order to mitigate it. But again, you need a reboot for that.

Or, you know, heaven forbid, the, the people that are running vulnerable versions of elastic and have large shard sizes, they could be looking at weeks to patch this. 'cause you can't fix it without a reboot. And your only strategy then is pattern based prevention, which they're already bypasses for. Like, we're, we're still on. We're, you know, we haven't, we haven't finished singing the National Anthem yet.

He was a very quote, You know, Andrew, what I'm sitting here is I'm listening to Ryan and I'm thinking, okay, this is Ryan telling us, right? You know, there's not many people with as much knowledge and perspective at an m MSSP as Ryan. Can we agree on that? Right? Sure. And, and he has a team and all they do is focus on security. Now I'm an MSP, uh, and I got eight employees, maybe five of them are techs, and I have 30, you know, 20, 30 customers.

Um, I'm just thinking about as Ryan's talking, I'm, if I'm in that spot, how difficult this is for me across all those environments with the resources I have, which are not as deep, which, you know, not as, uh, you know, educated, right? Uh, in terms of we're mature in the security process. It's, it's a, it's a steep hill. It's a concerning hill, right? Yeah.

The only thing goes back to what we're gonna talk about with Phyllis, which is, if at least before this you had done some of the basics, if you had an infor, you, that's probably the only thing, right? That can, that can kind of bring down the, the steep of that o of that curve. Otherwise, it's a tough situation For this one specifically tough, really tough for an MSP. Jason brought up a really good point, Slagel in the chat there.

You know, my, my hypothesis Friday morning was that a lot of the activity we were seeing was security research. But you, there's no way to differentiate researcher versus attacker. So any ex, any, you know, attack string that you get, you have to treat as a potentially malicious attack. And you don't know what that C two is doing unless you're analyzing whatever files it's downloading.

Like, my favorite were the lh, uh, sh and LH two sh files from Saturday, which were dropping the kin, kin kin sing kinin, I can't remember, um, malware kit for crypto mining. Um, and then there was a variant of the Mariah botnet that we saw too. So like, um, you know, my hypothesis, it's a matter of days before ransomware threat actors are including this. Um, right now what's happening is in the ransomware ecosystem, they largely outsource, uh, sourcing of victims to pen testers.

So they hire pen testers, the pen testers go and they find victims. So right now in the underground, all these pen testers are, are gaining access to as many people as they can. And then what they're gonna do, they're gonna go to every single ransomware operator and all of their affiliates, and they're gonna say, I have this access for sale, and they're gonna sell that access to every single ransomware operator. They're gonna get paid multiple times for the same information.

And then now we're in a race, every single ransomware operator is now in a race to act on that access before their competitor does in order to lock in the ransom payment. We're in a very short window here between when these attackers feel they have enough to start monetizing, and the fact that the mitigation is slightly complicated, I think is gonna delay that kind of tidal wave, but that tidal wave is gonna come just like we saw with half in the exchange breach.

Yeah, I, I have to agree with Ryan. Um, I think we've seen, I talked to Bryson in our CRU, we've seen about 56,000 or so hits. 90% of that activity is coming from, uh, Tor 90%. What does that typically tell you in most threat activity and threat landscape kinds of, um, perspectives is a lot of that's reconnaissance a lot of it's spray and pray. A lot of it's gathering this intelligence and gathering this data. So I think he's right.

Um, we have, just so people know, we have seen crypto mining, it's the only thing we've seen. I know Microsoft has reported some, some more broad specific threat activity, but we haven't seen that it's all crypto mining at this point, which is exactly what happened with proxy log on. Um, and, uh, it, it, it, I think the reason for that is the scale.

Uh, and in the, um, when you, when you have, when you have so much presence of vulnerability here, I think that's what you see a lot of this start from a scripted perspective is just mass crypto mining. Um, but then when they see opportunistic targets, that's where you'll see ransomware. So I think Ryan's exactly right. I think it's imminent. We need to treat it as imminent. So let's talk about Ryan West. What, and, and Phyllis, what should an MSP be doing right now?

Like if you had to give some top things what you know, both internally and then customer facing. Yeah. Um, so I, I think you've heard this from data a couple times. I, I think of how we, whenever we respond to something or we do something, we think about it in three rings. The first is yourself, right? The second is your customers, and then the third is the community or their customers. Um, if I'm an MSP, I'm focusing on customer zero. I'm making sure that my house is in order.

I'm, I'm, I'm trying to get answers from all of my vendors. I'm trying to get detailed information about their response. Um, I think I saw a post from slagel this weekend that was like, Hey, just because someone says they're not vulnerable doesn't mean they're not running like extremely out of date versions of Log four J.

And I'll raise my hand and admit, I got a couple environments where, where we weren't, uh, vulnerable because we're running a really old version and like my teams are already working on upgrading, right? But like, there's a, there's a whole nother set of information here that needs to be kind of discussed with vendors. Um, so I, I, I think this is a situation where it needs to be a little bit more than don't worry about it. You're, we assess that you're fine.

Um, like, I think enable gave a good amount of detail. Um, you know, uh, I told you, I told you Dave's an operator. Like he gets it. He's a smart guy. Um, and, um, yeah, so I think, you know, you gotta figure out your tech stack, um, which is tough, especially when you're relying on SaaS services.

So part of your response needs to be reaching out to every single one of your vendor, uh, vendors and asking them for the information they have about their response, their assessments, what they're doing, what they're continuing to do. Like if your vendors consider their response to this effort done, that should be a giant red flag for you. Um, because we are nowhere near done on this. Right? Got it.

And then you turn your Ryan, you know, we keep 'em inventory of software, um, like a tools matrix for our peer members. Yes. And you know, most MSPs, when you say vendors, they have like probably between 35 and 40 vendors now. Yeah. Uh, to some degree. I mean, maybe not all of them have exposure, but, uh, it's pretty big number, right? They, they're getting to the point where they have more, uh, vendors than customers.

Well, Gary, Jason Slagel put in 49 and yeah, you know, they're a good Ms P, but I know there's MSPs on here that are five and 10 times their size. I'm seeing who the chat coming through, maybe not 10 times, but there's, so, I mean, think about how many apps could be out there, you know, and, and again, so anyway, yes. Get the inventory going. ASAP. Yeah. Wes, any thoughts from you on this? I think Ryan answered it pretty clearly there, right?

But first is, is understanding what your exposure is, and that's more nuances we've talked about, right? The second thing is understanding, you know, where can I apply mitigations and patching immediately, both for myself and having a clear picture on exposed vendors and what they're doing.

I did post a list further up in chat on, it's not perfect and it's probably gonna be out of date, but a whole big list of like getting at least a starting point of some of your vendors and where they're at with all of this. Um, you know, even taking a look at, for example, what we've published here, what das published, what ConnectWise is published on, on this, you know, understanding where your vendors are at. I think that's an important piece. Let me just post ours so you have it.

Um, so that's the other place you go. Um, I think you can do some additional things in terms of while you're waiting for patching, um, and you're waiting for updates. There was, let me post this as well. This is highly updated and pretty awesome, is another GitHub repo, um, that has a bunch of like Yara based pattern matching that you can use to some Detection. Yeah, Flo, that stuff is great. Florian stuff is the, that's the gold standard right now.

Florian stuff is the stuff you should be using. That's the stuff we repackaged for your use. Oh, right on. Awesome. Awesome. Yeah. So, so those are some things, Andrew. Yeah. And I was just gonna put in here before I go ask Phyllis, um, uh, John Strand Black Hills information, they refer to Florian a lot. And they did a, um, thanks Tim for throwing that to me for net. Um, they did an emergency meeting on Friday. I put the YouTube in there.

They always, the HIS always has awesome recommendations and, and information as well. So that's in free you to watch. So this another question just popped into my head, right? One of the questions you should ask your vendor is, when did you start your investigation? And this, this came to mind because, um, I have a, a friend of a friend who works in security and he was, uh, he, him and I were talking on Saturday. His company didn't declare an incident until Saturday afternoon. Hmm.

And I think that says a lot about your vendor, about like, how, how close are they to what's happening? How quick are they to respond? Like I think that that in and of itself is a pretty interesting kind of metric. Um, you know, barring like the people in Australia because like, they were sleeping and that's not fair.

But, um, you know, I think, you know, that that was one thing that I was kind of shaking my head going, wow, we were, we were done with our initial assessment by Saturday afternoon. Um, I can't even imagine starting and like the, that was such a crazier environment to be starting your assessment in. So, um, anyway. Got it. Phyllis, any, anything from you? I mean, I think these guys have really said it all.

You know, having that software asset inventory, I think it's, I think it's, um, important to realize, I think everyone on this call realizes it's not possible just to patch everything right away. Um, you, you, you know, there, it's just so ubiquitous as well as, you know, it's gonna be hard to restart these services at times. So, you know, like what Jen Easterly says is minimizing, you know, the effects.

Do you have MSA enabled for all those external facing, um, apps just to make things harder as far as, you know? Do you have controls in place to limit, um, lateral movement? Lateral movement is just common across the board whenever it comes to ransomware. So there are all these other things and that you can put in place making sure you're limiting admin privileges so that hopefully, you know, the code when it executes at some point hits a wall.

It, it's unclear to me, you know, how effective that can be. But, you know, putting in these mitigations could possibly help, um, with, you know, the impact. Um, once you get hit or if you get hit, There's one more thing that you can do, um, egress and network access. So the way the payload works is it sends a command and control server with like a w get request, like, go get this file and run it.

So even if your server receives that request and processes and does the domain lookup, so you'll see the DNS lookup of the C two domain. Mm-Hmm. Um, you, if you, you won't see the subsequent network connection that having that egress network access that says, don't allow this server to download random stuff from the internet is probably by far and away the best compensating control that I've seen.

And like, we actually found, uh, a, uh, an Amazon S3 instance that was resolving the C two domain that we were using for testing, but their egress network access was blocking the C two from the Amazon web services tier. Um, and CloudFlare reported that they had the same thing. They were like, listen, yes, we were vulnerable, but also our container and Kubernetes infrastructure doesn't allow random internet access.

So like, even if we receive these payloads, so then really what you worry about is now you're not worried about some external actor taking over your server. You're worried about some sort of single packet destructive action. Um, 'cause you, the server is still processing that request. It's just not able to make network connection. So you haven't completely mitigated it.

But if you wanna not have the threat actor be able to instruct your server to open a door up for them to come back in through egress network control is a great way to protect servers.

So again, if you're, if you're in a situation where maybe you do have a vulnerable, a customer that runs a vulnerable elastic search instance, and it's gonna take you three weeks to patch it because you're gonna lose data if you go any faster, just slap a rule on that Elasticsearch cluster that says no internet access and you should be fine. Like, at least, well, you won't be fine, right?

But you at least won't have the, the, you know, the, the kind of the most serious flavor of this exploit, uh, knocking on your door. Cool. So let me, you know, well, the good thing is a lot of these questions, we, we've actually, you guys are mind readers. Wes, I was gonna ask you about the CRU. Um, so Wes, let me ask you and then maybe ask you to take, you know, some questions to Phyllis, and I'll let Gary do that also in the remaining time.

But Wes, you've alluded to a fusion center many times, and the idea of one, what if we had one, if it was functioning properly, what would it ideally be doing right now, you know, for MSPs? And, and can you maybe just first start off what it is? Um, and, uh, yeah, go from there. Uh, you're on mute, but I think I can't hear you. Sure am. There we go. So Ryan, I want you to add to this too.

Ryan and I have actually had a lot of discussions around this along with some others that I probably won't name yet, just for sake of, um, uh, just leave it until we're ready. But you know what a Fusion center does? I was exposed to this at fs-isac and the way that their financial services ISAC when I was a banker, and the way that their fusion center worked was, um, you, you're talking about a, a place where you have analysts embedded that all have sort of different, um, interests and stakes.

So we had access into, at the time, uh, DH S'S Ink, which has now become, uh, csa. So the ability for FS-ISAC analysts to be able to have direct communication with the federal government, um, also crossovers into some of the other major ISACs that are out there. And it simply just bridges multiple different parties together to be able to share information, um, much quicker and much more timely to understand what's happening around us. Right.

And so, you know, how could a Fusion center look in the ch in the channel in the MSP space? Well, you know, I think a few things. I think having all of the major vendors together, um, is a good starting point. I think making sure that we have access into MS. isac, for example, is a big piece of this. I think making sure that, um, we do have hook ins and tie-ins through the capabilities that CISA has given us into the federal government is, is important.

So this is gonna become a funding thing, it's gonna become a cost thing, but I think it's vitally important for us to speed of communication and speed of ability to adapt and what's happening in the attack life cycle. Uh, Ryan, you wanna add to that? No, I was just looking at what Cody posted and was gonna say something about that, but I, I, you know, you're, you're spot on as per usual. Yeah. So, so Andrew, I don't know how it's gonna look, right? How, how do we build this, how do we do it?

Just know for those that are on the call, there's a lot of, a lot of collaboration and a lot of discussion that's happening around, around this. Um, so if you're interested in the conversation, let let us know and join in. Um, this is not something that, you know, we shall tell you at Cyber Call how it shall be. It's not how this thing operates. Fair enough. So Wes, um, I have you kind of taken it here for a few questions and then over to Gary.

Gary, if we have time, by the way, I hope, would you put you on the spot? Maybe talk about how you as an MSP, which you've been multiple times, how you might communicate to a customer, you know, um, and, and use it as a sales potential. Um, so, uh, Wes, let me let you take it here with folks. Yeah, Phyllis, um, I wanna ask you a question around, um, Ms. isac, if I might, um, give us some wisdom and experience when this hit the scene. Can you just walk us through a timeline of what Ms.

isac did and is doing and how partners work? Give us your wisdom here that we could glean from this and, and eventually seize in, in the Ms. P space. Yeah, sure. So, um, like you said, the, um, MSIs SAC has, um, direct access to csa, um, as well as to the other ISACs, right? So the MSIs SAC is getting this data as well as they have sensors in front of all the states called Albert sensors.

So they're getting all this telemetry data, they're getting data from csa, they're getting data from the different ISOs, ISACs, and then they try to compile an advisory together to say, for my constituents, who is this relevant to? And then they'll put like, high, medium, of course, it's just high, high, high for this one. Um, they sent out an advisory on, um, Friday around noon.

So what they wanna do is try to, you know, whittle down that, um, noise for the end user so they don't have to read a million advisories. They only have to read one and then figure out what to do as well as, um, in the event that you can't patch, what can you do as compensating controls as we've talked about here. Additionally, um, you know, they have a SOC and they're helping with incident response.

So if organizations need help, they can reach out to the, um, isac, they have a 24 7 shop, as well as, um, they're also doing a pilot with, um, EDR called ESF. And so then they have a close partnership with, um, CrowdStrike. So they could say, let's deploy these signatures here, let's deploy signatures and Albert. So making sure that, um, they're sending out also actionable things to do. So organizations can also detect, um, and, um, hopefully respond accordingly. And so, Ms.

ISAC members, this will be the last thing I'll ask you for clarification. Sure. Then I'll let, um, Gary jump in. But, so Ms. ISAC members have the ability just to sort of follow your cadences and, um, I think that's what we need in the, in the channel, right? We have so many MSPs that don't have the capabilities and, and insights into security and how to respond and what to do.

Um, so you're giving your members sort of a cadence approach of here's what we're looking for, here's what you need to do, here's how you can participate. That's kind of how it all ends up impacting them. Yes, exactly. Um, you know, we are a small nonprofit, even though Ms. IAC is under the ca, you know, we, we have a very strong mission to serve the underserved. It's kind of like our tagline. We've been trying to do something like that.

And so we're very sensitive to the fact that there are those small, medium, you know, we call 'em enterprises to include those government organizations, um, that don't have IT staff, let alone cyber staff. And how is it that we can help those organizations? Got it. I love that, Gary. Yeah, so Phyllis, um, I was talking with Andrew when he was telling me that, uh, Ryan wrote a blog that's gonna be coming out soon.

And essentially, you know, talking about how MSPs and SMBs, one of the things they fall prey to is like, well, I I'm not a high value target, right? Like, I don't have anything that bad guys want. And so maybe if you could, you know, talk to why that logic, you know, doesn't hold up and why it's important to be prepared and have standards in place for every single customer. And this is a good example, this one, and the way that they're, they're hunting. This is a good example of it, right?

So it's kind of indiscriminate. I agree. And, um, what I like to do is always point to data. I have to say many times I don't dazzle people with data and logic, but, you know, um, I think it's important to note the most recent, the 2021 Verizon, Verizon data breach report, um, talked about businesses with less than a thousand employees. Those small mediums as well as businesses with more than a thousand employees, basically are experiencing the same amount and the same types of attacks.

There was like no differentiation when they reported out on it. So they actually reported out on those attacks in the same section. They just said, okay, they're experiencing the same amount of attacks, so we're just going to bundle them in the same section and talk about them in the same way.

Additionally, um, you know, we have seen often, um, if you look at the signaling from federal, the US government as well as from the states, um, everyone is pointing to small medium businesses as the backbone of this country and as the most vulnerable parts of our supply chain, right? In particular, people are also pointing to MSPs as, um, a place where you can, you know, it's kinda like your, the biggest bang for your buck, right?

You're gonna go to an MSP and then you can get, um, you can, um, compromise multiple organizations including that MSP. That's why I love it. Ryan always says, I'm gonna clean up my house first. I'm gonna clean up my house first. Which I think is so important because MSPs in particular are higher targets.

And we're also seeing, you know, quite honestly, I've been working with a lot of cyber insurance organizations and they are terrified, terrified about ransomware, um, with, um, especially the small mediums who don't have a lot of, um, you know, safeguards in place. What we'll see now, not only you see executive orders talking about small medium businesses and supply chain, you'll also see, um, state legislation around that as well.

So is that specifically in when you, when you look at, uh, CIS standards, are these the kind of things that like influence how you look at those, how they get updated, right? Like Yeah, definitely. Um, you know, just so you know, like the CIS um, controls have been pointed to as, um, in a couple safe harbor laws in a few states like Ohio and Connecticut, and Nevada's put has one on the docket to vote on, meaning if you implement CIS controls, then um, you will be protected from being sued.

And so that does shape what we do. It shapes our prioritization. We look at cyber insurance industry who typically set standards if you were to look like for fire and for these other things. Things get regulated implemented based on insurance industry. So you look at them for cyber and you see many of the controls with an IG one, backup, MFA, um, you know, the inventory, et cetera. These are prerequisites to cyber insurance.

Um, and, you know, they are not going to insure you if you do not have certain safeguards in place. Yeah, I, I, you know, I I think it's actually better that cyber insurance is gonna mandate CIS implementation group one than regulation. Um, I think it's, you know, I agree regulation would require for everyone, but I think everyone knows in this ecosystem, you don't wanna get flatfooted, uh, paying a ransom. And so it's a really good like carrot before the stick comes.

So like, I, I think this is the right move and I'm, I'm pumped about it. Uh, likewise. I mean, how, how, how much did we hear Justin rein with, like before even this, you know, the July 4th incident, and he was on many times before that, right? Gary just hammer away about how he does an assessment before, before he ever writes a policy with an MSP. Um, so it's, you know, the, the, it's nice to see this is finally starting to take foothold.

Um, and Gary, you have a special project you've done where you gathered assessments of cyber, uh, insurance companies, haven't you? And, and, Yeah, we did that and, and now we're doing, um, uh, for this quarter next week or tomorrow actually, um, we're doing, uh, taking it a step further. We're doing a workshop just just around the legal aspects. I saw Eric was on here and, you know, it kind of leads to the question you were asking, you know, how do you communicate with your customers?

And the answer is very carefully, that's the answer. And it's not something like, you know, we always say be prepared, have a relationship, understand what you should and shouldn't say, right? And you can't do that and think of that on Friday, Right? Correct. You execute that on Friday. So, um, because you do have to have a process in place on when and how and what you communicate to your customers. Um, and then you need time at that point.

You gotta focus on, you know, mitigation right o of the risk, then a postmortem. And then you have to be able to wrap all that into a conversation with every customer to push forward their understanding of an assumed breach world that they live in and the investments they need to make so that you can prepare them better in the future. Right? Yeah.

It, it, it just reminds me, and I played this last night because someone had asked a question in, in email, and I, I'm, I'm thinking of the four minute segment that you and Ryan, uh, are, are talking, oh, and we had Spencer on the breach attorney and Ryan's hammering away at what should you communicate to your customer? When should you communicate to your customer, you know, with all these things that should be in an IRP, right? Um, instant response plan.

Um, and, uh, I, I can drop that if you guys would like, I can drop that, uh, URL to that four minute, uh, video, um, in, in chat. I can grab that. But, uh, I know this is like the 10th time in the past four months that we have, you know, said this, but for you and also for your customers, how much more as an MSP do you need to see to understand your business model and your costing, your pricing has to change to deal with these? And how much more do your customers have to see?

And if any MSPs at this point are still hanging on to this notion in their head, like, yeah, well my customers just don't want to take my recommendations. They don't. It's, you have to assume responsibility. There's no way they can understand what we just talked about the last hour and be concerned about investing a little bit more in, in, in, into the, into their security maturity. You have to now take responsibility. It's not, write this down, it's not your customer's fault.

And, and so, um, you know, Phyllis, I know we've in in the closing moments here, um, and we'll have you back on to talk about it, but can you talk a little bit about what's new at CIS in terms of csat? So what is csat, the, you know, the CSAT Pro and what is CIS gonna be doing and, and some big news out there for MSPs 'cause typically if you wanted CSAT Pro, you needed to be a Secure Suite member, which was around seven grand. And, you know, it was expensive relative, you know, one time cost.

Um, I shouldn't say it was expensive. Could so bring us up to speed in, in closing moments here would be, it'd be great to hear. Yeah, sure. So, um, CSAT Pro is our on-prem version of the Control Self-assessment tool where organizations can self-assess. Um, it's, it's multi-tenant, so you assess yourself first, and then of course you can assess, you know, organizations that are relying on you. So it's primed really for, for consult, for the consultant use case as well as MSPs.

Um, yeah, you know, I've talked with a lot of organizations and they really are interested in CSAT Pro and CSAT Pro only, and just the control self-assessment tools. So we are offering really for Cyber Nation only at this point. Um, a, um, secure suite membership for the controls only, so for Csap Pro, and I think that's gonna be around 2 99 a month, um, to try to work with MSP pricing.

So if you're interested in that, um, please, please let me know, or I, let me, um, put in Yeah, Throw your, throw your email in there. I if you'll, I'll put an email and our controls email. So, um, Andrew, can I just, um, my last thought today? Well, yeah, go Ryan. Uh, Ryan, awesome. Today. Great job. So great to have you, uh, on these kind of days. But also Phyllis, thank you.

I mean, as, um, Wes said, can't say too much, but just how eager you are to partner, um, with the right intentions so that we can help this community more. Um, we're very appreciative and we hope there's, you know, more good things to come. Yeah. Thank you. I appreciate the opportunity. Yeah, and, and no, I couldn't, I think that's a really great point.

I mean, uh, for everybody out there still, this is kind of monumental that a, that you can now, now, you know, assess all of your customers against CIS, um, and it's in one consolidated platform, um, and that they're willing to take away, um, uh, the, the overall membership costs and just allow you to have the tool set. So thank you for that, Phyllis. That was, that was awesome. Um, alright, so wrapping things up here, um, Wes closing thoughts, uh, comments for anybody?

Hey, we we're not here yet guys. Uh, we got a long way to go. And even that last question from Jennifer really caught my ear. Um, and, and the question earlier on, like, how do we even have this conversation with our clients when it's wrapped up with stuff out of our control? Yeh, those are great questions that we don't have the answers to. We're not perfect here yet, but we're moving forward every single day, every single month, every single year. We're pushing this forward.

There are many vendors that are doing a great job of this. There are many that are far behind. You two are a vendor as an MSP. Keep that in mind, right? So, um, I'm, I love this call. We share this out with friends, um, and we'll see more unravel and we'll be here with you through that journey. Yeah, absolutely. Wes, I put it in the right of boom.

I'd love, you know, again, I'm not publicizing this in a large scale thing, it's filling up rapidly, but if you're an MSP that really wants to take your practice to the next level, we are going to have literally the top, um, it's pretty awesome speakers. And my mc, west Spencer, Ryan, Gary, um, Phyllis, thank you as always for all you do for us. Um, can't say enough of, of the partnership and uh, really cool to see. Um, Gary closing thoughts or comments and or Ryan?

Uh, nope, I think I've said it all. Okay, fantastic. Ryan, Don't forget to spend some time with your family over the holidays. Absolutely. All right, everybody, have a fantastic, uh, rest of your day and we'll see you soon. Take care. Bye now. Bye bye.