ConnectWise Cert Signing Security Concerns – the Real Story

Prince last week. Yeah. Secure. Okay. We are live and we are experiencing just a few technical difficulties. Yeah. Brian, sorry about that. And sorry, everybody, uh, we are running a bit behind. Um, let me make sure I can get everybody in. Um, bear with me all Bob can while You, while you're doing it. Yeah. We'll just take off while you're doing it. So, Phyllis, you were talking, we were chatting a little bit beforehand.

We, why don't we, did you want to talk about, um, IT Nation Secure last week? 'cause that was Oh, no, no. Well, I was not at it. T Nation Secure. I did see Snoop Dogg though over Oh, At Tax eight? Yeah, PAX eight. So, um, you know, I, I, I took a selfie. I mean, I was like in the front row, but I took a selfie with myself and Snoop Dogg, um, and sent it to my kids.

So, you know, it's, It's amazing how these, um, all these events really know their audience because, um, ideation Connect use or always has these, these 80 ish bands and like Snoop Dogg and all of these things. It's, it's, it, it's always like the, these event organizers really focus on, like, know your audience, know the people that are there, get the right people in. Man. Pretty Safe bet With Snoop Dogg, though, he covers such a, everything from my wife down my teenagers.

He covers that entire span. Yes. Yeah. Yeah. There was an eighties band. Well, um, uh, tell me about IT Nation. I thought PAX eight went really well. Um, I gave a two, three talks and it was, um, great to see a lot of MSPs there. A lot of folks interested also, um, in the sessions I attended in CMMC and what it takes to actually become CMMC certified. So they had some MSPs up there. And, um, you know, it was very interesting message.

I don't know how many MSPs on this call, um, are looking to get CMMC certified, but, you know, I think one of the messages that, you know, Lawrence Christiana gave was, you know, it's really difficult. So if you don't have a big customer base or aren't banking on one, then you know, don't do it. So that was like one of my big takeaways. I was like, okay, that's good to know. How was IT Nation or do you have any comments on PAX eight? IT Nation for me was pretty wonderful.

We had, we, I had a four hour session, which was the first time I actually had a session that long. And, um, it was educational about M 365 and how to implement security and, and how to, uh, make sure that the tenant is secure. And I, I, I lost my voice the day before 'cause I was having a terrible cold. I'm still suffering from it just slightly, but I remember halfway through thinking, holy crap, four hours is actually really long. But the, the feedback I've received was absolutely amazing.

As I understand it was one of the most visited sessions at IT Nation Secure and like having a captive audience of, I think, think we had 120 people in the room for four hours is absolutely amazing. I think one or two people left. That's it. Wow. That's great.

So what was your session on specifically how to lock it down or, Um, yeah, how to lock down your tenant, but mostly, um, from the perspective of how an MSP shouldn't accept the defaults, because we often too often make the mistake of taking what Microsoft is saying and saying like, you know what? Good enough. But M 365 has been designed for collaboration, and that means that security takes second place in that case. So there, there's third place, fifth place. There's a lot of balance.

All right. Thanks for joining. Hey. Hey, Patrick. Aaron, there's Jeff apologies. I am not sure that's a first for us. Um, We've had a lot of firsts lately, Andrew, so we're, that's, man, That's, it's a, it's a great theme, huh, Patrick. Sorry. Well played. So, um, well let, let's, um, let me, uh, in interest of time. So thanks for your, everybody watching. Thank you for your patience.

Uh, Patrick, Jeff, thank you for your patience, uh, dealing with the, uh, always fun technology problems that we can have. Um, so, um, you know, let me, I was gonna talk about a few things in, in banter, but in the interest of time, I'm not going to today. Um, let me kind of set the stage. Um, Patrick, Jeff, um, Gary Pika couldn't make it. So Bob Miller is gonna play that, those first questions and, uh, we'll get right on into it.

Um, but let me, let me kind of set things, uh, in, in doing so and introduce everybody. Um, so as always, uh, welcome to the cyber call. First off, happy Fathers, Jay. Um, Patrick, I hope you had a great one. I know you have a few boys. Jeff, do you have, you have kids as well, right? Yeah. Okay. Just the one. Yeah, this one. Awesome. And Kelvin, I never asked you kids as well, or No, no kids. Okay. No kids. No kids. Uh, my wife, my wife and I are called Dinks, as I Recently learned.

What's that? Dual income? No kids, meaning dual income? No kids. They just have a money room in their house and every once in a store around, you know, they're like, yeah, woo, I'm coming to the Netherlands. I'm hanging, I'm hanging with the Lars. This sounds pretty damn good. I got, um, I got one about to graduate high school in two days, and I'm told he doesn't get off the payroll for some time. Oh, count on that Patrick.

Plus, yeah, I just hit the, uh, I just hit the, um, uh, empty nester phase, Patrick. So, yes, it, it, it does same, but they're awesome. Alright, so, alright. So, you know, we have something that, you know, obviously impacted a lot of MSPs. Um, this is a, and I'm really glad Patrick and Jeff came on because a lot of times, you know, you'll hear something like this and immediately I think as knee jerk reaction, we immediately Kelvin go, oh, it's the vendor, right?

I mean, so often these days that's become the fair, the knee jerk reaction, um, with all the issues going on. So I appreciate these guys. Um, coming, coming here and talking to us. Um, the, the recent decision to rotate the, you know, the code signing certificates across, um, automate, um, um, screen Connect and RMM, um, triggered quote unquote security concerns, of course.

Um, you know, and, and this was seemingly started by some independent, um, uh, researcher that started making noise about the way you guys were doing things. Um, you know, it, although, um, you know, you guys went through the process and, you know, went through and updated everything, it seemed obviously would have some ripple effects in the industry, it impacted MSPs, et cetera.

So, um, in order to do that, MSPs obviously had to coordinate, you know, get their, you know, if, um, make sure everything was, uh, working properly, especially if they were, um, I believe Jeff, correct me if I'm wrong on-prem specifically, right? This was for, Yeah, I mean, we, we updated our cloud and on-prem, but, you know, the long tail for us is always making sure we can get in touch with the right people at the MSP or wherever to say, Hey, we gotta go get this done.

And, you know, things go into junk folders or clutter folders, or you got the wrong, or there's 1,001 reasons why that becomes difficult. Absolutely. Absolutely. So, so with that, um, let's do some quick intros. I'll turn it to Bob for questions. But starting off with you, Jeff, this is a first for you. Thanks for, thanks for joining us here. Probably not under the, like, hey, I'd love to come on under this circumstance, but I appreciate you doing it. Yeah.

And, and you know, but by the way, I think it's a hallmark of, of, of quality leadership to address things, um, in the way you guys are doing it. So thanks for doing so. Yeah, no, happy to. Yeah, Jeff Bishop's, a EEP over at product management side of the world at ConnectWise. I've been around here for, I, I don't know, about 10 years, and Andrew and I worked together in the past for a while on different projects and things.

So, um, and then, yeah, I originally was with, uh, one of the founders of Screen Connect back feels like a hundred years ago nowadays. So it's, uh, but yeah, look after the product management here at ConnectWise, along with, uh, Russ Humphreys. So, Awesome. Thanks for joining Jeff. Okay, Patrick, good to see you back. It's been a while. Thanks, Andrew. And we gotta tell you, we gotta do one of these. We don't have an event to talk about.

I can tell you about fun stuff that we're doing here at ConnectWise, but, uh, yeah, chief Information Security officer at ConnectWise, so happy. It's just, just always like being on this, Andrew, you have a, you know, you got a great crew, um, you know, we're, we're happy to, to, to maybe put out some education today. So, yeah, appreciate it, Patrick. And then Kelvin, thanks for filling in as a, a guest co-host.

As always, I, I wanted you on as somebody that, you know, builds code yourself, um, and in a short order, you've got, um, I guess, what is it, north of 8,000 now using sip. So, uh, it's, that's a staggering number in a short period of time, so good for you. But, uh, for those that may not know you, which is hard to believe anyone doesn't, uh, tell us a little about yourself. Um, wow, I, I, I, it's so weird to do these introductions these days.

Um, creator of Cyber Drain, uh, founder of sip, uh, SIP is an M 365 multi-tenant management tool, which still is quite a mouthful. I need to find a solution to say that a bit quicker. But, um, yeah, and, uh, at the same time, an MSP owner, uh, uh, CTO at Lamb Networks a, um, fast growing MSP in the Netherlands, uh, and parts of Europe, uh, we're expanding into Belgium and Germany, um, over time. Um, so I get the luck, I guess, to see everything from both sides of the fence.

Um, I'm very involved with Microsoft and a seven times Microsoft MVP. I'm actually today in Edinburg at a Microsoft event speaking for them. Um, so it's, it's, it's, um, I I'm very lucky that I get to be involved in the vendor side, the Microsoft side, the MSP side.

And because of that, I get, uh, I'm able to reflect on I'm lot of these incidents and it's, it's, this is actually one of those instance where I'm saying that it's, it's almost wasn't necessary, but we'll get into that during the actual questions. And Kelvin is being an, is that why we hear, hear a hint of a Scottish accent from you today is that it Might be, it might be. All right.

Um, and if you can, if it doesn't come up, would you being involved with Microsoft, can you at the right time, just explain the whole Microsoft it JUSTER type of thing, if Jeff might get to it sooner, but if you, if not, will you get to that as well? So, sure. So Bob, let me let you play Gary, since Gary could make it today, and welcome everybody out there again once, uh, out there watching. Thank you for your patience, Mr. Miller. I'll let you kick it off. All right, thank you.

So Jeff, I I think before we get too deep into this, that we're gonna have a bunch of people out here who actually do not actually know what code signing certificates do, what role they play. You wanna try and bring everybody up to, uh, spec on what they really kinda serve as part of the, from a Security. Sure, sure. Happy to. Yeah, and look, and I'm, I'm, you know, a novice, I think one of the jokes, like you stayed at a Holiday Inn Express last night kind of thing.

So that's, that's, I sit on the product management side of the house and dabble in the engineering side. But for co-signing, it's, it's, um, going out to a third party group, you know, a, a certificate authority and saying, Hey, I, I need to be a way to sign my software. So as I deliver it outbound to customers, and the, the biggest thing is agents that go down on the machines is what, where people typically see this. I need to be able to say, this is my code. I signed it.

There's hash, you know, created and public private key exchanges. But it's so that when, when the code gets delivered, you know where it came from and who it came from. And then also, if anybody tries to modify or change that in any way, shape, or form, then that invalidates that certificate. So you can tell if somebody, you know, malicious act or somebody hopped into that code and did something that they shouldn't have.

So it's, it's identification as well as making sure what they get is what we think it should be as it gets delivered outbound. Yeah. And, and I don't think most people know that sometimes they can have an expiration date, right? That's another thing. Yeah, they Do. Which if they didn't, they know now. Yeah. So, yeah. Yeah. So can you, I mean, can you walk us through what kind of prompted ConnectWise to have to rotate certificates?

Because I think that kind of gets to the heart of the question, right? Like, yeah, you had to do that for Screen Connect, automate and RMM. So what triggered that decision other than the public advisor? Yeah, so Andrew kind of keyed in on some of the high level aspects of it. And Patrick please, you know, copy in and correct me where I, I may miss something or forget something.

But, um, around the, you know, around the fourth or fifth of, uh, this month we got a, a notification that a, a researcher had had flagged something, and that our ca came back and said, Hey, looks like maybe there's something going on with your cert. Uh, we're gonna have to revoke it, and you're gonna have to resign everything and get it out the door. So we know how long that takes, especially for like a screen connect where there's tens of thousands of tenants that are out there.

So we gotta go back in and not just redo our build and get the new cert assigned and get it out the door. We gotta get it reinstalled across millions of endpoints and across tens of thousands of instances. Um, and then we were utilizing, which is, you know, not, I shouldn't say it's not a best practice, but probably not the, the smartest thing to do at our scale, which is also have the Sameer attached to our RMM product lines and Screen Connect.

Um, we didn't really know exactly what the problem was at that moment in time. You know, we were told that, uh, something to do with way some of the configuration information was, was being presented. Uh, initially we thought it was tied to like a, an XML file or that was on the machine. So we went in, did a bunch of checks, validations around that xml, we're like, no, we think we're good. We'll figure it out, but we, it's gonna take us three or four days just to get this all redeployed.

And we had a deadline of like, Tuesday. So we're like, all right, well, this is gonna be quick and fast, and we've gotta go get all this, all three of these products rebuilt out the door. So we started moving on Friday, we were able to sit down, I think that's the sixth. And my, from a timeline perspective, we were told like, Hey, uh, uh, you know, about midday our time East Coast, we're like, Hey, we, let's get together with, you know, let's talk, you know, ca ourselves and, and a few others.

And, and then we would say, well, and by the way, we've got the, a report for you. I don't think it was the formal report, but sort of a, a higher level analysis. Patrick, you would know better than me, But, and then I'll keep going on Timeline and I'll back up a little a second too, is we normally, we work with researchers every week, every month, um, all the time. We have a pretty, I'd like to think a pretty mature vulnerability disclosure program.

Shout out to Jason Ferguson and his team who, uh, who really, who kicks some butt and they, they work with researchers when, when either vulnerability or a defect is identified. And those are normally a back and forth for validation. Um, you know, and, and, uh, or, or, or, or lack thereof. And then we work with them on either just public disclosure or private disclosure or cd all that.

What Jeff is highlighting is we didn't, you know, on what Tuesday, we didn't even know why on Friday the researcher report was, was provided to us, but it had been provided directly to the other authorities of, um, so unfortunately we didn't get to have that timeline of back and forth for validation and, and some, and some, some other things that we did. So, So they did, basically did an end around Patrick instead of working with you, for lack of a better word.

I, I, I'm not gonna get, you know, um, I'm not gonna, I don't, I'm not gonna speculate on what their intent was at all. But, um, you know, again, no, No, no, not intent, but I'm just saying they didn't give you the courtesy, like a normally, like if Kelvin's playing White hat, he would come to you though, like, Hey, I see this. Right? Yeah. Normally, Yeah, Sometimes people will submit these types of things very briefly as abuse reports, Andrew, right?

And we do get a lot of abuse reports on Screen Connect, because a lot of time prem Screen Connect instances are popped and used for bad things. Right? Of course, Jeff knows that. So it, you know, that fortunately it wasn't submitted as a, you know, a vulnerability disclosure. So Yeah, to your point, they, they went straight to, to the, to the, to the certificate authorities.

So, By the way, I'm not sure if anybody uses this, just 'cause we mentioned Screen Connect and, and that has have Kelvin have you, have you heard of this one called Simple Help? Just quick tangent, Patrick? Yeah, yeah. Um, actually we have, we have a couple of, uh, uh, tools running well, scripts running on our endpoints using our RMM to make sure that these kind of tools don't get installed because there's so many bad actors. Uh, simple Help is one of them.

Um, there was another one Easy, something that was really popular, um, in, in exploiting workstations. They, they just convince someone to download it and Execute it their own. I'm taking a moment. Yeah, thank you, Kelvin. I'm just taking a moment to give this to everybody out there. To Kelvin's point about, you know, application, um, blacklisting, right? Kelvin, for this point, you're, you're looking for, you know, um, agents that shouldn't be installed.

And again, uh, the, um, you know, um, gosh, I'm drawing a blank on this, on the, on the terminology, but, uh, uh, living off, you know, for lack of a better word, living off the land. But this is ano a new one, Calvin, that I'm not sure if you saw, has another vulnerability right now that's being used and abused with ransomware.

It's called, uh, simple Help For those out there, again, make sure this one is in your, your list to, uh, don't allow installation of if you're not, clients are not running it. Um, it's a, it's an up and coming one. So, sorry, Jeff, go ahead, please. No, no.

Yeah, I think what's really interesting, especially because if you've been a programmer as long as I have since, since the eighties, then certificates have always been, I mean, they, they've been around for a very long time, but it's one of the older technologies that is in the stack, right? And so most people don't actually understand how all those interact with one another. And back when we were using them originally, there wasn't a cybersecurity terminology to be worried about.

So, you know, setting 'em up to be more segmented and, you know, isolated and just like you segmented network, those are all sort of new things that we have to consider as it relates to that older technology. And you know what, if you're hauling freight building software, it's probably not priority one to go back and re-engineer something that's been in place for a long time. So I can understand, you know, um, why you can't, why anybody, right? And so Kelvin, you write software, right?

You know exactly what we're talking about, right? You gotta be able to view that and maintain it. And, um, like you guys did, you know, you were able to kind of mitigate this other than anytime you make a change, coordinating with a million people is a problem. Coordinate with two people is a problem, much less a huge number like you guys have. So, yeah.

So can you, uh, can you, Jeff, can you kind of walk us through what prompted ConnectWise to go ahead and start rotating those code signing certificates? Well, I, I'll finish the very quick on timeline because I think that's, that was most important as we, yes. Actually got to see the, the report, we realized that what we were trying to fix actually wasn't the problem, right? It, it was that, and again, um, try to break it down into the way I can understand it as a product manager.

If I take the, um, the area that we signed, you know, uh, from a co-sign perspective said, yep, this is the code, this is the agent, you're good. Because we were, uh, an on-prem product first and then kind of moved to the cloud. Um, we were a multi-instance type product so that, you know, it could be installed on your servers. On our servers, uh, that required A URL. We had to be able to say, you know, like a relay in a URL, like say, here's how you get back from the agent to us.

And what we did was we passed some of that information along, uh, in the configuration, like a buffer area that, of the file, the agent that we were sending down, um, as far as I know that is not, um, you know, uh, illegal or, or, or we shouldn't be doing anything like it, it was, but potentially not the best practices today in the way you should probably be delivering some of those things. Um, but the, the technology capability was there and it was supported.

And we've been doing it for years and years and years and never had any issues with it. Um, the way the report read out was that if somebody got access to a screen connect tenant, they would be able to go in and modify that configuration and be able to set, set a passing down maybe our URL to the relay or whatever else they were potentially gonna pass down, uh, a link for like malware. Mm-hmm.

And, um, and then if they could get Kelvin or, uh, or Andrew or somebody to install that agent on the machine, then they would be able to bring that in. And because our, uh, our agent was co-signed by us, that it would potentially allow a bank into the system, uh, because of white listing and other things that had been set up.

So they were looking at it as less of like a, a vulnerability the way some of this kind of got portrayed or talked about in, in social media, other places, but more of a, this is a way to get stuff down into the, onto a machine, um, you know, kind of behind the scenes a little bit.

Now, again, that a a had to had their own screen connect tender, have access to yours, and then b talk somebody in somehow to putting an agent on their machine, which with any remote control tool, if you can do one or both of those, you probably already have a, a, a pretty major column on your hands. But I, not questioning whether or not we should do this or fix it, we all sat down and said, okay, let's go back and figure out a way to, to remediate this, uh, and get it out the door.

But, uh, in the middle of all that, we had to go redo all of our, they gave us until Tuesday or, you know, like that next to like the, uh, the thir, well, the, the 10th I think was the actual date. And then they gave us a little bit longer until the 13th before everything finally got completely revoked. Um, so we, we had to go redo all of our builds for all three product lines. We had to move 'em, we were using exportable cert, if anybody's familiar with that concept.

And we had to move to a, a hardware based, uh, an HSM cert for all these, which required us rebuilding the way or modifying the way all of our build processes worked on all three product lines. In the middle of all that, we had two other issues. One is we, I said, we started rolling out the screen connect fix, 'cause we thought it was gonna take us three or four, five days to get it done. And, uh, we got tagged again for two things.

One was, do not go out and keep using the same server on multiple products. 'cause that's gonna get red flagged and we're potentially gonna revoke again. And oh, by the way, we've already seen, you know, another report, uh, that from, uh, of the same type of utilization with that cert two that you just started kind of rolling out the door.

So both certs that we had within, you know, x number of days of each other getting flagged all at the same time, plus the, the requirement that we also separated and segregated all products into their own certs. So, um, we had until the 13th to get the first, uh, part of this all resolved, and now we have until the 20th, 21st, depending on your time zone, uh, to get the, the second aspect of all this resolved.

So we've gone through, we've redeployed all of our clouds, we've got all the builds out for all of our on-prem partners. If any of our automated on-prem partners are listening, please update. Uh, I think we're about 35, 30 8% of our partners have updated so far. So we need the other, uh, you know, roughly 60% of you to get there. So, uh, that's, that's what all we did, and that's how we kind of got to the procedure where we're at. So I'll pause there.

I-J-I-J-I, I just wanted to, to circle back a little bit on the start of your, uh, talk, because that, that's something I really want to emphasize. It's, it's the theoretical application of this, I don't wanna even wanna call it a vulnerability. It is a mis a potential misconfiguration issue, right? That could cause, uh, potential harm. It's, it's, it's not even a vulnerability.

And it's been, um, how did, uh, ConnectWise react internally when they found out like, Hey, um, it, it's not even something that, that is in the realm of possibilities directly, because it generated a lot of work for MSPs to immediately update all of their servers. Uh, as soon as your updates were available, the timeline was very short.

So, so I'm really interested in, in how internally at ConnectWise you guys reacted to, um, finding out that it's not an actual deep vulnerability, but more something that is a potential misconfiguration issue. I'll let Patrick hop in if he wants to here in a second, but I'll, I'll start and go say, look, we, when we saw it and our initial reaction was a little bit like skeptical, like, is this really a vulnerability? Is this really why we, you know, get our cert kinda revoked?

But, um, as we looked through and, and talked about it internally, we said, Hey, we've got to make sure that we hold ourselves to as high of a standard as possible considering our role in this space. And if this is going to be something that's gonna cause problems, uh, and and lead to malicious actors doing things, let's go get it fixed. Um, so skepticism a little bit of like, man, how hard is this gonna be?

Uh, all of my product engineering teams who worked pretty much 24 hours a day for better part, like four or five days, certainly would've preferred to maybe done this as a vulnerability report and, and had weeks or months to go solve it. But, um, so skepticism at first, but then the reality says in, look, this is the situation. It is what it's gonna be, we've gotta go get it fixed and get it done as quickly as possible. So more of a rallying around everything.

And now I'm sitting there, you know, uh, Kevin, you, you developed code and now I'm telling all of them like, Hey, everything that you've told me was gonna take a month, I now know you can do in three days. So we're going to, we're gonna speed up development cycles Easy, Jeff. Yeah, yeah, Yeah. Jeff, unless you wanna start writing code, you may wanna reel that back. They know, I'm kidding. They know. I know.

Jeff and I, um, we had, I mean, definitely, you know, this was, I will say from a, a threat modeling scenario, you know, validation and, and you know, alerts, we'll say like, we'll say, you know, mis not misconfigurations, but tampering with certs, right? Kelvin, you know, you get this right. Um, you know, we're, we're, we're looking to try to identify some, some, you know, some detection gaps, right? Um, around this theme.

This one, I'm not sure I can come up with a detection, uh, you know, either a process or technology that would detect it per se. Um, but, um, you know what, hindsight's 2020 now.

So that's what what we're gonna, we're gonna be doing is really hitting the whiteboard and saying, okay, working with Jess folks, working with David Raport, who, you know, runs all of, uh, product engineering is, hey, how, how can we, uh, how can we shorten this up, uh, from a playbook standpoint, you know, from a revoked cert. We got the playbook down. Now, I'll tell you that.

Um, and that's what you wanna do and learn from these things is, is be able to, uh, you know, shorten your response time as much as possible. This one was a pretty short one. Um, but you know, we're, we're, we're taking a look at, uh, you know, any, any detective or any processes that we could put in place. So, Patrick, quick question for you. You know, like if I look at like the Verizon DBIR this year, you know, and, and they look at, you know, edge device patching, right?

It was like only 54%, you know, even with patches out got remediated. Yep. Yeah. You guys are dealing with something. Again, arguably, and we've, the reason I wanted to bring on Kelvin is, you know, again, MSPs can be very skeptical. Sure. So that's why I have Kelvin on like, going, was this really necessary, probably wasn't a top priority for ConnectWise, but now we've got you guys more.

So I see the bigger problem, you know, you guys stepped up and did the right thing, but now we've got, like Jeff was saying, we've got like 60 some odd percent that we gotta make sure get things updated. That, so I see this more as an operational risk now than anything. And, and can you talk about that? 'cause it's, it's almost like, you know, race against patching the same way, you know, Hey, the patch is there, fix it, and people sit around and don't patch, Right?

Rewind, rewind to, you know, February, March, 2024, right? Um, with an externally facing vulnerable, an externally exploitable screen connect vulnerability. And Jeff, I know our, our patching numbers there weren't the best at first, right? So it took, it took a level of adoption, it took awareness, it took calls like this, it took folks to understand that, listen, you're, you know, there, there's an issue out there. And this, especially for MSPs.

This is your livelihood and this is your end user's livelihood. Uh, and I know all of you do that, right? Um, you know, there's different reasons why folks are a little bit, um, you know, slow. Maybe they're, they're testing things internally. I totally get that. Um, so this is, you know, a a lot of lessons learned, we leverage from crisis communication standpoint, which I can't say enough on these things.

It, it may not be a cyber issue per se, you know, to the core, but it's the heck of, it's not a crisis comms issue. And if you don't have a crisis comms playbook, you know, in place that now we have a chapter for, for this type of thing. And I know I'm, I'm not trying to def deflect here, but that's was such, such an important part of this response as well. So, Yeah. Yeah.

No, it's, it's like, like I said, it's, I, I, like I said, I not to beat a dead horse here, Patrick and and team, but like, this is the thing, you know, you sit there and when you look at, you know, I was literally reading the Verizon DBIR today for something else, and you sit there and go, wait, the majority of edge exploitation vulnerabilities, those patches are available, right? All day long. And so, you know, it says, now we've got this opera. It's, it's, it's frustrating.

And I know Kelvin, I think you share the same thing. Now we've created this fricking operational problem. Like, you know, we, we've, we've created work that I don't know necessarily need to be created. Yeah. The last thing I'll say is, you guys were Patrick, you guys were stuck in between a rock and hard place, right? Because once you're told about it, you gotta make a business decision. You know how we're gonna react to it. And if you don't, then that sends a message.

And if you, if you do, then you got the operational overhead. So I understand that, that it, it is what it is. And you, I think you guys did the exact right thing, given the circumstances you were placed in. But to everyone's point was this, was this juice worth the organ, the entire ecosystem squeeze to, to resolve it to everyone's satisfaction? I think that's where the real debate is, to be Honest. Kelvin, you were, thanks, Bob.

You were gonna say something Kelvin, Um, you and I had a talk and, and I had a talk before this, uh, during the incident. We, we called each other just to chat and catch up. And I already said that at that moment I was super frustrated. Um, I met Patrick Beggs during, um, secure, and we were talking about it. It, it was like, Hey, what's going on? Hey, thanks for not, No problem. Um, I con I contacted some of my friends at Microsoft even to like, talk about this.

Like, Hey, um, what exactly is, is this? And it was, no one knew exactly what it was, except that a security researcher was pretty much bullying everyone to get his will done. And that is not the way that we should be handling security in our industry.

It, it upsets me because you know what's going to happen next time a big issue is actually reported and an executive hears it, he is going to say, well, this might just be a, a storm in a glass of water as we have the Dutch adm and we're going to wait it out a little. And we're, and then we might have a bigger problem. Executives are going to have less trust in security researcher, because this was balloon into so much unneeded work.

It's actually a pretty sad event if, if we look at it that way. Because this could have been resolved in a month, in two months, in three months, without any added risk to MSPs. There was zero added risk. I mean, if you already control the server, you could already deploy ransomware everywhere. If you already controlled the endpoint or controlled what's installed on the endpoint, you could already deploy ransomware there. It wasn't an increased risk.

It's, it's truly a little bit of a sad event. Yeah. Interesting. Bob, do you have any others or do No, no. I'm about to hand it over to Calvin. I finished, we, we covered all our, so, alright, Mr. T. Mr. T, that's a new name for you, Kelvin. I just realized that you're the new Mr T. Oh God. Oh God, please. No. Um, actually, um, we were talking about, um, the, the, uh, involvement of, um, the con the code signing certificate itself.

Has ConnectWise made any other decisions based on this incident on, uh, what long-term improvements they should do and how you manage code signing or how you manage deployment security and all of these things have, have been made any decisions about that?

A a few and, and I I do wanna say like, look, while this was all regrettable and what we hate that all the work that we caused all of our partners and, and everybody using our products, uh, I I will say that the, the different technology companies that we were working with behind the scenes to try to get, you know, extended times to get more feedback and insights and maybe some people that could come in and, um, help us out with some of the questions that we had.

They, they did, they did help us and they helped us a ton on some of that. So, um, won't go into all the company names involved, but I really appreciate everybody who jumped in to help us out. The, um, the changes that we've made. So we had to make a quick change to the way the agent was being built for Screen Connect. So we've, we've removed that configuration information out, separated in its own file.

I'm sure a lot of people aren't, you know, overly excited with the way that deployment is going out the door. But, uh, we had to get something quick in place to, to separate those two. The team is already, I had, uh, some of the product engineering leadership, uh, by my office about a half an hour ago saying, we're already working on it. We think we can solve this. We already have a plan.

Give us another day or, so this is gonna be days, weeks, not weeks and months in order to get something else out the door that creates a better experience. The other thing that we did sit down on Friday, you know, Thursday and Friday this week was look as a remote control because we thought about it as a remote control tool. We need to hold ourselves to a higher standard, uh, in some regards.

Uh, we know that people have, we've seen our builds, you know, like not vulnerabilities, but we've seen people utilizing our software in malicious ways. Um, you know, I remember the first time I saw Screen Connect on Pirates Bay. It, you know, drove me and my old business partners pretty crazy. And we're trying to figure out how, what can we do to stop it?

And, um, this has also opened up a world of opportunities to talk to some people like yourselves and others who are like, Hey, why don't you try this? And hey, this company did that. And, and so we are starting to take actions around the way we do trials. We've done some of this over the years, don't get me wrong. It's not like we've just been sitting around on, you know, and doing nothing. But we're gonna change the way we do trials. We're gonna change the way we do purchasing.

We're going to get more information about the partners who are consuming our product lines. And we're gonna try to hold that level of accountability, both cloud and on-prem. Uh, we've had AI in place for years now. Um, uh, probably not the best name in the world for it, but, uh, we called it Scammer Hammer back in the day. And so we, how do we, uh, how do we try to continue to grow our, uh, ai, um, capability to prevent people from mis, you know, misusing the product.

Uh, so both from a, an operational and a business perspective, um, it, it may cost us a little bit 'cause we may, uh, actually fire some customers here or there 'cause they're not willing to give us a level of insights that we need to make sure the product isn't being maliciously used. Um, but also building in some additional, uh, telemetry data so that we can identify that better and faster that we can pass along to the security experts and out there.

So you'll see a lot of changes over the next, uh, three or four, five months from ConnectWise in all aspects around all of our products, but particularly our trial based products like Screen Connect. Yeah, I'll jump on what Jeff said. It's tight tightening a lot of those, those external controls and, and you know, and how, you know, how those are managed, right?

Uh, from a supply chain standpoint, we have a few and we're still, I'm not gonna do the le I don't, I don't quite have a formalized lessons learned yet 'cause we're still still working through it. But I think there's always gonna be room for in our, in our supply chain kind of, uh, risk identification for, for this type of thing as well. Especially the CER certificate behavior, uh, new products, uh, lemme tell you, it's gonna be something triple checked with any new products, right.

Uh, to ensure integrity. And it's in that they're, you know, we're, we're not falling back on any previous, um, say risk or behavior from a development standpoint. Yeah. Awesome. So, um, I actually have one more question and that's mostly around, um, so we already spoke about how this actually wasn't a real vulnerability, but more around configuration, uh, uh, a potential issue around configuration.

And we know that, um, the certificate revoking was completely preventive, but, um, just so we're absolutely 100% clear and that, that everyone who's watching this and, and rewatching this on Reddit or where, wherever they're, uh, getting this, um, can you clarify if you've seen any evidence that malicious actor obtained any of the older CER code signing certificate or that they like, like it's already clear from our entire conversation. It's just always good to ask a question directly.

Have you seen any evidence of the codes signing certificate Actually being promised? No, we haven't seen any. We, uh, we have not seen any direct evidence of that yet. So, Yeah, I, I'll echo Patrick, but he's the ciso so his voice carries way more weight on this than mine. So, uh, yes I do. Um, yeah, and we, you know, we, we made a public announcement on May 28th about an internal event that we contacted very small number of partners. This had nothing to do with that. Nothing So Good.

Yeah, absolutely. I absolutely love that answer. Just being able to heart saying no just clears up so much for people who are online spreading rumors about what type of incident this actually was. Yeah, I I would, I would not be in this career long if I sat in these platforms and misled you on that. 'cause that's just not how I roll. It's not how ConnectWise operates. Uh, and I know it's not how this community operates. You either be transparent or get the heck out, you know, that's, I love it.

So Good. Kelvin, do you have others? I, I forgot exactly if we got 'em all in or No, I, I think I got them all in and Jeff was very, um, um, expansive on his answers. So he covered a lot of my Question very early. So thank you for that, Jeff. Alright, we'll go over the, yeah, sure. So You said, said like around 40% of your customers have done what they need to do, which is interesting. Um, and surprisingly low.

So have you, you know, have you had much feedback from partners about service disruptions RMM agent failures or integrations breaking because of the certificate change? And then, um, you know, as you did that, how did your support team triage and respond, um, to all the issues? Yeah. Uh, so just a bit of a clarification. All of our cloud-based applications are fully deployed and agents are calling back in and those are, you know, they're a hundred percent deployed. Mm-hmm.

Um, and then, which is a large portion of our, our customer base. Right. Um, and then yes, there are agents that are still offline. I'm sure a lot of 'em are calling in today. Um, and so we're, I'm sure our support team is dealing with some of that right now. As far as integrations go, I'm not aware of any. I actually did, uh, reach out to our support team this morning and ask for what are some of the most common things we're seeing or hearing any sort of patterns.

Still the biggest pattern is, um, just, um, sort of one off, uh, environmental, trying to get something redeployed, having an issue with things like this version of Lennox or something to do with a Mac or a security setting or a, an EDR setting that they're trying to figure out. But most of these have still been one off.

I do expect as probably like everybody else here is that, um, when an agent, um, co-signing is revoked, uh, that the E Ds or the avs, depending on their configurative set up, we're gonna flag these things, which means they get quarantined, which means they can't call back in. And then you've gotta decide how you want to handle that redeploy, reinstall, um, uh, un quarantined for a half a minute, let it call in and get checked in. I, I'm not making any recommendations on any of that.

I'm just saying that we've, we've seen everything from like SECM to Intune, to using other RMMs our RMMs using remote control tools. 'cause while we are ConnectWise, a lot of our partners have, you know, a, a competing RMM, but with Screen Connect or they have, uh, you know, um, uh, you know, our RMM and maybe with a different, uh, remote control tool, a lot of them use Intune.

Um, so there's so many different ways to potentially redeploy these things, and I think everybody's starting to figure out what's their best practice for them, depending on their size and scale and their customer's, uh, uh, requirements from a regulatory and compliance perspective. Excellent. Just outta curiosity, um, what percentage of your customers are, um, on-prem versus cloud? Um, across the board it's a little different.

Like for our automate from an RMM perspective, um, it's, I'm, I'm just trying to do a quick math in my head. It's about 25%, 20 to 25%, and then the other 25%. And then, um, from a Screen Connect perspective, it's actually much smaller. Um, ten eight, uh, it's probably about four to 5%, uh, of screen connect customers that are on-prem. Okay. Cool. All right. Yeah, I'm just kind of curious Yeah. How to do the math in my head. That's a good question.

Well, I'm just always curious like what the percentages are, you know, from also just from a controls perspective, like what, what needs to be addressed, what doesn't need to be addressed. Sure. Um, so, um, we know that a number of MSPs rely on automation platforms. Um, and we briefly talked about, you know, third party, third party integrations. You said they were pretty low.

Um, did any, do you know, does anything stand out that, um, broke unexpectedly because they replied, they relied on certain specific, um, certificate, um, fingerprints and, um, you know, what's your message to MSPs that are managing these things that are so, um, brittle and can, can break so readily? Most of the integration partners that we have utilize the, like one of our agents for deployments or they integrate into the UI or the overall experience. Mm-hmm.

So they're, they're far less, um, reliant upon our agent in terms of their operation other than potentially where we might deploy their software like an EDR or a backup tool where we use our RMM to potentially deploy their agents on their behalf. Um, most are API based, um, integrations where they, they bring in. So, um, so from that regards, not too much from an MSP perspective, I think the, the biggest areas that we're seeing is how do you whitelist these applications?

How do you set up the controls? How are you doing your deployment in your, um, uh, of different agents and different tools, and what are your backups for when these kind of things occur? Hopefully it's not a code signing issue for us or any of the other, I don't wish this on any other vendor out there. I don't care who they are. Um, but if, but you might run into an issue, right? You might run into a technical issue.

Um, I can go back a dozen years ago and I can remember seeing where, you know, uh, agents getting uninstalled, uh, on different product lines. I won't go to which product lines, but I remember seeing it, it was like, well, how do you get these redeployed? Mm-hmm. That was your only method of doing so, you don't wanna run run trucks. You, you know, so having some of those backups is probably the biggest thing that I would recommend to make sure, you know, how would you handle this in a crisis?

Patrick kinda alluded to it earlier, um, crisis comms and crisis processes, and make sure you've got it well documented so you know exactly what you're gonna do. And, um, we're seeing, I, I know one, one individual, one company individually, uh, that tens of thousands of endpoints, tens and tens and tens, you know, short of a hundred thousand, but, uh, big and, um, they got everything redeployed in like an hour or so, so, wow. Yeah. Pretty impressive.

When I saw it, I was like, I want to go back and document and see how he did it a hundred percent, but I was pretty impressed when I heard those numbers. Can I get on my soapbox about the white listing and some best practices, Jeff? Sure.

I'm like, I, you know, obviously when a codes hiring certificate, like, you know, like, like the one we have is a revoked security tools, you know, your EDR tools are designed flag and block stuff, software signed, you know, with an invalid or expired certificate, right? In case it's where it's a critical insurer, some systems are running ConnectWise, screen Connect, remain operational, right? Um, obviously we want the late, you know, the latest version.

Um, you might have to explicitly, you know, instruct these tools to trust that that's executable. But, uh, and we worked, we worked with the EDR, the EDR vendors, um, you know, provided samples for the review. Um, it's, but it's still, you know, it still reinforces the independent verification on Nick, on the, the MSP or the, the end customer's part to ensure the integrity and trustworthiness of the newly signed code.

So as we're working with vendors to complete their new review and update their definitions, these, these false positives will begin to fall off. Um, so, you know, pre-approved, um, you know, around the pre-approved or whitelist news certificates, but it is important to know, when I put this out there, this should be the, you know, the, this all should be done on the thumb print hash or the file Shaw hash, not the entire directory.

Uh, so Kel, Kelvin, if you wanna put that in the, in the, the global chat, uh, just, it's just a friendly reminder that you don't have to do this on the entire directory from a white listing standpoint, just, uh, just the small part. So I know that might be a stating the obvious, but just kind of, um, my, my public service announcement.

So, and again, thanks, thanks to my team and Jeff's team for, you know, really diving deep and on the, the nuances of things that we assume people know, but, uh, you know, getting some feedback that they maybe didn't. So That's awesome. Thanks. You know, and I think we can see in the chat, and you two are here, we appreciate the transparency and you know, really, um, letting everybody know what's going on, what you know, and what's happening in this situation.

I mean, I think you already kind of gave out guidance, um, you know, organizations or the MSPs. Um, do you have any additional guidance you wanna give for, um, MSPs running RMM and remote access tools to, um, you know, handle a, a similar situation or, you know, supply chain that threats or anything?

You know, if something like this were to happen in the future, Always Phyllis, and I'm gonna say things that I know will resonate is please have, please enforce multifactor authentication on all privilege accounts and remote access tools. I'll say that again. MFA if possible on all privilege accounts and, and your arm, arm and tools.

Um, you know, saying this enough, we're, you know, we're, we're always our, our identity journey at ConnectWise is one that will never end from a maturity standpoint. Mm-hmm. Using role-based access controls, you'd be surprised a lot of folks just don't do it to limit access, um, for those, you know, for those specific job responsibilities. Also implementing just in time access if you can, to re reduce your attack surface.

Um, it goes back to what Andrew was saying from supply chain, um, you know, from a vulnerability attack surface, you know, know your critical assets and what, uh, you know, what the, what version you should be on. Which ties back to this, Phyllis, were to your question, about 40% patched, we, we ran this last year was a challenge too. Um, so, you know, really, really, it, it's, it's, it's the basics that are gonna keep folks, you know, up uptime and profitable. It really is.

Um, but I can't say this enough, is, you know, know where those are deployed, how they're configured, where do they sit. Um, and again, you know, like Jeff said earlier, um, for this to be truly exploited, it would require administrative access to that infrastructure or, and to your environment. It is not a, you know, it's not publicly exploitable, but doesn't mean it's not gonna happen. So, right. Patrick, can I just say something real quick because I thought your words are really important.

Um, you know, oftentimes, you know, we're always, a lot of times MSPs are struggling, right? With non-regulated customers about selling things like vt, so services, right? And our job, I feel like is like, you know, doing the utmost we can with within budget reason, right? The best security for, for that organization. And you know, the fact that you mentioned, you know, no understanding where your critical assets are, right? Mm-hmm.

What critical systems processes they support and obviously revenue for MSPs, right? We're talking about critical systems and processes that support revenue and, and you know, it's like those genre, those words about the conversations, right? We as MSPs, and maybe I'll just, you know, ask, you know, you Kelvin, right?

Like that's what, tho that's the kinds of conversations we have to really be driving with customers about critical systems, processes of revenue, not this is the security threat, this is that. 'cause again, it's not the language they speak and I talk about this week in and week out. Yeah. So we actually, um, we, we have a strong security team, a, a large security team at our MSP, um, it's now 14 FTE that is directly involved in security.

And whenever they go to a client, they don't talk to them about, oh, it's this specific CVE that, that you're, you might be suffering from. And, um, you, you might get hacked from. They, they, they just go to them and say like, Hey, um, currently your primary source of revenue, um, is a danger of being hacked with a medium priority, um, or, or a medium priority and impact, uh, potential issue. We need to go solve that.

And sometimes they go to them like, Hey, um, this is, this is something very minor to you. It won't have any impact on your revenue, but if you fix this, we'll be able to, uh, sort of chain things together and resolve things in a way that it will impact how much money you're making. It's, it's, you cannot have an executive conversation with your clients, no matter the size of the MSP, if it is a one person MSP, or if you have, like, like I do a large security team.

If you go to the client and you talk to them about CVEs and, um, potential damage and uh, uh, um, what the exploit actually does, they will, they will not understand anything. You actually have to go to them and explain to them like, Hey guys, this is going to have a direct impact on your line of business if we don't fix it now, if we don't fix it in a week, if we don't fix it in a month.

And if you start saying like, it might cost you a million dollars if this actually happens, they start understanding, oh, wait, I could lose a million dollars if I let, if I don't fix this, let's go take care of it. It's, it's, you have to actually speak their language and so many MSPs still come with technical details, leave those technical details behind. Mm-hmm. Especially when you're talking about a line of business app.

When you're talking about a line of business app, you go to the client and say, your most important application is having an issue, let's fix that. And then also to that internally as well, look at how much money it would cost you if your RMM got hacked because you were not looking at things, or if your, uh, uh, remote, uh, control tool got hacked, or if any other tool that has the ability to run something on your client's machines, what would it cost me if this would happen?

MSPs do not analyze that enough. And the moment that you do start looking at it, you're like, holy s**t, this is horrifying. I will need to react to any incident that happens with this, with the utmost priority instead of leaning back and waiting a couple of weeks to install an update because, you know, it might inconvenience or help this a little while. Mm-hmm. Yeah. Yeah. Well said Calvin. Well said Phyllis, I, I think I might have cut you off from Your request. Oh, no, I mean, it's fine.

I was gonna like go off script because this is the one I is, you know, Patrick, you triggered me when, you know, through your, and I'll ask everyone on the panel, you all deal with end customers, obviously. We're talking about MSPs themselves. What is the barrier to updating your software? Tell me what's the barrier for people to patch, to update, to do these things that week over week? We always say just do it. Like what's, I just need to know.

Tell me like, when you get that feedback, tell me what is it? I mean, I, I totally understand and respect that. I mean, I run a, you know, I run a, we have a, we're a 3,500 person company with, I, I like to think some pretty mature, um, you know, information security capabilities and, you know, we're, we're constantly scanning our environments and our application inventory, right? You know, we get a lot of, we have a lot of automation in place, right? But smaller MSPs don't have that, right?

So I think there is, there probably is, you know, a resource challenge, probably a tool challenge, right? You guys are operating on, on refine, you know, defined budgets, um, and that you don't necessarily have, you know, access or, or mature capabilities in place. But it, um, you know, but it does. So it might take more of a manual process or somebody wearing two or three different hats and that Wednesday they just, they did something else, right?

When they were supposed to, you know, check their asset inventory from a vulnerability standpoint. I totally understand that. Um, so it's just, it's why we're doing these types of things. It's why you guys are out up here, you know, you know, really preaching what you're preaching. But I think it's just, you know, it's a scalable, it's a scaling issue, I think is my best answer. So Patrick, and if I could just jump in behind you again, I'll beat the Verizon DBIR report this year.

Phyllis, remember, like there was real asymmetry, they called it between enterprise and SMB this year, like significant, I don't know if you saw that, Patrick, but it was like enterprises at like 39% versus MSPs at like 86% in terms of breach, uh, ransomware related to breaches, et cetera. So it was, to your point, right? It's re it's not only certainly process, but process and resources that, you know, SMBs are, are lacking.

We've done some incident responses for partners and, you know, I'll be like, Hey, listen, I thought where I'm talking to their security point of contact, I say, Hey, can you bring the IT administrator on the call? And he goes, oh, I'm here. I said, can you talk to who, who does your vulnerability? And I just, he just cuts me off. I'm here and he goes, oh, by the way, I run the budget for the company too. And I'm just like, oh man, I'm, I'm sorry buddy. Um, well, let's, let's work through this.

You know, that's, and that's a real anecdote. That actually is clear. Good. So, I mean, I think about that and I wonder like what we as security professionals can do to help with this. So I'll think on that. Yeah, Phyllis and, and, and kind of rounding things out, there was this other thing I saw today. You know, often we talk about our, you know, we, we, our, our frustration with our clients and this that, and the other.

There was this, um, there's an article, I'll put it in here about, you know, North Carolina and Georgia being disrupted, um, for cyber attacks. And literally the quote was that, you know, the, um, it says that it wasn't until the final month of the prior administration in December, 2024, this is after five years or so of, um, that efforts to secure a comprehensive backup system were initiated like that.

Like this is, you know, government and, and I, and, and again, I know we know, oh, government has limited resources, this, that and the other. But you know, you would, this is not, you know, Phyllis, we talk about it. It's not just enterprise. Mm-hmm. It's not just, it's, you know, MSPs are like, why am I my, my clients? It's, it's as u seemingly more ubiquitous, um, mindset of, you know, head in the sand, um, mentality often that we're dealing with.

Um, so yeah, it's a sign of the times and hopefully we can continue to fight the good fight and Right. Get people to change it. But, you know, I think it speaks to what everyone, you know, at least on the panel knows, is like you can't scare the bejesus outta everybody and expect them to make changes. Like, it just doesn't happen. So it's like that Doesn't work. The research Doesn't work work. Right. We have to motivate them and make the business case, Business case. Yes.

And, and even for governments, they need the business case as well. So a hundred percent, a hundred percent. Well, let me wrap it up there at the top of the hour. First, um, everybody in the audience, thank you so much for coming out. It was awesome having you. If you had a happy father, if you were a father. Happy Father's Day to you all. Um, Jeff, uh, thanks so much for coming on for our first time.

I hope we can have you back on and talk about the, uh, SDLC or things that are more, um, anything more, uh, positive, uh, secure by design with Phyllis Patrick, always good having you back. Thanks for, uh, for doing this in short order. Um, Kelvin, always great to have you, man. Um, I'm looking forward to seeing you next week. Um, gosh, you're putting on some miles. I expect you to be in first class on your way over here with all the, I I always remember he's got that room full of money, so.

Yeah, exactly. Dink, he's a dink. Exactly. Bob, thanks for filling in. Phil is always great to see. And we'll see you guys next, next week. Oh, there's Ash. Good to see you, Ash. Um, take care everybody. Thanks everybody. Cheers. Bye. Bye.