Expert Breach Council & Voice of the MSP (Kelvin, Ray & Kyle) & How to Properly Communicate a Cyber Incident

Welcome back, everybody. It is episode one 14 here on the Cyber Call. I hope you guys all had a fantastic weekend. And as you can see, we have a full Brady Bunch packed, uh, house today. Very unorthodox West Spencer in a not so good audio visual setting there. And I don't mean any disrespect to Jason Slagel, by the way, when I say that. Um, so Wes is en route, uh, he's with Jason. They're doing, uh, some, uh, uh, event together, which is really cool.

Wes, uh, and Jason, we're kind enough to join. I'll set the stage quickly, we'll get right on into it. And, uh, today is definitely a day of lots of chat, uh, as if it wasn't like last week. Um, okay, so quick announcements. Um, please, uh, know I put the write of boom, um, uh, thing in the bottom there. Um, it, uh, we order, we, we, uh, and I'll turn off my dinghy and Donny in a second. Again, love to be there.

Um, all I ask for you guys is if you're interested, if sign up, um, as you guys know, I don't, uh, uh, ask for patronage or any of those kinds of things. The one thing I would ask is if you, if you wanna be around, you know, your peers and many of them in chat and on the, just sign up and come, uh, it'll take a world of stress off me because I'm personally responsible for oil hotel rooms until they get filled. Uh, Kyle, you'll probably know about that very soon as well.

Percent Already, already aware. I'm already aware. All right. So, um, and then tomorrow, um, is a really cool, um, I'm just gonna put it in chat, a really cool webinar. Um, and that is gonna have, um, Chris Lahr, who is in, uh, chat right now, and I think most of you know him. Um, the EVP of solid Security and CFC response, a leading incident response firm, and Reed Wella, uh, fifth Wall.

And we're gonna be talking about how, um, cyber insurance is driving the need more and more for right of boom type services for your MSP specifically, IR plans are showing up over and over and over now, and becoming pretty much a requirement to get proper coverage and premium, uh, amongst a number of other controls as we know.

Okay, let me just get on into it and set the stage here, and then I'll do intros, ask a few questions, and I'm gonna turn it really over to, hopefully to, Wes can come on here. Uh, they're in and out. If not, I'll have to play, uh, Wes Spencer. So, um, okay. So regardless of what was communicated by Kaseya regarding the IT glue event, let's take them on their word. And that out of abundance of a caution out abundance of caution, uh, actions were taken to reset passwords.

Um, we're gonna proceed in this call today that all of that was true. Um, however, it's become clear that the communication was suboptimal and raised a lot of undue concern in the channel. Um, so what we want to do today is use this kind of as a case study to demonstrate critical security changes or potential threat activity.

Uh, and building on that like proper communication as if you know, what we're gonna be talking about, you know, changes in security breach management, if there was a, um, gosh, if there was a security incident breach. Um, but rather hearing from the co-hosts, as you normally do in us an answering and answering a, a bunch of questions, I really wanted to, to make it about the MSPs you all watching. And, um, Spencer, who is I'll introduce shortly.

Um, and again, the intent here, again, I, I'm just gonna stress this, it's not a bashing session. I really want to use this as something for you all to take as in essence, an asset so that in the event, and heaven forbid it does, but in the event this happens to you, um, and you have to communicate, you have some tools in your tool belt that can be affected. Is that fair enough? Kelvin? Let me just do a check-in with you. Is that well articulated? Yeah, absolutely. Absolutely. Okay. Okay.

So, um, by the way, yes, my shirt was, uh, called out on by there, Andrew. This, uh, was my birthday recently, and my daughter got me this, and he's like, how's your shirt, dad? And, uh, I said, I haven't worn it yet, but, um, I will in your honor for the cyber goal. So, um, that's why I'm wearing this and not my usual white. Okay. Um, so let's get into intros and I'm gonna start with the MSPs, and then we will end up with Spencer. Kelvin, welcome back. Always awesome to see you.

Thanks for joining. Thanks for having me. Yeah. Tell us a little about yourself and the refreshing Lyme networks. Exactly. Um, so I own an Ms P in the Netherlands, uh, mid-sized MSPs. Uh, lots of Dutch clients, but also a bunch of international clients. So we're pretty much spread out all over the world these days, or at least our clients are spread out over the world and we're supporting them. Um, we're Lyme networks, Lyme networks is refreshing in it.

So we try to be as refreshing as possible towards our clients. And, uh, just be that, be that breath refresher that everyone needs in it. We understand it's been commoditized years ago, and we just really want to help people in getting to that next stage with their, uh, technology infrastructure. Very cool. Always good to have you join us. And you, you have an awesome open source platform, CIPP.

Um, you can throw something in there about chat if you'd like, uh, in chat about that and what it does. Let's go to Kyle. Kyle, awesome to have you with us. I think this is a first, is that correct? This is my first time and I am super excited to be as a guest. I've been in the chat multiple times, but, uh, this is, this is always fun. Yeah, it's awesome to have you share a little about yourself, Kyle, and your company. So, uh, I am a director of operations at In Telecom Technologies.

Um, we're a, I'd say large, mid to large size MSP. Um, I, I, I forget, uh, I feel like we're still a small company, but, you know, people tell us we're bigger. Um, it's, uh, we are global, um, for staff wise and across the United States for our, uh, clients. Uh, and you know, it's, it's, I also happen to run, uh, MSP geek with several other fantastic admins who are in the chat, probably. Yeah. Um, uh, a free MSP community if anyone wants to come hang out. Yeah. Only 14,000 plus people.

Yeah, just a few small, Small community. Um, all love, all the work you do there and your, you got, it's all volunteer. So thank you for that, Ray. Awesome to have you with us for our first time as well. Tell us a little about yourself, my friend. Yeah. Long time listener, first time guest, but I'm always in the chat on Mondays. Uh, I'm Ray Rossini, uh, most importantly for this call, uh, one of the moderators, uh, of RMSP, uh, the largest MSP community on the planet.

Um, I'm also the CEO and founder of OIT VoIP, where many of you know, I was an MSP before that and just founded MSP News Network in April of this year. So doing a few things, keeping busy. Yeah, It sounds it, right, right. Congrats, thanks for having, and then I know we don't see Wes and Jason, but can we, or can you guys not, uh, come on camera or did we lose them, which is very feasible. Um, let me know in chat guys where you are and I'll try and pull you back.

Um, I think we, I think we lost them, so I'll try and get, uh, Wes back with us and, um, let's just see here. Let me bear, bear with me, guys. Uh, lemme try Jason, one more time. Apologize, everyone, lemme just try to get Jason's legal back. And Wes, uh, can you guys, I keep Popping in and outta chat. Yeah. Can you, when you see them, can you just let me know so I can try and pull them back, uh, for Kyle? Um, keep an eye out there, that would be appreciated. Come on.

All right, thank you, my friend. Okay, so I, although I do want to ask you something first, Kyle, um, Oh, they're back. Oh, they're back. Okay. We're here. We're here. Okay. I'm glad you're here. Okay. Hopefully we get everything perfect this time and, uh, we'll get going. Alright, so Kyle, um, and, and everybody in full transparency, I was in conversations with Kaseya following last week's cyber call. Um, we spoke about having the deer on.

Um, and since then they didn't want to come on or respectfully declined. And, um, they did however, ask that we talk about the FAQ video, um, that talks about what happened with IT Glue, et cetera. And Kyle, I think you just happened upon that, and I'll put it in chat for everybody if you haven't seen it momentarily. Um, I have it. Um, but can you share your perspective, Kyle, again, like you said, you know, you guys, you guys are a decent sized MSP, what, what are your thoughts on it?

And, and then we'll get into some questions here for everybody. Um, yeah, uh, it was, yeah, like an hour ago they released it. Um, it was the entire first part of the video is what should have been in the email that went out to everyone, is my opinion. Um, uh, it felt like he was reading off of a prepared statement, which is fine. Um, and then he went into, uh, frequently asked questions. Um, everything he answered in the frequently asked questions was okay.

Uh, I think a lot of the stuff was drawn out specifically because of the really terrible communication that initially was released. Um, and the non follow up of that communication with additional details, which should have happened as well. Um, the, it's like it released on Saturday morning, and then now we're getting an FAQ and update video.

Um, an email, much more clarification should have gone out much sooner than that, like later Saturday and Sunday, especially considering what would happen. But overall it was okay. Um, more clarification was wanted and it was given, uh, I am satisfied with the, with the response that he gave personally. Okay. Awesome. Um, I'm going to, I'm not gonna spend a lot of time on that again, folks. I did put the link in the chat. Um, if you just look up just a touch, you'll see it.

It's a, it's a, um, Vimeo there for you, and it's about 11 minutes or so, and it'll go over everything you may need to know. Um, okay. So with that, I want to ask, starting with you, Kelvin, um, each of you the, kind of like the following question, what was it about the communication that cause concerns or skepticism for you?

And then once we have some of those, you know, kind of out here, I'd like, Spencer, you kind of being kind of the listening up here for what's being said and then what maybe proper communication should have looked like in a situation like this.

And so, Spencer, just to set the stage for you, and I think you know about it, but for everybody who may not, it glue out of abundance of caution, 10,000 plus MSPs put out a statement that they saw in a massive amount, or, uh, my words, but a lot of activity on the dark web Spencer regarding, uh, increase, uh, attempts to log in, specifically credential stuffing and out of abundance of caution across the board, reset everybody's password enforced.

MFA obviously locked out a number of MSPs on top of it. The MSP obviously is the customer of Kaseya. And then there's something called My Glue Spencer. So if you were a customer of, let's just say Kyle, you would've gotten a notification also about this as Kyle's end customer. So Kyle didn't get a chance to. So does that set the stage enough for you to start Spencer? Okay, perfect. Okay. Kelvin, talk to us from your perspective real quick.

Yeah, so my, my first impression was, this is phishing, and I actually discussed it with some other guys inside of MS speaking like this. This isn't a real email. There were so many signs, it wasn't a real email. The entire email was riddled with typos. Uh, it was sent from marketing@itglue.com. First, the email on Thursday came saying, Hey, we now are mandating MFA for all accounts. And then on Saturday, a follow-up email came, it all pointed to just absolute vagueness.

It, it, it wasn't like I couldn't accept it was real for the first couple of hours because I was just like, okay, come on, someone's just trying to phish it. Glue users found a mailing list somewhere, and it's that, I mean, even in the email itself, the, the, the word compromised was misspelled or at least used in the wrong context. They used comprised instead of compromised. And all these little things that, that just make me feel like, Hey, what the hell is going on here?

So my first impression was no incident happened and we were getting phished. That's how bad the communications actually were at the very start. That's a really interesting perspective. Uh, Kelvin, um, in chat, if you did get the communication, could you just put a Y or n uh, in terms of, was that your initial thoughts as well? And then Spencer, I'm just gonna go one by one. So let's just say, you know, you've done write of Boom Consulting for Kelvin.

You are his breach counsel and you know, for his MSP, what, what's going, what, what do you, how do you interpret this? What's right, what's wrong was email the appropriate mechanism? And I will close my email so I don't keep annoying everybody. I'm sorry, I'm watching the chat box with a lot of why's coming up. Yeah. Um, so I mean, honestly, if I got that email, I would've thrown up too because I mean, just like my spidey senses would've gone off.

'cause you're right, I mean, I didn't see the email, but everything you're describing, um, it just sounds like a Bush League attempt to phish you. Uh, and which I, I mean look, I'm, I'm gonna try not to be, um, what's this? You don't have to be politically correct. I mean, we're, break it down for us. So if I'm, if I, was it glue and I'm kaseya, if I'm gonna Monday morning quarterback and look back Yeah.

Right to start, if I'm sending out an email, I'll tell you what, anytime a client sends out an email when I'm involved, we give it about 50 times over, right? There will not be a typo. There will not be a spelling mistake. They're, well, our word choices are very conscious because as you all now experience vague is one thing, but if I'm misspelling words out to my community out to my customers, that's a huge problem.

Um, and I think if they're playing money, morning quarterback right now, if I'm their breach coach, if I'm their compliance attorney privacy, I would probably strangle somebody. Um, and I think probably somebody got strangled because at least with that one, and coming from marketing, that's a little frightening. I mean, God, a lot of what you just said frightens me.

Um, so I mean, to break it down, the first thing that I would say, right, if I'm looking back, what I would tell them is never have your marketing people send out a compliance email ever again. Two, you should never be having typos in an email. I mean, that's just, that's a phishing email that's just is gonna look like it. Um, and then they gotta be really careful when they start throwing words out, like compromise, right?

Because obviously in the MSP community, your, your spidey senses are gonna kick up a notch at that point. Well, it wasn't just that they used the word compromised. It's that in place of it. It says com comprised, which makes the sentence read as if you normally, if it was spelled correctly, it'd have been you weren't compromised, which makes me feel better. However, misspelling it you weren't comprised Yeah. Changes the entire scope of what I'm looking at. Exactly. Yeah.

I mean, this is just sloppy work and you can't, as everyone on this call knows, right? As MSPs, you guys have so much responsibility downstream to all your clients. So this email, did this email go to your clients as well? Correct? Yeah, Spencer, that's, that's what I was trying to say. So obviously, let's just say Kelvin is a Kaseya client. So he gets the phishing quote quote, he got the communication thinks it's phishing, but not only that, but pretend all his clients are in IT glue. Okay.

Yeah. They also were emailed. So of all the companies in the world that should know better than this, that what scares me about to say an IT glue now, right? And this goes to the incident response plan, being able to one, differentiate between a compromise or an incident. Let's just use the word incident and a breach. I mean, you can call a breach. I wouldn't call it compromise a breach because I think it still gets a little hairy.

I like to say incident, lots of incidents happen every day, not all becoming a breach. But of all the companies out there that should be better prepared, um, a Kaseya and I'm assuming an IT glue, who's a subsidiary of Kaseya? Hold on. So, uh, the, one of the biggest problems we have with incidents is comms, right? So they should have had a much better plan in place for something like this.

And especially that they unfortunately went through a horrific experience a year and a couple months ago where they should know that comms are everything in this. And it's interesting because they were pretty open during that incident, right? With pushing out comms. And so I'm wondering if their wholly, if IT glue just was kind of behind the eight ball because they should, when you have an IR plan, you need to have this built in, right?

You need to, You can leave an air, you wanna see, you wanna see cat, Chris layer is gonna comment on the cat. That's all we can say, I'll say. But, um, you need to have it built in for these specific reasons because one, you, if you have it built in, your marketing's not gonna send it out. Two, they should know that they shouldn't be sending it to your clients, right? Because then that took it from one level to then escalating it to a whole new level.

Because in this community, what there's, I don't know, 5,400 people on here, 5,400 companies, hypothetically, how many clients do each of you have? I'm not saying each of them were with IT glue, but they need to be a little bit more conscientious of that. So the fact that they pushed this out without really thinking through the steps, um, was a huge error I think on their part.

I will say in the video, uh, Nadir addressed the fact that they emailed, uh, my glue and light users that that was a mistake on their part. They were That's very true, Kyle. Yeah, true. I think that was stand up the way they did that. But If you look at the entire video itself, um, just, just the content of the video should have been the very first email, just like the first 15 minutes of it, 15 minutes of it should have been the very first email they've sent.

It was well articulated exactly, uh, with reasoning behind why specific choices were made and not like this weird two, two phase plan where first on a random first day they enforce MFA for everyone. And then on a Saturday they, they send out an announcement, Hey, we reset all of your passwords. I mean, that stuff causes a panic. That was, that was the other thing I was thinking about when you, so when I've had clients in a similar situation where passwords are potentially a compromise, right?

What I have had them do is go out to the community and say, look, we don't see, we don't know of an actual compromise, but we're gonna be safe. So we are letting you know we're gonna reset your passwords, right? And so this is actually legitimate. If you have concerns, call us or, you know, write us rather than for you all. It sounds like they just literally did a forced reset and force enforced MFA right across the board without kind of communicating that with you.

And that, that was my, my initial concern was that my, my first question was, was this a prepared statement as part of a prepared plan of action for when you have to do something? Or was this just completely off the cuff? Those emails Saturday and Friday definitely felt like they were just off the cuff. That was somebody we need to get something out real quick. We already did the action and now we need to tell everybody what's going on.

Um, because we saw, uh, Kelvin remind me, we did see reports of the password resets happening before the emails went out, correct? Yeah, Exactly. Probably because the email marketing tool was still sending out the emails at the same time as the reset was happening, something like that. So some people didn't even receive the email yet.

And, and that's a worrying thing too, you know, it's, it's, they send out in their email, oh, and if you're using your password anywhere else, you should reset it, which immediately made my mind jump to, oh, hey wait, they, they have the, their, their da uh, password or user database got leaked or something. They did say it because they were worried about credential stuffing working and then that password being used elsewhere. So it, it was just very badly communicated poorly.

But that could have been, that could have been done so much better. How much differently would've been received if the message said in an abundance of caution, as per best practice, do not reuse your passwords. Yeah. If you are sharing your IT glue password, go ahead and reset it anywhere else just to be safe. Exactly. That, that eliminates panic. And you need to give the choice out there, right?

If there really wasn't a compromise, rather than doing the forced password reset, you can go out to people and say, look, we don't have, this is Adam an abundance of question. We're recommending you reset your passwords. We're not gonna force you to do that. Because when you do a forced password reset, it does take it up a notch.

Um, and I think there's just like a comedy of errors that happened here, which is a huge aspect of incident response where you don't have a plan or your plan is so lacking that we're now on a call with, I don't know, 5,500 people discussing how they screwed this up when if they had a good plan in place and knew how to do these comms and knew your escalation procedures, right?

So if this wasn't a breach and this wasn't a compromise, it was just an incident or they suspected an incident, there's just different ways to communicate each level. And you have to be really careful because when you're this vague, I mean, the fact that they had this many typos in an email, that's, that's pretty, That that was actually one of my original questions. Uh, Spencer, you're, you're best to answer this.

Um, when you're creating the list of comms that are gonna go out, do you already in advance expect to have to have an update of those comms later on? Like, are you planning that out? Event, event plus 24 hours event plus 72 hours? What we try to do, so the best case scenario is that I meet a client beforehand, right?

And we basically will map out comms for you for all these incidents because then the hope is when an incident happens, you call us, then we immediately push something out, you know, between something like this to an actual breach to a compromise. We don't know it's a breach. Um, obviously it gets updated somewhat based on the facts, but you want to at least one have that meet done and approved.

'cause if you imagine when somebody doesn't, when we draft their comms, they have to go through 15 levels and all this red tape to get approved, which then extends the time. Um, and then we wanna know what channels we're pushing it out to, right? So here they pushed it out to you all and your clients, which is just grossly inappropriate. Um, they shouldn't, I don't know what where in their minds they think that they should be discussing anything with your clients, but neither here nor there.

It brings a question as to why is the information in their marketing database to begin with. I mean, yeah, that's is a good question. I think it just emailed all users that were in accounts, which included the my glue and light account users. I think that's, they Just, but your blanketed it, but to your clients have that relationship where they know who, um, IT glue is Not always. No, no, No.

And are they, and, and this is my question, Calvin, could you've kind of raised this, have, did, did when they set up a My MILU account, have they opted in in essence for GDPR compliance? Like how does, Is there any, okay, so, so, so GDPR is is actually pretty simple. As soon as anything contains personal information of a European citizen, you have to be GDPR compliance. So you have to apply the right to be forgotten. You have to apply, uh, specific data protection acts.

You have to store data specific locations if you agreed on that with your client. So there's a lot of rules around that. But, um, clients themselves do not opt in for GDPR or opt out, it just applies. As long as you're a European citizen and your personal data is saved somewhere, it applies. But personal data is not always immediately all data, for example, an email address could be personal data if it contains the, uh, full first name and last name of a person.

But that doesn't mean that it also is immediately something that, that, um, you apply the full GDPR, uh, compliance or at least the, the, the framework to. So there, there's, there's, there's some, some caveats there. You can't just say like, okay, hey, um, just because a user ticked this box, you have to apply GDPR. It, it doesn't really work that way. I would suggest that every MSP on the call that works with IT glue and they're gonna hate me. I don't work with IT glue, nor do I know them.

I would go back to them after this and say, we want be more control over your ability to contact our client. And I would start asking some pointed questions about the technical safeguards they have in place. Protecting the client information, right? Because an email address, I mean, uh, it depends if they've got an email and password for it glue for your clients that could implicate different state laws. 'cause the state laws are very vague, right?

They say an email address and password that allows access to an account. Most likely the legislatures that did this wanted it for financial, but it's not defined. Um, I would just, and as a community, you're gonna have a lot more leverage now in going back to them rather than if it was a one-off.

Um, I would go back and start renegotiating some terms and really pushing them because I would be concerned the fact that they sent this out and they were so, it, it's a concern of mine hearing this, right? If I was your attorney, what I would tell you is that we need to go renegotiate terms and we need a lot more tight contractual provisions. And I wanna know how they're handling our client data.

'cause the clients are gonna blame you at the, if something bad should occur with their data with it glue. That makes sense. I see Jason in the slack, he's asked a couple times and a couple people were throwing around the words incident compromise breach. Spencer, from, like, from your perspective, how do you define the, because obviously they have different responses, right? The different way you, you go out and communicate. How are you defining those when you speak to your clients?

And Ray, can I just throw in there before Spencer answer? Spencer, let's just, again, let's just assume there's nothing, nothing aside from the communication. Let's assume all is good. Just can you also maybe just start up a level with change management around security? And do you do, can the comms first or do you do the change?

Like can we start there and then kind of maybe work all the way down to breach being like the, you know, you know, So when, when do you, when do you go out with the security? When you're having a security incident? No, no, I'm saying even there's a, let's talk with even change management. Like here, let's, again, nothing happened bad. Let's assume nothing happened bad, but we have a secure, we have, we're gonna change something in our security, um, control within it.

Glue of how, you know, instantaneously password reset. So what I'm saying is let's start at the highest level. There's nothing wrong, quote unquote. Got It. Okay. But we have to Saying, Yeah, I think in terms of an organization when this is happening, the first thing to do, right?

When you've determined it's not a breach or not a compromise, you just have a straight incident and you want to do the password reset, I think you need to go out and you need to frame it as such to the community, right? When you're, especially, you're asking them to take an affirmative action. Um, I see I'm getting blown up in the comments about me saying to renegotiate with it glue. I forgot it glue say Exactly. I'll Stand by my comment though. You guys have a powerful But beyond the point.

Um, the first thing you gotta do, right? Was when you're going out, you need to, if you're asking for a affirmative action, you're gonna get pushback. 'cause you're gonna say, well, why do we need to do this? So you need to basically explain it and put it in a light to make sure when you're transparent, you're upfront and you're not freaking people out. So here I probably would've said very much, clearly we have no evidence of a compromise of this death.

However, as you know, security risks are ongoing as such. We saw, I would wordsmith this a little better than on the fly. Uh, we saw activity on the dark web, um, that raised some concerns. However, once again, no indication that anything was compromised on our networks. We've checked X, Y, and Z, we've done scans, whatever it is, the technical side. However, once again, out of an abundance of caution, we are asking all users to reset their passwords.

And, you know, 'cause at that point, I think asking for a reset password reset, rather than doing the force one, the force one makes me feel very much uncomfortable rather than asking people to do it. So I, I spoke, I spoke to Nadira a while, uh, a couple of days back for the incident, and he was able to disclose that the reason they reset the passwords was also because they increased their password requirements.

So I guess the communication should have just been, hey, we've increased our password requirements and because of that you have to reset your password or set up a new password, something like that. Because they went from uh, eight character minimum to a 10 character minimum. So I understand the, the reasoning behind it. Yeah. But you don't wanna just have that be the reason for, you know, resetting it because you're getting, uh, a ton of password logins, right? Spray. Yeah, yeah, yeah, exactly.

So include that 100%. Yeah. And hey, as well as taking this precaution of resetting your password, um, we also have this, uh, we also increase the requirements for you to put in your new password. So be aware of those as well. I think that would've been helpful. Yeah. Yeah. Um, and I think the, the biggest issue, right, beyond all the type is beyond the miscommunication aspect. The forced password reset to me automatically means there's something wrong, right?

That automat, all I'm thinking about like is Mimi Cats and the ral harvesting credential stuff and whatever it is. If I'm forcing people to do that, then immediately, especially with a community like this that is pretty well tuned to these things, that's an issue, right? So if there wasn't an actual security compromise, it wasn't an actual breach, then they really should have walked this back pretty quickly, or not even got this far ahead of it and done a reset.

I think saying to the community, we've seen this no compromise. We're changing our password requirements to make it more secure. Um, as such, everybody's gonna need to reset their passwords. I mean, that's the thing, right? It's, it's examining all the potential questions that could come out afterward. Trying to answer them upfront. I know, and I know it's a losing proposition, but there's some basic ones. Why did you do this?

What we're doing, how it's gonna affect you, our plans going forward? I mean, any comms should have those things in place. Yeah. Spencer, so in hindsight, colleague, go, go one second. So just to clarify, Spencer and something like this, again, assume nothing again, nefarious happened. You want to give the MSP the customer that choice. You're explaining the whys behind what you're seeing. You're explaining our policy change from eight characters to 10.

We're enforcing MFA, whatever, whatever, whatever. But then I'm empowering Kelvin, I'm empowering Kyle to do or not do that. And then by a certain period of time, has that action not been taken, then we're gonna force it. Is that Yeah, like in this situation, right?

If we're changing our policy and we're saying, look, we're gonna change this in 30 days, please communicate it downstream because pastors are gonna get reset and changed as long as you're laying out everything and making sure, because we wanna look, once again, if I'm taking 'em at face value, nothing actually happened. I still wanna feel secure. I still wanna make sure that they are securing their environment to make me feel good about it.

So then if they lay that out and then come to you saying, look, we're changing the requirements because of changing security environment, whatever, it's, you got 30 days and then we're gonna do it. You know? 'cause that gives some time for you to get to your clients to explain that this is gonna get pushed out. And it at least gives, um, it glue, I guess in and k say as some buffer to kind of answer those questions within those 30 days.

'cause they come back and they said, okay, well just what you asked the manager, did you have a breach? No. Right? And then this is what we saw. And they could explain it all in that initial email. This is what we saw, here's what we're doing, here's what we recommend and here's what we're gonna require in the next 30 days. Cool. Kyle, you were gonna say something? Apologize. That's fine. Um, I think there's two things that we haven't really brought up that we probably should.

Uh, the first one is we don't trust it glue or kaseya, uh, as a, as a branch off of that. Um, and that's why everyone is so panicked, um, off of this, uh, bad communication. Uh, the second thing is they're sent a follow-up email asking us to look and see if our have, if we had any nefarious activity in our login details, um, which I think is, uh, more worrisome that they can't look at that data or asking, putting that on us for some reason, um, than anything else. Got it.

So, so how do you feel about that, Spencer? That they're asking Kyle to go through his logs to see if there's anything nefarious That's really suspicious, Right? That's what I was saying. I mean, once again, and I'm trying to be respectful of it, glue and KSA and believe what they're saying, but Yeah. And let's do that. So let's just say, but that act, let, let's begin. We're gonna go with the mindset, everything's fine here.

But what, but by doing that, what is, again, 'cause what I wanna do is say, let's pretend Kelvin is the MSP. Bear with me, Calvin Kelvin's, Kyle's MSP, and he says, Hey Kyle, we did this. Can you take a look at your logs? Like where, so take us through that. Uh, and this was a communication they sent out to you all basically saying they want you to look at your own log. Yep. An email. It, it's a follow up basically to the initial communication.

So the question then is, what would I have done instead of asking that question? It it, a, is it, is it warranted knowing and, and Kyle, is it, is it a cloud instance? Yeah, it's all cloud it. So it's all cloud. So it's their infrastructure. Now they're asking Kyle to look through his logs to see, first of all, it's obviously, you know, is he in, he's, they're assuming I'm Kyle's ingesting internet facing APIs and logs to pull it in. Fair, Kyle, that they're, they're taking that assumption.

He's doing it. And then so again, taking this outside of Kase, just in general, a supplier, a vendor of ours telling this and then taking it downstream, right? 'cause it, this is about us being better communicators. So just kind of walk us through your thoughts here. Oof. Yeah, that's tough because I mean, if you're asking someone to look at the logs and look into it to me, then, then you've, you've gone from a nothing burger to a potential burger, right?

So now we kind of escalated up from the first step. 'cause the first question is, we're gonna do a password reset. Okay, fine, I can get past that. But then when we're talking about the logging aspect, um, when you're discussing this with clients, I mean, if we're discussing with an MSB, you all are knowledgeable and you're educated and you're mature. So you understand that. But if you came to a client, they're not gonna get that. They're also gonna expect you to be able to do that for them.

Um, so I think it's you all looking at your client logs if you were in that position. Um, Well let's take a, this, lemme just interrupt. Let's just say Kelvin's got a co-managed customer with an IT staff and he says to the IT staff, Hey, would you take a look at your logs? We think, you know, nothing's happened, but can you start investigating your logs? I think it opens the door. I mean it, that would raise, Hmm. So how do effectively communicate? That would be an interesting conversation.

Um, it's almost like when you have a phishing email, right? And you click it and then it gets sent to a customer and it's talking to that customer basically saying, don't click this. If you click it, talk to your internal it. And then if the internal IT does call you for external MSB calls, you know them saying, well, what's going on? Um, you could just, This is a tough question. Yeah.

Because the second you've escalated like this to me as an incident response attorney, it's all the hairs on the back of my neck would go up and be like, hold on, let's take a step back and figure out why are we even looking at that? Because the question I'd have, right, okay, wait, I'm still gonna take 'em at their face value. So I would, I don't even know why we'd get there though. That's where, that's the problem with scene Andrews.

Like if there was nothing wrong to start, right, and it's a nothing burger, but we've reset the passwords out of an abundance of caution. Okay. There shouldn't be that next step then. I mean, unless you're talking about best practices of going through your logs, constantly looking for threats, um, I mean, I guess you could be doing that. Right? Right, right, right. And did Kyle, did they suggest any, like IOCs or any particular things Go?

No, they just said that, they just said like, look in your logs. And I was like, okay, what does that even mean? What does that mean? What, what do you want me to look at? You know, and I that go to your clients as well? No, no, no. That went just to it Glue, uh, clients. Got it. So the, my, the, my glue users got a separate email and it glue users got an email, uh, that, that had a little bit of different information in it, but pretty much the same stuff.

But yeah, it's, it's, it's worrying that you get an email from a, from a vendor like that, that's supposed to be able to access all these, uh, audit logs. You know, they're supposed to be able to just say like, these people, they, uh, they got breached. They have inside, they have an incident, however we call it, their, their credential stuffing didn't work and on these it didn't.

And it's, it's kind of worrying that they're telling us like, Hey, make sure you log in and check yourself if you haven't been breached it. Why? Yeah. Yeah. I mean, so For them, what I would've told them to do, right? If they wanted to go above and beyond, I probably would've done all this together. I would've said, look, here's the best practice. We're gonna change password policy. You got 30 days to do that.

You know, we also recommend as a security best practice, this would be so hard to massage you over as a message. But it would be, if I, Well, it's, it's phrasing. Like if, if Theta said in the initial email blast, Hey, look, if you are worried and you want to verify yourself before we can do a full audit, here's how to check your own personal logs. Not, hey, you should look at your logs for anything suspicious. And the sequence of events is problematic, right?

So if I, if I'm gonna play Monday morning quarterback then, and we're gonna believe there was no compromise or breach, I would've then done everything in one blast. I would've said change, we're gonna do password reset. Also, you know, wordsmith, if you don't believe us, I got it. You know, check out your logs, you know, just to verify, um, in a more, much more soft kind of way. But I wouldn't have done, uh, corrective active.

I, I think doing multiple emails, unless you're giving sub, uh, substative updates, is gonna cause problems like here, like clearly doing that first email, that's problematic. And then they followed it up with another kind of critical mistake by saying to do something that was even more alarming. So to me, it's like, if you're gonna get a message out, right, and you're on the fly, get one good one out rather than a bunch of little s****y ones. Excuse my language.

Um, because the problem is, as you see, you all are now highlighting you're compounding, right? You're just stacking issues on top of issues. So when you're getting a message out, if you're in this situation and you do wanna push this out like this, you wanna make it concise, clear, and only do one. 'cause if you imagine humans instinctually, if you keep throwing something in front of us, we're gonna keep looking at it, right?

So transparent up front, one message, and then substantive down the road. So, Spencer, Kyle, did you wanna say something? Well, I think, you know, another thing that's predates all of this is like two days before they sent an email out, forcing everyone to have MFA, that's, I believe technically the starting point for this entire situation Was, so They, everyone has to have MFA two days later it's Saturday, boom, reset everybody's passwords. Got it.

So let, let's, let's shift into more like how we help you all watching. And I appreciate all the chat and communications. Spencer, can we just take just a step back, talk to us about your role as a breach attorney. Not to, you know, 'cause we don't have a ton of time, but I would, you know, again, for, for all those people, but why is it important as we're seeing here? Like, most people would think, okay, I got an IR plan, I've tested my r plan.

Again, most people don't do it under duress and heavy comms and you know, that's the ideal situation. But can you talk to us about, you know, right, of boom planning Mm-hmm. In that, why would I don't have anything wrong? Why am I gonna bring you in and start to talk about these things as we're seeing? Yeah, it's a great question. So basically my role is twofold. It's right of boom and left of boom. I like, I love getting calls left at boom.

And don't get me wrong, I, I've done five or 600 breaches. My firm's done like 11,000 breaches, not even incidents. So we're talking that have escalated up. Um, but why it's important to know someone like me before an incident is exactly this, right? So I'm sure with all your IR plans, and I've worked with a bunch of MSPs and I've seen your IR plans, you all are very technologically savvy, right?

You know how to escalate on the technical side, but then the question then becomes what happens when you need to communicate? Um, and it's a really rapid transition. And what I've found a lot of times when I get brought, get brought in say a day in eight hours in very quick time, right? It's not like a month later, communications have already gone out. And so it's really important to know someone like me, one, to get us baked into your lives. And there's a lot of breach.

Well, there's like seven or eight breach coaches, or I'm sorry, attorney firms out there. They're on all the insurance panels, right? So go vet a couple. So beyond me being self-serving, what we'll do though beforehand is we'll get to know everything about you. So then we can actually build out your comms plan. Um, you can work with a PR firm, um, breach attorneys. Were pretty good with comms though, I will say.

Um, I've worked with PR firms though, uh, it it's just very important that you understand when you get past the technical, right? If you think about an incident, how does an incident really flow? What are the important pieces? Incident responders, forensics DFIR to get in first to get you button up secure and start the forensics that flows right into the legal portion, right? Because then at that point, we need to be able to help with comms notifications obligations.

Same thing with an IR plan left of boom, you're gonna start by building out your technical portion, but then you also need to understand your compliance portion and your, uh, notification, I'm sorry, your communication portion. So to be able to do that effectively, you have to get it done to start. And I mean, I, I'm a huge preacher about IR plans 'cause I've seen them really work.

And I think this incident really shows when they probably didn't have the best IR plan in place because they just were fumbling, they were falling over their own feet a bunch here, which then caused me to be on the call. Which, you know, in retrospect I appreciate because I love talking to you all. Um, but I shouldn't be here if I wouldn't be here.

If they had an effective plan in place and were able to actually communicate this where you all didn't get suspicious, understandably, it'd be it's a nothing burger. It'd be a nothing burger. Exactly. Exactly. So do you advise, uh, any MSP to have a breach coach available? Yes, I advise everybody. I think you all should have one. I think you should get a breach coach to your clients.

Um, you want to establish the rapport on both sides because when I come in on a breach or an incident, uh, I work directly with the MSP. And so if I know the MSS p it's a much better situation than when I don't know the M mss P. 'cause naturally I've had MSPs get really defensive, especially if I'm working with someone like Chris LA who just ends up yelling at them. Um, makes sense. So, but it's about making sure you have the rapport right? Even if I'm representing your client, right?

You're gonna, I need you, I need you in the IR plan, I need you in an incident. And then you also want to have your own too. You can share the client and you can share one As long as the breach, as long as the conflict doesn't come, that's a whole nother conversation. But yes, Chris, Larry doesn't yell at everyone. That's So can we, can we talk about the expense side, Spencer?

Because I think it's important, correct me if I'm wrong in my com conversations with you, it gets a lot more expensive not doing this first. So First, like they, that you have an again, whoever the, whoever the breach council is. Mm-Hmm. So what we know is that a breach costs, I think 4.4 million average global companies have got a plan, have a team in pre that saved like 2.66. This is up from 2019 where I think it was 1.3 million. So in terms of savings, it's huge. The numbers are huge.

Um, but especially for an MSP, you all are not Spencer's widget shop is one entity. If I have a breach, I just have my clients, right? My clients don't really have the customer aspect. I've got my commercial clients about their information potentially, or clients, individuals, right? If I'm producing widgets, you all though have 500 Spencer widget shops, which then have the downstream.

So then the communication aspect from the IR plan, I think for you all is so critical because you are, you already have half of this done. You understand the technical side of escalations, but you're gonna be saving yourself an immense amount of money and reputational damage if you have the calm side down with it. Hmm. Very interesting. Um, questions you guys might have. I'm gonna look, there was a few questions in ask a question section.

We got a, like I said, we got a top breach council here, please. Um, uh, I you, you're making me laugh with all the comments about Lair, but do you have some serious questions as, as well? Um, 'cause I can bring on Lair and we, you know, we do have a space here. He can just la la you know, just start yelling at everybody. Just Yelling at everybody. In fact, Um, in fact, hold on, let me see if I can get Chris here, uh, up because he, I'd love his take on this, uh, as well.

If we can Get, let me, let me turn down my volume. Yeah, yeah. Um, we wanna Take Okay, While, while, while Andrew, uh, gets Chris on, maybe it's nice to ask a question to the audience this time. Um, please, how many of you actually have a comp plan available, like for if an incident happens and just an incident According to the poll five, there are six. Six, yeah. Yeah. And there's, there's literally like right now hundreds and hundreds of hundreds on here yet.

I don't know why this is, but very few answer the poll. It takes literally a yes or no. It's, It's A click. It's A click, it's a click. It would really like serve everybody. This is for you too, not just me. Oh, It's like when I say YouTube, click like, and subscribe and like thousands of viewers, nobody clicks. Like, yeah, I gotta move. My mouse is drop that my LinkedIn, what's, is it okay if I drop my LinkedIn? Yeah, of course you can. Yeah, yeah.

Um, Chris, first could you turn down your volume, but second, any, any thoughts, you know, from watching other than, you know, um, you know, what's going on here in chat, but as you watch this, um, communications recommendations that, that stand out to you. Yeah. First of all, remember it Glue is Canadian so that that's already not a not good for them. Um, even we them with the acquisition, so don't, we can't miss the fact they're Canadian.

Uh, but what I would say in this thing is this is like overwhelmingly disappointing. I mean, we can, we could talk about the trust factor with K say all we want, but you know, I said it earlier on and, and Slagel said we should expand upon that is come on man. I mean these guys, you know, supposedly, you know, they hire a CIO from the FBI. They, they have been through kind of the worst of public relations issues, you know, over the last couple of years. None of this really makes any sense.

And even if it is just, and I don't think there's anything, um, coincidental about this, this password spray. It had to be triggered by something. Uh, it was when somebody got, you know, a, a whiff of air and just said, Hey, let's, let's password spray this weekend. Um, there's just that this, this whole thing is just kind of handled just incredibly goofy and the way that they've kind of taken care of it afterwards.

I mean, it's one thing to address this incident, but it's another thing to come out and just say, Hey, overall we're gonna do something better going forward. And I don't know if they said that in the video. I haven't watched the video, but it just, it's just everything about it smells bad. There's just not one piece of this where you can go, oh, you know, that was just somebody being a little trigger happy, just trying to do the right thing and get the message out.

It just is so like immature of a, of a or of a global organization that's that large. And we know that they have more than just MSPs as their clients. They have some very large government entities and stuff that are, that use Kaseya. It just, it, it, the whole thing is just goofy as hell.

Um, and you know, and then on that one other piece is, you know, I know Spencer's getting told a little bit about his comments about, uh, you know, we should go back to Kaseya, but this is back to this same thing with these vendors is even as large as this MSP community is, and I'll say it again and again and maybe I should yell about it, is we're still not acting like an industry should.

I mean, we have some great communities, we have some large communities like MSP geek, we got some great things, but we are still not operating like we should, like the banking industry does, like these other industries that have a lot of leverage. We're still just a, a bunch of little companies is what these guys see. And somehow we gotta figure out that mess and then we can apply leverage and start flipping things in the MSP's favor until we figure out that piece we're kind of screwed.

You need to become a consortium. You need to basically pull resources, um, and then you can start throwing your weight around. Because I can guarantee you if you get 5,000 MSPs working together under one banner, pushing with one message, um, you're gonna get traction. I mean, it's just, that's a five thou, I mean 5,000 MSPs, a thousand MSPs because say it might be big, but they will listen the more you get and the more centralized they want you to just be like broken up parts, right?

And just one angry man yelling with their hand raised versus a machine pushing them. Well that was one of the biggest frustrations that, you know, their response to the communications. It was awesome that, you know, Jason came on last week with, with Andrew to talk about it. Didn't get the answers we wanted. No, no fault of Andrews. I mean, you tried.

Um, but you know, that was one of the frustrating part was 'cause we each individually went back and said you know, to NIR and to others, Hey, we wanna have this conversation. We wanna ask the real questions that community's asking. And their answer was, no, we're gonna talk to our partners directly. 'cause they're giving us good feedback. My problem with that is you lose the, you lose the intelligence of the group. You lose the information sharing of the group.

I know when I wanna know if something's going on, I'll go straight to geek, I'll go straight to Discord or, or msp and I'll ask, and I'll be, I'll have a hundred other MSPs tell me, this is what's going on. This is, you know, this is what they're telling me. So you can fact check.

You can't do that when they're having one-on-one conversations in private, which is why I applaud any vendor or anybody that's willing to go in front of an audience and talk and say, we did this, this was our thinking. We may have screwed up, but this is why, and this is how we're gonna make it better. And that didn't happen here. And by the way, Spencer J, when he was first to Jason, their, their CISO did come on last week. Mm-Hmm. So, so just ffy.

Um, so we have a, that's really, Ray, your points are really well taken. Um, the, uh, consortium idea is a great one, Spencer, and, um, we've got a few minutes left. So let me ask everybody in, in the audience, how can we help, right? Like, are there some things you guys like an FAQ from maybe Spencer, that we could facilitate?

Like, you know, like to start, like top things you should be looking at in your comms plan, something like that, that, you know, Spencer could, you know, I know I'm putting this on you, but like a little blog that we could get out to everybody of things to Yeah. Review and look at what, what do you guys need and want? Again, I want this to be something constructive where we saw a case study of bad communication and what it did. This could, again, please don't take this the wrong way, right?

This could be us, right? Because, and Chris can, can I ask for your help with this? When you run tabletops and real incidences, a lot of times the person who's the, I don't know, CEO, you're like, that is not the person who should be doing the communication because of their personality. So as, as I look for comments about what I just asked, could you even talk about that?

Where you think, oh, well the leader of the company is the one that should do the comms talk to us about under a, you know, an RMM compromise, why sometimes that isn't the one or something like that. I mean, from, yeah, I mean we, we've seen it, we've seen it with CEOs just take it upon themselves. Just start, start calling clients and they just talk too much. And they reveal too much. They speculate too much. And so you gotta kind of know that and, and, and just know their personality.

And most people do. And so now trying to convince them to stand down in these situations, that's, that's not the easiest thing. But Spencer and I could talk in for hours with examples about that. And the other rule I would tell you is don't hire anybody that's related to anybody in the media. I mean, uh, Spencer and I have been involved in a number of cases where, uh, it's purely been that reason alone.

It, it was somebody worked for the company and they were their spouse or someone else was in the media. And that's how the media caught wind of it. So you can't underscore the importance of these types of little things that are the ones that bite you. And I, and, and this is a perfect example. I mean the, the one lesson to take outside of the whole vendor snafu thing here is how communications is incredibly key. Internal communications, external communications.

It is so important to the success of everything. And when Spencer was talking about earlier how those IR plans from MSPs are so technically good, the communications side is, is is really the, the piece that lacks the most. Chris, as somebody that owns a media company, I take great offense to that. Even if you're right, that's irrelevant. But Chris, I love the way, I know at the top of the hour here, I think you just summed up the whole thing really well, which is, again, everything.

Let's say it was all fine and really nothing bad happened. Our communication to our customers is so vitally critical. 'cause we see here as we talk about it two weeks later, right? Think about that, right? The impact it's had. Um, and I'll sum up with what Jason Slagel said in uh, in here. Spencer's a simple comms plan would be great. I have a plan in ours of, you know, who can talk roughly, uh, what they can say.

But I'd love to see more is like maybe an FAQ of what to, so you're not obviously doing their job, the job, right. That people can hire you if they want. But maybe an FAQ that I could give out to everybody, um, Spencer on what things should, should be in your, or you should you be evaluating in your current comms plan, like you're the top five things or whatever it is like that. Um, maybe Yeah, I'd be like do's and don'ts kind Of thing. Yeah. And then, then we'll have people post it.

We'll post it out on LinkedIn. I'll send it out to all the emails. And um, as a thanks to you obviously we'll put your LinkedIn and email in it. Oh no, thank you. I appreciate it. Yeah. Um, so I just want to say again for all of you on here, thank you so much. First, um, I want to thank Ray, um, Kyle and Kelvin. Um, I know we've all spoken over the weekend and, you know, many hours on this, so I appreciate your, um, your willingness to, to come on and, and, and share perspective.

Um, Spencer, thank you. Um, I know, um, you um, have a lot on your plate to say the least, and, and your, um, you know, being gracious and coming on here and, and such. I appreciate y'all Having me. Yeah, of course Chris. Um, I'm not even sure if I am gonna No, of course. Always awesome to see you, my friend. Um, that's always good to Be, I just try to see how many times the cats get on Spencer's lap. There's A count in the, in the corner. Yeah, I think it's at like seven.

Yeah, that's what to say. Um, I got one In front of me right now. She gave up. Um, so with that, um, as we wrap up here, episode one 14, I appreciate the community. Um, thanks for the awesome chat as always. I really appreciate your guys willingness to stay engaged and, and, and help each other out. Um, and again, um, love to see you all at right of boom and uh, look forward to seeing you all very soon. Anything else in closing you might wanna share Kelvin?

'cause you're the, the latest as far as hours being what, five or six ahead of us? Yeah, it's, it's is there Anything in the future you can tell us? Exactly. I think, I didn't think Europeans work this late. We, we actually don't. This, this is, this is my hobby. No. Um, I think the, the biggest thing I want to tell every MSP in here right now is hey, um, make sure you have a comms plan or an instant response plan available.

Contact, uh, a breach coach or anything like that to make sure that you don't end up in a situation where you can't communicate anything to clients. Just be prepared for the worst. Well, really well summed up. And then Ray, everybody's asking me to get your name tag. I'll get with you offline to see if somehow I can do that. Um, 'cause I know that's part probably what you do for a living. Um, yeah, I do my, I do pe I don't know if everybody knows here.

Um, like Ray, I get beaten up here on the cyber call and I'm kidding partially, but I do sometimes and like, um, I pay for this Ray, I don't get any money, so I pay to get the abuse every week. Uh, You know, you're a little masochistic, but you do it outta love. I do the same thing. Hey, look as a, as a moderator of RMSP, right? Like we all know what Reddit is and I still Kelvin and I do it with love, so I I completely get It. Yeah, you guys do a great job. It's like being, you know what it is.

Last thing I'll say, it's like being on the board of it, of your HOA, right? Oh, that's, that's pretty accurate actually. It Is. It's, That's Or state council or something. Just the government council, I mean, or just owning a cat. I mean, yeah, Yeah, Yeah. School Board. Jason got a school board. Yeah. Yeah, right. James. No, I, I appreciate you having us on Andrew. Um, thank you. It's an important conversation and this is a good note to vendors.

If you are not gonna have the conversation in public, the MSPs will so better to be part of the conversation than not Probably right after you have any communication. Thanks everybody. Have an awesome week. Look forward to seeing you all very soon and make it a great day. Thanks everybody.