Storm-0558 is a China-based threat actor M365 Compromise

Welcome, welcome everybody. Cyber Call, episode 1 48, and we've got, um, some new faces on with us calling Audibles left and right, but, uh, I'm very excited about who's with us. I'm gonna introduce everybody momentarily. A few quick announcements for you all. First off, um, the final back Doors and Breaches cloud, uh, episode with Bo Bullock, um, is gonna be this coming Thursday. Um, and Chip, this is your group putting this on, uh, sponsoring, uh, Bo.

And the cool thing about this is there's no sales pitches, there's no nothing. This is just straight up learning how to handle incidences through the back doors and breaches platform. And then the new, um, cloud, uh, initiative that, um, that, uh, black Hills has put together. What do you think about it so far, chip? Can you, I think it's been great.

I mean, there's been terrific engagement from, uh, you know, the participants, um, the attendees of these, of these first two, uh, features that we did. I love backdoors and breaches. I think it's such a good idea both for internal team training, but especially for customer training when you can get a customer to participate, uh, and, and work through it and just learn about the whole notion of table type exercises. It's a fun way to introduce people to it. Yeah. Yeah.

Chris, you, we'll introduce you momentarily, Chris, but you've, you've got a lot of experience on this as an incident handler working with the guys at, uh, back doors and breaches. Any quick thoughts on, you know, using back doors and breaches as well? Oh, absolutely. Using back doors and breaches and really any opportunity to, you know, war game, tabletop, whatever you want to call it, uh, just to flex those muscles that you don't get to use all the time.

And plus you don't wanna flex those muscles in a real life, uh, incident. Uh, so being able to do that in a nice, safe way is worth its weight in gold. Yeah. Yeah. Yeah. Absolutely. It's, it's, like I said, it, I always, I always reflect on this, Chris and, and Phyllis from the defender side at the NSA, I always reflect, I'm like, you know, no one thinks twice about, you know, if they play a sport right. Practicing. Except, you know, when we talk about cyber, we're like a tabletop.

Why would I want to do that? You know? I don't know. You beats me. Let's figure it out when all hell is breaking loose. All right. Right. Um, all right, so that's one thing going on. I put the URL in chat. The other thing going on, Chris, maybe you can chat a little bit about this, but you and Wes Spencer are doing a masterclass. Um, I think there's what, four of 'em around the country, give or take? Yep. Yep.

Uh, we're, so we're gonna do a, a road show focused on, uh, cybersecurity masterclass. So this is going to be everything from how do you position yourself to do cybersecurity as MSP to, how do you handle, you know, those conversations that back and forth with your customers all the way through, like, how do you handle something like a cybersecurity incident? And so we're gonna be doing four cities in August.

We're gonna start with Chicago, then we're gonna go down to Orlando, Dallas, and then la. Awesome. Awesome, awesome. And I put the URL in there for people to learn more there, Chris. Thank you. And, uh, to Danny's question, Danny, um, I've heard a ton of people use, um, uh, the cyber call for their CPEs. Um, you know, just, I, I'm not exactly sure how they've done it, but they just literally take, like, typically the agenda.

Like if it's A-C-I-S-S-P and you know, the class and send it in, they'll put the URL, uh, that they attended. Um, so if anybody has other, like I saw Steven made a comment as well, but if anybody has done that, but just, you know, if you could chime in there on how to do that, that would be, um, awesome. All right. Welcome, man. Welcome everybody. Okay, last thing, um, in the green bar below, um, I put in right of boom registration, early bird is live.

The, um, particular, ironically, both SaaS alerts, which has Bo Bullock and Chip, um, uh, chip, your pre-day with BO on M 365 compromises and everything. And Chris, your pre-day with Huntress are the two neck and neck right now. And for those out there that want to attend these, the way it works is there's the pre-days, um, there's, there's a few that have larger slots, but most of the pre-days only have 40 attendees max. And that's it.

And, um, so yours are, uh, running neck and neck right now, so we'll see who gets there, gets there first. So let me set the stage we'll get on into this year. This is a really, um, to me, a really fascinating one. And, and Chip, I'm really glad you've spent so much time on this.

Uh, I think it's important, um, for MSPs especially, because one of the good things about this that we're gonna talk about is how Microsoft changed course in terms of their allowing, um, just premium addition to get logging. That was huge. Yeah. You know, if there's any positive in this, MSPs are gonna benefit. And so if, you know, if you have friends that are in the MSP community, and they're not on this right now, I highly suggest, you know, reach out to them, tell 'em to join us.

Um, but, um, just let me, I'm gonna quote something from the, um, Washington Post, which was one of the earliest chip that you and I went back and forth on, um, that, that started talking about the story. Now again, they were like last month, who knows how long the Chinese were, you know, doing their thing here. I doubt it was last month. I'm sure it was a little longer, uh, than just last month.

But, you know, it says the, um, I'm just gonna paraphrase it here, but Chinese cyber spies, uh, were exploiting a fundamental gap in the Microsoft Cloud hacked email accounts at the commerce and state departments, including that of Commerce Secretary, Gina Raimondo, if I'm pronouncing her names right, whose agency had imposed stiff export controls on Chinese technology that Beijing had denounced as malicious attempt to suppress its companies.

So, you know, in, in terms of who they attacked, you know, it would certainly, you know, seemingly appear that this was cyber espionage, that there was obviously a strong motive behind what they did. Um, but let's dig into this a little deeper and see what other ramifications, 'cause I think Chip, um, what reason I wanted to have you on, you've like, I think read every piece of information available.

You've gone through Microsoft's literally their documentation, and your comment to me was, as I let you introduce yourself and then to Chris, your comment on paraphrasing chip is this could be one of the most sophisticated cyber attacks to date. So, um, I'm, we're gonna put that on a little dangler there to come to that momentarily with Phyllis. But Chip, tell us a little about yourself.

That folks that don't know you, a lot of 'em do, but tell us a little about yourself and your role currently. Sure. So thanks Andrew. Um, well, I'm, I'm Chip Buck. I'm, uh, CTO and co-founder at SA Alerts. Um, I've been selling, uh, technology services and products in the, in the MSP and SMB universe since, uh, the late nineties. So, you know, had a pretty long career, uh, in this stretch.

Have built a couple of companies, um, along the way, uh, prior to that, I've, you know, had experience as a software developer and things that we really don't do much development around anymore, at least not in the business world, embedded systems. Um, so, uh, that's the background. But, uh, you know, Mo, most of my career has been spent, um, in, in the MSP world, either directly or indirectly as an MSP myself, and then selling, uh, software products to help MSPs with their customer journey.

Yeah, it's really cool that you got, uh, traction on, you know, the experience on both sides, chip, which is always very helpful when you build something, Chris, new to the MSP space or new. Yep. But if anybody out there follows Chris Cochrane, I'll put his LinkedIn, uh, out there.

If you are, you know, somebody that follows, um, some of the top cyber, um, uh, evangelists and people that really, uh, have, uh, strong followings, um, I'm gonna let Chris tell you a little about himself, about, um, his, um, gosh, Chris, I'm drawing a blank on your platform. Hacker News. No, Ugh, almost, what is it? Hacker Valley Media Banker, hacker Valley Media. But, um, Chris, really cool that Huntress was able to bring you on. Hey, nerds back. Kelvin, good to see you.

Um, uh, lot of experience incident handling service to our country as a Marine. So awesome to have you with us. Um, why don't you share a little about your background, Chris? Yeah, for sure. Uh, thanks for having me on the call. Excited to be here. Uh, started my career, uh, as a United States Marine. I was at the National Security Agency for about five years doing technical intelligence. Spent a long time at, uh, US Cyber Command when it stood up.

And then I've been able to do some really cool things in the cybersecurity space. I've worked at organizations such as the US House of Reps, uh, Mandiant, Netflix, uh, a couple other places, largely leading security operations with a focus on intelligence. Uh, but now I'm the advisory CISO and chief evangelist at Huntress and alongside the, uh, the Hacker Valley stuff where we create content, podcast, web series, uh, for practitioners of, of all sorts. Very cool. Alright.

Thank you so much for, uh, the intros. Thanks for everybody being here. Let's make this chat lively, um, throw q and a into, if you have questions for any of the, um, audience, guests today, throw 'em into q and a. I'll try and keep my eyes on chat, but let's keep the chat as lively as ever. Phyllis, always wonderful to see you. Um, and I will let you, uh, kick things off here, uh, with Chip. Yeah, thanks, and thanks everyone for being here.

Um, so Chip, can you help, um, everyone understand, um, what actually, um, did storm, I don't know how to say it. Is it storm dash 0 5 5 8? I mean, how did they use the consumer product side of Microsoft Kane access, um, to these systems? Well, there's, you know, I think they stumbled across a mistake or a flaw in how Microsoft's, um, authorization and then session token, um, process works. Um, you know, there, the big unknown here is how did they get this signing key?

So for those who don't know, the way the authorization programs work, you put in your credentials into the Microsoft OAuth, uh, screen like you're used to doing. You get challenged for MFA, that bundles up a bunch of information inside of Microsoft's, um, authorization system, including your tenant ID and Scope and other things. And that gets sent over, uh, to Microsoft. And they check all this out and figure out who you are. And you've passed all this stuff.

And what comes back, um, is the session token. The session token contains information about what things you're allowed to access, um, and how long that SEC token is gonna be alive. And what Microsoft did, excuse me, what the hackers did is, uh, they managed to get a hold of, uh, a signing key that's used to sign that, um, that session token that comes back for validation, a private key. And they managed to use that to forge the session tokens.

So I think a lot of people in this call are probably familiar with the whole session token hacks that we've all gotten sort of used to, right? Even in Microsoft's, um, review on their blog of this particular incident, their initial reaction was, okay, well this was probably a proxy based session harvesting attack, and that's how they get in. But they realize that that's not what was happening. So they had to look deeper.

Um, the, the flaw and the consumer side of this, Phil, is to get your question, is the key that they acquired. And, um, Microsoft specifically uses in their commentary that these attackers acquired an inactive, uh, consumer key, which, you know, that term inactive, let's come back to that, or maybe Chris will want to come back to that. 'cause I think that's a very, very important, um, facet to this, uh, whole incident.

So they use that inactive, um, key to forge tokens that they subsequently use to access enterprise, uh, in this case government, um, domains, government tenants in Microsoft 365. So, you know, there's a cross, uh, there's a cross hatch there that probably shouldn't have occurred. I'm guessing that it was, um, not designed to work that way. Microsoft acknowledges that that was a flaw, uh, that they've since rec rectified, but that was the way they used the consumer side to work through.

The other interesting piece to note is, is that they didn't only focus on the government, um, email addresses or email accounts, um, they also went after the personal account. So Andrew, let's just say it's, you know, Andrew Morgan at, uh, state.gov or, or whatever the address would be. They also went after andrewMorgan@outlook.com knowing that you had a personal account. And we're looking, uh, on both sides of it.

So there's really two angles to this in terms of the consumer, uh, in the enterprise crossover. Yeah, I mean, that's a very good explanation. Thank you. You already talked about how they kind of forged the token, but can you kind of go into what is token harvesting and why is it that this attack seems, you know, from your perspective and really just it is, why was it so sophisticated? Yeah. Well, token harvesting is something that we've noticed, um, come on strong really in the past year.

I mean, it's been there for a while and it affects really any product that uses, um, OAuth two as a methodology to, to validate and maintain a session, um, is susceptible to this. This can happen with, with Google Workspace, it can happen with Outlook, um, could theoretically happen with really any product that's a SaaS product in the marketplace.

And it's there because, you know, um, people who write SaaS applications need a methodology, um, to allow people to use their browser to access, you know, a a web client. Like that's the whole purpose of SaaS. So, um, the, the session token itself is what's really regulating, uh, that authorization. And we've talked about this on other calls, that there's steps, steps people can take, um, to provide some protection around this.

You should expire session tokens regularly, rather than leave them open by default, Microsoft leaves their session tokens open for days or even weeks. You can be logged into your Outlook web access, uh, or, you know, OneDrive, uh, online for as long as you want practically unless your administrator changes that session. So the, the hackers have have sort of figured out, well, wait a minute. There's this little token that's sitting in the browser. Um, we can figure out ways to get that.

Um, there's a couple ways of getting it. One is simply by clicking a link, when you're using an insecure browser, it opens up another tab and can harvest your session token from the tab you open over there. Um, that's become less common because it's less reliable. The browsers themselves have gotten, uh, tighter.

Uh, the other though is it's what's called a proxy harvest attack, where someone clicks on a link that they get in a phishing email, um, and up pops what looks to them like a perfectly valid, uh, Microsoft or Google workspace authorization screen, and they go through the entire authorization process, username, password, hopefully they have an MFA challenge on there.

Um, and the attacker in the meantime is picking up all this information, including the session token that comes back, and now they're able to open a tab in their own browser and log in, impersonating that user, um, with that same session token. Wow. Thanks. Excellent explanation. So, um, someone actually, Derek posted this, the quote, the method by which the actor acquired the key is still a matter, I know, is a matter of ongoing investigation. Yeah.

And we were, we were just talking about this, um, right before we got on, you know, how is it that, um, that key got stolen in the first place, right? I mean, this is like 1 0 1, like, you know, guard your secret, guard your secret. Like that's like the number one thing that everyone is supposed to do. Yeah.

You know, when Andrew and I were bouncing around with this when it first popped out, um, you know, I think the reason that I made the comment to Andrea that I think this could be one of the most sophisticated attacks we've seen maybe ever against Microsoft, I wouldn't know that, uh, of course, but it, you know, in, in, in all the time that I've been following these things, it feels like that to me.

Um, I think part of me was mentally, um, prepped into that commentary to Andrew, because I Wired did a great article on this, you know, like a day after this news broke. Um, and they identified this as one of the most important is, what is the word they used, um, breaches in the Microsoft universe. Um, that inactive token part is, is the thing to me that makes this so intriguing and, and I think potentially makes it, um, such a, you know, a momentous, um, breach on the part of Microsoft.

You know, we all, as we all business owners, and I'm sure Chris in the military, now you're at Hunts, we've got procedures for everything, right? When you offboard an employee, you've got a checklist. Mm-Hmm. You know, if you have an inactive token, it's either inactive, waiting to become active, or it's inactive because it's been deactivated.

So we, I don't know which it is for sure on this side, if it's been deactivated and it was formally in use, there's gotta be a checklist somewhere to de to destroy that, right? If it's inactive waiting to be activated, there's gotta be a process for that as well to be making certain that these tokens are maintained in such a way in a key store. Um, I keep saying tokens. I mean, keys are maintained in a key store that's, you know, completely imponderable.

I mean, in our organization, you know, our key stores are locked down in myself and, and two senior developers, that's it. Out of everybody in, in the business, no one else has access to that. And I think, you know, we've all learned that that's the way you need to manage, um, properly manage key stores. So what happened here, you know, it, it feels to me total speculation, but this feels to me more like almost an inside job than it would anything else.

I have a really hard time believing that Microsoft doesn't have strong controls in place. So I just wonder, you know, what was the flaw that allowed this to slip out? Was it an innocent mistake on the part of a developer? I, I don't know. I don't, you know, Chris, I'd love your opinion on this one when they pull you into the conversation.

I just have a hard time imagining that anyone who would have access to this information, it's gonna be of the type of personality, um, that would make a mistake, that would do something foolish with this that would say, oh, I'm just gonna use this key and try it out. So, um, Chris, No. Let's, let's have, let's have Chris, Chris chime in. I mean, you, you've got so many reps onto you for incidents. Like, what, what's your gut tell You here?

Yeah, I mean, it is pure conjecture, but I, I think you nailed it. I, I think it, it, it's some type of exposure and, and largely from an insider's perspective, I would say that's probably the easiest way to go about it, is some type of insider that gave it away willingly, or maybe someone was coerced into doing it. I mean, this is one of those situations where someone really wanted to get to that, to that, that token, the, uh, to that ability to write basically as many passwords as they want.

Uh, many as, as many accesses as they want in this, uh, in this environment. And so I think it's pretty alarming in, in a lot of different cases. And what I'm thinking about right now is all of the folks that are doing incident response right now, because now you're thinking, you know, the first time you hear about something like, Hey, am I leveraging Microsoft? Yep. Check. So now how do I look for it? And so now you're figuring out, you, now you're looking up the, the disclosure for Microsoft.

You're looking at some of the articles, uh, Wiz put out a really good article about, uh, the whole situation, which I think was really cool. So now you're trying to figure out, alright, was this, uh, leveraged in my environment? What are the IOCs that are attached to it? And then I definitely heard that there was some other information that's gonna be coming out here relatively soon. And so I, I think it's kind of a nightmare for a lot of folks. You're gonna have a lot of churn.

Uh, some a I think a lot of folks are gonna probably be safe, but you just never know. So you're gonna have to wait. Are you seeing people going back to exchange? Chris, have you, have you Followed the Reddit threads? I mean, the Reddit threads in cybersecurity, you've got people all over, like, this is the end of cloud, we're all going back to on-prem. Well, it's funny, it's funny you say that going, no. Alright, so perfect timing.

The reason I wanna pull up, Kelvin, first of all, you know, internationally known has vast experience with the Microsoft APIs building his own open source with CIPP founder event, or one of the admins of Reddit, MSP. And, uh, so, uh, Kelvin as an MSP and certainly considered MSSP in terms of your incident handling. Like, give us your take on this. Uh, love it. Oh, give the, the thing is that I have very hot takes about this. Um, number one, we Don't care. Just let it rip, let it rip. Yeah.

Let's start with number one is that, um, no matter what anyone is saying, no one could have protected themselves against this. Uh, there were a couple of vendors that have started saying like, if you use this and that, you would've been safe. No, you wouldn't have been safe. This is one of those things where someone just throws a brick through your window and enters and leaves. Again, this is not a simple backdoor thing. No, this is the front door.

You a huge brick or a battering ramp to ram your door down at the enter. It's, it's, it's as simple as that. This is not something that anyone could have protected you from. And, um, even Microsoft, even Microsoft themselves has said like, okay, you know, this, this entire technique, the way they worked using those MSA keys, we had to make specific fixes, uh, making sure that our uhm SA keys are no longer stored in one specific location, but we're now sorting them in our enterprise key stores.

So they, they changed their entire backend infrastructure to make sure that this could never happen again. It's, it's not a simple, um, um, minor attack. And I was just saying some things like in the chat, um, Microsoft has acknowledged that not having the logging available would be a terrible situation for everyone. So they are now making that logging available everywhere. Mm-Hmm. Um, it's, it's, uh, I've seen a bit of confusion about that.

So I want to make sure that, that, that is addressed, because as a Microsoft MVP, Microsoft has asked us like, Hey, make sure you say this in the right way, using the right words, using the right, uh, things, because a lot of people started assuming, oh, we're now getting free P one, everyone is now getting a, uh, active directory, premium subscription. That is not the case. Everyone is getting access to advanced logging. Everyone is getting access to what's called purview basic auditing.

That is, you're getting access to an API to access audit events. You're getting access to more, uh, data, uh, instead of 30 days of logs or 40 days of logs, depending on your Azure subscription, you're getting 90 days by default, or 60 days by default, 90 days by a manual extension, something like that. And, um, Microsoft is making sure that those pieces of the P one subscription are available to you. This doesn't mean you, you're getting free conditional access.

This doesn't mean you're getting ways to protect yourself or better security. You're just getting access to that auditing data, and that's very important. Mm-Hmm. Um, So Kelvin, just a quick question. What does it mean, let's just say it was September, and we have a question in here, so I don't wanna blow up. I forget who asked who, but around the why September, but let's just say it was on right now as an MSP, how are you leveraging that?

Because you mentioned it be a ap, so, you know, 'cause again, you know, as MSPs, we've got a lot going on in our business. I'm assuming this, this doesn't show up, right? And now of a sudden alerts are coming through your door. Obviously I'm being a little facetious. So talk to us about what you would do to make sure that you, this is actually useful for you and your company. Yeah.

So, um, one of the things that we always do, uh, at my atmosphere specifically is, um, all the audit data gets collected at a central sim. That is, um, we just make sure that all the all data gets collected, analyzed, these kind of things. I have a very strong security team that works on that. But one of the things that would've been very noticeable in this specific attack is that, um, the key that is being used is being signed by a different application than the ones in your own tenant.

It is an unknown application ideas. It's called the, the specific part of the key that showed like, Hey, this application signed this key was unknown to your own tenant. And that is something that you get alert on. By default, very few applications should do that. Yes, there are some M ms P applications, I believe that, um, some SSO applications that are used, the MSP space, I wanna name the specific vendor, use a foreign principle as it's called.

But even then, if you start alerting on it and only excluding the foreign principles, you're not using it already gives you a lot more insight to where these logs are coming from. But Kelvin, as Eric Woodard asked, you know, does this logging, you know, just kind of enabled or default? And again, Eric, it's, it's not till September. Again, we're gonna come to that question to the team momentarily. Last thing I'm gonna ask you, Kelvin, but please stay, I'm gonna get back on track.

But you know, Kelvin, so you're saying, Hey, look it, pretend it's September 1st, this is available, but you still gotta take the steps to get that forwarded into your sim. Absolutely. It's not just gonna, all of a sudden, you know, as MSPs like, yeah, you know, here's a ticket in your, in your PSA that says, oh wow, here's some conditional access log that you should probably review. That's not gonna happen.

No, it's, you still have to enable the, that, that, that's one thing that I cannot stress enough. And, um, I'm going to plug sip, I'm sorry, but SIP is a M 365 multi-tenant, multi-tenant management application that allows you to enable the audit log for order tenants in one click. Please do that. Because by default, audit logging is disabled in all your tenants. It is not a default setting.

Microsoft says, we're not going to collect that data unless you explicitly tell us to start collecting that data. So in your M 365 tenant, it is off by default for all your clients. Make sure you go to the audit logging and you click yes, we're talking unified audit logging, someone just asking chip, make sure in the unified audit log, you click on that enabled button, otherwise you're not collecting the data. And it only starts collecting the data from that point forward.

It doesn't give you the data from before. If you've not hit the enabled button chip, What do you, why do you think that is the case? By the way? Why do you think mic, is it storage? Is it cost? I Mean, it's a lot of data, right? Mm-Hmm. So, you know, Microsoft is, is hosting hundreds of millions of, you know, enterprise user accounts.

And, um, I mean, look, we save this data at LERs longer than Microsoft does because we can, but we're, you know, we're, we're biting off the tiniest little fraction of this Apple. Um, if you're Microsoft, you, you can't just save data for people who are never gonna use it. That's wasteful. You know, like in our case, we know that our customers want this data, right? Um, and we go through the same conversation that, that Kelvin just laid out. You know, unified audit logging has to be turned on.

Um, auto logging, you know, even basic audit logging for older tenants isn't turned on by default. You have to jump in and do that. It is now, I think since, uh, fall of 2021 is, you know, newer tenants, it's automatically on. Um, but if I were Microsoft, I wouldn't just turn it on automatically. 'cause most of, let's face it, most of their customers in the MSB world, um, are not, not using the data. So why would they store it? It's just piling up. Got it. Got it.

And there's processor time to compute, time to create it. I mean, there, there's, there's cost. Yeah. Yeah. Yeah. Bottom line. So, Phyllis, thank you for, uh, let me jump in there. I, I really wanted to Kelvin's perspective. Kelvin, stay with us. I'd love for you to chime in when it's appropriate. If you can. I'd Love, I'd love to stay, but I have to hop off. Uh, is it dinner time? Come on. You got time, Time. I have to start cooking. What, what are we having tonight? Just to Cook?

I, I, I, I think it's gonna be something simple. I'll, I'll send a picture when I'm done. Yeah. Can you, you know, and the recipe if you would, in chat when you're done. All right. Of course. All right, bud. Great seeing you. Alright, Talk to you later, man. Thanks. Um, All right, Phyllis. Yeah. So let me circle back to Chris. We've, we've said a lot. Um, you've said a little bit about, you know, possibly an insider.

Just in general, what are your thoughts, um, from, you know, you've, you have a long career in incident response, so, um, what are your thoughts thus far on the conversation and some of the comments in the chat? Yeah, there's some secondary and tertiary things that people need to think about when it comes to this. Uh, activity is, of course, you know, there are IOCs that Microsoft put out. There's information that some of the other folks put out as well.

But once you get access, there are a lot of things that you can do internal to create additional footholds, create other situations that aren't gonna be great for MSPs nor your, your customers.

So you're gonna have to do even more diligence, even if you have access to all the logs that you need, even if you have all the information, you're able to do your, your investigation, you're gonna have to really do a little extra step just to make sure that there's nothing else that's going on on the backend that might give additional access to, you know, these, these criminal hackers.

So, I mean, it, it really is a, a huge mess for people to clean up, but it, it is, it's just gonna be one of those things that we all have to do together. Okay. I did wanna circle back and just point out, um, 'cause we got a little sidetracked with very interesting conversation. Um, when, you know, chip was talking, number one, someone actually had to get access to the keys, but then understand that the code was flawed. Right. Who would've thought that in? I think That was luck. Yeah.

I You think it was luck? Okay. Yeah, because I was like, you know, what are the odds? It's true. You know, that all these flaws were in place so that something that someone didn't think would ever happen was able to happen. Right? Mm-Hmm. Do you, do you think, you know, Phyllis, when you think about this, you know, obviously it's kind of certainly not SolarWinds right? But certainly reminiscent because of Microsoft's involvement and, and things like that.

Do you think there's more, like they're saying only a few organizations right? In our country and collectively less than, I mean, look, we're gonna talk about this, but less than 30. Like, I, I gotta find that hard to believe. Well, it, so it, you can look at it a couple ways. Mm-Hmm.

Number one, Microsoft should know how to track the, those, those conversations down because they know the keys and the signatures should be clear and they should be able to look in their logs, and they should be able to very clearly find out which organizations were compromised. Right? So, unless there's like another flaw or something else that was exploited that Microsoft doesn't know about, I feel like they should know. And they should know very clearly which organizations were hacked into.

Um, So, so, so to that, like, Matt asked the question about like, with these fixes, what was the end user impact chip? Is it, is there, other than quote unquote exfiltration of email, was there other things that you saw in the articles? No. Um, but there certainly could have been. I mean, you know, this is one of the themes I hound on all the time, Andrew.

And, you know, I know you hear me say it and or probably sick of hearing me say it, but a business email compromise is a Microsoft 365 compromise. It's a total account compromise. It's not just one piece of it. So certainly, you know, for as long as the attacker is, is in there, uh, living off the land as, as we like to say, um, who knows what else they could have done.

Um, now, you know, you would have logging data to indicate did they download files, did they upload file, did they delete files? Did they, you know, get in into a teams group? Like all of that information could be uncovered in terms of what the account did over the time span when the attackers, uh, were there. Mm-Hmm.

I'm, you know, Phyllis, I I tend to agree with your assessment that Microsoft should know, um, certainly based upon all the IOCs that we see here, the, you know, which SOX proxies were used, what the IP address ranges were, the, the $64 question is, you know, was this, is there a broader time span here that we haven't heard about yet?

Like, there were different proxies used, different IP addresses, um, you know, other IOCs that you could, that are still to be correlated that would indicate that this was a broader event. And I, you know, I don't think we, we know that yet. Microsoft hasn't really come out and said they've closed the book on this. I mean, I think they've done a very good job of being transparent Mm-Hmm. Um, in, in the information that they've provided, which is great.

That really helps the security community overall. It's helping my company, you know, derive, um, event tracking for this sort of thing that we didn't have before. Um, because, um, you know, this was a brick through the front door, right. Our, our team came to us immediately and said, uh, came to me and, and our senior developers and said, Hey, can, could we have caught this? You know, if people were using SAS words, would we have caught us? I said, absolutely not.

Like, I don't wanna hear anyone in our organization come out and make that claim. This was very, very novel. Um, And, and I think you're correct that Microsoft doesn't fully know just because in the pre-show you were saying everyone, everyone had to go and get a new key, right? That's a pretty drastic measure, and that is a pretty, um, uh, I would say that causes a burden on the end user that a big company typically wouldn't want To. Yeah.

Well, they didn't revoke anybody's session tokens on the enterprise side, or even on the consumer side. What they did do, do was revoke the keys from the, from the consumer side. Mm-Hmm. Um, so any, any keys that were in that key store were eliminated, recreated, and they moved the whole key store. Ah, okay. Got it. Got it, got it. Okay. So, um, back to Chris. We, we were just talking about the, um, premium security logging Mm-Hmm. It's not going to be available until, until um, September.

Right. What are your thoughts on this, and why, why, why is it that everyone has to wait until September? Right? Yeah, I mean, we kind of touched on it earlier, but first yeah. Congratulations to, uh, Microsoft for making this step. I think this step has been in the process for a while now.

Uh, Jen Easter, easterly director, Jen Easter easterly from cisa cisa, uh, talked about, Hey, we've been having this conversation for about a year now of enabling, uh, the, the industry to have access to this. Just off the, off the rip, I think this probably pushed the, the timeline up a little bit, but when you're talking about enabling logging for this many organizations, 'cause I mean, how many accounts have the standard account?

I mean, turning that on isn't just like hitting a button and everything's good to go. I mean, you gotta think about compute, you gotta think about storage. Uh, there, there's a lot that goes into it, and they're, they're just trying to do it in a way that makes sense for, for their infrastructure and just try not to, to bog everything down. Yeah, that makes sense. Over to you, Chris. Yeah. Yeah.

I definitely wanted to touch on a little bit about, uh, the, the organizations, right, that they said, they said the 10 or so. Um, when it comes to, uh, token harvesting identity impressions, uh, and the indicators of compromise that was sent over by Microsoft, how holistic would you say that that information is? Uh, chip?

Would you say that, uh, we touched on like, do we think it's more organizations, but do you think this is enough information to really get this incident to closure if, uh, MSP or even one of the customers are dealing with the, the incident itself? Um, well, for you guys, everyone keeps saying 10. I've read 25 in a number of places, so Right. Yeah. 10, 10 in the United States, but 25 altogether. Yeah.

And then I think there was a bunch of personal accounts, mainly overseas in the Taiwan area, which probably alluded to it being a potentially a Chinese actor. Yeah. Look, I mean, for me, my, my gut tells me that that piece of it we can close out and, and my logic is as follows. Um, the first piece of it is, is that Microsoft really doesn't have a reason to lie about it all that much. You know, if it's a hundred or if it's 25, it doesn't really make a difference.

Um, but what's most important is, is I feel confident that they've contacted anyone. Like there may be, um, there may be, I could see a scenario where, you know, the government of the UK or Germany would say, you know what? Don't include us in your account. Thanks for letting our people know. Um, but we don't even, we don't even wanna sniff that, you know, that we were part of this. So there's gonna be unknowns.

But I, but I do feel confident that anybody was impacted, at least through the IOCs, that, that are in hand, that you can track that in the data in his, in the archival data, the historical data. Um, you know, they can do a forensic analysis, find out which tenants were impacted, and contact those tenants. And I'm sure they've contacted every single tenant because it would be suicide not to. Right.

One thing I gotta bring up, I'm gonna be the one to, to, to ask about it, but artificial intelligence, uh, from a security operations standpoint, I've been working with a bunch of different companies that are trying to leverage AI to do security operations, looking at visibility, looking at intelligence and analysis, and all ultimately getting to actions.

Would you say from a security operations standpoint, leveraging something like AI might have facilitated, being able to detect it sooner, being able to stop the bleeding quicker? Uh, what is, what is your take there? I think AI is gonna improve, you know, threat hunting and, um, and pattern recognition and patterns that shouldn't be there, um, exponentially as we move through it and it's gonna get better and better and better and faster.

I mean, threat hunting, you know, without AI or machine learning is a really tedious job, right? It takes a lot of manpower, it takes a lot of skills and training and time being in the industry that, you know, what you're looking for. Um, machine learning has, has helped dramatically, and I think AI is gonna help all that much more. Mm-Hmm. And we, we touched on the, the indicators of compromise earlier. Uh, they did allude to there being potentially more indicators that they're gonna release.

Uh, any estimation as to why they might be waiting to release it? Is it just additional checks and analysis that they're waiting before they send it out to everybody before they have to retract it if they, they jump the gun too soon? Yeah, I think, you know, it's, it's not a simple task, um, to bottom out forensic data and make sure that you're a hundred percent correct and you want to be, you know, you can never be a hundred percent correct, but you want to be five nines. Correct. If you can.

What, what's one of the biggest problems that everyone faces who works, um, you know, in the direct cyber monitoring and then remediation industry is dealing with noise, like getting, getting the signal to noise ratio figured out is tough. And if I'm Microsoft, I don't want to throw anything out there that's gonna make that mission even more difficult. Um, you know, they have a huge security team as, as we all know, and they're super sensitive to it because they have to deal with it every day.

So I, I have, I have, I would much rather that they take 10 weeks to bring forth a set of IOCs, um, that are additional to what's already been put out that people can put the use in their sims and, uh, in, in products like ours. Um, then put out information that isn't fully baked. A hundred percent. I remember, uh, leading threat intelligence for a lot of organizations.

And you get these feeds and, you know, you get premium feeds and you got freemium feeds and sometimes you'll get Google's Quad eight and you don't wanna block Google's Quad eight, uh, from, uh, on your, at your firewall. So you definitely have to do your due diligence when it comes to IOCs. But Phyllis, I'd love to ask you, uh, the same question. I mean, you spent a lot of time at the National Security Agency, so you know, a thing or two about telemetry and, and leveraging it the right way.

Uh, what do you have to say about the, the indicators and even the, uh, activity itself? And, and by the way, Chris, can I just jump in Phyllis, anything with the MSI SAC and feeds too? Like does it any pieces parts to Chris' add a great question, but anything you might be hearing there? 'cause obviously you guys get some massive feeds into that ISAC as well. Yeah, sure. And I, I agree 100% with what CHIP says.

You really, when you put out these IOCs, you really have to make sure that they're valid. Um, the last thing you wanna do is send out something and then people are like, well that didn't work. Um, and certainly that has happened. Certainly US government has been accused of not putting out, um, helpful IOCs quite honestly. Um, and you know, the worst thing, it does seem like people don't like it when you hold them back.

However, I do think it's important to dot your i's and cross your T's because that's really how you build trust, right? Um, you know, a lot of, um, I would say ISACs and like the whole idea of threat sharing and sharing those in indicators of compromise. A lot of organizations, um, don't like to do that because they're worried that what they send out would be bad, perhaps doesn't work in your environment. A lot of folks test them out first, right?

To make sure they work out in their environment as well. So I agree that you really have to be careful. I think Microsoft is just being careful at this time. Um, for, um, the MSI sac, um, which is the other side of CIS, we do have, we do create those IOCs a lot of times. Um, we have, um, kind of like, um, those, um, idss in place and we'll put those automatically in for the states 'cause they're sitting on the boundary. Um, but for sure we test and vet those as well.

Um, it's, we, we do also put them out. There's like a sharing platform in which, um, you could subscribe and Ms. IEC put those out for the s ltts, but it's not easy, you know, not everyone also is equipped to actually understand what an IOC is and, and, you know, put it into their own system, et cetera, et cetera. And so you're hoping either, you know, your managed service provider or MSSP is going to do that on your behalf. Mm-Hmm. Yeah.

Wrangling IOCs and dealing with them that it could, it is a full-time job. It really isn't, unless you have something like AI supporting your, your, your mission and even then you wouldn't wanna completely offload that entire operation to, to a machine to begin with. Right. Chip, I definitely wanted to ask you, 'cause I'm, I'm sure there's a lot of MSPs that are listening right now.

They're thinking like, all right, you know, obviously it's tough on an individual customer, but now I'm an MSP that has multiple customers. Was there anything that an MSP could have done on the front end to either expedite the incident response? Uh, is there anything that they should be doing from an SOP perspective when it comes to this stuff?

Because I, I know that a lot of MSPs, they might be it first security might be their secondary language, but is there anything that they could have done on the, on the front end or back end to make it easier on them and their clients? Um, you mean if they were impacted by, by this particular event? Or you, you're saying in general? Just in general, but just keeping this event in mind. Yeah. And or going forward chip with identity.

Maybe just even take, take the Chris initial question and which you know, is a great one. But, you know, we, we, as MSPs, I think for both onboarding and offboarding, this is a, an important topic. Yeah. Look, if, if you are running an MSP and you have, you know, anything more than a small handful of cu of customers, you know, if you're more than one man show, you've gotta be thinking about some kind of automation in terms of, uh, at least monitoring, if not response.

And I'm, you know, I'm not plugging our product or anybody's product, but you've gotta have some kind of product in the background that is either machine learning or AI based. And there are products out there that, that fit that definition. Um, that's doing the yeoman's work of looking through all this log data and helping surface for you what you need to follow up on.

Um, I'd say the, you know, the biggest question that, that we deal with in, in our business, um, in terms of support is coming from MSPs who see something and it looks like a duck and it quacks like a duck, and they reach out to us and say, is this a duck? Um, and they do that before they contact their customer. And you know, that's the wrong way around. Security events are a great opportunity to reach out to customers, even if you're not a hundred percent sure that there's something going on.

And guess what, even though you know products like the products in the marketplace that support MSPs for this do a a decent job of reducing the noise, there's still gonna be noise. They're still gonna be signals that come in that need attention, and you have to reach out to your customer. You can't be embarrassed for them to say, oh yeah, you know, that, that, that is our CEO traveling in Mexico right now. He really is down there.

You know, this, this is a new thing, particularly in the, in the SMB space that people aren't necessarily accustomed to the risks that they face and to the fact that it's a two-way communication street with their support partners with their MSPs. So, you know, I think rethinking those things are, are gonna help and making sure that you have some kind of tool in place to take advantage of increased logging that's available. So that's coming.

There's logging that's available now you can turn it on on your own. It's pretty difficult to say, okay, we're gonna turn these logs on and now we're gonna put our whole team on a rotating schedule of having to sit there and go through 'em all day long. Like, that's not practical. You have to find a way of getting some automation to help surface, uh, the events that require follow up.

But chip, two things I'd like to interject here from sps, because a long for, I would say throughout the last eight years that I've watched this, uh, in terms of logging and sim, right? Historically it has been about regulated customers, right? In other words, I've got these X amount of regulated customers, I choose vendor X and I can't, you know, and as the vendor, I want, like, Hey, why isn't this being sold more? Well, it's hard to sell. It's expensive, right?

I can check a box with compliance, the auditors are coming in, I need it. This is a great, um, way to do challenge. We talk a lot about challenger sale, right? This is a real world situation to go to your customer and go, Hey, did you realize this happened? And for the first time, Microsoft is exposing, you know, this information. Let me tell you what it means to your business.

Again, don't go into what means and this and that, but talk to them about risk that this would've had to their business that they could understand, right? Like if they were a intellectual property law firm, if they did m and a as a law firm, put it in their language of what that exfiltration would've meant to their business. And oh, by the way, I'm not saying this is inexpensive Mr. Or Mrs. Customer, but this is the implications. Can we talk about it?

And now all of a sudden the law firm would go, wow, that's a lot of, you mean all of our m and a activity would've been exposed? Yeah, all of it. 'cause that's what happened. You know, there's a couple of things that our, um, our sales team and our account reps are trained to do. And, and one of them is to, to reinforce to MSPs all the time that the MSP's customer certainly, and maybe the MSP, have liability now if they aren't taking these steps.

This is one of those scenarios where the, the more people you protect, the better the protection gets, right? I mean, think about how the whole kill chain of a business email compromise works. You get into one business whatever way, and then you start extending yourself through next business email compromises using the original foothold that you grab. And it's kind of a community thing. I mean, if, you know, if you don't, even in a large city, it's a community thing.

If you, you community in the sense of you're running, I don't know, you're running a tire store, right? And all the tire stores around have similar vendors that they're working with, that they're buying product from and similar customers. So you go from one to the next and you just keep extending and extending and extending. So anyone that's not covered by, um, an MSP who is security conscious is putting other people in their direct community at risk.

If you're in the contact list, if you're a vendor or a customer, now you're at risk. 'cause someone got a foothold elsewhere. And Microsoft calls this out now directly in their user agreement. I mean, they tell you straight out, if someone else is damaged because your account was compromised, you're gonna pay for it. Yeah. Mm-Hmm.

You know, that is such an important, you know, component and it's a great way to help convince the MSP customer, you know, the, the law firm, the medical office, the tire company. It's a great way to help them understand there's cost and consequence now. So, you know, you can't, you can't run with the theory that, oh, my business is too small, no one's interested in it. Everyone's, every hacker's interested in any foothold they can get anywhere. 'cause that gets 'em to the next step. Yep. Yeah.

Now that, Now that you have automation, you have ai, now you can do things at scale. A lot of folks aren't talking a lot about business email compromise. We're doing a lot of work on the hunter side to protect folks, but people aren't talking about it as much as, say, ransomware. 'cause it, it isn't as, um, inflammatory right in, in the media. But if you really look at business email compromise, they're shuttering businesses in some cases.

Because I mean, if you miss out on a payment and you're a small business and this is an important client, you might be out and, and how are you gonna pay your people? So there, there's a lot to think about when it comes to business email compromise. Chris, if I could just interject there too, again, this again for MSPs, like again, we gotta be good storytellers. This is an opportunity to go down the path of supply chain, right?

Helping the client understand, as Chip was saying, like, let me help you understand that even if you don't think it'll happen to your business, let me unders, let me explain to you the downstream impact, right? Chip and, and, and so that because, because again, it used to be like, oh, well ransomware, well, it's, it's, it's just you as, as Chris is, is saying, well, these, I these identity or account takeovers, chip have mass or can have mass implications. Fair. Yeah. Absolutely.

Because again, they can keep extending and extending, but they can also act like ransomware. I, in the end, what, what are hackers after? They're looking for a way to make money. Yep. Um, and they make money via blackmail. You know, ransomware certainly is, is a, you know, it's kind of like getting punched in the face, right? Ransomware. But I would compare the, a business email compromise where someone is in an account for, for weeks or months or even years, that's a cancer.

Like you can get over getting punched in the face. You know, if you don't, if you don't root out the silent killer, sooner or later, it's gonna get you Yeah. And that's gonna see the difference between the two and Verizon data breach. Right? Phyllis pointed that out this year. Like not saying ransomware is going away, but it's, it certainly has dropped relative to social engineering. Mm-Hmm.

Yeah, I mean, look, people are using the cloud itself as an effective strategy to avoid the consequences of ran ransomware. I mean, if you're backed up properly, if you're using 365 or Google Drive or whatever it is that your product is, and most of your data is in the cloud, you're a lot less susceptible, uh, to the impact of ran ransomware. So I think that's part of, you know, why you're seeing the change.

Um, not that it, you know, not that it's ever gonna go away completely, but we've learned to mitigate against it. So hackers are looking for other ways to find, uh, you know, mechanisms to make money. That's what it's all about. They, they're running a business too, like, don't forget that they're on the other side of us, but they're running a business. That's what they're doing. And they're, they're pretty good at it. Chip, unfortunately. Yeah, they're Alright.

Chris, sorry to, to take that away from you. Where are we at here, chip? Uh, and so I think you were asking, you asked about MSP, so Phyllis back to you for the final few here. Yeah. Um, you know, we just mentioned the V-D-B-I-R, the Verizon Data Breach report, and, um, why is it, do you think that, and, and in that report says, you know, someone just recently said, you know, you can't say I'm too small, it doesn't affect me.

And that report was like, you know, large and small enterprises, they're seeing the same thing. They didn't even break out a separate section on that. And so why is it, you know, we have all this data, why is it that MSPs, um, have and haven't taken, you know, M 365 and cloud monitoring more seriously? We have the data saying that we should. So what do you think that barrier is J that's for you? Sorry. Oh, I thought That was for Chris. I can ask that too. Yeah, go ahead.

Yeah, on that, there's a couple things there. I would say. I think people are taking things seriously, but there's just so much data, there's so much to look at and it's not easy to analyze. Even if you look at something as seemingly simple as asset management, it's really hard to wrangle. You still have folks from an asset management standpoint, they're giving you a range of how many assets they have in their organization. Now you want to look at something that's pretty much virtual.

When you're looking at things like identities, you're looking at email, you're looking at cloud infrastructure, all that stuff changes very, very rapidly. So having a handle, and on number one, uh, one of my favorite lines from the Lion King is everything that light touches is our kingdom. As an MSP, now you're, you're not only having to understand just one customer and what is in their environment. You're understanding multiple customers in their environment.

And now once you understand even if you have a good handle on what that area of operation is now, how do you analyze it? How do you ask questions of that data to be able to take, um, that information, that context to make a decision or take an action in your organization? It's really tough to do unless you have some type of automation or some type of solution or someone that's wicked smart in this area to be able to pull that stuff together.

And they're probably gonna use one of those two things to do it. That's awesome. Thank you for that answer, Chris, in the, in the last couple minutes. Um, I'll ask actually both of you this, I'll start with Chip. Um, close this out with your thoughts on MFA your, your SS e report indicated, um, similar to CSA that only 32% of orgs have MFA fully implemented. Um, and what are your thoughts on this and how do you see this, um, affecting everybody?

We're just talking about business email compromise, which we know MFA could be a good mitigator towards that. Yeah, I mean obviously I, I don't think any of us need convincing that everything should be done, um, with a minimum of MFA. Um, we're seeing an uptick. I think, you know, by the time our SE report is out again next year, we're gonna see those numbers go up. Um, you know, hopefully that's a broader impact than just, uh, our customers.

You know, we've, we do a lot of education on this, Andrew, I know you do a ton of education on this. Um, so, you know, I I hope that the, the world is coming around and changing. Um, what I really hope is that people are recognizing that you need some other second factor authentication, whether, you know, we think of it as in the way Microsoft sort of traditionally does things.

Um, I really hope that most of the world does a leapfrog right over all of it and jumps into, into Fido, uh, and goes with, you know, hardware token authentication for everything. I mean, I think Passwordless authentication, you know, you know, on the Fido model, on the Fido two model is, is where we need to get to as fast as possible. And I believe it's actually more convenient than how people experience using MFA today.

So that's, that's my hope is that that's the way it will take off and we will reduce identity theft and be business email compromise and so many of these issues, um, you know, much more aggressively and effectively through through that technology. Thanks, Chris. Your thoughts on that? I've been playing around with this concept of, uh, being a bad neighbor. So like, let's say we lived in this like metaphorical neighborhood and we know that there's crime going on.

Uh, if I was the owner of a home, I might want to be a bad neighbor and solidify my house as much as possible, so they'll go to the next house. I don't want 'em to come to my house. I want 'em to go to the next house. But if we do that at scale, if I'm a bad neighbor and my neighbor's a bad neighbor and everyone starts to use things like MFA, they become a harder target.

We all become harder targets and it's gonna be that much more difficult for, uh, a criminal hacker to do the things that they want to do. They're gonna have to, you know, second guess their their operations, they're gonna have to move on to something else completely. So I think the more of us that are bad neighbors and we use things like MFA, we use things like canary tokens, uh, it's gonna be a lot harder for these, these criminal hackers that do the, the stuff that they're doing today. Yeah.

And, and the al uh, you know, ironically Hunters and neighborhood watch Chris, but, uh, neighborhood watches do work, right? Yeah. I mean, so, so it, it's, it's a great analogy to close us out with. I mean, you know, we've seen the efficacy and effectiveness of doing that kind of stuff together, so. Alright, well let's wrap it up there, everybody. Awesome. Awesome.

Chris Chip, I can't thank you enough for coming on Phyllis as always making it fantastic and look forward to seeing you guys all back here next Monday. Um, don't forget about Bo Bullock, Thursday, Chris and Wes on the road and right of boom, uh, you uh, we would love to see you guys at the MGM in Vegas this year. Take advantage of early bird and uh, we'll look seeing you all very, very soon. Take care. Thanks guys. Thank you. Thanks everyone. Bye-Bye.