Challenges of Managing M365 at Scale

Welcome everybody. Episode one 11 here on the cyber call. And it's cool 'cause I'm not in a hotel room back to back weeks. Uh, that was quite a, uh, stint there, hun Gary, we were going from show to show and Yeah. I'm sorry. I couldn't, uh, I couldn't jump on. You were literally right outside the room that I was presenting in, but I was double booked. Got it. Well, and you guys did a great job. Um, what was the, your perspective on Da Ocon? Did you, did you get some good feedback?

What, what was, I mean, from my perspective, it was Da Ocon. Um, you know, I've been running a track there for probably four or five years, so, Mm-Hmm. Uh, to me it was business, you know, Datto business as usual. Yeah. They did a really, I thought they did a really good job.

Um, I'll say the one thing that I shared, you know, in my weekly audio message to our, uh, customers is that, um, every time I go to these events, more and more, I'm sensing this growing gap between people that are seeing this amazing opportunity from all the changes right. In our industry. Mm-Hmm.

Um, and then other people that either don't see it or they're questioning solutions to problems that we solve 10 years ago, like they're living in 2012, but their competitors and their customers are living in 2022. We all gotta get moving. Yeah. Yeah. Yeah. Very good. Um, okay, so just a few quick things. Gar, I lo I know you love my announcements. Um, one, we're gonna have a, um, cyber cast out on Control 13. It's in production right now that's on network monitoring defense.

It was really good. Um, Phyllis, uh, Ryan was back in that, and Wes it was, it was really well done. So that'll be out. I'll get that to everybody shortly. Um, I just want to give everybody a quick purview. This is gonna be live this week. Let me know if you can see my screen. Yes. Okay. So write a boom is gonna go live this week, thrilled about it. Um, we're really taking a, a lens through a threat actor, how they attack, but then on the other side, how you can defend and take it to market.

So we're really gonna look at from a growth and profit profitability side, and Gary was really succinct and telling me, Andrew, don't make it complicated. Don't make it complex. So I took that to him. Um, we're gonna have an incredible pre-day event with John Strand and John Hammond for those of the more technically, um, noted.

And then Sun Neil is going to be doing something specific on how you build your tech stack and services to defend against the threat actors that we're gonna present in Write a Boom from a lodging perspective. We are going to have it at the beautiful, uh, Gaylord Texan in Dallas. So that is just what I wanted to show you guys that'll be out there, um, ideally later this week, barring any hiccups, which of course, as we know in technology never happen.

Um, all right, so with that, let's get right on into it. First off, thrilled to have Aaron Cherin in doing his Wes Spencer impersonation. Wes, uh, how are you bud? Good to see you. Hey, you've got Aaron here, Andrew, thanks for the invite and I'll try my best to be a Wes For you. No, Wes Spencer. All right. So you'll be talking a lot about crypto and uh, I'm not sure the ship. Yeah, yeah, yeah. We loves his chili, so I'm gonna be talking a lot about that skyline. Uh, fantastic. Okay.

So, um, I got the pleasure to meet Jeff and his team at Simeon Cloud, and, um, I thought it was really apropos to bring him on the cyber call because, um, I'm gonna go on a limb here, Gary. I mean, Microsoft in essence for SMB and MSP is pretty much taking over. Is that fair statement? I mean, yes. They continue to, I mean, it's, it's becoming ubiquitous.

And so being able to manage M 365 at scale, um, especially when it comes to configuration and, and regulation, um, change management and all the things that we are going to be dealing with and r dealing with, uh, as MSPs. I just thought it was very apropos. Um, also apropos to bring Aaron on because again, automation, uh, when I was talking to Jeff, I was like, I literally thought back to 2016 when I met Aaron and his story coming from large financial services. So does Jeff as well.

He is owning an MSP and he's like, yeah, if something, you know, we do it multiple times, we should probably automate that function. And that was Aaron's entree from it into security and, um, so it was really serendipitous. So that is my setting the stage here. Uh, Jeff, welcome. And it's awesome to have you tell us a little about yourself, um, and how you got into this crazy world as a vendor to MSPs. Sure. Well, uh, thanks for having me.

Um, I've been in the industry for about 20 years now, as you mentioned. Uh, I've been an MSP myself. I've also worked in software engineering as a developer and as an architect. Um, I've worked in large financial services most recently, and I've worked a lot with Microsoft 365, as you mentioned, it's really taking over, uh, stealing market share from previously independent solutions like Okta or VMware. Um, sorry, it, I'm getting a little bit of echo. Are you hearing any echo? We're not.

The only thing that may be Jeff, is if you have two windows of Crowdcast open, that could be the trick. Is there any chance you have two? Wow. You're definitely a good troubleshooter. That's all right. I can, I anything. It only happens every week. Any MSPs out there looking for a good technician? Call one 800. Ask Andrew. I'll be ready. All right, go ahead Jeff. Thank you, Andrew. Uh, that's much better. Good.

Um, so in working a lot with Microsoft 365, both in the enterprise as well as managing, uh, customer tenants as an MSP myself, I saw that there really was missing the kind of maturity in automation that exists for other areas using tools like Terraform and Lummi and Ansible and chef and ad nauseum.

And there really wasn't a tool set that covered the Microsoft 365 stack, and there wasn't a tool set that really was broadly applicable to a wide range of audiences, whether you are a DevOps engineer, a software developer, or an IT admin. And so that's why I started Simeon to bring the benefits of configuration as code to the Microsoft 365 stack and also to bring it to a broader audience. Very cool. Very cool.

Alright, Gary, you're big, uh, in configuration is code, so I really wanted to start things off. Um, yeah, hand, hands on keyboard kind of guy with Obviously the most technical person on the call. Absolutely. Yeah, Definitely. So, Um, so Jeff, I have some questions for you before that, Aaron. Uh, can I just take a minute?

We're assuming everybody here knows Aaron, um, maybe, maybe Aaron, can you just take two minutes and just maybe so everyone who doesn't know who you are, uh, can know your background. Yeah, That's a really good, good idea. Hey, everyone, you've got, uh, Aaron, I was the founder and CEO of, uh, perch Security. Um, recently, over the past 20 months or so, I've started a new company called Roost, and you can get to that@rewst.io.

And Roost is a automation platform for MSPs and we focus on integrate, integrating, and creating automations, um, around the products that MSPs use today. Um, it seems kind of weird that, you know, you would go from a cybersecurity company to an automation company, but my background is, uh, running teams called, uh, security automation, uh, within, uh, large financial services companies. Yeah, really cool. Gary, thank you for that. And I, yeah, feel awesome.

Completely inept now for missing that, so thank you. Thanks. So, um, that's what, what I'm here for Andrew, I mean, making you look inept. So Jeff, I mean, um, Microsoft Lighthouse, right? Uh, we've been hearing about this since, um, you know, probably shortly after the dawn of man that it's going to, you know, bring multi-tenancy management and really allow, uh, MSPs to scale their practice. Can you tell us, um, what's the latest, have they succeeded? Give us an update?

Well, uh, I I wouldn't say they've succeeded yet. Um, they'd have to have a, a fully functioning product, uh, in order to have succeeded. Um, so we're about two and a half years into it now, and what we can see is that they have some useful multi-tenant administration abilities to manage or get visibility into, uh, things like Intune across your tenants. And they have some very limited ability to apply, uh, concepts like baselines, which are extremely important to your tenants.

But the range of settings that those baselines cover is, uh, just, just a handful of settings when configuring a fully functioning environment takes hundreds if not thousands of settings across all different areas of M 365.

So what, what are some of those things, top things that Lighthouse should be, you know, automating or making available that is not So to, to really support the full range of the Microsoft 365 stack that an MSP needs to configure, uh, and, and secure and manage on an ongoing basis to have or to deliver value to their customers? Uh, you've gotta have teams management. You've gotta have O 365 management. You've gotta have end-to-end coverage of Azure ad.

Uh, and you have to have things like multi-tenant application packaging for Intune, uh, applying configuration policies, intune across, across tenants. And so right now the baselines really just include a handful of best practices that Microsoft, uh, wants you to be able to apply. So can you tell us, like with the MSPs you work with, um, of kind of what they're doing now in terms of managing multiple, you know, MSPs managing 10, 20, 50? I mean, my first MSP had 180 customers. It's difficult.

Uh, I, I've seen a really wide range. I've seen, uh, things like spreadsheets with thousands of line items of things that need to be configured, set up tasks for provisioning a new tenant. Um, I've seen a lot of people that have built homegrown PowerShell solutions to get them 60, 70% of the way there, uh, doing automation to provision a new environment.

And I've seen, uh, companies that have built in-house, uh, desired state configuration solutions with their own software development teams, uh, some really impressive solutions out there, but there really haven't been, uh, you know, a single thing that everyone is doing. Yeah. So can you talk specifically when it comes to security configuration, regulation, PII, retention, like those kind of specifics? Sure. So I mean, uh, it's really important to get the configuration of your tenants right?

And it's really important to maintain them, uh, on an ongoing basis. And to get coverage of PII, uh, DLP uh, secure authentication, you really need to cover all the areas of the Microsoft 365 ecosystem. Uh, there's not just one area, and it's not just at setup because as we know, regulations change and, uh, you need to not leave your old tenants that you configured a year ago, uh, in the dust. You need to keep them up to date with your best practice configurations. Yeah.

So before I pass over to Phyllis, you know, one, you know, statement that I'll make is what I hear sometimes from MSPs is, um, Andrew, they say, you know, some of our customers are pushing back, they don't have all their on-prem equipment. They're wondering why we, why we're raising prices when most of the stuff they're doing is in apps and in and in and in teams. Yeah. Because it's probably more complicated and they have more risk, not less risk. Right.

And we don't have the same level of process and automation and procedures in this world that we finally got to after two decades. Right? Uh, only now we don't have two decades, uh, to figure it out. We have to have process, and we're gonna have to have automation, um, and customers have to understand that they need to invest more and not less. That's where we are. Gary, quick question to you.

Are you, you know, because you built the standards platform, um, are you seeing, like in your peer groups, are they struggling with this? Are they talking about things like we're talking about right now? Yes. Yeah. Yes, absolutely. Yeah. And Calvin, awesome to see you and you certainly can, uh, be proud of CIPP and and what you're doing, so, yeah. Um, okay, Phyllis.

Yeah, no, I love this because, um, you know, at CIS we did work looking at top five attacks, and the number one thing you can do to help defend, um, is have a secure configuration, right? Um, it is so important for organizations to implement those. And, and you just heard Gary, which he says this whenever he's on the call, you know, implement those standards into your business. This is how you can differentiate yourself in the marketplace. He calls it your company way.

And so, you know what, very good Phyllis former You, I listen. Um, so, you know, what can MSPs do when they need to automate, you know, at large scale and they have so many customers? Are there open source, um, tools that they can use? Or are there cots tools? Like what is it that they can do? Well, Pretty much everything that you need to configure is exposed in some way or another via an API or by a PowerShell.

And so, as I was saying, I've seen a lot of people write some really impressive PowerShell tooling that, uh, configures the full range of the Microsoft 365 stack. There's an open source solution called Microsoft 365 DSC, which is, uh, very, uh, robust in its coverage, um, except with the area of Intune. Um, and frankly, there's also Simeon. And the reason I built Simeon is because of the gaps that exist in, in those other tool tool sets. So, um, you know, it's hard.

Many MSPs don't have a lot of automation. Some have the resources they can lean on, some don't. Um, can you talk to us about how difficult is it, how many resources do you need to automate M 365? You know, you have to have change control, all these sorts of things, um, possibly even measuring drift over time, right? You know, um, you mentioned like a year later, or maybe even sooner than that, you need to check in on your customers.

Um, how is it that organizations can do this, um, in any sort of manageable way? Well, uh, I mean, you, you need to be able to check on how your customers compare against your current best practices. At any point in time, you should be able to look at a dashboard and see that, uh, just, you know, visually, whenever, whenever you're, uh, whenever you're looking at the state of your tenants. Um, to do that, uh, is, is a really heavy lift.

Even if you are a PowerShell scripting expert, or if you use Microsoft 365 DSC to actually build the automation around it that's required to have a, a fully functioning, uh, or fully functioning continuous integration and continuous deployment and continuous monitoring, you have to be a PowerShell developer, and you have to build, uh, a lot of it yourself. And it's totally possible, um, that this is where Simeon comes in to make things easier and give you out of the box end to end automation.

Um, but, uh, you know, you can need, uh, if you use a tool like Simeon, you just need non-developers that are familiar with the admin portals. Um, if you're going to build it yourself, you need software engineers or DevOps engineers at least. So when you talk about continuous monitoring, this is a term government uses all the time. You see it in so many frameworks. Now, what do you mean by, I'm just, you know, I'm curious, like, is it daily?

Is it you're having, you've got agents that are just measuring Drift, you know, how often, like for an MSP in particular, who's managing, you know, Gary said his first MSP, you know, he is a high achiever, 180 different customers. Right? So, um, you know, how is it that you're going to continuously monitor 180 different customers? So this is, uh, it's a good question. Um, all of the solutions that I've talked about have some sort of synchronization process involved.

And so that runs on a schedule because getting push events from actual changes that are made in the tenant is very difficult. So it's important if you're going to be doing synchronizations, um, I mean, by default, uh, Simeon does them daily, but we allow our customers to configure, uh, whatever time period they want down to 10 minutes.

But in order to do that, it's extremely important that you're, uh, your inspection of the tenant can run quickly, extract the current state, and then you can run it frequently to make sure that you're at least, uh, as up to date as possible. Okay. Yeah. And so, um, in your experience when it comes to, um, Microsoft managing Microsoft in a cloud environment, um, how, how is that, is it harder, easier? How is it managing, you know, um, in a, in a different environment like Microsoft?

Oh, it's very different. Um, you know, if you look at, uh, on-prem administrations, uh, administration, all your software is installed on virtual machines or physical machines. It's entirely within your control. Uh, the Microsoft 365 platform is stitched together for many different products that were previously on-Prem. And they're, uh, you know, they, they, they look like they're working as one platform, but they're really many different products.

And so each product has its own form of administration, its own form of APIs to configure it and its own, uh, functionality to manage it. Uh, and usually also its own portal, uh, to configure it. Um, so, you know, you get the benefit that you're not working and maintaining your physical and your on-premises infrastructure. Um, you also get the complexity that comes with managing so many different products in the cloud. Yeah, I mean, it's, it's true.

Even just trying to figure out all of that, it's so complicated. Um, and so what, in your experience, how could you recommend, you know, what kind of processes, you know, what kind of, um, uh, you know, what, what's, what's the scope for these MSPs on how you really wanna manage that Microsoft stack and have a, well-defined SOP or standing standard operating procedure? How is it that organizations can really try to systematically look at the problem?

'cause it's very overwhelming, like you said, if you looked at M 365, it's ridiculous. All the different products that are out there, and it is, everything has their own way of, of, of managing and, and you know, honestly, they're different development teams as well, so it's probably like different dashboards and, and different ways of managing. So how is it that an MSP can tackle this problem? Sure. Well, you know, having a unified single pane of glass is sort of the holy grail here.

And that is what our, our product tries to offer. Um, that said, no matter what tool set you're using, you need to be cognizant of a couple of things. First, uh, first off is, is baselines or the concept of baselines or a golden image of your most recent best practices. And you need the ability to provision new tenants using that baseline, but you also need the ability to make sure that no tenant gets left behind.

Uh, and that every tenant can get the most up-to-date version of that baseline using a few clicks or a few lines of code or whatever it is that you're using to manage your environments. Um, also, you know, when, when you have, uh, the baseline management in place, you also wanna look at processes like having non-production environments where you can test things. I know a lot of people that are still testing, uh, new changes that they wanna make in directly in production.

And, uh, it's, it's been hard to have replica non-production environments without any automation to provision them and make them look just like production. Uh, so you want to, you definitely want to have processes, uh, for change control and change management and have those non-production environments. Uh, auditing, backup and restore are definitely other pieces of the puzzle.

Um, like I was saying with on-premises infrastructure, you could always just sort of revert your server back to a previous point in time. It's not so simple, uh, when you're managing a product like Microsoft 365 in the cloud. Got it. Thanks. Um, just a quick reminder to do the poll. I just have a quick question for the audience. I see like, um, some folks are saying that it's almost 50 50. Um, no, they're not struggling to manage M 365 at scale.

I'm curious if you could just respond in the chat, those orgs that are not having issues, how is it that you're actually managing it? Or is it that you're just not even touching it anyway? Yeah, Or if somebody has more than, you know, 40 customers and is saying it's not an issue, Andrew, maybe they can come on and talk, talk about what they're Doing. Yeah, that, that's great. That'd be cool. Yeah, if you have an MSP that is doing it, that'd be cool. So definitely. Yeah. Um, Yeah. Thank you.

Curious, go ahead. On on, because you guys obviously have your go, you know, you know your images and Yep. Is this something that you guys focus on in terms of, or is it just os because I know you have OS and browser and, but um, right. Perspective? Yeah, no, we are working on Azure services for sure. Um, M 365 being one of them.

So, um, we have a very strong partnership with Microsoft and Azure on doing exactly this, is offering secure configurations and offering those, um, secure baselines as, um, everyone is talking about so that, um, Microsoft can just give it to you out of the box. Someone said that in the, in the chat, um, it is a lot for organizations to manage, especially for an MSP.

Um, and so the best, the best would be, you know, the secure configuration out the box and um, then, you know, possibly measuring drift over time if you allow your customers to, to change their configurations or, um, change a configuration by exception only. Right. And so, um, and then document, why is it that you needed to roll back a security setting? So just one other question.

Are you, do you hear this challenge at like in MS isac, you know, or large organizations that you guys talk to as well? Oh yeah. This, this challenge happens is, is everywhere having a secure configuration and being able to maintain it, it is 100% a huge issue across the board. Small organizations, large organizations, et cetera.

So, I don't know if you remember years and years ago when Google had their big hack by administrators in China, and that was because, um, you have a lot of companies like Google who have software developers who wanna have admin privileges. And so those folks don't get a secure configuration. 'cause they'll argue that I need access to everything, right? I, you know, a best practice even at that time was don't do development on your real network, don't do development on the production network.

That's what happened with RSA when RSA keys were stolen. Like, you know, they were doing development on the actual real production network, right? And so, um, you know, while that was perhaps not a configuration setting, it was a best practice and they should have configured, you know, no admin, all these other things, right? So there, you know, if you had had secure configurations in place, then those exploits would not have happened.

So, you know, big companies that want to have the flexibility for their software developers, other companies that don't want secure configurations for convenience sake, and also, um, you know, you have to measure drift over time, right? And so this is like the classic problem where new software gets, um, loaded, you don't know what to do. Or another piece of security software.

So in the US government, when we had a contract with, um, McAfee, McAfee rolled back would not work with, um, DLP and I forget the other, um, pass the hash whatever with Microsoft. So what did the government do? They turned those security features off, even though the configuration was keep those on, right? And so we have so many, you know, products that we're trying to help also lock down our environments, especially these large enterprises. And then what do you have to do?

You have to up, you have to roll back other configurations so that product can work, right? Because that product wants to be, um, the admin or the super user or route on that box, that product wants to have access to all this data so it can, you know, pull back data to whatever, some central data stores. So it's so common, true across the board all the time, and organizations really need to understand, um, what's going on.

So it's nice to have the secure configuration, it's, um, important, but someone needs to also understand why those configurations are there, right? So roll back a configuration, what does that really mean? Phyllis, last I think, oh, sorry. I was just gonna say real quick, Jeff, and then to you Phyllis, I just wanna ask, were you in Belize with McAfee, you know, doing this consulting work? Right? Go ahead, Jeff.

I was just going to agree with you that, uh, you know, no matter what you're using to apply configurations, having an exception management process where you actually understand why exceptions were made when they were made and have visibility into that is extremely important. Cool. Aaron, so this is like, this is a, like, I mean, something, I hear you dealing with a lot, you know, like struggles that MSPs are having and, you know, automation, period, onboarding, offboarding.

Just do, do you see these kind of struggles that we're seeing? And then like, let lead on into your question. So are you seeing this as well? Uh, definitely. Uh, as a founder of a, a group of folks building an automation platform that has a ton of joy integrating with Microsoft APIs, um, we see, uh, a ton of that. And speaking of, um, fun of Microsoft APIs, here's a question for you, Jeff. Um, does Microsoft make it easy for you to detect, uh, the changes in client's environments?

I wish they did. Um, there are a lot of different APIs and every product and API has its own, uh, way of doing things. There isn't really one consistent standard, uh, even within Microsoft Graph, which is supposed to be their one unifying API things are very different things. Uh, you know, uh, they're, they're changed, uh, and breaking changes are made pretty frequently, um, you know, on a, a weekly basis. And, uh, so pulling data out of those Microsoft APIs can be pretty difficult.

Uh, and keeping it working over time, uh, can be difficult as well. And this is one of the difficulties if you're maintaining your own PowerShell scripts, uh, that provision and manage your Microsoft tenants is you also need to maintain them. Uh, and that can take a lot of effort. Uh, definitely. Um, so, uh, tell me a little bit about configuration as code. And if MSPs were to attempt to do that at some point, do you see like a maturity model or a, a method of adoption that's consumable?

Um, I, I never think of any new technology as like a light switch. One day you have it and you know, one day you don't have it. The next day you do. It's something that you ease your way into it. How, what is configuration as code and how could folks ease their way into it? Sure. So the way I like to think about configuration as code is that your code describes the what of your environment, what's in it, what state should it be in today?

And then a tool set or engine is responsible for aligning the actual production environment with that, what that's described in your code, and this is as opposed to something like writing PowerShell script where it's imperative and the scripts describe how to put the tenant in the state that you want.

Uh, and that can be simpler to write upfront, but when you're looking at things like drift management and realignment of existing tenants, if you're just describing the how, then it can become very difficult to understand the current state of the tenant. And your scripts need to become very complex to actually realign a tenant at any point in time with the desired state. And so this is where configuration as code and the underlying engine really helps you out and simplifies things.

Um, the text detection of changes and configuration as as code, all of this leads me to, like, think a lot about configuration compliance and best practices. Do you think there's maybe a high an, an o uh, even larger maturity model that may need to be met by an MSP before they even care about configurations? Do you think there's any prerequisites before they even get to configuration monitoring? Sure. So I, I, I think I look at it as multiple steps of maturity.

Uh, the first step is actually having standards. Um, so, you know, the, the customers that I've seen that have those giant Excel spreadsheets, that's a good first step because you at least are documenting what you want to be in all of your customer environments. Uh, and you know, if, if you have engineers following those steps accurately, you can put a customer environment in a secure state.

The next step along the maturity model, I think is writing those imperative PowerShell scripts to help you provision new environments. And then the next step of the maturity model is adopting something like a configuration is code engine, where you are modifying the what, uh, of the, that definition. It's almost like you've got that Excel spreadsheet in code, and the configuration is code engine can just apply it. Uh, it's also important to be cognizant of the different standards that exist.

So we see customers that are creating different baseline, uh, a NIST baseline, a uh, A-G-D-P-R baseline, and they're applying different baseline in different situations. And so that's really, you know, the, on the more mature side, um, where they're, they're actually looking at industry standards and creating baselines that correspond to them.

Aaron, question to you, if I could, do you, you know, that that maturity model is kind of interesting, like with the companies that do well with roost, are they, are you seeing that like they've documented process or using something and they're like, oh yeah, we thought this through because I thought I heard like Mic Michael Fishler as an example, what he did with some security automation. Uh, is that kind of synonymous with what you see Too? Yeah, there's two different, um, styles of maturity.

One comes from, uh, the cybersecurity world that ports, uh, immediately over to automation. And it's really, really basic. And I have so many conversations with folks. Um, for example, um, if you're buying a, a security solution, um, that covers a control like network security that you don't do today, it maybe it only meets 80% of your needs, um, 80% is better than zero. And you would get in a lot of those conversations with folks that, that decide to do nothing instead of something, right?

Um, and the same goes with automations. Um, if you, sometimes you can't automate something a hundred percent right out of the gate, um, but automating something to 70 or 80% is still less work. Um, at the end of the day, um, I wanted to go back to Steven's comment. It takes a village to manage configuration.

He's, he's kind of onto something there, uh, which is, I really feel like the prerequisite to configuration, monitoring and compliance is really teaching MSPs about this policy, about these configuration policies and how to sell them to their clients. Um, the client needs to be on board with the configuration policy just as much as the MSP is.

Because if the MSP detects a deviation and makes a change and it does disrupt business or impacts them because they can't get to their favorite BitTorrent site or something, um, then uh, the client's gonna be upset. But if you have an agreed upon policy and you show the deviation, it's a little bit, uh, better. So the client needs to be just as on board, um, uh, as the MSP with that Aaron for Sure.

It brings up a good point, like, 'cause Gary, you, you know, these, these are the, you know, risks to revenue conversations, right? That we hope MSPs are maturing to having, right? Because we, we see a lot of times controls implemented, but not a policy corresponding to the control. Do you have any thoughts on this as well, Bud? Yeah. And, and listen, I believe the solution is, is good. It look, technology is gonna be the easier part of solution. It's arriving, right?

Uh, we have some on this, on this call today, and it will continue to mature. The other side of it is, um, is the MSPs understanding focused diligence. Look, Andrew, if we go all the way back to like, you know, I was the first one to buy all the, every one of the RMMs literally went on to the next one, right? And invested a lot of time and energy upfront created a huge competitive advantage to the average MSP that I thought would only last for a year or two.

Turns out it lasted for the life of, uh, you know, of, of my MSP. So I believe an office 3, 6 5 is, you know, one big piece of it. What I'm telling our peer members is that's where we are right now with, uh, with a different way that people have to look and think about automation. Part of it is efficiency, but part of it is just what we're talking about today, right? Around managing security, around managing configurations. And this is a really big deal.

And the difference is we don't have 10 years to figure it out. Like when the first wave of automation came with RMM, we gotta get going, you know, sooner, sooner than later. So, Andrew, my more my concern is that second piece, which is MSP, slowing down to understand that the world has already changed and it's gonna change even more in terms of what we need to do. And we need to start thinking about the company that we need to be in six months in a year, in two years from now.

And it's really hard to do again. And, and I understand how hard it is 'cause I deal with so many people and they're buried in tickets and projects, but it's really important right now. Yeah. I mean, IG I'm glad you, you kind of ex put an exclamation point in that.

And one of the things we really wanna focus on and write a boom this year is literally talking, you know, the, the translation that business owners, you know, we can all sit here and talk about automation and configuration and the importance of all this, but the average business owner right there doesn't, but we've gotta get, we have to be able to translate you. Last thing I'll say to you, 'cause I love the way you did it was you have the true method sales translator, right?

We've been dealing, again, with the trying to talk the wrong, speak the wrong things to end customers. Can you just give us some sense of that? Yeah. And listen, when you get it right, the same way, like we were early on investing in proactive services, we learned by doing it the amount of time, effort, and focus that it took us, it was easy to sell the value of it. 'cause we did it. It's the same with this, it's the same with security. It's the same with automation.

When you invest and you actually do the work, it is easy to translate it to value because you know how hard it is for someone else to do what, you know, walk the same path, right? That, that you have walked. And we don't always get opportunities to create a competitive advantage. And we have a bunch of them, you know, a bunch of 'em right now. And, you know, I sit on the board of, you know, two private equity backed, you know, MSPs and we're all in, uh, you know, we're all in on this. Yeah.

Jeff, you had, you had a comment? I know I've stepped, stepped on it a few times. Oh, no, it's okay.

Uh, I, I was just going to, uh, uh, agree with what Aaron was saying, which is that when you have this kind of automation in place, the, the reliability, consistency and ease, which with which you can apply them allows you to focus on delivering greater value to your customers and giving them access to a wider range of services and, and bringing them into the, the modern cloud and, uh, and, and selling this, this value to your customer.

And, you know, I I've seen a lot of MSPs that have been really successful in, uh, both gaining customers and also, uh, you know, increasing the rates that they may charge to their customers to deliver all this additional value. So instead of upselling on a, a license, you, you can upsell on the value that you're delivering as an Cool, I just wanted to chime in.

I really love those perspectives because just at MSSP alert, I was talking to a couple of MSPs and what they said is, well, it's hard for me to sell security 'cause my customers don't want it. Right? And, and so I was like, where's, where's Gary? I Don't want security, don't want automation. They don't want anything. Yeah. That, that's not what they want. They want to be able to run their business more efficiently and not get breached. Yeah. Yeah. Yeah.

And that, and that, I'm glad you brought that up, Phyllis, because we talked about that on stage. You and I lost a version, you know, again, people don't wanna lose their revenue, their customers, their reputation and, and their business. And we've gotta learn to as, as MSPs to talk in their terms. 'cause 'cause you're right Gary, they don't care about it. They don't care about security, right? They, they care about their business.

Um, and that's, Listen, and that's why from a software perspective, and I spend time, you know, talking hours talking with, with Aaron, is that I feel like when you're developing, and I developed a, a, a software product, right? For, for MSPs, you have to, has to be based on meeting MSPs where they are.

And I feel like too much technology is aimed at some solution that that doesn't take into consideration the journey of where MSPs are and the constraints they have in their business that they need, that they need to change. Yeah. Very cool. Aaron, back to you. Um, you're, we've, we've stolen your, you know, time here. Sorry, Aaron. That's what I, I forgot what I was asking anymore.

Um, uh, let's move on to chat, like a different different line of, uh, of questions now that we've like detected a change, uh, because we love working with Microsoft APIs. Um, can you get those detections to a sim? Sure. So, uh, you know, if you're using a desired state configuration, configuration is code platform. Uh, when that platform extracts data from the tenant, you have the opportunity to act on that.

And so we see a lot of customers that wanna take that differential or that delta that's just been pulled out of the tenant and forward that on to their sim. Uh, so that, that's, you know, one of the key advantages of using configuration as code is you're getting a delta at any point in time and you can ingest, just ingest that data into a sim and it's structured in a consistent way.

Uh, I, I've seen a lot of direct integrations where it's pulling the Microsoft audit logs from various different audit sources directly and ingesting that into the sim. The challenge there is that those logs don't follow any kind of standard format. Uh, they're all different in their structure and their nature. And so, you know, with data, uh, you know, it's garbage data in garbage data out. You always want to have structured data that you can, uh, make intelligible and act on.

And so it can be really challenging to take the data out of the Microsoft audit logs directly and parse it in a standard way into the sim. Um, I'm stuck on configurations. I think. I think it's, uh, uh, I would love to go sell, um, best practice configuration policy to, to my clients. Um, but it's, uh, that is a big bite. That's a big thing to ask, um, because there's gonna be other MSPs that they could a client could do business with that doesn't force 'em to do that.

Um, do you see like a maturity model into implementing, uh, configuration monitoring that may not, uh, immediately impact the client, but still be beneficial to the MSP? Sure. So I, I fully believe and, and, and agree that you've gotta meet your customers where they're at. Uh, I think that that's a great way of putting it.

Um, and so the first thing that, uh, that, that we always do when we onboard a customer, and this should be true of whatever configuration is code platform you use, is you extract the current state of the tenant and you should have the ability to view that as a delta against your best practice configuration.

So you can then take that delta in a report and actually show it to the customer along with documentation of what changes you think, uh, or your best practices say should be made to that environment. And so you are already getting value from day one because you're at least capturing what's in there and you're capturing what's changing on a daily basis, uh, or, or on a 10 every 10 minute basis. But, uh, you can incrementally apply your best practices one at a time.

Uh, say, okay, today we're applying MFA and we're going to uncheck this box and bring in the MFA policies from our baseline. Uh, today we're going to apply DLP policies. We're gonna uncheck this box or check this box and apply our baseline policies. And so that, that, uh, incremental step or steps to getting your client's, uh, environment aligned with the baseline is extremely important. Otherwise, it's just too big a change all at once.

And you're guaranteed to get a p****d off client when you break something. Uh, extra credit question, do you have public APIs that we can integrate with and build automations off your platform? So we do not have any hosted APIs because we are at, at our core a configuration is code platform. And so we run in our customer's DevOps environment, we install into our customer's Azure DevOps managed instance. And so you can do essentially anything you want from there.

You can go and inject custom workflow steps. You can, uh, add custom workflows that run on top of the built-in synchronization workflows. So we're not hosting any of your data or any of your authentication information ourselves. So we don't have public APIs, but you can build pretty much whatever you want on top of our product. Great. Hey, hey, Kelvin. Hey Aaron. I see you. Hey, everyone can see Kelvin. I wanted to bring you on too 'cause you have a lot of perspective on this.

Um, wanted you to meet Jeff. Um, Kelvin comes from the other side of the pond, and I had the good fortune, Kelvin. We finally got to meet in person at that Ocon Yep. And sit next to each other. That was pretty cool. Um, Kelvin, what's your perspective on, you know, this, this, you have, you have a lot, is it challenging? And you know, why, why do you think more MSPs aren't, like when I look at the poll, is it that they're not just not doing it? What's your thoughts?

Um, so, so I'll, I'll circle back to first a little bit about that standards configuration that we were talking about a little bit before, because that's an important part of why I think a lot of MSPs aren't doing this or aren't struggling to do this. It's because they don't care. It's because they have other things to worry about at that moment. I mean, you have to manage, let, let, let's say the average MSP manages 50 clients. It's, it's a medium-sized shop.

They're, they're working hard, they're constantly fixing issues. They don't have time to be looking at the configuration. And a lot of MSPs are still doing a lot of reactive stuff. They're, they're looking at, at Microsoft 365, and they're going like, oh, hey, this person got hacked. Let's enable multifactor authentication. Oh, hey, this person got fished. Let's enable phishing policies. And it evolves like that.

It's not like, oh, hey, let's look at all our clients and deploy the best standard for them right now, because they simply don't have the time for that. It's, it's, um, it's, it's worrying. As a guy that worked on a tool like Simeon, it's, it's, it's stressful as an MSP because you see that a lot of other, um, MSPs aren't doing the same thing as you.

It's, it, it's a stressful situation because you know that a lot of your clients could be protected better if they would perform these sort of baselines. And I don't believe that the baselines should be coming 100% from MSPs. Um, kind of like, uh, Phyllis just said, um, they should be coming from the vendors themselves. There should be discussions with Microsoft.

There should be discussions with someone to create the default as secure as possible, or at least to create baselines as secure as possible. Now, I'm a Microsoft MVP, and I'm lucky to be talking to the Microsoft developers directly a lot. I mean, just this week I had like six or seven calls with them, like telling them I need this API to work with. I need this API to work with, I need to make sure that I can do this through graph, through PowerShell or whatever.

And they're very receptive, receptive of that. But there's no one that's, that's really sitting down with them and just telling them like, okay, hey look, these are MSPs. They're managing a lot of clients. They need specifically this, there, there, there's no one explaining to them what MSPs needs. And that's also why, why Lighthouse is almost a failure, I guess. It's, it's, it's developed very slowly. It's, it's not where it should be.

So, um, I, I think the biggest part of what we need to start seeing more in the MSP market is somewhat of a parental obligation by vendors themselves that are saying, these are the defaults live with it. And that's gonna happen in five days. For example, in five days Microsoft is finally disabling basic authentication after it being a source of security problems for years, they're finally saying, we're cutting this off.

And I'm noticing I'm rating a little bit, so I'll let other people talk because otherwise Yeah. I, I do, it's, I was just gonna say, it's really important to have an MSP's perspective and, and one that's security first. Like you, Calvin manages a lot of customers. Um, phys, I wanted to chat with Gary, like, I mean, the first thing he said was back to standards. And I have to give you kudos.

You, uh, it's, it's what you started, you know, what, two decades ago, but what's, you know, and, and, and Phyllis maybe handing it to you, like just in, in like the, the breaches we're seeing again, I mean like I was sitting there with John Strand and, and I said, John, in your IR team, again, you are the biggest of big, like, what are, how what percent were just confi, you know, credential, you know, MFA, just simple things that could have been stopped. And he looked at me, he goes, 98%.

So to, to your point, Phil. So let's just do that. Yeah, we're trying, right? That's the whole point of like CIS and Mplementation group one, right? Phyllis, but what were we gonna say phy? Yeah, I'm gonna say, you know, I also wanna say that it's not necessarily true that having a secure configuration is going to really degrade an end user's experience, right?

And so I think there's also, um, you know, at CIS, the benchmarks they have like a level one is in which it's not supposed to degrade the user experience. And so I think that MSPs could, could look at, um, any kind of secure configuration and you should be able to know, you know, if you turn on, for example, DLP or all these other things, um, that could help defend your environment, that that doesn't degrade the user's experience at all. Right?

So there are a lot of secure configurations that, um, or secure configuration settings that, um, are totally fine. And so it's not necessarily that you have to go through, um, a laundry list of, well, this is gonna be turned off, this is gonna be turned off. I mean, something like MFA of course, where you have to go an additional step perhaps, you know, you could talk about that. But I think there are many things, many security configurations that no one would even notice. Yeah.

But you know what I find, again, in talking to MSPs of all different maturity levels, the ones that are more mature in terms of the relationship that they have with their customers, they don't, they manage through this because the customers are understanding that anything that might affect them, and, you know, MFA's a good example. You know, the executives say they don't have time.

When you have the right relationship, they, they're understanding what that risk is and they, they do it and they move on to the next thing. I find it less Ms p matures, uh, less mature MSPs that don't have that same relationship. They just find it as a huge, that same thing is just a huge wall they can't get over. And then eventually just give up and say, well, customers don't want to do it. Customers don't want to be secure. It's, it's not true.

You, if that's what your customers are saying, it's, you have to say, it's me, it's not them. Yep. That's a great point. Um, I, sorry, go ahead. No, go Jeff, please. Oh. So yeah, having that trust with your customer is extremely important because when they trust you, you're partners with them. You're not the person making their life more difficult. And, uh, back to, um, what, what we were just talking about with, even with MFA, there are benefits to operating in that more secure model.

So when you turn on MFA and you stop using, uh, legacy authentication methods from your phone, you get single sign on from all the managed Microsoft apps without ever prompting you. Uh, and you can use Face ID to authenticate with your tenants, or you can use biometrics on your Windows computer.

And so when, when your customer understands that you can deliver value by enabling these features, and you're not just making their life more difficult, that's really the position you can be in where you can deliver more value to your customers. That's a really good point. Um, Phyllis, um, in a few minutes, I, I, I know we're gonna wrap up. Um, maybe just a prelude. Jeff, thanks a million for coming on Kelvin.

Also, I just wanted to ask you if you could give a prelude to next week why MSPs wanna show up for the ransomware task force. We're gonna have an incredible resource on Jen Ellis, but can you just give us a quick, I'm psyched for next week. Yeah. Can, can you, uh, can you give us a quick overview of it, um, as we wrap up here? And then again, I'll just thank, thank out everybody, but good. But please stay on 'cause this is important. Yeah, sure.

Um, IST, the institute for, um, I think, uh, security and Technology is a nonprofit. In 2021, they created the RA ransomware Task Force co-chaired by also Michael Daniel. He, he used to be the former cyber czar underneath the Obama administration. And, um, it was around, you know, hey ransomware, everyone is suffering from it. How is it that we can best defend against ransomware? So they created this big document they handed over to government.

It talked about policy and all these other things that, you know, um, you know, technology policy, everything needs to come together to help the nation, um, combat ransomware. 'cause they saw it as a national threat. And so, um, one of the recommendations was to create a, a framework to help defend against ransomware. So that's like the technology portion.

And so, um, we were a part of it at CIS, um, we created, we helped create the, um, ran, um, blueprint for ransomware defense, or Ransomware Defense Blueprint. I can't remember the title for some reason. I actually just was looking at it this morning, but I can't recall it right now. Um, I think it's a blueprint for ransomware defense, and that's really based off implementation group one. So we had participants from nonprofits from, um, you know, the, the commercial side.

We had like Microsoft, we had, um, cyber insurance, we had, um, tool suppliers, MSPs Ryan from Datto was, um, he reviewed our paper and Jen Ellis from Rapid seven, she was a big participant. And so it is about, you know, how is it the group settled upon implementation group one, we chose a smaller subset, we geared it towards, um, small medium enterprises. And we really just wanna talk about, hey, do these actionable things to help defend against ransomware.

We as a nation are concerned about it and we wanna help organizations defend. Awesome. Yeah, I'm excited. This is gonna be a big one. And it obviously is in, comes out of the community defense model, which is one of the top five attacks. So. Alright. This was awesome. So, um, first off, Aaron, thank you for, I, I, I have to say Wes, it might be on the hot seat here. What do you think, Gary? Could it, could this be a, Uh, Wes who, Awesome. Thanks for coming on, Aaron.

It was awesome to have you with us. No problem. It was fun. Yeah, you were great Calvin. Thanks for jumping in and as always, no sharing your perspective. It was great to see you Jeff. Thanks a million. That was really, really, uh, a lot of fun.

It was, uh, great to get your perspective on, and I think this is gonna be something Gary, wouldn't you agree that is gonna, just the way roost and automation is starting to take shape and we're seeing more MSPs gravitate to it, configuration is code and standards that things like Jeff, uh, is doing and Kelvin, uh, and others are, it's gonna be, it's gonna become the norm. It's All one thing, Andrew, where we have to get to with efficiency, but also with configuration, standards, security.

We need to approach things in a different kind of way and everyone needs to get on board. Yeah. Very good. Phyllis, always good to see you. Um, thanks for hooking us up with the ransomware task force. Um, Gary, always, um, great to have you back, man. It's, it's always great to see you. Everybody. Have a fantastic week. We'll look forward to seeing you all, um, this time next Monday. Take care everyone. Thank you. See you. Bye.