LA Fires – Lessons from the Frontlines of MSP Resilience

All right. So there's a slight delay. They're gonna hear that, but, um, you always have to wait for a moment. Let me just go to chat and, um, okay. Welcome, mark. Okay. Welcome everyone. Can you hear and see us? Okay. So I'll just check, do some quick checks, sound, video, everything going well for everybody out there. Uh, happy Monday to you all. Yeah, good, Eric. Sounds like we're all good, Mr. Monroe. Loud and clear. Okay, perfect. Thanks guys. And gals. Okay.

Um, it'll take some time for people to come in, but, um, I'll get us kind of situated with the, um, with, uh, setting the stage here and, and then doing the introductions. Obviously, um, these sessions are, um, I would, I would say, you know, they're never, never fun to say the least. However, um, I think it's really important, and I can't thank our guests enough for coming on and, um, you know, sharing what they're going through and maybe some early lessons learned.

Um, in fact, uh, one of the people on today, uh, Kemper Brown is the CEO of, um, Eric, if you're typing, can you mute for a second please? Um, Kembra Brown's on from, uh, the CEO of electronic office who, um, went through the Asheville North car, who's in Asheville, and went through the hurricane. So, um, he's gonna be a co-host sharing perspective as well. Um, okay, so let me, let me set the stage.

Um, like I said, it seems like yesterday, Kemper, we had you on talking about what you went through, um, the, uh, devastating hurricanes in Asheville. And, um, this week we wanna kind of set aside some time to discuss the, you know, horrific fires that are devastating, um, Los Angeles.

And, um, uh, I started to reach out to MSPs I knew in the area, and, and sure enough, um, there are, um, in fact, uh, MSPs, their customers and obviously families, um, being affected by this, uh, quite, quite, uh, quite significantly. Um, we'll try and get the background audio, uh, cleaned up a little bit, um, as far as where that's coming from. Alright, so, um, lemme start off, uh, with, with some intros, uh, real quick.

So, uh, first Brent, um, if you could tell us a little about yourself, your MSP, um, you've been in the field for, for quite some time. Um, have you ever seen anything like this starting off, Brent, and then some intro? Gosh, Yeah. This is, uh, this is new territory for me. You know, being living in California for, you know, I, I don't want to date myself, but it's been decades. Okay.

Um, you know, we we're not really, you know, we're pretty used to these natural disasters, fires, earthquakes and, and mudslides and different things. But this one hit so close to home and affected so many of our clients that it's, it definitely feels and is different, uh, this time. Um, it turns out that, you know, LA Kenyata, which is a, a small town, uh, north of Glendale and west of Pasadena, um, is where our office is, and it's where my home is.

And the, the e the Eaton Fire, which you've seen on the news, it's, it's devastated block after block of Beautiful Homes in elt. Dena is about a quarter a mile from my house, and it's about a mile and a half from our office in LaQuinta. So we were, uh, we were impacted pretty severely and we're still not back in the office yet. So that's, uh, that sets the stage for that. And you mentioned, um, having an MSP for a long time.

I, I, I owned, uh, DCG Technical Solutions, and we operate out of Greater LA just like Alan. Uh, our clients are mostly in, in the Valley and, and around Los Angeles. Um, and we, uh, I ran that for 27 years before joining new charter Technologies about five years ago. And, uh, and, and since then, we've become a national platform, uh, operating in about 45 cities.

And, um, uh, and, and that's been kind of an interesting aspect of this, uh, being part of a platform and having this happen has been kind of interesting. And maybe we can chat about. Sure, sure thing. Thanks, Brent. Appreciate you joining us under the circumstances. Uh, a uh, it's hard to believe I've known you, uh, over 20 years and, um, hopefully not in these circumstances too often again. But, uh, thanks for joining.

And if you could tell us a little about what's, you know, you're dealing with and, and a little about your, your yourself and your company. Yeah, thanks, Andrew. Yeah, it's been a couple decades for us, hasn't it? Yeah, Yeah. I'm a president and CEO of all connected. I've, uh, like Brent lived in the area for decades. Um, and we're at the far east of Ventura County. It's a, a city called Simi Valley.

We're adjacent to LA County, um, our weather, different kinds of wind events and storms and, uh, definitely fires. Um, but, uh, the impact of this, like, uh, like Brent was alluding to, has been, uh, pretty staggering. Um, our company has been in business for 26 years now. Um, there's about 30 of us.

Our organizations are really focused on aligning clients with the NIST Cybersecurity Framework and CNMC, and we primarily service municipalities, uh, financial businesses and manufacturer manufacturing sectors in the LA and Ventura counties. So it's like, uh, um, you know, all of us, I feel like our, our, our clients, our team members, everybody that that worked for us somehow knows somebody that was affected, you know, in this particular disaster.

Um, it's close to home because for me, uh, disaster recovery is pretty, you know, pretty passionate. I'm pretty passionate about disaster recovery and, and cybersecurity and, and, uh, that's, that's always been a mission of our company too, to help protect and connect to organizations in the area. Yeah. Good, Alan? Yeah. Uh, if we have time, we'll, maybe we'll talk a little about, but you were one of the first for the company. Just nostalgia everybody.

Um, Alan built a company called Xla Core, um, several, gosh, almost two decades ago, I think it was Alan and one of the first in the BD r space built a, you know, very amazing company, sold it. Um, so yeah, you, you know, business content fairly well. And, and I remember you being in the heart of, um, helping out people, um, during the, um, uh, Katrina, uh, incident as well. So, yeah, this is, uh, pretty, uh, pretty amazing.

So, Kemper last you, um, you, uh, appreciate you joining us as a co-host today. You, you've got, unfortunately, some incredible perspective on crisis communications and navigating, uh, natural disasters as the MSP that we had on a few months ago from Asheville, North Carolina. A a how are you doing? How's, how's spirits of everybody? And, uh, welcome to joining. Thanks, uh, thanks Andrew. Um, thanks Brent, now and for joining.

And, uh, yeah, I'm, I'm doing, uh, I'll say better, better than, uh, when we last talked on the, on the cyber call, uh, I think we're, we're making good progress. Um, still have a long way to go, but, uh, certainly, certainly things are, things are getting better, uh, around here, so thank you. Thanks for having me back. Yeah. Yeah. Thanks for joining. Okay, Ms. Phyllis, I'll let you kick things off. Yeah, sure. Thanks everyone for being here.

Um, and Alan, first of all, how is everybody, um, in your organization doing? Yeah. Well, fortunately, Phyllis, our, our team's doing well, um, we're still in, uh, red flag warning for high fire risk through Wednesday. Um, the, the wind speeds are, are picking up or, or moving a little bit west. Um, the active threat that we had was, was primarily from the Kenneth Fire, um, that had a potential path to our facility here in Simi Valley.

Um, and that was knocked down by the, uh, amazing firefighting that ended up happening over the weekend. Um, uh, it has been tragic and definitely heartbreaking to see the devastation around the areas and in LA that we love so much when these walls of fire, you know, descend on these cities. Um, like I said earlier, I, I think, you know, ev everybody on our team knows somebody that's, that's affected.

Um, and, uh, in Simi Valley here in our area, even though I, I'd say we're, we're on the outer perimeter, right, of where the fires are affecting. We have these, um, you know, PSPS public safety events that Edison issues that basically shut down power, um, whenever the wind risk is high, fire risk is high because they don't want fires to be caused by electrical transformers in the area.

Um, and so watching all, almost, uh, all the homes and businesses in our city, south of the one 18 freeway shut down for three to four days without power made it feel like a, it was a ghost town, you know, here in our city, you know, at night, in fact, right before the podcast we we're, we're, we're kind of all checking in, and three minutes before all the power in our building shuts down.

So it's, we just hit another, uh, power safety event, um, mandatory shutdown from Edison, and we're operating on generator power. Unfortunately, I got in just in time. Wow. Um, but, uh, yeah. Um, I, I, I think, you know, you're asking about our, our, our people. When when the power goes out, you gotta figure out a plan B pretty quick. Mm-hmm. Um, it wasn't just, um, you know, losing power in our homes and in our businesses, but the cell towers around on all the mountaintops start going down.

So you start losing at and t you start losing Verizon, uh, SMS messages don't work. Texting doesn't work. Most of that's been restored, but, uh, who knows in the next 10 minutes or 15 minutes what will happen as, as power starts getting shut off again. Mm-hmm. Wow. That's amazing.

I mean, you know, looking at it from over on the East Coast, of course, it just looks devastating and I just can't even imagine, um, being there and also trying to work and help others actively, you know, maintain their communications must be so, so difficult. Um, so Brent, um, my understanding is that, uh, 6,000 homes in your area have been lost, and over 50 companies you manage are off the line. Mm-hmm. Mm-hmm.

Um, so can you help us understand what you're going through right now and how you're dealing with everything? Yeah, it's, you know, the pe my people have been awesome. Um, we had two people besides myself and my family that were in evac zones, evacuation zones, and, um, they've, they're, they're, they've been allowed to return to homes, but we're still kind of in that, like, uh, Alan said, it's, it's definitely a red flags area. The, uh, the winds abated a bit.

Um, I don't, you may not, a lot of people don't realize this, but the Santa Ana winds got up to between an 80 and a hundred miles an hour last week, and that's what made the, the fires almost impossible to defend against. And, um, and so, so that was a little bit of a, you know, made it really hard, you know, to get on top of it. And it went on for days.

But when we woke up the morning of the, of all the evacuations, there was texts from our, you know, the county and the city text us, uh, on this emergency system that the areas that need to be evacuated. And we were getting texts starting at three in the morning, and we implemented our, um, our, our DR plan, if you will. Um, and the kind of, the interesting thing about DCG is that we've got four employees that are out of the state.

Uh, two of 'em are on the East Coast out of the state, and two of them are like north of California. And, um, we, they basically took over and our, our phone systems are set up so that they can receive the inbound calls and, and, and they, they were also tasked with calling. Every site that we noticed was down in the morning. Um, we use a, a, some software that gives us a geographic look at where the trouble spots are.

And by, by looking at ISP and ips, we can, we can tell like which areas are impacted. And of course, um, the, the San Gabriel, the San Gabriel Valley was completely lit up, as was, uh, some of our sites that were over on highway one, um, near the Palisades Fire. And, uh, there was a few in Allen's area that were, we actually had an employee in, in Allen's area in near the sea area where the sea fire was, that had to be evacuated.

So anyway, these four people that were out of the state really bore the brunt and worked, started early and worked late to help, uh, every, everyone that was impacted. But most of my employees were not, uh, in evac zones. And so they were back and working pretty quickly.

Um, I think the saving grace about this kind of a disaster, if there is one from an IT perspective, is that, um, as the power, you know, power goes out in these large areas like Alan mentioned, and they do that to prevent the winds from pushing the electrical lines together and arcing and causing transformers to blow and then start homes on fire. So they turn off the power proactively, but they turned it back on very gradually.

And so we had, we knew we had 55 sites down, but we had plenty of time to respond as each one of them came back up. And even though, um, uh, we have a full-time national help desk at new charter that was at the ready to help us, we were able to handle, uh, the turnups, uh, and of all the sites as they, as they came back on. And some, some have not yet come back on that were, that were in the, in the fire zones, but most have. And so we were able to handle it. Phyllis, um, interesting.

You know, uh, uh, lo lo lo Loden, uh, asked about, you know, Alan, I can ask you this. The, you know, obviously we've had a large digital transformation, the SaaS and cloud, um, uh, I'm assuming there's probably some still on-prem. Um, but can you talk to us a little about that?

His question like, Hey, is what's been the, maybe the delta there, uh, the people that are in cloud and SaaS, um, be better off in terms of scalability, availability, if you will, and data loss than the ones OnPrem or, or not at all, or not, not a difference. Yeah, I think you're asking me about that, right? Yeah, I'll ask, yeah. It depends on the type of business.

You know, when we serve municipalities or water districts or, you know, those kinds of industries, you, they've gotta be remotely survivable. So there's a element of data that has to be OnPrem. Mm-hmm. You know, for local operation. They can't have a situation where they're disconnected from the cloud and they can't operate and, you know, you know, have first responder units active and so on. So, so that's a, that's an important element, and we do support a lot of that type of infrastructure.

Um, you know, for, for clients that ended up migrating to the cloud. I mean, yeah, large majority of our clients, you know, have file data, application data in the cloud, different types of clouds. The challenge becomes, can you get to the cloud? Hmm. And, uh, you know, with, with, with so much pressure on local network infrastructure, it's, it's a real challenge.

When we had the Woolsey fire rip through here in 2018, you know, it's burning all the fiber lines that are running, and then you start finding out, okay, where's spectrum's weak spots? And where's at t's weak spots? And what's the, what's the line that failed that none of the smart people ever thought of was the most important, you know, and, and you learn that stuff when, when disasters converge on a particular area. So, um, yeah.

And, and then you got, you know, we got into situations where clients have their own generators, but maybe our engineers didn't know how to start their generators, you know, that was their electricians, but their electricians aren't available. So how do we get that infrastructure online? Some, some of that, you know, started to happening too. And then we're trying to patch in electricians and bring them into chats, you know, to, to help get customers infrastructure online.

Um, but, but by and large, I think, Andrew, it, it, it was helpful, you know, know, it's, it's helpful to have not as much on-prem and be able to have remote employees from different areas, log in from however they can get connected to the internet. Mm-hmm. More functional, especially if those systems are communication like O 365. Right. Right. Right. Interesting. Thank, thanks for sharing that now, Phyllis. Yeah. Awesome. Thanks.

So, Alan, um, you know, I'm sure it's hard to, to balance that personal, uh, versus professional here since so many folks in your organization may have been affected, um, by the fires. So how do you balance helping employees, giving them the grace they need, um, and time they need versus, Hey, you're managing a business and you also have to think about your clients and what they're going through and helping them get back up and running? Yeah. It becomes a lot harder to prioritize Phyllis.

Um, well, 'cause a lot starts to happen really fast, and you think you're on a track and then something else comes up. Um, you know, I've got a vivid memory of, of a client in the 2018 fire that burned down to their building, and they're packing equipment into trucks in St. Allen. We're on our way over to load it in your data center. You know, uh, for us, the trigger was when that Kenneth Fire started, because it's just a few miles away. And, uh, it's really interesting.

One of our employee's, neighbors actually pinned down a man carrying a, a torch that they thought lit the fire. And there's a bunch of commotion going on here in the office about, you know, whether it was arson or not. And, and then the wind's blowing fast and the fire's heading our way. And, and, uh, you know, meanwhile we got clients that are dropping. And, you know, I, I just think you gotta really rely well on our operations team. And our, our team is fantastic.

Our tams were fantastic because they take care of pods of customers, and that helps to, you know, almost isolate, you know, a, you know, a leader or commander kind of for that, for that unit. And balance emergency communication, you know, my assistant and also our, our client experience manager, um, helps to clear schedules and prioritize, um, make, making other calls as needed.

Um, you know, and, and that helps me out a lot because there's, there's some personal calls I had to make too, to right. To employees, to CEOs and, and IT directors just to make sure that they're okay. And then of course, we have our, our own families, you know, and my son actually just experienced a, a serious fall in Yosemite Valley a a few weeks ago and confined to a wheelchair. He'd only been back home a week. And that go bag was different than we'd ever packed before.

And, you know, my wife and I are working to make sure that that's going on. It's right. Just, it's one thing after another that's, it's hitting in a, in a very compressed period of time. Yeah. I mean, it's just devastating all around and, and like you said, you know, um, it's changing every day, so then your plan may have to change every day. Yeah.

I, I will say, I think that the real time communication services, I know a lot of, a lot of people use 'em, you know, whether it's, uh, you know, Microsoft teams or WebEx Chat, we use Cisco WebEx ourselves, but, but being able to have those groups or spaces that we can create really quickly to involve the team that needs to deal with this disaster or this disaster, and having the emergency response chat for all employees where you can actively see that they've read, it saves so much time because you know who's checking in and who's not, and who you need to reach out to.

So that was a big help for me to, to juggle, I guess you could say, a lot of personal and, you know, business related communication. Alan, you mentioned you had to go to starlink, and I know, maybe I could chime into Kemper here a little bit too. Can you first Alan talk about, Hey, it's, what about, you know, if you don't have, you know, your fiber to, and you know, you can't get on your phone? Ha has, has starlink helped?

And, you know, 'cause I remember Kemper saying one of the things that was, you know, they were really diligent in operational maturity and testing their plans, but one thing they never tested was, Hey, we're not gonna be able to communicate to employees for three days and even know if they're alive. Um, so maybe Alan, you could talk first about that and Kemper. Yeah. Um, just portable power stations are so important.

Just something that'll last through 20 phone charges, you know, and, and a little inverter to, to end up, uh, plugging things into like laptops, that's important. But if you've got the power and, and, and you're, you're losing, you know, your cellular service in the area and all the local businesses are down, you can't even drive up to a Starbucks to get a wifi hotspot. You start to figure out like, where am I gonna get internet from and communicate.

Um, we have, we have, uh, you know, emergency starlink kits here. It didn't start out that way. It really started out for, you know, people that wanted to go on vacation and they wanted to stay somewhat tethered, you know, on a longer trip, and they could take a starlink and activate it. And that was helpful. But, um, you know, starlink sent out a message for California saying, Hey, anybody that's got these star links, we'll turn 'em on for free and give you guys 30 days of internet.

So we said, Hey, we've got it available. You guys can use it. You know, just put it out there as, you know, one of the resources, um, that was available. Same thing with generators, same thing with our building. You know, it's, our building is one of a very few in a, in a pretty wide radius that's, you know, got all the, you know, got, got power.

So people actually come in here and recharge power stations and, you know, use that to, you know, help make sure they have power to get through the night for emergency communications. Got it. Ke Kemper, any thoughts on this? 'cause you know, you, I know you lived it. I remember you talking about this. Yeah. Um, absolutely. I mean, I, Alan I think that's a great point about, you know, real time communication techniques in terms of chats and teams, channels and, and things like that.

Um, Brent, I'm, I'm curious on your perspective on how you've, how you've been able to, uh, help your team stay connected. Uh, any other tips, techniques, things that you've learned or done? Yeah, so I think what really, we started this process probably during covid when so many of us were working remotely. But, um, you know, having a a, a comm system where it'll work no, no matter what, no matter, you know, from anywhere anyhow, I think is key. And it's kind of surprising.

A a few of our clients that we called, um, like, we didn't even get a busy signal. It was kind of like a weird, fast, busy, you know, so they were like completely shut off. Um, and, and I, I can't imagine an MSP having a situation like that where they clients couldn't reach them. But, um, our phones work, you know, on the east coast up north and in Washington where we've got some employees and, um, they work well around southern California.

You know, Southern California is really just, it's like a country in terms of size. And so, you know, we can go, we can go to our data center, which is, you know, kind of over in El Segundo, which is south of Santa Monica. Uh, we've got another one in la. We've got, um, you know, home offices across SoCal. Um, we feel like, you know, have, being able to run the comms is probably the most important thing in terms of, you know, email and, um, in, in inbound calls.

'cause a lot of our clients still call us. So we think that's, that's kind of key. And the, the, we, we have a combination. We have a hybrid system that uses three CX and teams for inbound calls. So we feel like we're pretty well covered, uh, with that. And so, and that, and that has served us well, not just in this disaster, but, uh, but some others. And I think that, that, that's helped us keep in touch.

Um, you know, listening to this call in, in, in, it does make me wonder if, if, you know, I think Kemper, you, you said that none of your cell phone, you know, cell phone towers were out and cell phones will stop working. Um, we haven't seen that yet. Not in this disaster or any other in SoCal, but I guess possible. And it makes me really think, kind of wonder about that and wonder how we could even be better prepared after listening. See you guys talk about that, that issue. Yeah. Yeah.

I think I, I think Brent, just to that point, um, I think starlink, starlink is a, is a great way to maintain that communication. Yeah. Um, uh, dual band radios. I don't know how many people on, on this call. I've, uh, you know, used dual band radios and things like that, that, that was actually a, a saving grace for a lot of people during, uh, during our hurricane mm-hmm. Was, was the use of dual dual band radios. So, you know, ham radios, things, um, things like that.

It's, it's definitely something that we're thinking more about. Um, satellite, satellite phones is an, is another, is another one. Yeah. Uh, although the, uh, it was kind of semi news here in Asheville because the major, uh, uh, hospital organization in West North Carolina, they, they had a bunch of satellite phones, but had not tested them. And so, you know, when, when they actually needed to use them, uh, they found them hard, hardly operational.

So, uh, you know, testing, uh, that goes, that goes back to testing our DR plans, uh, yeah. And the equipment that we rely on. Um, yeah, I was, before, prior to this call, I was talking to my team. I'm like, um, you know, I'm gonna be talking to Andrew and, and some others about, about this. And, um, John, who's now the president of DCG in Los Angeles, he's like, you, well, tell him about our, our pre-game of this and, and the, uh, um, the emergency response drills that we do.

'cause, you know, he says that pretty much we find things every time we do one of those, we find things that we could improve on. And we do one of those once a year. And I'm, I'm tempted to step that up a bit. Yeah. Yeah. That's a, that's a great segue, Brent, tell, tell me a little bit about your, um, uh, the, the DR plan that you had in place. I'm sure fires is something that you guys had had thought about. Mm-hmm. Um, but what, what was that, what was that planning look like?

And then, uh, anything that you're, you know, at least at this point, you're thinking like, that you would change about that. Yeah. So it's funny. Can I just too, that you, there's like 50 plus clients down with this, so could you kind of maybe flavor that in of, uh, Kemper's question around this? Yeah. So, um, I think our clients, I mentioned that a bit. I think our clients are less set, less prepared for this than of course we were.

Um, but most of them that have, have gotten their power back are back up. So I, I think, uh, it was 55 on Thursday, it dropped to, uh, I think 20 by the end of that day. And by Friday night, everybody was back up. Um, we had two sites. One had a UPS below, so they weren't comfortable, you know, bypassing it. So we had to go on site for that one. And we also had a raid not come back up. Um, it was a, there was a, a down drive, and then one that was in, in predictive failure.

And we had to, um, that was the only site we had to do a restore on, but otherwise, everybody is back up as of, uh, this morning.

Um, but as far as that, you know, pre-gaming the, the emergency and doing this emergency response drill, you know, we've always assumed that we were vulnerable because, um, of earthquakes, you know, in southern California, we have earthquakes, and when there's damage to a building, the fire department drives around and they yellow tape your building, and it makes it so you can't get in it.

So for years, we've assumed that we might not be able, we might wake up and not be able to get in our building. Um, this was a little different because there was a fire, like about a mile and a half away, and it caused the entire town to be evacuated. And they, because of the looting that was going on, they had police and, uh, sheriff volunteers on every corner that was preventing us from getting to that office even if we wanted to. And so, uh, that was, we got to, got to implement the plan.

And, you know, there were certain things in the plan that, um, worked really well. And I think I mentioned this earlier, the outstate employees really saved the day. They, you know, they didn't have family friends that were impacted. They didn't have school issues to deal with. They didn't have, you know, family that were, that were impacted. And they just, you know, I, I think the best thing you can do is have people out of the geographic area that's just able to just focus on the clients.

And I think that was a big, big plus for us. Um, when we pre-game this, we also assume that our main office, um, you know, anything that was in the main office over time, that I think the only thing that's in the office anymore is A DVR. That records, you know, the cameras that we have around the office, everything else has been moved to the cloud or to one of our data centers, uh, in LA in Los Angeles. So, um, we also have a DR data center in Las Vegas, Nevada.

So we, we feel like we, when we pregame this stuff, we could switch over to that, um, to that data center, should something happen to the LA data centers. And so there's just, each time we run it, we usually uncover new things. Um, if it's, you know, you mentioned starlink. I, I, we have through one of my other companies, we have, I think six or eight of those units, and I think we should probably store a few of them in Southern California.

I think it might make sense to do that so that we're more prepared for the next one. But yeah. Um, I think the best thing you can do is just practice, you know, work, do the drill and, and improve every time. And, uh, you'll be, you'll be more ready each time. Thanks.

Uh, Brent, you, uh, you kind of touched on some of the, some of the technology, uh, but were there, have there been any specific tools or platforms that have been, uh, you know, really helpful, uh, for your business or, you know mm-hmm. You know, keeping people on the same page, keeping clients on the same page mm-hmm. Communicating, um, communicating with clients. Tell me a little bit about that. Yeah. Um, the phone still works the best, I think, in, in these kind of situations.

We call, like I said, every client, and we have backup numbers, like cell phones and things, so that we are able to let the clients know that we are standing by and we're ready. Okay. Um, that really, that, that always, I think that worked really well for our clients and, and, and for us.

Um, there was, uh, uh, as far as like the stack and software stack, I, I, it's kind of interesting, you know, mostly we rely on our ConnectWise PSA tool, you know, that help us with, you know, up down and those sorts of things. And we, we've always had PRTG in place, um, which is a, I think used to be owned by SolarWinds, and now it's somebody else that owns them.

But it's kind of an, it, it really, it's, it's a great product for, for monitoring gateways and, and internet providers and their gateway equipment modems and such, along with, um, the firewalls. And we were able to use that to get a geographic map of where the DA of, of the two main places where we had clients concentration with outages.

And it was also really helpful that, that even when power came up, but if a provider, if, if the last mile was still down, like a concentrator was down, or there was a problem with somebody's, you know, cable connection or whatever they, or fiber, we were able to diagnose quite quickly that, that they weren't coming back up or, and why. So PRTG was great for that, and, uh, the guys relied heavily on that.

Um, and I, I think that, uh, the other thing that we relied on a lot was our three CX slash teams hybrid system, because, um, it never skipped a beat and worked perfectly. Uh, our clients could reach us every second, and we could reach them every second throughout the, the emergency. Oh, that's great. Um, I, I'm gonna just go off the script for two seconds for, for Brent and Alan. How much, like, how, in terms of the forecasting and things like that, how much notice did you guys have?

Did you guys know that you were ending a period where this, where something really bad could happen? Or, I know when the, you know, when the Palisades fire started, I mean, it just spread so quickly, but can you, can you maybe, uh, explain for some of us non-California Californians, what that's like, You want, you wanna start with, uh, Alan maybe on That one? Sure. Yeah, Yeah, sure. I, I'd say Edison's probably the, the, the one of the leading indicators.

I mean, you'll, you'll also get alerts from weather, you know, like we have now that it's a red flag warning and that you gotta be alert, but without any fire starting, you don't know what that means or where it's gonna pop up. But once Edison starts telling you, okay, we're, you know, your, your area has been flagged for a power shutdown alert, um, you know, that they're taking it very seriously.

And I, I'd say that's probably the leading indicator for us, uh, Kemper was, um, and Edison notifications to us that, um, you know, few thousand businesses and homes in the local area, were gonna be shut down. And how often does this happen in a given, you know, in a given month? Is this something that happens, you know, weekly or, you know, less frequently? I, I think, I mean, Before a disaster occurs, you know, I, I always felt like it happens too much.

Like, you know, we, we need our power and they're shutting down the power, and it's, and you just think, uh, okay, it's just windy outside, you know, uh, 30, 40 miles an hour. But, but the reality is that 30, 40 miles an hour up in the mountains turns to 50, 60, 70, and it, it, it can move things really, really fast.

You know, as, as, uh, you, you're starting to hear about the investigations that are going on with fire, they're, they're looking and they're pinpointing the specific areas, like we think it started here. And, and, and you look at that, and maybe it's in a valley and it can just move super, super fast.

Um, I was reading a little bit about the, that guy that sounded the alert, Brent, in your area, that guy's, you know, got a lot of publicity, 24-year-old, uh, um, uh, climate expert in your area, that, that, that really helped hundreds to evacuate early because he saw the warnings in advance, right? Mm-hmm. I know a lot of people got alerts from that too. Mm-hmm. Yeah. In, in Lata, we have an 8 9 1 1 system, and it's, they, they test it every year.

And I get this kinda lane text from them that just says, Hey, we're just testing the system. Well at 3:00 AM um, Pacific time, I got a, I got a text, and, um, I noticed it. I, I heard it and looked at it, and I was like, oh, we're we're being evacuated. Wow. Yeah. It was, it was surprising to get one that was real, and it, it worked as it was designed. Um, they, they were, it was pretty effective, I think. Yeah. On the other hand, we did have some alerts that were not so effective.

Uh, we got, we got the one on Thursday evening where, uh, an evacuation notice accidentally got sent out to 10 million residents. And, uh, that was at like 4:00 PM on Thursday. And like, our guys are out on the freeways and trying to get to clients, and it was noticeable. Like all of a sudden the freeways like overloaded with people thinking we, we got evacuations coming, and, you know, LA's a big area, and you got people that are trying to get in, people are trying to get out.

And, and that caused a, it caused a little panic, but, uh, yeah, I didn't, I didn't get that one. I guess That's crazy. Yeah, I didn't get that one. Yeah. Yeah. We all got it. Yeah.

But, uh, you know, it's, you're, you're always gonna, you know, have those kinds of issues trying to be overcautious and, you know, then they're trying to get the dispatches out and something went wrong in the software and actually tagged, not just our county, it was meant for our county for the Kenneth Fire, but it went out to everybody in every community. So, um, yeah. Can I, I guess not you, Brent, Brent, Brent.

Maybe starting with you, I just, a quick question from, uh, the, the chat or, you know, I know in, in Florida here, when we had back to back hurricanes, you know, you get these notices from your, uh, insurance provider, you know, it's like, Hey, you know, have you've been affected, this, that, and the other, not to the degree that you guys are dealing with. Uh, what do, are you hearing anything from insurance at this point, fema, anything like that? Or is it just crickets?

It's crickets right now. Yeah. It's crickets. I, you know, given, given the topic that some of the topics you cover on, on this, that your amazing, uh, uh, podcast, Andrew, I think that we're probably gonna see a lot of relief, like crazy emails about relief funds and families needing your help. And, you know, phishing links and I, I, uh, we're, we're preparing a, a, an email to our clients that'll go out this week about that.

Um, so It's, it's so funny you, you bring that, not funny, but you bring that up. I, when I was on the phone with Alan this Saturday, we were talking, I'm like, Alan, I, I go, this is a, this has got covid written all over it from the standpoint of, you know, there's a disaster and there's gonna be Yeah. Phish, social engineering, praying all over this. Um, so that's in, so you're taking some proactive steps on that. Yeah. That, yeah.

Um, and then, yeah, just Remind people that, you know, you, you can go to the Red Cross site and make donations, but don't respond to emails directly, you know, and those sorts of things. Um, we've had to do this, I think in one of the hurricane relief. Uh, I think it was the, uh, I think the south, either North, South Carolina, one of those, we did it then too. 'cause we were seeing a lot of, uh, there were just a lot of, of fishing around that one too.

Less about Florida for some reason, but a lot about that one. Interesting. Yeah. Yeah. We've seen it physically too out here with, I don't know if you saw the news about the, the looter that was dressed up as a fireman. Um, you know, so physically it's happening, right? Um, people that don't want to take advantage, and I think that's important. Brent, we, we've been talking about that too.

Planning specific releases to clients about being careful about GoFundMe, being careful about, you know, help or offers or aid that comes from unsolicited email addresses and getting out there and go into the actual domain name where, where you, you know, where the help is being offered from. It's super important. Yep. Yeah. Cool. Well, Brent, 1, 1, 1 more question, and then I'll, I'll pass it to Eric.

But, uh, uh, have there been any other contingency, uh, contingency plans that you guys have done, uh, either internally or for your, you know, for your clients, uh, as kind of this, this event has unfolded? You know, what, what, what were some of the things from your clients that you were maybe maybe worried about previously that you, that you, uh, plans that you enacted to try and help, uh, keep them safe or keep them up and running? Yeah.

You know, whenever power goes down in the night without being like, supervised, and you, you just, you just, you're hoping that the communication between the UPSs and the equipment works. And it almost always does. But we, you know, we have a, we have 175 clients in Southern California, and when 54 of 'em went down, we expected to have one or two where we were gonna have to, you know, possibly help.

And, and, and in that, that case with the, the raid 10 with two drives that were in trouble, we actually had to go into a restore. And so the only thing that we really had to do outside of just kind of hold people's hand as they waited for power to come on, was do a, a full re uh, a restore of one of the, of, of, uh, of one, of one server. And it was successful.

Um, but, you know, I think that's the only kind of thing that popped up that, you know, was as a result of the storm and, or the firestorms and, and the power, the subsequent power outages. Okay. Excellent. I appreciate it guys. Eric, I'm gonna, I'm gonna pass the baton to you. Thanks, Kemper. I appreciate it. So Alan, uh, a little while ago, Brent mentioned the importance of when possible having some geographic diversity amongst your employees.

Um, and I understand that, that during these fires, um, you had, uh, several out-of-state employees covering phones. Um, tell us about the, the nuts and bolts of, of, of how you pull that off. Um, well, it started with a, I guess a hallway huddle. You know, you're, you're running back and forth in the office. Are you guys okay? Are you guys okay? And, and I, I had one with our operations manager at the end of, uh, Thursday.

He was already, you know, stress was, was building up because employee after employee was getting evacuation orders. And then at, at 4:00 PM that alert I talked about with the evacuations, order orders that, that were supposed to be just for our area, went out to everybody and the freeway started loading up, you know, so, um, so you just, you start getting a little worried and, uh, how are we gonna deal with this? And he was worried, like, how are we gonna handle tomorrow?

How are we gonna handle Friday? As, as, as the, the threat of the Kenneth Fire, you know, potentially could hit us. So, um, you know, we've got employees in Illinois and Denver and Kansas, and, and like Bri was saying, like without the threat of a looming disaster in their local community, there we're way more freed up even mentally, right? To be able to think and handle things in a, a less, less chaotic way.

So, um, that's, you know, what was, what's kind of our approach could, could we use them? Yeah. So sometimes when you have those, those remote employees, you know, they might not be your, your level one help desk, or even level two help desk guys who are, are taking those first line calls. Can you, uh, can, can you tell us what challenges come with that and how do you have them adapt? How do you encourage them to adapt to this new role that they know nothing about? Yeah.

Well, I, I, one, one of our core values is preparedness. It has been for a long time, and we, we'll, we will, uh, you know, our monthly company meetings will always recognize somebody for, you know, an outstanding way of, you know, living a company core value. And I think over time that becomes important. 'cause people realize we've gotta step in and, and help each other.

And, um, so we discussed Curtis and I, how, how could we, you know, maybe have them tra take frontline calls and he agreed, jumped in and set it up. And, and I don't think it was that, that hard because even though the roles were maybe sales or recovery services, or even accounting, you know, one of them, uh, they readily helped out and, and jumped in to fill in an immediate need. And I, I think because they know our business, Eric, they know our core values.

Um, and most everybody on our team, clients are a lot happier knowing that they're talking to a team member here locally rather than like a dispatch center. 'cause we could have done that and just, you know, farm 'em all to a dispatch center. But when you're talking to somebody on the team that actually knows the people and the techs that work, you know, it, uh, I I I think it helps a lot more with disaster response. Yeah. So obviously hindsight is 2020, right?

But now that you've gone through the, this, this new remote work experience, particularly with these, this handful of employees that you have living outta state, how has that experience changed your view of number one, remote work and number two, the role of remote work in contingency planning?

Yeah, it's, um, it's a little bit of a wake up call, I think because so many of us have been in the mode of, let's, let's get everybody back to the office or, or try to build back the office culture or some of the office culture that maybe we lost during covid. And you can go a little too far, you know, and, and, and maybe not have enough diversity in, in geography.

Um, and it, and it's made me think more about senior engineering roles, you know, for higher level work as part of a larger business continuity plan. Something to to discuss with the management team. I feel like we've always been, you know, remote work enabled where people in the local area can jump on and connect wherever they're at. I mean, we all have, but I, I think the strategy is a big reminder on just how important it is for us to be plan planning on a serious disaster.

Brent mentioned earthquakes. I mean, that's always been hardwired into me. Mm-hmm. Scariest types of disasters. You just never know when they're gonna hit. And, um, you know where your people are gonna be and what they're gonna be doing at the time. So having a team outside of the 300 mile radius they talk about, right? You want that outside that, that different power grid, different transportation system. It's extremely, extremely helpful.

Hey, Eric, just, just curious for, for you or anybody, 'cause you're out in the California region too, but like Alan, I remember, you know, again, it was almost 20 years ago. I remember vividly you and I were at a, a Ingram meeting post Katrina, uh, Kemper's dad, um, as well. Yeah. Kemper, you might have been there with him. Um, but, um, it, it seems like these natural disasters are more cut, more, uh, what's the word I want coming more regularly.

Um, like I said, we had hurricanes this year and, and what Kemper went through without, you know, less than, you know, six months later we have this, um, it would appear that, you know, Dr. Physical, Dr is, is what's old, has to become new again now. And I mean, it was like, we gotta, you know, it's like outta dr and it was passe for a while, but it seems to be something that's becoming in vogue again, uh, that, that we really need to get, become better at or, or be diligent about.

Is that, do you, what are your thoughts? Yeah, it's changing a lot too. Um, you know, disaster recovery, because you've got apps in so many different places in different areas that can, you know, that, that, that can break or falter in, in some cases. Um, and the local network and connectivity becomes such an important aspect of all of this. Um, I think that's, that's what we're seeing.

While we can, we can work on identity management and authentication and make sure that people can get to the resources that they need to, um, secure connectivity in a disaster scenario is, is, uh, absolutely crucial. And, um, I think sometimes the resiliency that we think is there with, with the ISPs isn't there when you need it the most. Mm-hmm.

Um, you know, it's, it's made us go back and look at all the clients who don't have redundant connections and those redundant connections aren't tested. Um, and, you know, what's the plan for the employees of those customers? We always talk about incident response planning so much with cybersecurity, but I think you're right. We, we haven't talked a lot, as much as we probably should have about that fifth pillar, you know, and then, and the cybersecurity framework of recovery.

Like what, what about the employees and, and, and, and how to do instant response planning for them. And, and, uh, we talked a lot about practice, practice, practice. You gotta, you, you gotta keep repeating these things, whatever monthly call arounds, you know, for disaster communication and, and DR tests. But, but connectivity's a big, big piece of it, So, yeah. Yeah. Yeah. And, and Kemper again iterated that and his call. So, um, Eric, please back to you. Yeah. Yeah. Thanks Andrew.

That was a, that's a a great point. So, uh, Brent, we've, we've talked a lot today about how MSPs are supporting their customers through these disasters, right? But, but, but looking internally at your own employees, right? Some of whom are going through their own personal losses with evacuation orders and, and, and such, how do you support them as people forget about workers, but as people, um, through these situations at the same time we're, we're, we're trying to support our, our customers.

Yeah. So again, having the out-of-state team. So two, we have four dispatchers at DCG, and, and two of them are outta state. So we really didn't miss a beat on, like, covering the inbound calls. Um, and so what we were able to do, we had one engineer impacted, and one, um, what we call, uh, uh, kind of client service dispatcher type that was impacted, um, besides myself.

And, uh, we were able to, to just take your time, go do what you gotta do, uh, and, and, and they, they didn't, you know, they didn't have to pick up the night number. They didn't have to, you know, handle anything except what was in front of them. And each of them came up for air within by the next day and was back at work. Um, um, wow. Yeah, so they were both fine.

Um, I think that, uh, like there, you know, we all have, I, I think the bigger story around this is we all know someone who lost a home. And I think Alan alluded to that earlier, maybe on the pre-call. But, um, we all know somebody, and I think that's the bigger thing that, that, that is not gonna be, like, it's not gonna be taken care of. And, um, over very quickly, uh, we've got clients that were, that had places in the Palisades that have lost everything.

We also have, you know, sometimes it's, it's kind of close relative. Sometimes it's a, a mother-in-law or sister-in-law who is living in Altadena and, and is now, you know, home without a home and living like with two other families, with another family member. So our, our people are helping with those types of stories and those types of situations. And I think that that's the, the piece where we're not even sure how bad it is, but we all know somebody.

And so, um, as far as insurance coming in to help cover and, and rebuilding, like all of those things have gotta be sorted out in the next, you know, days and weeks. Yeah, definitely. Mark a, Alan told me something about the RVs, right? Alan, didn't you say something about that was an interesting to twist to this? Like, people with RVs and, you know, you know, it was like, Hey, that Covid, but now all of a sudden these are becoming homes. Yeah. Yeah.

That was really just a, that was an employee, you know, as we, as we kind of put it out there and said, Hey, keep us, keep us posted. Let us know what's happening with your evacuation orders and where you're going. And we had, we had five people that got evacuation orders, real ones, you know, uh, um, within a very, um, close time proximity to each other. And, and it was just an employee of ours that happened to have an RV that said, Hey, I got an rv, you guys can evacuate here.

You're welcome to use it. So, you know, it's, it's, it's part of helping each other as a team. And, uh, I I, I think it, it's stressful. I saw one of the live chats that was going through here, one of the comments that some of, some of the, the, the audience posted here about, you know, if you're operating in a state of hyper readiness 24 7, how do you prevent burnout?

And, and I think that can, that can definitely happen, you know, like it sure does with our firefighters, but even in our, our case, which is, you know, so much less scale compared to what they're doing out there, it can happen. And you gotta figure out ways to cover each other and help each other out. And, and, um, for us, I, I, I think it brings our, our teams together. Um, but you do gotta watch for it. Yeah. Yeah, I do. Yeah. True.

I mean, anecdotally, I'm, I'm a couple hours from LA and in the Palm Springs area right now, and, you know, we've, we've offered extra rooms in our homes, anyone who needs it. And, you know, you drive down the street and you see people with their lives and their car, uh, 'cause uh, 'cause they're just, uh, getting away from it. So that's crazy. But, uh, but Brent, you know, one, one last question.

If you could distill this down to you, you know, one piece of advice that you'd give other MSPs about preparing for this, preparing for this or something like this, um, what would it be? Create a plan and work the plan. Um, and that that means, you know, creating emergency response drills, um, actually implementing it on a day where you don't need it to find out what's not working. But yeah, it create a plan. Work the plan, and, and, and you'll be, you'll, you'll be able to get through it. Hmm.

Yeah. Yeah. We, we beat that one pretty hard for years on the cyber call, right. Phyllis about, you know, incident readiness, tabletops preparation. Yep. And, uh, Yeah. Yeah, exactly. I mean, I'm impressed, you know, as an east coaster who always saw disaster recovery plan on a test, you know, like your ci up your, this and the other. I mean, we don't, you know, we don't have very many natural disasters or any of that. Right.

And so, um, just, I'm so impressed by everyone on the call dealing with it, Brent, the fact that you already, you had your disaster recovery plan. You're, you, you've, you do it yearly. You're thinking about upping it. I mean, I think, you know, as Andrew pointed out more and more, we are getting these natural disasters, um, politics aside for whatever reasons. Mm-hmm. And so it is, um, uh, a sad reminder that, um, we, we have to do that planning.

And really, there's nothing magical what you hear on this call. There's no magical tool. There's no magical, um, anything that helped, um, these guys, um, prepare for the natural da disasters. It is have a plan, execute the plan, practice the plan. Mm-hmm. Fix the flaws in the plan, fix the gaps in the plan, and then when you go to execute the plan, you know, rinse and repeat. Right. And so, um, it's just a, it's just a harsh reminder to us all. Yeah. Thanks for summing that up, Phyllis.

And, you know, Alan Brent Kemper, um, you know, thank, thanks so much for joining, specifically Alan and Brent, you know, with what you're currently going through. Um, I, I, and, and, and, and, yeah. And Brent obviously Kemper what you went through and are still recovering from it. It speaks to, uh, I guess what I'm most impressed about and and optimistic about is how, uh, the maturity of MSPs on the margin is, is getting better and better and better, um, and how you guys are operational.

Um, it's, it's, it's pretty awesome to see. So, so thank you for set being exemplars, all of you for, for the channel and, and being on this, uh, this show. Thanks so much. Thank you, Andrew. Yeah, Thank you. Be safe. Thanks, Andrew. Yeah, Thank you Andrew. Thanks. Thanks for putting together such a nice community that helps us learn from each other too. No doubt. My pleasure. It's all about the community. Thanks guys. Yeah, you're absolutely right. Take care, everybody. Everybody.

Thanks for joining everyone. Take care here.