Security Risk from APIs via 3rd party integrations

All right, week 63. Welcome everyone to the cyber call. It's great to have everybody with us Co-host, Gary Pika, west Venture, and Ryan Weeks. Good to see you all. Uh, gonna be introducing our special guest shortly. Just a few quick announcements. Gary, you know, I'm always quick on the announcements, right? Yeah. You know, I love that about you. Um, okay. So just really one main announcement. It's in the call to action. I'm putting it up here. Um, Gary, uh, and, and, and, and Brian.

And Wes, by the way, why isn't this thing I'm hitting enter? It's not working there. It's, um, so Gary, real quick, thank you for, um, getting behind the security level Up challenge, getting everybody trained up. Um, appreciate you guys backing that. Um, Wes, um, ConnectWise and Perch and Ryan and Datto, um, take a look. Uh, if you have any questions, email me, uh, andrew@thecybernation.com. I'll put it in there. Um, this is coming up fast, and we've got a lot of people attending this.

There are some very specific things in terms of, you know, when you register with the cyber call code that give you, uh, things that, uh, only you get, uh, as the MSP group. So, um, with that, uh, I had a conversation with Danny late last week.

And, you know, for those of you that may be using Threat Locker, probably know about this, but for those of you that don't, they have a unique way in which they get threat intelligence, because oftentimes, uh, the way they see things is it's already bypassed, um, different security controls. So they're looking at different attacks. And we started to talk about APIs, and Danny was mentioning, wow, you know, I, it's interesting the API attacks that, that we see here at Threat Lock.

When we chatted a little bit, I'm like, I think it's a great topic, um, that we really need to explore. Uh, interestingly, you know, when you, uh, look and dig into it, 83% now of all traffic according to, uh, Akamai is service or API based, uh, which is only growing as they, as you read through the report with, uh, 5G.

And consequently, most of the detection tools focus on, and Ryan West, keep me honest here, according to the threat report, more on HDTP and HTPS traffic versus, uh, you know, these, these services. So detection tools on the aggregate, on the margin aren't necessarily doing a great job. And then internally on the server side, the same kind of thing with JSON and xml. So we use as MSPs, uh, we turn on a lot of third party apps.

Um, Wes, I know at Perch, you know that well, and, uh, our clients do also with Azure and oh 365. So that's setting the stage for today. Danny, thanks for joining us. Can you tell us who you are, a little bit about yourself, and, uh, uh, let's get right on into it. Yeah, so I, I'm Dan Jenkins, I'm the CEO of Threat Locker. Um, just to give you a bit about my background, I, I started off in the enterprise security space.

So in enterprise, you tend to approach things slightly different when you think about security than necessarily MSPs do. And, you know, I've, I've found in mul multiple businesses, and, you know, over the last few years we've really pivoted and focused, uh, on msp.

So I've done everything from preparing controls, NIST compliance, ISO compliance, uh, so compliance right through to ethical hacking where I've, uh, broken into probably about 50 or 60 networks at this point where with permission, uh, you know, including MSPs that gave me their permission to try and get onto their network. So I've kind of seen it from all angles on, from a security point of view. Fantastic. Again, thanks for joining us guys.

So he Probably, he probably knows the two main secrets to MSPs. The two things, the two things that MSPs, they're good at a lot of stuff, but there's two things they're not good at. Got thousand selling or doing things Spoken from, Spoken as an Ms p, as an msp, time MSP business owner. All right. Fantastic. Wes, I'm gonna hand this right over to you to kick things off. Coming to you from a beautiful hotel room there in Kentucky. If I, if I could, if I could guess west.

Yeah, I'm here in, uh, I guess the locals would say Cincinnati, but I'm on the Kentucky side with, uh, emerge it partner of ours and had a great event with them. And they actually, I can't believe I'm gonna do this, but we're all friends here, right? They got me some skyline socks. Tell me that's not the coolest thing ever. Wow. Yeah. Much better than the food itself. I was gonna say, the decor on that chair behind you is phenomenal. Yeah. I should be sitting at that chair, shouldn't I? Yes.

Alright, Take it over. Dan. Thanks for joining us. And you know, why don't we just start with the media conversation. Let's talk about APIs API security and why this should be a threat vector. And you know, it's really interesting, I had a conversation with a, an IT company just this morning at the event I was at, and they were like, Wes, let's talk about APIs. Why does it seem like no one is really talking about security from that side of the house?

And I think we are, but maybe not as much in the channel, maybe not as much in MSPs, and yet we leverage them highly. So let's just kick off the discussion today. Talk to us about your view and your angle around APIs as a threat vector and what MSPs should really be focusing on. So, so I think the first thing is, and there there's obviously a huge elephant in the room, which is the, a whole RMM APIs.

'cause that's, that's one of the biggest things we wanna be concerned about in, in terms of if someone gets into your RMM or someone gets into your remote control software of some description, then with an A anyway, then that's a problem. And we often think about, well I've got dual factor authentication, so when I log in, it gives me dual factor and that gives me a token. And then I can use that token for a period of time and it expires after a period of time, which is, which is great.

But when we think about APIs, you can't use dual factor on APIs. And if you do, it's normally to set it up and then that key is available. Uh, so if, if you have a API key going into your RMM or to your remote control software that API key is there, it doesn't change most of the time. Uh, it's open and it allows you certain levels of access. And the problem we constantly see is people tend to forget about them when they're thinking about their security.

That they go in with this mindset of, how can someone get into my system? Well, they can't get in through here. I have dual factor that they can't do this because, um, I have this piece of protection in place. But they forget about with their, all of these integrations I have, they all have API keys and, and quite often those API keys have far too much privilege. And the, there's two areas I I like to focus on. One is, what do we really need to give them?

And secondly, how do we remediate for the, whenever we have these APIs, we add threat to our business. And there's nothing wrong with adding threat or risk to our business. There's nothing bad, bad about it. Risk is something we have to deal with every day, but how do we remediate and reduce that risk?

'cause we have to take into account, okay, I've added a risk here, but I'm willing to add that risk because I want courage to be able to get this data, or I want someone to be able to get this data and feed it in. And, and that gives me a business benefit. What can I do to cross-reference that and how can I remediate it? Yeah. And, and these are, that's really good feedback. And I, I think this is certainly, do you agree that this is an area or just maybe we agree or not?

MSPs are probably not, most of us are probably not, um, fully aware of, um, both the risks they inherit with APIs and probably not even having that much visibility into the capability and um, uh, what's the word I'm looking for? The, the challenges they're gonna run into. Like they may not even know the full capability of what the APIs they consume every day, um, may bring them. Don't you, don't you think that's a challenge? Well, I, I think that's, that's the problem.

And as vendors, we all like to say, Hey, we need an API key into your system. Here's the instructions. Yes. And we, we, we we're all guilty of this not, not saying, Hey, give us as least amount of access as you can. 'cause we don't want more than that. So, and MSPs just tend to follow the kb follow the KB article and say, here it is. Here's the key, everything's fine.

Uh, I wanna give you an example, and I'm gonna not mention any product names, but we, we had a client who was using an RMM tool, and that RM tool used an external, um, uh, remote control software. And the external remote control software had more than the ability to add remote control. So, uh, we had a, a client call us up, uh, one day that someone had requested access to Mimi cas on their RDP server.

And they wanted to know, well, how did one get onto my server to request Mimi cas, they're trying to run it, it's being denied. Uh, and they've requested access. And this is the first I found out about it. And we spent quite a bit of time going through the logs. They'd actually been logged onto their server for three days. Um, and, and they, they didn't have any alerting set up to notify when things were being denied, which was remediated.

Their, their, their EDR hadn't alerted them for whatever reason, but they, we tried to figure out, well, how did they get onto your server? And the first thing we tried to figure out, well, first of all, they got an RDP session, which is another conversation, but how did they get a domain admin password? I mean, the domain admin password wasn't weak. It was, there was all sorts of other problems, but how did they even get that password?

And it, it transpired that they'd got in, we asked them how they connected it, they connect to the service and they had an RM, they said, the RMM. And we said, well that's, and we have dual factor on that. And I said, well, but that's using another product. And we could see these other products every time. Just before the domain admin account was created, we saw a DLL run from that other product, the remote control software.

And what it happened is they said, well, we don't have a login for that remote control software. We just do it through our MM. What happened is there was an API key between the two and the API key in between them had been compromised and they were using that to create a domain admin account, and they had no idea that even existed as far as they were concerned. No, that's impossible. 'cause we don't actually have a log into that platform.

So, well, something has a log into that platform and it's running, and they, they traced it back and they managed to diagnose it and thankfully nothing was actually able to be ran, which was good. But the point is, they're able to get into that system without even, they didn't even know that that existed as far I was, they were concerned, we bought an RMM, it had integration into something and now someone was able to create a domain admin from that. Y yes.

And I think this really goes into, like, we talk about principle least privilege all the time in security design, but it seems like APIs are like the redheaded stepchild there, right? This sort of gets left out in that discussion and we don't really consider the capabilities what the API allows and, and the risk inside of it. So, um, what if I'm an MSP Danny, like, like in your view, what should I be doing to understand that risk and be able to control it?

Like, should I, should I have my team or myself like dive into the API itself and understand, you know, do I have certain permissions that I can grant it? Um, do I understand the ramifications from the vendor themselves of like what it allows and doesn't allow? Like how do I go down this journey of like implementing lease privilege into my API? Yeah, so, so I, I think there's two parts to that. Uh, the first one is, uh, you have to question everything.

And that's a real problem because most MSPs don't have any time. So, so now we're asking them to look question, um, uh, call ConnectWise, call data and say, Hey, what does this API have access to? And just ask those questions so you know where you stand. And it may be that your, your API has more access than you want, or it may be that it needs to have that access. So it it because you are doing those functions with it. But if question, what does it have?

And then remove everything it doesn't need. So if you, if you don't need read a or write access, don't give it right access. If you don't need read access to other areas, don't give it read access. One of the things I always did, uh, when I was doing penetration testing was I was always, or, or ethical hacking was always trying to find out data. I, I love data. The more data I can get, the more I can find out about your business.

I suddenly, it's like light bulbs, they light up, oh, now I know that I can know I can do this. So take away even read access, which sometimes people think, well, it's read access. What damage can it do? Those little breadcrumbs allow attack attackers to move further along in the, in the progress. So always ask the question. The other thing is, you have to acknowledge that sometimes systems are gonna fail. You can have the dual factual authentication on everything.

You can have APIs configured to the least amount possible. You can have, um, everything else in place. So adding more checks and balances, assuming with there I have this, uh, I have an RMMI love my RM, it's one of the most important tools in my business that RMM has lots of power. I'm gonna add a check and balance on that.

I'm going to put security software, I'm gonna put whitelist, I'm gonna put EDR, I'm gonna put things on the endpoints that are gonna control that and make sure that if it does something outside its parameters, then we're gonna get notified.

It's gonna get blocked, and we're gonna be able to put a check and balance in there because you have to acknowledge sometimes I need this technology, I need to open up this API, but acknowledge that sometimes it's gonna go wrong and you've gotta have something stopping. You've gotta have an extra check in there. Yeah, that's good. I, that's really good feedback.

And I, I think that's exactly the steps that I think MSPs and really anybody but especially MSPs, owe it to themselves to really go down that journey and implement principle of least privilege and understand, it's kinda like what Ryan Weeks you've said before in threat modeling, you know, knowing ourself, knowing the enemy, knowing the battlefield, and API security has all three of those combined together and, you know, understanding, you know, what APIs do we have in place, the automation of these things is a wonderful thing to be able to, you know, transfer information from one platform to another and orchestrate all these wonderful things is great.

Um, but you know, what risks do I implement with this? What's the battlefield that I've set for myself here? And then how could a threat actor take advantage of it? So Danny, it really is important for us to spend some time thinking and understanding where does our data flow from what platform to another, and what are the cap, what kind of information's flowing through it, and what capabilities does that have? And, and, um, what permissions can I use to cool down and control all of that?

That's really at the essence what you're kind of saying as we go down thinking about building security by design into our api, into our APIs, right? Yeah, Yeah. Through the APIs, throughout your systems. And then assume that you assume that you don't know something as well, because if you always assume there's something I've missed you, you're in a, and I wanna account for everything else, you, you, that helps you catch the unknown. Yeah, yeah.

I, I completely agree with, with what, what's been said that the one thing I wanna remind folks is when you get to that threat modeling, you need to have that data flow diagram. That data flow diagram comes from your systems inventory and from sitting down and figuring out what has access to what via what APIs. Um, I've worked with MSPs for an entire day mapping out the API access that all of their systems have.

And at the end of the day, when we got it all figured out, the question I asked them was, does all of this need to access all of this stuff? And they kind of just looked at me and it's like, they were just enabling integrations because like, well, it works better if it's integrated. And I'm like, but does your business depend on it? Gimme a value. Yeah, that's an awesome point.

You know, when we were early in building MyKey process, the first question everyone asked, every MSP does it integrate with ConnectWise? And I would say, well, why? What, what do, what do you want from an integration? I don't know. But everything we did is in ConnectWise. So like there really wasn't a really huge value, but like master file and maybe be able to click on a ticket that was created, but like, we just did it.

We built it so that from a sales standpoint, when people said you integrate with ConnectWise and Autotask, we were like, yeah, yeah, Yeah. And look, we did the exact same thing. I mean, we, we added integrations into everything and there are some values associated with them, but in many cases when people say, does it integrate our, our our ans and it used to be no, but what, and we struggled so much, what do you want in the integration? We're gonna go off, we're gonna build this for you.

We're gonna, we're gonna put this onto our pipeline. Tell me what you want to achieve. And half the time it was like, well, I don't know, I just wanted to integrate Exactly it, man. And so sometimes building something to say yes is like, takes a main obstacle out of the sales, like friction outta the sales process.

But you've gotta ask yourself, why do I want, I mean, we have integrations with various platforms, but why when you turn that on, even from us, when you turn it on, what do I want to achieve outta this integration? And if you don't, if there's no business value, just don't do it. There's two things you've added together and there's there's no benefit to it. Hey, Brian, um, just because of what you were just saying, there's a question, how do you go about limiting the access an API has?

And, and so that's question number one. And, and you know, did, does documentation have to be there even at the start? 'cause that was another thing I'm just reading through the chat. Yeah. Uh, you know, so Yeah, there's, I mean there's, there's many layers to this question, right? So you're starting at the bottom of the question stack, in my opinion with that question.

Um, and the answer is usually, um, sometimes some platforms will just let you generate an API key and they give you the key and you have no idea what access it has. That's when you need to be talking to the vendor. Sometimes the vendor will let you, um, customize the access based off the platform. So I don't know a lot of platforms, so I'll talk about data products like Autotask, PSA, you can define very specifically the permissions that the API user has, right?

And then other platforms will be, here's an API key, you need to give it admin access. And that's where that lease privilege stuff falls apart. So the access is important, but the access is important for two reasons. One, the thing that the API is giving them access to does something dangerous, it lets me connect to all your servers. It lets me delete your backups. It lets me run arbitrary scripts. It lets me do something that an attacker would do in their, in their attack chain.

It makes it attractive for them as a tool. If that doesn't exist, uh, then you're a little bit safer. But if you're getting admin access, there's still, if there's a vulnerability in that API and I can make that API do something it's not supposed to do, it's gonna have the privileges to do it. And so you wanna really be understanding the access that that happens. And then there are other things further up the stack, which is how do I limit the accessibility of the API?

And this is the thing that I think as a whole in the IT channel all vendors suck at, including me, and we all have to be better on, is who is accessing APIs from where I actually have my team doing a data analysis study right now, the past year worth of API access of all APIs from all MSPs. And the question I'm trying to answer is, what would the impact be to MSPs if I restricted their APIs to certain sets of IP addresses? How frequently do they change?

And, um, and that, you know, we don't have that yet other people do. Um, and I think that the IPA, uh, restriction for API users can actually go a long way in, in mitigation. Now, again, it's not perfect because I could compromise, uh, an environment.

Like especially if you have an on-prem server that has an integration and you've permissioned your local onsite network to access that cloud, API and your network gets breached some other way, well now that attacker can leverage that, that IP access. So it's a really layered problem. And, and that's why you, and you gotta go back to that. What, what we, what I just did was a threat modeling exercise.

And what I did is I thought about three or four key things in a technology stack, your backup platform, your PSA platform, your RMM platform, and I didn't mention it documentation, but those four platforms in my experience have the most API access and API integration. The other thing that I'll note, I'm kind of going off script here, I'm pulling, Wes, you don't, I like it. Be worried about the API access to the system you need to be worried about.

So if I have a PSA and an RMM and the PSA has the R-M-M-A-P-I key, well you're worried about the access and the RMM and the IPA, uh, restrictions and the, uh, functionality of that API and then you just go shove that API key in the PSA, is that API key being securely stored in the PSA if are all users able to see that API key? How easy is that API key to steal if I get access to your PSA?

And that's what I'm saying, you really gotta think through what integrations you have enabled because every place you turn that integration on is another point for an attacker to steal your API key. That's awesome. Ryan, Danny, did you have anything to Add there? Well, I, I, I think, I think that's it.

If, if you have an RMM, which typically has godlike powers and you, you counteract that with whatever security software you have on your endpoints and on your RMM, you wanna make sure that the RMM isn't now accessible by five other the products. 'cause now you've got five points of vulnerability versus one for no or very little apparent reason.

If you can limit the access and it's not just limiting and you mentioned about vulnerabilities and exploits, it's also limiting, you know, we think about, on my API typically points to a SQL server of some description or some kind of database server and making sure that the permissions between your API and your databases are limited to the most amount possible.

So when, if you have some kind of vulnerability SQL injection or something in your API or your, or your UI or anything at that point that the amount of data it can put into your SQL server is limited. If, if, if it doesn't need to modify this table, don't let it modify this table. And then if it is compromised, it's not going to be able to go past where it's supposed to go. Excellent. That's good.

So, you know, as I turn over to Gary, uh, I think some big takeaways that hopefully everyone on the call is, is hearing is you've gotta put some due diligence into understanding, as Ryan said, data flow diagramming, understanding what data process flow I have from one to another.

And you can even start that exercise with what vendors do I have in place that store process or transmit my information and then begin to go to say, what, where does that information go and how does it flow from one to another in the workflows and processes we have? And, and I think the next step that you have to go down this journey is understanding, you know, what kinds of API permissions and capabilities do I have that I can call away that I don't need.

And then lastly, I think we're also illustrating a little bit more around, um, we've, we've gotta put some pressure on all of the vendors to do a better job of giving us better controls around that granularity and to introduce that into the system so that we can actually put the due diligence in place and not just force, you know, admin controls everywhere.

Because I think this is sort of like a prophetic, um, cyber call today in the sense of like, we're going come back and look at this in a year and be like, man, I remember when Cyber Call was talking about all of this back in the days of innocence and all of a sudden it's gonna come back and bite us all one day. So I I just really feel like this is a, uh, this is a time for us to really get in front of this versus, you know, like what we happened with all of the RMM breaches.

Um, you know, this kind of hit us with a storm and we didn't all fully see it coming. So, um, really good feedback. And um, Danny, thanks and Gary, ill turn it over to you my friend. I have a quick question for Danny. Have you seen an attack yet where the attacker leverages just the API access or are they leveraging the integration between two products and having gained access to one through something like credential breaches or lack of MFA or, uh, application security vulnerabilities?

So, so we've seen a attacks, um, where an a p key was used in an RMM or actually RMMs. I won't. And it's interesting 'cause we go back to July 4th weekend and we, we saw 45 clients, um, have ransomware attempted to be pushed out to them that 45 MSPs that weekend. Um, and it was, it's all terrible news and I mean, it was blocked, but it was, it was obviously big news. We typically see somewhere between seven and 10 in any weekend, uh, RMMs be used to push out ransomware to clients.

And quite often those are related to an API key. Uh, not always. Some of it is lack of dual factor authentication. Some of it is someone got onto an RMM server and reset something in a database. Uh, but quite often that happens because someone literally got an API key and then pushed out, uh, ransomware or attempted to push out ransomware using the RMM just that weekend. Um, and that happens, that's a normal week.

It's, we see it all the time and unfortunately sometimes it's an API key between two things, but other times it's someone just got an API key or they got into the RMM through a token that didn't expire or something along those lines. Yeah. So I wanna go one step deeper into that. Are they using the API key to create users that they then use to access the platform? Are they actually using API functions to deliver the ransomware?

I Think, um, and I don't think what it's, it's not one or the other. I've definitely seen, um, I've seen in personally, I've seen domain admins accounts created, which is not a user on the API, but I've also seen, um, I haven't seen a, a user being created on the RMM, but what I have seen is, um, that doesn't mean it hasn't been seen by somebody else, but I have seen the software being pushed out by the RMM through using an API function. Okay. Obviously It's a manual process.

If you create a user, the attacker now has to log into your server and they now have to, uh, push it out. If they can do it just through the API key, then that's even better. They can automate that and make a lot more money faster. Okay. Yeah. Sorry for the sidetrack. I just wanted to get some threat until there. Gary, you wanna, you wanna take us, uh, next? So I, I wasn't gonna say anything, but I guess this might be a good time to announce that Wes and I are about to launch our new software.

It's called API Locker soon, powered by Omics Baby, powered by Omics. So on the blockchain, lemme just gonna, I a, Its just a firewall. Just close the poll. I just wanna zoom out for a second because, um, I think I mentioned on a previous call, we looked across our couple hundred peer members and they average now 14 tools in their stack. So that means there's people that have between 20 and 30 tools in the stack. And so the first thing, and many of them are integrated, right?

Um, now we look at the poll question that says that 83% of MSPs that are on a security podcast, right? Really don't have a process around that. Um, I don't know if we can depend on vendors, right? Um, whether it's one of the four big vendors or whether it's startups, they all have a lot of other priorities, right? Uh, in terms of scaling their businesses and investors and, you know, those kind of things.

So this is something concerning and like, uh, you know, I want to just piggyback on something like, Ryan, you always boil things down right to the, like the kind of the, the steps and the least common denominators. But you said something, you know, R-M-M-P-S-A and documentation, like they're the center of integrations, right? So like, if you're gonna start somewhere, start there. And even if you can't go through in some of those key ones, even if you could look across your stack, right?

And say, of my 18 products, do all 80 of 'em need integrations with these three things? Or could I live, you know, could I just simplify my world, you know, uh, a little bit. Um, so it kind of goes back around at that point to what you said, if vendors and MSPs either can't or won't, or unable for a whole bunch of business reasons or to, to do as good of a job on this, then it really comes down to figuring out how do you know early on right. When something's happened, right?

So that's kind of what it goes to. I think that's what you're saying, Danny, right? Well, I, I think I, I mean, I, I think knowing and also controlling what happens, so don't look the end of the day, one of the best things about an RMM tool is it can push out software. One of the worst things about an RMM tool is it can push out software. And so it, it, it's, it's the best and the worst of it.

And, um, it, it, it's, so what you wanna make sure is that, look, that's something that's a feature that I'm buying. I need that. I need that feature. So control everything that can access the RMM to limit my exposure to just the RM itself, um, that can, can do that, but also then control what software the RMM can push out. So make sure you have other technologies in place to say, Hey, we've got white listing, so it's not gonna be able to push something out that isn't on my list.

Or we've got EDR as well to make sure that we're gonna detect on top of that. So knowing that early and controlling too. I don't think anyone, I don't think it's unreasonable for any, any MSP to have to approve every script through a third party or an external system, of course, if it's, or even better.

Um, but it's, I I think it's a, it's a reasonable step to say if we have a new piece of software we're pushing out through on RMM, we have a separate tool that is yeah, a check and a balance there. We've got Congress and the Supreme Court making sure that they, that, that this tool cannot do something without the approval of a extra step that has separate authentication systems, that has a separate, that doesn't have API keys that can just push from one to the other.

Because if you do, then that's a problem. So, zooming out from this, you mentioned what you see overall, like with all the other things that MSPs have to deal with today. Like how big of a threat do you think this is for MSPs right now? So it's huge. So it's the biggest thing, the biggest MSP business ending threat we block is stuff that happens from an API or from an RMM through an unexplained push out.

Uh, something happening for either, not just your RM but you gotta look at your antivirus software. Does this have the ability to push out software? And it annoys me when they do, but that's another conversation or other tools that may have the ability to run scripts on your system. It's the biggest thing. We are seeing the things people don't think about, how did someone get into my system? Because if, if I go interview a hundred MSPs, they're all gonna say the same thing.

Well, I've got dual factor on my RMM, I have restrictions in place. I have, um, uh, you know, I, I have, um, good policies. I have an EDR and yeah, it's the biggest business ending threat we see on a weekly basis and we block on a weekly basis because something happened. And sometimes what happens is they didn't have it on all the endpoints. They didn't protect all the endpoints that their RMM gets them. But I would say, and it's not just one vendor.

It, it's absolutely not one, just one vendor. But, and there's so many of them that's terrifying, that are unexplained. We never find out, well how did they get into RM? All we saw was this software got pushed out by this RM or this antivirus or this remote control software. And we see that was blocked. And like I've worked with MSPs on looking at logs and event logs and say, well, how did they get in? We don't even know how it happened.

And APIs often come back as this could be the API that's the only logical explanation. It could be a vulnerability that we didn't know about either in the UI or the API. Either way we know it happened. And I would say it's the biggest risk for MSPs now in business ending threats. Yeah. Something that's gonna your business across the board.

So, uh, Andrew, Ryan, Wes, do you think some point in the future, one of our quarterly kind of deep dives might be good to do a, like a workshop, you know, a around this, the way we've done around, you know, threat modeling and, and incident response? I don't know, you know, to give people like maybe a thought process or, or a reason. Um, you know, I'll share, I'll share something why I think of that now. 'cause people do go and they come to these and they watch 'em.

Uh, I had a, a customer that was, um, involved in was one of the unfortunate drew the a short straw, uh, in the Kaseya, uh, event. And, um, they're in our peer group and one of the things they said was, you know, we encouraged everybody to go do those IR workshops. We did two of 'em, um, with the people on this call and said that it was one of the things reasons why they got through, they're gonna survive it and they're gonna thrive, you know, on the other side of it.

But a few key things they took out of there ended up being game changers. So I wondered, Andrew, if maybe at some point we should bring that up and maybe it's something we can look at. Yeah, well, in fact, I don't think nfps know how to do this. Most of them, I have a call into the folks that do, um, the, uh, threat modeling and adversarial ation, um, uh, in, in that antis siphon training. They're, uh, one of 'em heads up threat at Walmart. And, um, the other is one of John's employees.

They do the course. So we'll see. But I think it's a great idea. Yeah, Yeah. The guy that heads up threat at Walmart is one of the best in the world. He is amazing. C can can I make one of other observations? Well, you might not be able to address all of this now, but you know, we, we run this philosophy, we always wanna keep improving. Um, always keep improving your security. So you might look at this and say, I've just got too many APIs. This is gonna take me too long.

You don't have to do everything in one day. It's not a case of saying this isn't gonna work perfectly, so I'm not gonna do it. Go off, pick one system and spend an hour on it, go through and spend an hour. How can I improve my security here? Um, how can I limit the API access? Do I need this API, can I take this away? Can I remove this software from my system?

If you operate your security on the basis, I'm just gonna improve one thing a week, you're gonna be a lot better off faster than you can imagine. If you try and solve every problem today, you're not. And you know, we do QBR with our customers. I'm always like, okay, let's look at what you've got here. Let's see if we can improve your security, reduce your privilege, limit access to your system. Just one thing.

Even if it's one thing now that you're gonna be one step better off than you were before and before you know it, you are a lot better off across the board. So even if you've got loads of APIs and you, you leave this call thinking this is just too much, I'm done. Um, don't just go, just go and pick one API go and pick something that's, um, you know, you think is, is critical your business. Have I got an API between my backup and my RMM? Let's just scrap that one. That's a good idea to remove it.

Like remove as much as you can in terms of connections where they're not needed. And if you, if you remove one thing today, you are, you are one step better off than you were yesterday and it might take you 10 minutes at the end of this call.

Hey, what just made me think, Gary, as, as what Danny said and what you said and looking at the, uh, people out here, what might be really cool is, um, even pulling in guys like Jason Slagel, Tim Farnet, Calvin onto that call, so that, you know, people that are in the trenches using them can kind of collaborate on, well we, you know, we don't really need that. Here's how I would approach it.

So maybe it's a combination of the, of the two, um, which could Again, Andrew, what I wanna say to people is if you go back over 63 calls, almost every one of them, there's something else that we're talking about where part of the answer is time, attention, and process, right? And we keep building on it. And so if you're an MSP today, that's set up like many of them, you don't really have a hundred percent dedicated proactive roles, right? People wear multiple hats.

Everybody is doing some tickets, everybody is billing some hours or working on, you know, delivering service. You're gonna have to at some point have a resource and eventually resources that are focused on this. That's number one. And two, you're going to have cost it into your model. That's why we're seeing, I'm watching three deals today come across from MSPs $187 a c $192 a C and I'm talking 40, 50, 80 users. This is the reason why.

And so we, when you hear Danny say like, he thinks this is the biggest threat, and 83% of people that are on a cyber call are saying they don't have a process mainly 'cause they probably don't have the resources, we've gotta start making the turn. Man. Good stuff Gary. Excellent point. You got anything else, my friend for Danny? Um, let me check my notes here. Yeah, sure. No, I think that's it. I think that's all I I had for him. Okay. Awesome. Over to you Senator Weeks.

Yeah, so I'm just, um, I'm thinking about that conversation, right? I referenced this question stack of things you need to think about for APIs and resources that MSPs can go and learn more. Uh, OASP is coming to mind. Um, and they have a lot on, uh, securing APIs and, and just material to help you actually build into a threat model for APIs. So Danny, um, for those that don't know who and what OSP is, can you give 'em a quick overview and um, I'll actually put a URL in chat while you do that.

So, sorry, who, who, who was is, sorry? Osp. O-W-A-S-P. You familiar with them? Oh, I, I I am not familiar with that. Okay, Perfect. I I feel like, um, I should, I I I'm missing out. Yeah, so oasp is basically a open web application security, um, consortium that's basically AppSec professionals. They get together and write best practices and standards. Um, so if you googled OASP and API security, you're very likely to come up.

There's actually a project, the Opsp API security project, um, from 2019. Uh, so you know, oasp API security top 10, and it has a listing of actual, um, of issues that they've identified with APIs. And so if you wanna go kind of like we, we think about a threat actor and Mitre as this database of things that attackers do.

Oasp has these classes of web application security vulnerabilities, and then they go a level deeper and they can actually highlight for you what the issues are with certain classes of application security issues. And it's a really great place, especially if you're interested in like web application hacking, which is Huge, but they're actually breaking down the security of the app. They're actually talking about the vulnerabilities in the API as opposed to So as a developer. Yes.

Uh, and that, and you, you, it does bring a good point though. We, we hire a lot of interview and hire a lot of developers. Um, I would say probably one in 15 developers that we interview are able to spot SQL injection. Now that should scare the crap out of everyone. Yes. Yeah. So, uh, so like these are basic score boy errors is what I would see, but I would say one in 15 that we interview can spot it. Yeah, that's definitely scary.

Um, especially if it's something as easy as concatenating the user supplied input into the query. Um, you know, that's hard coded in that should be pretty easy to spot. So, um, but yeah, there are resources out there, so if you're like, I I want to ask my vendors about API security, but I, I just don't know what to ask. Oasp is actually a great source to go learn more. Um, um, so they have the top 10 API security vulnerabilities, which um, I think I just popped in there too.

The, the wubs um, security project. Um, Gary mentioned common web apps like O 365 and Azure. Are there any recommendations you have for MSPs so they can mitigate shadow IT third party apps that might have APIs in their environments that they don't even know about that could be being used against their customers? So Office 365 is somewhat terrifying 'cause there's more and more apps getting added in there.

And it's important as an MSP that you go in and you do not allow apps that aren't approved by you to be added. Well, I mean obviously on your end points, but also into with the Office 365 limiting what has access to your Office 365. So you can set it up, you can configure which apps can be approved to access your data. OneDrive is a huge backdoor into your system. 'cause I, I wanna tell you how I got into someone's system once, uh, again, with their permission.

I, I was trying to get somebody to run malware on their machine and I was able to get in through an Office 365 account that I, I did a phishing campaign, got someone to gimme their password. They didn't have dual factor, all sorts other problems. But I was able to, um, and I've seen this over and over again, replace files in OneDrive with executables that had Excel icons on them.

So someone suddenly goes into their OneDrive on their computer, they double click on this Excel file, of course Windows hides file extensions. And now we had, and by the way, we publish this software, if you ever wanna see just how simple the code was, we publish it in our MSP resources. An example, um, we're seeing getting into your OneDrive, it is terrifying.

So whenever you allow access to Office 365 or you allow third party components, see what they're allowed access to, if they're allowed to create files on your OneDrive, if they're able to access your email and change things in your email, you're potentially leaving yourself open to someone using that as a back door into your system. Perfect. So I wanna go back to earlier when I was asking you the questions about have you actually seen attacks leveraging just an API key and not an integration?

And the answer sounded like yes. Um, if MSPs want to threat model on top of their own environment and their own data flow diagram, um, I think what would be great is we could actually give them three use cases to model. So one I think is, um, basically anything that has API access to your RMM is a good one to model, right? Um, and then, um, what else do you suggest that they they threat model out from an API perspective or integration security perspective?

So, so I, I think you probably wanna expand the word RMM to anything that has API access to anything that has, um, RMM like powers to your system. So include if you, uh, and look, some EDRs require consoles to do cleanups and, um, push out software and things like that. So making sure your EDRs or anything that has console access or any kind of access to your system should be included in that. Uh, that should be your, your f first thought as an Ms.

P 'cause that's what's gonna put you out of business. The other thing that's gonna cause problems is access to your customer's data. So, um, I I I, I feel like, uh, da getting two phones of this, uh, so your, your backups are your data. So anything that's got access to API access to your backup, and realistically, you should have very little API access into your, your, your backups. Absolutely. I mean, there's, there's not much integration that needs to be put into your backups.

So, uh, your, your backups or, or anything that holds massive amounts of data. So if you've got API access in if you're using AWS or something like that, but I would be focusing on that. But as an MSP, your tools are typically gonna be, I, where do I as an MSP hold my customer's data? Um, it's, or have access to my customers, my RMM or, or like-minded tools, my backup or similar tools, file sync tools and things like that. Yeah.

But again, most MSPs are, are dealing with, um, yeah, I refer Security software. Hey, Ryan, you know what I was gonna say? It's funny, we always talk about how MSPs have to have conversations right? With their customers about security versus is convenience. Yeah. And even sometimes productivity, but MSPs always are trying to get to convenience and productivity, uh, on, on our side. Yeah.

And I think there's, there's a balance between providing what the MSPs think they need and what they actually need. And then as Wes said, I think there's a problem where there's no, um, there's a lack of accountability. Um, and I, and I've, I've said this before, collective bargaining peer groups are a great way to get together and assess your vendors and, and do put some pressure on them to improve.

But frankly, I think, you know, there's a, there should be healthy interactions between vendors around APIs. Um, I think coming off of the July 4th weekend, we saw some unhealthy, uh, uh, situations between some vendors, um, and APIs. So, um, but I think there, you know, we're the ones ultimately building the integrations. And I think that there needs to be some accountability there too.

Like I think we, we need to be able to say, um, when we think there needs to be improvement, um, uh, in that as well. So do you, Do you mind me asking you a question and Yeah. I should know this as, as we have a data integration. Um, do, do you ever do review code reviews for Absolutely, Yeah. And like I said, I should know it, but I, I wasn't involved in the project. But you do code reviews of your integrations because yeah, I should be standardized in any organization.

So we have a, a product manager that's in charge of API integrations, and I sat with him two years ago and I said, here are all the things you're allowed to do. Here are all the things you're not allowed to do. And so a lot of what we've done now is actually said, you can use our API to build an integration. And for certain dangerous tools like backups and RMMs, I call those meta APIs. You can get metadata about the RMM or about the backups, but you can't access the backups.

You can't read what's in the backups. You can't destroy the backups. You can get information about device audits from RMM, but you can't run a script, you can't run a job, you can't do those things because those are too dangerous for APIs. If you need to do that, go into the tool, use the base functionality, don't do that by the API.

Um, but then you have a tool like PSA or IT documentation, those are places where you store data they're gonna read and, and, and those are more tricky, which is actually why we have a higher permission level on our PSA platform for API users, because we realize you're gonna need to customize it based off of what you needed to do. And so there's a highly customized level of permission there.

And, um, you know, we, we didn't advertise this, but the July 4th weekend we shut off our Kaseya integration too. But we did it just for PSA because we knew that our Kase integrations with BCDR and RMM we're totally safe because we understand the permissions that were given to them, right? And then once we, we followed the, the kind of threat intelligence that was coming out and we determined, um, we, we no entrust FireEye.

And so we said, we know FireEye has basically given them the all clear to turn it back on, we're gonna turn our integration back on. We didn't write an article about it. We didn't, you know, do anything. We just did what was right to protect the MSP. But frankly, I think it's, you know, and so that sounds great, but like we were figuring some of that stuff out on the fly.

And to me, like where, what, what we did that weekend, um, we created a survey, an inventory survey to all of our product managers and said, all of the APIs that your products have integrations into, list them which direction, who owns the API and a whole bunch of questions. And we're actually building out an entire inventory of APIs right now so that we can start to figure out how we're gonna attack this problem.

And we actually have an API security epic in our upcoming sprints in data RMM, um, to try and provide some additional security control around APIs. So this is a very timely conversation. I mean, I've often talked about, you know, playing battle with attackers as a chess game. And to me, API, you know, Kyle and I actually said in the first channel collaborative webinar in 2019 that APIs were next.

Here we are, APIs have been hit and now we're saying they're being hit, but it's really the next wave. It's the next big thing because we've gotten good at MFA now. So now they're gonna, now they're finally gonna shift. Frankly, I think it took too long for us to get to this point, but, um, it's gonna shift and it's gonna be a massive wave now. And, um, yeah, I think it's time.

Everybody really needs to start thinking really seriously about their API security and they should have a conversation with their vendors security team. Just like, just like I just had and said, here's where we're good. Here's where we need work, and here's what we're doing about it. Right? That's what you wanna hear from your vendor. So maybe this was a, you know, a little early, I'm jumping the gun to announce Ryan and i's launching of a new company called the API Consortium.

We're just gonna take over API development for all vendors. We're taking it so that Ryan can, we can make sure. That's Actually a great point. There are a lot of MSPs that use third parties. It's kind of like in the enterprise world, MuleSoft, MuleSoft is basically this giant software that understands all the enterprise APIs and lets you connect them all together. Magically we have those things in the MSP space who's doing diligence on them, right? Yeah.

Well that's a, that's a, that's one super that, that's a day that's gonna make a July 4th look like a, a real, a holiday again. So, uh, yeah, because Zapier, Zapier, yeah. So it's, it's, you've got all of this access to all these different things, and that's terrifying.

I think what's important, uh, from an API point of view is when, when you do issue API keys, um, when sure that you, you can get the least amount of privilege on it, just make sure that, right, like when we issue API keys, you can choose, okay, this API can do this one function and nothing else. It can, it can, and that function could be powerful. It could be to approve software, but nothing else. So you choose each granular control at that level.

Then you are not saying, okay, I've got an OPA open, API key that's open to everything and just turn on what you need. Yeah. So I, I mean, I, I wanna, I've never done this before, but maybe we can pull next cyber call. I wanna see every MSP on this call in the next, I don't know, let's say two or three weeks reduce their API integrations by 10%, not increase. I want you to reduce them. I want you to figure out where you have an integration where it's not providing business value.

And I want you to shut it down. I wanna see if you can do it. I wanna, I wanna see, and I think 10 percent's pretty achievable, right? I mean, that's like, if you have 10 integrations that's shutting one off. So that should be pretty easy to, to do. Um, so go see, go and I think this will get you asking questions. This will get you building your data flow diagrams and you'll just learn a lot in the process of trying to hit that goal. It's, it's not a pass fail.

If you get 7% fine, that works too. But like, I want you to, over the next two to three weeks to have your integrations actually decrease and not increase. That's excellent. Great stuff, man. And again, uh, I'll tell you one thing, one added benefit I get from being part of this is we have sessions like today, I, I take this, I'll be, I'll be creating an audio message for all true methods members. Um, probably a special, a future special project we'll do with our peer.

Like, so much of these things that come, you know, we have a little more influence right over customers than we do, uh, you know, radio listeners. But, um, really today I thought was really, really awesome. Yeah. Fantastic. Ryan, you got wrapped up your questions as well? I, I think so. Yeah. I kind of went, I, I I kind of stayed on script today, so I know this was, Brian was at, you're at your best today. Thanks. Must be the week of rest.

I had, Hey, there's a question here, um, from Lisa Mitchell. How, you know, what do you recommend in terms of rotating API piece? Brian, you wanna start like, you know, is, is on some like, so Yeah, it rotation is, is interesting. Um, RO rotation, it is kind of like rotating passwords, right? Uh, so how often to rotate or if to rotate is the question. And that's gonna be based off of a bunch of other questions are if someone steals the API key, can they use it from anywhere in the world?

You probably wanna rotate very quickly or, or frequently, somewhat frequently to avoid that happening or, um, you know, uh, avoid them being able to reuse that within a certain period of time. Now how long to rotate them is gonna depend on your tolerances. How long are you willing to have someone be able to misuse the API? Now also to rotate is gonna depend on how accessible is that API key. So again, if you're using one of those big four platforms and API keys are everywhere, right?

One MSP, they had A-P-S-A-A-P-I key being used in 14 different integrations. The same API key. That was painful to shut off. Uh, yeah. Especially when uh, we went back to them and we said, you need to at least have a single API per integration. That way you'll know which integration was the one that the attack came in through. 'cause right now it could be any of these 14 platforms and that's not gonna help you narrow it down. Right?

Um, and then rotation also comes down to um, uh, kind of those other compensating controls. Can you do something dangerous with the API or not? Is it one of those metadata APIs that I was talking about? Okay, is it a least privileged API? Okay, that's probably a lower risk. So you almost need to go through and ask questions about the API and the API. Um, but the API itself, what can you do? What type of access? Are there other access restrictions in place kind of tier them.

You can put them into two buckets just to make it easy, like high risk, low risk, or medium risk. 'cause I don't think any API is purely low risk. Um, and, um, you know, and then, and then say I'm gonna focus on the high risk ones and I'm gonna figure out what additional controls I can put in that maybe I haven't enabled that are available from the vendors.

I'm gonna have conversations about those with the vendors and ask them for the things that I need to do to keep them safe or ask them what their plans are. And if I don't like what I hear, then that's one of the things you can control is how often you rotate. Um, and especially if it's one of those really dangerous ones, I would say you probably need to be doing it at least monthly. But again, um, there have been attackers that have gained access to environments and attack within days.

And then we've seen ones where attackers have been latent for nine months. So you're not gonna mitigate all API based security threats, but you can certainly cut off the ones that are sitting in there using that API for many, many months at a time. Great. Great stuff. Danny, anything you'd like to add to that or what else?

So, it's funny, office, Microsoft changed Office 365 policy to consider you high risk if you have a password expiration policy now, which is, I'm sure there's someone in Microsoft that understands the logic behind that. But the, the, there is one thing I would say and, and I, I remember working with someone in the UK and I managed that you, well, it was a penetration test to an MSP and I managed to get his RM RMM password from him.

And um, and the reason being was because I was able to hijack his clipboard and he handled that password. So when you do recycle your API keys, be careful about where you're recycling them and how you are transacting that. Because essentially, you know, it's like moving money from one bank to another. If you drive a million dollars in the back of your car, one EV every week to another bank, it makes it harder for you to find your money.

But if you get rubbed along the way, it's, uh, you're in trouble. So when you are recycling them, or when you are setting any API keys, make sure you're doing it from secure systems systems. You're comf comfortable, no one can get access to that. You're not connected to other systems and things like that. Yeah. Don't, don't store them in your IT documentation store and plain text. Either your PSA, they don't belong there. Well think it should be deleted.

I mean, they don't need to be stored anywhere, Right. And I think a lot of MSPs, like, what's the worst thing that happens? You, you lose it. The integration breaks, you just generate a new one, right? It's, it's not like losing your MFA token, like don't store them because that's just creating more risk for you. Great. Well, we'll, we'll wrap it up there, Gary. So closing thoughts. You're gonna do a daily message on this. I think that's awesome. Anything? Yeah.

I'm gonna start making sure I promote awareness, uh, more and more do my part to amplify this. And, uh, Danny, thank you so much. Great job, uh, today, you know, on this, um, really appreciative. Yeah, there were some questions. Danny, can, there were some questions on, on Threat locker. Can they reach out to you about that? Yeah, and yes, They can, they can reach out to me if they, if I can't, uh, answer it, I'll, I'll forward it.

I know if someone will forward it on for me if they just email Danny at Threat Locker. Okay. Um, I know, I know there was some security questions that came up in the chat as well, and, uh, it's not that I, I avoided answer 'em. I just wanted to keep the, the topic on topic. So if anyone has those, you know, shoot 'em across as well. Yeah. So danny@threatlocker.com. Yeah. Um, and, uh, yeah, Danny, thank you so much for coming. Wes, Ryan.

Uh, Wes, how about you closing comments from Kentucky, that beautiful hotel room with an incredible disco decor. Uh, anything from, Yeah, it's the, this Hilton 1970s decor is quite, quite nice, isn't it? Uh, yeah, no, nothing else. He's read an old Kentucky, it's Just A chrome down here, Ryan. Um, I, I think this is an important conversation. It's a good one. This is a, again, a whole area spiderwebs.

Um, uh, and the, you know, I think start thinking about this and please take that 10% reduction challenge. Seriously, I'm doing that. Even if you don't achieve the reduction, you are going to learn something very important about your business. Yeah. Uh, being a student of your business is really important. Um, especially as a security practitioner. Your job is to be a student of your business all the time so that you can be helping it reduce risk.

These little challenges like this are a great way for you to learn. Um, the last thing is actually, uh, more of a direct message to Danny. Um, I'm assuming that if Datto was included in any of those vendors, you didn't wanna name that, I would probably be aware of it outside of this. Yeah. But if you have seen anything or do see anything, please reach out to me directly. Andrew can get you in contact with me.

Um, you know, I, I would, uh, you know, I think the challenge I have is, um, this is gonna not gonna come off right, but like a lot of what I learned about RMM attacks is actually coming from other people in positions that see what's happening with other vendor technology. And so I'm trying to consume that and stay ahead and learn the adapt from that. So whatever you're seeing, if you're willing to share I would, I would love Yeah, We'll, will do. We'll, we'll link up afterwards as well.

And anything we find, uh, we'll always, uh, share on, uh, and I, I'm just being very cautious not to, I, I mean, look, no, no one's house is spotless, so I, I'm just being very co uh, I know. I'm, I, yeah, I Very cautious not to name people or, or throw people into the bottom under these calls. Yeah. I mean, listen, and like Slagel's listening, he knows like we have a, we have a vulnerability disclosure program.

If you see a problem with our API and it's a vulnerability, submit it there, you know, even if it's feature requests don't belong in VDP programs, that needs to go through the, like, the community. But, um, by The way, I don't have anything just in case anyone's avoiding doubt on the call with me. Okay. Me, me annoying. Um, but I think we'll, we'll link up afterwards, so I've got your contact details. Yeah. So, um, awesome. Yeah.

Thank you so much and I really appreciate you coming on today and, and being part of this conversation. Okay. Thank you. Thanks Dan, everybody take care. We'll look forward to seeing you next week. Make it a great one. Awesome. Thank you. Bye-Bye bye.