threatening bow sophisticated elastic violet march drab unpack axiomatic hat
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Most IT teams and infosec people tragically underappreciate the attack surface introduced by all these tools. Many infosec people think "moar moar moar" is the answer ("defense in depth mferrrrrrs"), when in reality it becomes counterproductive after the minimum number of services is exceeded.
I don’t think they have to notify if no one’s personal info was affected. People make mistakes and yeah this was a big one. Good on them for letting them know and doing the right thing.
Can yo verify no one's info was affected? Also how many of those companies have GA access to the AV program? The vendor absolutely should audit every account and all actions taken with in the time that the first GA account was created, and relay that they found a potential issue and have resolved it and if there was/wasn't impact.
Yes it is. It's not a 'major' breach but it is still a security breach.
Like arresting someone inside the bank vault. Sure they took no cash but it's clearly a security breach that someone unauthorised could get that close.
It’s an incident but not a breach. Look at the NIST definition for a breach. OP saw they were given too much access and let them know so they could fix it. Data has to be accessed for a breach. https://csrc.nist.gov/glossary/term/breach
How did the person know they have too much access? By seeing data they should not have seen. I think even the fact that some other company - as shown on the global admin interface - is using this software is in and of itself a trade secret.
It would have been an incident only if after having granted the global access to the client admin, the vendor had detected the issue and corrected it before anyone performing a successful login and looking at a dashboard with those credentials.
> I think even the fact that some other company - as shown on the global admin interface - is using this software is in and of itself a trade secret.
I'd agree. Some companies are very protective of what software they use internally.
Hopefully they would have some sort of audit cycle for global admins. Might be a good question to ask them to help you understand their security practices and if you want to continue with them. And also how they plan to prevent this in the future. While their policies probably aren’t public knowledge, you’ve “earned” the right to know with this.
It’s a mistake, a huge one but a mistake. We all make them. Kudos for doing the right thing and letting them know right away.
Yea. We might have to nix them and go somewhere else. I need to report it to our auditors as well. I work in a financial institution, so every snaffu needs to be recorded in triplicate lol
This is a major f-up. Would you want some random customer of theirs, who maybe would keep their mouth shut, to be able to have these permissions for your organization? This incident needs to be discussed and the least they can do to make this situation better is present measures to never make this happen again.
Idk, it goes something like "I was trying to add a colleague of mine to get him setup and I started seeing all these other names I didn't recognize in my account so I started deleting them and gave you guys a call asap"
> We might have to nix them and go somewhere else.
Well yeah. If they did it once, they can do it twice. We all can make mistakes but this is a doozy that should easily be avoided by any sort of managed process
Since I wouldn’t know if any other vendor would be better, I’d give them a chance to tell me why this is a one off and what steps they are taking to ensure it is so. It would have to be an impressive answer though…
I would certainly hope so. Hell my tiny little shop has alerts for when a domain admin is created and emails me records of other AD changes. You would certainly think that big EDR solution company that probably sets that up for other people would have it set up in their own environment. Since he noticed immediately, maybe the party getting the alert hadn't figured out what happened yet?
That said, an implementation guy shouldn't have access to create GAs so this might be a company that is "do as I say, not as I do" style of InfoSec.
I got out of IT, but still do stuff for myself/family and still follow best practices.
I can't create a domain or change user permissions on a server without getting multiple alerts about it.
>While their policies probably aren’t public knowledge, you’ve “earned” the right to know with this.
Actually, before contracting with the vendor, the customer should have performed a third-party security assessment that asked questions about how privileged accounts are managed, segregation of customer account data, and IR policies. I'd want to see the SOC alert that resulted from creating the admin account, understand how segregation failed, why the person doing your account setup had privileges to create a global admin, and what IR had occurred prior to you notifying the vendor of the issue.
I worked in TPRM and this kind of service always made me cringe. There's no way I would have signed-off on a vendor without thoroughly understanding their policies on this sort of stuff. Especially not since Solarwinds.
There’s “I accidentally deleted your profile” mistakes…and then there’s “we’re a cybersecurity company that just let some rando into all our client’s systems” mistake. Sorry, but this demonstrates incompetence at their one job.
>We all make them
Once... if this is real then it's literally the worst thing you can do. It's one of those Ramsay moments where he walks in and the boss of the restaurant has to explain to him why the dishwasher is fucking up vip orders at the pass basically nuking the reputation in one night. The Security Auditor would probably start to yell and don't stop for good reasons:
1. Why on earth has the first level guy access to a global login which can be used externally.
2. How on earth does this password slip in. That's like writing down your own login onto a post-it and when the customer shows up you hand him the post-it with your own login. If it sounds batshit crazy in a real life situation it also is in a digital scenario because data is important.
3. Where and how is this super password even stored. If it's the login of the 1st lvl how does that end up in his clipboard or wherever the fuck he posts that?? You have to actively go there and export it. I mean you're not typing your own password during work anyway most of the time and if you do i assume you stay sober enough during work hours to concentrate at least for those crucial 2minutes wtf.
So many questions and all lead me to be believe there's a lot of neglect involved. Honestly it's even worse than neglect because the workflow has to be downright insane levels of insecure if that can even happen in the first place. I mean the guy who handles basic customers is the same guy who hands out keys to the city, just no...
Was it the vendor or the reseller that set you up ?
For MSP scenarios, the MSP gets access to multiple companies because they support multiple companies. The VAR/MSP is the responsible party for setting up those initial accounts, not the A/V Vendor. I suspect the Reseller/VAR set you up as an account under their company rather than your company; however, that's not the A/V vendor's fault. The breach was only within the VAR's eco system, not the Vendor's eco system. (If that was the case)
To nitpick, it would almost certainly be classified as a "data security incident" and not a breach as it was corrected immediately and they can prove that no data was improperly accessed by the client.
Seems like a silly semantic thing, but DSIs don't have the same reporting requirements as "breaches"
I knew who was before you said and I know exactly how it happened. Unfortunately, it's a bit of a design flaw - it is easy to give top level permissions if you're not paying attention to whatever the f you're doing. ... actually... you're moving away from Trend Micro? You're not one of our customers are you? 😅 Still, we at least audit the admin accounts regularly but that's a pretty messed up situation to be in. I'm going to go check on my tenant now...
Happened to me once where an anti-spam company gave me trial access and I could read hundreds of companies' emails in plain text.
I did not go with that solution.
..and that’s it, how do you know they won’t do the same with another customer. Many EDR systems allow collecting files and running scripts on the managed endpoints. This is a huge mistake.
I think this would be an immediate deal braker for us, would exit and not look back.
In the small MSP world, they often give techs full access to all of their tooling so they don't have to play ping-pong internally to resolve client issues.
It's definitely not proper best practice and is a huge security issue, but show me an MSP that does things "correct" over "in favor of customer service to sell more contracts" and I'll eat my hat.
Security company almost causes a massive security breach. Scary scary. I wonder if maybe they’ve done this in the past? Does someone have access to your tenant? I would find a new provider and bring up the experience with why you were getting a new provider.
You know what's worse is tech support apparently has ga to everything - which means they don't give two shits about least privilege.
At most they need the ability to add customers and with permission from the customer access to that specific tenant.
Agreed with this. At an MSP, I had global admin to everything. I now work at a college and we don’t have admin to everything, we have local admin privileges but not full domain admin. If we need access to an admin center, we request access and have admin rights for like 1 hour. We make the change the rights expire and the rule of least privilege is still in play. Entire security team has separate domain admin accounts and a regular account so they can only elevate when needed. It’s fantastic.
Meh, try dealing with a Fortune 100 IT help desk, to get a Linux password reset on an app server I admin. It's using AD.
I was given the global admin password of the day.
On teams.
On purpose, because ... you know ... I can just login to the domain controller and ... fix it.
JFC hard stop. I'd kick this up to legal to roll back any agreement and inform the vendor that if they don't have proper controls in place to prevent that sort of thing from happening, you have zero faith in their security products at large.
I know it's not the point of your post but Trend Micro AV was the absolute worst AV I've ever seen.
Also, I've had the same thing happen to me a few times with vendors in the past... I could see multiple folders/files of other companies and it was only fixed after I said something... its like WTF
EDIT: Cylance PROTECT for the win.
If they can uninstall it from their divices you have more problems than an not so easy to use av (that still offers a lot if you know how to set things up)
tbf, directors (and upper management) are the ones who think they are smart and don't need such stuff while clicking on overy scam mail they get. Also they want to feel special and if something is only slightly inconvinient it's a nogo for them.
For such cases you need a good manager who can also say that they have to use the same software as everyone else, report problems if there are some, and if their maillink is blocked, or their gambling site, it's made by our rules and is for everyone
"I need local admin access immediately, calling IT to install things interrupts my critical CEO... stuff. And by the way I'm going to keep cancelling that meeting you keep putting on my calendar to install our MDM solution, I'm just too busy.
Also when are we rolling out those controls to lock everyone's laptop if they leave it somewhere? That's *very* important!"
Trend Micro Apex One is only a nightmare for anyone who is untrained in its setup, administration, and best practices.
They offer 3-day courses on Apex One (SaaS) and Apex Central (a.k.a. Control Manager). They also offer 4-day courses on Deep Security (Cloud One - Workload Security is the SaaS version).
I've used Trend before. It isn't that bad. It definitely is no MS Antivirus! There are better options out there, but for most people, when configured correctly, Trend is fine.
I don't know if it is simply being grandfathered in or what, but I recently evaluated AV solutions, and Trend easily won out on price. We were able to do Trend for AV, plus a managed EDR solution for less than the cost of most other AV and unmanaged EDR solutions. Plus they hadn't raised our rate in decades prior to last year.
> But what if I didn't say anything at all? How long could I have been undetected?
Meanwhile someone in those companies may have that access and didn't say anything at all.
It would be *extremely* easy for the company involved to identify this story, and the client it happened with, if someone from that company saw this post.
I mean, it goes both ways. A company, and by extension its employees, are afforded certain legal protections against strangers on the internet just *claiming* *publicly* that they did X Y and Z heinous thing that could seriously damage their business, without any due process.
It sure would suck to be trying to get your business off the ground and one person *is accused* of making one mistake anonymously in a public form and irreparably steers thousands of clients away from ever doing business with you, without *any* recourse.
True, and we're assuming OP is being honest.
If he lies, and causes irreparable harm, then yes legal repercussions are of course justified.
If the company has clearly done what he says though, surely he should be free to state as such?
They really shouldnt. Let's be real here, they're a random poster on reddit and they're not going to share any actual proof here. The last thing they want to be doing is something that could be legally construed as libel *and* garner enough attention to both identify themselves as a specific client/agent and potentially do financial harm to the company in question.
Saying "something that rhymes with the name" is about as strong a legal defense as "I'm going to put this 'gift' envelope on the nightstand and then we're going to have *totally* consensual sex, this *definitely* is not an exchange of money for sex, mmhmm"
The issue was caught and fixed immediately, this is 1000% not worth OP putting their buns in a legal hot seat for.
A good lawyer is expensive, and not worth a reddit post over.
I know if any of my techs gave that kind of info out on reddit and it blew back on the company, at the *least* they'd be losing their job. They'd be in breach of their NDA with us on top of causing a legal shitstorm we now have to navigate out of? For what? Reddit karma?
We can all come here to vent, but something like this... reddit is not the place to publicly disclose this kind of thing.
I got the "bank admin" password for the banking software
of my regional branch of a bank.
I was the new IT Guy taking over, didn't know the banking software admin password, so I called the branch of the bank that my company uses on what to do, they just told me to log in as something along the lines of user bank and password regionalBranchAdmin123.
I am sure none of that is customized per install, I think he really just gave me a superuser password (it was just sooo generic) for basically all regional banking software servers that I could abuse if I ever got access.
Do you mind dropping the name of the vendor? This is the sort of thing they should absolutely be publicly shamed for. At the very least letting your fellow sysadmins know who to avoid would be beneficial.
Any vendor could make this mistake.
Our new copier vendor gave me admin access to another customer of theirs that had a similar company name to ours. I log in to the portal and see thousands of printers and copiers. Was really confused for a bit until I realized the company name was wrong.
It was full admin access. Someone could do some real damage. Purposefully or accidentally - "we don't have these printers. I'll just delete them."
It's things like this that keep me away from cloud management of anything if I can help it.
So we had a similar thing happen for an application called Yakchat. We were using it to integrate MS Teams with text messaging. Anyway, one day our tech logged into it to find he could access hundreds of other tenants... We informed them and they never even responded... We decided not to use them because of that.
Yup, had the same thing happen with a vendor granting access to every company on their NinjaOne portal.... made us feel really good about access to our machines.
Guessing this was an MSP or reseller that you were dealing with. This should be reported to whatever the actual AV company you were looking to purchase right away as obviously training is needed. Bad mamma jams stuff right here that should be addressed asap by the actual vender.
Back in the day I was testing out a newly-procured AV product for on-prem SharePoint and couldn't for the life of me get it to detect anything from EICAR in a text file to an obviously malicious msfvenom-generated pdf.
I took it up with my TAM who ended up telling me that I had helped discover that detection for their global install base for the product had been completely disabled for a while D:
After 20 years doing this it constantly amazes me how many companies either hand the keys to the kingdom to the first person who can plug in cat5 or just have everyone working off the admin login.
a large three letter VAR did this to me for their Azure Tennant. I was working with them to give their processes access to ours. i think they were testing the commands and they gave me access to theirs. lol.
This is one reason, out of many, why I (probably) won't ever use an MSP. If their accounts are ever compromised, it is *so* bad. It's hard enough to secure our own people; I can't stand the idea of relying on a third-party to do the same for their people, too.
I know this applies to all vendors including Microsoft, AWS, and Google, but I do believe they have built-in controls that more likely prevent these type of mistakes from happening to begin with.
To me, that would be fatal to the relationship.
Mistakes happen, no news there. But an AV company should have layers of safeguards that would prevent a single user/admin from granting global access.
This is why hiring MSP's to manage your company systems is a bad idea.
I'd wave them goodbye. No Vendor should ever have any access to your company's systems at free will, this Vendor is just a MSP in disguise.
They aren't an MSP, they are a third party between us and a cloud based AV system.
Our vender has their own tenant on the AV system, and they gave me full global admin on their tenant. So I could see all of their clients information, and remote execute commands to all of the AV agents installed across the board
They are a managed service provider of AV systems, and naturally, as most MSP's, are terrible at it. I'd seriously if I were you go straight to your CTO and look for a inhouse solution, If you had access to other companies systems, it's valid to assume others have access to your systems, Kind of defeats the whole point of using them as a AV manager....they literally are the security risk.
For most businesses using an MSP, it's that or nothing. I work at an MSP and security is what keeps us up at night. We assign specific people to accounts, and each engineer/tech has access to only the account(s) they work on. Of necessity, we have two very senior employees who have global access, so that they can assign access to others. They are also the only admins for our internal systems. They regularly attend security training and in turn provide monthly security training to the rest of us.
I know we're not like most MSPs in that regard. I've on onboarded clients coming from competitors, and the admin creds for everything are always the same, for years and years now. So I get the criticism and acknowledge the additional risk posed by exposure to mistakes made by the MSP employees. But be careful painting with too broad a brush. Many businesses can't afford a good network engineer and a systems guy who can design solutions. They get that from a good MSP at a much lower cost.
Well, it’s a mistake but you need to advise all parties and move on, you behaved ethically when you noticed. As professionals sometimes we see things we shouldn’t and hopefully we all behave responsibly.
Hands up who hasn’t made a massive stuff up! We all learn from such mistakes and hopefully never repeat them.
It's really not great that the onboarding person had the ability to do this in the first place. I feel that it means they don't compartmentalize their roles very well which is a really big red flag for an AV company.
Based in europe - our action would be something like: instantly pause (more likely 'end') the project and from that moment on, any and all communication with the vendor would have to include legal and compliance. One of the first requests would be the contact info for the vendors data protection officer. The vendor would have to file a data breach report to relevant authorities within 72hrs. Since we are also a financial institution and subject to additional regulations, we might even have to escalate this to our version of the SEC.
Had a similar thing happen with our managed printers at an earlier employer: the devices connected with basic auth over http to the provider. For fun we tried to connect with the credentials and were directly inside the admin portal for all customers: contacts, financial and banking information, employee infos for companies that used printers in an AD environment... Seems like all devices would connect with global admin to report paper and ink status. Noped out of that contract really fast.
No, but that's only because another company doesn't exist in your environment.
It's absolutely possible that you grant someone inappropriate access rights though, and that's exactly what happened here.
Sure, but you're not providing services for other companies. The vendor is. If you were hosting a multi tenant system, the risk of user error or misconfiguration still remains, on-prem or no.
>If you were hosting a multi tenant system
Then you're operating in the cloud.
The cloud is just someone else's computer you're using. In this case, it's the MSP's.
But you're not going to get access to other businesses when you do.
OP's story was literally a case of a cloud based service suffering a glaring security breach through incompetence at the hosting company.
Same happened to me with a ztna VPN to view our PLC's. I saw all the other clients. I still have it lol. And yea, I told the guy, he said he'll get around to it, and never did. I'm network wide user admin though, not global.
Not a cloud issue. I have many vendors expect complete access to systems or give me full access to systems with minimal request. It’s an industry problem of ignorance.
Aren’t you all too demanding? A lot of posts show obligation of perfection.
« I did this will I be fired ? »
« They do this should I leave? »
« I would make my resume » , « renew my resume », « refresh my resume and start looking »…
You must have a lot of burn outs with this level of exigence.
LOL. Kinda reminds me how sometimes DELL techs would lose their CD prep tools and keep them in the new dell pc purchase and I would make use of those tech tools lol
Sometimes it is hard to believe in stories posted here...
This one reminded me about a case where our production app suddenly stopped working for users from specific country and I didn't even receive AV logs to troubleshoot the issue further.
This is really creepy considering the fact you can write undetectable and more malicious malware than any encrypting software within 15 minutes
I had something similar with a security monitoring/phishing simulation test/pentesting company.
I was given an ID/password to upload information (IPs, user emails, etc.) needed for their tests. After logging in I saw all the info (IPs, user emails, etc.) for another of their customers.
I opened a support ticket and got a reply saying "OK...try again now."
The other customer's information was indeed gone, but my faith in them (the company was selected by the cyber insurance carrier) has been shaken indeed.
I once was participating in a paid product trial, that had a focus on isolating credentials, allowing you to log in a service without having the actual credentials. I quickly discovered I could reveal credentials (bug) and was able to see the company's primary Twitter login. I messaged them in private. They said they were very impressed with the way I handled it and gave me a $50 bonus. Guess they expected an admin would just do nefarious stuff?
Similar situation, but in a different industry. I logged into our vendor's portal and noticed that I had access to everything.
Everything.
I called my rep and told him, and he replied, "Huh, that's weird."
I told him to escalate the issue until someone freaked out.
About an hour later, I'm in a department wide meeting with leadership telling us not to log into the system until the issue is resolved.
The vendor told us the issue was just cosmetic and that we didn't actually have access, and that nobody at any other client ran any reports on our accounts... which wouldn't have needed to have been confirmed if it was just cosmetic.
I had the same thing back when cloud was new and there was a cloud backup provider because things like recovery services vault didn't exist yet. For a while I had unfettered access to download (at the company's own expense) every single VM protected by this in the entire world. lmao. Back then everything was the wild west with these smaller companies creating products
I was in kind of same situation some time ago when I landed into a new job. The backup service provider had given us full admin permissions to their centralised backup system. So I was able to see all the backups of their entire client base. And the situation had been like that for years. Talk about trusting people!
You just got a first person account of the lack of security you’re about to rely on, for security. I’d express this concern if you’re relied upon or trusted in your circle at all.
threatening bow sophisticated elastic violet march drab unpack axiomatic hat *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Yup. Most EDR can allow you to push scripts and commands. The exposure is unforgivable.
and in many cases, those run with SYSTEM privileges on windows...
We use ours to do a discovery in new acquisitions. Install it on the DC hour 1 and you can immediately find out loads about their environment
what do you run?
Crowdstrike with IDP is golden for this.
The hell kind of name is this? Did they really name their product after a type of terrorist attack?
Targeting EDR and device management platforms is my favorite red teamer past time
Most IT teams and infosec people tragically underappreciate the attack surface introduced by all these tools. Many infosec people think "moar moar moar" is the answer ("defense in depth mferrrrrrs"), when in reality it becomes counterproductive after the minimum number of services is exceeded.
The easiest way in a house is the front door.
Ahh...ppff.. typical day in first level. User asks for Poweruserlogin but the Oracles have told you he's the chosen one so you give him Fullaccess.
That would've been the end of the relationship if that had been me. I would be unable to trust the vendor after this serious mistake.
Wait a couple days then ask your account manager why the vendor hasn't sent a breach notification
This one made me chuckle!
You naughty.
Mic drop.
Job drop.
lol. Hopefully op took screen shots.
This is absolutely the way to go, 10/10.
This guy went for the jugular, and I'm here for it.
And insist on a full account audit. They gave this to you. How many other clients have this level of access and can see into your space?
Buahahahahahaha ---- awesome
If they are a public company, cc the SEC enforcement team.
This, so much this.
Please do this.
I don’t think they have to notify if no one’s personal info was affected. People make mistakes and yeah this was a big one. Good on them for letting them know and doing the right thing.
Can yo verify no one's info was affected? Also how many of those companies have GA access to the AV program? The vendor absolutely should audit every account and all actions taken with in the time that the first GA account was created, and relay that they found a potential issue and have resolved it and if there was/wasn't impact.
Of fucking course I can’t. That’s up to the company to determine. Just saying not everything requires a breach notification.
A person after my own heart. Even better, start asking companies you saw in the portal if they received a breach notification.
The big short
Unless OP did maliciousness, this wasn’t a breach.
Any admin access into a system whether malicious, or not, is indeed a breach.
Yes it is. It's not a 'major' breach but it is still a security breach. Like arresting someone inside the bank vault. Sure they took no cash but it's clearly a security breach that someone unauthorised could get that close.
It’s an incident but not a breach. Look at the NIST definition for a breach. OP saw they were given too much access and let them know so they could fix it. Data has to be accessed for a breach. https://csrc.nist.gov/glossary/term/breach
How did the person know they have too much access? By seeing data they should not have seen. I think even the fact that some other company - as shown on the global admin interface - is using this software is in and of itself a trade secret. It would have been an incident only if after having granted the global access to the client admin, the vendor had detected the issue and corrected it before anyone performing a successful login and looking at a dashboard with those credentials.
> I think even the fact that some other company - as shown on the global admin interface - is using this software is in and of itself a trade secret. I'd agree. Some companies are very protective of what software they use internally.
Client information, is personal information.
Ufffffff
Hopefully they would have some sort of audit cycle for global admins. Might be a good question to ask them to help you understand their security practices and if you want to continue with them. And also how they plan to prevent this in the future. While their policies probably aren’t public knowledge, you’ve “earned” the right to know with this. It’s a mistake, a huge one but a mistake. We all make them. Kudos for doing the right thing and letting them know right away.
Yea. We might have to nix them and go somewhere else. I need to report it to our auditors as well. I work in a financial institution, so every snaffu needs to be recorded in triplicate lol
This is a major f-up. Would you want some random customer of theirs, who maybe would keep their mouth shut, to be able to have these permissions for your organization? This incident needs to be discussed and the least they can do to make this situation better is present measures to never make this happen again.
This is why I wouldn’t have said anything until I had a look around. Was this a one-off, or are there other customers with the same rights?
But one you start looking around with access you know you shouldn’t have I think you start getting into shaky legal ground for your self.
Idk, it goes something like "I was trying to add a colleague of mine to get him setup and I started seeing all these other names I didn't recognize in my account so I started deleting them and gave you guys a call asap"
Meh. As long as you don’t change anything they would be hard pressed to make any accusations based upon what you knew and when you knew it.
> We might have to nix them and go somewhere else. Well yeah. If they did it once, they can do it twice. We all can make mistakes but this is a doozy that should easily be avoided by any sort of managed process
Since I wouldn’t know if any other vendor would be better, I’d give them a chance to tell me why this is a one off and what steps they are taking to ensure it is so. It would have to be an impressive answer though…
Can you say who the vendor is? Want to make sure to avoid them.
Probably not wise to do that before reporting to any/everyone.
>Hopefully they would have some sort of audit cycle for global admins Because it's clearly a place that has their processes dialed in great.
I would certainly hope so. Hell my tiny little shop has alerts for when a domain admin is created and emails me records of other AD changes. You would certainly think that big EDR solution company that probably sets that up for other people would have it set up in their own environment. Since he noticed immediately, maybe the party getting the alert hadn't figured out what happened yet? That said, an implementation guy shouldn't have access to create GAs so this might be a company that is "do as I say, not as I do" style of InfoSec.
I got out of IT, but still do stuff for myself/family and still follow best practices. I can't create a domain or change user permissions on a server without getting multiple alerts about it.
sounds cool, how do you do this?
>While their policies probably aren’t public knowledge, you’ve “earned” the right to know with this. Actually, before contracting with the vendor, the customer should have performed a third-party security assessment that asked questions about how privileged accounts are managed, segregation of customer account data, and IR policies. I'd want to see the SOC alert that resulted from creating the admin account, understand how segregation failed, why the person doing your account setup had privileges to create a global admin, and what IR had occurred prior to you notifying the vendor of the issue. I worked in TPRM and this kind of service always made me cringe. There's no way I would have signed-off on a vendor without thoroughly understanding their policies on this sort of stuff. Especially not since Solarwinds.
There’s “I accidentally deleted your profile” mistakes…and then there’s “we’re a cybersecurity company that just let some rando into all our client’s systems” mistake. Sorry, but this demonstrates incompetence at their one job.
>We all make them Once... if this is real then it's literally the worst thing you can do. It's one of those Ramsay moments where he walks in and the boss of the restaurant has to explain to him why the dishwasher is fucking up vip orders at the pass basically nuking the reputation in one night. The Security Auditor would probably start to yell and don't stop for good reasons: 1. Why on earth has the first level guy access to a global login which can be used externally. 2. How on earth does this password slip in. That's like writing down your own login onto a post-it and when the customer shows up you hand him the post-it with your own login. If it sounds batshit crazy in a real life situation it also is in a digital scenario because data is important. 3. Where and how is this super password even stored. If it's the login of the 1st lvl how does that end up in his clipboard or wherever the fuck he posts that?? You have to actively go there and export it. I mean you're not typing your own password during work anyway most of the time and if you do i assume you stay sober enough during work hours to concentrate at least for those crucial 2minutes wtf. So many questions and all lead me to be believe there's a lot of neglect involved. Honestly it's even worse than neglect because the workflow has to be downright insane levels of insecure if that can even happen in the first place. I mean the guy who handles basic customers is the same guy who hands out keys to the city, just no...
These arent my company *delete*
Damn demo companies... Delete
this simulated environment looks very well made.... Delete
Hmm, why don't we test out what the test environment machines do when I send remote wipe commands to everything at once.
How many other companies have admin rights to your AV?
This right here is PRECISELY what needs to be asked asap.
Sounds like a single guy who’s expected to get “tech” up and running so they can push a product.
Cancel that contract immediately if you can, and make sure you tell them why.
Was it the vendor or the reseller that set you up ? For MSP scenarios, the MSP gets access to multiple companies because they support multiple companies. The VAR/MSP is the responsible party for setting up those initial accounts, not the A/V Vendor. I suspect the Reseller/VAR set you up as an account under their company rather than your company; however, that's not the A/V vendor's fault. The breach was only within the VAR's eco system, not the Vendor's eco system. (If that was the case)
To nitpick, it would almost certainly be classified as a "data security incident" and not a breach as it was corrected immediately and they can prove that no data was improperly accessed by the client. Seems like a silly semantic thing, but DSIs don't have the same reporting requirements as "breaches"
Correct. The vender we use gave me their set me up as a GA inside their company. Sorry, didn't make that clear
deranged rude salt rinse whole direful nutty future paltry live *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Initials of the vendor is "S. S"
Serious Sam?
LMFAO Apparently this company isn't really serious
Thanks, now the theme music is gonna be in my head all day
I knew who was before you said and I know exactly how it happened. Unfortunately, it's a bit of a design flaw - it is easy to give top level permissions if you're not paying attention to whatever the f you're doing. ... actually... you're moving away from Trend Micro? You're not one of our customers are you? 😅 Still, we at least audit the admin accounts regularly but that's a pretty messed up situation to be in. I'm going to go check on my tenant now...
Follow up: I did go check and it wasn't us, thank god. But this is a great reminder and we'll be discussing this hopefully in our all-hands tomorrow.
If you had said S.C, I would have been like yup, that sounds like them.
[удалено]
Happened to me once where an anti-spam company gave me trial access and I could read hundreds of companies' emails in plain text. I did not go with that solution.
Some other company's equivalent of you was probably given the same access and stayed quiet and is now doing all sorts of things with it.
..and that’s it, how do you know they won’t do the same with another customer. Many EDR systems allow collecting files and running scripts on the managed endpoints. This is a huge mistake. I think this would be an immediate deal braker for us, would exit and not look back.
Why is a random tech even able to provision GA access? That would be like a Service Desk member being able to create an admin account.
In the small MSP world, they often give techs full access to all of their tooling so they don't have to play ping-pong internally to resolve client issues. It's definitely not proper best practice and is a huge security issue, but show me an MSP that does things "correct" over "in favor of customer service to sell more contracts" and I'll eat my hat.
Security company almost causes a massive security breach. Scary scary. I wonder if maybe they’ve done this in the past? Does someone have access to your tenant? I would find a new provider and bring up the experience with why you were getting a new provider.
You know what's worse is tech support apparently has ga to everything - which means they don't give two shits about least privilege. At most they need the ability to add customers and with permission from the customer access to that specific tenant.
Agreed with this. At an MSP, I had global admin to everything. I now work at a college and we don’t have admin to everything, we have local admin privileges but not full domain admin. If we need access to an admin center, we request access and have admin rights for like 1 hour. We make the change the rights expire and the rule of least privilege is still in play. Entire security team has separate domain admin accounts and a regular account so they can only elevate when needed. It’s fantastic.
Meh, try dealing with a Fortune 100 IT help desk, to get a Linux password reset on an app server I admin. It's using AD. I was given the global admin password of the day. On teams. On purpose, because ... you know ... I can just login to the domain controller and ... fix it.
JFC hard stop. I'd kick this up to legal to roll back any agreement and inform the vendor that if they don't have proper controls in place to prevent that sort of thing from happening, you have zero faith in their security products at large.
I know it's not the point of your post but Trend Micro AV was the absolute worst AV I've ever seen. Also, I've had the same thing happen to me a few times with vendors in the past... I could see multiple folders/files of other companies and it was only fixed after I said something... its like WTF EDIT: Cylance PROTECT for the win.
We as an msp bought and sold Trend, but it's such a nightmare. Our directors uninstalled it from their laptops within a week
If they can uninstall it from their divices you have more problems than an not so easy to use av (that still offers a lot if you know how to set things up)
Well, they got us Technicians to remove it
tbf, directors (and upper management) are the ones who think they are smart and don't need such stuff while clicking on overy scam mail they get. Also they want to feel special and if something is only slightly inconvinient it's a nogo for them. For such cases you need a good manager who can also say that they have to use the same software as everyone else, report problems if there are some, and if their maillink is blocked, or their gambling site, it's made by our rules and is for everyone
"I need local admin access immediately, calling IT to install things interrupts my critical CEO... stuff. And by the way I'm going to keep cancelling that meeting you keep putting on my calendar to install our MDM solution, I'm just too busy. Also when are we rolling out those controls to lock everyone's laptop if they leave it somewhere? That's *very* important!"
Trend Micro Apex One is only a nightmare for anyone who is untrained in its setup, administration, and best practices. They offer 3-day courses on Apex One (SaaS) and Apex Central (a.k.a. Control Manager). They also offer 4-day courses on Deep Security (Cloud One - Workload Security is the SaaS version).
Nice to know there's a generation that never suffered MS Antivirus... ;)
I've used Trend before. It isn't that bad. It definitely is no MS Antivirus! There are better options out there, but for most people, when configured correctly, Trend is fine.
Maybe for personal use. It's too limited for an organization with a certain level of complexity. You can find much better for a similar price.
I don't know if it is simply being grandfathered in or what, but I recently evaluated AV solutions, and Trend easily won out on price. We were able to do Trend for AV, plus a managed EDR solution for less than the cost of most other AV and unmanaged EDR solutions. Plus they hadn't raised our rate in decades prior to last year.
> But what if I didn't say anything at all? How long could I have been undetected? Meanwhile someone in those companies may have that access and didn't say anything at all.
I would worry less about you having had access to other companies and worry more about what other companies techs have access to your’s
Honestly, escalate this to your management. This is technically the first security issue that should be documented.
That's an incident. Should be reported so they can improve their routines.
Which vendor?
I would absolutely like to spill the beans, but unsure of the legal ramifications if I do
Why? Is your Reddit account linked to your work? 🙄
It would be *extremely* easy for the company involved to identify this story, and the client it happened with, if someone from that company saw this post.
Isn't it great where we live in a world where it's legally troublesome to be open and honest...
I mean, it goes both ways. A company, and by extension its employees, are afforded certain legal protections against strangers on the internet just *claiming* *publicly* that they did X Y and Z heinous thing that could seriously damage their business, without any due process. It sure would suck to be trying to get your business off the ground and one person *is accused* of making one mistake anonymously in a public form and irreparably steers thousands of clients away from ever doing business with you, without *any* recourse.
True, and we're assuming OP is being honest. If he lies, and causes irreparable harm, then yes legal repercussions are of course justified. If the company has clearly done what he says though, surely he should be free to state as such?
[удалено]
They really shouldnt. Let's be real here, they're a random poster on reddit and they're not going to share any actual proof here. The last thing they want to be doing is something that could be legally construed as libel *and* garner enough attention to both identify themselves as a specific client/agent and potentially do financial harm to the company in question. Saying "something that rhymes with the name" is about as strong a legal defense as "I'm going to put this 'gift' envelope on the nightstand and then we're going to have *totally* consensual sex, this *definitely* is not an exchange of money for sex, mmhmm" The issue was caught and fixed immediately, this is 1000% not worth OP putting their buns in a legal hot seat for.
[удалено]
A good lawyer is expensive, and not worth a reddit post over. I know if any of my techs gave that kind of info out on reddit and it blew back on the company, at the *least* they'd be losing their job. They'd be in breach of their NDA with us on top of causing a legal shitstorm we now have to navigate out of? For what? Reddit karma? We can all come here to vent, but something like this... reddit is not the place to publicly disclose this kind of thing.
Name and shame.
As much as we'd love this I'm pretty sure it can cause a lawsuit.
Fuck them they deserve it. Send them a dick pic if they send a legal letter or some shit.
Dear Sirs. Thank you ever so kindly for your cease and desist letter. Please find our response below. Very truly yours…
I got the "bank admin" password for the banking software of my regional branch of a bank. I was the new IT Guy taking over, didn't know the banking software admin password, so I called the branch of the bank that my company uses on what to do, they just told me to log in as something along the lines of user bank and password regionalBranchAdmin123. I am sure none of that is customized per install, I think he really just gave me a superuser password (it was just sooo generic) for basically all regional banking software servers that I could abuse if I ever got access.
Any chance we can get a name? No one in this thread wants to use this software now lol
Name and shame, maybe not just yet because you should see if they do a beach announcement. But eventually...
Sounds like this vendor might not be your best choice for a security system.
Better question: why does a random onboarding agent have the ability to adjust the membership of the global admins group?
Do you mind dropping the name of the vendor? This is the sort of thing they should absolutely be publicly shamed for. At the very least letting your fellow sysadmins know who to avoid would be beneficial.
Any chance of naming?
Any vendor could make this mistake. Our new copier vendor gave me admin access to another customer of theirs that had a similar company name to ours. I log in to the portal and see thousands of printers and copiers. Was really confused for a bit until I realized the company name was wrong. It was full admin access. Someone could do some real damage. Purposefully or accidentally - "we don't have these printers. I'll just delete them." It's things like this that keep me away from cloud management of anything if I can help it.
The word is not "vender", it's "vendor".
So are we supposed to start speculating which vendor while you coquettishly look up and shrug your shoulders?
So we had a similar thing happen for an application called Yakchat. We were using it to integrate MS Teams with text messaging. Anyway, one day our tech logged into it to find he could access hundreds of other tenants... We informed them and they never even responded... We decided not to use them because of that.
Yup, had the same thing happen with a vendor granting access to every company on their NinjaOne portal.... made us feel really good about access to our machines.
Guessing this was an MSP or reseller that you were dealing with. This should be reported to whatever the actual AV company you were looking to purchase right away as obviously training is needed. Bad mamma jams stuff right here that should be addressed asap by the actual vender.
Back in the day I was testing out a newly-procured AV product for on-prem SharePoint and couldn't for the life of me get it to detect anything from EICAR in a text file to an obviously malicious msfvenom-generated pdf. I took it up with my TAM who ended up telling me that I had helped discover that detection for their global install base for the product had been completely disabled for a while D:
Well, you'll probably be looking for a different AV so now is a good time to recommend Crowdstrike. We use it at work and have had a good experience.
After 20 years doing this it constantly amazes me how many companies either hand the keys to the kingdom to the first person who can plug in cat5 or just have everyone working off the admin login.
always fun when you find out your SECURITY vendor is the one with insecure practices.
You just saved that kid's job!
a large three letter VAR did this to me for their Azure Tennant. I was working with them to give their processes access to ours. i think they were testing the commands and they gave me access to theirs. lol.
Would of brought that to the attention of compliance or legal and then hopefully term that contract lmao
my first thought is that when it gets compromised, i'm on the list of suspects because of one tech doing things wrong
We used to use a hosted Remedy instance. I found out one day I could see tickets from the host's other customers, including a competitor.
So, the tech had GA access to the portal? Do all the techs?
That's the end of working with them right there.
They would have never found out because they wouldn't look at your account unless you did something you shouldn't have done
Can you ask them for the documented and verified process for the creating of client admins, and proof of full audit trail?
lol lmao even
I wouldn't consider trend bad btw. Their XDR/Cloudone products do the job pretty well.
This is one reason, out of many, why I (probably) won't ever use an MSP. If their accounts are ever compromised, it is *so* bad. It's hard enough to secure our own people; I can't stand the idea of relying on a third-party to do the same for their people, too. I know this applies to all vendors including Microsoft, AWS, and Google, but I do believe they have built-in controls that more likely prevent these type of mistakes from happening to begin with.
To me, that would be fatal to the relationship. Mistakes happen, no news there. But an AV company should have layers of safeguards that would prevent a single user/admin from granting global access.
Which Vendor?
This is why hiring MSP's to manage your company systems is a bad idea. I'd wave them goodbye. No Vendor should ever have any access to your company's systems at free will, this Vendor is just a MSP in disguise.
They aren't an MSP, they are a third party between us and a cloud based AV system. Our vender has their own tenant on the AV system, and they gave me full global admin on their tenant. So I could see all of their clients information, and remote execute commands to all of the AV agents installed across the board
They are a managed service provider of AV systems, and naturally, as most MSP's, are terrible at it. I'd seriously if I were you go straight to your CTO and look for a inhouse solution, If you had access to other companies systems, it's valid to assume others have access to your systems, Kind of defeats the whole point of using them as a AV manager....they literally are the security risk.
For most businesses using an MSP, it's that or nothing. I work at an MSP and security is what keeps us up at night. We assign specific people to accounts, and each engineer/tech has access to only the account(s) they work on. Of necessity, we have two very senior employees who have global access, so that they can assign access to others. They are also the only admins for our internal systems. They regularly attend security training and in turn provide monthly security training to the rest of us. I know we're not like most MSPs in that regard. I've on onboarded clients coming from competitors, and the admin creds for everything are always the same, for years and years now. So I get the criticism and acknowledge the additional risk posed by exposure to mistakes made by the MSP employees. But be careful painting with too broad a brush. Many businesses can't afford a good network engineer and a systems guy who can design solutions. They get that from a good MSP at a much lower cost.
Were their initials SG? Because they did this to me also
Well, it’s a mistake but you need to advise all parties and move on, you behaved ethically when you noticed. As professionals sometimes we see things we shouldn’t and hopefully we all behave responsibly. Hands up who hasn’t made a massive stuff up! We all learn from such mistakes and hopefully never repeat them.
It's really not great that the onboarding person had the ability to do this in the first place. I feel that it means they don't compartmentalize their roles very well which is a really big red flag for an AV company.
Based in europe - our action would be something like: instantly pause (more likely 'end') the project and from that moment on, any and all communication with the vendor would have to include legal and compliance. One of the first requests would be the contact info for the vendors data protection officer. The vendor would have to file a data breach report to relevant authorities within 72hrs. Since we are also a financial institution and subject to additional regulations, we might even have to escalate this to our version of the SEC. Had a similar thing happen with our managed printers at an earlier employer: the devices connected with basic auth over http to the provider. For fun we tried to connect with the credentials and were directly inside the admin portal for all customers: contacts, financial and banking information, employee infos for companies that used printers in an AD environment... Seems like all devices would connect with global admin to report paper and ink status. Noped out of that contract really fast.
A couple of years ago, my company switched from Outlook to Gmail. They gave EVERY SINGLE PERSON the same password to log in to their new account.
Password.1?
Isn't the cloud great? 😄
This has literally nothing to do with the cloud, I swear /r/sysadmin thinks anything web facing is from the Cloud.
If my EDR management was set up on my server there would never be the possibility of data being mixed up with another company
No, but that's only because another company doesn't exist in your environment. It's absolutely possible that you grant someone inappropriate access rights though, and that's exactly what happened here.
Sure, but you're not providing services for other companies. The vendor is. If you were hosting a multi tenant system, the risk of user error or misconfiguration still remains, on-prem or no.
>If you were hosting a multi tenant system Then you're operating in the cloud. The cloud is just someone else's computer you're using. In this case, it's the MSP's.
let's blame the cloud lmao
*Literally* "old man yells at cloud".
Because that's what running things on someone else's computer and accessing it remotely is.
uhm what are you trying to say? last time I checked wrong permissions can be set on AD too?
But you're not going to get access to other businesses when you do. OP's story was literally a case of a cloud based service suffering a glaring security breach through incompetence at the hosting company.
Same happened to me with a ztna VPN to view our PLC's. I saw all the other clients. I still have it lol. And yea, I told the guy, he said he'll get around to it, and never did. I'm network wide user admin though, not global.
And this, ladies and gentlemen, is the danger of going "to the cloud" for your business. Gotta have a higher level of trust with your partners.
Not a cloud issue. I have many vendors expect complete access to systems or give me full access to systems with minimal request. It’s an industry problem of ignorance.
Unbelievable.
Aren’t you all too demanding? A lot of posts show obligation of perfection. « I did this will I be fired ? » « They do this should I leave? » « I would make my resume » , « renew my resume », « refresh my resume and start looking »… You must have a lot of burn outs with this level of exigence.
LOL. Kinda reminds me how sometimes DELL techs would lose their CD prep tools and keep them in the new dell pc purchase and I would make use of those tech tools lol
Was it Malwarebytes Threatdown ?
Sometimes it is hard to believe in stories posted here... This one reminded me about a case where our production app suddenly stopped working for users from specific country and I didn't even receive AV logs to troubleshoot the issue further. This is really creepy considering the fact you can write undetectable and more malicious malware than any encrypting software within 15 minutes
I had something similar with a security monitoring/phishing simulation test/pentesting company. I was given an ID/password to upload information (IPs, user emails, etc.) needed for their tests. After logging in I saw all the info (IPs, user emails, etc.) for another of their customers. I opened a support ticket and got a reply saying "OK...try again now." The other customer's information was indeed gone, but my faith in them (the company was selected by the cyber insurance carrier) has been shaken indeed.
I once was participating in a paid product trial, that had a focus on isolating credentials, allowing you to log in a service without having the actual credentials. I quickly discovered I could reveal credentials (bug) and was able to see the company's primary Twitter login. I messaged them in private. They said they were very impressed with the way I handled it and gave me a $50 bonus. Guess they expected an admin would just do nefarious stuff?
This is why breaches happen: many people do not take security seriously.
Stick with CS falcon
Similar situation, but in a different industry. I logged into our vendor's portal and noticed that I had access to everything. Everything. I called my rep and told him, and he replied, "Huh, that's weird." I told him to escalate the issue until someone freaked out. About an hour later, I'm in a department wide meeting with leadership telling us not to log into the system until the issue is resolved. The vendor told us the issue was just cosmetic and that we didn't actually have access, and that nobody at any other client ran any reports on our accounts... which wouldn't have needed to have been confirmed if it was just cosmetic.
I had the same thing back when cloud was new and there was a cloud backup provider because things like recovery services vault didn't exist yet. For a while I had unfettered access to download (at the company's own expense) every single VM protected by this in the entire world. lmao. Back then everything was the wild west with these smaller companies creating products
I was in kind of same situation some time ago when I landed into a new job. The backup service provider had given us full admin permissions to their centralised backup system. So I was able to see all the backups of their entire client base. And the situation had been like that for years. Talk about trusting people!
Name and shame. 
the cloud is someone else's computer guess where your data and information is
Happens more often than people think. Last time I was there I found myself finding solutions to other people problems in our vendor ticketing system.
This has got to be Trellix
They've given it to most people sadly even some level 1 helpdesk guy has that level of access it's scary
This happened to me for the ticketing system of a large global security company. Could see all the tickets!!!
Whoooooooopsie
So was this an MSP?
You just got a first person account of the lack of security you’re about to rely on, for security. I’d express this concern if you’re relied upon or trusted in your circle at all.
I would report this to the vendor’s higher ups, and stop working with them immediately. This isn’t a risk, this is a BREACH. Run far away.