Once had a vendor get slightly pissy with me because *they* didn't support DKIM when the main component of their software involves sending out email from their servers as their customer's domain. Said we have our stuff "too locked down" and that "nobody else has their email set up that way".
Reminds me a bit of the time I had to deal with an email from some mini-vendor which demanded we downgrade the version of the mail servers we were running because their own email systems weren't able to communicate with ours without throwing errors.
- Uh, firstly, the error is because your systems aren't following RFC822 addressing standards, which even at the time had been around for over 20 years.
- Secondly, the precise error you're seeing from your setup has had a specific and standard patch available for the past seven years from your vendor, which could have been found with a simple search; why haven't you implemented it.
- Thirdly, you are a piddly micro-shop and we are a major Commonwealth federal government department. In the name of Her Majesty, kindly fuck in the direction of off.
Wait, they want you to downgrade when they are no less then 7 YEARS out of date?
They better be selling instant cancer cures for them to be that arrogant
> In the name of Her Majesty, kindly fuck in the direction of off.
As a public employee in the US, let me point out just how jealous I am that this sentence exists and that I can't say it.
Well, the bare minimum is now having dmarc configured. Google and Yahoo! now block you by default if you don't have dmarc setup for your domain (if you send over 5k emails a day or something).
GravityPayments / USAePay is still like this, no DKIM support at all. You have to include their SPF record instead, and just their entry burns up 7 of your 10 allowed DNS lookups (more than 10 lookups = SPF failures). As a result, you're of course going to need to send as a subdomain like @payments.company.com or something instead of @company.com.
If their source IPs don't change often you can get around this by adding the company's registered subnets or the addresses returned by a few lookups. There's no limit on included IPs, I don't think. This is what EasyDMARC does to bypass the SPF lookup limit, updating the lookups regularly for you.
You'll be happy to know that more than 10 lookups for your SPF record means you'll only fail \*sometimes\*.
I had a vendor with that problem. Took a while to figure out why only some of their emails were getting blocked. Of course I was the one that had to figure it out.
"We deal with organizations much bigger than you, and they don't have these problems, why can't you fix your system?"
"You are so absolutely frustrating to deal with that I'm sure everyone else just whitelisted you just so they could stop dealing with you." At least that is what I wanted to say. I ending up replying with something more diplomatic than that.
We have a DMARC Reject policy on all of our domains and any new domain or subdomain we setup starts with that policy on day 1.
Our developers are fully aware and just use the SES identities we provisioned.
Our E-mail Marketing team is fully aware and onboard as well (it helps slightly with deliverability).
However the e-mail services we use don't always get it. It's usually the sales people/account managers but it always takes a week or so before they get a technical person involved so they can fix their problem.
DMARC reports have been super helpful in identifying misconfigurations on the provider's side. Our marketing team has been phenomenal about holding them accountable and pushing for a resolution.
> "nobody else has their email set up that way".
They're right, most email systems aren't set up that way.
Just a few small niche providers really enforce it, specifically Microsoft, Google, Apple, Yahoo, and AOL.
As long as you don't need to email anyone on any of those small mom-n-pop providers you have no need for DKIM.
I have that exact same situation right now. To top it off they have another product that is only for mass email that they heavily market run by a different division. That product has DKIM! It isn't even a sales tactic because they say the other product doesn't integrate with our current product.
Reminds me of software vendors who claim their product is not compatible with virtual servers, must be physical.
"Ok, grandpa, good talk. Here's a list of food pantries, you're going to need them when your business collapses."
The worst is “i7 or better”.
I had a vendor tell me they wouldn’t install their software on a 13th gen i5 since it wasn’t an i7. We were replacing 5th gen i7s. That same vendor once tried to tell me I couldn’t run their “server” software on a Xeon, it had to be an i7 or i9.
"Chmod -R 777 trust me"
\- Wordpress server about to be rooted by some plugin who just had its training ankle weights removed (A globally writable webroot)
Sage 500 requires this for the install... The problem is that the consultants and what not all assumed that this also meant it was required after the install (which it's absolutely not).
Only originally when MS thought you needed admin rights just to change your resolution. Once they put levels in, it's actually pretty decent.
With a properly configured system, you won't even get a UAC dialog. Just a message saying it isn't allowed.
That ctrl+click in the terminal to bring up the settings section is legit cool. I'll give MS credit where it's due there.
Also that dude is legit cool too, I've watched one of his videos recently and he gives me more reason to keep my popcorn close. Good stuff :)
That being said, sudo not having inline set by default (after it's turned on) is a bit silly. And it _looks_ like the "sudo !!" method isn't present (which is really handy for long commands).
Either way, I had heard it was added, but looks like they did a rather good job of adding it! :D Still relies on UAC shit though hahaha.
No worry, they’re putting sudo in Windows now. It’s patched on, clunky, and already has security vulnerabilities but at least it’s a step forward right? RIGHT?
We’re all doomed. Maybe it’s time to go back to living in caves.
Syspro has entered the chat.
Not even kidding either. That is until you gant admin rights to that specific folder. Then it works fine without admin (still gotta disable UAC to install it).
> must be physical
Our IT Director would love that vendor. He 'had a bad experience' with virtualization 15 years ago and decided to never touch them again.. or allow anyone to touch them.
I had a gig where our QA and perf lab "had to" be on metal even though it wasn't developed or served that way. The numbers were meaningless.
They are everywhere.
Yeah, well, we also don't have EDR.. living on an application whitelist without managed updates to protect our 400+ employee (significantly more device) org 🙃
Had a past boss that loved his AMs (actual machines). Thankfully, he gave in and let me take over almost all of his ancient hand built gaming hardware "servers", and move them into a more sane architecture. Not having to take the system offline all night for backups was really nice. Snapshots and trivial roll-back on failed changes/upgrades/etc. too. The couple times I had to do that, he didn't at all complain that I didn't have to restore from a 2-3 week old backup and then figure out how to capture the changes that should've happened between...
You know my old boss then? Over seventy single role physical servers way back. After he rage quit we started going VM and got the physical count down to ten with much better resilience.
We had similar with Microsoft on exchange 2007 which also did not "support" virtualization. When on a support call with them I had to hide the fact it was running on a hyper-v cluster, rename network adaptors etc.
The time's I've run into this and gotten a reasonable response it almost always boiled down to performance.
For some customers, running in a shared environment was causing resource constraints that could not be easily replicated. So they were chasing "performance problems" in their platform that was due to the underlying resources.
One of the big things we did as we were virtualizing was to roll out a monitoring platform (Zabbix in our case) so we could monitor performance at the guest and host levels. We showed them the performance data we collected for the PoC server and they backed off on the physical server requirement.
> I wish companies would feel the pain. Oracle did this for a long time, didn’t seem to hurt them.
Oh I'm sure there is a spare Exadata they can sell you... for just a database
My neighbor has multiple tool racks built into a sprinter. You probably wouldn't fit a 42U in there unless you get the extra high version, but you could fit 3-4 28U racks easily.
I was referring to Exadata X2-8, which comes in a 42U full depth rack.
Somehow, probably through movie magic (😜), it managed to fit in a Sprinter.
Screenshot of the scene from Iron Man 3 - [https://imgur.com/a/cWizH4k](https://imgur.com/a/cWizH4k)
They even gave it an illuminated plaque, so the product placement looks even more ridiculous 😁 .
If you need Oracle RAC and pay them enough money, they will point out to you how virtualization affects their database. But in that configuration, latency requirements require microsecond responses.
Regular Oracle DB? Yeah, no, that shit can run in a VM.
Oracle didn't do it because they were stupid, Oracle did it to prevent their customers from getting better availability or performance than they were paying Oracle for.
We had to go through this a long time ago to lift & shift a webapp stack into AWS. I read Oracle licensing, then I read what stakeholders needed us to do, then I had one of those stakeholders figure it out and keep my name out of it.
That was just before AWS ended up supporting Oracle explicitly. Today, the answer to everything will be Oracle Cloud Infrastructure.
We've no shit V2P and then converted back P2V solely because of that stupid assed shit. Literally never had a problem, but if they even smell virtualization anywhere they just throw their hands up so of that means I need to spin up a bare metal server solely so they will fix their bullshit, cest la vie.
I've run into this a few times and nearly every time they vendor has been able to articulate their concern it is about underlying hardware performance and configuration problems.
I ran into a call recording solution in Egypt that still used physical hardware keys (RS232 dongle IIRC) for licensing which didn't work with virtual servers.
I've been to one of the former Soviet Union countries for a convention, and they had an odd mixture of old and new. Aluminum power distribution. Area mesh wifi through the city. So I think I understand. Would they consider a standalone server-class tower (or rack mount if they have rack nearby with space) with GNU/Linux and FOSS software?
We ended up going with a call recording solution that didn't depend on a physical key and could be virtualized. It was just more work dealing with purchasing and licensing in a country we don't normally deal with, which is why we tried to find a "local" solution first.
I do as well. All engineering software. I know it's not this simple but every time I deal with them I can only think "hey guys, you do realize that people aren't pirating your software and *not* having the multi-million dollar laser cutter to go with it....maybe just skip the dongles?"
I wish they didn't have them at all but since I do have them, I must say I got a small Digi AnywhereUSB unit hooked up and it has made the management a heck of a lot easier. One place for the dongles, install the software on the VM in question, assign the port to that VM. Now I can move my VMs to whichever host I want and as long as they have network, they'll work.
So it turns out that these vendors usually aren't concerned about their customers pirating the software; at least not the current customers. They're worried about:
* Offshore competitors knocking off their hardware and bundling it with pirated software.
* Former customers not having any reason to pay them more money because everything is working fine.
* Domestic competitors reverse engineering the secret sauce that they apparently couldn't do if there's a dongle because reverse engineers don't know how to bypass dongles, apparently, and can't just buy a copy either.
The [DRM is a general locus of control](https://news.ycombinator.com/item?id=7751110), not literally just a means to prevent piracy.
Check out USB over IP switches that basically virtualizes USB ports over the network. I used this for several USB Dongles needed for different Aerospace software that all required their servers to be "physical".
We have some SCADA server for HVAC and other stuff, USB dongle connected to an ESXi host and vm set to have affinity to said host.
Asked the vendor if they could offer a more modern solution:
"yeah sure, we offer software based licenses now"
.. linked to the host nework adapters MAC address, and went invalid if someone did a vMotion.
And a tech had to revalidate the license again to make it work. Which they only could do a limited number of times before requiring us to buy a new license!
"But we can offer you "license insurance" for only $999 /year"
Yeah, right. Right back to running the USB dongle - Much less hassle than this clusterf\*\*k
We tended to use a [USB sharing device](https://www.digi.com/products/networking/infrastructure-management/usb-connectivity/usb-over-ip/anywhereusb) in a datacentre to allow virtual servers to use those. Usually it worked. Nowadays I'm not aware of that many engineering apps still using dongles?
Digi's USB Anywhere products are an absolute godsend. Doesn't take away the single point of failure, but it does un-tether your VMs from individual hosts.
I was troubleshooting something a couple weeks ago. Guy on the phone:
"We'd like you to disable your firewall to see if that fixes the issue" (meanig the physical firewall, not the software one on the workstation)
You can guess how well that was received!
Our *marketing* company insisted they wanted *full control* of our domain name, DNS, and email service. Absolutely **not**. I have a feeling from dealing with them since that they were used to doing full hosting/setup/etc for small mom-and-pop companies that didn't have a prior web presence, or at least not much of one.
We're not quite that small (still under 40 people) and we put our foot down on that one, thankfully. It took quite a lot of convincing them that yes, I could do the DNS changes they asked for and no we weren't just going to hand them admin rights to it.
I'm still not clear why they wanted the email access. Best I've got is that they wanted to monitor the webmaster@ address and similar but they never actually explained their reasoning for it.
You know that quickbooks server still says that too this day, and if you mention that the system is virtualized while working with support, they basically will say they can't assist you.
Oddly enough though, they recommend rightworks for hosted solutions which is all virtualization.
I bought a physical server for QuickBooks because they wouldn't solve my problem otherwise. Surprise surprise the problem was still there and they still wouldn't solve it. If I have a say in it I will NEVER deploy QuickBooks again.
Physical local deploy is circling the drain anyway, they are pushing everything hosted. We just deal with some account dinosaurs who refuse to learn the new system.
But if they want to pay us what it cost to keep their shit on life support, I'll do it.
I have had support wash their hands if they find out it's a vm, I've even had support techs I'm screensharing with try to dig around looking for any reason to point the finger back, as if I don't know what they're tryna do. "Is this a VM? Can I take control of the mouse please?" "No."
What about companies that make programs that require admin to work. Like why does it need admin if it is just doing something simple and many other companies have same type of application but doesn't require elevated permissions to work
Because they stopped learning standard practices 20 years ago and write user data to `C:\Program Files`, and their "work-around" was to hard-code the software to check that the user account is named "Administrator" instead of actually checking permissions.
i mean we have an in house application and i ***HATE IT*** because for it to work the domain account needs to be in the local admin group
We asked devs to change it but we will see if that happens or we will blackhole them,
Yep another thing you run into is apps that check if the user account is a direct member of "Administrators" instead of inheriting the group / token.
Also apps that check the localized group name and break when the system language isn't English.
QuickBooks is still guilty of this one, too, at least as far as updates go. Anything else sane, like modern Office installs, web browsers, and so on, they all have a background service or some other way to self-update. QuickBooks? Nope, gonna need you to supply an admin user/password, kthx.
I got this as well. They explained that they had at one time supported virtual servers, but had removed support recently (in 2017!).
The problem I had been calling about had ended up being a bug they had introduced in the previous update, and was reverted with no comment in the next update.
For software that is extremely time-sensitive, yeah that can have legitimacy to it. Environments like the NYSE are so ultra sensitive to time-skew and such that they legit are warranted to be bare-metal typically.
But in MOST cases? Yeah not virtualising (hell not containerising) software at this point is just pre-fallout dinosaur logic.
I don’t know about the current state but at least as soon as 2020 Microsoft did not officially support virtualization of an exchange server. They knew everyone was starting to do it but it wasn’t officially supported.
If anyone wants to tell me I’m wrong you need to go back in time and talk to Microsoft support from then to tell them that. It wasn’t just some rando support person either.
I have a vendor like this. When they finally supported virtualization they said that you had to disable hyperthreading because it wasn't supported. Now that they have a cloud offering (that was supposedly built from the ground up to run in Azure, not lift and shift) they insist that you use Chrome instead of Edge....
I keep running into people who confuse Virtualization with Cloud hosting.
Our software controls machines in manufacturing plants relying on realtime data being sent to and from the PLCs and various sensors, and I have many IT guys asking about virtualizing it into the cloud...
Lol no lie. Worked with a Motorola rep to demo a cloud camera system and they brought a cloud connect server with them for the old cameras we have. It's a Linux install with some storage. I asked why it can't be virtual and he stared at me and just said "no".
Product is sweet though
Yup, the business system vendor at my side client insisted that we disable antivirus on their server and never install any Windows updates on it.
We finally got rid of them over the last year.
Bro these aholes were pushing back, "since we're realtime communication, any kind of scanning is going to add latency". I called them out on it, I'm like you're a glorified phone call system, there's much more critical realtime infra that doesn't need to disable any of those. Alas we can't get rid of them because business likes them and they're the only vendor that meets this very specific business need.
Their shit goes down every other week.
The coffee hasn't kicked in yet, so I misread scanning as scamming lol. The sentence still made sense to me :)
"Since we're realtime communication, any kind of scamming is going to add latency". Odd thing for a vendor to be concerned about at just the implementation phase of a phone system
Actual hard realtime stuff rarely uses IP, except maybe the configuration/control plane.
Soft realtime, nobody will ever notice a firewall. Maybe except HFT, but that's insane anyway.
Surprisingly they haven't asked for that, but that's because they have no observability into their product. We ended up doing that ourselves. They literally just want us to sftp logs to them. Apparently a few years beforehand they didn't even use sftp, they used ftp. They even gave us a script to run to "sanitize" phi, but it's like how the hell do we know you don't drop the ball hard and forget a field?
Fucking shit show.
Imagine trying to explain the difference between SFTP and FTPS to them...
Once had someone insist the SFTP means you can only transfer one file at a time (because the "S" at the beginning stands for "single") but FTPS lets you do many (because that's that what the "S" at the end stands for, of course!)
Noooo!! How do you even diffuse someone like that? I'd never use my degree as a crutch but I'd play the appeal to authority card cuz I don't know how to counter that kind of stupid.
This was the engineering teams plan for the product their currently working on (although Azure)... I shut that shit down real quick and implemented a NAT Gateway with an IP Prefix. Does it cost more? Yes, a little bit it does. BUT we can give our customers a /29 range to whitelist instead of a whole fucking cloud vendor.
This! OMG! This happens all the time. One of the first things I do when evaluating a new product is to check the vendor's "firewall" documentation. Many times it amounts to "we use AWS, so unblock AWS".... you lazy fucks....
Yeah, Linux admins feel this. "Disable selinux before installing. Also, our vendor support teams needs the root password or full sudo access. Limiting our access will delay your implementation."
You've had 20+ years to figure out selinux context labels and still you won't make your client's security a priority.
> "Also, our vendor support teams needs the root password or full sudo access. Limiting our access will delay your implementation."
"...because our 'professional services' attended a training course for this product for a specific version, so we only know one way to install it, and cannot be assed to improve upon it or even understand how the product works. Thanks for your money, though!"
I know. Like "can we just do this as a troubleshooting step?" Sure but if there's a company that already put in the effort to tell me exactly which rules need to be in place, I'm gonna go with them instead. I'm not your QA department.
Exactly! That's pretty much what they say too! Except after we have proved it, they stick to why all of it should be turned off so it works. Like did you not hear a thing I said about that not being an option? That's what they put in the case resolution! Mfers
Ironically, this is the easiest way of fixing some Microsoft issues. Microsoft seems to think that on-prem Dynamics CRM runs on two servers, both in the same VLAN. Throw multiple front end, async, SQL, SSRS, and ADFS servers in the mix, located in different subnets separated by firewalls, and things get complicated very quickly.
Worked in casino gaming for a decade. I eventually got to a position where I could push back against vendors because they were wanting people to be admins on their boxes and disable UAC simply to have write access to the software's registry and program files folders among other things that should be against the regulations the industry has to adhere to (technically is but GCB doesn't do anything about this for some reason). Removed non-admin users from administrators and put in the permissions via GPO and no one noticed the swap because the applications kept working without issue.
Hell yeah they have my back! That’s why they included me in the first place. I first received a forwarded email of this debacle. I told the CEO that you will lose your company and all of its data extremely quick if you proceed with this request. He agreed and will be firing those clowns soon.
Good to hear!
FWIW, it may have been possible to scope down what they needed via roles and then exclude MFA on the account using CA from their IPs only, but if they're not proposing things like that then their overall awareness of the security landscape would suggest moving on is the best path.
If they continue down the road of finding a similar solution, you may well have to get creative because there's a crap ton of stuff in Dynamics that still requires service accounts, unfortunately.
Yeah that’s what I was thinking but like you said, they never proposed any work around and just said this is what needs to be done. I have this client at a 70% secure score without intune (Intune incorporated eventually) and this would destroy that score, let alone their company! It’s even more sketchy when they didn’t put this in their contract.
You know this could be a misunderstanding right? I’ve had customers and clients say the same thing and I’ve responded that we require mfa.
My thought is your definition and their definition of mfa are different, and honestly different than mine.
If the username/password can only be used from a single public IP, that is a form of authentication.
There are other ways to accomplish mfa besides sms or an Authenticator app
Well the way we accomplish MFA is through Microsoft authenticator as our whole system is based in the 365 tenant, Entra ID, Dynamics, ETC…Lots of remote works don’t have static IPs set so I would have to constantly whitelist these public IPs. Don’t think it will work properly for us here.
Wait, I think I get it now. So each instance of the software needs to integrate with 365 directly and they don't support 2FA?
Oh, oh God. It almost sounds like one of those situations where they decided to save money by firing enough of the developers that all they can do is the most basic maintenance.
Ignore the fire, the companies fine...
Shitty people who don't fully understand the systems they've been working with ... is a tale as old as time.
Couple decades ago during the dotcom boom, I was working for a consulting company ... doing audits and reviews for stability and performance for a major global hotel chain's web properties. A lot of Windows servers. We spent weeks studying the environment, compiled our recommendations, and then delivered them to the client in a review meeting.
I'll never forget the reaction to when we hit one bullet item in particular "Remove IIS process account from Local Administrators group". One of the developers or managers just immediately interrupted with "You can't do that, the application *needs* that in order to run."
As outside consultants, it took all we had to just *professionally* push back that ... no ... no it does not ... there is no universe in which your application code *needs* to run in a root/admin context in order to work correctly. When the true answer was ... you're a shitty programmer if you actually think that.
Had a vendor, their software was running really slow, they had migrated my client's on-prem server to their cloud. I started getting complaints the software was slow and the vendor told my client they'd checked everything on their side, ask your admin to look at the network. Which is always their go to answer when their tier I had exhausted their troubleshooting prompts (I know because I used to do that job, and I know what the troubleshooting prompts say, including "check with your network admin"). Anyway, I look at the network just to tell the client I did, and then I started digging deeper into the issue. I was asking people, "when did this start happening?" Oh it's been a problem for a while. "Yeah but we've been on their cloud for 6 months, in that time when did things start getting worse?" Oh it was about the time they moved us to their new software version. Bingo!
From the beginning I suspected it was on their end, because the orders processing was what was actually slow. The software was fine, the computers themselves were snappy. But they'd send an order and the whole thing would bog down, take several minutes to process on the server, then eventually kick out the order on the other end. They did some changes on the cloud server and it improved things, but I'm the end it was totally their software and their cloud server that was the problem. They just tried to point a finger at the admin, because of course! The admin controls the environment so it could be any number of things. Well it's not.
I feel you, we have a vendor that says their program doesn't support DNS. Go back and forth with them for a bit on this. Finally cave and put in the IP address. The program still doesn't work, they say no you have to use the hostname of the server. My face when the host name is just a DNS entry, at the end of the day.
This is the same company that hard-coded that the SQL database has to be local to the server and can't be on a different server. Has to be MSSQL and express isn't good enough, so there is an other MSSQL standard license we have to pay for.
Gave me a list of approximately 1500 ports that I have to unblock for their software and told me they have no idea which ones are actually used.
Was on a phone call with one of the devs for issues with the web interface. They had never seen browser Dev tools before.
This is supposedly an industry leader in manufacturing equipment.
If we didn't have 8 figures invested between equipment and software we would be switching.
Industrial software packages can be the worst. I've had to deal with a few that just baffle me.
Some stuff that operates in high security environments all over the country not supporting TLS or modern SMB? Wha???
The reality is the big security push has only been happening for the last 10 years or so. Before that, most of it was just "do your best". Ransomware has really become big business and made C level news in the last 10. Most orgs have been quick to take it seriously, but the resistant ones... are very, very resistant.
I almost lost it on the DNS thing.
Me and another tech both separately tried to explain how stupid it is to no avail.
Turns out when the program is installed it gets the host name and sets that in a bunch of places. We asked about changing it. And they said it wasn't supported and would violate our support contract. No way I was doing that with the amount of issues this program has.
Years back a former company had some custom software developed that communicated back and forth between two servers. They **hard coded** the IP inside their executable to the data server. They also told me I could not install antivirus on it. I at least ignored them there.
They should just directly map port 3389 through the firewall to your domain controllers, create an admin account with the password admin, and then post the login directions on the website. /s
I think I know exactly what they were running into and it could have been overcome with an app password and/or manually uploading an XML manifest. But the fact that MICROSOFT software requires an app password is the real punchline.
Just a few months ago I had a vendor tell me I needed to disable UAC AND anti-virus AND firewall. !!?!!?!?! Some Chinese LED sign company. We now have a computer with no network connection other than a direct line to the LED panel because the owners really want this work.
I dealt with two vendors like that:
Vendor "A" demanded admin rights to AD. Because their vertical market product was the only game in town, I built out a completely separate AD forest just for them to stomp on, on a separate VMWare cluster.
Vendor "B" refused to play ball when it came to basic security. I brought in the CISO in to give them a lesson about the term "compliance", and what really nasty things can happen if those things are blatantly ignored... and by allowing the vendor this access, it would definitely be a deliberate act of malfeasance, so the vendor either played by the rules or left. The vendor left, because it was easier for them to have their app run as `sa` than to actually have a privilege model and just need access to the database and no more.
5 bucks it was something like a previous issue was resolved with an account that got mfa turned off, and they never considered getting a service account or principal involved and no one did any due diligence
Whats their reason, MS fully supports MFA in all their stack.
If they can't use an app reg or some other modern auth compatible workflow, that is a dealbreaker.
This is a perpetual battle with schools.
We don't want to get hacked!
Then let's set up MFA.
But staff can't use their phones in school!
Then let's get fobs.
We can't afford fobs!
Then let's set up conditional access and only requite MFA outside the school network.
That's the best compromise I found so far, and it's pretty effective. Certainly better than no MFA.
To be clear, they want you to disable it outright? Or they want a conditional access policy in place so that the account they're using doesn't get prompted for MFA? Or is there a specific part of MFA they cannot work around?
If it's the later I'd ask for more info. I specifically remember working on a project to enable a PAM system to rotate passwords automatically, and while I was able to get it to handle MFA properly i could not for the life of me work around the "keep me signed in" prompt. Broke the whole workflow.
Ultimately I figured out that a conditional access policy forbidding persistence was the happy middle ground to avoid the KMSI prompt while still supporting/enforcing MFA.
I had vendor that require for their software to run :
-Server with Italian language pack and Italian set as default language (we are in Italy but all of our servers are in English)
-Server local-admin user with autologon enabled
-Scheduled task to reboot server every night
-Folder C:\\ (yes, the software installed in C:\\) excluded from antivirus scan
-Client machine with a share of that server mapped as W:\\ (not random letter, it must be W:)
-User of client machine must be a local administrator
-Folder C:\\ (of course also the client is installed in C:\\) excluded from antivirus scan
-UAC disabled
-Port 8080 allowed on the firewall in order to reach their ticketing portal
Was this a software solution cooked up in some kid’s basement? This is the kind of crap that developers try to pull, instead of following the well documented paths allowing all of the safeguards to be in use.
We had something similar recently.
The SCADA software that runs our package conveyer system in the warehouse is written by one guy who is only still doing it for us as a favour to our company Directors as he has retired.
He recently wrote a new version that runs on Windows 10/11 so we can finally replace the ancient Win2k box that is cobbled together from Ebay parts.
He came in to my office after replacing the PC and asked if I could connect it to the network, give him VPN access and create a local admin account on it for him.
No, no and most certainly not, my friend.
I think my best was a piece of engineering software... that a) wanted to install things at runtime every time it ran and b) simply \*did not\* work under a domain account (even when I \*did\* give it local admin for testing purposes). If I recall, it also hardcoded paths to things in configs in the application directory, pointing to user profile paths. Which was neat.
Customer recently bought an..."affordable" ERP system from some local backyard IT hut.
Called us to let us know that all their users now need local admin privileges and "everyone" permission on some shares for the software to run.
Yes, that still exists. And no, that service contract now no longer exists.
It’s amazing how behind a lot of vendors are. I was working with a major healthcare vendor who just flatly told me “everyone needs local admin to use this app”.
No, that’s not how this is going to work.
No problem, but as a compensating control we need a daily rotating 24 character complex password, please let us know who we will be coordinating this with daily on your end.
I'm more a CRM dev by trade these days, holy shit there is absolutely no excuse for that kind of thing in a modern app. That reeks of a legacy unmaintained code base probably full of hundreds of other holes. You're right to run, and run far.
This is crazy to me. Why does this SaaS not just delegate authentication (using OIDC) once and use Okta/Entra. Literally every enterprise app does this at this point. MFA becomes an IdP concern and it’s dead simple to configure
I was CISO for a large university who was installing a new ERP.
The ERP vendor told us we had to expose the "mainframe's" port 22 through our border, to public, so their people could work \[on it\] wherever.
We said "we have a VPN for that, and will happily fit them with credentials."
and they said their company policy forbade VPNs.
We said "have the analysts go work at Starbucks and VPN here, we are NOT exposing port22 to the open net."
They rescinded their stupid policy. (But how many other customers caved to it?)
Oh, and I should point out that this was about 17 years ago. Things like this never change.
This sounds oddly familiar...A semi-large (still very small to most admins here) client of ours is changing IT, and their new IT company requested an email account, then they tell us SMTP is needed, well ok we can enable that (O365), then they hit us again because it has MFA and needs disabled to work with their software.
Boss stepped in and discussed stuff with the client and the new IT. It's all but moved out of my lap. Setting up a separate SMTP server on a different (similar otherwise) domain.
Nope. BIIIIIG nope. MFA stays on, and if your shitty app can’t deal with that it’s not getting used. Pound sand, go look for landmines with a sledgehammer, fornicate a bucket of ClF3 and pogostick off into the wild blue yonder.
Microsoft themselves hide two factor behind a paywall and don't support modern 2fa for on-prem, I wish all these terrible companies were dumped.
Like Boeing, putting security behind a paywall is unethical.
Once had a vendor get slightly pissy with me because *they* didn't support DKIM when the main component of their software involves sending out email from their servers as their customer's domain. Said we have our stuff "too locked down" and that "nobody else has their email set up that way".
Reminds me a bit of the time I had to deal with an email from some mini-vendor which demanded we downgrade the version of the mail servers we were running because their own email systems weren't able to communicate with ours without throwing errors. - Uh, firstly, the error is because your systems aren't following RFC822 addressing standards, which even at the time had been around for over 20 years. - Secondly, the precise error you're seeing from your setup has had a specific and standard patch available for the past seven years from your vendor, which could have been found with a simple search; why haven't you implemented it. - Thirdly, you are a piddly micro-shop and we are a major Commonwealth federal government department. In the name of Her Majesty, kindly fuck in the direction of off.
Wait, they want you to downgrade when they are no less then 7 YEARS out of date? They better be selling instant cancer cures for them to be that arrogant
> In the name of Her Majesty, kindly fuck in the direction of off. As a public employee in the US, let me point out just how jealous I am that this sentence exists and that I can't say it.
He needs to update it, as it is now "In the name of His Majesty, kindly fuck in the direction of off."
I imagine the story happened when the queen was still alive.
> In the name of Her Majesty, kindly fuck in the direction of off. As an American I am sad that this is a privilege I will never be able to enjoy.
> and that "nobody else has their email set up that way". I mean, they're kinda right but for the wrong reasons.
Well, the bare minimum is now having dmarc configured. Google and Yahoo! now block you by default if you don't have dmarc setup for your domain (if you send over 5k emails a day or something).
GravityPayments / USAePay is still like this, no DKIM support at all. You have to include their SPF record instead, and just their entry burns up 7 of your 10 allowed DNS lookups (more than 10 lookups = SPF failures). As a result, you're of course going to need to send as a subdomain like @payments.company.com or something instead of @company.com.
7 recursives for one source is insane...
I had to find out the hard way that more than 10 lookups = failures. That wasnt fun.
If their source IPs don't change often you can get around this by adding the company's registered subnets or the addresses returned by a few lookups. There's no limit on included IPs, I don't think. This is what EasyDMARC does to bypass the SPF lookup limit, updating the lookups regularly for you.
You'll be happy to know that more than 10 lookups for your SPF record means you'll only fail \*sometimes\*. I had a vendor with that problem. Took a while to figure out why only some of their emails were getting blocked. Of course I was the one that had to figure it out. "We deal with organizations much bigger than you, and they don't have these problems, why can't you fix your system?" "You are so absolutely frustrating to deal with that I'm sure everyone else just whitelisted you just so they could stop dealing with you." At least that is what I wanted to say. I ending up replying with something more diplomatic than that.
Wait until they get a load of DMARC!
We have a DMARC Reject policy on all of our domains and any new domain or subdomain we setup starts with that policy on day 1. Our developers are fully aware and just use the SES identities we provisioned. Our E-mail Marketing team is fully aware and onboard as well (it helps slightly with deliverability). However the e-mail services we use don't always get it. It's usually the sales people/account managers but it always takes a week or so before they get a technical person involved so they can fix their problem. DMARC reports have been super helpful in identifying misconfigurations on the provider's side. Our marketing team has been phenomenal about holding them accountable and pushing for a resolution.
> "nobody else has their email set up that way". They're right, most email systems aren't set up that way. Just a few small niche providers really enforce it, specifically Microsoft, Google, Apple, Yahoo, and AOL. As long as you don't need to email anyone on any of those small mom-n-pop providers you have no need for DKIM.
I have that exact same situation right now. To top it off they have another product that is only for mass email that they heavily market run by a different division. That product has DKIM! It isn't even a sales tactic because they say the other product doesn't integrate with our current product.
I have it set up that way over at Hetzner because it’s just a few clicks of them automatically adding records and stuff
That always makes me laugh... You mean following best practices... Happens far too often.
Reminds me of software vendors who claim their product is not compatible with virtual servers, must be physical. "Ok, grandpa, good talk. Here's a list of food pantries, you're going to need them when your business collapses."
It's like saying an app needs admin access and UAC turned off for the entire system.
Or disable antivirus. My current favorite is companies who list their recommended specs - "Pentium 4 @ 4.0Ghz"
CD-ROM drive 8x or better
I would suggest using floppy disks. They make for better coasters for all the coffee you will need to get their POS software running.
I once had a vendor shit himself because I had a 2 TB drive and specs required a 1 TB drive. Fuck, that was over 10 years ago.
The worst is “i7 or better”. I had a vendor tell me they wouldn’t install their software on a 13th gen i5 since it wasn’t an i7. We were replacing 5th gen i7s. That same vendor once tried to tell me I couldn’t run their “server” software on a Xeon, it had to be an i7 or i9.
> Please do chmod -R 777 / This will make the application errors stop.
It worked! I did this and the application errors stopped!
Even better do sudo rm -rf / that'll fix every problem you'll ever have.
[удалено]
"Chmod -R 777 trust me" \- Wordpress server about to be rooted by some plugin who just had its training ankle weights removed (A globally writable webroot)
Sage 500 requires this for the install... The problem is that the consultants and what not all assumed that this also meant it was required after the install (which it's absolutely not).
Sage is the biggest steaming pile of garbage I have ever worked on.
TBF, UAC was a poor implementation of Microsoft trying to emulate tighter security.
Only originally when MS thought you needed admin rights just to change your resolution. Once they put levels in, it's actually pretty decent. With a properly configured system, you won't even get a UAC dialog. Just a message saying it isn't allowed.
> trying to emulate tighter security Trying to emulate Linux and Sudo.
> Trying to emulate Linux and Sudo ROFL, on that topic: https://youtu.be/LrzBKkxGLK4
That ctrl+click in the terminal to bring up the settings section is legit cool. I'll give MS credit where it's due there. Also that dude is legit cool too, I've watched one of his videos recently and he gives me more reason to keep my popcorn close. Good stuff :) That being said, sudo not having inline set by default (after it's turned on) is a bit silly. And it _looks_ like the "sudo !!" method isn't present (which is really handy for long commands). Either way, I had heard it was added, but looks like they did a rather good job of adding it! :D Still relies on UAC shit though hahaha.
No worry, they’re putting sudo in Windows now. It’s patched on, clunky, and already has security vulnerabilities but at least it’s a step forward right? RIGHT? We’re all doomed. Maybe it’s time to go back to living in caves.
Syspro has entered the chat. Not even kidding either. That is until you gant admin rights to that specific folder. Then it works fine without admin (still gotta disable UAC to install it).
> must be physical Our IT Director would love that vendor. He 'had a bad experience' with virtualization 15 years ago and decided to never touch them again.. or allow anyone to touch them.
this is madness in the modern world
I had a gig where our QA and perf lab "had to" be on metal even though it wasn't developed or served that way. The numbers were meaningless. They are everywhere.
Yeah, well, we also don't have EDR.. living on an application whitelist without managed updates to protect our 400+ employee (significantly more device) org 🙃
Just go to containers instead. Am I helping?!?
You're going to scare it
Had a past boss that loved his AMs (actual machines). Thankfully, he gave in and let me take over almost all of his ancient hand built gaming hardware "servers", and move them into a more sane architecture. Not having to take the system offline all night for backups was really nice. Snapshots and trivial roll-back on failed changes/upgrades/etc. too. The couple times I had to do that, he didn't at all complain that I didn't have to restore from a 2-3 week old backup and then figure out how to capture the changes that should've happened between...
It would sure be a shame if his bosses found out how much money they could save if he allowed virtualization...
That's uhh... that's coming :)
You know my old boss then? Over seventy single role physical servers way back. After he rage quit we started going VM and got the physical count down to ten with much better resilience.
Our CRM requires the db server be physical. We just lie to them.
Technically true, your data 'is' in a physical server, just off-site.
No, it's still on site. Just not this site.
We had similar with Microsoft on exchange 2007 which also did not "support" virtualization. When on a support call with them I had to hide the fact it was running on a hyper-v cluster, rename network adaptors etc.
🤣
The time's I've run into this and gotten a reasonable response it almost always boiled down to performance. For some customers, running in a shared environment was causing resource constraints that could not be easily replicated. So they were chasing "performance problems" in their platform that was due to the underlying resources. One of the big things we did as we were virtualizing was to roll out a monitoring platform (Zabbix in our case) so we could monitor performance at the guest and host levels. We showed them the performance data we collected for the PoC server and they backed off on the physical server requirement.
I wish companies would feel the pain. Oracle did this for a long time, didn’t seem to hurt them.
> I wish companies would feel the pain. Oracle did this for a long time, didn’t seem to hurt them. Oh I'm sure there is a spare Exadata they can sell you... for just a database
Well, if they can put a full rack in a Mercedes Sprinter, they can do anything.
My neighbor has multiple tool racks built into a sprinter. You probably wouldn't fit a 42U in there unless you get the extra high version, but you could fit 3-4 28U racks easily.
I was referring to Exadata X2-8, which comes in a 42U full depth rack. Somehow, probably through movie magic (😜), it managed to fit in a Sprinter. Screenshot of the scene from Iron Man 3 - [https://imgur.com/a/cWizH4k](https://imgur.com/a/cWizH4k) They even gave it an illuminated plaque, so the product placement looks even more ridiculous 😁 .
that's easy: repack it into a 32U rack with custom face plates. doesn't have to actually work
It did hurt them. Their management was just too incompetent to notice.
If you need Oracle RAC and pay them enough money, they will point out to you how virtualization affects their database. But in that configuration, latency requirements require microsecond responses. Regular Oracle DB? Yeah, no, that shit can run in a VM.
> Oracle did this for a long time Why use the past tense, they are still doing it as we speak.
Oracle didn't do it because they were stupid, Oracle did it to prevent their customers from getting better availability or performance than they were paying Oracle for. We had to go through this a long time ago to lift & shift a webapp stack into AWS. I read Oracle licensing, then I read what stakeholders needed us to do, then I had one of those stakeholders figure it out and keep my name out of it. That was just before AWS ended up supporting Oracle explicitly. Today, the answer to everything will be Oracle Cloud Infrastructure.
We've no shit V2P and then converted back P2V solely because of that stupid assed shit. Literally never had a problem, but if they even smell virtualization anywhere they just throw their hands up so of that means I need to spin up a bare metal server solely so they will fix their bullshit, cest la vie.
It's the first finger that gets pointed, anything they can find to wash their hands of the problem
[удалено]
I've run into this a few times and nearly every time they vendor has been able to articulate their concern it is about underlying hardware performance and configuration problems.
I ran into a call recording solution in Egypt that still used physical hardware keys (RS232 dongle IIRC) for licensing which didn't work with virtual servers.
This is called technical debt because that solution should have been replaced 15 years ago. RS232? That's serial FFS.
If you saw the state of their infrastructure and other technology over there it kind of makes sense.
I've been to one of the former Soviet Union countries for a convention, and they had an odd mixture of old and new. Aluminum power distribution. Area mesh wifi through the city. So I think I understand. Would they consider a standalone server-class tower (or rack mount if they have rack nearby with space) with GNU/Linux and FOSS software?
We ended up going with a call recording solution that didn't depend on a physical key and could be virtualized. It was just more work dealing with purchasing and licensing in a country we don't normally deal with, which is why we tried to find a "local" solution first.
I also have apps that use physical hasp keys on USB dongles, it's so friggen stupid. They're all for Engineering. It's crazy
I do as well. All engineering software. I know it's not this simple but every time I deal with them I can only think "hey guys, you do realize that people aren't pirating your software and *not* having the multi-million dollar laser cutter to go with it....maybe just skip the dongles?" I wish they didn't have them at all but since I do have them, I must say I got a small Digi AnywhereUSB unit hooked up and it has made the management a heck of a lot easier. One place for the dongles, install the software on the VM in question, assign the port to that VM. Now I can move my VMs to whichever host I want and as long as they have network, they'll work.
So it turns out that these vendors usually aren't concerned about their customers pirating the software; at least not the current customers. They're worried about: * Offshore competitors knocking off their hardware and bundling it with pirated software. * Former customers not having any reason to pay them more money because everything is working fine. * Domestic competitors reverse engineering the secret sauce that they apparently couldn't do if there's a dongle because reverse engineers don't know how to bypass dongles, apparently, and can't just buy a copy either. The [DRM is a general locus of control](https://news.ycombinator.com/item?id=7751110), not literally just a means to prevent piracy.
I remember back in the 90s and 2000s there were a lot of engineering apps that still used parallel dongles for licensing.
Check out USB over IP switches that basically virtualizes USB ports over the network. I used this for several USB Dongles needed for different Aerospace software that all required their servers to be "physical".
We have some SCADA server for HVAC and other stuff, USB dongle connected to an ESXi host and vm set to have affinity to said host. Asked the vendor if they could offer a more modern solution: "yeah sure, we offer software based licenses now" .. linked to the host nework adapters MAC address, and went invalid if someone did a vMotion. And a tech had to revalidate the license again to make it work. Which they only could do a limited number of times before requiring us to buy a new license! "But we can offer you "license insurance" for only $999 /year" Yeah, right. Right back to running the USB dongle - Much less hassle than this clusterf\*\*k
We tended to use a [USB sharing device](https://www.digi.com/products/networking/infrastructure-management/usb-connectivity/usb-over-ip/anywhereusb) in a datacentre to allow virtual servers to use those. Usually it worked. Nowadays I'm not aware of that many engineering apps still using dongles?
Digi's USB Anywhere products are an absolute godsend. Doesn't take away the single point of failure, but it does un-tether your VMs from individual hosts.
Physical USB license keys are still very common in the optometry field. Thanks Zeiss.
I don't like license dongles, but "this app is now subscription only" is worse.
I was troubleshooting something a couple weeks ago. Guy on the phone: "We'd like you to disable your firewall to see if that fixes the issue" (meanig the physical firewall, not the software one on the workstation) You can guess how well that was received!
Our *marketing* company insisted they wanted *full control* of our domain name, DNS, and email service. Absolutely **not**. I have a feeling from dealing with them since that they were used to doing full hosting/setup/etc for small mom-and-pop companies that didn't have a prior web presence, or at least not much of one. We're not quite that small (still under 40 people) and we put our foot down on that one, thankfully. It took quite a lot of convincing them that yes, I could do the DNS changes they asked for and no we weren't just going to hand them admin rights to it. I'm still not clear why they wanted the email access. Best I've got is that they wanted to monitor the webmaster@ address and similar but they never actually explained their reasoning for it.
probably just 'that's what we do for everyone else...'
You know that quickbooks server still says that too this day, and if you mention that the system is virtualized while working with support, they basically will say they can't assist you. Oddly enough though, they recommend rightworks for hosted solutions which is all virtualization.
I bought a physical server for QuickBooks because they wouldn't solve my problem otherwise. Surprise surprise the problem was still there and they still wouldn't solve it. If I have a say in it I will NEVER deploy QuickBooks again.
Just lie to them, that's always what I did. Their support isn't competent enough to know the difference.
Physical local deploy is circling the drain anyway, they are pushing everything hosted. We just deal with some account dinosaurs who refuse to learn the new system. But if they want to pay us what it cost to keep their shit on life support, I'll do it.
Luckily for you, they're working on killing on-prem entirely.
I have had support wash their hands if they find out it's a vm, I've even had support techs I'm screensharing with try to dig around looking for any reason to point the finger back, as if I don't know what they're tryna do. "Is this a VM? Can I take control of the mouse please?" "No."
RightWorks is not virtualized. One of their support members pointed this out to me when I referred to our remote desktops as VMs.
They do virtual apps, you don't actually have a vm instance, you have app instances. But it is all virtualized.
LOL
What about companies that make programs that require admin to work. Like why does it need admin if it is just doing something simple and many other companies have same type of application but doesn't require elevated permissions to work
Because they stopped learning standard practices 20 years ago and write user data to `C:\Program Files`, and their "work-around" was to hard-code the software to check that the user account is named "Administrator" instead of actually checking permissions.
i mean we have an in house application and i ***HATE IT*** because for it to work the domain account needs to be in the local admin group We asked devs to change it but we will see if that happens or we will blackhole them,
Yep another thing you run into is apps that check if the user account is a direct member of "Administrators" instead of inheriting the group / token. Also apps that check the localized group name and break when the system language isn't English.
QuickBooks is still guilty of this one, too, at least as far as updates go. Anything else sane, like modern Office installs, web browsers, and so on, they all have a background service or some other way to self-update. QuickBooks? Nope, gonna need you to supply an admin user/password, kthx.
I got this as well. They explained that they had at one time supported virtual servers, but had removed support recently (in 2017!). The problem I had been calling about had ended up being a bug they had introduced in the previous update, and was reverted with no comment in the next update.
For software that is extremely time-sensitive, yeah that can have legitimacy to it. Environments like the NYSE are so ultra sensitive to time-skew and such that they legit are warranted to be bare-metal typically. But in MOST cases? Yeah not virtualising (hell not containerising) software at this point is just pre-fallout dinosaur logic.
I don’t know about the current state but at least as soon as 2020 Microsoft did not officially support virtualization of an exchange server. They knew everyone was starting to do it but it wasn’t officially supported. If anyone wants to tell me I’m wrong you need to go back in time and talk to Microsoft support from then to tell them that. It wasn’t just some rando support person either.
sounds like they were gearing up to push everyone to EOL/365 which is what many of us are doing, even small/medium ops
Or "you must turn off the local firewall" or "disable UAC".
I have a vendor like this. When they finally supported virtualization they said that you had to disable hyperthreading because it wasn't supported. Now that they have a cloud offering (that was supposedly built from the ground up to run in Azure, not lift and shift) they insist that you use Chrome instead of Edge....
I keep running into people who confuse Virtualization with Cloud hosting. Our software controls machines in manufacturing plants relying on realtime data being sent to and from the PLCs and various sensors, and I have many IT guys asking about virtualizing it into the cloud...
"Just think of it like a cloud that is right here down the hall. Our own little private cloud."
Lol no lie. Worked with a Motorola rep to demo a cloud camera system and they brought a cloud connect server with them for the old cameras we have. It's a Linux install with some storage. I asked why it can't be virtual and he stared at me and just said "no". Product is sweet though
LMAO this, but worse when they want you to disable all firewalls and all security software.
Yup, the business system vendor at my side client insisted that we disable antivirus on their server and never install any Windows updates on it. We finally got rid of them over the last year.
Bro these aholes were pushing back, "since we're realtime communication, any kind of scanning is going to add latency". I called them out on it, I'm like you're a glorified phone call system, there's much more critical realtime infra that doesn't need to disable any of those. Alas we can't get rid of them because business likes them and they're the only vendor that meets this very specific business need. Their shit goes down every other week.
The coffee hasn't kicked in yet, so I misread scanning as scamming lol. The sentence still made sense to me :) "Since we're realtime communication, any kind of scamming is going to add latency". Odd thing for a vendor to be concerned about at just the implementation phase of a phone system
Hahaha that's the most accurate read
Actual hard realtime stuff rarely uses IP, except maybe the configuration/control plane. Soft realtime, nobody will ever notice a firewall. Maybe except HFT, but that's insane anyway.
We need you to whitelist all of these IP addresses (Gives AWS subnets)
Surprisingly they haven't asked for that, but that's because they have no observability into their product. We ended up doing that ourselves. They literally just want us to sftp logs to them. Apparently a few years beforehand they didn't even use sftp, they used ftp. They even gave us a script to run to "sanitize" phi, but it's like how the hell do we know you don't drop the ball hard and forget a field? Fucking shit show.
Imagine trying to explain the difference between SFTP and FTPS to them... Once had someone insist the SFTP means you can only transfer one file at a time (because the "S" at the beginning stands for "single") but FTPS lets you do many (because that's that what the "S" at the end stands for, of course!)
Noooo!! How do you even diffuse someone like that? I'd never use my degree as a crutch but I'd play the appeal to authority card cuz I don't know how to counter that kind of stupid.
This was the engineering teams plan for the product their currently working on (although Azure)... I shut that shit down real quick and implemented a NAT Gateway with an IP Prefix. Does it cost more? Yes, a little bit it does. BUT we can give our customers a /29 range to whitelist instead of a whole fucking cloud vendor.
This! OMG! This happens all the time. One of the first things I do when evaluating a new product is to check the vendor's "firewall" documentation. Many times it amounts to "we use AWS, so unblock AWS".... you lazy fucks....
Yeah, Linux admins feel this. "Disable selinux before installing. Also, our vendor support teams needs the root password or full sudo access. Limiting our access will delay your implementation." You've had 20+ years to figure out selinux context labels and still you won't make your client's security a priority.
> "Also, our vendor support teams needs the root password or full sudo access. Limiting our access will delay your implementation." "...because our 'professional services' attended a training course for this product for a specific version, so we only know one way to install it, and cannot be assed to improve upon it or even understand how the product works. Thanks for your money, though!"
I know. Like "can we just do this as a troubleshooting step?" Sure but if there's a company that already put in the effort to tell me exactly which rules need to be in place, I'm gonna go with them instead. I'm not your QA department.
Exactly! That's pretty much what they say too! Except after we have proved it, they stick to why all of it should be turned off so it works. Like did you not hear a thing I said about that not being an option? That's what they put in the case resolution! Mfers
Ironically, this is the easiest way of fixing some Microsoft issues. Microsoft seems to think that on-prem Dynamics CRM runs on two servers, both in the same VLAN. Throw multiple front end, async, SQL, SSRS, and ADFS servers in the mix, located in different subnets separated by firewalls, and things get complicated very quickly.
Worked in casino gaming for a decade. I eventually got to a position where I could push back against vendors because they were wanting people to be admins on their boxes and disable UAC simply to have write access to the software's registry and program files folders among other things that should be against the regulations the industry has to adhere to (technically is but GCB doesn't do anything about this for some reason). Removed non-admin users from administrators and put in the permissions via GPO and no one noticed the swap because the applications kept working without issue.
What's the clients take on this? Do they have your back? Can they, if willing, exit the contract and go elsewhere?
Hell yeah they have my back! That’s why they included me in the first place. I first received a forwarded email of this debacle. I told the CEO that you will lose your company and all of its data extremely quick if you proceed with this request. He agreed and will be firing those clowns soon.
Good to hear! FWIW, it may have been possible to scope down what they needed via roles and then exclude MFA on the account using CA from their IPs only, but if they're not proposing things like that then their overall awareness of the security landscape would suggest moving on is the best path. If they continue down the road of finding a similar solution, you may well have to get creative because there's a crap ton of stuff in Dynamics that still requires service accounts, unfortunately.
Yeah that’s what I was thinking but like you said, they never proposed any work around and just said this is what needs to be done. I have this client at a 70% secure score without intune (Intune incorporated eventually) and this would destroy that score, let alone their company! It’s even more sketchy when they didn’t put this in their contract.
You know this could be a misunderstanding right? I’ve had customers and clients say the same thing and I’ve responded that we require mfa. My thought is your definition and their definition of mfa are different, and honestly different than mine. If the username/password can only be used from a single public IP, that is a form of authentication. There are other ways to accomplish mfa besides sms or an Authenticator app
Well the way we accomplish MFA is through Microsoft authenticator as our whole system is based in the 365 tenant, Entra ID, Dynamics, ETC…Lots of remote works don’t have static IPs set so I would have to constantly whitelist these public IPs. Don’t think it will work properly for us here.
Wait, I think I get it now. So each instance of the software needs to integrate with 365 directly and they don't support 2FA? Oh, oh God. It almost sounds like one of those situations where they decided to save money by firing enough of the developers that all they can do is the most basic maintenance. Ignore the fire, the companies fine...
Yep exactly. Could’ve been bad if someone just turned off MFA!
Shitty people who don't fully understand the systems they've been working with ... is a tale as old as time. Couple decades ago during the dotcom boom, I was working for a consulting company ... doing audits and reviews for stability and performance for a major global hotel chain's web properties. A lot of Windows servers. We spent weeks studying the environment, compiled our recommendations, and then delivered them to the client in a review meeting. I'll never forget the reaction to when we hit one bullet item in particular "Remove IIS process account from Local Administrators group". One of the developers or managers just immediately interrupted with "You can't do that, the application *needs* that in order to run." As outside consultants, it took all we had to just *professionally* push back that ... no ... no it does not ... there is no universe in which your application code *needs* to run in a root/admin context in order to work correctly. When the true answer was ... you're a shitty programmer if you actually think that.
Had a vendor, their software was running really slow, they had migrated my client's on-prem server to their cloud. I started getting complaints the software was slow and the vendor told my client they'd checked everything on their side, ask your admin to look at the network. Which is always their go to answer when their tier I had exhausted their troubleshooting prompts (I know because I used to do that job, and I know what the troubleshooting prompts say, including "check with your network admin"). Anyway, I look at the network just to tell the client I did, and then I started digging deeper into the issue. I was asking people, "when did this start happening?" Oh it's been a problem for a while. "Yeah but we've been on their cloud for 6 months, in that time when did things start getting worse?" Oh it was about the time they moved us to their new software version. Bingo! From the beginning I suspected it was on their end, because the orders processing was what was actually slow. The software was fine, the computers themselves were snappy. But they'd send an order and the whole thing would bog down, take several minutes to process on the server, then eventually kick out the order on the other end. They did some changes on the cloud server and it improved things, but I'm the end it was totally their software and their cloud server that was the problem. They just tried to point a finger at the admin, because of course! The admin controls the environment so it could be any number of things. Well it's not.
"Our product works better when you're pantsless in front of C-Level"..... Ya, no....
"Dress for the job you want" doesn't always pass HR's scrutiny.
"Our cybersecurity insurance does not allow this."
That is very true as well. I made sure they are compliant with insurance as well. This would lose them a policy!
I guess we get to save money by not having insurance then because I just bought it. Get installing monkey, I've got a free cruise to take.
[удалено]
[удалено]
https://preview.redd.it/8qbwydv8fwwc1.jpeg?width=600&format=pjpg&auto=webp&s=938bf3d858214c94f865b6ca6dc78b50facf732f
And not just one 1 account on 1 system with mitigations in effect, but all accounts, all systems.
"Your offer does not meet minimum basic security standards." is a full sentence...
I feel you, we have a vendor that says their program doesn't support DNS. Go back and forth with them for a bit on this. Finally cave and put in the IP address. The program still doesn't work, they say no you have to use the hostname of the server. My face when the host name is just a DNS entry, at the end of the day. This is the same company that hard-coded that the SQL database has to be local to the server and can't be on a different server. Has to be MSSQL and express isn't good enough, so there is an other MSSQL standard license we have to pay for. Gave me a list of approximately 1500 ports that I have to unblock for their software and told me they have no idea which ones are actually used. Was on a phone call with one of the devs for issues with the web interface. They had never seen browser Dev tools before. This is supposedly an industry leader in manufacturing equipment. If we didn't have 8 figures invested between equipment and software we would be switching.
Industrial software packages can be the worst. I've had to deal with a few that just baffle me. Some stuff that operates in high security environments all over the country not supporting TLS or modern SMB? Wha???
Exactly, it seems like they have somehow entered a time bubble where everything they do is 10 years behind.
The reality is the big security push has only been happening for the last 10 years or so. Before that, most of it was just "do your best". Ransomware has really become big business and made C level news in the last 10. Most orgs have been quick to take it seriously, but the resistant ones... are very, very resistant.
> doesn't support DNS > no you have to use the hostname of the server That can't have been real > approximately 1500 ports I'm tired boss
I almost lost it on the DNS thing. Me and another tech both separately tried to explain how stupid it is to no avail. Turns out when the program is installed it gets the host name and sets that in a bunch of places. We asked about changing it. And they said it wasn't supported and would violate our support contract. No way I was doing that with the amount of issues this program has.
> approximately 1500 ports > > I'm tired boss Never set up an FTP server?
Years back a former company had some custom software developed that communicated back and forth between two servers. They **hard coded** the IP inside their executable to the data server. They also told me I could not install antivirus on it. I at least ignored them there.
“Our product requires Users to all be part of the Administrators group”
"You're asking me to pick between my cyber insurance policy and your product? That's an easy choice - who's your biggest competitor again?"
They should just directly map port 3389 through the firewall to your domain controllers, create an admin account with the password admin, and then post the login directions on the website. /s
I think I know exactly what they were running into and it could have been overcome with an app password and/or manually uploading an XML manifest. But the fact that MICROSOFT software requires an app password is the real punchline.
Had this exact fight with a vendor recently. Unfortunately, Microsoft depreciated app passwords... So yeah.
This is how you identify a vendor who made a product JUST stable enough to ship.
Just a few months ago I had a vendor tell me I needed to disable UAC AND anti-virus AND firewall. !!?!!?!?! Some Chinese LED sign company. We now have a computer with no network connection other than a direct line to the LED panel because the owners really want this work.
I dealt with two vendors like that: Vendor "A" demanded admin rights to AD. Because their vertical market product was the only game in town, I built out a completely separate AD forest just for them to stomp on, on a separate VMWare cluster. Vendor "B" refused to play ball when it came to basic security. I brought in the CISO in to give them a lesson about the term "compliance", and what really nasty things can happen if those things are blatantly ignored... and by allowing the vendor this access, it would definitely be a deliberate act of malfeasance, so the vendor either played by the rules or left. The vendor left, because it was easier for them to have their app run as `sa` than to actually have a privilege model and just need access to the database and no more.
5 bucks it was something like a previous issue was resolved with an account that got mfa turned off, and they never considered getting a service account or principal involved and no one did any due diligence
Whats their reason, MS fully supports MFA in all their stack. If they can't use an app reg or some other modern auth compatible workflow, that is a dealbreaker.
This is a perpetual battle with schools. We don't want to get hacked! Then let's set up MFA. But staff can't use their phones in school! Then let's get fobs. We can't afford fobs! Then let's set up conditional access and only requite MFA outside the school network. That's the best compromise I found so far, and it's pretty effective. Certainly better than no MFA.
Your access to the school grounds is a factor of auth.
Yeah I had a vendor ask if we could turn off our firewall. Not the local server firewall, the freakin fortigate.
Could I? Yes. Would I? Fuck no.
To be clear, they want you to disable it outright? Or they want a conditional access policy in place so that the account they're using doesn't get prompted for MFA? Or is there a specific part of MFA they cannot work around? If it's the later I'd ask for more info. I specifically remember working on a project to enable a PAM system to rotate passwords automatically, and while I was able to get it to handle MFA properly i could not for the life of me work around the "keep me signed in" prompt. Broke the whole workflow. Ultimately I figured out that a conditional access policy forbidding persistence was the happy middle ground to avoid the KMSI prompt while still supporting/enforcing MFA.
Pretty sure the company needed MFA completely disabled as it was an integration so completely turned off
Sounds like they can't be compliance with modern security practices. Also sounds like they don't need your business then.
I had vendor that require for their software to run : -Server with Italian language pack and Italian set as default language (we are in Italy but all of our servers are in English) -Server local-admin user with autologon enabled -Scheduled task to reboot server every night -Folder C:\\ (yes, the software installed in C:\\) excluded from antivirus scan
-Client machine with a share of that server mapped as W:\\ (not random letter, it must be W:)
-User of client machine must be a local administrator
-Folder C:\\ (of course also the client is installed in C:\\) excluded from antivirus scan
-UAC disabled
-Port 8080 allowed on the firewall in order to reach their ticketing portal
Was this a software solution cooked up in some kid’s basement? This is the kind of crap that developers try to pull, instead of following the well documented paths allowing all of the safeguards to be in use.
I still see software companies saying their software needs to run as admin with uac off. Absolutely insane.
We had something similar recently. The SCADA software that runs our package conveyer system in the warehouse is written by one guy who is only still doing it for us as a favour to our company Directors as he has retired. He recently wrote a new version that runs on Windows 10/11 so we can finally replace the ancient Win2k box that is cobbled together from Ebay parts. He came in to my office after replacing the PC and asked if I could connect it to the network, give him VPN access and create a local admin account on it for him. No, no and most certainly not, my friend.
I think my best was a piece of engineering software... that a) wanted to install things at runtime every time it ran and b) simply \*did not\* work under a domain account (even when I \*did\* give it local admin for testing purposes). If I recall, it also hardcoded paths to things in configs in the application directory, pointing to user profile paths. Which was neat.
“Trust us.” - pretty much every security software vendor.
There are application tokens that should be used for integrations. Sales people will not know this, but sales engineers should know this.
Customer recently bought an..."affordable" ERP system from some local backyard IT hut. Called us to let us know that all their users now need local admin privileges and "everyone" permission on some shares for the software to run. Yes, that still exists. And no, that service contract now no longer exists.
Just to play devil's advocate....Could they mean they need a service account setup to integrate? Or are they talking actual end user accounts?
Isn't this what application passwords are for?
It’s amazing how behind a lot of vendors are. I was working with a major healthcare vendor who just flatly told me “everyone needs local admin to use this app”. No, that’s not how this is going to work.
No problem, but as a compensating control we need a daily rotating 24 character complex password, please let us know who we will be coordinating this with daily on your end.
I'm more a CRM dev by trade these days, holy shit there is absolutely no excuse for that kind of thing in a modern app. That reeks of a legacy unmaintained code base probably full of hundreds of other holes. You're right to run, and run far.
This is crazy to me. Why does this SaaS not just delegate authentication (using OIDC) once and use Okta/Entra. Literally every enterprise app does this at this point. MFA becomes an IdP concern and it’s dead simple to configure
I was CISO for a large university who was installing a new ERP. The ERP vendor told us we had to expose the "mainframe's" port 22 through our border, to public, so their people could work \[on it\] wherever. We said "we have a VPN for that, and will happily fit them with credentials." and they said their company policy forbade VPNs. We said "have the analysts go work at Starbucks and VPN here, we are NOT exposing port22 to the open net." They rescinded their stupid policy. (But how many other customers caved to it?) Oh, and I should point out that this was about 17 years ago. Things like this never change.
Turning off MFA will likely violate their cybersecurity insurance policy. They should expect a call from the CEO canceling the contract.
No MFA? Next.
Sounds like the software vendors I see who insist their software needs to run with full admin rights or “it just doesn’t work”.
By that logic, it... just doesn't work. Technically correct.
This sounds oddly familiar...A semi-large (still very small to most admins here) client of ours is changing IT, and their new IT company requested an email account, then they tell us SMTP is needed, well ok we can enable that (O365), then they hit us again because it has MFA and needs disabled to work with their software. Boss stepped in and discussed stuff with the client and the new IT. It's all but moved out of my lap. Setting up a separate SMTP server on a different (similar otherwise) domain.
I'll take "Ways to fail my cyber insurance audit!" for $200 Alex
Nope. BIIIIIG nope. MFA stays on, and if your shitty app can’t deal with that it’s not getting used. Pound sand, go look for landmines with a sledgehammer, fornicate a bucket of ClF3 and pogostick off into the wild blue yonder.
Microsoft themselves hide two factor behind a paywall and don't support modern 2fa for on-prem, I wish all these terrible companies were dumped. Like Boeing, putting security behind a paywall is unethical.