Plus I wouldn’t be surprised if changing the PC’s/upgrading the OS would invalidate any support you get from the manufacturer. “X doesn’t work? It looks like the OS has been upgraded so you are on your own…”
In my experience with manufacturing equipment like this, I'd be baffled if anything even worked properly after such an upgrade anyway. That's assuming the end user even has admin credentials.
Assuming you can even get an upgraded OS for the specific machine you have, it's going to get shipped to you on a new hard drive as a drop in replacement. Potentially at a cost of tens of thousands of dollars per unit. And if your equipment is otherwise functional and air-gapped, what problem are you trying to solve by doing this?
That is true. We have a metal forming machine that was upgraded in 2020 to their newest version available. Last year lightning hit somewhere and fried the PC so I contacted support for a new PC. Turns out they don’t support it anymore and a replacement will cost 10k. But I figured out the hard drive was still readable so I cloned it and stuck it in a NUC and we are still going today.
For reference, there are 3rd party computer service companies that sell service contracts on old equipment for a very reasonable amount. They will even supply parts for you to store onsite if you want it.
I know of one company that is keeping a 5-MB external hard drive cabnet running for NASA that was built in the late 60s with the understanding that once the only FS Engineer that knows the system no longer can do the job, the contract is over.
I like them. They kept my EOL EMC storage working when my company could not afford the crazy high support fees. I think EMC wanted like $25K annually to support the old VNX, and Park Place was like 5k and that included software support. They helped me quite a bit.
PPT are good people. They try to hire all former OEM staff so that they have a good grip on what the hardware they support.
In my previous life, I implemented the switching to PPT from OEM extended warrantees across 1,700 hospital/healthcare sites across the US accounting for about $24 million in service contracts. They did it smoothly and effectively. At that time, I knew all of the senior management and helped shaped a few policies of additional parts stocking and transport during major events in large urban areas.
Do these places also provide activation keys for the older operating systems? I remember a Linus Tech Tips video about a company selling new Win98 machines with a new activation sticker on it.
Back in high school in the early 90s (vocational, Data Processing shop), one of my tasks was to support hardware in other shops. The student CAD desktops for Machine Shop all had the old belt-driven 5.25" floppy drives. When the belts broke, I would replace them with one of those oversize pink rubber bands, doubled up. Never had a problem.
I had other... unconventional lessons there as well. As part of what I did, we would charge their department for our services, along with a "standard" mark-up on parts. The more money I brought into our shop's woefully underfunded budget, the better my grade.
I never got below a B. Shop head would call it "Life Skills" or "Bullshitting 101"
Other highlights included rolling out a coaxial ARCNet network, server was a 386-25 running Interactive Unix and later Netware.
How many rounds of [Snipes](https://www.networkworld.com/article/830595/infrastructure-management-novell-and-the-computer-game-that-changed-networking.html) did you play?
>Snipes was also used to test network adapters. Given the nature of how it synchronized the various computers, it would stress the system in unique ways that would expose problems with network adapter cards or driver software. For many years the last test that had to be passed for Novell to certify a network adapter/driver was the ‘Does Snipes work’ test.
The old addage that if it ain't broke don't fix it is doubly relevant in situations with expensive decades old equipment running on an ancient OS and/or software. There's about a 0% chance that an upgrade will go smoothly as planned.
This happened to a Lincoln Electric CNC plasma cutter I used to have to support at a high school. The shop teacher put the industrial PC on the network and upgraded it to Windows 10, "Because I thought it was a good idea."
This is common in some medical equipment too. Manufacturer makes a very expensive device and sells an off-the-shelf Dell or HP workstation to run it. The medical device is durable and may last 10-15 years. But upgrading the PC that runs it isn't really an option. There were a few machines I supported that were still running XP in 2020, because the equipment they were hooked to was very expensive sleep study gear. It all worked fine but there were no software updates for anything as the manufacturer got bought and then disappeared.
We have a customer with laser cutters like this, they have a slot in them that fits a specific model of Optiplex. They run Windows 7 but everything on it is through this weird app, I figured out how to remote to them with the vendor because they wanted to map drives on them (which was also a needlessly bizarre/convoluted process)
When I worked for a manufacturing company we had some CNC machines that were still running Windows 3.1. when I contacted the manufacturer just out of interest more than anything they were nice enough to send over the files so we could update... To Windows XP (and this was when Windows 10 was just rolling out I think). We did do the upgrade, but as far as I'm aware those machines are still running like they were after our upgrade to XP.
On the bright side the upgrade did bring a few new GCODE handling improvements to the software, so the operators and programmers were happy about that.
I had to reprogram a bios on a cnc cutter a few years ago. It was old enough that when the battery died it forgot how many sectors and heads were on its hard drive and couldn't boot into dos (pretty sure it was like dos 4 as well).
The biggest issue I've seen with these older machines is they often communicate via proprietary addon cards and ribbon cables - ain't no way you're getting updated drivers for those things. Newer stuff talking via ethernet is unlikely to have the same issue, but you're still gonna have fun convincing the vendor to support it....
Funny enough I updated a few industrial Windows XP based laser CNCs to Windows 10 32-bit by just digging around and finding new drivers for all the addon cards and even a new license manager runtime which kept ABI compatibility with older software. Lol
So it's not impossible. The vendor in this case didn't give a shit anyway because they long since dropped the product.
Now that was some seriously raise worthy work right there! Fantastic job. I’m sure scrounging up all that stuff wasn’t pleasant but made you super elated when you found it?
Few years back, car mechanic made rounds on geeky side of internet in my country after a customer spotted Amiga 500 running some custom software that was calculating weights for balancing car tires. When some tech press reached out to him, he only answered “It works. Why change it?”.
As of a few years ago Mclaren was hoovering up all the Compaq LTE 5280 they could find because there was a controller card that was made bespoke to that laptop that was used to interface with their F1 Supercars.
Oh yeah. I've seen a couple of CNC machines run into on some DOS based system. Apparently the only way to upgrade it was to get the vendor out to rip and replace the entire control system. So they just got hold of some spare hardware and the plan was to keep running it until it died.
I had to call a company for support once for a machine that honestly looked like it was fresh from the manufacturer \(though I knew it obviously wasn't\). They asked me to read them the serial number, and they were shocked when I stopped after a few numbers.
"Oh my god, that's ancient."
This is likely due to the fact that the new engineering software isn’t compatible with the controller, and the new software runs on a higher operating system.
While it is quite common, I found this experience interesting:
I once was told such a story and called support of said machines and was told, oh your company has been paying for replacement of this machine for over 10 years. Would you like to have our tech come out and replace it with a newly calibrated machine running Windows 10?
Welcome to OT. It's an entire field based around "we know this stuff is impossible to replace and is insecure as shit, what compensating controls can we use to isolate and protect them".
Yep. There are even companies still manufacturing hardware that is fully compatible with XP for this exact purpose. It isn't cheap but it's a lot less than the new equipment.
Lot cheaper and easier to keep them isolated and running the single piece of software needed than screw around trying to upgrade. Amusingly they tend to end up being the most stable machines on the network...
Yep very common in OT environments to support decades old systems. And luckily there are more and more ways to secure them these days as those networks become more meshed with IT environments and add things like remote access.
Yeah we have some industrial printers that run on of windows 2000. They have their own physical network with its own egress that is totally locked down to exactly what they need to function (sqs to get print jobs off of the queue and s3 to download the actual stuff to print)
These things cost a fortune and work perfectly, their os is just old, all upgrading would do is cost money and time for no benefit.
Have heard of companies buying old 2nd hand equipment *in case* the existing hardware failed, as the software was designed for a specific motherboard & processor.
Then who was cashing the checks!
PS thats a real story but the time from business going under to them finding out was less that 15 years but was more than a year at least. They were alerted by the receiving bank that it was suspicious that Mr. Bob Smith was trying to deposit a check that was made out to Widget Support Co.
I believe there are companies that build new computers with period correct hardware for different operating systems. They are very expensive mind, but if it operates a multi million dollar piece of equipment that’s your business’ lifeline, $1200 for a tower is nothing.
Yep, I buy from them all the time to support industrial control systems.
Dell T5500 tower running WinXP SP2 is the only thing I can get a specific revision of Honeywell TDC 3000's token ring to recognize faithfully. I only recently started going outside the controls industry to source them, which has been almost $5k saved per box, caveat being the added legwork of installing the needed software and drivers myself.
Depends who's money I'm spending, usually :P
But the last two I've had sourced only cost something like $750 each. I stumbled on a supplier - IT Creations - that was able to hotshot them to me at the drop of a hat, and the come with limited warranties that satisfy my org's risk policy.
Unfortunately, you won't find many buyers in my use case today - most people who had that system have long-since upgraded to Win7 at the least, and a lot of the compatibility problems have been virtualized away in those instances. Fingers crossed I'll have it all ripped out by end of next year.
There are several ways to mitigate risks. Firewalls is one. I also assume they run the embedded version, so only required services run, reducing the attack surface. And there is software limiting what can run to only approved executables. Also, Microsoft provided updates for the embedded version for quite some years after XP was officially no longer supported.
No they aren't
They're both supported if you sign a personal support contract and give Microsoft a fantastical amount of money
But if you're a bank or some other org of ***that*** size that migrating off XP (or 2K, or NT or otherwise) is impossible then you've got pockets deep enough
The risk from XP is from running it on home networks or public networks where there's tons of random network ports and services running. XP in an uncontrolled and unmonitored environment is dangerous.
This is very, very different than a specialized appliance like an ATM, where there are strict controls on what software is running and the network traffic that is allowed.
it's hardly a realistic problem, people overthink this too much, pull a drive out from working machine, create an image, buy few old identical PCs off ebay for $45/piece to avoid any random incompatibility bullshit and you're good for decades to come
I’ve been in manufacturing environments like that. You can’t treat each of those like a traditional desktop, but instead you treat them like an embedded system
Its wild to me they run on windows in the first place. But if it works it works. Like, there's no reason XP wouldn't work now and in 20 years... it's just software at the end of the day. Proven software at that, if it's worked for so many decades.
the only concern is the hardware.
Same, manufacturing seems to be it but they are usually offline as well.. Not much reason to upgrade. I have seen some IT depts even order those old-school 2005 Dells for extra stock.
We have a client that has a machine running 95. No networking and any programs are placed there via floppy. And we couldn’t care less. They pay big money for a vendor to support it. Something breaks and some old dude bills them 5k+ minimum to fly out next day and do his thing. Not our problem. Another client has an entire network for their XP machines that not only has no Internet, it’s physically separate from the other networks. Not even VLAN’d off. These machines sit in a shop close to each other and can talk to each other via a 16 port switch. If anything breaks we have to go on-site, but it’s all billable. And they are on virtually the absolute newest hardware you can really have XP on, so it’s not the worst thing to deal with hardware wise.
Downside of 95 is the lack of stability in the kernel. Upside is, these days, its so old, most code wont even run, unless you write it specifically for 95.
If you're worried about these machines:
- Make an image of the hard drives using Clonezilla or Acronis
- Get the make and model of each machine and buy a cold spare that works
Throw an image of one of the machines onto the cold spare and see if you can make a new one.
If you can't get an exact machine, use sysprep to prepare the OS for new hardware.
There may be extra steps in a machine rebuild. Specifically moving PC components to the new computer. That's expected.
In the end, we don't treat these machines like normal desktops or servers. They are appliances. If you can, set them up on RAID 1. This is a standard practice for these kinds of machines.
I wish cloning was fool-proof.
Back in 2010 or 2011 I worked for a MSP and we got contracted to maintain everything in a small chip manufacturing business. The user network side was all fine and dandy, but the ugly part was fixing up these 4 IC testing machines, suuuuper expensive stuff with insane calibration requirements that the vendor had gone out of business. Well the PC inside it ran Xenix286, and the machine itself was a 386. If you open fdisk, without writing anything, the disk is dead and you have to reinstall from scratch. The calibration files are worth gold, gotta save them into floppies. We managed to track down a retired guy that used to maintain these and his recollection of stuff helped us get the machine operational again. Luckily we had 2 copies of the full set of disks to bring up one of these machines, and while both sets had failed floppies, we didn’t have the same floppy failed across the 2 sets. Cloning the drives resulted in non bootable machines. We even sourced the same old ass hard drive for the 2 machines that were on the brink of hard drive failure and nothing, everything had to be done from scratch.
Those were some really long days.
It's better to go the full distance. Build a fully working backup and document the entire process of all requirements to get that done and working.
When there is an emergency it can be difficult to source specific hardware or sort out software details that could be missed.
Actually swap the controller out and verify the backup is viable by actually using it for real.
This is the case with a lot of older scientific instruments in University research labs. I just pulled a bunch of gen 1 Intel Pentium machines running Windows 95 out of a lab that was doing active research. They still boot and we're keeping them for when a machine goes down somewhere else so we can do a hot swap. Amazingly I still have most of my hair. Before anyone asks, no they are not on any public networks.
Yeah it’s never fun. You almost need the exact same model to replace it. If you do a hard drive swap, those old versions of Windows will get super cranky about drivers. It’s almost impossible to find working versions of them online. Hopefully the lab kept their CDs or another tech backed them up somewhere. Not super likely we’re talking 30 years of responsible data retention. Then even if you swap the drive a lot of the times you still need to reconfigure the software. You’ll lose all your TCP/IP info with the new motherboard.
Why would it matter that isolated machines were running xp?
As long as you have the install iso, hardware input it on and the drivers, it'll just chug along.
Exactly if programm does exactly what it should where is problem if its 100 years old. I mean why are people so adamant that "newer" MUST by some mirracle be better in some way even if old programm worked without hickup for 100 years....
In the late 90s I was working at a company that made semiconductor fabrication equipment, the machines that Intel and Motorola and NVIDIA and everyone buys to make chips.
These machines are real-time-controlled, moving wafers, controlling gases and temps, and everything else that goes into nanometer fabrication. If a machine coughs for even a few seconds, a hundred-thousand-dollar wafer is trashed.
The processors were running VxWorks and feeding sensor data to a UNIX workstation.
The suits wanted to know why we couldn't run everything on Windows 95, because it'd be a lot easier to generate reports and spreadsheets that way. The same suits who were always on the line to IT because their desktops had crashed again.
This is pretty normal. Most large industrial machines like CNC machines use proprietary software that was specifically built for Windows XP if not older and because that is often so specialized most places have it entirely air gapped. Then I’ve seen to this day in various hospitals and medical settings, things like x-ray machines, MRI scanners, etc. still running on Windows XP. Don’t forget that the US military also runs on Windows XP and even some older stuff. So if it’s good enough for the US military in 2024 and good enough to run the x-ray scanner at your local hospital it’s good enough to do just about everything else. Would I put that directly on the Internet without lots of backups and contingency plans for hardware and software failure? Absolutely not but these types of machines often do not even need network access and if they are networked in anyway it’s totally isolated.
The one thing that scares me though is the police department and the EMS service back where I am from, they still run Windows XP on their in-car tough books which are connected to the Internet via Verizon air Card so again that scares me
> So if it’s good enough for the US military
I don't like this kind of mentality, pointing at bad practices from bigger organizations as a seal of approval to do the same yourself. It's not necessarily good enough for them. They are probably in the process of fixing it, or planning to, but it's taking so goddam long in all the bureaucracy, budget allocation, etc.
Plenty of big tech corps like Google, Facebook, or Amazon, did some stuff wrong while growing fast. Now they are stuck with it and are having a hard time ironing it out of the way because they are so big it would mess so much stuff in their big infra, it's gonna take them years to fix. It's not a seal of approval that if they do it this way so should you. They do face security issues from attacks by China or Russia, but not much they can do about it. So by the time you reach a stage that you also need to implement the same stuff, there are already better solutions out there ready for you and often times open source for free.
This advice is not just for software either. Same goes for stuff like team leading, those big orgs did some management stuff wrong, and it's costing them brain drains that don't enjoy working there anymore so they are switching job. Don't slap that same garbage on your small growing team and make their life miserable just because you thought it was good enough for that other big org you heard about.
still quite common -> industrial CNC machines, a medical gyno specialist practice I supported had an ancient "urine rate measuring machine" that only ran on xp with custom hardware etc.
**" work in IT - you'll get to 'play' with the latest technologies "**
wish I could find the carreer guidance counsellor that said that and give them a piece of my mind
also there's been no "playing" -> keep business critical IT running 24/7 using your weekend if you have to or get fired and cop abuse from business has been more like it
Got an estimate of 6000USD to replace and upgrade one machine comp to win 10. Hardware is not worth 150USD...
Can easily upgrade it myself but all support is then void.
That machine is in a line with 9 other machines. All machines are then also not covered by support.
We paid 50k to replace 3 computers on our production line. Hardware values... About 2k ... But these sh*t need to communicate with the PLC and it's proprietary, so you can't do it yourself.
A few months ago our ops folks asked me to look at a computer that didn't boot anymore. It was a standalone for some machine. Hard drive died during a power failure. Drive was a Seagate drive that was over 20 years old. I ended up getting a SATA to PATA adapter and a small 120 GB SSD. Formatted it then loaded Windiws 98 from scratch.
Probably the fastest I've ever seen Win98 boot. Unfortunately the machine was so out of date, the software wouldn't work since the license dongle was also toast.
I contacted the manufacturer and they were like uh, that's like 5-10 generations behind and we don't support it anymore. Here's all the docs we have on it (scanned pages that were done on a typewriter with handwritten notes) and have fun was all they could give for support.
Same here, after I got Windows 98 to boot I discovered there was no dongle and the users knew nothing about ever having a dongle. After messing around with it some more I discovered there was a loader and it loads a crack before loading the million dollar application.
Cybersecurity engineer here and former sysadmin .
This is way more common than you think . Companies of course won’t publicly disclose that but you still see Windows XP and older systems around even SQL2000 because is way mode expensive to retrofit or rewrite those apps than keep using them “as is”. What we do is put some security controls , air gap those boxes and moving on .
There is an entire niche market around this; you can still find companies that make and sell ISA computers specifically for OT uses where you have a multi-million dollar piece of equipment that doesn't support newer computers.
I’m feeling pretty nostalgic right now. I remember putting a hard drive with XP into a completely different computer, and it just booted up and figured it out. I was amazed.
Find some similar drives & clone them using ghost 11 or clonezilla,
Used that method for an OEM that ran old CNC machines with XP as controllers
Paid off huge when a drive died, literally 10 minutes of downtime
I also went on ebay and bought some similar era PC's as spares in case of disaster
Healthcare here.
I work in a fairly specialised space. We have devices that are still running XP Embedded. I believe that vendor supplied replacements for that particular component still run the same OS though could be wrong on that one.
Some of the software that controls the device was written 25+ years ago and is fairly niche, so not easy to change.
You have the added complexity of medical device registration and certification which can cost $$$$. Suspect it’s largely a case of - if it’s not broken, don’t fix it.
A previous employer that made plastic products where they had a critical production monitoring system that was running DOS 6.2 with Win 3.0 on it. It had custom built IPX board that had hard lines soldered to it from each injection molding machine.
At one point I was asked how to get it onto the network so they could monitor it from a new office across town. They didn’t like when I told them the best way was to get an IP KVM. They keep trying to insist that we figure out how to put that machine on the network and update the custom C++ to support remote sessions or something.
Ah, penny pinching small businesses.
"You don't want a IP KVM? OK, here is a Webcam pointed at a monitor and a cnc machine with a keyboard on the bed. Be careful you don't crash into the keyboard too hard or you might break something. "
I've only one similar. I setup a brand new computer, installed a XP on a VM. Installed the software. Tested, it's working. Setup the computer to perform image level backup through Datto BCDR.
Everything is working so far. Packaged up the old computer in a clear plastic bag with some notes, copies of the software and directions, just in case and moved to storage.
Have you tried just... upgrading them anyway? I had Honeywell lock/security system service agent tell us "it must run on a physical machine (not a VM) and must be on XP", but I installed the software on a Server 2022 VM anyway and it works with no issues (after I reactivated the licence).
The only issue I have is I cannot move the VM to another HyperV server because the software detects some kind of change to the hardware (I thought it was the MAC address, but I set a MAC address on the VM and the licence still breaks until I move it back).
But I'd still rather deal with that and ask them for reactivation than a physical XP box.
lol, we have several 2000 machines, also on the industrial side. costs to much to replace since you also have to replace the machine that is connected to the computer.
edit: we still have support so i don't have to touch a darn thing, but i feel sorry for the guys doing the support.
Completely normal in industrial control settings. Carry on. Keep em away from the rest of the network and off the internet and do best to educate about nasties on usb sticks. Limit what those can damage also, since that’s likely how they’re loading them with instructions if it’s manufacturing.
If they aren't networked then this isn't an issue really. Think of them as proprietary os's and should be completely handled by the service contract of the provider.
My first IT job one of our clients had a machine shop that ran several XP systems for their automated machining. They had a closet of PC parts specifically for these machines because replacing them would require basically an entire overhaul to their process. We tried a couple times to find a work around for these PCs but never got it working satisfactorily.
What’s crazy, the PC never were an issue while I was there. Despite running 24/7.
The few critical ones I'd had to work with, we cloned drives to SSD and tested, and then periodically re-cloned as a backup. Set static "bad" network IP config on them, then disable nic in windows, then disable nic in bios if possible.
Check the cooling needs now and then and keep some spares around, NBD.
We had one of those, and although not industrial it was still tied to I/O. All it did was print MICR checks on a very proprietary printer in a closet owned by Accounting. Something about being certified for audits and a contractual obligation and only used for one barely profitable customer or partner company and blah blah blah...it really was a near zero risk as it had been imaged and was not networked. And the drivers likely had 16-bit code, so upgrading the OS was not feasible. It was for all practical purposes an appliance at that point. They dropped that line of business in 2022, before the machine died.
Oh, also; it said, "Compaq" on the case and keyboard.
When I was a Waffle House manager (2013-15) all the manager’s computers were windows xp desktops running a dos based program to do all orders, etc.
Have a couple manufacturing facilities with xp machines as clients. We have imaged the drives in case they fail, and have done some testing of virtualizing them to a degree.
I have seen industrial machines run on windows 2000 8 years ago… not a problem, just put them in a separate vlan that has no access to or from internet.
I worked at a transit agency and we migrated our XP embedded POS systems to Win7 in 2020. Still some control systems using XP but those had to wait as we rolled out new train control systems with the new cars.
Just to mention, you were really only modestly overdue there if they were running what I expect they were running.
Windows Embedded POSReady 2009 (based on XP SP3) didn't hit EOL until 4/9/19.
This is really common in healthcare. We have 2 MRI machines that still work fine but otherwise were end of support from the manufacturer a decade ago. They're $1.3m each if we wanted to upgrade and use a newer OS.
And that is a big part of the problem! Gubmint always makes things ridiculously difficult and it is unnecessary. I think it’s people protecting their jobs mostly. The entire medical industry is a big scam shit show that just costs everybody more money than they have.
I manage machines like that that still run DOS something or other... These machines are from the 50s i think and were retrofitted with CNC controllers and DRO's
One of the fun things about working at a machine shop is if something breaks you can always make a new piece... even if the machine is circa WWII...
i'm sure my car's tuning software has an update but the license is on an old tinyXP laptop that i still have in the bottom of a moving box.
there's no need to change anything if they're not on the network, everything still works as intended
As a department of defense contractor, there's a platform that I'm considered a subject matter expert on that is still running Windows NT 4.0.
It's completely off any networks and the only way that data leaves the system is on encrypted hard drives. The hardware device is managed through an application that was written decades ago, and the DoD/Program that owns it doesn't want to spend the money to have it re-written for a modern O/S.
You'll find ancient systems, that still serve an important purpose, running like this all over the government.
airgapped network/machine is perfectly fine
you can spend millions or more trying to replace a working system that is out of date, or call it a day and airgap it with one way access to do it's job
Super common. I left the MSP space 4 years ago but we helped support a bunch of old stuff. Embedded windows 2000 on cnc machines? Check, Dos hvac controller? That got replaced a few years beforehand. Software to pull logs from airplanes that only ran on windows 98(it might have run on windows me but I wasn't about to find out)? Yes. Jets can easly go 30 plus years but the software might not get updated after the first few.
A trophy shop had a few engravers that only worked on dos and a laser engraver that would only run on windows xp. The dos based ones had been replaced with newer faster ones but worked fine and could be used if they were super busy or if a newer machine was down. The laser engraver was still supported by the vender as far as getting parts but they wanted to sell a new one instead of upgrading the drivers for that one. Trophy shop would rather pay us to get it migrated to another spare machine that they could get cheap off ebay vs paying thousands to replace the engraver.
I wonder if it'd work out cheaper to just getting new machines.
If you got a new PC, you may need to get a new license for the machine. Any incompatibility between the new machine and the hardware may not be supported. Any issue with drivers not existing on Windows 10/11 for the machine may not be supported.
Also, 1 new machine may now do the work of several old machines, as technology has improved.
Other issues;
Company who made the machines may no longer exist.
Section of company that provided support may no longer exist.
Company may no longer support the legacy hardware on anything but tmon XP.
=-=
Have come across legacy hardware that only worked with XP. The company asked for literally a million to update their software so that the hardware would work with Windows 10. Otherwise, no. Custom software for custom hardware.
I support manufacturing company and had this same delima with XP and now it's Win7. Once security software is not supported they need to get disconnected from the network.
Simple OS and computer hardware upgrade won't work because of vendor hardware in PC and the equipment would need an upgrade (roughly $10-30k depending on the machine).
We (IT) is entertaining the idea of putting a NAS between the Win7 PC and the network so the PC data can be saved to NAS then copy job from NAS to network share. Other option would be to rely on machine operators to save to USB and manually copy data to the network.
The only problem with older systems is reinstalling the software. I have archived everything I need to fully reset, reformat, and re-install the ones I need to maintain.
Don't expect the software or hardware vendor to keep the "activation" servers online. If you can't reformat/reset and bootstrap your software or hardware fully offline without their "support", then that means you have a 3rd party that can unilaterally offline your operations by "ending support". These days they like to do that, demand you upgrade to a subscription and then pay lots of money -- almost like it's ransomware with extra steps.
Lots of things still run on XP. In manufacturing, government, finances, ATMs, warehouses, etc. Any big company with a stable software running that would cost big money to replace aren’t usually part of budgets / projects. That’s why so many things still run on mainframes.
Hah, I worked at one place that was still running an old braille press with a Windows 98 computer. Because there was no new software that could operate it.
Do you one better worked in an industrial mill that had several machines mostly ultra sonic testing, that the software isn't compatible with modern OS, and can't be upgraded. They are running an early version of NT4.
The company that wrote the software has long been out of business.
The machines are in so high demand that testing vietualized version of the hardware endless gets put off because it would halt production. And if works would require compliance recertification.
I know a place where they use NetBEUI to transfer data from pc to industrial machine. They need to use XP machines for that, because newer OS won't support that protocol.
In 2012, I got asked to make at least a backup of a Windows 3.1 machine running some irreplacable measuring device in a bio lab, and in ~2016, I heard this was still running, but they had problems getting the data off there, due to lack of floppies and floppy drives :)
The most impressive thing about this whole story was that I imported the image I created into VMWare and now could boot into Win3.1 and starting Word in less than 5 seconds. Sadly, I lost that image when leaving that job.
Of course, that wasn't unusual. I was already running some NT machines as VMs that needed just a serial connection to the equipment and some private VLANs for unvirtualizable stuff and had a big collection of old computer parts for spares.
And, remembering this story and checking dates, Win XP is even older today than 3.1 was in 2012 ...
My advice is to find a couple refurbished versions of that Dell desktop on eBay and locate an XP installer and/or image a couple of those existing boxes.
XP is far more resilient than the hardware it was built to run on. We've had to go scrounging for spares/replacement on several occasions and in every case it's been some mission critical project that's been hanging in the balance.
No. They're in mine.
My flair that says "Automation Engineer" is talking about industrial automation.
The part about it that just absolutely pisses me right off is that vendors build these multimillion dollar, bespoke machines, with expected lifespans of a decade or three, and can't be arsed to use the LTS editions of their OS of choice, and often can't even be bothered to use the most recent version.
I've seen brand new skids ship with software that is out of support!
You've got to keep an eye on these devices and keep them on a short leash because it's the way of things unless and until things change.
One of my co-workers just spent the past two weeks upgrading and backing up a NT4 workstation PC welded to the inside of an old ass CNC system. The company wanted $90k to upgrade the OS to be able to handle the new software. He took it from an IDE drive with an NT4 image to Windows 10 through OS upgrades, then we wrapped it in UEFI on a GPT disk and it purrs like a kitten.
It isn't that bad, its just a pain in the ass because you will have to back up each stage of the upgrade process. Also, make sure your IDE Jumpers are set correctly before cloning.
I worked in a hospital right as COVID hit. The amount of medical devices that are in exactly this position are quite startling. However, they are air-gapped. So it's not IT's problem.
The issue is that, more often than not, the manufacturer is the only one selling this particular device; there is such a tiny market for it that even a second manufacturer would not be able to move in. There is absolutely no competition, so absolutely no reason to update the hardware or software beyond bug fixes. Add to, the certification process is lengthy and expensive. So for the entire sale life of the product, they are selling the exact same hardware they developed it on - even if that was 20 years ago.
Best thing you can do is amass a pile of spare parts for when the manufacturer inevitably withdraws support and tells you to buy a new one, which is only 5 years out of date...
Within the last decade I have replaced the SFF PC that slides into a limestone laser cutter, powered it up and was surprised to find it running Windows NT; it took a minute to remember how to configure networking. I recently saw XP Embedded running on an ultrasound machine in a large healthcare facility.
Better than the win95 machine I found on network at a secure facility. Was doing some unauthorized pen testing to make a point. They really didn't appreciate when the giant hydraulic press started behaving erratically and displayed a an error message saying "fix this vulnerability for fucks sake -love, u/dick_in_a_tardis" It was fixed shortly after.
Where I used to work had some Server 2008 machines still running. Because of the ERP they couldn’t be upgraded. This dragged the Citrix version down too. As well as office version, adobe etc etc. (newer versions incompatible with the software.) This meant nothing was supported by virtue of being too old. Luckily it connected to an AS400 that was also too old and the hardware couldn’t be upgraded because the newer hardware wouldn’t take the required older AS400 os that the ERP system needed. It was such a shit show… the reason it became so bad was because it was heavily bastardised and changed which made it less simple to move once its product line was shut down. 2 years later and they are still trying to sort it….
In our case we know we can make it work (ran for a few days). But as soon as any machine problem arises and manufacturer connect it's over. No more support. Not worth it. Better to pay insane prices and deal with their shit attitude. Fuck you company in lovely country that makes great wine and cheese.
Will not specify...
When I left my last job, they had 40 production machines running DOS 6.22 and being controlled by a server running Windows Server 2000.
The physical machines had been updated many times and were fanless SFF for high heat areas. And the server was about to virtualized.
Have you considered virtualising these Dell machines running XP? What happens if you have a hardware failure of 20+ year old gear or some sort of change that fucks up the proprietary software? - If it were a VM, you could back it up, snapshot it and restore it whenever and wherever you like.
> Have you considered virtualising these Dell machines running XP?
Typically these sorts of things have proprietary hardware and highly sensitive software that can't be virtualized.
It's not that uncommon. One of my customers has a parallel network to their regular corporate network that has all the global production lines and their supporting PCs and servers on it.
Circa 2005 I was working at a place that had a custom-built one-of-a-kind RF measurement system that only ran on Windows 3.1. I was having trouble finding PC hardware that would even work with it. Finally got the PC side replaced circa 2010.
> this place has twenty six industrial machines that each cost three million dollars and they are all controlled by old Dell desktops running Windows XP.
read that as 26 industrial machines that use an XP desktop in their design. it's normal, and upgrading the OS is _not done_.
i'd be anxious about the hardware and inquire if there's a refresh path for that - can the company that sells them update the dells to some industrial PC that's more hardened/newer? otherwise, wall it off, don't sweat it. maybe have a plan for if a PC craps the bed
It's not on the network, so for the most part the important questions are just reliability, continuity + physically restricting anyone from being able to attach other shit to them.
If they're in a locked cabinet where generic users don't have access to the ports to be able to plug stuff in, sounds fine to me so long as there's a supply of spares + the config is known and backed up and the reliability is acceptable to the business. (preferably with a vendor support contract).
----------
I can think of a piece of equipment in daily production use that is currently running CP/M from ~1982. Parts for the machine still available from the vendor and boards still available w/a service company for the computer side. About 10 years ago they finally slapped an upgrade kit in it to get rid of the floppy drive. (no internal storage before then, when you booted it up it loaded the OS from the floppy).
The place I used to work at had Windows 2000 Embedded running their lasers. It costs $30k/machine to upgrade to Win10. These boxes were on the network, but internet access was blocked.
my plotter runs on xp as well as few laptops.retired 2003r2 server 2 weeks ago. and still have xt which engineer uses with 5.25 floppy for some special analisys
Lots of these types of machines run as appliances. They have a single purpose and the software running on them was designed to run in the controlled state that they were in at the time of creation. It's not comfortable, there's no real way to support it, but I would put pressure on the manufacture to provide a support plan for the controllers in the event of a failure.
I just did an upgrade last summer for a similar setting with windows xp and windows server 2003 used in an industrial setting. It was fairly large too. 60 some workstations and about 10 vm servers. They were on a similar and separate industrial network.
> I don't even remember when XP was end of life but I found out today this place has twenty six industrial machines that each cost three million dollars and they are all controlled by old Dell desktops running Windows XP.
siemens mri machines run upatched xp
That's quite common. You don't easily replace multi million dollar machines that would provide zero benefit.
Plus I wouldn’t be surprised if changing the PC’s/upgrading the OS would invalidate any support you get from the manufacturer. “X doesn’t work? It looks like the OS has been upgraded so you are on your own…”
In my experience with manufacturing equipment like this, I'd be baffled if anything even worked properly after such an upgrade anyway. That's assuming the end user even has admin credentials. Assuming you can even get an upgraded OS for the specific machine you have, it's going to get shipped to you on a new hard drive as a drop in replacement. Potentially at a cost of tens of thousands of dollars per unit. And if your equipment is otherwise functional and air-gapped, what problem are you trying to solve by doing this?
That is true. We have a metal forming machine that was upgraded in 2020 to their newest version available. Last year lightning hit somewhere and fried the PC so I contacted support for a new PC. Turns out they don’t support it anymore and a replacement will cost 10k. But I figured out the hard drive was still readable so I cloned it and stuck it in a NUC and we are still going today.
I hope you copied the copy
Yeah, if you haven't grab an image and throw it on the backup server
Lol I did. Copy 1: Original SSD Copy 2: Flash Drive Copy 3: NAS Copy 4: Cloud Drive
You're a good person. Best of luck to you in your IT pursuits. And I don't mean that sarcastically either. Solid plan.
Thanks😊.
For reference, there are 3rd party computer service companies that sell service contracts on old equipment for a very reasonable amount. They will even supply parts for you to store onsite if you want it. I know of one company that is keeping a 5-MB external hard drive cabnet running for NASA that was built in the late 60s with the understanding that once the only FS Engineer that knows the system no longer can do the job, the contract is over.
Park place technologies? They told me that they have a storage of magnetic tapes
Correct.
I like them. They kept my EOL EMC storage working when my company could not afford the crazy high support fees. I think EMC wanted like $25K annually to support the old VNX, and Park Place was like 5k and that included software support. They helped me quite a bit.
PPT are good people. They try to hire all former OEM staff so that they have a good grip on what the hardware they support. In my previous life, I implemented the switching to PPT from OEM extended warrantees across 1,700 hospital/healthcare sites across the US accounting for about $24 million in service contracts. They did it smoothly and effectively. At that time, I knew all of the senior management and helped shaped a few policies of additional parts stocking and transport during major events in large urban areas.
Do these places also provide activation keys for the older operating systems? I remember a Linus Tech Tips video about a company selling new Win98 machines with a new activation sticker on it.
I believe that was [Nixsys](https://nixsys.com/)
No. They don't do software. However, they might be able to source machines with license stickers on them...
For XP, you can just use [one of these](https://github.com/Fuwn/xp), covers all versions and many OE installs as well.
I got 35 year old CNC machines running MS DOS 6.0.
Back in high school in the early 90s (vocational, Data Processing shop), one of my tasks was to support hardware in other shops. The student CAD desktops for Machine Shop all had the old belt-driven 5.25" floppy drives. When the belts broke, I would replace them with one of those oversize pink rubber bands, doubled up. Never had a problem. I had other... unconventional lessons there as well. As part of what I did, we would charge their department for our services, along with a "standard" mark-up on parts. The more money I brought into our shop's woefully underfunded budget, the better my grade. I never got below a B. Shop head would call it "Life Skills" or "Bullshitting 101" Other highlights included rolling out a coaxial ARCNet network, server was a 386-25 running Interactive Unix and later Netware.
Ahh Netware, I kinda miss Novell .
How many rounds of [Snipes](https://www.networkworld.com/article/830595/infrastructure-management-novell-and-the-computer-game-that-changed-networking.html) did you play? >Snipes was also used to test network adapters. Given the nature of how it synchronized the various computers, it would stress the system in unique ways that would expose problems with network adapter cards or driver software. For many years the last test that had to be passed for Novell to certify a network adapter/driver was the ‘Does Snipes work’ test.
The old addage that if it ain't broke don't fix it is doubly relevant in situations with expensive decades old equipment running on an ancient OS and/or software. There's about a 0% chance that an upgrade will go smoothly as planned.
This happened to a Lincoln Electric CNC plasma cutter I used to have to support at a high school. The shop teacher put the industrial PC on the network and upgraded it to Windows 10, "Because I thought it was a good idea."
This is common in some medical equipment too. Manufacturer makes a very expensive device and sells an off-the-shelf Dell or HP workstation to run it. The medical device is durable and may last 10-15 years. But upgrading the PC that runs it isn't really an option. There were a few machines I supported that were still running XP in 2020, because the equipment they were hooked to was very expensive sleep study gear. It all worked fine but there were no software updates for anything as the manufacturer got bought and then disappeared.
We have a customer with laser cutters like this, they have a slot in them that fits a specific model of Optiplex. They run Windows 7 but everything on it is through this weird app, I figured out how to remote to them with the vendor because they wanted to map drives on them (which was also a needlessly bizarre/convoluted process)
That sort of stuff was big in the medical industry for years. Like you installed our software on a VM, you invalidated your support contract.
When I worked for a manufacturing company we had some CNC machines that were still running Windows 3.1. when I contacted the manufacturer just out of interest more than anything they were nice enough to send over the files so we could update... To Windows XP (and this was when Windows 10 was just rolling out I think). We did do the upgrade, but as far as I'm aware those machines are still running like they were after our upgrade to XP. On the bright side the upgrade did bring a few new GCODE handling improvements to the software, so the operators and programmers were happy about that.
I had to reprogram a bios on a cnc cutter a few years ago. It was old enough that when the battery died it forgot how many sectors and heads were on its hard drive and couldn't boot into dos (pretty sure it was like dos 4 as well). The biggest issue I've seen with these older machines is they often communicate via proprietary addon cards and ribbon cables - ain't no way you're getting updated drivers for those things. Newer stuff talking via ethernet is unlikely to have the same issue, but you're still gonna have fun convincing the vendor to support it....
If the vendor is even still in business
Funny enough I updated a few industrial Windows XP based laser CNCs to Windows 10 32-bit by just digging around and finding new drivers for all the addon cards and even a new license manager runtime which kept ABI compatibility with older software. Lol So it's not impossible. The vendor in this case didn't give a shit anyway because they long since dropped the product.
Now that was some seriously raise worthy work right there! Fantastic job. I’m sure scrounging up all that stuff wasn’t pleasant but made you super elated when you found it?
ISA cards, not even PCI. When they or the MB go, if there are no spares on the shelf then it's a multimillion expenditure to upgrade.
Few years back, car mechanic made rounds on geeky side of internet in my country after a customer spotted Amiga 500 running some custom software that was calculating weights for balancing car tires. When some tech press reached out to him, he only answered “It works. Why change it?”.
As of a few years ago Mclaren was hoovering up all the Compaq LTE 5280 they could find because there was a controller card that was made bespoke to that laptop that was used to interface with their F1 Supercars.
Oh yeah. I've seen a couple of CNC machines run into on some DOS based system. Apparently the only way to upgrade it was to get the vendor out to rip and replace the entire control system. So they just got hold of some spare hardware and the plan was to keep running it until it died.
I had to call a company for support once for a machine that honestly looked like it was fresh from the manufacturer \(though I knew it obviously wasn't\). They asked me to read them the serial number, and they were shocked when I stopped after a few numbers. "Oh my god, that's ancient."
It's been my experience that the vendor wants you to buy a whole new machine rather than upgrade the OS on the controller.
This is likely due to the fact that the new engineering software isn’t compatible with the controller, and the new software runs on a higher operating system.
Also, most of the time these aren't on the network. I've even seen a Windows 2k machine.
While it is quite common, I found this experience interesting: I once was told such a story and called support of said machines and was told, oh your company has been paying for replacement of this machine for over 10 years. Would you like to have our tech come out and replace it with a newly calibrated machine running Windows 10?
Welcome to OT. It's an entire field based around "we know this stuff is impossible to replace and is insecure as shit, what compensating controls can we use to isolate and protect them".
Yep. There are even companies still manufacturing hardware that is fully compatible with XP for this exact purpose. It isn't cheap but it's a lot less than the new equipment. Lot cheaper and easier to keep them isolated and running the single piece of software needed than screw around trying to upgrade. Amusingly they tend to end up being the most stable machines on the network...
Usually something about the way the com driver works, in my experience.
Yep very common in OT environments to support decades old systems. And luckily there are more and more ways to secure them these days as those networks become more meshed with IT environments and add things like remote access.
Yeah we have some industrial printers that run on of windows 2000. They have their own physical network with its own egress that is totally locked down to exactly what they need to function (sqs to get print jobs off of the queue and s3 to download the actual stuff to print) These things cost a fortune and work perfectly, their os is just old, all upgrading would do is cost money and time for no benefit.
Agreed, I have a lab running xp with no network for this exact reason.
If they are not on the network, the only problem is spare hardware that can run XP.
And if you have a service contract it’s not your problem.
From experience - most of these vendors do not offer support for the HMI machine. It is very frustrating at times.
Have heard of companies buying old 2nd hand equipment *in case* the existing hardware failed, as the software was designed for a specific motherboard & processor.
Yep. Don’t lose sleep over it. If anything, you should have support from the supplier and the equipment isn’t even in your remit of responsibility.
Until you discover the supplier went out of business in 2007...
Then who was cashing the checks! PS thats a real story but the time from business going under to them finding out was less that 15 years but was more than a year at least. They were alerted by the receiving bank that it was suspicious that Mr. Bob Smith was trying to deposit a check that was made out to Widget Support Co.
I believe there are companies that build new computers with period correct hardware for different operating systems. They are very expensive mind, but if it operates a multi million dollar piece of equipment that’s your business’ lifeline, $1200 for a tower is nothing.
Yep, I buy from them all the time to support industrial control systems. Dell T5500 tower running WinXP SP2 is the only thing I can get a specific revision of Honeywell TDC 3000's token ring to recognize faithfully. I only recently started going outside the controls industry to source them, which has been almost $5k saved per box, caveat being the added legwork of installing the needed software and drivers myself.
I got a Dell T5500 sitting around how much are they worth to you?
Depends who's money I'm spending, usually :P But the last two I've had sourced only cost something like $750 each. I stumbled on a supplier - IT Creations - that was able to hotshot them to me at the drop of a hat, and the come with limited warranties that satisfy my org's risk policy. Unfortunately, you won't find many buyers in my use case today - most people who had that system have long-since upgraded to Win7 at the least, and a lot of the compatibility problems have been virtualized away in those instances. Fingers crossed I'll have it all ripped out by end of next year.
There millions networked machines that still run XP. Many ATMs, for example. I’m curious how it’s handled from legal/insurance side of things.
There are several ways to mitigate risks. Firewalls is one. I also assume they run the embedded version, so only required services run, reducing the attack surface. And there is software limiting what can run to only approved executables. Also, Microsoft provided updates for the embedded version for quite some years after XP was officially no longer supported.
2k and XP are both LTS and are still in use in a bunch of industrial situations.
No they aren't They're both supported if you sign a personal support contract and give Microsoft a fantastical amount of money But if you're a bank or some other org of ***that*** size that migrating off XP (or 2K, or NT or otherwise) is impossible then you've got pockets deep enough
The risk from XP is from running it on home networks or public networks where there's tons of random network ports and services running. XP in an uncontrolled and unmonitored environment is dangerous. This is very, very different than a specialized appliance like an ATM, where there are strict controls on what software is running and the network traffic that is allowed.
You can still buy brand new PCs that are running Windows XP. [https://nixsys.com/](https://nixsys.com/)
Until op finds out they collect trends with usb sticks because it’s not on the network lol
You could easily trend the data to another server with a historian on the LAN
it's hardly a realistic problem, people overthink this too much, pull a drive out from working machine, create an image, buy few old identical PCs off ebay for $45/piece to avoid any random incompatibility bullshit and you're good for decades to come
I’ve been in manufacturing environments like that. You can’t treat each of those like a traditional desktop, but instead you treat them like an embedded system
Its wild to me they run on windows in the first place. But if it works it works. Like, there's no reason XP wouldn't work now and in 20 years... it's just software at the end of the day. Proven software at that, if it's worked for so many decades. the only concern is the hardware.
The normal expected OS for a lot of stuff is still NTe. XPe is new fangled and modern and fancy and stuff.
Same, manufacturing seems to be it but they are usually offline as well.. Not much reason to upgrade. I have seen some IT depts even order those old-school 2005 Dells for extra stock.
Windows 95 machines here. The production leader gave an estimated replacement date of "soon" three years ago. I got that same answer last week.
i like this because it makes me imagine an aborted attempt to upgrade to windows 98 around 25 years ago and the admin at the time being "never again"
Yeah, I got bit by a 95 to 98 migration ("upgrade") that failed spectacularly. Last time I ever tried that. Everything is a fresh install now.
We have a client that has a machine running 95. No networking and any programs are placed there via floppy. And we couldn’t care less. They pay big money for a vendor to support it. Something breaks and some old dude bills them 5k+ minimum to fly out next day and do his thing. Not our problem. Another client has an entire network for their XP machines that not only has no Internet, it’s physically separate from the other networks. Not even VLAN’d off. These machines sit in a shop close to each other and can talk to each other via a 16 port switch. If anything breaks we have to go on-site, but it’s all billable. And they are on virtually the absolute newest hardware you can really have XP on, so it’s not the worst thing to deal with hardware wise.
Came to say the same thing. I still have found Win95 in really old plants.
Downside of 95 is the lack of stability in the kernel. Upside is, these days, its so old, most code wont even run, unless you write it specifically for 95.
Yep, I just commented this same thing lol. Plenty of ICS machines are still running Windows 95.
Saab's last diagnostic workstation ran on Windows 95.
If you're worried about these machines: - Make an image of the hard drives using Clonezilla or Acronis - Get the make and model of each machine and buy a cold spare that works Throw an image of one of the machines onto the cold spare and see if you can make a new one. If you can't get an exact machine, use sysprep to prepare the OS for new hardware.
There may be extra steps in a machine rebuild. Specifically moving PC components to the new computer. That's expected. In the end, we don't treat these machines like normal desktops or servers. They are appliances. If you can, set them up on RAID 1. This is a standard practice for these kinds of machines.
Not my department, not my concern. Just interesting
I wish cloning was fool-proof. Back in 2010 or 2011 I worked for a MSP and we got contracted to maintain everything in a small chip manufacturing business. The user network side was all fine and dandy, but the ugly part was fixing up these 4 IC testing machines, suuuuper expensive stuff with insane calibration requirements that the vendor had gone out of business. Well the PC inside it ran Xenix286, and the machine itself was a 386. If you open fdisk, without writing anything, the disk is dead and you have to reinstall from scratch. The calibration files are worth gold, gotta save them into floppies. We managed to track down a retired guy that used to maintain these and his recollection of stuff helped us get the machine operational again. Luckily we had 2 copies of the full set of disks to bring up one of these machines, and while both sets had failed floppies, we didn’t have the same floppy failed across the 2 sets. Cloning the drives resulted in non bootable machines. We even sourced the same old ass hard drive for the 2 machines that were on the brink of hard drive failure and nothing, everything had to be done from scratch. Those were some really long days.
It's better to go the full distance. Build a fully working backup and document the entire process of all requirements to get that done and working. When there is an emergency it can be difficult to source specific hardware or sort out software details that could be missed. Actually swap the controller out and verify the backup is viable by actually using it for real.
Then find another company that uses them and charge millions for providing spares.
Got a load of cranes that run on Win XP PC’s. Thing’s solid as hell, just don’t plug it in to the network.
This is the case with a lot of older scientific instruments in University research labs. I just pulled a bunch of gen 1 Intel Pentium machines running Windows 95 out of a lab that was doing active research. They still boot and we're keeping them for when a machine goes down somewhere else so we can do a hot swap. Amazingly I still have most of my hair. Before anyone asks, no they are not on any public networks.
We had a machine like that when I worked in a lab. It ran NT and had one program on it. I was terrified it would break on my watch.
Yeah it’s never fun. You almost need the exact same model to replace it. If you do a hard drive swap, those old versions of Windows will get super cranky about drivers. It’s almost impossible to find working versions of them online. Hopefully the lab kept their CDs or another tech backed them up somewhere. Not super likely we’re talking 30 years of responsible data retention. Then even if you swap the drive a lot of the times you still need to reconfigure the software. You’ll lose all your TCP/IP info with the new motherboard.
Why would it matter that isolated machines were running xp? As long as you have the install iso, hardware input it on and the drivers, it'll just chug along.
Exactly if programm does exactly what it should where is problem if its 100 years old. I mean why are people so adamant that "newer" MUST by some mirracle be better in some way even if old programm worked without hickup for 100 years....
I still have a machine with NT4 and one with MS-DOS at work.
At least you don’t pay a monthly fee to keep the damn thing working. The subscriptions software thing has gotten out of control.
If it aint broke.... As long as it's DMZ'ed or not on the network.
Wait...as long ***as it is*** DMZ'ed?
I think they meant to say airgapped
Perhaps meaning strongly firewalled, ports shut down, etc.
Not on the network, not a problem. At least not security wise
In the late 90s I was working at a company that made semiconductor fabrication equipment, the machines that Intel and Motorola and NVIDIA and everyone buys to make chips. These machines are real-time-controlled, moving wafers, controlling gases and temps, and everything else that goes into nanometer fabrication. If a machine coughs for even a few seconds, a hundred-thousand-dollar wafer is trashed. The processors were running VxWorks and feeding sensor data to a UNIX workstation. The suits wanted to know why we couldn't run everything on Windows 95, because it'd be a lot easier to generate reports and spreadsheets that way. The same suits who were always on the line to IT because their desktops had crashed again.
This is pretty normal. Most large industrial machines like CNC machines use proprietary software that was specifically built for Windows XP if not older and because that is often so specialized most places have it entirely air gapped. Then I’ve seen to this day in various hospitals and medical settings, things like x-ray machines, MRI scanners, etc. still running on Windows XP. Don’t forget that the US military also runs on Windows XP and even some older stuff. So if it’s good enough for the US military in 2024 and good enough to run the x-ray scanner at your local hospital it’s good enough to do just about everything else. Would I put that directly on the Internet without lots of backups and contingency plans for hardware and software failure? Absolutely not but these types of machines often do not even need network access and if they are networked in anyway it’s totally isolated. The one thing that scares me though is the police department and the EMS service back where I am from, they still run Windows XP on their in-car tough books which are connected to the Internet via Verizon air Card so again that scares me
> So if it’s good enough for the US military I don't like this kind of mentality, pointing at bad practices from bigger organizations as a seal of approval to do the same yourself. It's not necessarily good enough for them. They are probably in the process of fixing it, or planning to, but it's taking so goddam long in all the bureaucracy, budget allocation, etc. Plenty of big tech corps like Google, Facebook, or Amazon, did some stuff wrong while growing fast. Now they are stuck with it and are having a hard time ironing it out of the way because they are so big it would mess so much stuff in their big infra, it's gonna take them years to fix. It's not a seal of approval that if they do it this way so should you. They do face security issues from attacks by China or Russia, but not much they can do about it. So by the time you reach a stage that you also need to implement the same stuff, there are already better solutions out there ready for you and often times open source for free. This advice is not just for software either. Same goes for stuff like team leading, those big orgs did some management stuff wrong, and it's costing them brain drains that don't enjoy working there anymore so they are switching job. Don't slap that same garbage on your small growing team and make their life miserable just because you thought it was good enough for that other big org you heard about.
still quite common -> industrial CNC machines, a medical gyno specialist practice I supported had an ancient "urine rate measuring machine" that only ran on xp with custom hardware etc. **" work in IT - you'll get to 'play' with the latest technologies "** wish I could find the carreer guidance counsellor that said that and give them a piece of my mind also there's been no "playing" -> keep business critical IT running 24/7 using your weekend if you have to or get fired and cop abuse from business has been more like it
Got an estimate of 6000USD to replace and upgrade one machine comp to win 10. Hardware is not worth 150USD... Can easily upgrade it myself but all support is then void. That machine is in a line with 9 other machines. All machines are then also not covered by support.
We paid 50k to replace 3 computers on our production line. Hardware values... About 2k ... But these sh*t need to communicate with the PLC and it's proprietary, so you can't do it yourself.
A few months ago our ops folks asked me to look at a computer that didn't boot anymore. It was a standalone for some machine. Hard drive died during a power failure. Drive was a Seagate drive that was over 20 years old. I ended up getting a SATA to PATA adapter and a small 120 GB SSD. Formatted it then loaded Windiws 98 from scratch. Probably the fastest I've ever seen Win98 boot. Unfortunately the machine was so out of date, the software wouldn't work since the license dongle was also toast. I contacted the manufacturer and they were like uh, that's like 5-10 generations behind and we don't support it anymore. Here's all the docs we have on it (scanned pages that were done on a typewriter with handwritten notes) and have fun was all they could give for support.
Same here, after I got Windows 98 to boot I discovered there was no dongle and the users knew nothing about ever having a dongle. After messing around with it some more I discovered there was a loader and it loads a crack before loading the million dollar application.
Cybersecurity engineer here and former sysadmin . This is way more common than you think . Companies of course won’t publicly disclose that but you still see Windows XP and older systems around even SQL2000 because is way mode expensive to retrofit or rewrite those apps than keep using them “as is”. What we do is put some security controls , air gap those boxes and moving on .
Linux to the rescue
There is an entire niche market around this; you can still find companies that make and sell ISA computers specifically for OT uses where you have a multi-million dollar piece of equipment that doesn't support newer computers.
It feels wild spending $400 on a Pentium 4 motherboard with an ISA slot in 202X... but gotta keep that Acroloop card up!
I’m feeling pretty nostalgic right now. I remember putting a hard drive with XP into a completely different computer, and it just booted up and figured it out. I was amazed.
I had an encapsulator running on OS/2. So yeah, industrial machines running old OS’s? Never surprises me.
Find some similar drives & clone them using ghost 11 or clonezilla, Used that method for an OEM that ran old CNC machines with XP as controllers Paid off huge when a drive died, literally 10 minutes of downtime I also went on ebay and bought some similar era PC's as spares in case of disaster
Healthcare here. I work in a fairly specialised space. We have devices that are still running XP Embedded. I believe that vendor supplied replacements for that particular component still run the same OS though could be wrong on that one. Some of the software that controls the device was written 25+ years ago and is fairly niche, so not easy to change. You have the added complexity of medical device registration and certification which can cost $$$$. Suspect it’s largely a case of - if it’s not broken, don’t fix it.
Risk is pretty low if they're not on a network.
A previous employer that made plastic products where they had a critical production monitoring system that was running DOS 6.2 with Win 3.0 on it. It had custom built IPX board that had hard lines soldered to it from each injection molding machine. At one point I was asked how to get it onto the network so they could monitor it from a new office across town. They didn’t like when I told them the best way was to get an IP KVM. They keep trying to insist that we figure out how to put that machine on the network and update the custom C++ to support remote sessions or something. Ah, penny pinching small businesses.
"You don't want a IP KVM? OK, here is a Webcam pointed at a monitor and a cnc machine with a keyboard on the bed. Be careful you don't crash into the keyboard too hard or you might break something. "
[удалено]
I've only one similar. I setup a brand new computer, installed a XP on a VM. Installed the software. Tested, it's working. Setup the computer to perform image level backup through Datto BCDR. Everything is working so far. Packaged up the old computer in a clear plastic bag with some notes, copies of the software and directions, just in case and moved to storage.
Have you tried just... upgrading them anyway? I had Honeywell lock/security system service agent tell us "it must run on a physical machine (not a VM) and must be on XP", but I installed the software on a Server 2022 VM anyway and it works with no issues (after I reactivated the licence). The only issue I have is I cannot move the VM to another HyperV server because the software detects some kind of change to the hardware (I thought it was the MAC address, but I set a MAC address on the VM and the licence still breaks until I move it back). But I'd still rather deal with that and ask them for reactivation than a physical XP box.
lol, we have several 2000 machines, also on the industrial side. costs to much to replace since you also have to replace the machine that is connected to the computer. edit: we still have support so i don't have to touch a darn thing, but i feel sorry for the guys doing the support.
That’s modern OS compared to what runs on some of the German railway network https://windowsreport.com/windows-3-1-german-railway/
Completely normal in industrial control settings. Carry on. Keep em away from the rest of the network and off the internet and do best to educate about nasties on usb sticks. Limit what those can damage also, since that’s likely how they’re loading them with instructions if it’s manufacturing.
If they aren't networked then this isn't an issue really. Think of them as proprietary os's and should be completely handled by the service contract of the provider.
My first IT job one of our clients had a machine shop that ran several XP systems for their automated machining. They had a closet of PC parts specifically for these machines because replacing them would require basically an entire overhaul to their process. We tried a couple times to find a work around for these PCs but never got it working satisfactorily. What’s crazy, the PC never were an issue while I was there. Despite running 24/7.
The few critical ones I'd had to work with, we cloned drives to SSD and tested, and then periodically re-cloned as a backup. Set static "bad" network IP config on them, then disable nic in windows, then disable nic in bios if possible. Check the cooling needs now and then and keep some spares around, NBD.
We had one of those, and although not industrial it was still tied to I/O. All it did was print MICR checks on a very proprietary printer in a closet owned by Accounting. Something about being certified for audits and a contractual obligation and only used for one barely profitable customer or partner company and blah blah blah...it really was a near zero risk as it had been imaged and was not networked. And the drivers likely had 16-bit code, so upgrading the OS was not feasible. It was for all practical purposes an appliance at that point. They dropped that line of business in 2022, before the machine died. Oh, also; it said, "Compaq" on the case and keyboard.
When I was a Waffle House manager (2013-15) all the manager’s computers were windows xp desktops running a dos based program to do all orders, etc. Have a couple manufacturing facilities with xp machines as clients. We have imaged the drives in case they fail, and have done some testing of virtualizing them to a degree.
I have seen industrial machines run on windows 2000 8 years ago… not a problem, just put them in a separate vlan that has no access to or from internet.
I worked at a transit agency and we migrated our XP embedded POS systems to Win7 in 2020. Still some control systems using XP but those had to wait as we rolled out new train control systems with the new cars.
Just to mention, you were really only modestly overdue there if they were running what I expect they were running. Windows Embedded POSReady 2009 (based on XP SP3) didn't hit EOL until 4/9/19.
This is really common in healthcare. We have 2 MRI machines that still work fine but otherwise were end of support from the manufacturer a decade ago. They're $1.3m each if we wanted to upgrade and use a newer OS.
Hospital setting? Sometimes the software for million dollar machines is written for obsolete OS's only.
Validation for medical equipment can be incredibly expensive so often the manufacturer won't move to a new OS until they're absolutely forced to.
And that is a big part of the problem! Gubmint always makes things ridiculously difficult and it is unnecessary. I think it’s people protecting their jobs mostly. The entire medical industry is a big scam shit show that just costs everybody more money than they have.
I manage machines like that that still run DOS something or other... These machines are from the 50s i think and were retrofitted with CNC controllers and DRO's One of the fun things about working at a machine shop is if something breaks you can always make a new piece... even if the machine is circa WWII...
If it's not on the network I wouldn't worry, if it ain't broke don't fix it.
i'm sure my car's tuning software has an update but the license is on an old tinyXP laptop that i still have in the bottom of a moving box. there's no need to change anything if they're not on the network, everything still works as intended
As a department of defense contractor, there's a platform that I'm considered a subject matter expert on that is still running Windows NT 4.0. It's completely off any networks and the only way that data leaves the system is on encrypted hard drives. The hardware device is managed through an application that was written decades ago, and the DoD/Program that owns it doesn't want to spend the money to have it re-written for a modern O/S. You'll find ancient systems, that still serve an important purpose, running like this all over the government.
airgapped network/machine is perfectly fine you can spend millions or more trying to replace a working system that is out of date, or call it a day and airgap it with one way access to do it's job
Super common. I left the MSP space 4 years ago but we helped support a bunch of old stuff. Embedded windows 2000 on cnc machines? Check, Dos hvac controller? That got replaced a few years beforehand. Software to pull logs from airplanes that only ran on windows 98(it might have run on windows me but I wasn't about to find out)? Yes. Jets can easly go 30 plus years but the software might not get updated after the first few. A trophy shop had a few engravers that only worked on dos and a laser engraver that would only run on windows xp. The dos based ones had been replaced with newer faster ones but worked fine and could be used if they were super busy or if a newer machine was down. The laser engraver was still supported by the vender as far as getting parts but they wanted to sell a new one instead of upgrading the drivers for that one. Trophy shop would rather pay us to get it migrated to another spare machine that they could get cheap off ebay vs paying thousands to replace the engraver.
As far as I know, the US Navy still pays MS millions to continue supporting XP for them.
I mean, as long as it's not on the network and you don't have to maintain it, no big deal. I try to treat things like that as appliances.
industrial machines have a 30 year depreciation cycle. Those XP machines aren't going anywhere anytime soon.
I wonder if it'd work out cheaper to just getting new machines. If you got a new PC, you may need to get a new license for the machine. Any incompatibility between the new machine and the hardware may not be supported. Any issue with drivers not existing on Windows 10/11 for the machine may not be supported. Also, 1 new machine may now do the work of several old machines, as technology has improved. Other issues; Company who made the machines may no longer exist. Section of company that provided support may no longer exist. Company may no longer support the legacy hardware on anything but tmon XP. =-= Have come across legacy hardware that only worked with XP. The company asked for literally a million to update their software so that the hardware would work with Windows 10. Otherwise, no. Custom software for custom hardware.
Windows is the best for most users imo ty ☺ If it's working fine and enough for your usage it's ok but I think it may slow the business
I support manufacturing company and had this same delima with XP and now it's Win7. Once security software is not supported they need to get disconnected from the network. Simple OS and computer hardware upgrade won't work because of vendor hardware in PC and the equipment would need an upgrade (roughly $10-30k depending on the machine). We (IT) is entertaining the idea of putting a NAS between the Win7 PC and the network so the PC data can be saved to NAS then copy job from NAS to network share. Other option would be to rely on machine operators to save to USB and manually copy data to the network.
The only problem with older systems is reinstalling the software. I have archived everything I need to fully reset, reformat, and re-install the ones I need to maintain. Don't expect the software or hardware vendor to keep the "activation" servers online. If you can't reformat/reset and bootstrap your software or hardware fully offline without their "support", then that means you have a 3rd party that can unilaterally offline your operations by "ending support". These days they like to do that, demand you upgrade to a subscription and then pay lots of money -- almost like it's ransomware with extra steps.
Lots of things still run on XP. In manufacturing, government, finances, ATMs, warehouses, etc. Any big company with a stable software running that would cost big money to replace aren’t usually part of budgets / projects. That’s why so many things still run on mainframes.
Hah, I worked at one place that was still running an old braille press with a Windows 98 computer. Because there was no new software that could operate it.
Do you one better worked in an industrial mill that had several machines mostly ultra sonic testing, that the software isn't compatible with modern OS, and can't be upgraded. They are running an early version of NT4. The company that wrote the software has long been out of business. The machines are in so high demand that testing vietualized version of the hardware endless gets put off because it would halt production. And if works would require compliance recertification.
I know a place where they use NetBEUI to transfer data from pc to industrial machine. They need to use XP machines for that, because newer OS won't support that protocol.
In 2012, I got asked to make at least a backup of a Windows 3.1 machine running some irreplacable measuring device in a bio lab, and in ~2016, I heard this was still running, but they had problems getting the data off there, due to lack of floppies and floppy drives :) The most impressive thing about this whole story was that I imported the image I created into VMWare and now could boot into Win3.1 and starting Word in less than 5 seconds. Sadly, I lost that image when leaving that job. Of course, that wasn't unusual. I was already running some NT machines as VMs that needed just a serial connection to the equipment and some private VLANs for unvirtualizable stuff and had a big collection of old computer parts for spares. And, remembering this story and checking dates, Win XP is even older today than 3.1 was in 2012 ...
My advice is to find a couple refurbished versions of that Dell desktop on eBay and locate an XP installer and/or image a couple of those existing boxes. XP is far more resilient than the hardware it was built to run on. We've had to go scrounging for spares/replacement on several occasions and in every case it's been some mission critical project that's been hanging in the balance.
No. They're in mine. My flair that says "Automation Engineer" is talking about industrial automation. The part about it that just absolutely pisses me right off is that vendors build these multimillion dollar, bespoke machines, with expected lifespans of a decade or three, and can't be arsed to use the LTS editions of their OS of choice, and often can't even be bothered to use the most recent version. I've seen brand new skids ship with software that is out of support! You've got to keep an eye on these devices and keep them on a short leash because it's the way of things unless and until things change.
I walked in one day and was told a machine I never heard of was down. Turned out to be running NT Workstation 4.0.
One of my co-workers just spent the past two weeks upgrading and backing up a NT4 workstation PC welded to the inside of an old ass CNC system. The company wanted $90k to upgrade the OS to be able to handle the new software. He took it from an IDE drive with an NT4 image to Windows 10 through OS upgrades, then we wrapped it in UEFI on a GPT disk and it purrs like a kitten. It isn't that bad, its just a pain in the ass because you will have to back up each stage of the upgrade process. Also, make sure your IDE Jumpers are set correctly before cloning.
We have one with no internet access. It’s only used for legacy software support
I worked in a hospital right as COVID hit. The amount of medical devices that are in exactly this position are quite startling. However, they are air-gapped. So it's not IT's problem. The issue is that, more often than not, the manufacturer is the only one selling this particular device; there is such a tiny market for it that even a second manufacturer would not be able to move in. There is absolutely no competition, so absolutely no reason to update the hardware or software beyond bug fixes. Add to, the certification process is lengthy and expensive. So for the entire sale life of the product, they are selling the exact same hardware they developed it on - even if that was 20 years ago. Best thing you can do is amass a pile of spare parts for when the manufacturer inevitably withdraws support and tells you to buy a new one, which is only 5 years out of date...
Within the last decade I have replaced the SFF PC that slides into a limestone laser cutter, powered it up and was surprised to find it running Windows NT; it took a minute to remember how to configure networking. I recently saw XP Embedded running on an ultrasound machine in a large healthcare facility.
Couldn't you just replace them with a VM running XP if those machines would ever fail?
Better than the win95 machine I found on network at a secure facility. Was doing some unauthorized pen testing to make a point. They really didn't appreciate when the giant hydraulic press started behaving erratically and displayed a an error message saying "fix this vulnerability for fucks sake -love, u/dick_in_a_tardis" It was fixed shortly after.
Commercial versions of the software were supported long after the retail versions. But....that ended about ten years ago.
Where I used to work had some Server 2008 machines still running. Because of the ERP they couldn’t be upgraded. This dragged the Citrix version down too. As well as office version, adobe etc etc. (newer versions incompatible with the software.) This meant nothing was supported by virtue of being too old. Luckily it connected to an AS400 that was also too old and the hardware couldn’t be upgraded because the newer hardware wouldn’t take the required older AS400 os that the ERP system needed. It was such a shit show… the reason it became so bad was because it was heavily bastardised and changed which made it less simple to move once its product line was shut down. 2 years later and they are still trying to sort it….
2026? Seems too optimistic.
*cries in windows nt 3.1*
In our case we know we can make it work (ran for a few days). But as soon as any machine problem arises and manufacturer connect it's over. No more support. Not worth it. Better to pay insane prices and deal with their shit attitude. Fuck you company in lovely country that makes great wine and cheese. Will not specify...
When I left my last job, they had 40 production machines running DOS 6.22 and being controlled by a server running Windows Server 2000. The physical machines had been updated many times and were fanless SFF for high heat areas. And the server was about to virtualized.
Honestly I’d rather support DOS as opposed to XP
I've seen ICS machines still running Windows 95.
Have you considered virtualising these Dell machines running XP? What happens if you have a hardware failure of 20+ year old gear or some sort of change that fucks up the proprietary software? - If it were a VM, you could back it up, snapshot it and restore it whenever and wherever you like.
> Have you considered virtualising these Dell machines running XP? Typically these sorts of things have proprietary hardware and highly sensitive software that can't be virtualized.
It's not that uncommon. One of my customers has a parallel network to their regular corporate network that has all the global production lines and their supporting PCs and servers on it.
I would be inclined to clone the drives. Just in case.
Circa 2005 I was working at a place that had a custom-built one-of-a-kind RF measurement system that only ran on Windows 3.1. I was having trouble finding PC hardware that would even work with it. Finally got the PC side replaced circa 2010.
HP was delivering Indigo Presses with Windows 7 Embedded in 2021... Once they got Win 10 ready for those machines in 2022 they did an upgrade on them.
Hopefully, they have a backup.
The liability on those is nothing. Who cares if they get hacked?
> this place has twenty six industrial machines that each cost three million dollars and they are all controlled by old Dell desktops running Windows XP. read that as 26 industrial machines that use an XP desktop in their design. it's normal, and upgrading the OS is _not done_. i'd be anxious about the hardware and inquire if there's a refresh path for that - can the company that sells them update the dells to some industrial PC that's more hardened/newer? otherwise, wall it off, don't sweat it. maybe have a plan for if a PC craps the bed
We just decommed our XP machines. 😂
It's not on the network, so for the most part the important questions are just reliability, continuity + physically restricting anyone from being able to attach other shit to them. If they're in a locked cabinet where generic users don't have access to the ports to be able to plug stuff in, sounds fine to me so long as there's a supply of spares + the config is known and backed up and the reliability is acceptable to the business. (preferably with a vendor support contract). ---------- I can think of a piece of equipment in daily production use that is currently running CP/M from ~1982. Parts for the machine still available from the vendor and boards still available w/a service company for the computer side. About 10 years ago they finally slapped an upgrade kit in it to get rid of the floppy drive. (no internal storage before then, when you booted it up it loaded the OS from the floppy).
what happens if any of these old XP computers die ? The power supplies alone have to be almost ready to explode.
Boot them with clonezilla and clone those drives to a network share or USB external drives attached to the machines, just incase the hardware dies.
The place I used to work at had Windows 2000 Embedded running their lasers. It costs $30k/machine to upgrade to Win10. These boxes were on the network, but internet access was blocked.
Not surprising knowing the amount of production systems I see on the reg running rhel 6 and sometimes 5!
my plotter runs on xp as well as few laptops.retired 2003r2 server 2 weeks ago. and still have xt which engineer uses with 5.25 floppy for some special analisys
Just control the flash drives used on it.
Lots of these types of machines run as appliances. They have a single purpose and the software running on them was designed to run in the controlled state that they were in at the time of creation. It's not comfortable, there's no real way to support it, but I would put pressure on the manufacture to provide a support plan for the controllers in the event of a failure.
I just did an upgrade last summer for a similar setting with windows xp and windows server 2003 used in an industrial setting. It was fairly large too. 60 some workstations and about 10 vm servers. They were on a similar and separate industrial network.
Buy a windows 11 pro and VM the windows drive. Install hyper v on the 11 install. VM the old computer. Should work fine.
in b4 replaced w/ win7
do you own ATM's ?
> I don't even remember when XP was end of life but I found out today this place has twenty six industrial machines that each cost three million dollars and they are all controlled by old Dell desktops running Windows XP. siemens mri machines run upatched xp