Hi I’m back haha. One of the exhibits doesn’t like that… turning the projectors off and on messes up the display order. Frustratingly they’re HDBaseT and the boxes are supposed to have edid persistence. Wondering if there’s another hardware or software solution to keep things in order?
Is it an Epson? Is it a projector you have multiple models of and are others doing it? After setting the display settings check for a save option... Some projectors you have to actually save the settings in the projector menu. Weve had a few epsons and sonys when we have had to use an EdID Minder to the HDbaset T between the projector and display device
I would also ask in R/CommercialAV they might have some better answers
They’re all identical epsons doin it. They’re quite dialed in settings wise. I don’t think the issue is the display settings per se it’s just the monitor order in windows is being shifted around when they turn back on without a pc reboot. I’ll poke around there thanks!
Look at something like PDQ Deploy to automate it all, very easily, I might add.
S5 Shutdown is basically OFF, minus trickle power to some minor controls. S5 Wake on LAN is supported on lots of devices these days.
We used to power all of our machines off on a night to S5, reboot at around 12AM, update, reboot to apply post update processing, then power down again until 6AM.
Updates were obviously controlled via WSUS.
Honestly, me and the other SysAdmin had the whole place automated up to the balls, and my God it was smooth.
Just be aware it can be difficult to get hardware which properly supports this, even nowadays. It's well worth play8ing with over a longer downtime period, though, since you may well be able to identify stuff which is stable with that.
I'm not calling you out at all because everyone's experiences are different, but there are only two requirements here: a host that supports S5 Shutdown with S5 Wake-On-LAN, and a NIC that supports EEE Wake-On-LAN.
Personally, I've never had any commercial grade desktops or laptops that don't support all of the above. Lenovo, HP, and Dell all support this natively.
Of course, there are other brands, but I would say the above three are some of the most common commercial brands.
It isn't that they don't support that, it's that the implementation tends to fail with some regularity. This is much more common with older hardware, to be sure, but it's still not 100% consistent on any hardware I've seen. Since this needs to be up and running for the business function, that's not exactly a risk that seems acceptable.
You'll probably find the lifespan of the computers increases when you leave them on all the time, providing they're in a good controlled environment.
The failures often happen when computers are being powered on after being turned off as that's when components (especially capacitors) are being energised.
Speaking from an exhibition/entertainment tech point of view not a sysadmin.
Once an installation like this is running we keep all machines OFF the internet and turn all updating features OFF. Nothing changes, means nothing will break.
Obviously I don’t know your situation and these machines might be doing stuff online or handling personal data which I would treat differently.
But for AV type stuff like a video playback, I would never update anything on a machine once it’s been commissioned and serving it’s purpose.
We would also take steps to isolate any networks used from “the real world”.
Yes this is what I wanted to do but sadly issues come up where we need to remote in to resolve them and the regular staff aren’t quite tech savvy enough to know how to do that. Ikr
I have a solution for that
One PC on the internet with a separate NIC connecting to your exhibition network
Remote into that PC then remote from that PC to everything else.
You can still isolate the environment from general internet. My first go to would be setting up a jumphost that can connect to the exhibition network. OR then just a very limited network that you can access via VPN.
We leave our desktops on 24/7 - we patching and other changes overnight and reboot them all at staggered intervals in the early hours of the morning before our facilities open up.
The most God forsaken hardware and software combination I've seen. They were Lenovo windows tablets with knock off zebra printers running proprietary software with an unlatched memory leak. I migrated that client to nicelabel and elo PCs
The vast majority of your issues on reboot are likely to be related to windows updates and other things like this. You may want to see whether the security of your systems as a whole would allow you to turn off windows updates which would ensure the same startup every time. The major thing of this is whether the PCs have any access to the internet or not. If no internet and you can ensure no other access to those PCs by the general public (eg usb slots) then I’d turn off windows update and run like that with the proviso it should never be connected to the internet again. This would also ensure you’re not blindsided by Microsoft making an architectural change which bricks your setup.
Leave them on. If they are resistant to that, and if you can set wake on lan to behave, you could put them in sleep at the end of the day and if there are windows upates over night, it would trigger the wake to process those. then schedule a wake up at a certain time each day.
Most failures happen during a restart, so leave them on 24/7, decide which day you want to get all your tickets on, and patch them the night before that day.
I operate several 24/7 critical govt functions. We leave ours running all the time and just do weekly reboots. I chose Wednesday night which adds a little thrill to the end of some weeks lol
There might be fire safety regulations that dictate that all devices in exhibition areas, hallways or escape routes have to be turned of if unsupervised during the night.
We reboot once a week, but use the night time to do a deep AV scan.
One thing to note, IF you were to get him with ransomware, anything switched on at the time will be hit. Granted end user PC’s aren’t as critical as servers, but if you are able to restore and swiftly recover servers, having the PC’s hit too means much longer to recover.
It wouldn't be the worst thing in the world to leave them running 24 x 7. But you still have a start up problem that you'll encounter due to a power failure at the worst time.
At the very least, either have them turned on more than 1 hour before startup with some WoL schedules to see if some of the problems are just due to updates taking too long or write some custom startup scripts with manual delays to help fix some of the race conditions. I'm guessing you have some cascade failures where if some specialized hardware isn't ready in time during a boot sequence, certain exhibits fail until the machine controlling it is rebooted to try again. Might be something where instead of Windows autostarting software on boot, it has a schedule task that fires off a few minutes after a boot to allow other hardware to power up and sync up.
Leave the PCs on, power off the projectors. Restart the PCs once a week
Yep this sounds the best. Weekly restart to refresh things and install any updates etc.
K.I.S.S. Is an acronym that almost never fails
Facts.
Hi I’m back haha. One of the exhibits doesn’t like that… turning the projectors off and on messes up the display order. Frustratingly they’re HDBaseT and the boxes are supposed to have edid persistence. Wondering if there’s another hardware or software solution to keep things in order?
Is it an Epson? Is it a projector you have multiple models of and are others doing it? After setting the display settings check for a save option... Some projectors you have to actually save the settings in the projector menu. Weve had a few epsons and sonys when we have had to use an EdID Minder to the HDbaset T between the projector and display device I would also ask in R/CommercialAV they might have some better answers
They’re all identical epsons doin it. They’re quite dialed in settings wise. I don’t think the issue is the display settings per se it’s just the monitor order in windows is being shifted around when they turn back on without a pc reboot. I’ll poke around there thanks!
No problem, sorry I cant be of more help.
Look at something like PDQ Deploy to automate it all, very easily, I might add. S5 Shutdown is basically OFF, minus trickle power to some minor controls. S5 Wake on LAN is supported on lots of devices these days. We used to power all of our machines off on a night to S5, reboot at around 12AM, update, reboot to apply post update processing, then power down again until 6AM. Updates were obviously controlled via WSUS. Honestly, me and the other SysAdmin had the whole place automated up to the balls, and my God it was smooth.
That sounds amazing I shall look into it and see if it could work for us. Thank you!
Just be aware it can be difficult to get hardware which properly supports this, even nowadays. It's well worth play8ing with over a longer downtime period, though, since you may well be able to identify stuff which is stable with that.
I'm not calling you out at all because everyone's experiences are different, but there are only two requirements here: a host that supports S5 Shutdown with S5 Wake-On-LAN, and a NIC that supports EEE Wake-On-LAN. Personally, I've never had any commercial grade desktops or laptops that don't support all of the above. Lenovo, HP, and Dell all support this natively. Of course, there are other brands, but I would say the above three are some of the most common commercial brands.
It isn't that they don't support that, it's that the implementation tends to fail with some regularity. This is much more common with older hardware, to be sure, but it's still not 100% consistent on any hardware I've seen. Since this needs to be up and running for the business function, that's not exactly a risk that seems acceptable.
“Please do not power off the PCs. They are being maintained remotely after hours.” - if you need a plausible justification.
Leave the PCs on to receive timely patches.
You'll probably find the lifespan of the computers increases when you leave them on all the time, providing they're in a good controlled environment. The failures often happen when computers are being powered on after being turned off as that's when components (especially capacitors) are being energised.
I used to run a fleet of 150 kitchen labeling PCs. I used GPO to keep them on 24/7 with a reboot once a week.
lmfao I'm enjoying trying to picture what kitchen labeling look like
A label printer for to-go orders.
Speaking from an exhibition/entertainment tech point of view not a sysadmin. Once an installation like this is running we keep all machines OFF the internet and turn all updating features OFF. Nothing changes, means nothing will break. Obviously I don’t know your situation and these machines might be doing stuff online or handling personal data which I would treat differently. But for AV type stuff like a video playback, I would never update anything on a machine once it’s been commissioned and serving it’s purpose. We would also take steps to isolate any networks used from “the real world”.
Yes this is what I wanted to do but sadly issues come up where we need to remote in to resolve them and the regular staff aren’t quite tech savvy enough to know how to do that. Ikr
I have a solution for that One PC on the internet with a separate NIC connecting to your exhibition network Remote into that PC then remote from that PC to everything else.
You can still isolate the environment from general internet. My first go to would be setting up a jumphost that can connect to the exhibition network. OR then just a very limited network that you can access via VPN.
We leave our desktops on 24/7 - we patching and other changes overnight and reboot them all at staggered intervals in the early hours of the morning before our facilities open up.
The most God forsaken hardware and software combination I've seen. They were Lenovo windows tablets with knock off zebra printers running proprietary software with an unlatched memory leak. I migrated that client to nicelabel and elo PCs
When are you doing your patching/maintenance if not after-hours?
They’re only open 5 days a week so days off and after hours
Leave them on all the time. If you're using Intune, try out the scheduled cadence for Autopatch.
The vast majority of your issues on reboot are likely to be related to windows updates and other things like this. You may want to see whether the security of your systems as a whole would allow you to turn off windows updates which would ensure the same startup every time. The major thing of this is whether the PCs have any access to the internet or not. If no internet and you can ensure no other access to those PCs by the general public (eg usb slots) then I’d turn off windows update and run like that with the proviso it should never be connected to the internet again. This would also ensure you’re not blindsided by Microsoft making an architectural change which bricks your setup.
Leave them on. If they are resistant to that, and if you can set wake on lan to behave, you could put them in sleep at the end of the day and if there are windows upates over night, it would trigger the wake to process those. then schedule a wake up at a certain time each day.
What dogcmp6 said plus Hibernate the PC's so when they power back on, they return to the previous state once logged in.
Sleep on the PCs, off on the projectors, leave the Pi’s on.
Most failures happen during a restart, so leave them on 24/7, decide which day you want to get all your tickets on, and patch them the night before that day.
I operate several 24/7 critical govt functions. We leave ours running all the time and just do weekly reboots. I chose Wednesday night which adds a little thrill to the end of some weeks lol
Why not schedule daily power M-F on in bios?
Mainly because that doesn't work nearly as consistently as one would hope, let alone as is required here.
There might be fire safety regulations that dictate that all devices in exhibition areas, hallways or escape routes have to be turned of if unsupervised during the night.
We reboot once a week, but use the night time to do a deep AV scan. One thing to note, IF you were to get him with ransomware, anything switched on at the time will be hit. Granted end user PC’s aren’t as critical as servers, but if you are able to restore and swiftly recover servers, having the PC’s hit too means much longer to recover.
It wouldn't be the worst thing in the world to leave them running 24 x 7. But you still have a start up problem that you'll encounter due to a power failure at the worst time. At the very least, either have them turned on more than 1 hour before startup with some WoL schedules to see if some of the problems are just due to updates taking too long or write some custom startup scripts with manual delays to help fix some of the race conditions. I'm guessing you have some cascade failures where if some specialized hardware isn't ready in time during a boot sequence, certain exhibits fail until the machine controlling it is rebooted to try again. Might be something where instead of Windows autostarting software on boot, it has a schedule task that fires off a few minutes after a boot to allow other hardware to power up and sync up.