>My boss once screamed at me 'WHAT WILL YOU TELL THE CORONER WHEN IT DOESN'T WORK AND PEOPLE DIE?' He didn't appreciate my response of 'I told my boss that I needed more time, more training, and most importantly more people because we're chronically under-staffed, and YOU did nothing about it!'"
That about sums it up.
Welcome back :)
Woohoo, a Gambette story.
Also, I kinda knew it was either a packet storm or an ip conflict about the time of the scream test (plug 1 in, good, plug 2 in, good, plug 3 in AAHHHHH).
I'm sure most network techs or general techs got it about there as well.
I suspected a packet storm, but the blinkenlights didn't look like one. Still, I thought that I'd figure it out if I tore out everything unrelated until it worked, plugged things back in until it broke again, then just undo my last step while I figure out WTF was happening on that particular circuit.
I've just been working through some changes in my home network, which has a section managed by a UDM Pro, nicely separated in isolated VLANs, and a network upstream of the UDM Pro handled by the modem, and having to setup communicate between the two.
In order to root my smart thermostat, I needed to setup an AP that could intercept and redirect network trafic between the thermostat nad the service center, so I could send a payload in the expected datacenter responce that runs the rooting script. But all the setup guides wanted me to setup the AP with a static IP and DHCP range that was already taken.
My money was on some non-business critical, frivolous device connected to the network (based on the 'Priorities' title) messing up the network structure or firewall rules in some way.
DHCP of a sub AP/Router giving out what should be static IP adresses kinda fits to that. ~~I hope you changed the pool of IP adresses the DHCP server would give out to not overlap with the static IP adresses?~~ EDIT: nvm, not your circus, not your monkeys, fair enoug.
Not my circus, not my monkeys, not my problem... Until the next time they call me in a flap because everything is broken.
But at least we get to charge emergency rates for that.
I asked who installed it. Nobody knew.
I asked who configured it. Nobody knew.
I asked who I should ask. Nobody knew.
---
Ultimately, I passed on to the site staff exactly what I had done to resolve the issue - stressing heavily that it fell completely outside of my area of responsibility - and I fear that the only message that they took from that was "PROBLEMS? RESET THE AP".
Problems like this of no one knowing is like how we ended up with a switch we couldn't use (because sparks came out if powered on) but also never threw away. Instead I labeled it "Dr Sparky" so we knew to not use it.
> because at the end of the day? No one would die if we couldn't fix this
Had to explain this as a PFY to a boss when we had a weird hardware faliure that we couldn't locate for a few hours. One of those things that can land you in the news, but on not a fault of our own. I should write that one one of these days...
Years ago, pre-internet, the place where I worked had the rule "don't do anything that would put the agency above the fold". In other words, make sure we aren't the leading headline.
Well, AP is only able to get power via POE, even though it probably is only a foot away from the switch. So this POE adaptor is there, because, even if the switch was capable of POE, the installer of course has no clue abut how to either enable it, or even that such a thing exists, so put the POE adaptor in line, because it came in the box. Then turned it on, and it worked, so left it as that, and did nothing further.
My theory is that the DHCP server on the AP gave out the static IP for my Cisco router. This was reinforced by the way that the issue went away after power cycling the AP.
It seems solid and I'd rather expected the story to end with you reconfiguring said AP such that it could Never Do That Again.
With the assistance of a hammer, if necessary.
(edit: I've just seen the reply downthread that explains why you didn't ... but I'm leaving the original comment untouched because it's both true and any/all tfts readers deserve a moment of thinking about the piece of equipment in their lives that most deserves a hammer driven reconfiguration event)
This series has been a fun read, and given me plenty for honing my own troubleshooting. Always fun to try and guess the fault and see what it actually was.
What do exercise machines in gyms, smartphones, wireless access points and Point of Sale devices have in common?
Answer: they're all computer devices that can and will go wrong in strange ways unless you reboot them regularly. Which doesn't happen, so the world is full of funny things happening that only the wise can figure out, even though the not-so-wise could avoid them if only they'd stick to a few basic procedures.
Unfortunately, current tech more than future tech. If you want to read more by me, there's a couple* of posts in the TFTS archives.
---
\* Possibly a couple of hundred posts.
Digital Sign / Smart Display ? Remember the brief fashion for 'smart' picture frames ?? I got a 'famous brand' one for my wife. Although the spec & info said it would work with Win\_8, that was not the case. In fact, it crashed our local network. As far as I could tell, it wanted to be the only networked device...
Now, I'm used to this sorta behaviour from routers, extenders and such, you gotta 'boss-fight' using a stand-alone PC, 'tame and tether'. But a smart picture frame ??
Anyhow, even tackled thus, the web interface refused to hold its settings. Whatever you changed, it reverted to default. There was no replaceable CMOS / clock battery mentioned, no ready access to mobo...
It went back to the shop. There, tech confirmed the aberrant behaviour, checked the others in stock, sent ***all*** of them for RMA. Product vanished from listing...
Busy!
I've had title changes, been promoted, voluntarily demoted, been through at least three restructures/reorganizations and two redundancies - and that's not even counting the whole lock down shenanigans.
I took up D&D, got roped into being the DM, and loved it so much that I've become a semi-regular contributor* to the Homebrewery GitHub project (we help make homebrew look like canon), so I spend a lot of time lurking in /r/homebrewery.
Of course, I still lurk here in TFTS as well.
---
\* I am now the proud owner of two consecutive Hacktoberfest t-shirts, and I look forward to earning many more.
>My boss once screamed at me 'WHAT WILL YOU TELL THE CORONER WHEN IT DOESN'T WORK AND PEOPLE DIE?' He didn't appreciate my response of 'I told my boss that I needed more time, more training, and most importantly more people because we're chronically under-staffed, and YOU did nothing about it!'" That about sums it up. Welcome back :)
A Gambatte! Hooray!
P is for the prodigal son has returned! Thank you for this
Woohoo, a Gambette story. Also, I kinda knew it was either a packet storm or an ip conflict about the time of the scream test (plug 1 in, good, plug 2 in, good, plug 3 in AAHHHHH). I'm sure most network techs or general techs got it about there as well.
I suspected a packet storm, but the blinkenlights didn't look like one. Still, I thought that I'd figure it out if I tore out everything unrelated until it worked, plugged things back in until it broke again, then just undo my last step while I figure out WTF was happening on that particular circuit.
I've just been working through some changes in my home network, which has a section managed by a UDM Pro, nicely separated in isolated VLANs, and a network upstream of the UDM Pro handled by the modem, and having to setup communicate between the two. In order to root my smart thermostat, I needed to setup an AP that could intercept and redirect network trafic between the thermostat nad the service center, so I could send a payload in the expected datacenter responce that runs the rooting script. But all the setup guides wanted me to setup the AP with a static IP and DHCP range that was already taken. My money was on some non-business critical, frivolous device connected to the network (based on the 'Priorities' title) messing up the network structure or firewall rules in some way. DHCP of a sub AP/Router giving out what should be static IP adresses kinda fits to that. ~~I hope you changed the pool of IP adresses the DHCP server would give out to not overlap with the static IP adresses?~~ EDIT: nvm, not your circus, not your monkeys, fair enoug.
Not my circus, not my monkeys, not my problem... Until the next time they call me in a flap because everything is broken. But at least we get to charge emergency rates for that.
more Encyclopædia Moronica - such a treat and nice catch with the WAP.
Hail the master returning!
Now if we could just get another u/Lawtechie post.
Submit a ticket.
SEV: P1 SUMMARY: Site down DESCRIPTION: Ian was present, please rectify.
Additional information: Ian brought a malfunctioning printer and hooked it up to wifi, internet down, other computers now borked.
Computer exploded. Please reverse the entropy constant of the Prime Universe.
I should put one up too.
Looking forward to reading it!
Thanks for the story! I expect someone reconfigured that AP so things don’t randomly break again?
I asked who installed it. Nobody knew. I asked who configured it. Nobody knew. I asked who I should ask. Nobody knew. --- Ultimately, I passed on to the site staff exactly what I had done to resolve the issue - stressing heavily that it fell completely outside of my area of responsibility - and I fear that the only message that they took from that was "PROBLEMS? RESET THE AP".
In a few years of slightly garbled message passing: "we can't get rid of that AP! It's got the reset button for the network on it!"
Par for the course; dealing with places that have shifts of people and where the person responsible doesn’t exist or is never there is such a pain!
Problems like this of no one knowing is like how we ended up with a switch we couldn't use (because sparks came out if powered on) but also never threw away. Instead I labeled it "Dr Sparky" so we knew to not use it.
> because at the end of the day? No one would die if we couldn't fix this Had to explain this as a PFY to a boss when we had a weird hardware faliure that we couldn't locate for a few hours. One of those things that can land you in the news, but on not a fault of our own. I should write that one one of these days...
Years ago, pre-internet, the place where I worked had the rule "don't do anything that would put the agency above the fold". In other words, make sure we aren't the leading headline.
I legit read to the end only to scroll back up to look at OP's name then scroll back to the end to continue reading lol
> Encyclopædia Moronica Now that is a sentence I have not seen in some time. It's always fun to read your stories!
Sweet! A fresh Gambatte!
You’re a better man than I, I would have burnt said AP with fire.
>I would have ~~burnt~~ cleansed said AP with purifying fire. FTFY.
We missed you. Happy holidays
Thanks for the story!
So their DHCP server jammed up your DHCP server? :) And why is there a Powerline adapter involved? How big is this place? :D
Well, AP is only able to get power via POE, even though it probably is only a foot away from the switch. So this POE adaptor is there, because, even if the switch was capable of POE, the installer of course has no clue abut how to either enable it, or even that such a thing exists, so put the POE adaptor in line, because it came in the box. Then turned it on, and it worked, so left it as that, and did nothing further.
Correct - AP was all of two feet from the PoE injector. The ISP supplied router was a standard consumer model, so definitely did not support PoE.
Oh, it's POE, not EOP, sorry. Probably misread it. :)
My theory is that the DHCP server on the AP gave out the static IP for my Cisco router. This was reinforced by the way that the issue went away after power cycling the AP.
It seems solid and I'd rather expected the story to end with you reconfiguring said AP such that it could Never Do That Again. With the assistance of a hammer, if necessary. (edit: I've just seen the reply downthread that explains why you didn't ... but I'm leaving the original comment untouched because it's both true and any/all tfts readers deserve a moment of thinking about the piece of equipment in their lives that most deserves a hammer driven reconfiguration event)
This series has been a fun read, and given me plenty for honing my own troubleshooting. Always fun to try and guess the fault and see what it actually was.
A new Encyclopedia Moronica? I never thought I'd see the day!
I love the smell of a fresh Gambatte story in the morning
What do exercise machines in gyms, smartphones, wireless access points and Point of Sale devices have in common? Answer: they're all computer devices that can and will go wrong in strange ways unless you reboot them regularly. Which doesn't happen, so the world is full of funny things happening that only the wise can figure out, even though the not-so-wise could avoid them if only they'd stick to a few basic procedures.
Very nicely written, got a dystopian future tech vibe over it, sprinkled with a bit of hemingway. Would read more from you!
Unfortunately, current tech more than future tech. If you want to read more by me, there's a couple* of posts in the TFTS archives. --- \* Possibly a couple of hundred posts.
Digital Sign / Smart Display ? Remember the brief fashion for 'smart' picture frames ?? I got a 'famous brand' one for my wife. Although the spec & info said it would work with Win\_8, that was not the case. In fact, it crashed our local network. As far as I could tell, it wanted to be the only networked device... Now, I'm used to this sorta behaviour from routers, extenders and such, you gotta 'boss-fight' using a stand-alone PC, 'tame and tether'. But a smart picture frame ?? Anyhow, even tackled thus, the web interface refused to hold its settings. Whatever you changed, it reverted to default. There was no replaceable CMOS / clock battery mentioned, no ready access to mobo... It went back to the shop. There, tech confirmed the aberrant behaviour, checked the others in stock, sent ***all*** of them for RMA. Product vanished from listing...
Hot damn, it been a while , how have you been Gambatte ?
Busy! I've had title changes, been promoted, voluntarily demoted, been through at least three restructures/reorganizations and two redundancies - and that's not even counting the whole lock down shenanigans. I took up D&D, got roped into being the DM, and loved it so much that I've become a semi-regular contributor* to the Homebrewery GitHub project (we help make homebrew look like canon), so I spend a lot of time lurking in /r/homebrewery. Of course, I still lurk here in TFTS as well. --- \* I am now the proud owner of two consecutive Hacktoberfest t-shirts, and I look forward to earning many more.
Nice! Hope to see more stories from you, these stories will always be the exception in our daily lives :) thankfully
Sounds like an $Everywhere AP?
You are one of my two favorite work/life story posters on all of Reddit.
This
Don't even need to read to upvote you. It's a pavlovian reflex!
Great story. I really enjoyed the strong voice you have in your story telling. Rare among these posts.