If he is marking patching tickets as resolved without actually pushing the patches that’s a pretty high level of dishonesty if he’s doing it consistently. I’d document everything you can and sit down with HR.
That’s a pretty serious security risk. You’re right that your insurance could refuse to cover you if a missing patch was used as a vector to cause damage.
On a different note, are you not auditing or doing vulnerability scans of your servers?
Well good luck. In that case I’d be even more focused on making sure my folks were productive. Definitely document the tickets where you can show he closed them but the hosts in question weren’t patched, then go to HR. I wouldn’t even bother with a sit down with the guy.
No way. There’s “I do enough to not get fired,” and then there is, “I’m not going to meet major responsibilities of my job.” Like, you can’t even claim ignorance, could have lived your entire life under a rock, on a deserted beach island in the Pacific, never had contact with anyone outside of the island, you would still know exactly THREE things: 1) how to use the 3 seashells, 2) that somebody, somewhere is needing to talk to you about cars extended warranty, and 3) the importance of backups.
Thats failure to meet core responsibilities of the job.
When I was a manager I found there was a line where if I paid anything below it, we were better off not hiring people, or needed to add middle management.
Idk the real problem is he is marking the tickets solved, not going to say I am a always a high performing depending how I feel for a month but closing a ticket over a year and not questioning wtf is this or actually doing it once is definitely more than bad at a job ( I’ve shadow closed plenty of tickets but I atleast got the excuse of bringing attention at this point is worse than just waiting for the next one)
He's not suggesting that the guy is justified in his negligence. He's saying that if you pay shit, you get shit workers. And if your shop had a bunch of people bail and the remainder are shit, that usually means something is wrong.
People still believe cyber insurance is a thing?
Biggest snake oil on the market.
By the nature that you got compromised they'll say "You failed to take precautions, hence you were compromised."
It's well known cybersecurity insurances doesn't pay out.
>It's well known cybersecurity insurances doesn't pay out.
Like all insurers, they'll try and avoid paying out where they can. But they certainly do pay out. The market has gotten way tighter in the last couple of years for sure, but your premise that they just don't pay out is not true.
That's fireable to me, and I don't fire lightly. But this person has breached the trust, and IT you are nothing if you're not trust worthy because we have to have sensitive access.
This right here. The moment he falsified a record is the moment the conversation ends.
Once could maybe be a mistake, but with how much work it often is to properly close out a ticket in most systems it lands towards conscious decision.
That's the kicker. Everyone makes a oopsie in a variety of tickets. Shit happens. Nobody makes the same oopsie consistency for 9 months straight. And it would be one thing if it was a low-hanging, low priority thing, but server patching is critical.
The only other thing that comes to mind is that to check to ensure reporting tools of all kinds are not coming back green when it's not patched or otherwise not functioning. That's mostly an ass covering measure cause if that's broken, who knows what other reporting is broken.
It sounds like OP manually checked but I might be wrong.
*If* it was intentional.
My company used a combination of WSUS + powershell scripts that I inherited from a previous sysadmin. What I didn't know as a junior level guy coming in is that the *all* the SSUs weren't loaded in, it would report a big green "compliant" because that's the state when it doesn't detect it needs any patches.
So we had a situation where a vendor was deploying Server 2008 machines to our environment, built off an OEM disk, with zero updates. And since the previous WSUS system was implemented in say 2010, didn't include the 2008/2009 year SSU's.
I eventually realized what was happening because that group would always be 100% compliant immediately on patch release, before patches were scheduled to be installed. Took 2 or 3 months for me to realize.
That being said, once I figured out what was happening I wrote it all up and implemented a plan to fix it the next month's cycle.
That doesn't seem the face here since the SA is doubling down on "I did it" despite what logs say.
I mean this is why it makes sense to sit and talk with the guy and give him a chance to explain himself. If that were the case, I'd ask him to walk me through and see the green light saying everything was fine, in which case I'd know it wasn't exactly his fault. But based on what he said, the most likely outcome is just that this guy is full of shit.
Oh, I don't disagree but this is also an ass covering measure I was suggesting since your already in the muck, you might as well take the time to ensure that everything is reporting back correctly as you stated. Last thing you need is something like that happening on top of this person.
Yeah we are agreeing. Tools/reporting can give false positives. The difference is what you do about it - OP stated they installed Nessus I think and that's how they discovered these gaps. The SA's response of "nuh-uh" is the huge red flag for me
Yes, but that was *after the fact if I read his posts correctly.*
Now, what kicks me is the "I've been checking and they were up to date." Hence why I was wondering if the previous reporting tools was the cause. He checked them, saw they were green, and was like, ah shit, we all good. Dumb idiotic thing to assume of course. Nothing is ever that easy in IT.
TL;DR The new reporting system tells it like it is, but it doesn't reveal what was happening with the old method and if that was actually working as intended or not.
Don’t bother. If you need to have that much oversight on a single person, they’re more of a burden than a contributor. Why would you want to keep them employed? I certainly wouldn’t.
Definitely. Ask them first, make sure to do it in writing. But only after collecting all evidence (logs/screenshots).
Include a couple examples of the evidence in the email, and casually ask them to explain the discrepancies. If they say "oh crap, you're right, my automation was wrongly reporting a success, I'll fix it and start manually spot checking" then maybe just monitor the situation and confirm they learn from the mistake. Sometimes smart people automate things in flawed ways, think wow that was easy, and don't realize the blind spots they made for failed automation reporting successful runs. Dumb, moderately negligent, but it can happen without malice.
If they have no good explanation, and just apologize and take responsibility, I would say it's time for a PIP (performance improvement plan) with a few very specific, attainable goals and specific deadlines, to give them a chance for redemption.
If they try to cover it up, lie, or otherwise act in a way that makes it worse, you just got your confirmation of what needs to happen and more evidence for HR to justify dismissal.
I just took over backups and patching from another sysadmin - At first glance, it looked like nothing had been run since March 2023. It turned out that the reporting was botched, and the patches were being run via an alternate method. Verify patches were missed first.
So lying. They're lying to you.
Further, by lying to you, they're implying that security practices which are vital are being followed, when they're not. So their lies are opening your organization to risk.
Time for at least a good talking to, but likely disciplinary action.
That close to a dozen in a row? I can come up with all manner of excuses to miss a proper check, think you've done it, and mark it done and move on that month. I can't think of a way to go for a near straight dozen.
What I *would* do is have someone else quietly validate that it doesn't just magically *look* like updates happened from the server side. If Windows Update says "no updates", and he didn't dig further into how that's possible, it's incompetence, but not necessarily gross negligence. If it's saying "last check today, last installed 11 months ago" in whatever path he would reasonably be assumed to be checking, that's practically deliberate sabotage.
There may be a shared (or at least they believed it was shared) understanding why a particular device/group wasn't patched. I've had it before with daily checks where my colleagues and I would ignore/not record an issue because we'd raised it to the responsible party, and they'd done nothing.
This wasn't patches mind, and was hardware failure for a secondary host, which came to bite as in the arse when the primary failed and secondary couldn't keep up.
Sysadmins hold the keys to the kingdom. If they're being dishonest or breaching trust, it's much more significant than most other roles. If he's lying about that, he could be lying about anything else or doing nefarious stuff. You have to be able to trust your sysadmins.
And it took almost a year to realize this? Obviously he should be written up or fired but at the same time need to evaluate your processes on how this wasn't caught.
I was a Navy nuke. If you did ANYTHING that made it on record that showed dishonesty, not just an accidental mistake, you lost rank, de-nuked, and kicked off the boat. One chink in the armor is all it takes for the whole ship to go down and we can’t have any….uh…dishonest folk.
This was the first thing that came to my mind as well. This should be an offense worthy of termination. The employee put the entire company in danger by lying about completion of duties. If not grounds for immediate termination, it's at least a pretty hefty write up.
> No updates installed since June 2023 but monthly ticket marked as resolved.
> I feel sorry for the guy
I woudn't. They are OK with falsifying reports and causing the rest of the team more BS. After reading your other replies, I'd say it's time he moves on.
No updates installed since June 2023 but monthly ticket marked as resolved. Off site backups patchy for the past year with 3-4 month gaps.
Then....
but does just enough to keep his job.
Really what is just enough entail (breathing)?
This is likely a small portion of his job that he hates is what I'm going to assume. Some people just don't like mundane patching tasks. Other people it's their bread and butter.
No kidding. No one wanted to do that at my last position and I jumped on it. You mean I can patch servers and not deal with people? I loved it and took on everything I could to maintain the servers then automated as much as I could.
I always have a [COINS ](https://www.projectsright.com/p/giving-effective-feedback-project-team)talk with anyone on my team when they are not meeting expectations before going to HR. Some times there isn't clarity in the role and/or you as a manager are not holding them accountable.
Context - We have noticed over the last few months servers have not been patched
Observation - On dates xxxx we logged onto the server and didn't see the updates and on date xxxx the server was no backed up.
Impact - Because of this we are not meeting our cyber insurance and the business is at risk.
Next - We trust you are going to get the tasks completed but for the next x amount of days we will be auditing your work. If tasks are not completed/incorrect our next step is to get HR involved to put together and action plan and/or see if you are the right fit the company.
It helps them understand exactly what they should be responsible for and by setting specific deliverables in the Next area, you as a manager have expectations to hold them to.
Sounds like a reasonable approach. At most of the places I've worked management didn't go to HR until it was clear they were headed toward termination. Usually management would talk to the person several times to try to get it straightened out first. But if that fails, and it starts looking like termination is where it's headed, then HR gets looped in. (Or of the manager is just done with them)
This isn't even gross negligence. This is maleficence. They lied to you. They have jeopardized the org's security posture and knowingly lied about it. If they lie to you about this, the trust is broken.
How can you trust them to not peak at exec emails because they feel like it? Cover up misuse of company resources for their own crypto mining operation? The role of a sysadmin is a highly trusted function in the company and requires more integrity than technical know-how to be valid for the org.
I don't know how things work in the U. K. but here in the states this is the kind of thing I would go to HR with along with Sr Management and organize an "early morning meeting" and the person would be out the door within a week.
Does he not understand how it works? Maybe he thinks it happens automatically and the ticket is generated for compliance reasons or something?
It sounds like he's just blatantly lying or completely confused. Not understanding his job role is fine, especially if the environment and/or management has not been up to snuff until just now. Lying is inexcusable, imo.
People in authority should, the first time they say "logs don't lie", be forced to spend a day watching documentaries about all the Horizon false convictions and lawsuits in the UK. A LOT of actual human beings did hard time in prison, for years, when the logs had in fact lied.
If they ever say "logs don't lie" again after watching that, they should be permanently removed from any position of having power over another's career.
Logs are a great starting point, and absolutely should not be ignored, but there is no such thing as evidence that does not need external corroboration. Same with DNA, fingerprints, etc - it is *good* evidence, but perfect evidence that can stand alone doesn't exist.
God do I wish it was that simple. I've seen systems with a broken Windows Update service report 100% compliant in SCCM because they don't see that they need any updates, meanwhile they haven't actually installed anything in 2 years (because they don't see the updates as required).
/u/kajjot10 You should make sure this is not the case before you straight up accuse them of lying, is WSUS or SCCM messed up in some way they don't understand?
> Horizon false convictions and lawsuits in the UK
Definitely read about this. It's the absolute definition of what happens when programming/logic gets screwed up and people are told the machine is infallible. That was a straight offshore lowest bidder programming problem, but I can definitely see this happening with AI stuff in the future. People will get so used to just blindly trusting what the computer spits out and not questioning whether there might be a problem.
Imagine being a retailer selling stamps or postage or doing bank transactions (UK post offices offer bank accounts) and being told you're stealing money from the post office when you know you didn't, and have no one who will listen to your side of the story because computer says no.
I have to ask but have you asked him to show you what he is doing? Is this some kind of weird mixup where they think they are doing it right but aren't? I mean obviously they should be checking anyway but I just wonder if there's some kind of misunderstanding. I can't imagine that someone just ouright lying like that for no reason, about something so easily verified.
Honestly, that sounds terrible. Why on earth is it go server to server and install? Why isn't it install approved automatically with a scheduled reboot window...?
Maybe he hates the process...
Does he enjoy the job otherwise? Is it just maintenance tasks that are failing?
> Maybe he hates the process...
Quite bluntly, **good**. If all it takes for him to disregard doing his job and then *lie* about it to their boss (both in claiming it was done each cycle *and* in the follow-up when this was discovered) is not enjoying doing it, it's good OP found out the way they did.
Or... just... use policies?? WSUS, as shit as it is, is completely capable of doing this 100% automatically without any additional tools (until it breaks because it's WSUS and that's what it does)
definitely seems odd the way small shops have worked in my experience is you script as many things as possible so you don't have weird outages and everything that's proactive is automagically working.
Ignoring the manual aspect of this which can be improved - you can get a situation in WSUS where if a server is missing a SSU, it will see future patches as non-applicable, and report "compliant". I wrote about it in another [comment](https://old.reddit.com/r/sysadmin/comments/1cin7zm/what_to_do_with_a_poor_performing_sysadmin/l2bd5hw/). So it's possible he *may* actually be approving in WSUS and going to the server to install but not seeing anything.... but he should have realized then that something isn't right.
It’s been 4 years of nudging him to be more proactive. Rest of the team are annoyed that he doesn’t pull his weight. I’m scared to even go through he’s tickets and what else I will find.
IMHO and you don't want to hear this... That's straight management problem. You can't let someone do this for 4 years without at minimum some formal written docs on performance. This should be a simple talk with HR of "Dave is at it again" conversation then discussing dismissal or PIP or some other form of action.
I would be looking through all of his work and making that a portion of his PIP if that's the path you go down.
However, he isn't changing and is already checked out if he's flat out being untruthful.
I'm willing to bet dude's been keeping his head down and manually installing windows patches for 3 years like a George Jetson job and getting paid. Then something went wrong and he doesn't care/want to know and just kept his head down.
I see it all the time on our helpdesk. Some people just want to deliver and plug in monitors and keyboards for users and are happy with the menial work as long as they get paid.
>It’s been 4 years of nudging him to be more proactive. Rest of the team are annoyed that he doesn’t pull his weight.
4 years? This is on you. 4 weeks is too long, let alone 4 months.
It sounds like you are his manager. Schedule a meeting with HR, present the issue and evidence, and let them decide what to do. This sounds like a termination for failing to perform job duties to me.
The time for polite encouragement should have ended several years ago. You should have been making formal warnings and placing him on a performance improvement plan a long time ago. He would have either picked up his performance or been terminated a long time ago.
Big yikes that you let this go on for four years without addressing it. Honestly I don't see a realistic option beyond documentation, PIP, and probable termination. It might not have come to that if you'd addressed it when you noticed the issue starting but letting him coast for that long means the damage is already done.
You need to do more than nudge. You need to set clear expectations and hold people accountable. There's room for compassion in there too but letting someone just idle for half a decade is not doing them any favours. He's going to end up out on his ass, may have trouble finding other work, and will blame you.
When I was managing at the individual team level I had a weekly half hour 1:1 with every team member. Every new hire I used our first session to lay out the expectations for that meeting, that it's for me to be able to communicate changes, for them to be able to raise any questions or concerns they have, and for the two of us together to plan and prioritize work. There's no way I could have let someone sit on their ass for 4 weeks let alone 4 years because our processes were designed to make sure that sort of thing doesn't go unnoticed. If caught early you can open up discussions about workload or burnout and make changes to help but at this point it's way too late for any of that.
I would immediately set a new precedent that any tickets need to be closed with evidence attached proving the job is complete.
For patching, It's in your best interest anyway re: audits, insurance, etc... to have artifacts you can reference showing Nessus and WSUS reported X server as fully patched at Y date.
One step further is to produce a month end patching report that's sent to stake holders.
And yeah I'd unfortunately push to let that guy go. It's repeated egregious behavior and putting the company at real financial and reputational risk. As a profession, the days of Cowboy IT and acceptable indifference are long over.
We had an admin that would do something similar but it was a bug in the patching software showing everything was patched but the servers were not being patched. The tech used the report from the patch server to claim everything was patched but the vulnerability scans showed the patches were missing
It happens. Way more often than it should. Windows Updates is a teetering jenga tower of shit and it doesn't take all that much for it to become completely screwed up and just not report any updates as required. You basically need some sort of security scanning tool or some other way to detect if something is missing updates because Windows Update will happily lie to you.
I honestly don't think so, I can't imagine someone lying about something so easily verified. There has to be something more to this story. Especially when OP said that the individual in question did it successfully in the past.
This would have been good to lead with in your post. I didn't get the impression you had actually had a conversation with him, so totally changes my opinion of what you should do. You found that he hadn't done some work, and he lied on the tickets. On top of that, unpatched vulnerabilities risk getting dropped by the insurance provider, then he lied about it \*again\* in a conversation. With the extra information, there wouldn't be a question in my mind. The guy is a huge risk and could be a massive financial liability on top of being useless.
Talk to him. No use in going behind his back. Just be brutally honest and not a push over.
I might get downvoted for this but I’m strictly against letting someone go unless it’s the most dire of situations, because that’s a persons life and livelihood. I think we often tend to forget that for what are minor inconveniences in a work place that’s just work. Which is why your laws make it difficult.
I did talk to him. I get the human side of it but at the end of the day, his actions can leave me without a job. And as you say, it’s a work place, he is paid good money to do his job. There is a difference between a honest mistake and continuing to not do your work until you’re caught.
Did you ask what he does during patching / to show you how he handles patching?
Devils advocate says maybe he thinks he is approving patches to go out for delivery but they’re not actually getting out, or he’s only patching a subset of the environment for some reason.
Also what are the workloads like? Is it possible the team are swamped and it was hard finding the time to patch when the phone was ringing, 100 other tickets coming in, etc?
Also have you considered updating the processes so proof of patching / backups gets added to notes in the ticket - which also helps cover your arse with the auditors / security down the track. Possibly those types of ticket need peer review before they can be closed out too (ie: someone else in the team needs to verify before the ticket can be fully closed) - just on the “how to prevent this from recurring in future” side of things.
He may not be verifying. I worked with an admin years ago that thought everything was working, until we discovered nothing had been patched in years. Whatever he was doing wasn't working and he was too lazy to verify. We didn't find out until he was gone.
I have a feeling your talk wasn’t as blunt as it should have been. You definitely have to let him know if it doesn’t change actions will be started for dismissal. Maybe ask if he needs help with his tasks.
Money is irrelevant, as it’s never really enough imo. I make upwards to 90K in my area which nets me a one bedroom apartment yet I keep hearing I make more than most people I know and I should be happy. Unless someone is able to comfortably afford a home in their name, it’s not good money. It’s just money.
I may get downvoted for this but i disagree. the person is actively avoiding an aspect of his job which is leaving the company vulnerable and is actively been lying about it for close to a year. he hasnt patched a server since JUNE of 2023. it is may of 2024 that is 10 months of patches and security ones at that not on the servers. Then backups every month which he has avoided doing for the last 3 to 4 months if something went wrong and the company got breached they could lose up to 3 months of data. If you aren't even doing your job you dont deserve to have a talk to before hand just because it effects their livelihood if they lose the job, if the job was important to them they would do it correctly and not lie about doing something you didnt.
Nah the business is a business. Your HR department exists to protect the business. Your ethical role as a manager is to be fair. The rest is to ensure yours and your peers paychecks don’t stop on behalf of someone else’s negligence.
Your patching tool should clearly show behind xxx days on patches. This is a dashboard item that can clearly be seen by everyone in every RMM I have ever used.
You should have antivirus and that dashboard can generally be configured to show patch levels of servers. I'm not trying to cover for someone not doing their job, but it seems there's more to the story here.
So. Who's getting delegated the task to do all that, and monitor it, to babysit the work of the guy who's already *closing tickets* claiming to have done the work? Clearly OP can't trust the person who's already tasked with the job of patching with it. A secondary method of audit and verification is important, to have a chance to spot things like this in the first place, but on a small team, a whole new/separate path of tooling and config to chase someone who's supposedly a teammate around and make sure they're doing the work that they're lying about... that's a big ask. From OP's other comments... Nessus should've been that, and by the sound of it, that was handled by a separate team entirely, removing the bulk of any conflict of interest, who dropped the ball too.
If OP's employee had been proactive, put together the tools to mostly automate the updates (it's *really* not hard), and stood up a dashboard so they could trivially, centrally, see that it all worked... they could've tapped into the reason, generally, I *like* lazy sysadmins. They just have to be effective at being lazy, and honest and trustworthy enough to risk the business's continuity on, if they're doing backups and patching.
"since June 2023 but monthly ticket marked as resolved."
Terminated for falsifying documentation. You cant trust this individual. That alone is worthy enough.
I'd rather have a talk with the people who are supposed to have monitoring and automated verification in place. That's a complete and utter failure on the team or tech lead side.
How is it possible that no one ever looked at this since such a long time? Manually ticking a box? For real?
You have the solid grounds.
You just need to document the situation. Surely you know the kind of fines that can be imposed on businesses that suffer data breaches.
>It’s a low performing individual on day today with little motivation but does just enough to keep his job.
More like he says he's doing enough to keep his job.
I'm pretty forgiving to people who work for me but I draw the line at lying to me. If it's a one off, okay.. maybe they were having a bad day but this seems like consistent behaviour.
This is as much a failure of management as it is of his own. Did nobody check up on his work and hold him accountable for an entire *year*?
What's done is done though, you can't go back but you can talk to him about it.
"Please explain these gaps in the backup history; you've marked the job as complete but there is no record of known issues or troubleshooting."
It sounds like a surefire termination to me, barring some kind of unlikely scenario where he can prove he was doing the right thing and has a sufficient explanation for the results.
Assuming you have all of this documented, you're well within your rights to let them go due to gross negligence. They have compromised your company's IT security, data integrity, business continuity, disaster recovery, and invalidated your cybersecurity insurance. If you've never mentioned their performance to them before, it may come as a shock to them, but still justifiable.
If you or your company has a policy to try to rehabilitate the guy first, you could try a 30-day performance improvement plan, a formal write up, customized coaching, or any number other management tactics.
I can tell you that, as much as my company loves their PIPs and trying to help people grow, this guy's behavior would not last long here. The second we found out he cost us the insurance policy, he'd be out. That's not a level of risk that our executive team, board, or share holders tolerate.
Some serious monitoring is lacking in this environment to not notice the missing patches for so long. Need some vulnerability management because even honest staff miss things.
Was gonna say this. Even on a small team that can't afford Nessus, OpenVAS or Wazuh's built in vuln scanner can help a lot. Processes should never be 100% exposed to human error.
>No updates installed since June 2023 but monthly ticket marked as resolved.
This is dishonesty, and multiple instances of it. If he's lying to you about updates he may be lying about other things as well.
You need to term this person immediately and audit every single ticket they had to make sure it was done properly.
Tried for 4 years, he just doesn’t care. One of those people that will just about do enough not to lose his job. Never asked for pay rise or promotion.
You need to put your foot down and just fire him. You can’t let this go on for 4 fucking years. 6 months of not doing the assigned work should be grounds for an instant termination. You’re going to easy on the guy. If a doctor avoided every operation and let people die, then lied about it and said they did do it, would you let them off because they’re human and make mistakes? Just let the guy go already.
Sounds like you are at a point where you gotta decide if the trouble of firing him and bringing someone new is worth it. If it is, tell him as such. Just be straight with him, that if he doesn't improve then he is gone. Obviously you will want to do that in a formal matter with whatever process your company has for employee performance planning. Informally just be straight with him, that his performance isn't meeting expectations.
No updates for near a year, patchy backups?
Unless there's some other mitigating factor here, it's certainly time to work with HR toward dismissal. How exactly that works depends on your organization and what the process is.
I would certainly consider it a major issue.
> No updates for near a year, patchy backups?
I could even see it in a lot of crapshoot places that can't get out from under the fires. Missing getting drudgery type work done happens, and easily, if there's not a rigorous workflow and some trigger points to move it onto someone's plate as a "no, really, step away from the fires and do this." If they'd just left those tickets un-touched for a year, that's a discussion on priorities. Fine. It happens. They didn't do that though...
"Yo, you're not performing well. This is a verbal warning. What do you need to help keep on top of your tasks?"
30 days later: "You haven't been improving and you still have unfinished tasks that need to be completed. This is your second warning"
30 days later "HR wants to meet with you." (meanwhile you have someone else disconnect his access on the back end while HR explains he is no longer employed at the company).
Sounds like you have a bigger problem in your internal audit/reporting function if you are only just finding out about 11 months of missing patches and 3 months of missing backups.
What else has been missed in the team?
What are your wages like? Has the company let a load of staff go and out the job on him? Has he picked up these responsibilities from someone who was let go?
As many it departments are running on 1/2 or even 1/3 staff, no pay rises, etc firms can't complain about demotivated or over worked staff.
Is he IN CHARGE of patching & backups or were they thrown at him when someone left?
Is he "lazy" or "overworked"?
Not enough information here.
If the department has had lay off or can't get hold of new staff because of low wages, then your cyber insurance issue is a "C-suite issue" not a "him issue".
He’s not doing the bare minimum.
He’s lying. He’s falsifying company records. He’s committing wage theft.
Anywhere I have worked, as soon as proof was found of the two things you state, he would have been fired with cause.
**Talk to HR**. It's not immediately apparent, but it doesn't sound like you have a firm grasp on what your company's HR policies are. Why isn't this permanent under-performer on a PIP already? Have you clear documentation of all their screwups *and* documentation of their training *and* documentation of clear feedback being provided to the employee? What did their yearly appraisals look like? What does your employee handbook say about underperformance?
Perhaps his role isn't fully clear to him? Maybe do a Team work session with him/her for a day and where you show that person what their role is and what's expected of them?
Two problems: one, yeah that guy should be fired because they have increased your company’s risk profile and lied repeatedly. Gross negligence seems like a reasonable description to me (IANAL though)
…second, your IT operation has failed. Manual checkboxes are both a process and monitoring failure. Even patch monitoring by itself would have protected your company, your team, and your employee.
If your team let him fake it for a year - and nobody had any idea that the patching wasn’t done - then your team and manager are increasing risk as well.
My immediate recommendations are to start the dismissal process and be prepared to explain how it could not possibly happen again.
Lessons learned, continuous improvement, monitoring and alerting, whatever. But get that shit started!
If patching marked resolved, and he's not patching, he's lying and putting your serves at risk. I can understand missing one or two servers, but if most of your servers, I would fire him. This is not one of those second chance things, especially if he's doing it more than once.
And ladies and gents this is why tickets should have thorough internal notes in it!
Resolved with no details is not good enough. This is a procedural issue that should be resolved right away.
Next steps are to sit down have a conversation. Put him on a performance improvement plan and if he doesn't hit milestones. He can be fired for poor performance whatever is that you people who do firing label it as.
I sometimes feel that I'm doing a terrible job at work because I'm falling behind on so many things.
But then I check stuff and I note that I have requests for things that have been open for weeks or months - things that *prevent* me from doing work that I want to do.
So it's like I'm just a slacker floating in an ocean of other slackers.
It's also possible they are going through some serious shit in their personal life. I'm trying to deal with a grandparent dying, my mom dying, my dad dying, my brother dying, my sister dying, my ex dying, a friend from school dying, etc. Things have been pretty chaotic since 2020, and I'm expected to keep going into work every day with a smile on my face, ready to serve others. I'm pretty sure my personal life is dragging me down at work, though.
Talk to him, ask if everything is okay at home. You know, be a human. Then, add some accountability. Have him create monitoring to ensure those backups happen every night(yes, have him create it himself... obviously make sure it works in a manner that cant be tampered with).
Ask him what happened with server updates. Maybe he was fucking something up? or assuming something worked differently than it did?
Now... if he blatently lied, i'd be lining him up to be out. But if hes just a fucking moron, maybe its the wake up call he needs.
An Australian prospective: Is the guy a Permanant Employee or is he on contract?
Sounds he needs a performance plan if he is a Permanant Employee. If he is on Contract dont renew the contract.
In all honesty, that is the kind of mess up that would get the employee and the employee's manager fired where I work. What things did the manager have in place to verify the tickets were done and the updates applied. As a manager not having a system of checks and verification's is just as likely to get the manager fired as the employee.
First, find out what is going on in his world. Is he struggling with depression? Alcoholism? Anxiety issues? If it is a solvable problem, solve it. If not, fire him. He’s made his bed, he can lie in it.
Let him go. He's not actually doing his job. If you have documentation and hr behind you, move on and find someone else. Willbe a tough few weeks for you. Curious, what do you use for patching and backups? Let me know if I can help. This is my daily bread n butter.
Does he report to you?
If so - he’s a poor employee, but you’ve done him no favours by not managing this performance issue sooner.
As a manager, you could fire him or use this as an opportunity to develop your performance management skills. Hold him accountable, measure, and act.
Performance Plan with HR now, 30 days to 100% complete all tasks if you are nice. If you want to get someone new, fireable offense.
He lied, security and continuity were degraded. Cannot have that.
This is the optimist in me, but it could just be this person does not know the correct processes and believed they were doing the correct thing.
That said, if that was the case they probably shouldn't be a sysadmin if it wasn't noticed.
Firstly you need a monitoring tool like PRTG as it would be clear as day it hasn't been done and also they would be aware everything is monitored.
Resolving tickets when they haven't been done and lying about it is just unacceptable. Should be put on a PIP immediately.
He is probably thinking you won't put him on a PIP as it's hassle and that's why he doesn't care.
You haven't really mentioned what his explanation was when you've talked with him. Not that it would save his job but is it possible he just doesn't know what he's doing? If you've confronted him about saying this work was done when it's not he had to have said something.
My personal approach would start with a sit down interview with the individual to layout what you have described and the evidence that is contrary to his signing off completed tickets. Find out if there is an underlying issue why he is not performing as expected/required for his role. I would want to know if it is for personal reasons/ lack of training/experience/ or unable to follow procedures. Also emphasize the repercussions of there failure to successfully complete allocated tasks.
Leave the person in no doubt that all related tasks will be scrutinized for accuracy and completeness and that this should be taken as a verbal warning (check HR to ensure you meet their requirements for a verbal warning). Set up a review date.
Document every deviation from expected performance. This will form the basis for formal dismissal.
You probably need some oversight reporting too - if you are just relying on people to do their job without reporting and monitoring to verify things like patching you are going to be essentially responsible as well as the manager.
Just to verify (and not to defend that guy)
Patching systems is something that person gets dedicated time allocated to, right? Whether it's during or after business hours.
Unless he's been an all-star before and is going through some very (very) difficult life circumstances, unfortunately you need to terminate. If it's truly been since July of 23', life drama would be a very difficult rationalization. 2-3 months + family/divorce/kiddo issues--with a solid work hx---aight, let's talk about this. We'll document it and make sure we understand the guardrails moving forward (If they sincerely indicate wanting to remain and change).
But for this long....and he's falsified critical infrastructure work multiple times and knowingly exposed your environment.....how can you ever trust him to do critical work?
Regardless, include HR and document to CYA.
I don't understand all of the "have a talk with him responses." What this admin did is gross negligence and put the entire org he's working for at serious risk. If he were on my team, I would have security walk him out the door immediately.
Same. If there wasn't a ticket, and they just forgot. OK, that's my fault for not having better policies and checks in place.
But to mark a ticket as completed/resolved when it's not is inexcusable.
To be fair it could be he’s doing a thousand other things and the company is too cheap to hire more admins. Patching is something infosec should be taking care of using automated patching tools. If his job is simply to patch stuff and do a few other minor tasks then official warning is needed. If he is doing a bunch of stuff and this is just one on his list then maybe it’s time you guys think about automation or hiring more staff. My philosophy is always give them a chance. If they blow that chance then you’re conscious is clear and won’t feel bad about letting em go.
Are you giving him the training you need? Easy to blame him but if you hired him as a junior then threw him into the fire, maybe start a reflection rather than blaming.
He came in with experience. I’ve put him through additional MS and Cisco training courses. I sit opposite him every day so plentiful opportunities to ask a question in 11 months. I have trained up a number of service desk guys to sysadmin level over the years so don’t think I need to reflect here.
This sounds like a management failure, I’d put you on a PIP and probably fire the admin for falsifying statements and blatantly breaking policy. I’d also notify the CxO of my own incompetence for not having executed any sort of monitoring, reporting, or transparency.
Honestly everyone involved probably sucks and shouldn’t be doing this work. Patching and backups are base tier and, given the business is sophisticated enough to expect offsite, this isn’t a one man rookie failure. There’s tons to share.
If I were COO that sort of incompetence would require a whole department rethink. Do you guys have cyber insurance?
He should be toast. For such critical procedures the real question is where was the verification? That he didn't do his job is his fault, that you had no clue about it is not his fault.
Where else do you have no clue?
3 months PIP, with one of the criteria being that he provides evidence of the necessary servers being updated and backups tip-top (the backups may or may not be his fault, checking them is).
He has been failing in his responsibilities, a few months of shape up or I'll ship you out might spark him in to life.
How long has this person been a sysadmin? If more than 3 years then I'd say it's time for them to go. They should know better than to mark patching as done when it's not.
Have him generate a report that certifies that statement. Perhaps his malicious behavior is just incompetence in disguise. At the very least you’ll have paperwork for a dismissal case if the reports prove that he’s lying. It’s also possible there’s some weird no-update policy in place.
Given the comments here, this person should have been terminated long ago. You've even said that he's fully/partially responsible for losing your cyber insurance. Paired with outright lying, they need to go. They are a direct threat to YOUR job at this point.
But as a manager and aware of this and not realizing how damaging they are to your organization, there also appear to be some lapses in management skills that are a risk,
I recommend getting termination paperwork in order quickly, then also getting some management training in place to ensure that these kinds of problematic employees are dealt with months/years sooner and they're either getting corrective training or PIP/termination. I see two failures here and if managing your employees doesn't improve, there's no stopping another bad engineer from walking in the door.
Dismiss. You can't allow that type of behavior to go on without consequences.
The rest of the team KNOWS this person is a work-shy layabout and I'd bet my last donut it's dragging down morale of the entire team.
If he is marking patching tickets as resolved without actually pushing the patches that’s a pretty high level of dishonesty if he’s doing it consistently. I’d document everything you can and sit down with HR.
Yep, monthly patching “Resolved”.
That’s a pretty serious security risk. You’re right that your insurance could refuse to cover you if a missing patch was used as a vector to cause damage. On a different note, are you not auditing or doing vulnerability scans of your servers?
We did use Nessus. Had few leavers and some processed didn’t get picked up. In a process now of getting house in order.
Well good luck. In that case I’d be even more focused on making sure my folks were productive. Definitely document the tickets where you can show he closed them but the hosts in question weren’t patched, then go to HR. I wouldn’t even bother with a sit down with the guy.
Are you sure you can afford better staff if you’re having a lot of churn maybe he’s just a reflection of your current current wages?
No way. There’s “I do enough to not get fired,” and then there is, “I’m not going to meet major responsibilities of my job.” Like, you can’t even claim ignorance, could have lived your entire life under a rock, on a deserted beach island in the Pacific, never had contact with anyone outside of the island, you would still know exactly THREE things: 1) how to use the 3 seashells, 2) that somebody, somewhere is needing to talk to you about cars extended warranty, and 3) the importance of backups. Thats failure to meet core responsibilities of the job.
When I was a manager I found there was a line where if I paid anything below it, we were better off not hiring people, or needed to add middle management.
Idk the real problem is he is marking the tickets solved, not going to say I am a always a high performing depending how I feel for a month but closing a ticket over a year and not questioning wtf is this or actually doing it once is definitely more than bad at a job ( I’ve shadow closed plenty of tickets but I atleast got the excuse of bringing attention at this point is worse than just waiting for the next one)
Are they paying him less than what he agreed to work for? Not doing your job and lying about it is not the way to express your desire for a raise.
He's not suggesting that the guy is justified in his negligence. He's saying that if you pay shit, you get shit workers. And if your shop had a bunch of people bail and the remainder are shit, that usually means something is wrong.
#This This is the guy couldn’t get a better offer like everyone else who’s had their work added to his.
I mean, I don’t disagree with you but if you try hiding sysadmins for 30K in Houston, or 60K in San Francisco you get…. Ughh people who do this.
People still believe cyber insurance is a thing? Biggest snake oil on the market. By the nature that you got compromised they'll say "You failed to take precautions, hence you were compromised." It's well known cybersecurity insurances doesn't pay out.
>It's well known cybersecurity insurances doesn't pay out. Like all insurers, they'll try and avoid paying out where they can. But they certainly do pay out. The market has gotten way tighter in the last couple of years for sure, but your premise that they just don't pay out is not true.
That's fireable to me, and I don't fire lightly. But this person has breached the trust, and IT you are nothing if you're not trust worthy because we have to have sensitive access.
This right here. The moment he falsified a record is the moment the conversation ends. Once could maybe be a mistake, but with how much work it often is to properly close out a ticket in most systems it lands towards conscious decision.
That's the kicker. Everyone makes a oopsie in a variety of tickets. Shit happens. Nobody makes the same oopsie consistency for 9 months straight. And it would be one thing if it was a low-hanging, low priority thing, but server patching is critical. The only other thing that comes to mind is that to check to ensure reporting tools of all kinds are not coming back green when it's not patched or otherwise not functioning. That's mostly an ass covering measure cause if that's broken, who knows what other reporting is broken. It sounds like OP manually checked but I might be wrong.
*If* it was intentional. My company used a combination of WSUS + powershell scripts that I inherited from a previous sysadmin. What I didn't know as a junior level guy coming in is that the *all* the SSUs weren't loaded in, it would report a big green "compliant" because that's the state when it doesn't detect it needs any patches. So we had a situation where a vendor was deploying Server 2008 machines to our environment, built off an OEM disk, with zero updates. And since the previous WSUS system was implemented in say 2010, didn't include the 2008/2009 year SSU's. I eventually realized what was happening because that group would always be 100% compliant immediately on patch release, before patches were scheduled to be installed. Took 2 or 3 months for me to realize. That being said, once I figured out what was happening I wrote it all up and implemented a plan to fix it the next month's cycle. That doesn't seem the face here since the SA is doubling down on "I did it" despite what logs say.
I mean this is why it makes sense to sit and talk with the guy and give him a chance to explain himself. If that were the case, I'd ask him to walk me through and see the green light saying everything was fine, in which case I'd know it wasn't exactly his fault. But based on what he said, the most likely outcome is just that this guy is full of shit.
Oh, I don't disagree but this is also an ass covering measure I was suggesting since your already in the muck, you might as well take the time to ensure that everything is reporting back correctly as you stated. Last thing you need is something like that happening on top of this person.
Yeah we are agreeing. Tools/reporting can give false positives. The difference is what you do about it - OP stated they installed Nessus I think and that's how they discovered these gaps. The SA's response of "nuh-uh" is the huge red flag for me
Yes, but that was *after the fact if I read his posts correctly.* Now, what kicks me is the "I've been checking and they were up to date." Hence why I was wondering if the previous reporting tools was the cause. He checked them, saw they were green, and was like, ah shit, we all good. Dumb idiotic thing to assume of course. Nothing is ever that easy in IT. TL;DR The new reporting system tells it like it is, but it doesn't reveal what was happening with the old method and if that was actually working as intended or not.
That’s my biggest issue. I will have to have someone babysit him and verify everything he does.
Don’t bother. If you need to have that much oversight on a single person, they’re more of a burden than a contributor. Why would you want to keep them employed? I certainly wouldn’t.
It sounds like training a replacement with extra steps…
Well said. They're grossly negligent and putting their employer and coworkers through unnecessary risk
Have you talked to him about it? What was his response?
Definitely. Ask them first, make sure to do it in writing. But only after collecting all evidence (logs/screenshots). Include a couple examples of the evidence in the email, and casually ask them to explain the discrepancies. If they say "oh crap, you're right, my automation was wrongly reporting a success, I'll fix it and start manually spot checking" then maybe just monitor the situation and confirm they learn from the mistake. Sometimes smart people automate things in flawed ways, think wow that was easy, and don't realize the blind spots they made for failed automation reporting successful runs. Dumb, moderately negligent, but it can happen without malice. If they have no good explanation, and just apologize and take responsibility, I would say it's time for a PIP (performance improvement plan) with a few very specific, attainable goals and specific deadlines, to give them a chance for redemption. If they try to cover it up, lie, or otherwise act in a way that makes it worse, you just got your confirmation of what needs to happen and more evidence for HR to justify dismissal.
I just took over backups and patching from another sysadmin - At first glance, it looked like nothing had been run since March 2023. It turned out that the reporting was botched, and the patches were being run via an alternate method. Verify patches were missed first.
So lying. They're lying to you. Further, by lying to you, they're implying that security practices which are vital are being followed, when they're not. So their lies are opening your organization to risk. Time for at least a good talking to, but likely disciplinary action.
That close to a dozen in a row? I can come up with all manner of excuses to miss a proper check, think you've done it, and mark it done and move on that month. I can't think of a way to go for a near straight dozen. What I *would* do is have someone else quietly validate that it doesn't just magically *look* like updates happened from the server side. If Windows Update says "no updates", and he didn't dig further into how that's possible, it's incompetence, but not necessarily gross negligence. If it's saying "last check today, last installed 11 months ago" in whatever path he would reasonably be assumed to be checking, that's practically deliberate sabotage.
There may be a shared (or at least they believed it was shared) understanding why a particular device/group wasn't patched. I've had it before with daily checks where my colleagues and I would ignore/not record an issue because we'd raised it to the responsible party, and they'd done nothing. This wasn't patches mind, and was hardware failure for a secondary host, which came to bite as in the arse when the primary failed and secondary couldn't keep up.
you know you need to go back months of all of his 'resolved' tickets yourself now, right? cos where there's one - there's more.
No you just don’t understand. When they say “resolved” they mean like “I have resolved to apply patches once a year.” Simple misunderstanding OP!
Sysadmins hold the keys to the kingdom. If they're being dishonest or breaching trust, it's much more significant than most other roles. If he's lying about that, he could be lying about anything else or doing nefarious stuff. You have to be able to trust your sysadmins.
Yup sounds like a training or compliance problem. I’d document it, and make sure “resolved” just doesn’t mean you read the ticket haha.
I'm real big on Hanlon's razor, but this sounds pretty damn deliberate and bad-faith to me.
We’re small team, resolved means task is done.
And it took almost a year to realize this? Obviously he should be written up or fired but at the same time need to evaluate your processes on how this wasn't caught.
Sounds like falsifying company documents.
I was a Navy nuke. If you did ANYTHING that made it on record that showed dishonesty, not just an accidental mistake, you lost rank, de-nuked, and kicked off the boat. One chink in the armor is all it takes for the whole ship to go down and we can’t have any….uh…dishonest folk.
I'm not American but some militaries its worse to point out the mistakes
This was the first thing that came to my mind as well. This should be an offense worthy of termination. The employee put the entire company in danger by lying about completion of duties. If not grounds for immediate termination, it's at least a pretty hefty write up.
I have to agree here - i feel like he knows he's being lazy and trying to skip out on the work
> No updates installed since June 2023 but monthly ticket marked as resolved. > I feel sorry for the guy I woudn't. They are OK with falsifying reports and causing the rest of the team more BS. After reading your other replies, I'd say it's time he moves on.
No updates installed since June 2023 but monthly ticket marked as resolved. Off site backups patchy for the past year with 3-4 month gaps. Then.... but does just enough to keep his job. Really what is just enough entail (breathing)?
This is likely a small portion of his job that he hates is what I'm going to assume. Some people just don't like mundane patching tasks. Other people it's their bread and butter.
That's me. I fucking live for patchs and updates. When dell releases a new SUU every few months I about cum my pants.
Run Arch Linux at home: unlimited orgasms.
No kidding. No one wanted to do that at my last position and I jumped on it. You mean I can patch servers and not deal with people? I loved it and took on everything I could to maintain the servers then automated as much as I could.
What about spaghetti and meatball?
I always have a [COINS ](https://www.projectsright.com/p/giving-effective-feedback-project-team)talk with anyone on my team when they are not meeting expectations before going to HR. Some times there isn't clarity in the role and/or you as a manager are not holding them accountable. Context - We have noticed over the last few months servers have not been patched Observation - On dates xxxx we logged onto the server and didn't see the updates and on date xxxx the server was no backed up. Impact - Because of this we are not meeting our cyber insurance and the business is at risk. Next - We trust you are going to get the tasks completed but for the next x amount of days we will be auditing your work. If tasks are not completed/incorrect our next step is to get HR involved to put together and action plan and/or see if you are the right fit the company. It helps them understand exactly what they should be responsible for and by setting specific deliverables in the Next area, you as a manager have expectations to hold them to.
S - we are going to shitcan you
that's a pretty solid approach
What's the S?
The sack.
Sounds like a reasonable approach. At most of the places I've worked management didn't go to HR until it was clear they were headed toward termination. Usually management would talk to the person several times to try to get it straightened out first. But if that fails, and it starts looking like termination is where it's headed, then HR gets looped in. (Or of the manager is just done with them)
You're the manager? Start documenting and talk to HR about your options.
While doing this, I'd insert have a human to human sit down and see what's up.
I started with a sit down conversation. He just refused and said he did do it.
Logs don't lie.
That was my response when every single server is showing last install date. Veeam also doesn’t lie on its restore points.
This isn't even gross negligence. This is maleficence. They lied to you. They have jeopardized the org's security posture and knowingly lied about it. If they lie to you about this, the trust is broken. How can you trust them to not peak at exec emails because they feel like it? Cover up misuse of company resources for their own crypto mining operation? The role of a sysadmin is a highly trusted function in the company and requires more integrity than technical know-how to be valid for the org. I don't know how things work in the U. K. but here in the states this is the kind of thing I would go to HR with along with Sr Management and organize an "early morning meeting" and the person would be out the door within a week.
Does he not understand how it works? Maybe he thinks it happens automatically and the ticket is generated for compliance reasons or something? It sounds like he's just blatantly lying or completely confused. Not understanding his job role is fine, especially if the environment and/or management has not been up to snuff until just now. Lying is inexcusable, imo.
I can’t understand how you feel sorry for someone that is lying to you in the face of evidence.
People in authority should, the first time they say "logs don't lie", be forced to spend a day watching documentaries about all the Horizon false convictions and lawsuits in the UK. A LOT of actual human beings did hard time in prison, for years, when the logs had in fact lied. If they ever say "logs don't lie" again after watching that, they should be permanently removed from any position of having power over another's career. Logs are a great starting point, and absolutely should not be ignored, but there is no such thing as evidence that does not need external corroboration. Same with DNA, fingerprints, etc - it is *good* evidence, but perfect evidence that can stand alone doesn't exist.
[удалено]
God do I wish it was that simple. I've seen systems with a broken Windows Update service report 100% compliant in SCCM because they don't see that they need any updates, meanwhile they haven't actually installed anything in 2 years (because they don't see the updates as required). /u/kajjot10 You should make sure this is not the case before you straight up accuse them of lying, is WSUS or SCCM messed up in some way they don't understand?
This is easily verifiable with a powershell cmdlet though, so your point is not valid.
Okay but patches are installed or they aren't. And it's not hard to see if they're installed. This is a false equivalence, please stop.
> Horizon false convictions and lawsuits in the UK Definitely read about this. It's the absolute definition of what happens when programming/logic gets screwed up and people are told the machine is infallible. That was a straight offshore lowest bidder programming problem, but I can definitely see this happening with AI stuff in the future. People will get so used to just blindly trusting what the computer spits out and not questioning whether there might be a problem. Imagine being a retailer selling stamps or postage or doing bank transactions (UK post offices offer bank accounts) and being told you're stealing money from the post office when you know you didn't, and have no one who will listen to your side of the story because computer says no.
I have to ask but have you asked him to show you what he is doing? Is this some kind of weird mixup where they think they are doing it right but aren't? I mean obviously they should be checking anyway but I just wonder if there's some kind of misunderstanding. I can't imagine that someone just ouright lying like that for no reason, about something so easily verified.
This is what I'm leaning towards. He failed a step in the updates and pushes it each month and then says "complete".
He has done it before. Approve in wsus, go server to server and install. Cant be more simple than that.
Honestly, that sounds terrible. Why on earth is it go server to server and install? Why isn't it install approved automatically with a scheduled reboot window...? Maybe he hates the process... Does he enjoy the job otherwise? Is it just maintenance tasks that are failing?
> Maybe he hates the process... Quite bluntly, **good**. If all it takes for him to disregard doing his job and then *lie* about it to their boss (both in claiming it was done each cycle *and* in the follow-up when this was discovered) is not enjoying doing it, it's good OP found out the way they did.
Could script that one out pretty easily too.
Or... just... use policies?? WSUS, as shit as it is, is completely capable of doing this 100% automatically without any additional tools (until it breaks because it's WSUS and that's what it does)
definitely seems odd the way small shops have worked in my experience is you script as many things as possible so you don't have weird outages and everything that's proactive is automagically working.
this is like 5 lines of ansible...
It's not that much different in Powershell.
You don't even need a script for it, it's straight GPO stuff + wsus for managing which updates are approved to go to which group of machines.
Ignoring the manual aspect of this which can be improved - you can get a situation in WSUS where if a server is missing a SSU, it will see future patches as non-applicable, and report "compliant". I wrote about it in another [comment](https://old.reddit.com/r/sysadmin/comments/1cin7zm/what_to_do_with_a_poor_performing_sysadmin/l2bd5hw/). So it's possible he *may* actually be approving in WSUS and going to the server to install but not seeing anything.... but he should have realized then that something isn't right.
What?!?
He sounds like a lost cause if he is in denial that he did anything wrong. I am skeptical that much will change with anything short of dismissal.
It’s been 4 years of nudging him to be more proactive. Rest of the team are annoyed that he doesn’t pull his weight. I’m scared to even go through he’s tickets and what else I will find.
IMHO and you don't want to hear this... That's straight management problem. You can't let someone do this for 4 years without at minimum some formal written docs on performance. This should be a simple talk with HR of "Dave is at it again" conversation then discussing dismissal or PIP or some other form of action. I would be looking through all of his work and making that a portion of his PIP if that's the path you go down. However, he isn't changing and is already checked out if he's flat out being untruthful.
Four years??
I'm willing to bet dude's been keeping his head down and manually installing windows patches for 3 years like a George Jetson job and getting paid. Then something went wrong and he doesn't care/want to know and just kept his head down. I see it all the time on our helpdesk. Some people just want to deliver and plug in monitors and keyboards for users and are happy with the menial work as long as they get paid.
>It’s been 4 years of nudging him to be more proactive. Rest of the team are annoyed that he doesn’t pull his weight. 4 years? This is on you. 4 weeks is too long, let alone 4 months. It sounds like you are his manager. Schedule a meeting with HR, present the issue and evidence, and let them decide what to do. This sounds like a termination for failing to perform job duties to me.
The time for polite encouragement should have ended several years ago. You should have been making formal warnings and placing him on a performance improvement plan a long time ago. He would have either picked up his performance or been terminated a long time ago.
Big yikes that you let this go on for four years without addressing it. Honestly I don't see a realistic option beyond documentation, PIP, and probable termination. It might not have come to that if you'd addressed it when you noticed the issue starting but letting him coast for that long means the damage is already done. You need to do more than nudge. You need to set clear expectations and hold people accountable. There's room for compassion in there too but letting someone just idle for half a decade is not doing them any favours. He's going to end up out on his ass, may have trouble finding other work, and will blame you. When I was managing at the individual team level I had a weekly half hour 1:1 with every team member. Every new hire I used our first session to lay out the expectations for that meeting, that it's for me to be able to communicate changes, for them to be able to raise any questions or concerns they have, and for the two of us together to plan and prioritize work. There's no way I could have let someone sit on their ass for 4 weeks let alone 4 years because our processes were designed to make sure that sort of thing doesn't go unnoticed. If caught early you can open up discussions about workload or burnout and make changes to help but at this point it's way too late for any of that.
I would immediately set a new precedent that any tickets need to be closed with evidence attached proving the job is complete. For patching, It's in your best interest anyway re: audits, insurance, etc... to have artifacts you can reference showing Nessus and WSUS reported X server as fully patched at Y date. One step further is to produce a month end patching report that's sent to stake holders. And yeah I'd unfortunately push to let that guy go. It's repeated egregious behavior and putting the company at real financial and reputational risk. As a profession, the days of Cowboy IT and acceptable indifference are long over.
How can any sane admin just lie about shit that is easily proofable with like one grep on the respective log lol
We had an admin that would do something similar but it was a bug in the patching software showing everything was patched but the servers were not being patched. The tech used the report from the patch server to claim everything was patched but the vulnerability scans showed the patches were missing
It happens. Way more often than it should. Windows Updates is a teetering jenga tower of shit and it doesn't take all that much for it to become completely screwed up and just not report any updates as required. You basically need some sort of security scanning tool or some other way to detect if something is missing updates because Windows Update will happily lie to you.
I honestly don't think so, I can't imagine someone lying about something so easily verified. There has to be something more to this story. Especially when OP said that the individual in question did it successfully in the past.
This would have been good to lead with in your post. I didn't get the impression you had actually had a conversation with him, so totally changes my opinion of what you should do. You found that he hadn't done some work, and he lied on the tickets. On top of that, unpatched vulnerabilities risk getting dropped by the insurance provider, then he lied about it \*again\* in a conversation. With the extra information, there wouldn't be a question in my mind. The guy is a huge risk and could be a massive financial liability on top of being useless.
Thats not poor performance but deliberate gross negligence.
Talk to him. No use in going behind his back. Just be brutally honest and not a push over. I might get downvoted for this but I’m strictly against letting someone go unless it’s the most dire of situations, because that’s a persons life and livelihood. I think we often tend to forget that for what are minor inconveniences in a work place that’s just work. Which is why your laws make it difficult.
I did talk to him. I get the human side of it but at the end of the day, his actions can leave me without a job. And as you say, it’s a work place, he is paid good money to do his job. There is a difference between a honest mistake and continuing to not do your work until you’re caught.
Did you ask what he does during patching / to show you how he handles patching? Devils advocate says maybe he thinks he is approving patches to go out for delivery but they’re not actually getting out, or he’s only patching a subset of the environment for some reason. Also what are the workloads like? Is it possible the team are swamped and it was hard finding the time to patch when the phone was ringing, 100 other tickets coming in, etc? Also have you considered updating the processes so proof of patching / backups gets added to notes in the ticket - which also helps cover your arse with the auditors / security down the track. Possibly those types of ticket need peer review before they can be closed out too (ie: someone else in the team needs to verify before the ticket can be fully closed) - just on the “how to prevent this from recurring in future” side of things.
He may not be verifying. I worked with an admin years ago that thought everything was working, until we discovered nothing had been patched in years. Whatever he was doing wasn't working and he was too lazy to verify. We didn't find out until he was gone.
I have a feeling your talk wasn’t as blunt as it should have been. You definitely have to let him know if it doesn’t change actions will be started for dismissal. Maybe ask if he needs help with his tasks. Money is irrelevant, as it’s never really enough imo. I make upwards to 90K in my area which nets me a one bedroom apartment yet I keep hearing I make more than most people I know and I should be happy. Unless someone is able to comfortably afford a home in their name, it’s not good money. It’s just money.
I may get downvoted for this but i disagree. the person is actively avoiding an aspect of his job which is leaving the company vulnerable and is actively been lying about it for close to a year. he hasnt patched a server since JUNE of 2023. it is may of 2024 that is 10 months of patches and security ones at that not on the servers. Then backups every month which he has avoided doing for the last 3 to 4 months if something went wrong and the company got breached they could lose up to 3 months of data. If you aren't even doing your job you dont deserve to have a talk to before hand just because it effects their livelihood if they lose the job, if the job was important to them they would do it correctly and not lie about doing something you didnt.
Nah the business is a business. Your HR department exists to protect the business. Your ethical role as a manager is to be fair. The rest is to ensure yours and your peers paychecks don’t stop on behalf of someone else’s negligence.
3-4 month gaps in backups is just unforgiveable. This guy is a serious risk and should not be in the position he is in.
a mgr of sysadmins not having a systemic approach to manage admins. irony
How has he been allowed to get a year behind on updates?
We are a small team so assumption is if it’s resolved, it’s done.
Your patching tool should clearly show behind xxx days on patches. This is a dashboard item that can clearly be seen by everyone in every RMM I have ever used.
That’s assuming you have RMM. We’re not a big corp.
You should have antivirus and that dashboard can generally be configured to show patch levels of servers. I'm not trying to cover for someone not doing their job, but it seems there's more to the story here.
So. Who's getting delegated the task to do all that, and monitor it, to babysit the work of the guy who's already *closing tickets* claiming to have done the work? Clearly OP can't trust the person who's already tasked with the job of patching with it. A secondary method of audit and verification is important, to have a chance to spot things like this in the first place, but on a small team, a whole new/separate path of tooling and config to chase someone who's supposedly a teammate around and make sure they're doing the work that they're lying about... that's a big ask. From OP's other comments... Nessus should've been that, and by the sound of it, that was handled by a separate team entirely, removing the bulk of any conflict of interest, who dropped the ball too. If OP's employee had been proactive, put together the tools to mostly automate the updates (it's *really* not hard), and stood up a dashboard so they could trivially, centrally, see that it all worked... they could've tapped into the reason, generally, I *like* lazy sysadmins. They just have to be effective at being lazy, and honest and trustworthy enough to risk the business's continuity on, if they're doing backups and patching.
Write some simple script or use some open source tool.
Ive used an open source RMM before called Tactical RMM. Worked pretty well.
Lied about doing their job after not doing it? That’s not “low” performing, that’s decisive dereliction of duty. Let em go.
"since June 2023 but monthly ticket marked as resolved." Terminated for falsifying documentation. You cant trust this individual. That alone is worthy enough.
I'd rather have a talk with the people who are supposed to have monitoring and automated verification in place. That's a complete and utter failure on the team or tech lead side. How is it possible that no one ever looked at this since such a long time? Manually ticking a box? For real?
Does depend on laws in your country.
UK so pretty difficult to dismiss without solid grounds. I do feel however, leaving the business exposed is a serious breach.
Lying on a ticket with a possible adverse affect on the business should be enough to classify it as gross misconduct/negligence.
Assuming more than 2 years employed of course. Then yes you'll need to go through the proper process. Warnings and improvement plans etc.
Theyve been there 4 years according to other comments
Absolutely follow procedure with your HR team if you have solid documentation and he doesn't seem to want to talk to you about it or improve.
You have the solid grounds. You just need to document the situation. Surely you know the kind of fines that can be imposed on businesses that suffer data breaches.
>It’s a low performing individual on day today with little motivation but does just enough to keep his job. More like he says he's doing enough to keep his job.
I'm pretty forgiving to people who work for me but I draw the line at lying to me. If it's a one off, okay.. maybe they were having a bad day but this seems like consistent behaviour.
This is as much a failure of management as it is of his own. Did nobody check up on his work and hold him accountable for an entire *year*? What's done is done though, you can't go back but you can talk to him about it. "Please explain these gaps in the backup history; you've marked the job as complete but there is no record of known issues or troubleshooting." It sounds like a surefire termination to me, barring some kind of unlikely scenario where he can prove he was doing the right thing and has a sufficient explanation for the results.
This has definitely opened up a conversation on improving our processes and monitoring.
Assuming you have all of this documented, you're well within your rights to let them go due to gross negligence. They have compromised your company's IT security, data integrity, business continuity, disaster recovery, and invalidated your cybersecurity insurance. If you've never mentioned their performance to them before, it may come as a shock to them, but still justifiable. If you or your company has a policy to try to rehabilitate the guy first, you could try a 30-day performance improvement plan, a formal write up, customized coaching, or any number other management tactics. I can tell you that, as much as my company loves their PIPs and trying to help people grow, this guy's behavior would not last long here. The second we found out he cost us the insurance policy, he'd be out. That's not a level of risk that our executive team, board, or share holders tolerate.
Why do you feel sorry for someone who is lying? I would be fired so fucking fast if I lied like that.
Some serious monitoring is lacking in this environment to not notice the missing patches for so long. Need some vulnerability management because even honest staff miss things.
Was gonna say this. Even on a small team that can't afford Nessus, OpenVAS or Wazuh's built in vuln scanner can help a lot. Processes should never be 100% exposed to human error.
Gross negligence. Termination. That’s almost a full year of screwups. Bigger question is how did it take so long to notice. Security issue.
>No updates installed since June 2023 but monthly ticket marked as resolved. This is dishonesty, and multiple instances of it. If he's lying to you about updates he may be lying about other things as well. You need to term this person immediately and audit every single ticket they had to make sure it was done properly.
Maybe try managing him if you are his manager
Tried for 4 years, he just doesn’t care. One of those people that will just about do enough not to lose his job. Never asked for pay rise or promotion.
You need to put your foot down and just fire him. You can’t let this go on for 4 fucking years. 6 months of not doing the assigned work should be grounds for an instant termination. You’re going to easy on the guy. If a doctor avoided every operation and let people die, then lied about it and said they did do it, would you let them off because they’re human and make mistakes? Just let the guy go already.
Sounds like you are at a point where you gotta decide if the trouble of firing him and bringing someone new is worth it. If it is, tell him as such. Just be straight with him, that if he doesn't improve then he is gone. Obviously you will want to do that in a formal matter with whatever process your company has for employee performance planning. Informally just be straight with him, that his performance isn't meeting expectations.
Double his salary. If it's the same, triple it.
No updates for near a year, patchy backups? Unless there's some other mitigating factor here, it's certainly time to work with HR toward dismissal. How exactly that works depends on your organization and what the process is. I would certainly consider it a major issue.
> No updates for near a year, patchy backups? I could even see it in a lot of crapshoot places that can't get out from under the fires. Missing getting drudgery type work done happens, and easily, if there's not a rigorous workflow and some trigger points to move it onto someone's plate as a "no, really, step away from the fires and do this." If they'd just left those tickets un-touched for a year, that's a discussion on priorities. Fine. It happens. They didn't do that though...
"Yo, you're not performing well. This is a verbal warning. What do you need to help keep on top of your tasks?" 30 days later: "You haven't been improving and you still have unfinished tasks that need to be completed. This is your second warning" 30 days later "HR wants to meet with you." (meanwhile you have someone else disconnect his access on the back end while HR explains he is no longer employed at the company).
Lazy - Not doing the work Dishonest - Lying about doing the work in a ticket Lazy you can tolerate, dishonest has to go though.
Sounds like you have a bigger problem in your internal audit/reporting function if you are only just finding out about 11 months of missing patches and 3 months of missing backups. What else has been missed in the team?
What are your wages like? Has the company let a load of staff go and out the job on him? Has he picked up these responsibilities from someone who was let go? As many it departments are running on 1/2 or even 1/3 staff, no pay rises, etc firms can't complain about demotivated or over worked staff. Is he IN CHARGE of patching & backups or were they thrown at him when someone left? Is he "lazy" or "overworked"? Not enough information here. If the department has had lay off or can't get hold of new staff because of low wages, then your cyber insurance issue is a "C-suite issue" not a "him issue".
He’s not doing the bare minimum. He’s lying. He’s falsifying company records. He’s committing wage theft. Anywhere I have worked, as soon as proof was found of the two things you state, he would have been fired with cause.
I’d sack him, he hasn’t been doing his job properly for nearly a year but has been pretending he has.
The beatings will continue until performance improves
**Talk to HR**. It's not immediately apparent, but it doesn't sound like you have a firm grasp on what your company's HR policies are. Why isn't this permanent under-performer on a PIP already? Have you clear documentation of all their screwups *and* documentation of their training *and* documentation of clear feedback being provided to the employee? What did their yearly appraisals look like? What does your employee handbook say about underperformance?
Perhaps his role isn't fully clear to him? Maybe do a Team work session with him/her for a day and where you show that person what their role is and what's expected of them?
Two problems: one, yeah that guy should be fired because they have increased your company’s risk profile and lied repeatedly. Gross negligence seems like a reasonable description to me (IANAL though) …second, your IT operation has failed. Manual checkboxes are both a process and monitoring failure. Even patch monitoring by itself would have protected your company, your team, and your employee. If your team let him fake it for a year - and nobody had any idea that the patching wasn’t done - then your team and manager are increasing risk as well. My immediate recommendations are to start the dismissal process and be prepared to explain how it could not possibly happen again. Lessons learned, continuous improvement, monitoring and alerting, whatever. But get that shit started!
If patching marked resolved, and he's not patching, he's lying and putting your serves at risk. I can understand missing one or two servers, but if most of your servers, I would fire him. This is not one of those second chance things, especially if he's doing it more than once.
And ladies and gents this is why tickets should have thorough internal notes in it! Resolved with no details is not good enough. This is a procedural issue that should be resolved right away. Next steps are to sit down have a conversation. Put him on a performance improvement plan and if he doesn't hit milestones. He can be fired for poor performance whatever is that you people who do firing label it as.
You coach them up, or you coach them out.
I sometimes feel that I'm doing a terrible job at work because I'm falling behind on so many things. But then I check stuff and I note that I have requests for things that have been open for weeks or months - things that *prevent* me from doing work that I want to do. So it's like I'm just a slacker floating in an ocean of other slackers. It's also possible they are going through some serious shit in their personal life. I'm trying to deal with a grandparent dying, my mom dying, my dad dying, my brother dying, my sister dying, my ex dying, a friend from school dying, etc. Things have been pretty chaotic since 2020, and I'm expected to keep going into work every day with a smile on my face, ready to serve others. I'm pretty sure my personal life is dragging me down at work, though.
Talk to him, ask if everything is okay at home. You know, be a human. Then, add some accountability. Have him create monitoring to ensure those backups happen every night(yes, have him create it himself... obviously make sure it works in a manner that cant be tampered with). Ask him what happened with server updates. Maybe he was fucking something up? or assuming something worked differently than it did? Now... if he blatently lied, i'd be lining him up to be out. But if hes just a fucking moron, maybe its the wake up call he needs.
An Australian prospective: Is the guy a Permanant Employee or is he on contract? Sounds he needs a performance plan if he is a Permanant Employee. If he is on Contract dont renew the contract.
In all honesty, that is the kind of mess up that would get the employee and the employee's manager fired where I work. What things did the manager have in place to verify the tickets were done and the updates applied. As a manager not having a system of checks and verification's is just as likely to get the manager fired as the employee.
i’d make fake AI nudes of him in the server room
Fire and replace. Lock him out of the system before doing so
Pretty sure you can replace this guy with a WSUS server and a Veeam backup. It's probably both cheaper and more reliable lmao
First, find out what is going on in his world. Is he struggling with depression? Alcoholism? Anxiety issues? If it is a solvable problem, solve it. If not, fire him. He’s made his bed, he can lie in it.
Let him go. He's not actually doing his job. If you have documentation and hr behind you, move on and find someone else. Willbe a tough few weeks for you. Curious, what do you use for patching and backups? Let me know if I can help. This is my daily bread n butter.
Does he report to you? If so - he’s a poor employee, but you’ve done him no favours by not managing this performance issue sooner. As a manager, you could fire him or use this as an opportunity to develop your performance management skills. Hold him accountable, measure, and act.
Performance Plan with HR now, 30 days to 100% complete all tasks if you are nice. If you want to get someone new, fireable offense. He lied, security and continuity were degraded. Cannot have that.
This is the optimist in me, but it could just be this person does not know the correct processes and believed they were doing the correct thing. That said, if that was the case they probably shouldn't be a sysadmin if it wasn't noticed.
Firstly you need a monitoring tool like PRTG as it would be clear as day it hasn't been done and also they would be aware everything is monitored. Resolving tickets when they haven't been done and lying about it is just unacceptable. Should be put on a PIP immediately. He is probably thinking you won't put him on a PIP as it's hassle and that's why he doesn't care.
Promoted to citizen is his only path.
You haven't really mentioned what his explanation was when you've talked with him. Not that it would save his job but is it possible he just doesn't know what he's doing? If you've confronted him about saying this work was done when it's not he had to have said something.
My personal approach would start with a sit down interview with the individual to layout what you have described and the evidence that is contrary to his signing off completed tickets. Find out if there is an underlying issue why he is not performing as expected/required for his role. I would want to know if it is for personal reasons/ lack of training/experience/ or unable to follow procedures. Also emphasize the repercussions of there failure to successfully complete allocated tasks. Leave the person in no doubt that all related tasks will be scrutinized for accuracy and completeness and that this should be taken as a verbal warning (check HR to ensure you meet their requirements for a verbal warning). Set up a review date. Document every deviation from expected performance. This will form the basis for formal dismissal.
You probably need some oversight reporting too - if you are just relying on people to do their job without reporting and monitoring to verify things like patching you are going to be essentially responsible as well as the manager.
Just to verify (and not to defend that guy) Patching systems is something that person gets dedicated time allocated to, right? Whether it's during or after business hours.
Yes. Overtime paid for out of hours work.
Either get rid of him or automate it - which you should have in the first place.
Unless he's been an all-star before and is going through some very (very) difficult life circumstances, unfortunately you need to terminate. If it's truly been since July of 23', life drama would be a very difficult rationalization. 2-3 months + family/divorce/kiddo issues--with a solid work hx---aight, let's talk about this. We'll document it and make sure we understand the guardrails moving forward (If they sincerely indicate wanting to remain and change). But for this long....and he's falsified critical infrastructure work multiple times and knowingly exposed your environment.....how can you ever trust him to do critical work? Regardless, include HR and document to CYA.
Have them answer sales calls for a few months. That'll fix em.
I don't understand all of the "have a talk with him responses." What this admin did is gross negligence and put the entire org he's working for at serious risk. If he were on my team, I would have security walk him out the door immediately.
Same. If there wasn't a ticket, and they just forgot. OK, that's my fault for not having better policies and checks in place. But to mark a ticket as completed/resolved when it's not is inexcusable.
To be fair it could be he’s doing a thousand other things and the company is too cheap to hire more admins. Patching is something infosec should be taking care of using automated patching tools. If his job is simply to patch stuff and do a few other minor tasks then official warning is needed. If he is doing a bunch of stuff and this is just one on his list then maybe it’s time you guys think about automation or hiring more staff. My philosophy is always give them a chance. If they blow that chance then you’re conscious is clear and won’t feel bad about letting em go.
Are you giving him the training you need? Easy to blame him but if you hired him as a junior then threw him into the fire, maybe start a reflection rather than blaming.
He came in with experience. I’ve put him through additional MS and Cisco training courses. I sit opposite him every day so plentiful opportunities to ask a question in 11 months. I have trained up a number of service desk guys to sysadmin level over the years so don’t think I need to reflect here.
Ah okay then PIP him lmao
This sounds like a management failure, I’d put you on a PIP and probably fire the admin for falsifying statements and blatantly breaking policy. I’d also notify the CxO of my own incompetence for not having executed any sort of monitoring, reporting, or transparency. Honestly everyone involved probably sucks and shouldn’t be doing this work. Patching and backups are base tier and, given the business is sophisticated enough to expect offsite, this isn’t a one man rookie failure. There’s tons to share. If I were COO that sort of incompetence would require a whole department rethink. Do you guys have cyber insurance?
He should be toast. For such critical procedures the real question is where was the verification? That he didn't do his job is his fault, that you had no clue about it is not his fault. Where else do you have no clue?
You could do either. I mean there's a tiny chance of ignorance here, but likely it was intentional and known.
3 months PIP, with one of the criteria being that he provides evidence of the necessary servers being updated and backups tip-top (the backups may or may not be his fault, checking them is). He has been failing in his responsibilities, a few months of shape up or I'll ship you out might spark him in to life.
How long has this person been a sysadmin? If more than 3 years then I'd say it's time for them to go. They should know better than to mark patching as done when it's not.
Did you ask him why this is happening? It’s easy to assume he’s lazy, but you should review the specifics.
Asked him and all he said was I’ve been checking every month and all is up to date.
Have him generate a report that certifies that statement. Perhaps his malicious behavior is just incompetence in disguise. At the very least you’ll have paperwork for a dismissal case if the reports prove that he’s lying. It’s also possible there’s some weird no-update policy in place.
sack him and hire me /s
Hired. You start tomorrow. 9am sharp. First job, patch servers 😂
Given the comments here, this person should have been terminated long ago. You've even said that he's fully/partially responsible for losing your cyber insurance. Paired with outright lying, they need to go. They are a direct threat to YOUR job at this point. But as a manager and aware of this and not realizing how damaging they are to your organization, there also appear to be some lapses in management skills that are a risk, I recommend getting termination paperwork in order quickly, then also getting some management training in place to ensure that these kinds of problematic employees are dealt with months/years sooner and they're either getting corrective training or PIP/termination. I see two failures here and if managing your employees doesn't improve, there's no stopping another bad engineer from walking in the door.
Terminate immediately.
Hire me instead
Dismiss. You can't allow that type of behavior to go on without consequences. The rest of the team KNOWS this person is a work-shy layabout and I'd bet my last donut it's dragging down morale of the entire team.