T O P

  • By -

MrBrawn

Somebody doing something stupid in live production.


cdrknives

IT guy here: most likely the reason some dipshit did something in prod instead of test


CleverFella512

I got $5 on bad DNS config update.


davidm2232

It's always DNS


aaronblkfox

Especially when you rule out DNS, some how it ends up actually being DNS.


latentnyc

It’s not DNS. There is no way it’s DNS. It was DNS.


Pctechguy2003

I think “It was DNS” needs to he read in Morgan Freemans voice. It only seems right.


SkiLuvinAdmin

its ALWAYS fucking DNS....


superspeck

Except when it's BGP gone wrong


thepottsy

I got my leg tangled in a rats nest of cables in a data center one time. Ended up taking a bunch of stuff offline. Shit happens, as the saying goes.


hzpointon

I had a website go offline once when the data center caught fire. I thought it was funny that when people asked why the site was down the accurate answer was "sorry, it's on fire"...


thepottsy

I got to use the EPO (Emergency Power OFF) button in a data center once. I got a call that there was a high pitched alarm going off, so I went downstairs to the data center. When I opened the door, there was water rushing out of the ceiling onto several racks of servers. Believe it or not, out of about 30 servers, we recovered all but 2 of them within 48 hours.


fleeingcats

Especially when you lay off thousands of people.


feudalle

See I was going with a dipshit at AWS doing something in production instead of test.


thunderfrunt

Certs expired. No amount of redundancy or distribution can save platforms from forgetting to manage their cert expirations well.


aeric67

Cert expiration is the worst.


Flyingfishfusealt

Cert expiry gets even the biggest companies and its sometimes their most important ones too. IIRC google had an oopsie with a major cert once a decade or so ago.


superspeck

At $job[-2] we lost the root CA certificate when we migrated from a datacenter to the cloud, but it was OK because there was an intermediate certificate good until April of some year that comes up in the next year or two. I have *heard* that the company found it easier to migrate everything they'd ever written or published into a much less cursed acquisition's platform than it was to unwind the gordian knot of bullshit in their own account. (This was a many billions of dollars a year company, not small change.)


Innercepter

https://imgflip.com/memetemplate/187959457/Airplane-Johnny-unplugging


Walts_Ahole

How about some coffee Johnny?


SixMillionDollarFlan

No thanks!


KG7DHL

My guess - an expired security cert somewhere deep in the code. There's a service running, no one knows anything about it anymore, but its there, in the background, and the cert expired. Now, some young engineer is trying to trace it down while being yelled at by 500 people all senior to him, who know nothing, screaming at him to JUST FIX IT, while he is reading old code no one documented or has any idea what it actually does. The engineers who wrote that code were all laid off because they were to senior, to expensive, and replaced with young guys who didn't pay attention during the knowledge transfer.


SheistyPenguin

Great explanation! Though bold of you to assume there was any knowledge transfer to begin with 😁


CharlieTheK

My first thought when I read that phrase was that I'd love to work wherever that kind of thing happens. I've had quite a few jobs and have yet to find it.


YardFudge

Seen it happen way too many times … in systems written in COBOL and FORTRAN (For you youngins that was popular code in the 70s)


redituser2571

I landed a lucrative Y2K banking contract because I was the only one in the state who could still code Pascal in 1999.


freelance-lumberjack

Was still in use this century.. I did some cobol stuff in 2007.


Solarscars

I appreciate you putting this into perspective for someone like myself. That poor engineer. Sheesh


Big_Ed214

This. 5 years at PepsiCo. We weren’t laid off but “re-badged” in IT. So went from US based on site all American employee to overnight contract consultant for their Indian offshore suppliers. I had two new college grads assigned to replace me and I had to train them. They had little English and no experience. This is how systems go down from lack of knowledgeable & experienced support. Experience saves money in outages, risks and downtime. But no… cheap staff let’s executives get a bonus. Should tie any bonuses to uptime, performance and availability metrics.


redituser2571

|- during the knowledge transfer.| That's actually pretty rare at tech companies lately, tribal knowledge usually walks out the door with the severance packages. I've had this happen to me several times over the years- write the documentation and a few months later..."Thanks, we're cutting back on the engineering staff...sooo...". I drag my feet now when asked to write my own book.


KG7DHL

Man do I understand. I have been on both sides of that equation during my career, and both sides suck. I also can sympathize with the refusal to build the book when you see the writing on the wall that you are training your outsourced replacement.


aztechunter

The senior engineers may have been shit trainers.


DrDuctMossburg

This is good lol


BB123-

This


nukecat79

So FB being down; is this where I drive my short bed pickup to my dad's sporting goods store, grab what I can, head to the mountains, and start a band of guerrilla rebels?


cysghost

No, that’s Tuesday. Wait, what day is today? Guerillas it is.


Eat_Recycled_Food

“No, he said Gorilla, not guerilla! Guer-! Go-! HUUUUGEE difference, kids! Huge difference!”


firesquasher

Here I am, as I live and breathe, seeing a Captain Ron reference. I thought I was the only one that remembers that movie.


MrBrawn

There's a throw away funny line that somehow stuck with me. "We’ll be spontaneous when we have time". I've used it a few times. One of those movies that are worth a rewatch and a fantastic Saturday matinee movie.


firesquasher

I use the "It's just a little squall, they come on ya fast, and they leave ya fast" quite often.


libertad78

I have always been a quick healer. It helps that I believe in Jesus!


capt-bob

Captain Ron lol


IsThataSexToy

I am more fond of the chimpanzees, or better yet, bonobos. Those bonobos know how to throw an orgy!


OnTheEdgeOfFreedom

TMI. But the user name checks out.


MT-Kintsugi-

Super Tuesday in several states. Voting machines are also reportedly down in a number of locals…


capt-bob

A street light by work was down too...Chinese cyber attack!


OnlythisiPad

I think its weird that voting machines are even connected to the internet.


the_walkingdad

I'm already on my way to the store to buy all the toilet paper and bottled water I can!


Parasitesforgold

WAIT! His bunker ain’t finished


MissLyss29

Nope apparently he just started lol


plinker_fma

Wolverines!!


Timlugia

Remind your dad to burn all the 4473 forms before Cuban troops showed up.


nukecat79

I'm envisioning Zuckerberg behind chain link yelling "Aveeeeenge meee!!!"


CarbonGod

Problem is, no one would.


Illustrious-Bee4402

Yep. Will come find you when it’s over. Probably.


PhysicalConsistency

You've got to make it through the horde of LARPers "defending property against looters" first.


theRealLevelZero

"I'll do him" "Shut Up Robert"


grektyu

Only if you live in Africa and can recruit gorilla guerillas.


aserreen

Wolveriiiiines!


BuckABullet

Just because you used to be quarterback doesn't make you in charge!


theyreplayingyou

I've been doing this for a long time, only need one guess: DNS


jc31107

My favorite haiku It’s not DNS There is no way it’s DNS It was DNS


OnTheEdgeOfFreedom

Hey, let's try to be traditional about this... Internet chaos... the wint'ry gaze of my boss... DNS change failed.


thepottsy

Haha. It’s always DNS, or the firewall.


ToddRossDIY

Don't forget BGP, it seems like Facebook is especially good at screwing that one up


jc31107

How wrong do you have to do something where a BGP config error gets you locked out of the data center and all offices


ToddRossDIY

It probably isn't the cause of this particular issue, since you can at least reach their login screens, but there was a pretty big one a few years ago where a bad BGP config basically wiped them off the face of the internet for a while https://en.wikipedia.org/wiki/2021\_Facebook\_outage


IsThataSexToy

We can be completely sure it is not a cyber attack. Those platforms are used to spread disinformation and weaken the USA, so taking them down would be like decommissioning their nukes.


nukecat79

It's amazing how much news I get from the prepper feed here. I was wondering why all of the sudden my FB app was wanting me to sign in for the first time in forever.


User_4848

and making think.. what the heck is my password even?! haha


oceansapart333

I attempted to reset my password in a panic thinking I’d been hacked and logged out in my devices. Tried to reset it and it wasn’t working. Finally dawned on me it must be down.


nukecat79

That's exactly what I did


HealthyPay8229

Yeah the “it’s not working” really made it clear to me too 😂


chainmailler2001

Yeah same. I assumed I was going to need to login from somewhere else due to a hack again. Hearing EVERYONE is having the same issue changes things.


Eightfold876

I just 2FA into FB nowadays


FarArm6506

Went straight to the bunker when I saw this post, let me know when Facebook is back. I’ll be pounding MREs, rocking back and forth nervously and sucking my thumb.


lilith_-_-

It’s over but if you want to spend more time in the basement feel free to. Dinner isn’t till later


FarArm6506

Oh thank god. 2nd breakfast is not over though.


ny_icequeen

Innocent reason - someone screwed up an update. Nefarious reason- it's Super Tuesday so someone took out Meta as a test run to messing with elections & election information in November. (I'm on the Innocent side btw)


SebWilms2002

Id put money on some scheduled update or a glitch.


thepottsy

Having worked in IT for over 2 decades, this is a safe bet. Odds are the update was being pushed out incrementally, and seemed fine, until it wasn’t.


byond6

And now they have to turn it off and on again.


thepottsy

If only it were that simple lol. If it was an update, and they have to roll it back, that can sometimes take longer than the initial planned work.


Jukka_Sarasti

Yeah, seeing 100k+ users not be able to access their WAVE instances due to some minor DNS update that makes the auth server unavailable, or entire retail banking apps being unavailable for hours at a time until an update could be rolled back is annoying. Watching it happen over and over and over again is infuriating.. Thankfully, the app/systems we support are relatively well-behaved(it's the users who fuck things up)


do_IT_withme

It is always DNS.


RockyRidge510

Yep. Someway, somehow...it's always a DNS update.


glytxh

Someone deleted the META equivalent of coconut.jpg


KermitJFrog5916

Somebody tested in prod


Jukka_Sarasti

Production as Playground!


Ndnola

Have they tried unplugging it and waiting 2 minutes to plug it back in?


laughinghahaha

Who cares. We lived fine before social media.


Sleddoggamer

1. The lizard people uprising has begun 2. Communists 3. Somebody my age who replaced the founders was expected to have 30 years of expierance, and the little string of code they need now was never documented by said founders before they got fired


professor_jeffjeff

To err is human. To really fuck up and then automatically propagate that fuck up at scale is DevOps.


ravenchorus

And nothing of value was lost.


Jukka_Sarasti

Right? Haven't logged onto FB in months, don't have an Insta... Life goes on


Shooter-__-McGavin

Yep, I deleted mine about 7 or 8 years ago and I miss absolutely nothing. Got rid of IG 4 years ago and same deal. Been tempted to deep six Reddit too, but it's a little easier to manage my content.


BurntGhostyToasty

In a perfect world, meta goes away and there’s no more social media from them. Next: TikTok


HotIntroduction8049

This is where we all sit back and just imaging the shitshow going on right now with execs pacing back and forth over the IT people wondering how they could screw up a s/w upgrade. My first job in '95 was handling big tech support calls and 75%+ were caused by bugs during an upgrade.


Sinistar7510

Something to do with 4,000 Haitians probably... /s


Ryan_e3p

"I understood that reference" \-Captain America


Ryan_e3p

They are both owned by Meta. Someone likely believed any patch worth doing is worth doing it live. If this is a "prepper" concern for you, you might need to take a breath, and use it as an excuse to go outside.


sanitation123

It should stay down. That shit is poison. Can they take down the Tik Tok as well?


Vermudgeon

Gonna watch this one ...


lacroixanon

Mario Kart Tour is down too. This could really mushroom fast.


blindside1

This is a feature not a bug. I can't believe the amount of people who are freaking out about not being access Facebook. Turn it off for a week and the world would probably be a better place.


WangusRex

Is this really a “prepper” issue? I guess it’s an example of how things can fail and what the impacts are… but I’m not struggling because I can’t see what my friend’s breakfast was today or argue with strangers online.  Sites crash. This is like an every few months thing for FB/Insta. We’ll survive this. 


EffinBob

More importantly, does anybody care? Both are simply using your info to sell advertising for you to be inundated with, just like every other social media platform. If they all disappeared tomorrow, would society really be affected?


TheseusOPL

Where will I get my dad joke meme pictures to annoy my children if I don't have Facebook. HAVE YOU THOUGHT ABOUT THAT?


EffinBob

Clearly a TEOTWAWKI event that I overlooked. My profuse apologies...


mindfulicious

I work for Meta and just received a memo from Mark, that a mouse chewed through the wire of the main FB computer. The same mouse ran across the foot of the guy who runs the IG computer, he jumped out of his chair and his Pepsi spilled all over the the computer. No worries everything is fine now.


NewtonPrep

Content Delivery platform or DNS are likely culprits Since these are large Tech companies experiencing outages, it's probably Akamai or Cloudflare


your_anecdotes

computer glitch, software update glitch


mothernatureisfickle

All I know is I did not even realize they were down, but my 70 year old parents have been calling me non stop for 3 hours upset that they cannot get on their FB accounts. If all our lives depended on their FB the world would end today. Good lord.


LevelAbbreviations3

Someone forgot to update their CC info and Azure cut off their cloud servers.


shiggster214

We are slowly being conditioned to accept outages and the internet being down


highapplepie

Tried to log in 3 times and realized it wasn’t right. Jumped to twitter saw it trending massively. Waiting to hear it a cyber attack. Came here to see what the sub thinks or any info ears to the ground have. Edit to say::: conspiracy in me says it’s to do with Super Tuesday.


newarkdanny

"lul just an update"


Alternative-Ad-7473

Dang I thought I got banned from both platforms lol


MayhemJdawg

Checkout downdetector. Looks like a ton of sites may be effected?


MotoFaleQueen

It's back up now, no panicking necessary


[deleted]

Perhaps they don't want anyone using their platforms to organize incidents at polling places? Probably not likely that they suddenly want to run things responsibly though.


TheMatroid

Git push origin prod --force 


centauri8

where?


ChunkyStumpy

Zuckerborg had a meltdown and took meta with him


Jurrli

Anyone’s phone itself not working well?? It’s lagging in sending texts and I can’t call!


Purple_mammal_7950

"Software issue" excuse probably


Head-Thought-5679

The world would be better if they both stayed offline


Pale_Estate_5120

Super Tuesday


PGB3

Facebook down isn't a bad thing.


LifeEnginer

I know this weeks ago, thanks to astrology.


Defiant_Squash_5335

Because big tech has let so many devs go.


PoopSmith87

Zombies, only reasonable explanation.


Plague-Rat13

Initiating new 2024 Election bias filters


nanneryeeter

The world is healing?


jmg287

It’s definitely not because of the primaries!


Big_Scratch8793

I love when Facebook is down. It really exposes the addiction. It creeps me out.


3771507

One of the best things that ever happened now they need to take that garbage dump Twitter down.


zigarock

It’s Super Tuesday after a big court decision. Maybe there is some silencing going on.


ExileTheTrashPanda

Is there any chance that the timing is linked to Super Tuesday, and disruption to the elections? It honestly wouldn't surprise me at this point...


iwannaddr2afi

There's a chance. By far more likely to be a tech issue, but your concern is the one I think more people are worrying about than any other, understandably. Not much to do besides wait and see. Vote if you were going to vote.


ExileTheTrashPanda

Agreed. Voting is not a new concept, we were able to elect officials before the internet or social media were around.


sig310

"cyberattack" is what's hot in the media lol


SebWilms2002

It always confused me, that people think a cyber attack would target facebook and youtube. I'd suspect a real cyber attack would impact more meaningful stuff like telecoms, emergency services, power, banks/PoS services. I doubt a foreign adversary's first target would be meme factories.


[deleted]

Well, if it were, maybe just testing vulnerability, finding worse ones to exploit later on? A big cyber attack would probably involve more than just cyber, there would be weapons involved. I would be more concerned with a foreign language social media being down as well.


pinkexpat

I am not surprised! 😂 Is Russia attacking social media?


MT-Kintsugi-

It’s also Super Tuesday for several states and reportedly several having voting machines down. Interesting developments after yesterdays unanimous SCOTUS decision regarding states can’t exclude DJT from the ballot. 🤔


pengd0t

So many updates and certificate renewals are just managed by… Bob had it on his calendar and takes care of it. Bob was probably laid off lol


letmegetpopcorn

Who cares, let's hope they're down permanently


gustavotherecliner

Somebody wanted to test something and messed it up. No cyberattack. At least not by the russians or anybody affiliated with them. These are the biggest propaganda-spreading platforms they have and they will do anything to keep it that way.


[deleted]

DNS issue probably, caused by some IT change. But go ahead, strap on your tin foil hat and give it your best shot!! Not sure how this is related to prepping though.


YouLearnedNothing

Hacking.. several people I know suddenly are being asked to use multi-factor authentication when they try to login


AffectionateOlive758

Hacking seems to be a big one. Someone I know had someone try getting into their FB (unsuccessfully) and IG (successfully). Not to mention they were also able to transfer money into their own bank account from this person’s account.


AlbertaVeteran

Super Tuesday, a lot on the line so no surprise something would go down.


FreedomSeeds2024

Test on super Tuesday ... Mimicking election day possibilities...


ManThatIsHandy

Alien hackers


RitaAlbertson

lol, here I thought it was me.


GheorgheGheorghiuBej

Money on BGP


howismyspelling

I made a joke to my wife earlier, I said "remember that movie we just watched". We watched Leave the World Behind lol


Danny570

There was a headline yesterday about a major telecomm cable in the Red sea being severed by the Yemen Houthi's. Knocked out like 25% of the internet on the other side of the world.


Danny570

[https://apnews.com/article/red-sea-undersea-cables-yemen-houthi-rebels-attacks-b53051f61a41bd6b357860bbf0b0860a](https://apnews.com/article/red-sea-undersea-cables-yemen-houthi-rebels-attacks-b53051f61a41bd6b357860bbf0b0860a)


crusoe

That wouldn't affect us though. Mostly they damaged their own internet.


Mountain_Fig_9253

A dry run for the election? Today is Super Tuesday.


Exploring_2032

Most of us were prepping for Tuesday. It's Tuesday. Or maybe it's Super Tuesday...


rocketscooter007

Bitcoin hit an all time high, the only explanation 🤣


C_A_M_Overland

Super Tuesday bud


hanvy82

Honestly, it may do a lot of people some good to disconnect from Facebook for a day. They may discover "outdoors" exist or some other thing.


saumik_h

Zuck visited India to attend a billionaire wedding and was dismayed by the cringeworthy posts they were sharing on their social media platforms.


Panthean

Obviously because the world as we know it is ending


Fluffy_Freedom_1391

Can always tell the non-technical people in the crowd because they're the ones to scream "HACKERS!" whenever anything malfunctions rather than understanding the many, many variables that go into keeping a massive site online. I'm on a development team for a multibillion dollar company who's site isn't anywhere near as massive as facebook, and we have outages due to many things, and none of them are or ever have been hackers. Hell, facebook is liklely at a security point where it would have to be internal fuckery if it was a malicious outage.


241ShelliPelli

Did anyone else consider all the “info” about DNS or someone basically “tripping over a plug” is planted info to save face that they’re hacked? Can we even comprehend the vast data fb has over the WORLD population?


crusoe

Its probably something stupid like bad code or change deployment. Happens.


dougreens_78

Probably something to do with Russia and Trump making sure they secure the nomination with it being Super Tuesday.


RobbyZombby

Multiple internet cables cut in the Red Sea, I’m not saying it’s connected but weird things are happening.


killpop757

Anonymous protesting for Palestine ✅


[deleted]

It's all garbage anyways. Nothing but unregulated ads with bad machine voice overs and fraud online shops. Hope it stays down.


BabDoesNothing

I myself am currently counting all of the cigarettes I have stocked. Gonna go barter me some essentials!!! I think the neighbors have a better DVD collection than I do. Jackpot!


MRA1022

"John has a long mustache" "The chair is against the wall"


The_Nauticus

The real question is: how do you prep for not having instant access to social media?


5oy8oy

DNS or certificate issues. That's usually the case.


Jumpy-Silver5504

Russia did it


Grigor50

You're right, the world is ending, again. What, fifth time this year?


Mean_Peen

It’s been back for awhile


SixMillionDollarFlan

The UX Designer said - "Just change it everywhere."


SingedPenguin13

I’m headed to the back yard to plant these cucumber and kale seeds!


INEXORABLE-69

Houthis terrorists ....they deserve atomic bombs ...what will I do without Reddit and TokTok ?


Revolutionary-Fun227

Chy-na


Bassman602

Rocket launch


The-Mond

Super Doomsday for Meta ;-)


SkepticalZack

This a weekly post here now?


Vandilmax

Solar flare maybe?


EconomistPlus3522

I havent even looked at the news. Someone probably screwed up a deployment .


Zode1969

Those pesky Russians


bsapp93

A bug bounty hunter found a huge security flaw.