Random photos, comics, etc.

[Tony Bandiera Jr]

"The more they overthink the plumbing, the easier it is to stop up the drain." - Montgomery Scott (James Doohan, RIP)

Microdork's Winblows OS is a prime example.

I have gone through the steps to disable and remove all the bloatware from Winblows 11, only to have it magically reappear every time it tries to do updates, which I have set to do manually....but somehow it reinstalls/re-enables services that I have disabled.

My next step will be to block connections to Winblows updates on my firewall.

[Mark Gulbrandsen]

Randy, I love that his last name is Plumber... very fitting!

[Randy Stankey]

Okay, the guy who dropped it onto the server.

I watched a video, yesterday, that explained the problem and, oh boy! Somebody's got some 'splainin to do!

https://youtu.be/wAzEJxOo1ts?si=ZK5XXCFc1rdGxJHZ

Apparently, the Crowdstrike software runs as a driver, inside kernel space, but updates are sent out as P-code which resides outside of kernel space. Essentially, this allows unsigned code to be run inside kernel space without authentication.

This isn't just an "Oops! My bad!" moment! Somebody screwed the pooch!

[Mark Gulbrandsen]

I certainly wouldn't want to be the guy who pushed the button that sent out that update!

Oh, they'll say... that was done by a computer, automatically. Not by a person!!

[Randy Stankey]

You're right. Sic the lawyers on 'em!

Just as you suggest, I'm sure that there are plenty of lawyers already knocking at their proverbial doors.

From a quick check of the internet, the company's net income for FY-24 was close to $90 million. Damage estimates for this incident are likely to top a billion! I know that finances don't work exactly like this but, from some thumbnail math, Crowdstrike has likely screwed themselves for the next ten years.

I certainly wouldn't want to be the guy who pushed the button that sent out that update!

[Leo Enticknap]

My second call would be to the guy in charge of CrowdStrike... The entire conversation would be something like, "We're done." <click>

I suspect that in many of these scenarios, the second call will be to an attorney, with his or her brief being to get medieval on CrowdStrike's ass. CrowdStrike's liability insurer has likely received a few calls by now, too.

[Martin McCaffery]

"Oops! My bad," is NOT an acceptable answer!

Fortunately, customer service is NEVER allowed to say the company made a mistake.

[Randy Stankey]

In your opinion, what would be an acceptable answer?

When I was managing the stage at Mercyhurst, I had to train new students to work in the loading gallery, 50 ft. above deck. I taught them how to load 20 kg., cast iron counterweights onto the appropriate line sets. The rule was that you NEVER held weights with only one hand. Always two hands. I explained that, if a weight falls fifty feet to the deck, equipment is going to be damaged and people could easily be killed.

I also told students, "There is no punishment for dropping counterweights." When people ask, "Why? What do you mean?" my answer is, "Because it doesn't happen!"

In the same respect, I can't think of an acceptable answer because it shouldn't have happened in the first place.

I'm not trying to be a hard ass. I don't expect things to be perfect all the time. Yes, I understand that people make mistakes. So do I.
All my life, since I can remember, people told me to work hard, to do things right, to try my hardest. All my life, I struggled to get things right. Come to find out, I have a learning disability that I didn't discover until I was over forty, almost fifty. However, I got through, okay, because I tried my hardest. I learned to cope with problems and work around them. I know how hard it can be but it pisses me off when people who don't have problems like I do slack off, fuck up and don't seem to care.

If I can learn to do things right when I have to try twice as hard just to be thought of as half as good, why can't other people, who don't have disabilities, do their jobs right the first time?

Airlines, at least, should have a separate data center of their own at each airport. If this country ever had a terrorist attack we'd be at a giant shand still again.

Yeah! You, me and a bunch of other people at FT have been saying the same kinds of things for how long?

How long will it be before we get another, "Oops! My bad!"

[Mark Gulbrandsen]

Frank, Southwest likely has their own data centers. I think you'll see more airlines building their own centers.

[Frank Cox]

"Oops! My bad," is NOT an acceptable answer!

In your opinion, what would be an acceptable answer?

Southwest is still flying high, unaffected by the outage that’s plaguing the world today, and that’s apparently because it’s using Windows 3.1.

This article is highly misleading. Southwest is still flying because they are not a Crowdstrike subscriber. Just like any other company that's not a Crowdstrike subscriber, regardless of whether they're using an obsolete version of Microsoft Windows, a current version of Microsoft Windows, or Linux or anything else.

[Mark Gulbrandsen]

Screw Crowd Strike! Airlines, at least, should have a separate data center of their own at each airport. If this country ever had a terrorist attack we'd be at a giant shand still again.

[Randy Stankey]

That was more of a rhetorical statement, of course.

Still, I would have my people looking into alternatives beside CrowdStrike and I would be calling CrowdStrike up to say, "You fucked up! What are you going to do about it?" I wouldn't accept any pat answers and I certainly wouldn't let them hide behind any flimsy terms in some EULA.

Their stupid and avoidable screw up cost just cost thousands of companies, uncountable millions of dollars that are far in excess of the money they charged for their software. Sure, people shouldn't allow software to be updated automatically but, when you're talking about critical infrastructure the stakes are a lot higher and everybody who's got skin in the game needs to take responsibility.

When it all comes out in the wash, though, the buck stops at Crowdstike's doorstep. They are the ones who, ultimately, need to make things right.

"Oops! My bad," is NOT an acceptable answer!

[Mark Gulbrandsen]

Randy,
There may not be an option for many of these system users to move, and no one updates millions of computers all at once, that is beyond stupid. It will take weeks, if not months to get all this mess cleaned up. End of story.

[Randy Stankey]

The bigger problem is that it's millions of computers that are all over the world...

No, the problem is that it happened on mission critical computers, all over the world.

I would agree that the chances of something like this happening are small... less than 1%. However, if something happens, the chances of people being hurt, killed or negatively affected in some way is 100%.

If computers at banks go down, people can't access money and their lives grind to a halt. If somebody needs that money for an important purpose, they could be harmed. Even if that doesn't happen, businesses could be negatively impacted.

If computers at the airlines go down, millions of people are affected or, in an extreme situation, people could be killed.

If this problem only happened to home computers, it wouldn't be such a big deal. In a situation like that, edge cases don't matter so much. However, if you're talking about critical infrastructure, edge cases DO matter!

It's common for people to decide how much time, effort and money to put into preventing problems based on the probability of something bad happening. That makes sense. People don't hire armed guards to protect their houses, 24/7 because the chances of being burgled are pretty low. However, I'm certainly going to have armed guards protecting the White House because, even though the chances of somebody getting in are relatively low, IF somebody gets in, it could cause problems for the whole country.

In other words, gauge the amount of trouble that something might cause to determine how much of your resources to put into prevention of a problem, not just the chances of it happening.

If I was the boss of a company that got affected by this malfunction at CrowdStrike, I'd be making two phone calls:

My first call would be to my the head of my IT department to tell them to get something else, beside Crowdstrike, immediately and forthwith.

My second call would be to the guy in charge of CrowdStrike... The entire conversation would be something like, "We're done." <click>

[Mark Gulbrandsen]

Leo, there are mini computers today, used for ticketing a lot in Cinemas, that could easily serve as local computers on a big network. I got rid of my big workstation several years ago and run an HP Z2 G6 Mini computer. It has most of the features of the larger workstations. There are also monitors from all the big manufacturers that have the Mini computer built in.

Now if it's a VM setup, that makes the fix far more difficult as you pointed out. I suspect that in the end computer techs are going to have to go to every site and do the fixes locally, then verify the fix.

What a nightmare!