DigitalDilemma @ digdilem @lemmy.ml

Posts

2

Comments

549

Joined

2 yr. ago

I lost a day's holiday, and our team spent 8 man days on this entirely preventable mistake.

$10? Try extending our licence by another year for free, that might start going towards it.

We recently researched these for work.

They tick a lot of boxes - lots of space, reasonable speed, great cold storage figures. Reasonably priced tapes. Agree, they're the best thing. The slow read speed isn't quite as bad as expected (They can go extremely fast in seek mode), but definitely something to consider. We were okay with that for our needs.

But damn, the price of the hardware was horrendous - we got priced (I think) close to £20k for a suitable drive that met our needs. Completely killed the project. And remember that if you're doing site replication for DR, you'll need at least two of them. Sadly, it looks like we'll be using external HDD's for a while longer...

Small number of machines?

Disable unattended-upgrades and use crontab to schedule this on the days of the week you want.

Eg, Monday each week at 4 am - every combination of dates and days is possible with crontab. 2nd Tuesdays in a month? No problem.

0 4 * * MON apt-get update && apt-get upgrade && reboot

(You can also be more subtle by calling a script that does the above, and also does things like check whether a reboot is needed first)

Dozens, hundreds or thousands of machines? Use a scheduling automation system like Uyuni. That way you can put machines into System Groups and set patching schedule like that. And you can also define groups of machines, either ad-hoc or with System Groups, to do emergency patching like that day's openssh critical vuln by sending a remote command like the above to a batch at a time.

All of that is pretty normal SME/Enterprise sysadminning, so there's some good tools. I like Uyuni, but others have their preference.

However - Crowdstrike on Linux operates much like CS on Windows - they will push out updates, and you have little or no control over when or what. They aren't unique in this - pretty much every AV needs to be able to push updates to clients when new malware is detected. But! In the example of Crowdstrike breaking EL 9.4 a few months ago when it took exception to a new kernel and refused to boot, then yes, scheduled group patching would have minimised the damage. It did so for us, but we only have CS installed on a handful of Linux machines.

• HomeAssistant and a bunch of scripts and helpers.
• A number of websites, some that I agreed to host for someone who was dying.
• Jellyfin and a bunch of media
• A lot of docker containers (Adguard, *arrs)
• Zoneminder
• Some routing and failover to provide this between main main server and a much smaller secondary (keepalived, haproxy, some of the docker containers)
• Some development environments for my own stuff.
• A personal diary that I wrote and keep track of personal stats for 15 years
• Backup server for a couple of laptops and a desktop (plus automated backup archiving)

Main server is a ML110 G9 running Debian. 48G/ram. 256 ssd x2 in raid1 as root. 4tb backup drive. 4tb cctv drive. 4x4tb raid 10 data drive. (Separating cctv and backup to separate drives lowers overall iowait a lot). 2nd server is a baby thinkcentre. 2gb ram, 1x 128gb ssd.

Edit: Also traccar, tracking family phones. Really nice bit of software and entirely free and private. Replaced Life360 who have a dubious privacy history.

Edit2: Syncthing - a recent addition to replace GDrive. Bunch of files shared between various desktops/laptops and phones.

Nice analogy, except you'd check the script before you tried to use it. Computers are really good at crc/hash checking files to verify their integrity, and that's exactly what a privileged process like antivirus should do with every source of information.

you lose benefits of economy of scale.

I think you mean - the shareholders enjoy the profits of scale.

When a company scales up, prices are rarely reduced. Users do get increased community support through common experiences especially when official channels are congested through events like today, but that's about the only benefit the consumer sees.

Linux is that it is today thanks to Redhat.

Mmm, maybe - but only if you allow that the same can be said for the tens of thousands of other companies and individuals who have contributed.

Yes. Is it moist under there?

Agreed.

shareholders ... worship money

Well, that literally is the only reason to become a shareholder, right?

I mean, technically you're participating in the management of the company and can influence decisions such as environmental benefits, but it feels like that only happens when there's secondary benefits that also improve profit.

It has a privileged service running locally - csagent.sys - that was crashing causing the BSOD.

And Macs, we have it on all three OSs. But only Windows was affected by this.

The fault seems to be 90/10 CS, MS.

MS allegedly pushed a bad update. Ok, it happens. Crowdstrike's initial statement seems to be blaming that.

CS software csagent.sys took exception to this and royally shit the bed, disabling the entire computer. I don't think it should EVER do that, so the weight of blame must lie with them.

The really problematic part is, of course, the need to manually remediate these machines. I've just spent the morning of my day off doing just that. Thanks, Crowdstrike.

EDIT: Turns out it was 100% Crowdstrike, and the update was theirs. The initial press release from CS seemed to be blaming Microsoft for an update, but that now looks to be misleading.

Am on holiday this week - called in to help deal with this shit show :(

Fair enough, it's good that there's choice.

Interesting, thanks. Those I've spoken to moved from Centos to Rocky when that was killed, and I know of more that moved to Debian.

Half of what you’re writing isn’t really true.

'tis, you know.

Also, sorry, but is it disrespectful when a company drops a project? We could make that same comment about every project. Also, CentOS is open source, as you said, so anyone can download it . They didn’t.

Dropped a project? It wasn't actually their project. I think you're missing some history there. CentOS was a distro started by Greg and Rocky entirely separate from RHEL and ran for many years. Redhat took over CentOS through methods that may be seen as underhand until they had sufficient seats and influence over the Board to have control of it, and then they took/stole the CentOS name. CentOS Linux is an example of a FOSS project that got taken over by a corporate entity, and then killed. (Anyone heard of embrace, extend, extinguish before?) Now CentOS only exists as CentOS Stream, which is repositioned upstream of RHEL and is a staging area/testbed between Fedora and RHEL. Redhat say it's not suitable for production use, so nobody other than testers uses it, afaik.

It seems to be crowdstrike reacting to the new update.

We have got ours up by the very manual process of:

1 Boot into safe mode.

2. Navigate to C:\windows\system32\drivers\crowdstrike
3. Delete C-00000291*.sys
4. Reboot normally