It is definitely an under provisioning problem. But that under provisioning problem is caused by the customers usually being very very stingy about what they are willing to spend. Also, to be clear, it isn't buckling. It is doing exactly The thing it was designed to do. Which is to stop writes to the DB since there is no disk space left. And before this time, it's constantly throwing warnings to the end user. Usually these customers tend to ignore those errors until they reach this stop writes state.
In fact, we just had to give an RCA to the c-suite detailing why we had not scaled a customer when we should have, but we have a paper trail of them refusing the pricing and refusing to engage.
We get the same errors, and we usually reach out via email to each of these customers to help project where their data is going and scale appropriately. More frequently though, they are adding data at such a fast clip that them not responding for 2 hours would lead them directly into the stop writes status.
This has led us to guessing what our customers are going to end up at. Oftentimes being completely wrong and eating to scale multiple times.
Workload spikes are the entire reason why our database technology exists. That's the main thing we market ourselves as being able to handle (provided you gave the DB enough disk and the workload isn't sustained for a long enough to fill the discs.)
There is definitely an automation problem. Unfortunately, this particular line of our managed services will not be able to be automated. We work with special customers, with special requirements, usually fortune 100 companies that have extensive change control processes. Custom security implementations. And sometimes even no access to their environment unless they flip a switch.
To me it just seems to all go back to management/c-suite trying to sell a fantasy version of our product and setting us up for failure.
That is exactly what we do. The problem is that as a managed service offering. It is on us to scale in response to these alerts.
I think people are misunderstanding my original post. When I say that customer cluster will go into stop writes, that does not mean it is not functional. It is an entirely intended function of the database so that no important data is lost or overwritten.
The problem is more organizational. It's that we have a 5 minute SLA to respond to these types of events and that they can happen at any random customer impulse.
I don't have a problem with customers that can correctly project their load and let us know in advance. Those are my favorite customers. But they're not most of our customers.
As for automation. As I had exhaustedly detailed in another response, we do have another product that does this a lot better. And it's the one that we are mass marketing a lot more. The one where I'm feeling all the pain is actually our enterprise level managed service offering. Which goes to customers that have "special requirements" and usually mean that they will never get as robust automation as the other product line.
Our database is actually pretty graceful. It just goes into stop writes status. You can still read any data and resolving the situation is as easy as scaling the cluster or removing old records. By no means is the database down or inoperable.
Essentially our database is working as designed. If we rate limited it further then we have less of a product to sell. The main feature we sell of our database technology is its IOPS and resiliency.
Further, this is just for a specific customer, it has no impact to any other customers or any sort of central orchestration. Generally speaking the stop writes status only ever impacts a single customer and their associated applications.
Also, customers can be very stingy with the clusters they are willing to buy. We actually are on poor terms of the couple of our customers who just refuse to scale and just expect us to magic their cluster into accepting more data than its sized for.
Probably not feasible in our case. We sell our DB tech based on the sheer IOPS it's capable of. It already alerts the user if the write-cache is full or the replication cache is backing up too.
The problem is, at full tilt, a 9 node cluster can take on over 1GB/s in new data. This is fine if the customer is writing over old records and doesn't require any new space. It's just that it's more common that Mr. customer added a new microservice and didn't think through how much data it requires. Thus causing rapid increase in DB disk space or IOPs that the cluster wasn't sized for.
We do have another product line in the works (we call it DBaaS) and that can autoscale because it's based on clearly defined service levels and cluster specifications. I don't think that product will have this problem.
It's just these super mega special (read: big, important, fortune 100) companies have requirements that mean they need something more hand-crafted. Otherwise we'd have automated the toil by now.
I don't get it either. My brother-in-law is like this. And he refused to take his kids to see Buzz Lightyear because of its "political" nature. I was a dumbfounded when I heard that. To think that representation is just some nebulous political aim.
At this rate, we should just consider any media with a kiss in it "political media."
And I even grew up with this dude in the early 2000s. He didn't seem like this before.
I try to forget about the guy, but it's kind of hard because he won't let me see the nieces because I'm too "liberal".
I agree. I think 1440p+HDR is probably the way to go for now. HDR is FAR more impactful than a 4K resolution and 1440p should provide a stable 45ish FPS on Cyberpunk 2077 completely maxed out on an RTX 3080Ti (DLSS Performance).
And in terms of CPU, the same applies. 16 cores are for the gentoo using, source compiling folks like me. 8 cores on a well binned CPU from the last 3 generations goes plenty fast for gaming. CPU bottlenecking only really show up at 144fps+ in most games anyways.
Agree, most mainstream distros have it all handled for the most part and it normally "just works".
Now, myself on Gentoo testing on the other hand... Sometimes I shoot myself in the foot and forget to rebuild my kernel modules and wind up needing to chroot to fix things - all because I have an NVidia card.
As somebody with autism. I find this take lacking nuance. You see for me these tools represent a huge leap and accessibility for me. I can turn a wall of stream of consciousness text into something digestible and represents myself.
I find myself constantly exhausted with the societal expectation that I review, edit, and adjust my own speech constantly. And these tools go a long way to helping me actually communicate.
I mean, after all nothing changes for me. People thought of me as a robot before. And I guess they can continue to think I'm still a robot. I've stopped giving a crap about neurotypical expectations.
I mean I take a less extreme take. But I definitely resonate. As somebody with autism, it's really nice to have an impartial chat assistant to turn my stream of consciousness wall of text into something far more digestible. Trying to do so myself often takes hours to construct a message a couple paragraphs long. Where I checked and double check and triple check for anything that might offend somebody or come across strange or not flow well. Etc etc etc.
A lot of these articles don't really investigate the accessibility aspect of these tools. And I really wish they did. I know if one of my friends used chatgpt to help with their messages, I would be completely fine with it.
My favorite thing is that they are all inter-compatible. SWN/WWN/CWN are all essentially the same system and can be used interchangeably.
Sure, as @tissek mentioned, the system is pretty simple and the options for martial characters are a bit lackluster compared to other games, but that doesn't stop us GM's from homebrewing things. In fact, I think that's where Sine Nomine RPG's really shine, the ability to homebrew much of what you need and ready made resources to do it.
I'm rolling with a Halfling Rogue, it's always the class I try to play first in D&D games and versions. I'm also surprised at the lack of Halflings in the stats!
My wife is playing a Githyanki the only race that's less popular. Lol!
Forever GM's unite! (But like, once we get the schedule finalized /s)
Yeah, this is probably the best D&D game in existence now. It definitely has some pretty fun mechanics and a lot of depth that other D&D vidya games just lack.
Step one, do not be in downtown, inner SE, inner NE, Gateway, or anywhere near a Max line or bus station after dark.
Step two, carry mace and a stun gun.
Step three, leave Portland for good and only return if I must << We are here.
We got a lot of hate from certain left leaning folks in Portland for leaving "because of the homeless". It's like, "No, dude, I'm leaving because my wife was assaulted by homeless no less than 3 times (twice physically, once was almost a rape), and that's even when she was "safely on TriMet. You can 'but not ALL homeless' all you want. My wife is traumatized and we want nothing to do with this shithole of a city".
Yeah, after the 3rd one we left, and we can say with certainty that we'll never ever come back to live in PDX.
It is definitely an under provisioning problem. But that under provisioning problem is caused by the customers usually being very very stingy about what they are willing to spend. Also, to be clear, it isn't buckling. It is doing exactly The thing it was designed to do. Which is to stop writes to the DB since there is no disk space left. And before this time, it's constantly throwing warnings to the end user. Usually these customers tend to ignore those errors until they reach this stop writes state.
In fact, we just had to give an RCA to the c-suite detailing why we had not scaled a customer when we should have, but we have a paper trail of them refusing the pricing and refusing to engage.
We get the same errors, and we usually reach out via email to each of these customers to help project where their data is going and scale appropriately. More frequently though, they are adding data at such a fast clip that them not responding for 2 hours would lead them directly into the stop writes status.
This has led us to guessing what our customers are going to end up at. Oftentimes being completely wrong and eating to scale multiple times.
Workload spikes are the entire reason why our database technology exists. That's the main thing we market ourselves as being able to handle (provided you gave the DB enough disk and the workload isn't sustained for a long enough to fill the discs.)
There is definitely an automation problem. Unfortunately, this particular line of our managed services will not be able to be automated. We work with special customers, with special requirements, usually fortune 100 companies that have extensive change control processes. Custom security implementations. And sometimes even no access to their environment unless they flip a switch.
To me it just seems to all go back to management/c-suite trying to sell a fantasy version of our product and setting us up for failure.