How does AI use so much power?

Imagine someone said "make a machine that can peel an orange". You have a thousand shoeboxes full of Meccano. You give them a shake and tip out the contents and check which of the resulting scrap piles can best peel an orange. Odds are none of them can, so you repeat again. And again. And again. Eventually, one of boxes produces a contraption that can kinda, maybe, sorta touch the orange. That's the best you've got so you copy bits of it into the other 999 shoeboxes and give them another shake. It'll probably produce worse outcomes, but maybe one of them will be slightly better still and that becomes the basis of the next generation. You do this a trillion times and eventually you get a machine that can peel an orange. You don't know if it can peel an egg, or a banana, or even how it peels an orange because it wasn't designed but born through inefficient, random, brute-force evolution.

Now imagine that it's not a thousand shoeboxes, but a billion. And instead of shoeboxes, it's files containing hundred gigabytes of utterly incomprehensible abstract connections between meaningless data points. And instead of one a few generations a day, it's a thousand a second. And instead of "peel an orange" it's "sustain a facsimile of sentience capable of instantly understanding arbitrary, highly abstracted knowledge and generating creative works to a standard approaching the point of being indistinguishable from humanity such that it can manipulate those that it interacts with to support the views of a billionaire nazi nepo-baby even against their own interests". When someone asks for an LLM to generate a picture of a fucking cat astronaut or whatever, the unholy mess of scraps that behaves like a mind spits out a result and no-one knows how it does it aside from broad-stroke generalisation. The iteration that gets the most thumbs up from it's users gets to be the basis of the next generation, the rest die, millions of times a day.

What I just described is NEAT algorithms, which are pretty primitive by modern standards, but it's a flavour of what's going on.

It's mostly the training/machine learning that is power hungry.

AI is essentially a giant equation that is generated via machine learning. You give it a prompt with an expected answer, it gets run through the equation, and you get an output. That output gets an error score based on how far it is from the expected answer. The variables of the equation are then modified so that the prompt will lead to a better output (one with a lower error).

The issue is that current AI models have billions of variables and will be trained on billions of prompts. Each variable will be tuned based on each prompt. That's billions to the power of billions of calculations. It takes a while. AI researchers are of course looking for ways to speed up this process, but so far it's mostly come down to dividing up these billions of calculations over millions of computers. Powering millions of computers is where the energy costs come from.

Unless AI models can be trained in a way that doesn't require running a billion squared calculations, they're only going to get more power hungry.

This is a pretty great explanation/simplification.
I'll add that because the calculations rely on floating point math in many cases, graphics chips do most of the heavy processing since they were already designed for this pipeline in mind with video games.
That means there's a lot of power hungry graphics chips running in these data centers. It's also why NVidia stock is so insane.
- It's kinda interesting how the most power-consuming uses of graphics chips — crypto and AI/ML — have nothing to do with graphics.
  (Except for AI-generated graphics, I suppose)
Would AI inferencing, or training be better suited to a quantum computer? I recall thouse not being great at conventional math, but massively accelerates computations that sounded similar to machine learning.
- My understanding of quantum computers is that they're great a brute forcing stuff, but machine learning is just a lot of calculations, not brute forcing.
  If you want to know the square root of 25, you don't need to brute force it. There's a direct way to calculate the answer and traditional computers can do it just fine. It's still going to take a long time if you need to calculate the square root of a billion numbers.
  That's basically machine learning. The individual calculations aren't difficult, there's just a lot to calculate. However, if you have 2 computers doing the calculations, it'll take half the time. It'll take even less time if you fill a data center with a cluster of 100,000 GPUs.

will there be huge power plants just dedicated to AI all over the world soon?

Construction has started(or will soon) to convert a retired coal power plant in Pennsylvania to gas power, specifically for data-centers. Upon completion in 2027 it will likely be the third most powerful plant in the US.

The largest coal plant in North Dakota was considering shutting down in 2022 over financial issues, but is now approved to power a new data-center park.

Location has been laid out for a new power plant in Texas, from a single AI company you've probably never heard of.

And on it goes.

Data is the new oil. Collecting it, refining it, and distributing it.

My understanding is that traditional AI essentially takes a bruteforce approach to learning and because it is hardwired, its ability to learn and make logical connections is impaired.

Newer technologies like organic computers using neurons can change and adapt as it learns, forming new pathways for information to travel along, which reduces processing requirements and in turn, reduces power requirements.

https://www.techradar.com/pro/a-breakthrough-in-computing-cortical-labs-cl1-is-the-first-living-biocomputer-and-costs-almost-the-same-as-apples-best-failure

https://corticallabs.com/cl1.html

Machine learning always felt like a very wasteful way to utilize data. Even with ridiculous quantities of it, and the results are still kinda meh. So just dump in even more data, and you get something that can work.
Lower voltage chip advancement along with better cooling options may come along some day.
They should consider building their super centers underwater in places like Iceland.
- Thats only a short term solution, global warming will negate those benefits.
  Southern ocean currents just reversed and will likely cause rapid warming of water temps.
  Southern Ocean circulation reversed
  Southern Ocean current reverses for first time, signalling risk of climate system collapse
  France and Switzerland just had to shutdown their nuclear reactors due to the water sources they use for cooling being too warm.
  France and Switzerland shut down nuclear power plants amid scorching heatwave
  When heat halts power: Europe’s nuclear dilemma

OpenAI noticed that Generative Pre-trained Transformers get better when you make them bigger. GPT-1 had 120 million parameters. GPT-2 bumped it up to 1.5 billion. GPT-3 grew to 175 billion. Now we have models with over 300 billion.

To run, every generated word requires doing math with every parameter, which nowadays is a massive amount of work, running on the most power hungry top of the line chips.

There are efforts to make smaller models that are still effective, but we are still in the range of 7-30 billion to get anything useful out of them.

will there be huge power plants just dedicated to AI all over the world soon?

It takes time to build a power plant. A more realistic scenario is that we'll continue as we have: AI centers will be built wherever local governments approve them for the taxes, without regard for the strain they put on the aging electrical grid and, given the massive amount of electricity they need, everyone's electrical bill will just massively increase.

They've been building a large number of data centers and AI centers in Virginia, and it's been straining and raising prices across the entire PJM interconnector region, to the point where at least a couple states are considering leaving it. Microsoft has bought the rights to and is reactivating part of the Three Mile Island nuclear plant, because they wanted dedicated power, and they're still going to be pulling power from the grid.

Also water, they consume heaps of fresh water which is used for important meat bag things like, oh I don't know, eating and drinking perhaps.
No one is really challenging them on this, but water scarcity is going to be a big deal as climate change worsens.
Cook the planet and take all the water.
- And growing plants that we eat.
- Isn't water mainly used for cooling? I think you can still drink that water, unless they pump it full of chemicals.

The current algorithmic approach to AI hit a wall in 2022.

Since then they have had to pump exponentially more electricity into these systems that result in exponentially diminishing returns.

We should have stopped in 2022, but marketing teams had other plans.

There's not a way to do AI and use less electricity than the current models, and there most likely won't be any more advances in AI until someone invents a fundamentally different approach.

Supercomputers once required large power plants to operate, and now we carry around computing devices in out pockets that are more powerful than those supercomputers.

There’s plenty of room to further shrink the computers, simplify the training sets, formalize and optimize the training algorithms, and add optimized layers to the AI compute systems and the I/O systems.

But at the end of the day, you can either simplify or throw lots of energy at a system when training.

Just look at how much time and energy goes into training a child… and it’s using a training system that’s been optimized over hundreds of thousands of years (and is still being tweaked).

AI as we see it today (as far as generative AI goes) is much simpler, just setting up and executing probability sieves with a fancy instruction parser to feed it its inputs. But it is using hardware that’s barely optimized at all for the task, and the task is far from the least optimal way to process data to determine an output.

Supercomputers once required large power plants to operate, and now we carry around computing devices in out pockets that are more powerful than those supercomputers.
This is false. Supercomputers never required large [dedicated] power plants to operate.
Yes they used a lot of power, yes that has reduced significantly, but it's not at the same magnitude as AI
It is also a very large data set it has to go through the average English speaker knows 40kish words and it has to pull from a large data set and attempt to predict what’s the most likely word to come next and do that a hundred or so times per response. Then most people want the result in a very short period of time and with very high accuracy (smaller tolerances on the convergence and divergence criteria) so sure there is some hardware optimization that can be done but it will always be at least somewhat taxing.
Your answer is intuitively correct, but unfortunately has a couple of flaws
Supercomputers once required large power plants to operate
They didn't, not that much anyways, a Cray-1 used 115kW to produce 160 MFLOPS of calculations. And while 150kW is a LOT, it's not in the "needs its own power plant to operate" category, since even a small coal power plant (the least efficient electricity generation method) would produce a couple of orders of magnitude more than that.
and now we carry around computing devices in out pockets that are more powerful than those supercomputers.
Indeed, our phones are in the Teraflops range for just a couple of watts.
There’s plenty of room to further shrink the computers,
Unfortunately there isn't, we've reached the end of Moore's law, processors can't get any smaller because they require to block electrons from passing on given conditions, and if we built transistors smaller than the current ones electrons would be able to quantum leap across them making them useless.
There might be a revolution in computing by using light instead of electricity (which would completely and utterly revolutionize computers as we know them), but until that happens computers are as small as they're going to get, or more specifically they're as space efficient as they're going to get, i.e. to have more processing power you will need more space.
This is an astute answer. Bravo.

imagine that to type one letter, you need to manually read all unicode code points several thousand times. When you're done, you select one letter to type.

Then you start rereading all unicode code points again for thousands of times again, for the next letter.

That's how llms work. When they say 175 billion parameters, it means at least that many calculations per token it generates

That’s how llms work. When they say 175 billion parameters, it means at least that many calculations per token it generates
I don't get it, how is it possible that so many people all over the world use this concurrently, doing all kinds of lengthy chats, problem solving, codegeneration, image generation and so on?
- that's why they need huge datacenters and thousands of GPUs. And, pretty soon, dedicated power plants. It is insane just how wasteful this all is.

If people continue investing in AI and computing power keeps growing we would need more than dedicated power plants.

It takes a lot of energy to do something you are not meant to do, whether that’s a computer acting like a person or an introvert acting like an extrovert