Our evolution was geared towards a world of linearity. You can cover a distance in an hour of walking. You can cover twice as much distance in two hours. The savannah was a good place to use this intuition. It fails catastrophically when it comes to AI and its core exponential trends.
Since I started working on AI in 2010, the training data used to create the most advanced AI models has increased by an astonishing 1 trillion-fold. The amount of data, which is used to train the models, went from 1014 floating-point operations (the core computation unit) of early systems, to 1026 flops of today’s biggest models.
It’s an explosion. This is the basis for all other AI.
They keep on predicting wall. They keep getting it wrong in spite of the epic ramp up in computing power. They often point out the slowing of Moore’s Law. Also, they mention the lack of data or cite energy limitations.
The exponential growth of this industry is quite obvious when we look at all the factors that are driving it. It’s important to look at the fast-moving and complex reality below the headlines in order to understand the reasons.
Imagine an AI classroom full of calculator-using people.
Adding computational power to the room meant more calculators. Many of those employees sat inactive, tapping their fingers against desks while they waited for numbers for their calculations. Each pause wasted valuable time. The revolution of today is not just about better and more calculators, although they are certainly available. It’s also about making sure that these calculators work as one and never stop.
This is now possible because of three advances. The basic calculators became faster.
Nvidia chips increased raw performance by over 7 times in six years. From 312 Teraflops to 2,250 Teraflops. The Maia 200, which was launched in January this year, offers 30% more performance for every dollar spent than other hardware. The numbers are delivered faster by a new technology known as HBM (high bandwidth memory), which allows chips to be stacked vertically, like small skyscrapers. HBM3 triples bandwidth over its predecessor and feeds data into processors at a rate that keeps them occupied all day. The room with the calculators was transformed into an office, and later a campus or city. Technologies such as NVLink or InfiniBand allow hundreds of thousands GPUs to be connected into supercomputers the size of a warehouse that operate like a single entity.
This was unimaginable just a few years ago.
All of these gains add up to a dramatic increase in computing power. In 2020 it took 167 mins to train a language-model on 8 GPUs. Today, the same training takes less than 4 minutes. Moore’s Law predicted only a 5-fold improvement in this time period.
This was 50x. From two GPUs to train AlexNet in 2012 (the image recognition model which sparked the current deep learning boom), we now have over 100,000 GPUs on the largest clusters. Each GPU is far more powerful.
There’s also the software revolution. Epoch AI’s research suggests that computing power required to achieve a certain performance level is halved every 8 months.
This is much quicker than Moore’s Law’s traditional double in 18-24 month intervals. Annualized, the costs for servicing some models has fallen by up to 900 times. AI deployment is getting cheaper.
These numbers are also staggering for the future. Take into account that the capacity of leading laboratories is growing at a rate nearly four times per year. The compute required to train the frontier models has increased by 5x each year since 2020. By 2027, global AI-relevant computing is expected to reach 100 million H100 equivalents. This represents a 10 fold increase within three years.
All this adds up to a 1,000x increase in computing power by 2028. By 2030, we could bring 200 gigawatts more of computing online each year. This is equivalent to peak energy consumption in the UK, France Germany and Italy combined.
All this leads to what? I believe it will drive the transition from chatbots to nearly human-level agents–semiautonomous systems capable of writing code for days, carrying out weeks- and months-long projects, making calls, negotiating contracts, managing logistics.
Let go of basic assistants who answer your questions. Imagine teams of AI workers who collaborate and decide. We’re still in the early stages of the transition. The implications go far beyond technology. The entire industry built around cognitive work is going to be transformed.
Energy is the obvious problem. One refrigerator-sized AI rack uses 120 kilowatts of power, which is equivalent to the energy consumption in 100 homes.
This hunger is matched by another exponential. Solar costs fell nearly 100 times in 50 years, and battery prices dropped 97% within three decades. A path to clean scale is in sight.
Capital is being deployed.
Engineering is on the job. These $100 billion clusters are not science fiction. These projects are now being implemented in the US as well as around the globe. We are now on the road to true cognitive abundance.
This is the future world that our Microsoft AI superintelligence lab plans and builds.
Skeptics used to living in a world of linearity will continue to predict diminishing returns. The will be surprised. This is our technological story, period. It is only the beginning.
Mustafa Suleyman, CEO of Microsoft AI.