Nobody knows how little computation intelligence needs.

Nearly every AI chip on the market runs the same networks as a GPU, in the same number formats, so the answer built into the hardware is: about as much as a GPU uses. We believe the real answer is orders of magnitude lower than what today’s chips use, and we are going to find out.

For messages, the question was answered in 1948. Morse code assigned E, the most common letter in English, the shortest signal. Claude Shannon proved how far that goes1. Every source of messages has a hard lower limit on how compactly its messages can be encoded without loss. Shannon named that floor entropy, measured in bits per symbol. If some loss is allowed, the floor drops, and he worked out that limit too. In the 1990s MP3 and JPEG cut songs and photographs to about a tenth of their size by leaving out what the ear and eye cannot detect, and both could move over the early internet. Each spends fewer bits for the same result.

For neural computation, nobody has reached the floor, and today’s networks run far above it. A network stores and computes every value in a few fixed formats, independent of whether the value carries a lot of information or almost none, because those are the formats GPUs run fast. Sara Hooker named this the hardware lottery2: an idea wins because it fits the hardware and software that exist, not because it is the best idea. Deep learning won the lottery on chips built for graphics. The result is a network that moves and processes far more data than its output needs. Ten years ago, a trained network could already be made 35 to 49 times smaller on disk with no loss in accuracy3. Today’s chips can use only a small part of these savings: they expand every compressed value back into one of their fixed formats before computing with it.

The industry has learned to live with the gap, but each way of living with it is reaching its limit. Faster chips depend on shrinking transistors, and the free part of that gain faded out between 2007 and 2012, when a smaller transistor stopped using proportionally less power4. Each new node gains less than the last, and scaling is nearing its physical limits. Alternatives like analog, thermodynamic and other new-physics chips make the arithmetic cheaper, but each gives up part of what digital silicon has: mature foundries and design tools, predictable yield, devices that behave the same from one chip to the next, and a chip that can be reprogrammed when the model changes. None of them is ready at scale.

Number formats went from 32 bits to 4 in about a decade, but each further step costs more accuracy, because the networks were designed around wider ones. And narrower formats help only where they apply. In a language model, quantized weights make the matrix multiplications cheaper, while normalization and the other operators still run at high precision. However much one part of a computation is sped up, the total gets no faster than the parts left unchanged allow5: even if the matrix multiplications cost nothing, the rest would cap the gain. To go further, the operators themselves have to change, and new operators run efficiently only on a chip built for them.

So the gap has to be closed at design time. A network should use the smallest representation that preserves its capability, carrying no more information than its task requires. Hardware built for that representation should then compute it with the fewest operations and, above all, the least data moved per answer. Reading a value from off-chip memory costs several orders of magnitude more energy than adding two numbers6. The size of the representation therefore sets both the energy and latency of an answer. This is Shannon’s question, asked of computing an answer instead of transmitting a message. Compute has to be Entropy Proportional.

The gap cannot be closed by approaching it from one side. A smaller representation gains nothing on a chip that cannot execute it efficiently. A chip built for a new representation gains nothing without networks that use it. So we develop both, in a feedback loop between network and chip. That changes two things at once: how the arithmetic of a neural network is done, and how data moves between memory and compute. Both work on the information a value carries, which is usually far less than the width of its format. Narrower formats close part of that gap. Closing the rest means changing how networks are trained and what they compute in the first place. The incumbents can do the same only partly. Their chips have to run existing models, so their formats can move only as fast as the whole industry moves. In our case, we convert the models customers already have and preserve their accuracy. We fix the capability and are free to change everything else, on standard digital silicon, with no new physics.

Our performance model7 indicates that the first design, built on an FPGA, will be more than one order of magnitude ahead of the most efficient AI chips on the market. On a chip of our own, the gap would grow to about three to four orders of magnitude. For the complete product, from a robot to a server rack, our target is orders of magnitude lower cost, energy and time per answer than the best system on the market when ours arrives.

A gap that size is what ends an era8: it is the difference between renting intelligence and owning it where it is needed. The answers that matter most can’t always be rented. A bank’s client data, a hospital’s patient records and a ministry’s secrets may leave the building only on strict terms, if at all. A trading race is won by microseconds. A small drone over a jammed field has a few watts and no link. A humanoid’s brain shares a battery with its legs. Wherever cost, control, power, latency or place rules out renting, intelligence has to be owned.

Once an answer takes a tenth of the energy, a bank’s existing server room runs what would otherwise need a new data center, a humanoid carries a brain ten times larger on the same battery, and a model that can see and talk moves off a graphics card and into a camera. Intelligence cheap enough to own will not stay where it is: it will move into every room and every machine that could not afford it before. It will favor whatever computes it with the least energy and cost. We are building that machine.

We are a small team at the very start, with backgrounds in research at ETH, Stanford and in the chip industry. If you want to help find out how little computation intelligence needs, whether as an engineer, researcher or investor, write to us.