OpenAI Jalapeño: what a dedicated AI inference chip changes

OpenAI published the first results of Jalapeño, its first chip designed specifically for inference. Behind the flow and consumption figures is an important development: the performance of an artificial intelligence service no longer depends solely on the model, but on how hardware, memory, network and running software were designed together.

An inference chip, not a generalist accelerator

Training and inference are two very different moments. Training mobilizes a lot of calculation to adjust the parameters of a model. Inference corresponds to its daily use: receiving a request, reading the context, producing a response and, in the case of an agent, chaining several calls to tools. It is precisely this second work that Jalapeño is targeting.

An exchange with a language model itself involves several phases. The prefill processes the prompt and strongly requests the calculation units. The decode Then generates chips one by one and depends more on memory bandwidth. In between, moving the model state and communication between chips can immobilize some of the resources. Optimizing a single step is not enough to make the complete answer faster.

What the published results really measure

OpenAI reports testing Jalapeño with InferenceX, a public benchmark of SemiAnalysis, on three open models of different sizes and architectures: GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T. The objective was not only to measure a theoretical peak per chip, but the amount of useful work provided for a given power while respecting a latency level compatible with interactive use.

According to these measurements, Jalapeño provides between 1.5 and 1.9 times more work per watt at maximum flow and reduces end-to-end latency by between 1.7 and 3.6 times compared to comparison systems. Of the most interactive loads, the reported performance benefit ranges from 2.1 to 4.1. These results remain those of the manufacturer and will have to be confronted with real deployments, but the public protocol and the variety of models tested give more useful benchmarks than an isolated figure.

The chip is given for 700 watts, while OpenAI states that its sustained consumption remained less than or equal to 550 watts during the estimated loads. The grade is important: in a data centre, we reason in useful capacity per kilowatt, in cooling and in rack density, not only in nominal power of a component.

Reducing data movements in the centre of architecture

The technical principle presented is to maintain as much as possible locally the condition necessary for the generation, including the KV cache. The system then activates the appropriate combination of calculation, memory and network for each phase. The network is no longer a device added after the chip: it is part of the architecture.

This approach aims at a very concrete difficulty for large models. The longer a context and the more an agent runs the steps, the more waiting times add up. Dozens of milliseconds won on an operation can become noticeable when a task requires searching, file reading, API calls and several cycles of reasoning. Latency is therefore not just a matter of comfort; it can determine whether an agentic path remains usable.

L

OpenAI indicates that it has passed from the original drawing to tapout in nine months using its models to explore implementations, shorten verification loops and optimize certain arithmetic circuits. The hardware was also designed as a predictable programming target: tensor placement, explicit communication and deterministic synchronization can be optimized by both humans and code agents.

Three open models that were not included in the original production plan would have been elevated to a high level of performance in two months with the help of Codex and GPT-Astra. On some blocks of attention and expert mixing of GPT-OSS, the implementations generated would have been 1.5 to 1.8 times faster than the versions previously written by specialists. OpenAI points out, however, that this figure refers to selected blocks, not the full execution of the model.

What it changes for a company that builds an AI product

A client company will probably not buy a Jalapeño chip for its local server. Rather, the impact lies in the provision of services: better availability at points, faster responses and a possible reduction in the cost of a successful task. More efficient infrastructure can also make functions today limited for latency or budget reasons viable.

However, this announcement does not exempt from optimising the application. The choice of the model, the size of the context, the cache, the limitation of the agentic loops and the measurement of cost per result remain essential. Inefficient architecture will simply consume faster infrastructure that has become faster.

• Measure latency from end to end, not just the generation time of the model.
• Follow the cost per completed trade route rather than the only price per million chips.
• Provide several classes of models depending on the actual difficulty of each step.
• Evaluate the stability of performance under load and over long contexts.
• Maintain an architecture that is portable enough to absorb vendor developments.

A first step, not yet a generalisation

OpenAI plans to start the deployment of Jalapeño in its own infrastructure before the end of 2026. Production qualification, operating software and validations on more models continue. A second generation is already in advanced development and a third is beginning to take shape. The company also states that it will continue to deploy NVIDIA accelerators and those of other partners.

The point to remember is therefore not that one chip would immediately replace all others. It is that inference becomes a complete engineering discipline. As agents perform long tasks, the flow-latency-energy torque becomes as structuring as the quality of the model itself.

Consult the technical results published by OpenAI · Discover AI & RAG SDX accompaniment →