Glossary · simply explained

AI inference & GPU computing

Inference is the productive operation of AI models — and GPUs the infrastructure behind it. Where it runs decides cost and privacy.

Inference is the use of a fully trained AI model: every chat answer, every image analysis, every completion is an inference operation. While training a model consumes enormous compute once, inference accrues continuously — in steady operation it determines cost, latency and energy demand of AI applications.

GPUs are the workhorses for this: their thousands of parallel compute units match the mathematics of neural networks. The strategic question for companies: buy inference as an API, rent it in the cloud or run it on own infrastructure — with direct consequences for privacy, cost and network design.

Why GPUs — and what counts in inference

Neural networks consist of matrix operations that parallelise massively — exactly the strength of GPUs over CPUs. When running large language models, graphics memory (VRAM) becomes the limiting factor: the model must fit. Quantisation — computing at reduced precision — shrinks models considerably and makes many open-weight models runnable on single cards or workstations.

Inference is measured by latency to first token, throughput (tokens per second) and cost per request. Optimisations such as batching, caching and specialised inference servers extract multiples from the same hardware — and decide whether self-operation beats API prices.

The three operating models

API sourcing: top models without own infrastructure — fastest start, data flows to the provider, costs scale with usage.
GPU cloud: rented accelerators (hyperscalers or EU providers) for own models — control over model and data without hardware investment.
On-premises/colocation: own GPU systems for maximum data sovereignty and predictable steady costs — including power, cooling and operational responsibility.

Frequently asked questions about AI inference & GPU computing

What is the difference between training and inference?

Training creates the model — once, extremely compute-intensive, mostly the business of large providers. Inference uses the finished model in operation — continuously, latency-critical, cost-relevant. Companies practically always run inference (plus occasional fine-tuning), not foundational training.

When does own GPU infrastructure beat the API?

At high, steady volume (API costs exceed depreciation plus operations), strict privacy requirements or latency/offline needs on site. For fluctuating loads and frontier models, API or cloud sourcing usually stays more economical; many run hybrid.

How much GPU does an own LLM need?

Rule of thumb: model size in parameters × bytes per parameter must fit into VRAM — an 8B model quantised runs on a single professional card, 70B demands several large accelerators or heavy quantisation. Embedding and specialised models are far more frugal.

What does inference at the edge mean?

Models answer close to the user or machine — on local devices, industrial PCs or in the provider’s edge network. That lowers latency, keeps raw data on site and works offline; model size is limited by the edge hardware. Typical: small specialised models outside, large ones central.

Which infrastructure questions does GPU computing raise?

Substantial ones: GPU racks draw multiples of classic servers in power and heat — many existing server rooms fail at this, making high-density colocation an option. Add fast network connectivity for models and data plus access protection: GPU clusters holding company data belong segmented and reachable via Zero Trust.