Key takeaways
- CUDA is Nvidia's parallel computing platform and programming model, launched in 2006, that lets programs run thousands of calculations at once on Nvidia GPUs.
- Most AI code reaches CUDA through frameworks like PyTorch and Nvidia's CUDA-X libraries, such as cuDNN for neural networks and cuBLAS for linear algebra.
- The CUDA Toolkit costs nothing to download, but its license limits development to systems with Nvidia GPUs; AMD's ROCm and HIP and the open-source Triton are the main alternatives.
CUDA is Nvidia's parallel computing platform and programming model: the software that lets ordinary programs run thousands of calculations at once on an Nvidia GPU instead of a few at a time on a CPU. Nvidia launched it in 2006, and it is the main way AI frameworks such as PyTorch put Nvidia GPUs to work.
The name came back into the news on November 16, 2026, when Nvidia announced Rubin HPC at the SC26 supercomputing conference. Nvidia says the GPU has "a record number of CUDA cores," more than 50,000 on a single chip, and expects it to be available in 2028. Our Rubin HPC news story has the details.
What CUDA is, in plain words
A CPU has a few powerful cores that work through tasks one after another. A GPU has thousands of simpler units that do the same operation on lots of data at once. That suits graphics, and it suits AI, where a model multiplies huge grids of numbers.
CUDA is the layer that lets a programmer use that hardware for general work. The CUDA Programming Guide calls it "a parallel computing platform and programming model developed by NVIDIA," used in deep learning, scientific computing and high-performance computing. In practice it is three things:
- Language extensions. You write normal C++, Python or Fortran and mark the parts that should run on the GPU, according to Nvidia's CUDA Zone page.
- The CUDA Toolkit. A compiler, a runtime library, debugging and profiling tools, and GPU-accelerated libraries. The current release is CUDA 13.5.
- The libraries built on top, which Nvidia groups as CUDA-X (more on those below).
What CUDA cores are
CUDA cores is Nvidia's name for the basic arithmetic units inside its GPUs: the parts that add and multiply ordinary numbers. Nvidia's Tensor Core page lists the number formats its CUDA cores handle on Rubin: FP64, FP32, INT32, FP16 and BF16.
They are not the whole chip. The programming model docs describe a GPU as "a collection of Streaming Multiprocessors (SMs)," and each SM holds a set of functional units, the CUDA cores among them. Modern Nvidia GPUs also have Tensor Cores, which Nvidia describes as "specialized GPU cores that speed up AI and HPC math workloads." Tensor Cores run mixed-precision math, which is what AI models lean on; CUDA cores handle the general-purpose arithmetic.
That helps explain why Rubin HPC leads with CUDA cores: Nvidia pitches it at simulation and data processing for science and engineering, not at AI models. Our Vera Rubin vs Blackwell comparison covers the AI side of the same family.
A core count alone does not say how fast a chip is; the cores also need data. Nvidia pairs Rubin HPC's count with a sparse data acceleration feature meant to keep them fed.
Kernels, threads and blocks
- Kernel. A function that runs on the GPU. The guide says it is called a kernel "for historical reasons." You mark it with
__global__. - Thread. One copy of the kernel running on one piece of data. Launching a kernel starts many threads at once, and each one works out which piece of data is its own.
- Block. A group of threads that runs together on one SM and can share fast on-chip memory. The guide lists a limit of 1,024 threads per block for Blackwell-generation GPUs (compute capability 10.0).
- Grid. All the blocks started by one kernel launch.
Inside a block, the hardware runs threads in groups of 32 called warps, and every thread in a warp executes the same instruction at the same moment. Nvidia calls this SIMT, single instruction, multiple threads.
The guide's own example adds two lists of numbers:
__global__ void vecAdd(float* A, float* B, float* C, int vectorLength)
{
int workIndex = threadIdx.x + blockIdx.x*blockDim.x;
if(workIndex < vectorLength)
{
C[workIndex] = A[workIndex] + B[workIndex];
}
}
There is no loop. Each thread finds its position from its block number and its place in the block, then adds one pair of numbers. The <<<blocks, threads>>> launch syntax says how many to start.
Blocks can run in any order, so the same program runs on a small GPU or a huge one: a bigger chip just runs more blocks at once.
Why AI runs on CUDA
Training and running a neural network is mostly matrix multiplication repeated billions of times. The many small multiplications inside it can run side by side, which is the kind of work CUDA spreads across thousands of threads.
Most AI developers never write a kernel. They use a framework such as PyTorch, which Nvidia's CUDA Zone page names as a way to reach CUDA, and the framework calls Nvidia's libraries underneath. When a chatbot answers a question, a process called AI inference, the work on an Nvidia GPU passes through those same libraries. Techniques such as quantization, which shrink models into lower-precision numbers, exist partly to make better use of the Tensor Cores.
CUDA-X: cuDNN, cuBLAS and the rest
CUDA-X is Nvidia's name for the collection of libraries built on CUDA.:
| Library | What it does |
|---|---|
| cuDNN | Building blocks ("primitives") for deep neural networks |
| cuBLAS | Basic linear algebra, including the matrix multiplications at the heart of AI |
| NCCL | Fast communication between GPUs and across servers, used to train one model on many chips |
| TensorRT | Optimizes trained models and runs them for production inference |
| cuFFT | Fast Fourier transforms for signal processing and simulation |
| cuDF, cuML | Speed up data frames (pandas, Polars, Spark) and classic machine learning |
Why CUDA matters for Nvidia's lead
Nvidia sells hardware, but CUDA is a large part of why customers keep buying it. Its own CUDA explainer cites more than 150 CUDA-based libraries, SDKs and tools. Years of research code, tutorials and tuned kernels assume an Nvidia GPU. A rival chip has to match not only the silicon but that pile of software.
The license reinforces it. The CUDA Toolkit license says the SDK "is licensed for you to develop applications only for use in systems with NVIDIA GPUs." It also bars reverse engineering CUDA's compiled output "for the purpose of translating such output artifacts to target a non-NVIDIA platform."
That is the real moat. Rivals can build fast GPUs; getting existing code to run well on them is the slower job.
The alternatives: ROCm, HIP and Triton
AMD ROCm. AMD describes ROCm as its "open software stack for GPU-accelerated computing": runtimes, compilers and libraries for AMD GPUs. Many of its libraries mirror Nvidia's: rocBLAS and hipBLAS for linear algebra, MIOpen for deep learning, RCCL for multi-GPU communication. AMD names PyTorch among the frameworks that run on ROCm.
HIP. HIP is AMD's C++ runtime API and kernel language, and AMD says CUDA programmers "will find the HIP API straightforward." AMD's HIPIFY tools convert CUDA API calls into HIP calls. AMD notes that porting usually still needs some manual coding and tuning.
Triton. Triton is an open-source language and compiler for writing fast GPU kernels, installed as a Python package and released under the MIT license. Its stated aim is to be more productive than CUDA. It runs on Nvidia GPUs (compute capability 8.0 and newer) and on AMD GPUs with ROCm 6.2 or later, so a kernel written once in Triton can target both vendors.
Is CUDA free
Yes, to download and use. Nvidia lists the CUDA Toolkit under "Free Tools and Training," and there is no charge to download it. The catch is the hardware: the license limits development to systems with Nvidia GPUs, so the cost of CUDA is the cost of an Nvidia card or cloud GPU time.
The next date to watch is 2028, when Nvidia expects Rubin HPC to ship. Until then, the bigger story is whether ROCm and Triton narrow the software gap faster than Nvidia widens it.
Frequently asked questions
Is CUDA free?
Yes. The CUDA Toolkit is a free download, listed by Nvidia under its free developer tools. Its license limits development to systems with Nvidia GPUs, so the real cost is the Nvidia hardware or cloud GPU time.
What are CUDA cores?
CUDA cores are the general-purpose arithmetic units inside Nvidia GPUs, grouped into streaming multiprocessors. Nvidia's Rubin HPC, announced on November 16, 2026, has more than 50,000 of them on one GPU.
What is the difference between CUDA and ROCm?
CUDA is Nvidia's platform and runs only on Nvidia GPUs. ROCm is AMD's open software stack for its own GPUs, and its HIP language is close enough to CUDA that AMD's HIPIFY tools can convert CUDA code, though porting usually needs some manual tuning.
Is CUDA a programming language?
Not exactly. CUDA is a platform and programming model: extensions to C++, Python and Fortran, plus a compiler, runtime and libraries in the CUDA Toolkit.
Sources
- CUDA Programming GuideNVIDIA
- CUDA Programming Guide: Programming modelNVIDIA
- CUDA Programming Guide: Writing SIMT kernelsNVIDIA
- CUDA ZoneNVIDIA
- CUDA ToolkitNVIDIA
- CUDA Toolkit: Free Tools and TrainingNVIDIA
- CUDA FAQNVIDIA
- CUDA Toolkit End User License AgreementNVIDIA
- CUDA-X GPU-accelerated librariesNVIDIA
- NVIDIA Tensor CoresNVIDIA
- What Is CUDA?NVIDIA
- NVIDIA Rubin HPCNVIDIA
- What is ROCm?AMD
- What is HIP?AMD
- Tritontriton-lang



