Profiling and Kernel Optimization

Which dependency prevents earlier completion?

A slow operation is not automatically the operation that delays completion. It may overlap another operation, belong to a nested wrapper, or execute outside the interval you care about. Start by defining the result: request latency, tokens per second, step time, memory peak or useful training throughput.

Next predict the work that should occur. A mismatch becomes a hypothesis: a view materialized, a copy blocked overlap, a host launch arrived late, or a rank waited for another rank. The trace supplies evidence; a controlled change tests the explanation.

Keep three boundaries distinct: what the framework requested, what the host submitted, and what the accelerator executed. An optimization must improve the target metric while preserving numerical and training behavior. Fewer names, launches or instructions are intermediate observations, not success criteria.

Choose a profiler by the question

Start at the smallest scope that can answer the question. Escalate to a system timeline when input supply, streams, synchronization or collectives matter. Use detailed kernel analysis after identifying the kernel whose behavior can affect completion.

Do not collect every tool’s most expensive mode for an entire job. Large captures can hide the relevant interval and perturb the workload. Tool availability, permissions and supported counters depend on the platform.

Capture a bounded, representative workload

Use a few representative steps after the application reaches the state you intend to measure. Warm the model, allocator, libraries, compiler and input path. Include the actual shapes, dtype, masks and training/inference mode.

import torch
from torch.profiler import profile, ProfilerActivity, record_function

def train_step(batch):
    with record_function("forward"):
        loss = model(batch)
    with record_function("backward"):
        loss.backward()
    with record_function("optimizer"):
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)
    return loss

The sketch assumes an already configured model, optimizer and loader. It preserves semantic ranges and gradient reset order; it is not a complete training program. External warmup uses the same training step before collection.

for _ in range(10):
    train_step(next(loader))

def save_trace(prof):
    prof.export_chrome_trace(f"trace-{prof.step_num}.json")

schedule = torch.profiler.schedule(
    wait=1, warmup=2, active=3, repeat=1
)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule, on_trace_ready=save_trace,
    record_shapes=False, profile_memory=False, with_stack=False,
) as prof:
    for _ in range(6):
        train_step(next(loader))
        prof.step()

Call prof.step() once at the intended iteration boundary. Export each cycle in its callback with a distinct filename. In distributed runs, also distinguish workers. Supported accelerator activities and trace collection must be checked in the installed environment. These GPU snippets are not executed by the note.

Application warmup and profiler warmup are different

External warmup moves the application toward steady state. Scheduled profiler warmup prepares collection. Neither proves that every production shape has compiled or that recurring launch cost disappeared.

For wait 1, warmup 2, active 3, repeat 1, zero-based steps 0, 1–2 and 3–5 are respectively wait, warmup and active. The final active step requests saving. Later steps are outside this finite schedule. Repeat 0 in the API means continuing cycles until collection ends; it is not zero captures.

The visualization changes schedule phases only. It does not model clocks, CUDA initialization or a specific number of physical training steps per second. A delayed capture chosen by seconds needs separate evidence that it lands in the intended workload phase.

Collect only the evidence the question needs

The API also supplies tensorboard_trace_handler for directory-based traces. A direct Chrome JSON export gives a portable trace for compatible viewers such as Perfetto. Trace-export success is not proof that all requested device activity was collected.

Record software and profiler versions. Compare a short capture against an unprofiled measurement. If collection changes the behavior significantly, diagnose that perturbation before promoting the trace into a performance claim.

Attribution tables are not a stopwatch

An aggregate table answers where attributed time and call count accumulate. Self time excludes nested child attribution; total time includes it. Average time divides the aggregate by calls. A parent such as aten::linear can have little self device time while its child GEMM carries substantial work.

print(prof.key_averages().table(
    sort_by="self_cuda_time_total", row_limit=30
))

The current CUDA documentation uses this self-time key in its example. Check the available columns for the installed version and device. Group by shape or stack only when that grouping serves the question.

For a host interval[0,10] with immediate children covering[2,6] and[7,9], exclusive time is4, not10. If a hierarchy includes grandchildren, do not subtract a grandchild twice. This host nesting calculation is distinct from asynchronous device attribution.

CPU and accelerator totals are not generally additive. A synchronization event can spend time waiting for earlier device work. Adding that wait to the kernel duration may count the same elapsed interval twice.

Overlap makes summed time misleading

A timeline places events in time and separates lanes. It can show what launched a kernel, what waited, and what did not overlap. A kernel-duration sum cannot answer all of those questions.

Two 4 ms intervals at[0,4] and[2,6] sum to 8 ms, but occupy 6 ms of their shared time axis. Moving the second interval to[4,8] removes overlap. In the fixed 12 ms window, busy union becomes 8 ms and idle becomes 4 ms.

Union length is still not hardware utilization. Concurrent kernels can share resources, wait internally, or perform unnecessary work. The example assumes supplied intervals and makes no performance prediction about launching real kernels concurrently. Use dependencies and the target completion boundary to identify the critical path.

Follow an operation to its device work

ProfilerStep / semantic forward range
  → Python or nn.Module call
    → aten::linear
      → aten::t and aten::addmm or aten::mm
        → CUDA runtime submission
          → cuBLAS / Triton / generated device kernel

Each layer has a different role. Python expresses the program. ATen dispatches and composes tensor operations. CUDA submits asynchronous work. A library or generated implementation supplies the device kernel. The GPU lane shows when that kernel ran.

One operation can launch several kernels, one fused kernel can implement several operations, and some operations launch none. Views can change metadata only. A reshape or downstream consumer can instead require materialization, copying or a different kernel because of strides.

A CPU range’s duration is not the device execution duration of all work it submitted. Correlation identifies ownership; temporal overlap and dependencies determine completion. The launch visual below makes this distinction concrete.

Read host intent and device execution together

Nsight Systems records processes, CPU threads, CUDA API calls, kernels, copies, OS-runtime activity and NVTX ranges when enabled. Read the host and device lanes together. A cudaLaunchKernel interval describes host submission, not the kernel’s GPU duration.

Expand the process, Threads, CUDA API and CUDA HW views. Typical host calls include cudaMemcpyAsync, cudaStreamSynchronize and cudaEventRecord. The last records an event; it does not itself prove a blocking host wait. GPU work can include GEMMs, attention, layer normalization, elementwise operations and DALI input processing. Inspect a representative interval first. Return to host threads when a launch, input or wait hypothesis makes their behavior relevant; do not discard a possible host bottleneck because its lane is unfamiliar.

Make capture duration and application lifetime explicit

A basic capture launches the application under nsys profile. Short captures can disable sampling and context-switch collection when those are not needed. Start with the basic command. For a delayed, time-bounded window, use the second command:

nsys profile -o profile_report python train.py --arg1 value1 --arg2 value2

The following window records CUDA, OS-runtime and NVTX activity without CPU sampling or context-switch collection:

nsys profile --trace=cuda,osrt,nvtx \
  --sample=none --cpuctxsw=none \
  --delay=10 --duration=0.8 --kill=none \
  -o /tmp/train_profile python train.py --arg1 value1 --arg2 value2

The delay and duration are seconds, not iteration counts. A 0.8 s interval contains about four 0.2 s steps only if that timing actually holds. Startup duration can vary. Prefer explicit NVTX or CUDA-profiler ranges when phase boundaries matter.

Application lifetime is a separate setting. The current Nsight guide says duration-limited launches terminate when collection ends unless continued execution is selected. --kill=none makes continued execution explicit here. Check the installed version and range-end behavior; do not casually profile a valuable long-running job with a command that may stop it.

These are examples, not commands executed on your training environment. Preserve an existing report and select a new output name rather than overwriting evidence. See the Nsight Systems User Guide ↗ for capture modes and application-lifetime options.

Retain raw captures before diagnosing conversion

A lone .qdstrm does not prove the application was interrupted. Automatic report generation may not have run or completed. The CLI includes nsys import; deferred collection also has a finalization path. Inspect status and version compatibility before recommending another run.

Keep the raw artifact recoverable. A missing processed report is an analysis problem until the capture state is established. Do not turn it into permission to terminate, restart or delete a live workload.

Use summaries to choose a timeline investigation

nsys-ui
nsys stats profile_report.nsys-rep
nsys stats --report cuda_gpu_kern_sum profile_report.nsys-rep
nsys stats --report cuda_api_sum profile_report.nsys-rep
nsys stats --report osrt_sum profile_report.nsys-rep

Kernel summaries identify aggregate device costs. CUDA API summaries identify host-side submission and waits. OS-runtime summaries show host runtime activity. Use a high count or duration to select an investigation, then return to the timeline.

Many short kernels can suggest fragmentation. A long stream synchronization can instead expose waiting for already-submitted work. Many blocked host threads can be normal. Report names and availability are version-specific; inspect the installed tool’s report list.

A repeated pattern is not an annotated training step

Without semantic markers, a repeated launch/kernel pattern is a candidate step boundary. It can instead be a microbatch, accumulated-gradient substep, decode token or inner loop. Confirm the program’s iteration contract.

Annotate data transfer, forward, loss, backward, gradient synchronization, optimizer and checkpoint/logging as needed. Avoid ranges so broad that every event belongs to the same label. Good annotation reduces the guesswork in matching a trace to the intended workload.

One wider recorded step can be an application event, an initialization artifact or a profiler event. Compare surrounding steps and the unprofiled behavior before assigning its cause.

A trace signature suggests a test, not a verdict

These are hypotheses, not diagnoses. A dense lane can be memory-bound. A gap can reflect a dependency rather than slow Python. A lower kernel count can hide a slower fused implementation. Change one variable and check whether the predicted timeline feature and completion metric both move.

Recurring launch overhead changes the schedule

The toy submits five independent 2 ms kernels to a serial GPU lane. Host launch cost is adjustable. With asynchronous submission, CPU launch work can run while earlier GPU work executes. With a wait after every kernel, the host cannot submit the next launch until the current kernel finishes.

At 1 ms launch cost, asynchronous completion is 11 ms while GPU durations sum to 10 ms and CPU launch durations sum to 5 ms. Those sums do not imply 15 ms elapsed. Waiting after every kernel instead gives 15 ms because it inserts serialization.

At 3 ms launch cost, recurring submission gaps remain even after initialization. Hiding startup does not remove them. The visual reveals launch ownership progressively; its reveal stage is not a simulation clock or a measured trace.

The model omits transfers, queues, contention, multiple GPU streams and real dispatch machinery. It isolates one causal distinction: asynchronous submission versus an added completion dependency.

Inspect the distributed world, not only the local process count

NCCL kernels and ranges can reveal collectives and point-to-point work. Their presence alone does not prove that communication determines completion; some work overlaps computation and some waits for another rank to become ready.

--nproc_per_node=1 describes the local process count. A multi-node job can still have several ranks and inter-GPU communication. Check world size, devices, ranks and process placement before calling a run single-GPU.

For a truly single-device execution with no relevant peers, inter-GPU collectives are not the main question. For distributed execution, inspect critical-path ranks and dependencies, not only the fastest rank’s dense lane. Parallelism owns placement and schedule contracts.

Separate startup and profiler artifacts from recurring work

Startup can include CUDA initialization, lazy library loading, allocator growth, library heuristics, JIT compilation, autotuning, compiled graph generation and input-pipeline initialization. Profiler activity buffers add another possible source of perturbation.

A final synchronization can flush earlier work so complete timings can be emitted. It is not automatically an application synchronization defect. Clocks, thermals, power limits and background activity can also affect observed durations.

External warmup excludes some startup costs from a steady-state question. It does not optimize those costs away. If startup latency is the target, measure startup explicitly. If repeated launch cost is the target, test the hot path after warmup.

Compare several bounded windows and the application without collection. One unusual interval is evidence to investigate, not proof of a specific buffer allocation or compiler event without supporting correlation.

Measure completion outside the profiler

import time
import torch

def elapsed_cuda(step, iterations, device=None):
    if not isinstance(iterations, int) or isinstance(iterations, bool) or iterations < 1:
        raise ValueError("iterations must be a positive integer")
    torch.cuda.synchronize(device)
    start = time.perf_counter()
    for _ in range(iterations):
        step()
    torch.cuda.synchronize(device)
    return time.perf_counter() - start

Pass a callable for one representative iteration after external warmup. The function returns total seconds, not per-step seconds. The two synchronization boundaries target the same device; None selects the current device. Synchronizing after every operation can change the overlap you meant to measure.

This function assumes a CUDA workload on that device. Work on other devices, unsynchronized external processes, or later background submissions needs a broader completion contract. The local tests check ordering with a fake device; they do not execute a CUDA benchmark.

For thread ownership, memory coalescing, shared-memory banks and tiled kernel construction, see GPU Architecture and CUDA Programming. Use a timing method matched to the question. End-to-end host timing includes host/input work. CUDA events can address a declared stream/device interval. Neither automatically measures an entire distributed job’s critical path.

Report warmup, iteration count, shapes, batch size, dtype, hardware, software and timing boundaries. Recheck numerics and training behavior after an optimization. Profiler observations help explain a measured improvement; they do not replace the outside-profiler result.

Predict the operator and kernel inventory

g = gate_proj(x)
u = up_proj(x)
h = torch.nn.functional.gelu(g, approximate="tanh")
m = h * u
y = down_proj(m)

An eager gated MLP suggests gate and up GEMMs, a GeLU operation, a multiplication, and a down GEMM. Compare that expectation with the actual trace. Did a view materialize? Did allocation or copying appear? Did GeLU and multiplication fuse? Did shape change the GEMM implementation?

Equal-FLOP GEMMs can select different tiles and pipelines. A five-kernel implementation can beat one poorly shaped fused kernel. Keep the expected operation inventory separate from the exact kernel inventory until dispatch is observed.

Four meanings of fusion

For GeLU followed by multiplication, an eager path can write the activation to HBM and read it again. A fused kernel can keep that intermediate on chip and write the final output. The benefit depends on the implementation and workload, not the word “fused.”

Ask six questions: did operator count decrease;did launch count decrease;did kernel identity change;did allocations or copies disappear;did bytes moved decrease;and did steady-state completion improve? FlashAttention is an algorithmic IO example, not merely a renamed softmax operator.

Compare compiled execution, not only operator names

A compiled path can include TorchDynamo guards/cache lookup, AOTAutograd wrappers, TorchInductor generated or external calls, and library/Triton/generated kernels. Compilation can remove dispatch work, fold metadata operations, fuse pointwise work and specialize for shapes.

A lone Linear already mapped to one tuned GEMM may gain little or acquire wrapper overhead. Compare actual kernels and steady-state timings, not only the disappearance of ATen rows.

Use identical inputs and synchronization boundaries. Inspect recompiles, graph breaks, cache hits, layouts, dtypes and the real shape distribution. A highly specialized kernel and a generic kernel solve different deployment constraints.

An optimization that wins on one repeated shape can lose when production inputs trigger compilation or fallbacks. Record which execution path the improvement actually covers.

A static decode path makes launch replay practical

PyTorch’s historical GPT-Fast example starts with batch-1 autoregressive decoding dominated by recurring host overhead. The intervention compiles the single-token decode function as a full graph:

decode_one_token = torch.compile(
    decode_one_token, mode="reduce-overhead", fullgraph=True
)

A growing concatenated KV cache changes shapes, reallocates storage and copies prior contents. A static cache preallocates the supported maximum, updates the current slots, and masks or indexes the logical valid length. Stable shapes and addresses help make replay practical.

Prefill and decode remain separate. Prefill processes a variable prompt length; single-token decode repeats a fixed token dimension. Compiling them separately avoids making dynamic prompt work the contract for the repeated decode path.

reduce-overhead can use CUDA Graphs when the workload is capturable; it does not promise replay for every program. fullgraph=True asks for whole-function capture and makes unsupported breaks visible rather than silently splitting the hot path.

Keep the historical speedup attached to its setup

The November 2023 article uses Llama-7B, batch size 1, and an A100 80 GB limited to 330 W. The first increase is about 4.2×. Compilation also generates specialized decode kernels; the gain is not solely fewer CPU launches.

The 244.7 stage precedes tensor parallelism. Multi-GPU examples have a separate resource budget and must not be folded into that single-GPU comparison. The conclusion separately states 241 tokens/s for Llama-7B. The article does not explain that difference; retain the stage figure without equating them.

These are historical results, not a prediction for a new workload. Quantization changes weight traffic; speculation amortizes target work across candidate tokens. Verify output behavior and outside-profiler timing. GPT-Fast case study ↗.

Attention dispatch depends on the complete call

Scaled dot-product attention dispatch can choose math/reference, memory-efficient, FlashAttention or cuDNN implementations according to the available stack. The supported set depends on device, software, dtype, layout, head dimension, sequence lengths, masks, dropout, GQA and backward requirements.

from torch.nn import functional as F
from torch.nn.attention import SDPBackend, sdpa_kernel

with sdpa_kernel(SDPBackend.FLASH_ATTENTION):
    output = F.scaled_dot_product_attention(
        query, key, value, is_causal=True
    )

Force a backend to test a diagnosis, not to assert a permanent ranking. Check warnings or eligibility failures and compare the actual shapes. Automatic selection is a reasonable starting point; pinning needs workload evidence and version-aware support checks.

A copy or overwritten value can change the diagnosis

A naive path can expose $QK^{\top}$, scaling, masking, softmax and $PV$ as separate work. Here Q and K are query and key matrices; P is the normalized attention-probability matrix and V contains values. An out-of-place mask can introduce a copy. An in-place replacement can be unsafe when autograd needs the overwritten forward value.

Flash-style algorithms use tiled computation and online normalization to avoid materializing large score/probability matrices in HBM. That changes the IO contract, not the desired attention result. Attention owns the exact operator and online-normalization explanation.

Inspect both forward and backward. A forward-only improvement can add recomputation, extra memory or a slower backward path. Kernel names and intermediate allocations support the explanation;full training-step measurement tests its value.

Occupancy is not the optimization objective

Occupancy describes resident execution resources, not useful work per second. Registers and shared memory can reduce residency while enabling a kernel to avoid more expensive global-memory traffic.

Low occupancy is a clue about resource use, not proof of poor performance. A reported 0% can also reflect a launch-path or attribution limitation. Cross-check duration, resource footprint and the tool’s metric definition.

Do not optimize a single metric in isolation. A higher occupancy kernel can move more bytes, perform more work or take longer. Relevant roofs and measured completion provide the context.

Training, prefill and decode need separate evidence

Training includes forward, loss, backward, gradient synchronization and optimizer work. Checkpointing, logging and transfers can add dependencies. Backward can choose other kernels, recompute values and require forward intermediates.

Inference separates prompt prefill from token decode. Batch-1 latency and batched throughput also require different measurements. A trace with one convenient static input is not evidence for a production shape distribution.

Checkpointing, compilation and distributed overlap can shift the critical path. Profile the complete phase that defines the target metric and inspect all relevant ranks when stragglers matter.

An allocation is not the same as reserved memory

Profiler allocation tables attribute tensor allocations and releases to events. Separate self allocations from child allocations, and live tensor memory from allocator-reserved memory. Freed tensors can leave reusable reserved blocks.

An allocation-heavy operator need not own the long-lived data. A temporary library workspace and persistent cache have different roles. Peak memory depends on overlapping lifetimes, not the sum of all allocations across a step.

Use allocator statistics or memory snapshots for persistent reservation and fragmentation questions. Keep the memory boundary, device and timing phase explicit. Less allocation traffic does not automatically mean a lower peak or faster completion.

Predict, inspect, intervene, verify

First reproduce the issue without collection and define the metric. Warm the representative workload;capture a bounded window;annotate phases;and write the expected operators and kernels.

Then inspect table costs/call counts and timeline gaps, copies, overlap and ownership. Distinguish metadata views from materialization. Compare shapes, layouts, dtypes, kernel identities and synchronization. Classify startup and profiler artifacts separately.

Finally change one variable with a predicted outcome. Re-measure without collection and validate numerical/training behavior. Keep failures in the investigation log. A trace explanation is strongest when both the predicted event and the completion metric change as expected.

Continue with Speculative Decoding for serial generation work,Precision and Quantization for representation/traffic tradeoffs, and Compute, Memory, and Performance for performance accounting.

Sources and connected topics

• PyTorch profiler API ↗ and profiler recipe ↗.
• Nsight Systems User Guide ↗. Capture options and lifetime semantics are version-specific.
• Profiling in PyTorch, Part1 ↗; Part2:Linear to fused MLP ↗; Part3:attention ↗.
• GPT-Fast(2023) ↗. Historical diagnosis and optimization progression.
• SDPA tutorial ↗.

The interactive traces are original assigned examples. They are not screenshots, measurements or reconstructed benchmarks from these sources. Use the Serving Models cache-state section for ownership and capacity while interpreting the static decode path.