Abstract

In large-model inference optimization, one of the easiest things to misinterpret is treating a local speedup of a certain kernel directly as an improvement in service throughput. Even if a kernel is 1.08x faster in isolated tests, the complete service may be only a few percent faster, or even almost unchanged.

This article takes Qwen3.5-27B BF16 inference on the Hygon DCU gfx936 path as an example, using profile data, kernel microbenchmarks, and same-container endpoint A/B data already completed locally, to answer three questions:

  1. What operators limit Prefill and Decode respectively;
  2. Why BF16 M=1 GEMM/GEMV becomes the main performance wall for Decode;
  3. What verification a local optimization must go through before it can be judged to have actually improved the complete inference service.

This article only discusses DCU inference performance engineering. All performance numbers in the article come from existing measurement records in a fixed local environment; standalone kernel data, profile data, and end-to-end service data are labeled separately and are not mixed into a single conclusion.

1. First Split Inference into Prefill and Decode

A generation request can usually be divided into two stages:

  • Prefill: processes a longer input at once, converting the prompt into an initial state;
  • Decode: then generates token by token; each step usually processes only one new token while reading previously saved state.

These two stages look like they belong to the same model, but their hardware behavior is very different.

Prefill has a larger query dimension, matrix multiplication is closer to a large GEMM, and Attention can also be parallelized over larger token blocks. Decode usually has a query dimension of 1, and many matrix operations degrade into M=1 GEMM or GEMV: the amount of computation is not large, but each step must read very large weights and bears a large amount of kernel launch and runtime scheduling overhead.

A simplified time model can represent the complete request:

1
2
3
4
5
6
request time = prefill GEMM
             + prefill attention
             + other prefill linear layers
             + decode linear layers
             + decode state access
             + sampling and runtime overhead

Therefore, “Attention is fast” or “a certain GEMM is fast” only explains one part of the total time. What really deserves attention is: what proportion this part accounts for in the target workload, and whether it recurs in every generated token.

2. Experimental Environment and Measurement Protocol

The local measurements cited in this article use the following environment and definitions:

Item Configuration
Accelerator Hygon DCU, gfx936-class path
Model Qwen3.5-27B
Weight compute type BF16 original weights
PyTorch 2.10.0
Inference framework vLLM local runtime environment
Standalone warmup 10 times
Standalone measured iterations 40 times
Endpoint test fixed request set, concurrency 1, same-container A/B
Main metrics output throughput, mean/p99 TPOT, TTFT, output text, failure count

Measurements are divided into three layers:

  1. Profile: answers where time is spent;
  2. Standalone/CUDAGraph microbench: answers the local upper bound of the target kernel;
  3. Endpoint A/B: answers how much benefit the complete service actually receives.

These three layers cannot replace each other. Profile is suitable for locating the problem, not for directly serving as a throughput result; microbench is suitable for screening implementations, not for directly representing the service; only endpoint A/B can verify the full-chain effect.

The profile, RPB2 endpoint, and Tail-M counterexample in this article come from different historical experimental subjects: they are used to explain performance phenomena at different layers, not additive results from the same source combination, and the improvement numbers in this article cannot be added together.

3. What a Long-Context Profile Showed

The local profile recorded the stage timings of an official 8K prompt-path request with page832, prefix caching disabled, and all-Tail-Q vendor attention:

1
2
3
4
q=4096 prefill total: 2720.145 ms
q=1685 prefill total:  464.801 ms
decode total:        2586.460 ms
decode average:        43.108 ms/token

3.1 Prefill: After Attention, Large GEMM Becomes the Main Time Bucket

In q=4096 Prefill, the main time items are as follows:

Time item Duration
Attention 238.718 ms
aten::mm 1840.596 ms
aten::mm share of q=4096 Prefill 67.67%

This set of data gives a very direct judgment: after Attention has already been reduced, continuing to make small adjustments to Attention boundary parameters will have gains limited by its share of total time. At this point, what is more worth studying is the data access, tile organization, and actual call shapes of large GEMM.

Here aten::mm is not an abstract “model time”, but the aggregate of a batch of real matrix multiplication calls. To continue optimizing, it must be split into specific M, N, K shapes, dtype, call frequency, and whether it is covered by other operators, rather than only looking at the total bucket name.

3.2 Decode: Linear/GEMV Takes the Vast Majority of Time

The Decode statistics from the same profile are as follows:

Time item Time per token
Linear total 38.078 ms
LLMM1 23.470 ms
rocBLAS GEMV 14.607 ms
Linear share of Decode 88.33%

Observed per Decode token:

1
2
LLMM1:        176 launches/token
rocBLAS GEMV: 129 launches/token

The sources of the 129 GEMV calls can be summarized by module as:

1
2
3
4
64  MLP down projection
48  GDN output projection
16  attention output projection
1   lm_head

This shows that the optimization focus for Decode differs from Prefill. Decode is not simply about finding a “faster Attention”; it must handle a large number of small matrix operations: they are repeated at every step, the saving per call is small, but after multiplying by the number of layers and generated tokens, they may form an appreciable end-to-end difference.

4. Why BF16 M=1 Operations on DCU Are Especially Difficult

When the M dimension of matrix multiplication is 1, compute parallelism is relatively limited, while the weight matrix is still large. In this case, the kernel is often closer to being limited by weight reads and memory access efficiency, rather than by theoretical FLOPS.

A simplified M=1 computation can be written as:

1
y[j] = sum_k x[k] * W[j, k]

Each output row must traverse the K-dimensional weights. Performance is affected jointly by the following factors:

  • whether weight loading is contiguous and whether accesses can be coalesced;
  • how many output rows a block processes;
  • how threads within a wave cooperate during K-dimensional reduction;
  • conversion overhead between BF16 reads and FP32 accumulation;
  • whether output writeback causes extra synchronization;
  • whether kernel launch and graph replay have already become the main cost.

Therefore, optimization of an M=1 kernel is usually not simply increasing the thread count. Too many threads increase reduction and synchronization cost, while too few threads cannot sufficiently hide weight-read latency. Truly effective configurations often only suit a fixed set of model shapes.

The RPB2 experiment in this article keeps BF16 original weights and retains the verified FP32 pair accumulation, only adjusting the output-row blocking of the same LLMM1 implementation. Here RPB can be understood as rows per block: a block is responsible for a different number of output rows, and the organization of weight reads and reduction changes accordingly.

5. Local Standalone Results for BF16 M=1 GEMM

On the same DCU, with the same BF16 inputs and the same compiled extension, RPB4 and RPB2 are compared:

Weight shape RPB4 RPB2 Speedup Max difference
14336 x 5120 0.114160 ms 0.106480 ms 1.0721x 0
16384 x 5120 0.131040 ms 0.120960 ms 1.0833x 0
34816 x 5120 0.272480 ms 0.250400 ms 1.0882x 0

The test conditions were warmup 10 times and 40 formal measurement iterations. The outputs of all three shapes are bitwise identical to the reference path, and repeated runs are stable. Estimated from the weight matrix bytes divided by the measured kernel time, the effective weight bandwidth is about 1.38-1.42 TB/s; this is an effective bandwidth estimate for this kernel, not a direct measurement of DCU peak HBM bandwidth.

This set of data shows that the RPB2 gain is a real local execution gain, but it has not yet answered the service-layer question. Standalone only measured the target matrix multiplication; complete Decode also includes other Linear operations, state reads, sampling, and runtime overhead.

6. From Kernel to Service: Same-Container A/B Is the Key Gate

To determine whether the local gain can propagate to complete inference, A/B was performed locally in the same container, with the same model, the same request set, and the same runtime conditions. Both the baseline configuration and RPB2 use BF16 original weights. The endpoint metrics are as follows:

Tier Baseline output tok/s RPB2 output tok/s Throughput change Baseline p99 TPOT RPB2 p99 TPOT
4K-8K 18.8369 19.5961 +4.03% 46.9627 ms 44.9923 ms
8K-16K 14.5837 15.0306 +3.06% 47.5920 ms 45.5662 ms
16K-32K 11.7495 12.0286 +2.38% 49.8507 ms 47.8545 ms

The relationship between endpoint results and standalone is typical: the local kernel is 1.07x-1.09x, while the service layer does not show the same 7%-9% change, but rather 2.38%-4.03%. The reason is that RPB2 only affects part of the repeatedly executed Linear work, and the remaining time is still composed of other operators and runtime.

The correctness and stability checks are as follows:

1
2
3
4
4K-8K:   2644 vs 2644 output tokens, 10/10 text byte-identical
8K-16K:  1768 vs 1768 output tokens, 10/10 text byte-identical
16K-32K: 1877 vs 1877 output tokens, 10/10 text byte-identical
failed:  0

TTFT is basically unchanged:

Tier Baseline p99 TTFT RPB2 p99 TTFT
4K-8K 2049.317 ms 2047.251 ms
8K-16K 4620.050 ms 4614.045 ms
16K-32K 7044.241 ms 7037.622 ms

This matches the expectation for the optimization location: RPB2 mainly acts on Decode M=1 Linear, and the Prefill path for the first token has no obvious change, so TTFT is basically unchanged while TPOT improves.

7. Why a 1.08x Kernel Does Not Automatically Become a 1.08x Service

This phenomenon can be understood with a rough Amdahl model. Suppose the target kernel accounts for a proportion f of total time and the local speedup is s. Under ideal conditions, the overall speedup upper bound is approximately:

1
service speedup ≈ 1 / ((1 - f) + f / s)

When s = 1.08, if the target kernel covers only part of the total time, the overall gain is necessarily less than 8%. The actual service is also affected by more factors:

  • the matrix shapes of different layers are not exactly the same;
  • some calls still use the original implementation;
  • kernel launch, graph replay, and state access cannot be eliminated by the target kernel;
  • the large GEMM in Prefill and the small GEMM in Decode are located in different time phases;
  • measurements also contain fluctuations caused by request length, cache, and system scheduling.

Therefore, the correct question for end-to-end A/B is not “how many times faster is this kernel”, but:

1
2
3
How many real calls does it cover?
What share of total time do these calls account for?
Does it change the behavior of other paths?

The reason the local RPB2 results are meaningful is precisely that it simultaneously satisfies three points: the target calls appear many times in Decode, the local gain repeats across multiple real shapes, and the output and request behavior remain consistent.

8. A Counterexample: Standalone Gain, Endpoint Almost Unchanged

The same batch of local experiments also provided a very representative counterexample. After screening 145 shapes and M combinations across five types of large GEMM, 106 shapes simultaneously satisfied:

1
2
BF16 output hash consistent with default path
Standalone CUDAGraph speedup >= 1.01x

Looking at these shapes alone, they all look like optimizations worth integrating. After integrating them into the complete service, however, the result for the fixed mid-tier request set was only:

1
2
3
4
5
output throughput: 16.583027 -> 16.586315 tok/s
increment:         +0.0198%
texts:             10/10 exact
tokens:            1809 -> 1809
failed:            0 -> 0

Another gate/up GEMM scheme that preserves exact output also achieved an average gain of only +0.3250% in steady-state same-container A/B. This does not mean the standalone measurements were wrong; it means that the time share, call coverage, or overlapability of these calls in the complete request is insufficient to form an obvious service gain.

This counterexample is very important:

Standalone speedup is a local upper bound; endpoint A/B is the product-layer gain.

If schemes are selected only based on microbench, it is easy to stack dozens of small positive gains into a seemingly large prediction, but at runtime they may overlap each other, be drowned out by other time items, or hit only very few real requests.

9. How Correctness and Stability Should Be Verified

Performance optimization cannot just record a duration number. For BF16 kernels on DCU, at least the following should be checked simultaneously.

9.1 Kernel Layer

  • whether the output is finite;
  • whether the output shape and stride are correct;
  • the maximum absolute difference or hash compared with the reference path;
  • whether repeated runs are stable;
  • whether it is still correct for different batches, different M values, and boundary shapes.

9.2 Service Layer

  • whether each request completes;
  • whether the output token count is consistent;
  • whether the generated text is byte-identical, or meets a preset accuracy standard;
  • whether mean/p99 TPOT improves;
  • whether TTFT regresses;
  • whether there are failed requests, timeouts, or abnormal retries.

9.3 Measurement Layer

  • whether the two configurations use the same model and the same extension;
  • whether they are compared in the same container and with the same request set;
  • whether compilation, first load, and cache setup times are handled separately;
  • whether sufficient warmup is used;
  • whether repeated runs confirm the direction, rather than trusting a single result.

In particular, avoid treating “the code path exists” as “the kernel has been hit”. The most reliable approach is to record the real dispatch, input shapes, and selected implementation at runtime, then correspond the markers with performance results one-to-one.

10. A Reusable DCU Inference Optimization Workflow

Step 1: First Distinguish Prefill and Decode

Measure TTFT, TPOT, and stage timings separately. Do not use Decode optimization methods to explain Prefill, and do not use conclusions about Prefill large GEMM to replace Decode M=1 analysis.

Step 2: Use Profile to Find the Time Wall

First confirm whether the cost is large GEMM, GEMV, Attention, state reads/writes, or runtime overhead. Only by knowing the time share can you know whether a local speedup has enough physical budget.

Step 3: Restore the Real Call Shapes

Record M, N, K, dtype, batch, call count, and output writeback method. Microbenchmarks on generalized matrices can only serve as references and cannot replace real shapes.

Step 4: Screen in Standalone

First use fixed warmup and multiple iterations to measure local time, while checking correctness and determinism. For implementations clearly below the threshold, do not rush to integrate them into the service.

Step 5: Do Same-Container Endpoint A/B

Keep the model, dataset, concurrency, sampling settings, and runtime environment consistent, and record throughput, TPOT, TTFT, output length, and failure count separately. A candidate must repeatedly show a positive signal across multiple tiers or multiple real shapes.

Step 6: Keep Negative Results

A scheme that is positive in standalone but has no endpoint gain is also a valuable engineering conclusion. It can help subsequent work avoid repeated testing and bring the performance budget closer to the real service instead of stopping at the kernel table.

11. Conclusion

The local measured data presents a relatively clear DCU inference performance chain:

  1. The main contradiction in Prefill can fall on large GEMM; in the q=4096 profile, aten::mm accounts for 67.67%;
  2. The main contradiction in Decode is a large number of Linear/GEMV operations; Linear accounts for 88.33% of Decode in that profile;
  3. A small local improvement in the BF16 M=1 kernel can propagate to the endpoint after repeated calls; RPB2 brings 2.38%-4.03% throughput improvement across three same-container A/B tiers;
  4. Local speedup does not guarantee end-to-end gain; 106 standalone-positive and exact tail large GEMM shapes only produced +0.0198% after integration;
  5. Correctness, dispatch hit, repeatability, and measurement protocol are as important as the kernel’s own time.

For inference optimization on Hygon DCU, the most reusable thing is not a fixed block parameter, but a verification order:

First use profile to find the real time wall, then use real shapes for kernel microbench, and finally use same-container endpoint A/B to verify whether it improves the complete service.

Only when the three layers of evidence point in the same direction does local “fast” deserve to be called service-layer “fast”.

The performance numbers in this article come from existing measurement records in a fixed local environment and are only used to illustrate engineering observations under this model, software stack, and DCU path. Different DCU models, drivers, DTK, compilation caches, request distributions, and system loads may produce different results; standalone kernel speedups should not be directly extrapolated as general service throughput.