Hygon DCU Profile in Practice: How to Reconstruct a Token from Sixty Thousand GPU Kernels
Opening
When profiling large models on Hygon DCU, the hardest part is usually not whether a trace can be generated, but how to read it after it has been generated.
A local sampling run of Qwen3.5-27B BF16 produced:
1 | 总事件: 235,964 |
Sorting directly by total kernel time only shows a long list of Tensile, Triton, Attention, and custom kernel names. It does not proactively tell you:
- which ones belong to Prefill;
- which ones belong to Decode;
- how many times a kernel is called per token;
- whether the call counts can map back to the number of model layers;
- which time bucket is the real bottleneck worth optimizing.
This article does not discuss a new kernel. Instead, using this measured Hygon DCU trace, it demonstrates how to reconstruct the complete inference structure from event context and call counts.
1. First, Limit the Scope of the Profile
This sampling covers only one request with clear boundaries:
1 | 3 个 q=4096 Prefill chunk |
This is a very important precondition.
If the profile covers multiple concurrent requests, model loading, first compilation, and long generation, the number of events will expand rapidly, and it will be difficult to attribute a given kernel to a specific phase.
For Hygon DCU large-model services, a more reliable workflow is:
1 | 服务先完成加载和必要 warmup |
The goal of profiling is to answer “where is the time spent,” not to reproduce maximum throughput.
2. First Cut: Split Prefill and Decode by Execution Context
Each batch of GPU events in the trace has an upper-level execution context. Using the token count and generation state, they can first be divided into:
| Context | Total GPU event time |
|---|---|
| Three q=4096 Prefill chunks | 2720.145 ms |
| One q=1685 Prefill tail | 464.801 ms |
| 60 Decode steps | 2586.460 ms |
Decode average per step:
1 | 2586.460 ms / 60 |
This step is more important than looking directly at the top kernels.
The same GEMM name may appear in both Prefill and Decode, but the M dimension, parallelism, and optimization methods are completely different in the two phases. Only after splitting by context do the later statistics have physical meaning.
3. Second Cut: Identify Model Structure Using Call Counts
3.1 LLMM1 kernel
In the Decode interval, LLMM1 totals:
1 | 10,560 launches |
Divided by 60 Decode steps:
1 | 10,560 / 60 = 176 launches/token |
176 is not a random number; it maps completely back to the model structure:
1 | 64 次 MLP gate/up |
When the trace call count and the model layer count fully close, the kernel attribution is basically credible.
3.2 rocBLAS/Tensile GEMV
In the same Decode interval, rocBLAS/Tensile GEMV is:
1 | 7,740 launches |
After conversion:
1 | 7,740 / 60 = 129 launches/token |
129 can likewise be mapped back to modules:
1 | 64 次 MLP down projection |
This kind of “call-count closure” is very valuable. It not only tells us whether a kernel is hot, but also checks:
- whether some type of Linear was missed in the statistics;
- whether a fusion actually reduced launches;
- whether runtime markers are consistent with the actual call count;
- whether the model structure and the trace come from the same version.
4. Third Cut: Convert Time into a Per-Token Budget
LLMM1 and rocBLAS combined:
1 | 23.470 + 14.607 |
Relative to 43.108 ms/token for Decode:
1 | 38.078 / 43.108 |
This shows that in this Hygon DCU trace, the main Decode bottleneck is Linear, not some scattered pointwise kernel.
Other major Decode time includes:
| Module | Time per token | Decode share |
|---|---|---|
| Page-contiguous Attention | 1.630 ms | 3.78% |
| Packed GDN recurrent | 0.623 ms | 1.44% |
After seeing this ratio, optimization priorities become clear:
- a small fusion that saves only 2-3 us per layer is unlikely to change total TPOT;
- an implementation covering hundreds of Linear calls may accumulate endpoint gains even if each call is only a few percent faster;
- continuing to optimize a bucket that accounts for less than 2% requires first proving a sufficiently large absolute saving.
The role of profiling is not to automatically provide a solution, but to constrain the physical upper bound of the solution.
5. Prefill Cannot Reuse Decode’s Conclusions
The main time items for the three q=4096 Prefill chunks are:
| Time item | Total time | Share of q=4096 Prefill |
|---|---|---|
| aten::mm | 1840.596 ms | 67.67% |
| Attention | 238.718 ms | 8.78% |
The same model presents different shapes in Prefill and Decode:
1 | Prefill: |
Therefore, in trace analysis, “Linear accounts for 88.33% of Decode” cannot be understood as a unified share for the entire request, nor can Decode’s M=1 kernels be used to explain q=4096 Prefill.
The correct approach is to establish two separate budgets:
1 | TTFT 预算 -> Prefill 上下文 |
6. Why Looking Only at Kernel Rankings Misleads
Suppose the first place in the top kernel list is LLMM1 and the second place is some Tensile kernel.
Without context splitting, you will encounter at least three problems.
6.1 The Same Name May Correspond to Different Shapes
The same library kernel may be reused by multiple Linear modules. You need to combine M, N, K, call count, and parent events to map it to a model layer.
6.2 High Total Time May Just Mean Many Calls
A 130 us kernel called ten thousand times will rank ahead of a 5 ms kernel. When optimizing, look at all of:
1 | 单次耗时 |
6.3 The Profile Itself Is Not an Endpoint Benchmark
The profiler introduces recording, synchronization, and buffering overhead. The 43.108 ms/token in the trace is used to establish the time composition and cannot directly replace the service TPOT without a profiler.
7. A Real Profiler Shutdown Failure
The target request completed normally, and the trace file was successfully written; but after stop_profile was called, the ROCm runtime triggered a VMFault, and EngineCore subsequently exited.
Therefore, this run was handled as follows:
1 | 已写出的 trace: |
This boundary is very important.
“A service crash when stopping the profiler” does not mean that all previously written events are invalid; but it does mean that the profiler lifecycle has changed the service state, and the same process cannot continue to be used for performance A/B.
When collecting long traces on Hygon DCU, it is recommended to treat the profiler service as a one-time diagnostic instance.
8. A Reusable Hygon DCU Trace Analysis Workflow
Step 1: Fix the Request
Choose a single request that can cover the target Prefill shape and enough Decode steps.
Step 2: Record the Coverage Scope
Explicitly write down:
1 | Prefill chunk 数量与 token 数 |
Step 3: Bucket by Context
At minimum, divide into:
- large Prefill;
- tail Prefill;
- Decode;
- sampling and copies outside the execute context.
Step 4: Count Kernel Calls and Total Time
Do not keep only the top-N times; also keep the count and average time.
Step 5: Divide by Decode Steps
Convert the total call count and total time into:
1 | launches/token |
Step 6: Map Back to Model Layers
Check whether the call count can be fully summed from the MLP, GDN, Attention, and lm_head layer counts.
Step 7: Calculate the Physical Upper Bound
Estimate for candidate optimizations:
1 | maximum endpoint saving |
Step 8: Start a Clean Service for A/B
Profiling is responsible for locating; the endpoint in the same environment without a profiler is responsible for confirming real throughput, TTFT, TPOT, and output consistency.
9. What This Means for Hygon DCU Developers
This trace proves that in the gfx936 local software stack, the PyTorch/ROCm-compatible trace path can simultaneously observe:
- Tensile GEMM;
- custom HIP kernels;
- Triton kernels;
- CUDAGraph replay;
- Attention and GDN paths;
- Prefill and Decode contexts.
But the data volume is large. What is truly useful is not generating a trace file, but compressing 60,124 kernel events into a verifiable model structure:
1 | 176 次 LLMM1/token |
Only when the call count, layer count, and time all close does the profile turn from a “pretty timeline” into engineering evidence that can guide Hygon DCU optimization.
10. Conclusion
From 235,964 trace events, the local reconstruction yielded:
- three q=4096 Prefill chunks;
- one q=1685 tail;
- 60 Decode steps;
- 176 LLMM1 calls per token;
- 129 rocBLAS GEMV calls per token;
- Linear totaling 38.078 ms/token;
- Linear accounting for 88.33% of Decode.
The core of this article is not a kernel ranking, but a Hygon DCU profiling method:
First split phases by execution context, then reconstruct the model structure using call counts, and finally convert total time into a per-token physical budget.
Profiling is used to decide where it is worth doing work; endpoint A/B is used to decide whether the work is actually effective. The two kinds of evidence cannot be mixed.
The data in this article comes from a local gfx936 diagnostic trace with a clear coverage scope. Profiler execution has additional overhead; the profile times in this article are only used for operator attribution and budget analysis and do not represent the final throughput of a service without a profiler.
