The biggest waste of time I encountered while modifying vLLM kernels on Hygon DCU was not a compilation error.

That time, compilation ended normally, the service could start, the endpoint could return results, and the performance test even produced a change that looked explainable. Only after rechecking the runtime environment did I discover a more troublesome problem: the source code, build artifacts, and the files actually loaded by the service were not in the same state. All previous judgments about performance had to be discarded.

This changed my debugging order. Now every time I finish modifying a ROCm kernel, I do not immediately look at throughput; instead, I first answer a more basic question:

Is what is actually executing on the Hygon DCU really the code I just modified, compiled, and was about to test?

This article records a set of vLLM incremental compilation and runtime verification methods that I have actually used on a Hygon DCU with the gfx936 architecture. The focus is not introducing some faster operator, but rather opening up one by one the links between .cu source and real kernel dispatch where stale state can easily hide.

Experimental Environment

The measured environment involved in this article is as follows. The timings and binary information later are only valid for this environment and should not be extrapolated as fixed results for other DCU, DTK, or vLLM versions.

Item Measured environment
Accelerator card Hygon DCU, gfx936 architecture, device name shown as BW
DTK DTK 26.04 series environment
Python 3.10
PyTorch 2.10.0
Build tools CMake, Ninja, hipify, ccache 4.5.1
vLLM ROCm extension vllm._rocm_C

In this vLLM source tree, _rocm_C consists of three files:

1
2
3
csrc/rocm/torch_bindings.cpp
csrc/rocm/skinny_gemms.cu
csrc/rocm/attention.cu

skinny_gemms.cu and attention.cu first go through hipify, then DTK’s compiler generates target code for gfx936, and finally they are linked together with the operator registration code into _rocm_C.abi3.so.

A kernel has at least six “identities” before it runs

Looking only at the source directory, it is easy to mistakenly think that a modification has already taken effect. The actual chain is much longer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
skinny_gemms.cu
        |
        | hipify
        v
skinny_gemms.hip
        |
        | clang++ --offload-arch=gfx936
        v
cmake-build-rocm/_rocm_C.abi3.so
        |
        | cmake --install / manual install
        v
vllm/_rocm_C.abi3.so in the Python environment
        |
        | loaded by the service process
        v
loaded shared library in /proc/<pid>/maps
        |
        | runtime conditions such as shape, dtype, layout
        v
kernel branch actually selected for this request

The six-layer identity chain from source code to real dispatch for a vLLM kernel on Hygon DCU

If any layer here is not aligned, a very dangerous state appears: the program runs, but it is not running the implementation you think it is.

Therefore, “compilation succeeded” can at most prove that the third layer generated a file. It cannot prove that the file was installed into the current Python environment, still less that the already-started vLLM process reloaded it, and finally it cannot prove that the target request actually hit the new dispatch.

The incremental compilation I actually got working

Given an existing build directory, after modifying csrc/rocm/skinny_gemms.cu, I use the following two commands:

1
2
cmake --build --preset rocm-release --target _rocm_C
cmake --install cmake-build-rocm --component _rocm_C

In this environment there is a not-so-obvious difference: when only CMakeUserPresets.json was placed, the current image’s build process did not recognize it correctly; it worked stably only after switching to the actually used CMakePresets.json. The preset must at least specify the following information:

1
2
3
4
VLLM_TARGET_DEVICE=rocm
PYTORCH_ROCM_ARCH=gfx936
ROCM_PATH=/opt/dtk
CMAKE_HIP_COMPILER_LAUNCHER=ccache

A real incremental build log can be compressed into the following lines:

1
2
3
4
5
6
7
8
9
[1/4] ... hipify.py ... skinny_gemms.cu attention.cu
skinny_gemms.cu -> skinny_gemms.hip [ok]
attention.cu -> attention.hip [skipped, already hipified]
Successfully preprocessed all matching files.
Total number of unsupported CUDA function calls: 0
Total number of replaced kernel launches: 19

[2/3] clang++ ... --offload-arch=gfx936 ... skinny_gemms.hip
[3/3] ... -o _rocm_C.abi3.so

These lines are more useful than the final build success. They simultaneously prove four things:

  1. The modified .cu was re-hipified;
  2. The unmodified attention.cu was not pointlessly re-converted;
  3. The compilation command really targeted gfx936;
  4. A shared library relink actually occurred.

The same local experiment also recorded the state of the generated extension:

1
2
3
4
size: 21018624 bytes
sha256: 0e3280734c80bf7d33e101c04eecd9cdee0844feb3447b2141a3326ad6ef7549
torch: 2.10.0
LLMM1_registered: True

This hash is only used to identify that particular binary at that time and does not mean that other correct builds should produce the same hash.

A shared directory can slow down the configuration phase

I also compared the CMake configure time for the same source tree in two locations:

Build location Measured configure time
Shared directory about 111 seconds
Container-local /root 13.5 seconds

CMake configure time comparison between a shared directory and a container-local directory

The difference is about 8.2x. This result does not mean that /root will be 8.2x faster on any machine; it means that when the source tree, CMake probes, and a large number of small-file operations all land on high-latency shared storage, the configuration phase itself can become an iteration bottleneck.

My approach is to put the temporary source overlay and cmake-build-rocm on the container-local disk, and after compilation only copy the logs, source patches, and final .so that need to be preserved back to the persistent directory. This keeps the evidence while avoiding repeated metadata access on the shared directory for every configure.

Four “looks successful” failure states

I have encountered all four of the following situations in actual debugging. They are troublesome because the service does not necessarily report an error.

1. import vllm succeeds, but the ROCm extension does not exist

After reinstalling a wheel once, the Python package was restored and import vllm worked normally, but the default installation did not contain _rocm_C.abi3.so. If you verify the environment only with import vllm, you will mistake “the Python package exists” for “the custom ROCm extension is installed”.

A more reliable check is to import the extension directly and confirm that the target operator has been registered:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
python3 - <<'PY'
import importlib.util
import os
import torch

spec = importlib.util.find_spec("vllm._rocm_C")
print("extension:", None if spec is None else spec.origin)
if spec is None:
    raise SystemExit("vllm._rocm_C is missing")

import vllm._rocm_C  # noqa: F401
op_name = os.environ.get("DCU_VERIFY_OP", "LLMM1")
print("registered:", op_name, hasattr(torch.ops._rocm_C, op_name))
PY

Here LLMM1 is the operator name I was verifying at the time. When switching to another project, set DCU_VERIFY_OP to the registration name you actually need to test.

2. After reinstalling the wheel, only one currently modified file was copied

In another round of experiments, reinstalling the wheel restored site-packages to the wheel’s default state. At that time I copied only the Python file I was currently modifying, but missed other modified files inherited by the candidate implementation. The service could still start and the benchmark could complete, but that was not the complete candidate, and the results had no comparative value.

Since then, I no longer decide the synchronization scope by “how many files I manually changed this time”; instead, I decide it by the complete source diff:

1
git diff --name-only <baseline>...HEAD

All differing files that can affect runtime must be installed, hashed, and included in the backup and restore list. For Python files, python -m py_compile should also be run before starting the service, to avoid leaving transport corruption, encoding anomalies, or incomplete files to be discovered only at the model-loading stage tens of minutes later.

3. The .so on disk is updated, but the old process is still using the original mapping

Overwriting _rocm_C.abi3.so will not cause an already-running Python process to automatically unload the old shared library. After replacing the binary, the service must be fully restarted; if CUDAGraph is used, graph capture should also be redone, and the graph state from the old process must not be reused.

I once encountered a more insidious situation: after the previous service exited, a VLLM::EngineCore process was still alive and occupying GPU memory. Although the next service started, the environment was no longer clean.

Before starting a new service, I now perform at least two read-only checks:

1
2
ps -eo pid,ppid,cmd | grep -E 'vllm|VLLM::EngineCore' | grep -v grep
ss -ltnp | grep ':8000 '

Confirm that the old service, EngineCore, benchmark driver, and listening port have all been cleaned up, then restart through the normal service management method. After startup, you can also inspect the mappings of the target process:

1
2
PID=$(pgrep -n -f 'VLLM::EngineCore')
grep '_rocm_C.abi3.so' "/proc/${PID}/maps"

It can at least answer “from which path did the current process load the extension”. If the path is not as expected, there is no need to continue analyzing the subsequent performance data.

4. Same filename, but different binary origin

Once, while cleaning up the environment, I recorded that the hash prefix of _rocm_C in the runtime environment was:

1
7aee47576aa8...

while the hash prefix of the known extension prepared as a control was:

1
f2c5393af460...

Both files are named _rocm_C.abi3.so, and the target operators can run in both, but they are not the same binary. Without a hash record, this difference is easily masked by the filename.

I now record the hashes of both the build directory and the actual installation location:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
BUILD_SO=cmake-build-rocm/_rocm_C.abi3.so
INSTALLED_SO=$(python3 - <<'PY'
import importlib.util
spec = importlib.util.find_spec("vllm._rocm_C")
if spec is None:
    raise SystemExit(1)
print(spec.origin)
PY
)

sha256sum "$BUILD_SO" "$INSTALLED_SO"
cmp -s "$BUILD_SO" "$INSTALLED_SO" \
  && echo "binary match" \
  || echo "binary mismatch"

Hash alone is not enough: you must prove that the request hit the new dispatch

Consistent hashes only prove that the service loaded the target extension; they do not prove that a request selected the new kernel. Dispatch in vLLM is often affected simultaneously by shape, dtype, layout, sequence phase, and cache configuration. If even one condition is missed, the program may silently fall back to the original path.

My approach is to keep a diagnostic marker controlled by an environment variable in the candidate branch, printing only on the line where the candidate kernel is actually selected:

1
2
3
4
5
6
7
if os.getenv("VLLM_DCU_KERNEL_DIAG") == "1":
    logger.warning(
        "DCU_KERNEL_DIAG selected=%s shape=%s dtype=%s",
        selected,
        tuple(x.shape),
        x.dtype,
    )

The key to the marker is not “the feature is enabled”; rather, selected=True must come from the actual dispatch result. My automated runner extracts the marker from the service log and treats “expected shape appears, key parameters are correct, selected=True” as a precondition for a valid benchmark.

This is especially important for ROCm kernels on Hygon DCU. The fallback in many experiments is still a usable and reasonably fast vendor or Triton path, so “the service did not report an error” absolutely cannot prove that the new kernel was executed.

The seven verification gates I now use

The following table is the check order I finally settled on. If the previous item does not pass, do not proceed to the next.

Order What to prove Minimum evidence
1 Source state is complete Complete diff list, not just currently dirty files
2 Compilation target is correct Log shows --offload-arch=gfx936 and recompilation of the target source file
3 The installed binary is new SHA256 of build .so matches the actual import path
4 Operator registration succeeded Directly import vllm._rocm_C, target op is True
5 Process state is clean No leftover EngineCore, old service, old driver, or occupied port
6 Real request hits Service log shows selected=True marker with real shape
7 Results are usable for comparison Output, request count, and failure count meet the pre-agreed conditions before discussing performance

This process looks like a few more steps than “compile and then run the benchmark directly”, but most checks take only seconds. What really takes time is skipping them and then spending hours analyzing a set of data that actually came from an old binary, half the source tree, or a fallback path.

A read-only verification script

The following is a compressed version of the common checks I use. It does not modify files and does not stop processes; it is suitable for running once before and once after service startup. The operator name can be replaced through an environment variable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/usr/bin/env bash
set -euo pipefail

export DCU_VERIFY_OP="${DCU_VERIFY_OP:-LLMM1}"

python3 - <<'PY'
import importlib.util
import os
import torch
import vllm

print("vllm:", vllm.__file__)
spec = importlib.util.find_spec("vllm._rocm_C")
print("_rocm_C:", None if spec is None else spec.origin)
if spec is None:
    raise SystemExit("missing vllm._rocm_C")

import vllm._rocm_C  # noqa: F401
name = os.environ["DCU_VERIFY_OP"]
ok = hasattr(torch.ops._rocm_C, name)
print("op:", name, "registered=", ok)
if not ok:
    raise SystemExit("target op is not registered")
PY

SO=$(python3 - <<'PY'
import importlib.util
print(importlib.util.find_spec("vllm._rocm_C").origin)
PY
)

ls -l "$SO"
sha256sum "$SO"

echo "--- active vLLM processes ---"
ps -eo pid,ppid,cmd | grep -E 'vllm|VLLM::EngineCore' | grep -v grep || true

If the .so in the build directory, the .so actually imported by Python, the .so in the process mapping, and the dispatch marker can form a closed chain of evidence, only then will I write the subsequent results into the experimental conclusion.

The boundaries of this method

There are three points that need to be made clear in this article:

  1. 111 seconds -> 13.5 seconds is an observation under specific container and shared-storage conditions, not a fixed performance metric for Hygon DCU;
  2. 21018624 bytes and the SHA256 in this article belong to one specific build and are only used to illustrate how to record binary identity;
  3. This process proves “the test object is correct”; it does not automatically prove that the kernel is faster or numerically correct. Performance and correctness still need separate verification.

However, for me, this step has already become the engineering habit most worth keeping in Hygon DCU kernel development. HIPify, the DTK compiler, the CMake build tree, the Python installation directory, the long-running vLLM process, and runtime dispatch—each layer may retain stale state. Verifying them layer by layer is far more reliable than staring at the final line build success.

Now after I finish modifying a kernel, my first reaction is no longer to see how much the number has increased, but to first confirm: is the source correct, is the binary correct, is the process correct, and did this request actually reach it? The extra cost is only a few minutes, and what is often saved is an entire round of erroneous experiments.