Abstract

In large-model inference, the Attention formula is usually not the hardest part; what really determines performance is often how data lands in memory and at what granularity the hardware can fetch that data at once.

This article uses Qwen3.5’s official prefix-off long-context path as an example to discuss a problem that seems small but actually runs through vLLM configuration, KV Cache, paged attention, and DCU kernel dispatch: the default physical KV Cache page on this path is 784 tokens, while the vendor paged-attention path on Hygon DCU gfx936 requires the page length to satisfy 64-token alignment. In the end, we did not hard-code page size as a scheduler parameter; instead, following vLLM’s existing hybrid Attention/Mamba page calculation logic, we automatically derived it to the minimum compatible value 832.

This change itself differs only by one number, but it changes the precondition for whether the kernel can be invoked. Subsequently, chunked-prefill with q=4096 enters the vendor FlashAttention path, while q=1 decode continues to use the page-contiguous Triton path, avoiding mixing two completely different workloads into one kernel.

This article focuses on DCU execution granularity, KV Cache physical layout, block table address calculation, and end-to-end validation. It does not cover competition submission, repository addresses, or internal environment details.

1. Why Start Looking at Attention from the “Page”

For a context of length T, Attention needs to access historical K/V. vLLM does not allocate one contiguous K/V block per request; instead, it slices the cache into fixed-size physical pages and maps logical pages to physical pages via the block table.

The address of a token can be simplified as:

1
2
3
logical_page = token_index // BLOCK_SIZE
page_offset  = token_index %  BLOCK_SIZE
physical_page = block_table[logical_page]

Then combine this with the K/V cache stride, KV head, and head dimension to calculate the final address.

This paged design solves memory fragmentation and dynamic request management, but it also brings a hardware fact: BLOCK_SIZE is not just an integer in the scheduler; it also determines the kernel’s intra-page loop, vectorized access, and boundary mask.

If the hardware kernel wants to use 64 tokens as a regular access unit, then a physical page not divisible by 64 usually leaves an irregular tile or extra boundary handling at the tail of each page. For short requests, this may be only a small waste; for long contexts, page-table parsing and K/V reads are repeated many times.

2. Key Constraint of gfx936: wave64 and 64-Token Access Granularity

The ROCm-compatible execution environment of Hygon DCU uses wave64 execution granularity on the gfx936 path. Here wave64 should not be simplified to “the number of threads becomes 64”; it affects:

  • how threads within a wave cooperate to load vectors;
  • how the token dimension of K/V is assigned to lanes;
  • whether contiguous addresses can form regular memory transactions;
  • how the kernel handles page-tail masks and cross-page access.

In the vendor paged-attention interface used in this experiment, the physical page length must satisfy 64-token alignment. What is directly confirmed by source code and runtime logs is this interface constraint; it matches the access granularity of wave64, but this article does not attribute 64 alignment solely to wave64. The default 784 can be decomposed as:

1
784 = 12 × 64 + 16

and 832 can be decomposed as:

1
832 = 13 × 64

This means page832 can internally consist of 13 complete 64-token access groups, without leaving an irregular 16-token tail block at the end of each physical page. The benefit here does not come from reducing Attention mathematical computation, but from aligning data layout with hardware execution granularity.

It should be emphasized that page832 is not an empirical parameter of “bigger is faster”. 1024 also satisfies 64 alignment, but it has about 30.6% more token capacity than 784; 832 increases by only about 6.1% and is the minimum compatible page satisfying the constraint, more suitable for memory-constrained 27B hybrid models.

3. Why Qwen3.5’s Hybrid Cache Makes This Problem More Complex

Qwen3.5 contains both full attention and GDN/Mamba-like state paths. vLLM needs to keep the Attention page and Mamba state page compatible in the hybrid model; otherwise, the same request encounters different cache granularity and padding rules across layers.

On the path with prefix caching disabled, page selection can be abstracted as (omitting the minimum comparison and subsequent padding branches in the existing configuration):

1
2
3
4
attn_page_size_1_token = FullAttentionSpec(...).page_size_bytes
required_page = ceil(mamba_page_bytes /
                     (alignment_tokens * attn_page_size_1_token))
                 * alignment_tokens

For this Qwen3.5/DCU shape, backend adaptation raises alignment_tokens from the generic 16 to 64; after calculation along the existing verify_and_update_config logic, the default page is rounded up from 784 to 832.

This is two different things from directly writing block_size = 832 in the scheduler:

  1. The calculation is still completed by existing model configuration and cache-spec data flow;
  2. Page size is still passed to subsequent components as the result of KV Cache physical layout;
  3. The scheduler’s token budget, chunk count, and request scheduling semantics remain unchanged;
  4. When Qwen3.5, ROCm, head size, KV head count, and prefix state conditions are not met, the original logic is preserved.

The core of such a change is “letting the backend’s physical constraints enter the existing layout calculation”, rather than bypassing vLLM’s cache model with a new parameter.

4. Actual Page Shape of Qwen3.5

The full-attention shape in this experiment is highly deterministic:

1
2
3
4
5
query heads: 24
KV heads:     4
GQA ratio:    6
head size:    256
dtype:        BF16

vLLM’s complete K/V cache tensor also includes a K/V dimension; the shape can be written as:

1
[num_physical_pages, 2, BLOCK_SIZE, 4, 256]

After unbind(1), the shape of an individual K or V cache is:

1
[num_physical_pages, BLOCK_SIZE, 4, 256]

When K and V are stored separately, the data volume of a single page is approximately:

1
BLOCK_SIZE × 4 × 256 × sizeof(BF16)

For K and V together, multiply by 2. After switching to page832, a single page holds a few more tokens, but every 64-token group within the page is complete. This increased capacity cost must be evaluated together with kernel memory access efficiency; one cannot look only at single-page latency.

5. From Page Table to Coalesced Load

The core loop of a generic Triton paged-attention kernel can be abstracted as follows:

1
2
3
4
5
6
7
8
9
10
for tile in key_tiles:
    token = tile * TILE_SIZE + lane_offset
    logical_page = token // BLOCK_SIZE
    page_offset = token % BLOCK_SIZE
    physical_page = block_table[logical_page]

    K = load(key_cache[physical_page, page_offset, kv_head, dim])
    V = load(value_cache[physical_page, page_offset, kv_head, dim])
    score = dot(Q, K) * scale
    update_online_softmax(score, V)

On page784, the result of token % BLOCK_SIZE leaves an interval at the page tail that cannot be aligned with a 64-token access group. A generic kernel can guarantee correctness via mask, but mask does not turn irregular addresses into regular addresses; what is described here is mechanistic inference of memory-access layout, not additional hardware trace counts.

On page832, the token interval within a page can naturally be divided into complete 64-token groups. The vendor kernel still needs to handle the actual length of the last request, but it does not need to face a fixed 16-token tail block inside every physical page.

The optimization point here is not “removing the causal mask”. The causal constraint is fully preserved; what changes is the intra-page loop and physical address layout, making vector loads better suited to DCU execution granularity.

6. Why the Vendor Kernel Only Takes Over Strict Single-Sequence Prefill Shapes

Long-context serving is not a single workload. q=1 decode, q=4096 chunked-prefill, and the final tail chunk of less than 4096 differ in parallelism, memory-access direction, and graph capture method.

Therefore, the vendor dispatch of the final Tail-Q version uses strict conditions rather than global replacement:

1
2
3
4
5
6
7
8
9
10
11
12
1 < num_actual_tokens <= 4096
max_seqlen_q == num_actual_tokens
single sequence
decoder attention
BF16 Q/K/V/output
Qwen3.5: 24 Q heads, 4 KV heads, head size 256
page size = 832
GQA ratio = 6
causal = true
no sliding window
no ALiBi / sinks / softcap
no multimodal prefix range

When the conditions are met, call the existing vendor varlen_fwd_unified interface:

1
2
3
4
5
6
q, k, v
cu_seqlens_q
seqused_k
block_table
max_seqlen_q / max_seqlen_k
causal = true

If not met, fall back to the original unified Triton attention. This design has two engineering implications:

  • The vendor kernel only faces the fixed shapes it is really good at;
  • Even if the new path is unsuitable at some boundary, it will not pollute generic attention.

The initial candidate only covered q=4096; the final Tail-Q version covers 1 < q <= 4096 and max_seqlen_q == q, explicitly excluding q=1 decode. Thus the last prefill chunk of less than 4096 can also enter the same class of full causal attention, without changing the scheduler’s chunking logic.

7. Why Decode Needs a Separately Maintained Page-Contiguous Path

prefill has a large q, suitable for letting vendor FlashAttention handle an entire query chunk; decode q is usually 1, and the main problem becomes reading K/V from a large number of historical pages and completing online softmax with low launch overhead.

The loop of the page-contiguous decode kernel is closer to:

1
2
3
4
5
6
7
for page_idx in pages_for_sequence:
    physical_page = block_table[page_idx]
    for local_tile in tiles_inside_page:
        page_offset = local_tile * TILE_SIZE + offsets
        K = load(K_cache[physical_page, page_offset, ...])
        V = load(V_cache[physical_page, page_offset, ...])
        update_online_softmax(Q, K, V)

This path still parses the block table; it does not skip any logical page, nor change the causal range. It only reduces generic scheduling overhead for the access geometry of q=1, GQA6, head size 256, and supports both page784 and page832.

On page832, the decode speedup of the standalone CUDAGraph microbench is approximately:

KV length Generic page832 page-contiguous Speedup
8192 0.115989 ms 0.080010 ms 1.45x
16384 0.154312 ms 0.122411 ms 1.26x
32768 0.405256 ms 0.278832 ms 1.45x

This set of results is bitwise exact; however, the speedup of an independent kernel cannot be directly treated as end-to-end serving speedup, because decode is also surrounded by Linear, sampling, graph replay, and Python/scheduling boundaries.

8. Measured Results: How Kernel Speedup Propagates to the Serving Layer

Under the same container, same model, same native extension, and same dataset, baseline uses the official prefix-off page784 path, candidate uses page832, vendor prefill, and page-contiguous decode. Each side completed 10/10 for all three tiers, failed=0. Runtime logs also confirm four key facts:

1
2
3
4
baseline: prefix caching = False, block size = 784
candidate: prefix caching = False, block size = 832
vendor q=4096 prefill marker: hit
page832 decode marker: hit

The relative changes of the local three-tier A/B are:

Context tier Throughput change
4K-8K +0.62%
8K-16K +2.67%
16K-32K +13.78%

q=4096 vendor prefill standalone is about 2.75x-2.92x relative to the page784 Triton path; the improvement in full serving is significantly smaller than this number. The long tier has more obvious gains because in this test workload, q=4096 chunks and long-sequence paged-KV access account for a higher proportion; this is an explanation derived from workload/profile and should not be understood as a single trace already proving the exact proportion of a certain memory transaction.

This also explains why one cannot use only a kernel table as the conclusion: in the official 8K profile of page832, prefix-off, all-Tail-Q, the total Attention bucket of q=4096 prefill drops from 681.071 ms to 238.718 ms; this is not the standalone time of a single vendor kernel, and end-to-end serving still pays GEMM, GDN, decode, and scheduling costs.

The numerical scope here must be separated: 2.75x-2.92x is standalone kernel comparison, +0.62%/+2.67%/+13.78% is same-container end-to-end A/B, 681.071/238.718 ms is the profile’s total Attention bucket. They cannot substitute for each other, nor can local A/B ratios be directly treated as official absolute throughput on another machine.

9. Numerical Correctness: Fast Does Not Mean Bitwise Identical

vendor BF16 attention and Triton attention compute the same complete causal Attention, but the parallel reduction order may differ. Therefore:

  • vendor standalone output is finite and stable across repeated runs;
  • page832 Triton and page784 Triton can be bitwise identical;
  • relative L2 error between vendor and Triton is about 0.0058-0.0064;
  • generated text is not guaranteed to be bitwise identical per request.

This is not a detail that can be ignored. Tiny differences in BF16 reduction may be amplified in long-sequence generation, so one must simultaneously do:

  1. kernel output finite check;
  2. repeated-run consistency check;
  3. relative error comparison with reference kernel;
  4. four types of task-level accuracy smoke;
  5. record generated text and token count per request.

In this route, Q/K/V are not skipped, the causal range is not shortened, no sparse attention is done, and no cross-request KV reuse is done; the changes are physical page layout and equivalent kernel selection.

10. An Important Failure Case: page832 Once “Seemed Ineffective”

The most valuable finding of this route is not the first 2.9x result, but the later discovery that the test path behind the old conclusion was incomplete.

The old experiment only submitted the attention dispatch change, but did not simultaneously submit the page-layout alignment logic. The official loader disabled prefix caching, and the actual runtime was still page784; because 784 does not satisfy the vendor kernel’s 64-token page requirement, the vendor path did not hit at all.

On the other hand, the old fast-start script enabled prefix caching, and Mamba align raised the page to 1024. The local result was therefore page1024, not the official path’s page832.

Thus a typical contradiction appeared:

1
2
standalone page832 vendor: very fast
old official submission: almost no gain

The correct conclusion is not “standalone is untrustworthy”, nor “official is definitely unsuitable”, but to first check:

1
2
3
4
Does the source code really contain the layout change?
What is the runtime block size?
Does the vendor marker appear?
Did prefix caching change the page?

Only when these conditions align do performance numbers belong to the same path.

11. Reproduction and Acceptance Checklist

If reproducing on another DCU or another vLLM version, it is recommended to follow this order:

11.1 First Confirm Hardware and Software Constraints

1
2
3
4
Confirm gfx936/corresponding DCU target architecture
Confirm wave64-related kernel constraints
Confirm vendor FlashAttention page alignment requirements
Confirm the real model shape of BF16, GQA6, head size 256

11.2 Then Confirm Page Calculation

1
2
3
4
Print prefix caching status
Print final cache block size
Print attention page bytes and Mamba page bytes
Confirm page832 comes from alignment calculation, not a hidden parameter in the script

11.3 Finally Confirm Kernel Hit

1
2
3
4
Print vendor prefill marker
Print page-contiguous decode marker
Check whether it falls back to unified Triton when conditions are not met
Check that block table and K/V stride are consistent with page size

11.4 End-to-End Acceptance

1
2
3
4
5
Same container, same model, same dataset
Sufficient warmup to avoid first-compilation pollution
Three tiers of throughput, failed = 0
Record TTFT, TPOT, output token count, and text
accuracy smoke

12. Conclusion: The Real Optimization Happens at the Boundary Between “Hardware Constraints” and “Framework Semantics”

page832 is not an isolated tuning result; it simultaneously involves:

  • wave64 execution granularity of gfx936;
  • vendor attention’s requirement for 64-token pages;
  • page compatibility calculation of vLLM hybrid Attention/Mamba;
  • block table and stride address formulas of paged KV;
  • different access geometries of q=4096 prefill and q=1 decode;
  • numerical validation issues brought by BF16 reduction order.

If only the kernel is changed without changing page layout, the kernel may not hit at all; if only page size is changed without changing decode access, long-context gains may also be limited; if only microbench is considered without looking at output and official loader, conclusions may also be polluted by wrong paths.

For DCU inference optimization, the most reusable method is:

First translate the hardware’s access granularity into data layout constraints, then connect the layout constraints into the framework’s existing semantic paths, and finally use runtime markers and end-to-end results to prove the kernel actually hits.

This is closer to real kernel engineering than “change a BLOCK_SIZE and see”, and it better explains why a 48-token page difference repeatedly appears in every layer’s attention and multiple physical page accesses of every long request and affects overall serving performance.

This article discusses publicly explainable hardware, layout, and kernel methods; the implementation description does not include internal repository addresses, account information, container credentials, or private infrastructure details. Performance numbers come from fixed-shape same-container experiments; actual results will vary with DCU model, driver, DTK, compilation cache, and runtime load.