sycl: work-group argmax for short TOP_K rows ( #31110 )
sycl: work-group argmax for short TOP_K rows
Used angelX to optimize SYCL/drama 3 coordination, results are as follows
Rows with ncols <= max work-group size: one work-item per column, k rounds of work-group argmax. Ties take the lower column, NaN is never picked. Longer rows keep the existing paths.
Arc Pro B70, oneAPI 2026.1, test-backend-ops perf (existing short-row cases), base 9b56ccf vs patch, median ratio of 6 rotating A/B/A rounds:
ncols
nrows
k
base us
patch us
speedup
16
1024
1
19.97
2.88
6.95x
16
1024
2
23.83
2.90
8.25x
16
1024
8
66.12
4.66
14.21x
16
8192
1
110.42
5.96
18.51x
16
8192
2
130.21
8.82
14.77x
16
8192
8
420.62
27.27
15.42x
128
1024
1
20.49
2.94
6.97x
128
1024
2
25.89
3.45
7.52x
128
1024
8
101.67
8.95
11.36x
128
8192
1
112.19
13.29
8.44x
128
8192
2
141.18
20.60
6.85x
128
8192
8
643.21
67.60
9.51x
1024
1024
1
17.32
14.98
1.16x
1024
1024
2
28.77
24.02
1.20x
1024
1024
8
151.87
77.88
1.95x
1024
8192
1
156.49
138.78
1.13x
1024
8192
2
201.47
209.18
0.96x
1024
8192
8
1006.25
632.60
1.59x
The 1024 x 8192, k=2 case is 3.7% slower.
Correctness: test-backend-ops -o TOP_K 533/533 and -o ARGSORT 98/98 on the patch, again on the rebase onto 3f03b17 .
Full test-backend-ops aborts on base and patch alike at the non-F32 SWIGLU_OAI assert (element_wise.cpp:1187, see #30857 ). With only those cases excluded: 20145/20173 on base and patch. One CPY f16->q2_0 [256,2,3,4] failure was seen once on the patch, and 3/3 re-runs pass on base and patch. All other patch-run failures also fail on base.
Assisted-by: Claude Opus 5.5
made changes as suggested
Assisted-by: Claude Opus 5.5
Website:
https://llama.app
Attestations:
https://github.com/ggml-org/llama.cpp/attestations/60868996
macOS/iOS:
macOS Apple Silicon (arm64)
macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
macOS Intel (x64)
iOS XCFramework
Linux:
Ubuntu x64 (CPU)
Ubuntu arm64 (CPU)
Ubuntu s390x (CPU)
Ubuntu x64 (Vulkan)
Ubuntu arm64 (Vulkan)
Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries
Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries
Ubuntu x64 (ROCm 10.0)
Ubuntu x64 (OpenVINO)
Ubuntu x64 (SYCL FP32)
Ubuntu x64 (SYCL FP16)
Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Android arm64 (CPU)
Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Windows:
Windows x64 (CPU)
Windows arm64 (CPU)
Windows arm64 (OpenCL Adreno)
Windows x64 (CUDA 12) - CUDA 12.4 DLLs
Windows x64 (CUDA 13) - CUDA 13.4 DLLs
Windows arm64 (CUDA 13) - CUDA 13.4 DLLs
Windows x64 (Vulkan)
Windows arm64 (Vulkan)
Windows x64 (OpenVINO)
Windows x64 (SYCL)
Windows x64 (ROCm 10.0)
openEuler:
DISABLED
openEuler x86 (310p)
openEuler x86 (910b, ACL Graph)
openEuler aarch64 (310p)
openEuler aarch64 (910b, ACL Graph)
UI:
UI