Clarification on Hardware Reproducibility and Vectorization Score Averaging

[aditya78970]

I would like to confirm two points related to NPUEval reproducibility:

1. Cycle count hardware independence

The paper reports results on a system with a Ryzen 9 7940HS. I’m testing on a Ryzen 7 8845HS
My question: Should the cycle counts (AIE clock cycles) be identical/reproducible across different devices.

2. Average vectorization score calculation
   In Figure 5 of the paper, the “average vectorization score” per model is shown. Could you clarify how this is calculated?
   Two plausible methods:
   1. Average over all 102 kernels (including those that failed/vectorization = 0)
   2. Average only over kernels which succeeded (i.e., exclude failed/null scores)

[skalade]

Hi @aditya78970,

1. Since Phoenix and Hawk point have the same NPU (one is clocked higher) the cycle counts should be near identical. However, if you were to test on Strix or Kraken I expect wildly different results because the underlying NPU architecture will be different.
2. It's the first option, we average over number of tests and treat fails as 0, it's the same math as this notebook https://github.com/AMDResearch/NPUEval/blob/main/notebooks/05_measuring_speedup.ipynb

[aditya78970]

@skalade Thanks for the clarification on averaging (treating fails as 0 and averaging over total test cases). I recalculated the scores using the same method mentioned in the notebook. However, the results I am getting are still different from the values reported in the paper. Below are the average scores I obtained:

Model | Without RAG | With RAG

gpt-40-mini | 1.9 | 2.3
gpt-4.1 | 1.653 | 7.6368
qwen-2.5 | 1 | 2.3
deepseek-v3 | 4.2 | 5.9

Let me know if there are any additional scoring variations that I should consider to match the reported results.

[skalade]

Hi @aditya78970 are these results with re-compilation attempts or zero-shot? Paper results had N=10

[aditya78970]

Hi @skalade These results were obtained with re-compilation attempts using N=10, consistent with the evaluation setup described in the paper.

[skalade]

Hmm this could be a number of things... Unfortunately I don't have a Hawk Point machine on hand to test reproducibility - maybe underlying MLIR-AIE layer does treat it differently even if the NPU is the same.

As sanity check - when you run this notebook do your results match exactly what's in cell 12?

[aditya78970]

@skalade My results for gpt-4o-mini match exactly with the values shown in cell 12.

However, for gpt-4.1, I’m seeing a slight difference — in the notebook, abs_int8 shows total_cycles=540 and vector_cycles=31, but my run produced:
abs_int8: total_cycles=255, vector_cycles=16

4o-mini |   | total_cycles | vector_cycles

abs_int8 |   | 9990 |   | 0
add_offset_int8 | 2502 |   | 0
add_offset_uint8 | 9990 |   | 0
argmax_bfloat16 | 616 |   | 63
argmax_int32 | 1792 |   | 0
  
gpt4.1 |  

abs_int8 |   | 255 |   | 16
add_offset_int8 | fail |   | fail
add_offset_uint8 | 540 |   | 31
argmax_bfloat16 | 3529 |   | 0
argmax_int32 | 1351 |   | 0

The run was performed with the following parameters: temperature=0.0, top_p=1.0, attempts=1 as mentioned in [https://github.com/AMDResearch/NPUEval/blob/main/notebooks/02_evaluating_solutions.ipynb]

[skalade]

@aditya78970 the code generated by the models may also not be exactly the same iirc. But I don't think it should make so much of a difference with the results you're seeing. It might just be the underlying tools optimizing it differently because it treats 7940HS and 8845HS differently...

My recommendation would be to look at the blog (which has more recent results) https://amdresearch.github.io/NPUEval/blog.html and evaluate the speedup you're seeing with canonical solutions being run on your setup.