Quanti onnx and Arm result Strict check FAILED

malakas · 2022 年10 月 25 日 07:39

用户您好，请详细描述您所遇到的问题：

系统软件版本： x3_ubuntu_v1.1.0
OE版本号：ai_toolchain_2.3.3
问题涉及的技术领域： ai
问题描述：板端验证bin，结果和docker中的不一致。docker中通过inference.sh测试quantized onnx结果和origin onnx基本一直。hb_model_verifier测试结果也显示不一致。
复现概率：必现
提供必要的问题日志：

2022-10-25 14:58:18,873 INFO ================== Sim infer log end ==========================

2022-10-25 14:58:18,873 INFO Sim infer finish

2022-10-25 14:58:18,873 INFO Result compare starts

2022-10-25 14:58:20,223 INFO mismatch line num: 10078

2022-10-25 14:58:20,223 INFO total line num: 10080

2022-10-25 14:58:20,224 INFO mismatch rate: 1.0

2022-10-25 14:58:20,224 WARNING Quanti onnx and Arm result Strict check FAILED

2022-10-25 14:58:21,571 INFO mismatch line num: 9920

2022-10-25 14:58:21,572 INFO total line num: 10080

2022-10-25 14:58:21,572 INFO mismatch rate: 0.984

2022-10-25 14:58:21,572 WARNING Quanti onnx and Sim result Strict check FAILED

2022-10-25 14:58:21,812 INFO mismatch line num: 1334

2022-10-25 14:58:21,813 INFO total line num: 10080

2022-10-25 14:58:21,813 INFO mismatch rate: 0.132

2022-10-25 14:58:21,813 WARNING Arm and Sim result Strict check FAILED

2022-10-25 14:58:21,813 INFO Result compare finish

docker中我是参考horizon_model_convert_sample/04_detection/yolov3_darknet53中的代码修改并运行的我的模型。

板端代码是修改的horizon_runtime_sample/code/00_quick_start的代码。

附件无法上传大于10m的文件，我的模型是30mb。

地瓜橙 · 2022 年10 月 27 日 13:33

你好，可以把hb_model_verifier 的日志发出来，我们进行验证

malakas · 2022 年10 月 28 日 06:03

2022-10-28 14:00:35,535 INFO Model Verifier Starts…

2022-10-28 14:00:35,535 INFO verifier tool version 1.9.9

2022-10-28 14:00:35,535 INFO PLEASE NOTE: Verifier tool only works for single input model

2022-10-28 14:00:35,535 INFO PLEASE NOTE: Verifier tool will only check the first model output

2022-10-28 14:00:35,697 WARNING Please note: this model has been modified, comparison results may not be correct

2022-10-28 14:00:35,697 INFO check dnn tool version on host machine

2022-10-28 14:00:36,057 INFO check dnn tool version on arm board

The authenticity of host ‘192.168.200.10 (192.168.200.10)’ can’t be established.

ECDSA key fingerprint is SHA256:8q+y1ZW0IgoK2LdZJFs/yKcbzyE5J/7gQWu58mBozgQ.

Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Warning: Permanently added ‘192.168.200.10’ (ECDSA) to the list of known hosts.

root@192.168.200.10’s password:

2022-10-28 14:00:45,493 INFO ================== input info parsed =============================

2022-10-28 14:00:45,493 INFO quantized onnx model: ./model_output/pp_nv12_quantized_model.onnx

2022-10-28 14:00:45,493 INFO bin model: ./model_output/pp_nv12.bin

2022-10-28 14:00:45,493 INFO march: bernoulli2

2022-10-28 14:00:45,494 INFO actual input type : nv12

2022-10-28 14:00:45,494 INFO actual input layout: NHWC

2022-10-28 14:00:45,494 INFO model train input data type: rgb

2022-10-28 14:00:45,494 INFO model train input data layout: NCHW

2022-10-28 14:00:45,494 INFO input shape: [‘?’, 384, 640, 3]

2022-10-28 14:00:45,494 INFO compare digits: 5

2022-10-28 14:00:45,494 INFO ================== input info end ==========================

2022-10-28 14:00:45,610 INFO Quanti infer starts

2022-10-28 14:00:50,346 INFO Quanti infer finish

2022-10-28 14:00:50,346 INFO Arm infer starts

2022-10-28 14:00:50,404 INFO shell file generated

root@192.168.200.10’s password:

2022-10-28 14:01:10,181 INFO ================== Arm infer log start ==========================

2022-10-28 14:01:10,182 INFO + dirname /userdata/model_verifier_test/infer.sh

+ cd /userdata/model_verifier_test

+ hrt_model_exec infer --model_file=pp_nv12.bin --input_file=input.bin --enable_dump=true --dump_format=txt --dump_precision=5

hrt_model_exec infer --model_file=pp_nv12.bin --input_file=input.bin --enable_dump=true --dump_format=txt --dump_precision=5

I0000 00:00:00.000000 3468 vlog_is_on.cc:197] RAW: Set VLOG level for “*” to 3

[BPU_PLAT]BPU Platform Version(1.3.1)!

[HBRT] set log level as 0. version = 3.14.14

[DNN] Runtime version = 1.9.9_(3.14.14 HBRT)

Load model to DDR cost 159.912ms.

file length: 368640

I1028 14:01:07.655822 3468 main.cpp:758] get model handle success

I1028 14:01:07.656008 3468 main.cpp:764] get model input count success

I1028 14:01:07.656612 3468 main.cpp:780] read file success!

I1028 14:01:07.656678 3468 main.cpp:786] get model output count success

I1028 14:01:07.656823 3468 main.cpp:792] prepare output tensor success

I1028 14:01:07.657794 3468 main.cpp:812] infer success

---------------------Frame 0 begin---------------------

I1028 14:01:07.690109 3468 main.cpp:818] task done

Infer time: 33.195 ms

---------------------Frame 0 end---------------------

I1028 14:01:10.038221 3468 main.cpp:887] Dump path: ./model_infer_input_0.txt

I1028 14:01:10.103890 3468 main.cpp:971] Dump path: ./model_infer_output_0.txt

2022-10-28 14:01:10,183 INFO ================== Arm infer log end ==========================

root@192.168.200.10’s password:

2022-10-28 14:01:12,787 INFO Arm infer finish

2022-10-28 14:01:12,788 INFO Sim infer starts

2022-10-28 14:01:12,805 INFO shell file generated

2022-10-28 14:01:18,383 INFO ================== Sim infer log start ==========================

2022-10-28 14:01:18,384 INFO ++ dirname /data/ping/model_verifier_output/sim_model/infer.sh

+ cd /data/ping/model_verifier_output/sim_model

+ hrt_model_exec infer --model_file=pp_nv12.bin --input_file=input.bin --enable_dump=true --dump_format=txt --dump_precision=5

hrt_model_exec infer --model_file=pp_nv12.bin --input_file=input.bin --enable_dump=true --dump_format=txt --dump_precision=5

I0000 00:00:00.000000 173 vlog_is_on.cc:197] RAW: Set VLOG level for “*” to 3

core[0] open!

core[1] open!

[HBRT] set log level as 0. version = 3.14.14

[DNN] Runtime version = 1.9.9_(3.14.14 HBRT)

Load model to DDR cost 129.673ms.

I1028 14:01:13.106722 173 main.cpp:758] get model handle success

I1028 14:01:13.106810 173 main.cpp:764] get model input count success

file length: 368640

I1028 14:01:13.107219 173 main.cpp:780] read file success!

I1028 14:01:13.107228 173 main.cpp:786] get model output count success

I1028 14:01:13.107270 173 main.cpp:792] prepare output tensor success

I1028 14:01:13.107301 173 main.cpp:812] infer success

WARNING: cnn_core_fc_avl_cap is ignored in pseudo_firmware

I1028 14:01:17.601486 173 main.cpp:818] task done

---------------------Frame 0 begin---------------------

Infer time: 4494.19 ms

---------------------Frame 0 end---------------------

I1028 14:01:18.355247 173 main.cpp:887] Dump path: ./model_infer_input_0.txt

I1028 14:01:18.376161 173 main.cpp:971] Dump path: ./model_infer_output_0.txt

2022-10-28 14:01:18,384 INFO ================== Sim infer log end ==========================

2022-10-28 14:01:18,384 INFO Sim infer finish

2022-10-28 14:01:18,384 INFO Result compare starts

2022-10-28 14:01:19,794 INFO mismatch line num: 10079

2022-10-28 14:01:19,794 INFO total line num: 10080

2022-10-28 14:01:19,794 INFO mismatch rate: 1.0

2022-10-28 14:01:19,794 WARNING Quanti onnx and Arm result Strict check FAILED

2022-10-28 14:01:21,167 INFO mismatch line num: 9920

2022-10-28 14:01:21,168 INFO total line num: 10080

2022-10-28 14:01:21,168 INFO mismatch rate: 0.984

2022-10-28 14:01:21,168 WARNING Quanti onnx and Sim result Strict check FAILED

2022-10-28 14:01:21,406 INFO mismatch line num: 1348

2022-10-28 14:01:21,407 INFO total line num: 10080

2022-10-28 14:01:21,407 INFO mismatch rate: 0.134

2022-10-28 14:01:21,407 WARNING Arm and Sim result Strict check FAILED

2022-10-28 14:01:21,407 INFO Result compare finish

地瓜橙 · 2022 年11 月 1 日 02:52

你好，可以加我一下，扫右下角的二维码，我们check一下