模型精度损失过大问题

我们自己对yolov5检测模型做了集成,在上板计算精度时,损失有点大,至少降了10个点,请问会是什么原因呢?下面是我们的参数配置。

1、mapper目录下面

yaml文件配置如下:

     # Copyright (c) 2020 Horizon Robotics.All Rights Reserved.## The material in this file is confidential and contains trade secrets# of Horizon Robotics Inc. This is proprietary information owned by# Horizon Robotics Inc. No part of this work may be disclosed,# reproduced, copied, transmitted, or used in any way for any purpose,# without the express written permission of Horizon Robotics Inc.# 模型转化相关的参数model_parameters:  # 浮点网络数据模型文件  onnx_model: './yolov5s-visible-lwir.onnx'  # 指定模型转换过程中是否输出各层的中间结果,如果为True,则输出所有层的中间输出结果,  layer_out_dump: False  # 用于设置上板模型输出的layout, 支持NHWC和NCHW, 输入None则使用模型默认格式  output_layout: NHWC  # 日志文件的输出控制参数,  # debug输出模型转换的详细信息  # info只输出关键信息   # warn输出警告和错误级别以上的信息  log_level: 'debug'  # 模型转换输出的结果的存放目录  working_dir: 'model_output'  # 模型转换输出的用于上板执行的模型文件的名称前缀  output_model_file_prefix: 'yolov5_mutil_modal_back'# 模型输入相关参数, 若输入多个节点, 则应使用';'进行分隔, 使用默认缺省设置则写Noneinput_parameters:  # (可不填) 模型输入的节点名称, 此名称应与模型文件中的名称一致, 否则会报错, 不填则会使用模型文件中的节点名称  input_name: 'img;lwir'  # 网络实际执行时,输入给网络的数据格式,包括 nv12/rgbp/bgrp/yuv444_128/gray/featuremap,  # 如果输入的数据为yuv444_128, 模型训练用的是rgbp,则hb_mapper将自动插入YUV到RGBP(NCHW)转化操作  input_type_rt: 'yuv444_128;yuv444_128'  # 网络训练时输入的数据格式,可选的值为rgbp/bgrp/gray/featuremap/yuv444_128  input_type_train: 'rgbp;rgbp'  # 网络输入的预处理方法,主要有以下几种:  # no_preprocess 不做任何操作  # mean_file 减去从通道均值文件(mean_file)得到的均值  # data_scale 对图像像素乘以data_scale系数  # mean_file_and_scale 减去通道均值后再乘以scale系数  norm_type: "data_scale;data_scale"  # (可不填) 模型网络的输入大小, 以'x'分隔, 不填则会使用模型文件中的网络输入大小  input_shape: '1x3x672x672;1x3x672x672'  # 图像减去的均值存放文件, 文件内存放的如果是通道均值,均值之间必须用空格分隔  mean_value: 0 0 0;0 0 0  # 图像预处理缩放比例,该数值应为浮点数  scale_value: 0.003921568627451 0.003921568627451 0.003921568627451;0.003921568627451 0.003921568627451 0.003921568627451calibration_parameters:  # 模型量化的参考图像的存放目录,图片格式支持Jpeg、Bmp等格式,输入的图片  # 应该是使用的典型场景,一般是从测试集中选择20~50张图片,另外输入  # 的图片要覆盖典型场景,不要是偏僻场景,如过曝光、饱和、模糊、纯黑、纯白等图片  # 若有多个输入节点, 则应使用';'进行分隔  cal_data_dir: './calibration_data_rgbp_visible;./calibration_data_rgbp_lwir'  # 如果输入的图片文件尺寸和模型训练的尺寸不一致时,并且preprocess_on为true,  # 则将采用默认预处理方法(opencv resize),  # 将输入图片缩放或者裁减到指定尺寸,否则,需要用户提前把图片处理为训练时的尺寸  preprocess_on: False  # 模型量化的算法类型,支持kl、max,通常采用KL即可满足要求  calibration_type: 'max'  # 模型的量化校准方法设置为promoter,mapper会根据calibration的数据对模型进行微调从而提高精度,  # promoter_level的级别,可选的参数为-1到2,建议按照0到2的顺序实验,满足精度即可停止实验  # -1: 不进行promoter  # 0:表示对模型进行轻微调节,精度提高比较小  # 1:表示相对0对模型调节幅度稍大,精度提高也比较多  # 2:表示调节比较激进,可能造成精度的大幅提高也可能造成精度下降  promoter_level: -1# 编译器相关参数compiler_parameters:  # 编译策略,支持bandwidth和latency两种优化模式;  # bandwidth以优化ddr的访问带宽为目标;  # latency以优化推理时间为目标  compile_mode: 'latency'  # 设置debug为True将打开编译器的debug模式,能够输出性能仿真的相关信息,如帧率、DDR带宽占用等  debug: False  # 编译模型指定核数,不指定默认编译单核模型, 若编译双核模型,将下边注释打开即可  # core_num: 2  # 设置每个fuction call最大执行时间,单位为us  # max_time_per_fc: 500  # 优化等级可选范围为O0~O3  # O0不做任何优化, 编译速度最快,优化程度最低,  # O1-O3随着优化等级提高,预期编译后的模型的执行速度会更快,但是所需编译时间也会变长。  # 推荐用O2做最快验证  optimize_level: 'O2'

data_transformer.py:

import syssys.path.append("../../../01_common/python/data/")from transformer import *def data_transformer(input_shape=(672, 672)):    transformers = [        PadResizeTransformer(input_shape),        TransposeTransformer((2, 0, 1)),        ChannelSwapTransformer((2, 1, 0)),        # ScaleTransformer(1 / 255),    ]    return transformers

2、tools/send_tools目录下部门内容

data_loader.py

  if algo_name == "yolov5":    transformers = [        PadResizeTransformer((672, 672)),        TransposeTransformer((2, 0, 1)),        ChannelSwapTransformer((2, 1, 0)),        ColorConvertTransformer('RGB', 'YUV444'),    ]    return transformers, 672, 672

def image_loader(image_path, algo_name, image_type):    def image_load_func(x):        return skimage.img_as_float(            skimage.io.imread(x)).astype(np.float32)    count = 0    transformers, dst_h, dst_w \        = get_transformers_by_algo_name(algo_name, image_type)    '''     if algo_name == "yolov5":            transformers = [                PadResizeTransformer((672, 672)),                TransposeTransformer((2, 0, 1)),                ChannelSwapTransformer((2, 1, 0)),                ColorConvertTransformer('RGB', 'YUV444'),            ]            return transformers, 672, 672    '''    if algo_name == "yolov2" or algo_name == "yolov3" or algo_name == "yolov5" or algo_name == 'lenet_onnx' or algo_name == 'lenet':        image_read_method = cv2.imread    else:        image_read_method = image_load_func    image = image_read_method(image_path).astype(np.float32)    org_height = image.shape[0]    org_width = image.shape[1]    if image.ndim != 3:  # expend gray scale image to three channels        image = image[..., np.newaxis]        image = np.concatenate([image, image, image], axis=-1)    image = [image]    for tran in transformers:        image = tran(image)    image = image[0].transpose((1, 2, 0)).reshape(-1).astype(np.uint8)    return org_height, org_width, dst_h, dst_w, image

send_image.py:

# Copyright (c) 2020 Horizon Robotics.All Rights Reserved.## The material in this file is confidential and contains trade secrets# of Horizon Robotics Inc. This is proprietary information owned by# Horizon Robotics Inc. No part of this work may be disclosed,# reproduced, copied, transmitted, or used in any way for any purpose,# without the express written permission of Horizon Robotics Inc.import google.protobufimport zmq_msg_pb2import zmqimport timeimport sysimport randomimport numpy as npimport timeimport astimport osimport argparsefrom data_loader import *class ZmqSenderClient:    def __init__(self, end_point):        self.context = zmq.Context()        self.socket = self.context.socket(zmq.PUSH)        self.socket.setsockopt(zmq.SNDHWM, 2)        self.socket.connect(end_point)    def __del__(self):        self.socket.close()        self.context.destroy()    def send_image_msg(self, img, image_name, image_type, org_h, org_w, dst_h,                       dst_w):        zmq_msg = zmq_msg_pb2.ZMQMsg()        image_msg = zmq_msg.img_msg        image_msg.image_data = img.tobytes()        image_msg.image_width = org_w        image_msg.image_height = org_h        image_msg.image_name = image_name        image_msg.image_format = image_type        image_msg.image_dst_width = dst_w        image_msg.image_dst_height = dst_h        zmq_msg.msg_type = zmq_msg_pb2.ZMQMsg.IMAGE_MSG        msg = zmq_msg.SerializeToString()        while True:            try:                self.socket.send(msg, zmq.NOBLOCK)                break            except zmq.ZMQError:                continue        return 0    def over(self):        time.sleep(2)        while True:            try:                zmq_msg = zmq_msg_pb2.ZMQMsg()                zmq_msg.msg_type = zmq_msg_pb2.ZMQMsg.FINISH_MSG                msg = zmq_msg.SerializeToString()                self.socket.send(msg, zmq.NOBLOCK)                break            except zmq.ZMQError:                continuedef send_images(ip, algo_name, input_file_path, is_input_preprocessed,                image_count, image_type):    end_point = "tcp://" + ip + ":6680"    print("start to send data to %s ..." % end_point)    zmq_sender = ZmqSenderClient(end_point)    paths = os.listdir(input_file_path)  # 列出文件夹下所有的目录    if image_count > len(paths):        print('image count is too large')        image_count = len(paths)    # TODO random select images when image_count < len(paths)    path_list = np.arange(image_count)    current_count = 0    for i in path_list:        path = os.path.join(input_file_path, paths[i])        if os.path.isdir(path):            print("input path contains a dir !!!")            return        if not is_input_preprocessed:            org_h, org_w, dst_h, dst_w, data = image_loader(                path, algo_name, image_type)            image_name = paths[i].encode('utf-8')        else:            res = str.split(paths[i], '_')            data = np.fromfile(path, dtype=np.float32)            # {name}_{org_h}_{org_w}_{dst_h}_{dst_w}.bin            image_name = '_'.join(res[:-4]).encode('utf-8')            org_h = int(res[-4])            org_w = int(res[-3])            dst_h = int(res[-2])            dst_w = int(res[-1].split('.')[0])        t = time.time()        start_time = int(round(t * 1000))        zmq_sender.send_image_msg(data, image_name, image_type, org_h, org_w,                                  dst_h, dst_w)        end_time = int(round(t * 1000))        current_count = current_count + 1        print(            current_count, 'image name is: %s' % image_name.decode('utf-8'),            ' send image data message done, take %d ms' %            (end_time - start_time))    zmq_sender.over()if __name__ == "__main__":    parser = argparse.ArgumentParser()    parser.add_argument(        '--algo-name',        type=str,        required=True,        help='model type, such as mobilenetv1, mobilenetv2, '        'yolov2, yolov3')    parser.add_argument(        '--input-file-path',        type=str,        required=True,        help='input files of preprocessed dir path')    parser.add_argument(        '--is-input-preprocessed',        type=ast.literal_eval,        default=True,        help=        'input image is preprocessed or not preprocessed. If is preprocessed, each file '        'each file must be named {name}_{org_h}_{org_w}_{dst_h}_{dst_w}.bin')    parser.add_argument(        '--image-count',        type=int,        required=False,        help='image count you want to eval')    parser.add_argument(        '--ip', type=str, required=True, help='ip of development board')    parser.add_argument(        '--image-type', type=int, required=False, help='image type')    args = parser.parse_args()    print(args)    # TODO random seed    send_images(args.ip, args.algo_name, args.input_file_path,                args.is_input_preprocessed, args.image_count, args.image_type)

network_data_iterator.cc

bool NetworkDataIterator::Next(ImageTensor *visible_image_tensor,ImageTensor *lwir_image_tensor){     std::cout<<"network -------------------------------"<<std::endl;    while (true) {        int visible_ret_code = network_receiver_->RecvImage(*visible_image_tensor);        int lwir_ret_code = network_receiver_->RecvImage(*lwir_image_tensor);        if (visible_ret_code == TIMEOUT || lwir_ret_code == TIMEOUT) {          LOG(WARNING) << "Receive image timeout";          continue;        }        if (visible_ret_code == FINISHED && lwir_ret_code == FINISHED) {          is_finish_ = true;        } else {          visible_image_tensor->frame_id = NextFrameId();          lwir_image_tensor->frame_id = visible_image_tensor->frame_id;        }        return !(visible_ret_code == FINISHED);    }}

精度损失:

麻烦帮忙看下

1.先确认数据预处理是否正确-
2.通过vec_diff 工具看 那些层输出差异较大-

然后发出相关log

抱歉,我是新手,请问你们这个vec_diff工具在哪