关于x3派官方c++例子media_cdev/rtsp2display，怎么获取解码后的vps缩放图像

lijiahui · 2023 年1 月 11 日 03:32

用户您好，请详细描述您所遇到的问题：

系统软件版本： x3_ubuntu_v1.1.0
问题涉及的技术领域：其他
问题描述：根据官方的例子，程序中对rtsp进行解码，程序管道绑定了decoder跟vps，vps跟display，但是我通过sp_vio_get_frame函数获取vps缩放图像时返回-1，获取图像失败。
复现概率：高已进行的排查措施、分析及结果：
例子中分别对decoder，vps跟display模块进行初始化并绑定，正常情况应该通过vps模块直接getimage能获取缩放后的图像
提供必要的问题日志：

问题日志：

软件上是否有做自定义修改：

新增修改代码段：

sunkaikai · 2023 年1 月 11 日 03:49

您好：

先确认一下你的目的，原有程序是能够获取rtsp解码后的数据，你想把解码后的数据回灌到VPS，进行缩小操作，对吧？

ping01wang · 2023 年1 月 11 日 03:39

您好，

没有出错的时候的dmesg日志吗，看着是打开节点的时候出现了不允许的操作的

lijiahui · 2023 年1 月 11 日 03:43

有的，运行时的dmesg日志如下：

[ 4992.390291] LT8618_GET_EDID_RESOLUTION_RATIO

[ 4992.560338] [VPUDRV]vpu_open:1944: enter

[ 4992.560365] hobot_dmcfreq_target: dmcfreq->rate:2666000000, target_rate:2666000000

[ 4992.560376] buf:performance

, powersave_rate:2666000000, dmcfreq->pre_state:0

[ 4992.560399] [VPUDRV]vpu_open:1983: leave

[ 4992.560536] [VPUDRV]vpu_ioctl:2291: [+]VDI_IOCTL_GET_INSTANCE_POOL

[ 4992.560558] [VPUDRV]vpu_ioctl:2337: [-]VDI_IOCTL_GET_INSTANCE_POOL

[ 4992.560591] [VPUDRV]vpu_mmap:3104: enter

[ 4992.560597] [VPUDRV]vpu_map_to_instance_pool_memory:3048: enter

[ 4992.560605] [VPUDRV]vpu_map_to_instance_pool_memory:3081: leave

[ 4992.560740] [VPUDRV]vpu_ioctl:2593: [+]VDI_IOCTL_GET_REGISTER_INFO

[ 4992.560753] [VPUDRV]vpu_ioctl:2601: [-]VDI_IOCTL_GET_REGISTER_INFO vpu_register.phys_addr==0xa8000000, s_vpu_register.virt_addr=0xffffff8010930000,s_vpu_register.size=65536

[ 4992.560768] [VPUDRV]vpu_mmap:3104: enter

[ 4992.560773] [VPUDRV]vpu_map_to_register:2982: enter

[ 4992.560780] [VPUDRV]vpu_map_to_register:3002: leave

[ 4992.560866] [VPUDRV]vpu_ioctl:2276: [+]VDI_IOCTL_SET_CLOCK_GATE

[ 4992.560873] [VPUDRV]vpu_ioctl:2285: [-]VDI_IOCTL_SET_CLOCK_GATE

[ 4992.560919] [VPUDRV]vpu_ioctl:2355: [+]VDI_IOCTL_GET_COMMON_MEMORY

[ 4992.560926] [VPUDRV]vpu_ioctl:2388: [-]VDI_IOCTL_GET_COMMON_MEMORY

[ 4992.560946] [VPUDRV]vpu_mmap:3104: enter

[ 4992.560951] [VPUDRV]vpu_map_to_physical_memory:3012: enter

[ 4992.560975] [VPUDRV]vpu_map_to_physical_memory:3032: leave

[ 4992.561357] [VPUDRV]vpu_release:2881: enter

[ 4992.561364] [VPUDRV]vpu_free_instances:1317: enter

[ 4992.561369] [VPUDRV]vpu_free_instances:1450: leave

[ 4992.561374] [VPUDRV]vpu_free_buffers:1461: enter

[ 4992.561380] [VPUDRV]vpu_free_buffers:1484: leave

[ 4992.561386] [VPUDRV]vpu_release:2918: free instance pool

[ 4992.561422] hobot_dmcfreq_target: dmcfreq->rate:2666000000, target_rate:2666000000

[ 4992.561432] buf:performance

, powersave_rate:2666000000, dmcfreq->pre_state:0

[ 4992.561453] [VPUDRV]vpu_release:2954: leave

[ 4992.561956] [VPUDRV]vpu_open:1944: enter

[ 4992.561973] hobot_dmcfreq_target: dmcfreq->rate:2666000000, target_rate:2666000000

[ 4992.561982] buf:performance

, powersave_rate:2666000000, dmcfreq->pre_state:0

[ 4992.562001] [VPUDRV]vpu_open:1983: leave

[ 4992.562032] [VPUDRV]vpu_ioctl:2291: [+]VDI_IOCTL_GET_INSTANCE_POOL

[ 4992.562052] [VPUDRV]vpu_ioctl:2337: [-]VDI_IOCTL_GET_INSTANCE_POOL

[ 4992.562070] [VPUDRV]vpu_mmap:3104: enter

[ 4992.562075] [VPUDRV]vpu_map_to_instance_pool_memory:3048: enter

[ 4992.562082] [VPUDRV]vpu_map_to_instance_pool_memory:3081: leave

[ 4992.562125] [VPUDRV]vpu_ioctl:2593: [+]VDI_IOCTL_GET_REGISTER_INFO

[ 4992.562137] [VPUDRV]vpu_ioctl:2601: [-]VDI_IOCTL_GET_REGISTER_INFO vpu_register.phys_addr==0xa8000000, s_vpu_register.virt_addr=0xffffff8010930000,s_vpu_register.size=65536

[ 4992.562148] [VPUDRV]vpu_mmap:3104: enter

[ 4992.562153] [VPUDRV]vpu_map_to_register:2982: enter

[ 4992.562159] [VPUDRV]vpu_map_to_register:3002: leave

[ 4992.562193] [VPUDRV]vpu_ioctl:2276: [+]VDI_IOCTL_SET_CLOCK_GATE

[ 4992.562199] [VPUDRV]vpu_ioctl:2285: [-]VDI_IOCTL_SET_CLOCK_GATE

[ 4992.562207] [VPUDRV]vpu_ioctl:2355: [+]VDI_IOCTL_GET_COMMON_MEMORY

[ 4992.562213] [VPUDRV]vpu_ioctl:2388: [-]VDI_IOCTL_GET_COMMON_MEMORY

[ 4992.562230] [VPUDRV]vpu_mmap:3104: enter

[ 4992.562235] [VPUDRV]vpu_map_to_physical_memory:3012: enter

[ 4992.562257] [VPUDRV]vpu_map_to_physical_memory:3032: leave

[ 4992.562403] [VPUDRV]vpu_ioctl:2609: [+]VDI_IOCTL_ALLOCATE_INSTANCE_ID

[ 4992.562412] [VPUDRV]vpu_ioctl:2631: [-]VDI_IOCTL_ALLOCATE_INSTANCE_ID id = 0

[ 4992.575195] channel id is 0, enable is 0, reg value is 0x44ef00f.

[ 4992.575210] channel id is 1, enable is 0, reg value is 0x44ef00f.

[ 4992.575217] channel id is 2, enable is 0, reg value is 0x4ef00f.

[ 4992.575224] channel id is 3, enable is 0, reg value is 0x4ef00f.

[ 4992.575411] channel id is 1, enable is 1, reg value is 0x24ef00f.

[ 4992.575448] iar_output1_reqbufs.

[ 4992.575921] iar_output_stream.

[ 4992.576540] channel id is 1, enable is 1, reg value is 0x24ef00f.

[ 4992.577958] hobot_dev_ipu: [S0][V4] ipu_bind_chain_group done

[ 4992.577976] hobot_dev_ipu: [S0][V4] ipu_set_group_leader

[ 4992.577979] hobot_dev_ipu: ipu_video_init group->leader:0

[ 4992.578092] hobot_dev_ipu: [S0][V4]ipu_video_init done

[ 4992.578109] hobot_dev_ipu: [S0][V4]ipu_video_reqbufs buffer number 4 first index 0

[ 4992.587413] hobot_dev_ipu: [S0][V0] ipu_bind_chain_group done

[ 4992.587420] hobot_dev_ipu: reuse_shadow0_count = 1

[ 4992.587460] hobot_dev_ipu: [S0][V0] ipu_set_group_leader

[ 4992.587463] hobot_dev_ipu: ipu_video_init group->leader:0

[ 4992.587467] hobot_dev_ipu: [S0][V0]ipu_video_init done

[ 4992.587580] hobot_dev_ipu: [S0][V0]ipu_video_reqbufs buffer number 32 first index 0

[ 4992.587694] vio_group_api: Stream0 path: G0=>G2

[ 4992.587698] hobot_dev_ipu: [S0][V0] ipu_init_end done

[ 4992.587791] hobot_dev_ipu: [S0][V0] ipu_video_streamon

[ 4992.587804] hobot_dev_ipu: [S0][V4] ipu_video_streamon

[ 4992.589091] [VPUDRV]vpu_ioctl:2396: [+]VDI_IOCTL_OPEN_INSTANCE

[ 4992.589296] [VPUDRV]vpu_ioctl:2464: [-]VDI_IOCTL_OPEN_INSTANCE core_idx=0, inst_idx=0, s_vpu_open_ref_count=1, inst_open_count=1

[ 4992.593176] vio bind info dev open.

[ 4992.593197] vio bind info dev close.

[ 4992.593224] vio bind info dev open.

[ 4992.593229] vio bind info dev close.

[ 4992.604213] hobot_dev_ipu: [S0][V0]ipu_video_streamoff:proc 0, fst_ind 0, num:32

[ 4992.604238] hobot_dev_ipu: [S0]IPU close node V0 proc 0

[ 4992.604268] hobot_dev_ipu: [S0][V4]ipu_video_streamoff:proc 0, fst_ind 0, num:4

[ 4992.605334] [S0] ipu last process close disable channels

[ 4992.605339] [S1] ipu last process close disable channels

[ 4992.605344] [S2] ipu last process close disable channels

[ 4992.605349] [S3] ipu last process close disable channels

[ 4992.605356] hobot_dev_ipu: [S0]IPU close node V4 proc 0

[ 4992.610648] [VPUDRV]vpu_release:2881: enter

[ 4992.610655] [VPUDRV]vpu_free_instances:1317: enter

[ 4992.610662] [VPUDRV]vpu_free_instances:1337: vpu_free_instances detect instance crash instIdx=0, coreIdx=0, vip_base=ffffff800a38c000, instance_pool_size_per_core=1656, interrupt_flag=0

[ 4992.610667] [VPUDRV]vpu_close_instance:650: [+] core=0, inst=0, product_code=0x521c

[ 4992.610670] [VPUDRV]wave_close_instance:590: [+] core=0, inst=0

[ 4992.610674] [VPUDRV]vpu_check_is_decoder:641: vpu_check_is_decoder is_decoder=0x1

[ 4992.610699] [VPUDRV]vpuapi_close:1140: [+] core=0, inst=0, product_code=0x521c

[ 4992.610725] [VPUDRV]vpuapi_close:1175: [-] ret=0

[ 4992.610727] [VPUDRV]wave_close_instance:622: [-] ret=0

[ 4992.610730] [VPUDRV]vpu_close_instance:658: [-] success=1

[ 4992.610736] [VPUDRV]vpu_free_instances:1450: leave

[ 4992.610738] [VPUDRV]vpu_free_buffers:1461: enter

[ 4992.611139] [VPUDRV]vpu_free_buffers:1484: leave

[ 4992.611142] [VPUDRV]vpu_release:2918: free instance pool

[ 4992.611163] hobot_dmcfreq_target: dmcfreq->rate:2666000000, target_rate:2666000000

[ 4992.611168] buf:performance

, powersave_rate:2666000000, dmcfreq->pre_state:0

[ 4992.611182] [VPUDRV]vpu_release:2954: leave

lijiahui · 2023 年1 月 11 日 03:53

对的

ping01wang · 2023 年1 月 11 日 05:35

驱动没有明显的错误，如果是节点打开的错误的话，可能错误还在上层，需要logcat信息的

sunkaikai · 2023 年1 月 11 日 04:15

从你上面的dmesg来看，没有错误log，那应该是在上层就报错了，需要用logcat抓一下log

lijiahui · 2023 年1 月 11 日 06:15

请问如何调出logcat，或者有没有实现思路，在哪里将解码后的数据截取以及回灌到vps进行缩放，我理解不是绑定了decoder跟vps，解码后的图像会自动流向vps，只需要调用vps获取图像方法就可以了吗？

sunkaikai · 2023 年1 月 11 日 07:04

1.先执行logcat > log.txt &，然后再运行测试程序，就会把hal log保存到log.txt；

2.你调用绑定API是哪个，我看下是否已经实现了数据传递的功能

sunkaikai · 2023 年1 月 11 日 07:18

你把绑定，设置vps输出，get frame的代码一起贴下，从你最开始的log看，”no vps chn can be get“有可能是VPS输出配置没配对；

lijiahui · 2023 年1 月 11 日 08:04

这是相关代码段

 // init module void *decoder = sp_init_decoder_module(); void *display = sp_init_display_module(); void *vps = sp_init_vio_module(); int output_size = FRAME_BUFFER_SIZE(rtsp_w, rtsp_h); char *frame_buffer_output = malloc(output_size); // get display resolution sp_get_display_resolution(&disp_w, &disp_h); printf("rtsp_w:%d,rtsp_h:%d\ndisplay_w:%d,dispaly_h:%d\n", pVideoCodecCtx->width, pVideoCodecCtx->height, disp_w, disp_h); // decode setting // Setting the stream_file to null means that the decoded source is manually called sp_decoder_set_image func    ret = sp_start_decode(decoder, "", 0, SP_ENCODER_H264, rtsp_w, rtsp_h); // if rtsp resolution doesn't match display's,using vps to scale // refering our decoder2display sample if (ret != 0)    { printf("decoder start error!\n"); goto EXIT1;    }    ret = sp_start_display(display, 1, disp_w, disp_h); if (ret != 0)    { printf("display start error!\n"); goto EXIT1;    }    ret = sp_open_vps(vps, 0, 1, SP_VPS_SCALE, rtsp_w, rtsp_h,                      &disp_w, &disp_h, NULL, NULL, NULL, NULL, NULL); if (ret != 0)    { printf("[Error] sp_open_vps failed!\n"); goto EXIT1;    } printf("sp_open_vps success!\n"); // bind decoder and vio(vps)    ret = sp_module_bind(decoder, SP_MTYPE_DECODER, vps, SP_MTYPE_VIO); if (ret)    { printf("[Error] sp_module_bind failed, ret = %d\n", ret); goto EXIT1;    } // bind vio(vps) and display    ret = sp_module_bind(vps, SP_MTYPE_VIO, display, SP_MTYPE_DISPLAY); if (ret)    { printf("[Error] sp_module_bind failed, ret = %d\n", ret); goto EXIT1;    }    /*End*/ while (!is_stop)    { do        {#ifdef SP_DEBUG printf("start av_read_frame\n");#endif            ret = av_read_frame(ifmt_ctx, &pkt); // read frames#ifdef SP_DEBUG printf("end av_read_frame\n");#endif if (pkt.stream_index == video_st_index)            {#ifdef SP_DEBUG printf("video_st_index\n"); printf("pkt.size=%d, pkt.pts=%lld, pkt.data=0x%x.\n", pkt.size, pkt.pts, (unsigned int)pkt.data);#endif sp_decoder_set_image(decoder, pkt.data, 0, pkt.size, 0);//sending stream frame to decoder             }        } while (ret == AVERROR(EAGAIN) && (!is_stop)); if (ret < 0)        { printf("Could not read frame ---(error '%s')\n", av_make_error_string(errbuf, sizeof(errbuf), ret)); goto EXIT;        } av_packet_unref(&pkt); int output_size = FRAME_BUFFER_SIZE(disp_w, disp_h); char *frame_buffer_output = malloc(output_size);        ret = sp_vio_get_frame(vps, frame_buffer_output, disp_w, disp_h, 2000);//get frame,672*672 is for bpu input tensors if (ret < 0)        { printf("Could not read frame ---(error '%s')\n", av_make_error_string(errbuf, sizeof(errbuf), ret)); goto EXIT;        }    } free(frame_buffer_output); sp_module_unbind(vps, SP_MTYPE_VIO, display, SP_MTYPE_DISPLAY); sp_module_unbind(decoder, SP_MTYPE_DECODER, vps, SP_MTYPE_VIO);

lijiahui · 2023 年1 月 11 日 08:06

运行环境貌似没有logcat工具，调用绑定的api是sp_module_bind(decoder, SP_MTYPE_DECODER, vps, SP_MTYPE_VIO);

sunkaikai · 2023 年1 月 11 日 08:22

2种方案尝试一下：

换成 672x672那个试试

2.如果没有显示器，去掉ret = sp_module_bind(vps, SP_MTYPE_VIO, display, SP_MTYPE_DISPLAY);这行代码去掉试下

lijiahui · 2023 年1 月 11 日 08:24

“no vps chn can be get”有可能是给getframe分配的buffer空间不够大，我将size改成3000000000后没有报“no vps chn can be get”，但是还是返回-1

lijiahui · 2023 年1 月 11 日 08:33

你好，我按照你提供的方案修改了如下代码段

这是运行的输出

getframe还是返回-1

sunkaikai · 2023 年1 月 11 日 08:38

如果这两种方案都不行，那只能使用非绑定模式，用send的方式，先send，再进行get；rtsp解码模块bind vps的有可能内部没有测试到，存在bug

lijiahui · 2023 年1 月 11 日 08:45

我尝试了将绑定关系去掉，然后直接用decoder去getframe，也是返回-1，相关代码段如下

注释绑定代码

在解码器setframe后使用getframe

这是程序输出

sunkaikai · 2023 年1 月 11 日 08:51

从decode获取数据之后，需要先调用send函数给VPS，然后才能get，你的代码里面没有调用send函数给VPS发数据

sunkaikai · 2023 年1 月 11 日 08:57

先sp_vio_set_frame，再sp_vio_get_frame

lijiahui · 2023 年1 月 11 日 08:57

我理解你的意思，decode调用set函数传数据进去解码后先要用decode调用get获取解码后的数据，再用vps调用setget来获取缩放的图像，但是代码在decode调用get时就报错了喔