Product Info

I replaced a different company's GPU with SiMa.ai! [Implementation]

SiMa.ai (hereinafter referred to as SiMa) is a device with strengths in edge AI, rivaling the performance of Alpha Inc., a company that provides GPUs (graphics processing units). There are also voices expressing interest in actually using SiMa's MLSoC (Machine Learning System-on-Chip). In this article, we will introduce how to port a model that was running on Alpha Inc.'s GPU to SiMa and actually run it.

Overall Table of Contents

The transplant procedure is as follows:

Step 1: Overview

• Overview

Step-2: Preparation

• Preparation

Step 3: Environment setup

• Environment setup part 1
• Environment setup part 2
• Environment setup part 3

Step 4: Implementation

• Implementation

Goal of this chapter "Implementation"

• Learn how to build a system using the SiMa development kit

The results of this demo video are as follows:

Implementing the application

This time, we finally tracked the movements of people running on Alpha's GPU.
We will use SiMa to build a system that will perform heat map analysis of stay patterns.

The general process leading up to implementation

1. Clone YoloV7 demo
   There are some demo programs provided by SiMa in the app_zoo directory. In this case, the YoloV7 demo is the closest, so we will duplicate that folder and add the desired functions.
2. Implemented heatmap drawing method in CvHelpers
   In the plugin_zoo directory, there is the source code for the C++ Gstreamer overlay plugin provided by SiMa. Since it uses a helper class called CvHelpers that utilizes OpenCV, add the draw_heatmap method to the CvHelpers class to enable heatmap drawing processing.
3. Added heatmap option to overlay plugin
   The specification is that the overlay plugin takes the render-rules argument and performs arbitrary drawing based on the results processed by SiMa's MLA. Here, we will modify the overlay plugin in application.json to specify the heatmap option when calling it.
4. Build and deploy
   We will build the modified overlay plugin and build and deploy the application using the MPK tool.
5. Play video sent via UDP on the host machine
   This time, video data is sent to UDP port 9000 to the host machine's IP address, so it is received and played by Gstreamer.

Note that all of the above procedures are assumed to be executed within the Docker container created in the "Building the host machine environment" chapter.

Implementing heatmap in library and calling it

Clone YoloV7 demo

To clone the YoloV7 demo located in the app_zoo directory, run the following command:
In addition, since many folders have fixed file path values, we also perform a mass replacement of path names.

sima-user@docker-image-id:/home$ cd /usr/local/simaai/app_zoo/Gstreamer/ cp -r YoloV7 YoloV7_heatmap_demo sima-user@docker-image-id:/home$ find YoloV7_heatmap_demo -type f \( -name "*.json" -o -name "*.sh" \) -exec sed -i 's/YoloV7/YoloV7_heatmap_demo/g' {} \; -print sima-user@docker-image-id:/home$ find YoloV7_heatmap_demo -type f \( -name "*.json" -o -name "*.sh" \) -exec sed -i 's/yoloV7/yoloV7_heatmap_demo/g' {} \; -print

Implemented heatmap drawing method in CvHelpers

Add the draw_heatmap method to the CvHelpers class to handle heatmap drawing.
The source code modifications are as follows:

Since we are only using NV12 this time, the only thing we are actually implementing is the concrete class CvHelpersNV12 class, but it can also be used with YUV420 by making a similar implementation for the CvHelpersYUV420 class.

Added heatmap option to overlay plugin

We will modify the overlay plugin to specify the heatmap option when calling it.
The source code modifications are as follows:
Please modify the camera's rstp server information, such as location=rtsp://192.168.2.10:5005/routecam, in the gst of application.json as appropriate to match the specifications of your camera.

Build and deploy

To build after adding the plugin, run the following command.
Plugins need to be built separately from the normal build procedure using the MPK tools.

sima-user@docker-image-id:/home$ cd /usr/local/simaai/plugin_zoo/a65-apps/build && source /opt/poky/4.0.10/environment-setup-cortexa65-poky-linux && cmake -DCMAKE_INSTALL_PREFIX=/opt/poky/4.0.10/sysroots/cortexa65-poky-linux/usr/ .. && sudo make install

*The path specified in DCMAKE_INSTALL_PREFIX is the same as the path where the MPK tool searches for the built so files of the plugins required for building, so it is specified to be output to that path.

*This time we are building them all at once, but it is also possible to build only the plugins you want to build.

Next, to deploy, run the following command:

sima-user@docker-image-id:/home$ PIPELINE_PATH='/usr/local/simaai/app_zoo/Gstreamer/YoloV7_heatmap_demo' && mpk device connect -t sima@192.168.2.10 && mpk kill -i yoloV7_heatmap_demo_Pipeline && mpk remove -a com.sima.awesome_app -t 192.168.2.10 && mpk create -s $PIPELINE_PATH -d $PIPELINE_PATH && mpk deploy -f $PIPELINE_PATH/project.mpk -t 192.168.2.10

The overlay plugin is built automatically when you create it with the MPK tool, so there is no need to run the make command separately to build it.

If the build or deployment does not go well, it is a good idea to check the error logs that are output to /var/log/simaai/.

Play video sent via UDP on the host machine

To play the video data, please execute the following command.

sima-user@sima-user-machine:~$ gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, encoding-name=H264, payload=96" ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

Please check whether port 9000 is allowed on the host machine. If it is not allowed, please change the port or allow it.

The result is the following video:

By completing the steps above, the porting from Alpha's GPU to the SiMa.ai MLSoC has been completed.
I believe that Alpha has confirmed that the porting can be completed successfully.
We hope that you will be able to experience the actual performance throughout all chapters and use this information in your future development.

If you have any questions about this chapter, please contact us using the information below.