Edge Device Test and Diagnostic Tool (EDTDT)

Overview

Edge Device Test and Diagnostic Tool (EDTDT) Command Line Interface (CLI) tool provides customers with the capability to run an Intel provided test suite at the target system, with the goal of enabling partners to test their platform for a specific Intel® Edge Software Hub (ESH) package.

EDTDT and the software listed in table 1 are installed when selected as part of the Edge Insights for Vision package. If you have not installed the package yet, you can download it from EDTDT for Vision and follow the installation instructions.

Table 1
Time to Complete Approximately 1 hour 15 mins for installation and execution (including Vision + Intel® Vision Accelerator test installation) 
Software
  • EDTDT CLI tool binary – compiled Python* package using  
    • Python 3.6 for Ubuntu* 18.04 
    • Python 3.8 for Ubuntu 20.04 

Target System Requirements

  • One of the following processors: 
    • 6th, 7th, 8th, 10th, 11th generation Intel® Core™ processor. 
    • 6th, 7th, 8th generation Intel® Xeon® processor. 
    • Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics. 
  • At least 16 GB RAM. 
  • At least 64 GB hard drive. 
  • A stable Internet connection. 
  • Ubuntu* 18.04 LTS with Kernel 5.0 or 5.4  
  • Ubuntu* 20.04 LTS with Kernel 5.4  
  • Kernel of 5.4 and above needed to support Intel® 10th and 11th generation Intel® Core™ processor. 
  • Intel Vision Accelerators – Intel® Movidius™ Myriad™ X Vision Processing Unit (VPU) (Optional) 

How It Works

EDTDT package contains the CLI, Python library, and Test Module Interfaces. The CLI tool launches various test modules corresponding to the package. The test modules in scope are Intel Vision test module and Intel® Vision Accelerator test suite. The test module/test suites are dependencies for the EDTDT CLI execution and are downloaded by the Edge Software Hub CLI onto the target system.  

EDTDT package and its test module dependencies will be enlisted in the edgesoftware-configuration.xml when the customer selects the option. ESH CLI will use the same mechanism to download and install the EDTDT components and test suite onto the target system with the other package components. 

The test modules interact with the EDTDT CLI through a common test module interface layer (TMI), which is part of the EDTDT binary. EDTDT generates a complete test report in html format, along with detailed logs packaged as one zip file, which the customer can manually choose to email to the ESH support team. 

NOTE: EDTDT CLI should be launched after the package is successfully downloaded and installed by the Edge Software CLI.  

 

Figure 1: Architecture Diagram

 

Intel® Edge Insights for Vision Test Module Overview

Intel® Edge Insights for Vision Test Module is a validation framework for Intel® Media SDK, Intel® Container OpenVINOTM and Intel® Runtime OpenVINOTM

The Intel products mentioned above are intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to have customized derivative builds of Intel’s reference designs, and their performance number may vary from that of Intel’s designs. This calls for a base framework that allows partners to compare their design’s performance and generate a hardware health status report.  

Edge Insights for Vision test suite is a set of test scripts to validate the Edge Insights for Vision package. The details of the Edge Insights for Vision package can be found here.  

The test suite serves the following purpose:  

  • Health Check: It runs a health check to make sure all the components that are part of the vision package listed below are properly installed and configured correctly. 
    • Reference Implementation - Social Distancing Detection for Retail Settings
    • Intel® Distribution of OpenVINO™ toolkit 2021.2 in a Container 
    • Intel® Distribution of OpenVINO™ toolkit 2021.2 Runtime 
    • Deep Learning (DL) Streamer Pipelines 
    • ONNX* Runtime 
    • EdgeX* Foundry 
    • Docker Community Edition (CE)* 
    • Docker Compose* 
    • K3s* (Lightweight Kubernetes*) 
    • Amazon Web Services Greengrass Prerequisites 
    • Azure* IoT Edge Prerequisites 
    • Connect Edge Devices to Azure IoT* 
  • Sanity tests: The test module contains sanity tests for AWS Greengrass, Azure/Kubernetes A3S and ONNX. 
  • Functionally tests at component level: Apart from the sanity test for AWS greengrass, Azure/Kubernetes A3S and ONNX , for each of the mediaSDK and OpenVINOTM components, It runs a series of sample applications such as decode, transcode, encode & object detection on CPU, GPU and Intel Vision accelerators (if present) and validates the output of the sample applications to make sure that all the functionality of the components works as expected. 
    • MediaSDK_decode_h264 
    • MediaSDK_transcode_h264_mpeg2 
    • MediaSDK_encode_h265 
    • MediaSDK_setcommands_h264 
    • MediaSDK_setcommands_h265 
    • Openvino_CPU_reshape 
    • Openvino_CPU_styletransfer 
    • Openvino_GPU_reshape 
    • Openvino_GPU_ styletransfer 
    • Openvino_HDDL_reshape 
    • Openvino_HDDL_styletransfer 
    • Openvino_CPU_objectdetection 
    • Openvino_CPU_segmentation 
    • Openvino_GPU_objectdetection 
    • Openvino_GPU_segmentation 
    • Openvino_HDDL_objectdetection 
    • Openvino_HDDL_segmentation 
    • Openvino_container_CPU_objectdetection 
    • Openvino_container_CPU_segmentation 
    • Openvino_container_GPU_objectdetection 
    • Openvino_container_GPU_segmentation 
    • Openvino_container_HDDL_objectdetection 
    • Openvino_container_HDDL_segmentation 

Intel® Vision Accelerator (Intel® VATS) Test Suite Overview

Intel® VATS is an OpenVINO™ based extensible validation framework for Intel® Vision Accelerator solutions. Intel® VATS is a scalable and portable automated test framework for Intel® Vision Accelerator products such as Intel® Movidius™ Myriad™ X VPU platform. 

Intel® Vision Accelerator Design products are reference designs intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to build custom derivatives of Intel's reference designs, whose performance number may deviate from that of Intel's designs. This calls for a base framework that allows partners to compare their designs' performance and stability numbers to those advertised by Intel. 

Intel® VATS Test Suite has categorized test cases into the following categories to test different functionalities of Intel® Vision Accelerator Test Suite:  

Usability tests: This set of tests will ensure Intel® Movidius™ Myriad™ X VPU firmware is loaded in a reasonable time, and is able to reset successfully. After the reset the test ensuresIntel® Movidius™ Myriad™ X VPU driver is responding to the hardware ping and can receive VPU IO responses from OpenVINO™. 

  • firmware_loading_time 
  • reset_1 
  • hardware_ping 
  • io_ping_test 

Performance tests: This set of tests will run OpenVINO™ benchmark_apps such as alexnet, densenet, googlenet, mobilenet, squeezenet and resnet to acquire latency and throughput performance numbers 

  • performance_alexnet 
  • performance_densenet-121 
  • performance_googlenet-v1 
  • performance_mobilenet-ssd 
  • performance_mobilenet-v1-1.0-224 
  • performance_mobilenet-v2 
  • performance_squeezenet1.1 
  • performance_resnet-50-pytorch 

Reliability test: This test will capture the thermal reading of the Intel® Movidius™ Myriad™ X VPU device after loading the neural network and running inference via benchmark_app. 

  • thermal_test 

Test cases being run using edtdt test: 

1. firmware_loading_time 

  • This test will load the model to the Intel® Movidius™ Myriad™ X VPU device through OpenVINO™ Inference Engine library. It will measure the time taken to successfully load the model and print the result. The offload time can be acquired from output/firmware_loading_time.log. 
  • Passing Criteria: The test passes if the models can be offloaded to Intel® Movidius™ Myriad™ X VPU and time taken to offload the model can be acquired successfully. Below is a snippet of the test result.  

2. reset_1 

  • This test will reset the Intel® Movidius™ Myriad™ X VPU device to its initial state. The status of the Intel® Movidius™ Myriad™ X VPU device after reset can be acquired from output/reset_1.log. 
  • Passing Criteria: The Intel® Movidius™ Myriad™ X VPU is successfully reset to its initial state. Below is a snippet of the test result.

3. hardware_ping 

  • This test will ping Intel® Movidius™ Myriad™ X VPU driver information before and after firmware is loaded with a specific model. The status of driver can be acquired from output/hardware_ping.log. 
  • Passing Criteria: The test passes if the Intel® Movidius™ Myriad™ X VPU driver information is present before and after the firmware is loaded. Below is a snippet of the test result.

4. io_ping_test 

  • This test will check Intel® Movidius™ Myriad™ X VPU IO responses via OpenVINO™ inference engine API. The ping result can be acquired from output/io_ping_test.log. 
  • Passing Criteria: The test passes if the Intel® Movidius™ Myriad™ X VPU device can successfully receive VPU IO responses from OpenVINO™ inference engine API. Below is a snippet of the test result.

5. Performance test 

  • This test will run OpenVINO™ benchmark_app to acquire latency and throughput performance numbers. 
  • Passing Criteria: The test passes if inference for specific neural networks can be run without error and Latency and Throughput values can be acquired. VATS will not do benchmarking of inference performance. The performance_<MODEL>. log can be acquired from output/performance_<MODEL>.log. Below is a snippet of the test result.

6. thermal_test 

  • This test will capture the thermal reading of the Intel® Movidius™ Myriad™ X VPU device after loading the neural network and running inference via benchmark_app. The thermal reading can be acquired from output/thermal_test.log
  • Passing Criteria: The test passes if the thermal reading can be captured. Below is a snippet of the test result.

Get Started

The EDTDT 2.0 CLI tool is installed as part of the Edge Insights for Vision package that uses OpenVINO™ 2021.2 and is an upgrade from EDTDT 1.0.  

EDTDT 2.0 Download and Install 

1. UX Landing Page selection and EDTDT Package download:

As shown in the below screenshot, the latest EDTDT Download version is 2.0 which supports 2 flavors of Target System OS, namely: 

  • Ubuntu 18.04 LTS  
  • Ubuntu 20.04 LTS

2. View the edgesoftware_configuration.xml included in the edge software zip package and look for the EDTDT ingredient and test modules for Intel® Vision Accelerator Test Suite and Vision package.

NOTE: Since Intel® VATS is not supported on Ubuntu 20.04, VATS_Test_Module will not be listed in the xml for Ubuntu 20.04 package. 

3. Execute the command below to download and install the EDTDT 2.0 and test modules (Vision and Intel® Vision Accelerator Test Suite).

NOTE: Intel® VATS is available only on Ubuntu 18.04.

./edgesoftware install 

4. Once the installation is completed, you should see a status similar to the screen below.

NOTE: Make sure all the modules installed successfully before you start using the EDTDT.

5. Once the installation is completed, reboot the system.

The EDTDT CLI and the test modules are already available in the target system. To go to the directory for the EDTDT:

1. Find the Module ID for the EDTDT by running the command:

./edgesoftware list --default  

2. Edit the change directory command below before running it:

  • EDTDT 2.0 is the downloaded version for Edge Insights for Vision. It can be found in the readme file.
cd $HOME/edge_device_test_and_diagnostic_tool/Edge_Device_Test_and_Diagnostic_Tool_2.0/edtdt

Run the Application

1. Change directory to the location of EDTDT folder by using the command below:

cd <EDTDT Location> 

2. Check the edtdt version by using the below command: 

./edtdt --version 

 Ensure the version is 2.0.

3. Run EDTDT Help:

./edtdt --help 

Usage: EDTDT [OPTIONS] COMMAND [ARGS]... 

  A CLI wrapper of Intel® Edge Device Test and Diagnostic Tool. 

Options: 

  -v, --version  Show the version and exit. 

  --help         Show this message and exit.  

Commands: 

  systeminfo  Display the system information 

  run         Start the test module execution including systemInfo 

  genreport   Generate the complete report

4. Run EDTDT System Info gives a snapshot of the system OS version, kernel, and hardware info such as device manufacturer, processor architecture and type, memory size and hard disk drive configurations. It also gives information about HW accelerators if any. 

./edtdt systeminfo

5. Run EDTDT Test execution: This below command runs the test modules namely the Vision Test Module and Intel® VATS relevant for that vision package. 

./edtdt run 

NOTE: If the tool asks for super user permission, please provide the sudo password when prompted.

6. Run EDTDT Test execution and generate the report:

./edtdt run –r 

OR 

./edtdt run --report 

7. Once the tests are completed, the intermediate artifacts are available at edtdtReport folder.

8. Run EDTDT Report Generation:

./edtdt genreport  

9. This will generate the final report and prepares the zip package with all the required artifacts. Zip Package Example:edtdtReport_2020-07-28_00:29:19.zip.

10. The final report should look like the following example:

 

11. Navigate to the artifacts location by clicking to the respective test module name as shown below.

12. The test artifacts locations are as follows:

<ZIP File location>/EDTDT/edtdtReport/Vision_Test_Module/ 

<ZIP File location>/EDTDT/edtdtReport/VATS_Test_Module/ 

Release Notes

Version: 2.0 

New in this Release 

  • EDTDT 2.0 has incorporated feedback from customer from EDTDT 1.0 release.  
    • EDTDT skips the Intel® VATS execution if Intel® Vision Accelerator card is NOT present. 
    • EDTDT report for Intel® VATS provides host telemetry data such as host average CPU utilization, maximum CPU utilization, average memory utilization and maximum memory utilization in addition to pass/fail status. 
    • EDTDT README.txt is updated to capture the version information of the python modules being used. 
  • EDTDT 2.0 supports Ubuntu 20.04 in addition to Ubuntu 18.04 for Edge Insights for Vision. 
  • EDTDT 2.0 integrates Intel® Vision Accelerator Test Suite and Vision test module with OpenVINO™ 2021.2 support.  
  • EDTDT 2.0 Vision Test Module with Ubuntu 20.04 is tested on 11th generation Intel® Core™ processors with kernel 5.4. 

Known issues 

  • Intel® Vision Accelerator Test Suite currently runs only on Ubuntu 18.04.  
  • Intel® VATS will not provide backward compatibility with older OpenVINO™ versions. 
  • Report generation in PDF has been deprecated in EDTDT 2.0 due to known issues.  
  • On EDTDT 1.0, there is an issue with VATS tests failing for OpenVINO™ 2020.4 due to the following models being removed from the OpenVINO™ 2020.4 release. 
    • deeplabv3 
    • googlenet-v3 
    • resnet-101 
    • resnet-50  
  • There is no fix for the issue seen in EDTDT 1.0. The recommendation is to move to EDTDT 2.0 as the Intel® VATS is updated to remove these models and to support OpenVINO™ 2021.2. 

Version: 1.0 

New in this Release 

  • Initial Features for Recommended Configuration. 
  • Installs all the prerequisites and dependencies for the Edge Device Test and Diagnostic Tool. 
  • Supports below use cases: 
    • Run – launches the Vision test module and Vision Accelerator test module. If there are no Vision accelerator card present in the system, the HDDL test cases will be failed. 
    • System info – collects the system information where the test is run. 
    • genreport – Generates a summary test report in html and pdf for all the test modules that were executed.  

Known Issues 

  • The pdf type of edtdt report is not showing the color format for Status.   

Summary and Next Steps

With the EDTDT CLI tool, you were able to run an Intel provided test suite at the target system, with the goal of testing your platform for a specific ESH package. EDTDT generated a complete test report in html format, along with detailed logs packaged as one zip file. 

As a next step, customers can submit the final result package (Example: edtdtReport_2020-07-28_00:29:19.zip) by email to the eshsupport@intel.com.

Learn More

To learn more about the Edge Insights for Vision test suite and Intel® Vision Accelerator Test Suite, please refer to the below links. 

Troubleshooting

EDTDT  

If EDTDT binary is run on incompatible python version, the functionality might be affected. 

  • Solution: In the edgesoftware_configuration.xml file, verify the edtdt version. 
    • If version is 2.0.18, then ensure Ubuntu 18.04 with python 3.6 is being used in the system. 
    • If version is 2.0, then ensure Ubuntu 20.04 with python 3.8 is being used in the systemuser invokes the command edtdt genreport.  

Error Reported : Exiting EDTDT Execution !! Make sure to run the tests using 'edtdt run' before generating the report. 

  • Solution : perform edtdt run or edtdt run -r 

If one or more modules in the vision package are not downloaded in the target system, EDTDT can still be launched but it will likely to fail the test cases that are dependent on that module. 

Intel Vision Test Module 

Intel® Vision Accelerator Test Suite Prerequisite  

Please ensure that the OpenVINO™ package is being installed in the machine. The OpenVINO™ version used as part of this EDTDT package is 2021.2.0-1877. It can also be downloaded from here.  

Dependencies: 

OpenVINO™ toolkit: This assumes all required OpenVINO™ toolkit and Intel® Vision Accelerator Test Suite/VPU drivers and dependencies are installed by installing the Edge Insights for Vision package from the Edge Software Hub. If any of the modules are missing on the platform, follow the guides below to ensure the needed dependencies are installed. 

Run the edtdt command as mentioned in the section EDTDT 2.0 Download and Install to install all the dependencies and initiate the test execution. The approximate time for the Intel® VATS test to complete is 10 minutes. 

Below are the troubleshooting steps if encounter with the following error in Intel® VATS. 

Running inference is failed with the following error generated: 

E: [ncAPI] [ 965618] [MainThread] ncDeviceOpen:677 Failed to find a device, rc: X_LINK_ERROR

 Possible solution 

  1. Reboot the host system and re-run. 
  2. Kill the HDDL Plugin backend service (hddldaemon) and reset all Intel® Movidius™ VPUs before running an application. 
kill -9 $(pidof hddldaemon autoboot) 
pidof hddldaemon autoboot # Make sure none of them is alive 
source /opt/intel/openvino/bin/setupvars.sh 
${HDDL_INSTALL_DIR}/bin/bsl_reset 

For more information, please refer to this site.      

Support Forum

If you're unable to resolve your issues, contact the Support Forum

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.