Edge Device Test and Diagnostic Tool (EDTDT)

Overview

Edge Device Test and Diagnostic Tool (EDTDT) Command Line Interface (CLI) tool provides customers with the capability to run an Intel provided test suite at the target system, with the goal of enabling partners to test their platform for a specific Intel® Edge Software Hub (ESH) package.

EDTDT and the software listed in table 1 are installed when selected as part of the Edge Insights for Vision package. If you have not installed the package yet, you can download it from EDTDT for Vision and follow the installation instructions.

Table 1
Time to Complete Approximately 1 hour (Vision + Intel® Vision Accelerator Test Suite test execution)
Programming Language Python* 3.6
Software EDTDT CLI tool binary – compiled python package

Target System Requirements

  • One of the following processors: 
    • 6th, 7th, or 8th generation Intel® Core™ processor. 
    • 6th, 7th, or 8th generation Intel® Xeon® processor. 
    • Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics. 
  • At least 16 GB RAM. 
  • At least 64 GB hard drive. 
  • An Internet connection. 
  • Ubuntu* 18.04.3 LTS. Kernel 5.0 

How It Works

EDTDT package contains the CLI, python library, and Test Module Interfaces. The CLI tool launches various test modules corresponding to the package. The test modules in scope are Vision test and Intel® Vision Accelerator Test Suite. The test module/test suites are dependencies for the EDTDT CLI execution and are downloaded by the Edge Software Hub CLI onto the target system.

EDTDT package and its test module dependencies will be enlisted in the edgesoftware-configuration.xml when the customer selects the option. ESH CLI will use the same mechanism to download and install the EDTDT components and test suite onto the target system with the other package components.

The test modules interact with the EDTDT CLI through a common test module interface layer (TMI). EDTDT generates a complete test report in html and pdf formats, along with detailed logs packaged as one zip file, which the customer can manually choose to email to the ESH support team.

 

Figure 1: Architecture Diagram

 

Vision Test Module Overview

Edge Insights for Vision Test Module is a validation framework for Intel® Media SDK, Intel® Container OpenVINOTM and Intel® Runtime OpenVINOTM.

The Intel products mentioned above are intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to have customized derivative builds of Intel’s reference designs, and their performance number may vary from that of Intel’s designs. This calls for a base framework that allows partners to compare their design’s performance and generate a hardware health status report.

Edge Insights for Vision test suite is a set of test scripts to validate the Edge Insights for Vision package. The details of the Edge Insights for Vision package can be found here.

The test suite serves the following purpose:

  • Health Check: It runs a health check to make sure all the components are properly installed and configured at a component level.

  • Functionally tests at component level: It runs a series of sample applications and validates the output of the sample applications to make sure that all the functionality of the components works as expected.

  • Functionally test across components: It runs a few sample applications that have functionality to verify across components.

Intel® Vision Accelerator (aka HDDL) Test Suite Overview

Intel® Vision Accelerator (aka HDDL) Test Suite is an OpenVINOTM based extensible validation framework for Intel® Vision Accelerator solutions. Intel® Vision Accelerator Test Suite is a scalable and portable automated test framework for Intel® Vision Accelerator products.

Intel® Vision Accelerator Design products are reference designs intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to build custom derivatives of Intel's reference designs, whose performance number may deviate from that of Intel's designs. This calls for a base framework that allows partners to compare their designs' performance and stability numbers to those advertised by Intel.

HDDL Test Suite has categorized test cases into four categories: Usability, Performance, Reliability and Stability Test.

Intel® Vision Accelerator Test Suite Dependencies:

OpenVINOTM Toolkit: This assumes all required OpenVINOTM toolkit and Intel® Vision Accelerator Test Suite/VPU drivers and dependencies are installed by installing the Edge Insights for Vision package from the Edge Software Hub. If any of the modules are missing on the platform, follow the guides below to ensure the needed dependencies are installed.

Download apt packages for Intel® Vision Accelerator Test Suite:

```sh sudo apt-get install -y usbutils libboost-filesystem1.58.0 libboost-thread1.58.0 libjson-c2 libboost-program-options-dev ```

 

EDTDT downloads the Intel® Vision Accelerator Test Suite; installs all test suite dependencies; downloads and converts all required models; and builds the required sample applications and required public models using Model Downloader from OpenVINOTM toolkit. It converts the downloaded models to needed format, runs the test, and prints the logs.

The 13 models required for the test scripts are as follows:

  • AlexNet
  • DeepLab-v3
  • DenseNet-121
  • GoogleNet-v1
  • GoogleNet-v3
  • MobileNet-SSD
  • MobileNet-v1-1.0-224
  • MobileNet-v2
  • ResNet-101
  • ResNet-50
  • Se-resnext-50
  • SqueezeNet1.1
  • SSD300

By default, the Test Suite provides eight test cases categorized into different configuration to test different functionalities of Intel® Vision Accelerator Test Suite.

The test cases are:

  • usability_test
    • firmware_loading_time
  • bootup_speed_test
    • reset_1
    • hardware_ping
  • io_ping_test
  • stability_test_config
  • stability_test_1
  • reliability_test_config
  • thermal_test
    • reliability_72hour
  • performance_test_config
    • performance_alexnet
    • performance_densenet-121
    • performance_googlenet-v1
    • performance_googlenet-v4
    • performance_inception-resnet-v2
    • performance_mobilenet-ssd
    • performance_mobilenet-v1-1.0-224
    • performance_mobilenet-v2
    • performance_resnet-101
    • performance_resnet-50
    • performance_se-resnext-50
    • performance_squeezenet1.1
    • performance_ssd300

Get Started

The EDTDT CLI tool is installed as part of the Edge Insights for Vision package.

EDTDT Download and Install

1. View the edgesoftware_configuration.xml included in the edge software zip package and look for the EDTDT ingredient and test modules for Intel® Vision Accelerator Test Suite and Vision package.

 

2. Execute the command below to download and install the EDTDT and test modules (Vision and Intel® Vision Accelerator Test Suite).

./edgesoftware install 

 

3. Once the installation is completed, you should see a status similar to the screen below.

 

NOTE: Make sure all the modules installed successfully before you start using the EDTDT.

4. Once the installation is completed, reboot the system.

The EDTDT CLI and the test modules are already available in the target system. To go to the directory for the EDTDT:

1. Find the Module ID for the EDTDT by running the command:

./edgesoftware list --default  

2. Edit the change directory command below before running it:

  • 1.0 is the EDTDT downloaded version for Edge Insights for Vision. It can be found in the readme file.
cd $HOME/edge_device_test_and_diagnostic_tool/Edge_Device_Test_and_Diagnostic_Tool_1.0/EDTDT 

Run the Application

1. Change directory to the location of EDTDT folder by using the command below:

cd <EDTDT Location> 

2. Run EDTDT Help:

./edtdt --help 

Usage: edtdt [OPTIONS] COMMAND [ARGS]...  

  
A CLI wrapper of Intel® Edge Qualification Tool. 

Options: 

  --help  Show this message and exit. 

Commands: 

  genreport   Generate the complete report 

  run         Start the test module execution including systemInfo 

  systeminfo  Collect the system information 

3. Run EDTDT Test execution:

./edtdt run 

 

NOTE: If the tool asks for super user permission, please provide the sudo password when prompted.

4. Run EDTDT Test execution and generate the report:

./edtdt run –r 

OR 

./edtdt run --report 

 

NOTE: If the tool asks for super user permission, please provide the sudo password when prompted.

 

5. Once the tests are completed, the intermediate artifacts are available at edtdtReport folder.

6. Run EDTDT System Info:

./edtdt systeminfo 

7. Run EDTDT Report Generation:

./edtdt genreport  

8. This will generate the final report and prepares the zip package with all the required artifacts. Zip Package Example: edtdtReport_2020-07-28_00:29:19.zip

9. The final report should look like the following example:

10. Navigate to the artifacts location by clicking to the respective test module name as shown below.

10. The test artifacts locations are as follows:

<ZIP File location>/EDTDT/edtdtReport/Vision_Test_Module/ 

<ZIP File location>/EDTDT/edtdtReport/HDDL_Test_Module/ 

Release Notes

Version: 1.0

New in this Release

  • Initial Features for Recommended Configuration.

  • Installs all the prerequisites and dependencies for the Edge Device Test and Diagnostic Tool.

  • Supports use cases below:

    • Run – launches the Vision test module and Intel® Vision Accelerator test module. If there is no Vision accelerator card present in the system, the HDDL test cases will be failed.

    • System info – collects the system information where the test is run.

    • Genreport – Generates a summary test report in html and pdf for all the test modules that were executed.

Known Issues

The pdf EDTDT report is not showing the color format for Status.

Summary and Next Steps

With the EDTDT CLI tool, you were able to run an Intel provided test suite at the target system, with the goal of testing your platform for a specific ESH package. EDTDT generated a complete test report in html and pdf formats, along with detailed logs packaged as one zip file.

As a next step, customers can submit the final result package (Example: edtdtReport_2020-07-28_00:29:19.zip) by email to the ESH Support team.

Learn More

To learn more about the Edge Insights for Vision test suite and Intel® Vision Accelerator Test Suite, please refer to the links below.

 

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804