Release Notes for Intel® Distribution of OpenVINO™ Toolkit 2020

By Andrey Zaytsev, Andrey Zaytsev, Published: 02/05/2020, Last Updated: 04/11/2020

Note For the Release Notes for the 2019 version, refer to Release Notes for Intel® Distribution of OpenVINO™ toolkit 2019

Introduction

The Intel® Distribution of OpenVINO™ toolkit is a comprehensive toolkit for quickly developing applications and solutions that emulate human vision. Based on Convolutional Neural Networks (CNNs), the toolkit extends CV workloads across Intel® hardware, maximizing performance. It accelerates applications with high-performance, AI and deep learning inference deployed from edge to cloud.

The Intel Distribution of OpenVINO toolkit:

  • Enables deep learning inference from the edge to cloud.
  • Supports heterogeneous execution across Intel accelerators, using a common API for the Intel® CPU, Intel® Integrated graphics, Intel® Gaussian & Neural Accelerator (Intel® GNA), Intel® Movidius™ Neural Compute Stick (NCS), Intel® Neural Compute Stick 2 (Intel® NCS2), Intel® Vision Accelerator Design with Intel® Movidius™ Vision Processing Unit (VPU), Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA Speed Grade 2.
  • Speeds time-to-market through an easy-to-use library of CV functions and pre-optimized kernels.
  • Includes optimized calls for CV standards, including OpenCV and OpenCL™.

New and Changed in the Release 3 LTS

Executive Summary

New and Changed in the Release 2

Executive Summary

  • Introduces the new Deep Learning (DL) Streamer component, which uses the GStreamer* multimedia framework. This component enables the creation and deployment of analytic pipelines across Intel architecture from edge to cloud.
  • Includes bug fixes and stability fixes in memory leak issues in the Intel® Threading Building Blocks (Intel® TBB) library.
  • Improves support for noise suppression use cases by adding new layers and bug fixes to the Intel Gaussian & Neural Accelerator (Intel GNA) plug-in.
  • Adds support for new FP32 and INT8 models to enable more vision and text use cases:
    • Original models: 3D U-Net, MobileFace
    • Low precision (that is, INT8) models: EAST, OpenPose, RetinaNet, and FaceNet
  • Introduces new pretrained models on Open Model Zoo to reduce time to production, which include:
    • Multi-camera tracking for person reidentification
    • Cascade R-CNN for person detection
  • Published new performance benchmarks.

Backward Incompatible Changes Compared with v.2020 Release 1

  • List of Deprecated APIAPI Changes
  • Part of the Inference Engine shared library has been moved to the Inference Engine Legacy shared library, which contains obsolete routines needed for Intermediate Representation (IR) v7. You should link custom applications against both libraries, which is done automatically if you use the CMake* interface. 

Model Optimizer

Model Optimizer

Common Changes

  • Implemented generation of a compressed OpenVINO IR suitable for INT8 inference, which takes up to 4 times less disk space than an expanded one. Use the --disable_weights_compression Model Optimizer command-line parameter to get an expanded version.
  • Implemented an optimization transformation to replace a sub-graph with the Erf operation into the GeLU operation.
  • Implemented an optimization transformation to replace a sequence of Split-Concat operations to the Interpolate operation.
  • Fixed a number of Model Optimizer bugs to generate reshape-able IRs of many models with the command-line parameter --keep_shape_ops.
  • Fixed a number of Model Optimizer transformations to set operations name in an IR equal to the original framework model operation name.

ONNX*

  • Added support for the MeanVarianceNormalization operation if normalization is performed over spatial dimensions.

TensorFlow*

  • Added support for the TensorFlow* Object Detection models version 1.15.X.
  • Added support for the following operations:
    • BatchToSpaceND
    • SpaceToBatchND
    • Floor

MXNet*

  • Added support for the Reshape operation with input shape values equal to -2, -3, and -4.

Inference Engine

Inference Engine

Common Changes

  • Refactored the Inference Engine Extension API. The new API allows to create Network with custom nGraph operations and load custom CPU execution kernels for custom nGraph operations.
  • Added the Inference Engine Transformation Library, which contains nGraph transformation for the conversion of opset2 operations to opset1, and opset1 operations to the legacy operation set.
  • Added the Inference Engine LP transformations Library, which contains low precision transformation routines. The library is linked directly to plugins and not a dependency of the Inference Engine shared library.
  • Improved the support of AVX2 and AVX512 instruction sets in the CPU preprocessing module.

Inference Engine C API

Inference Engine C API

  • Added support of I420, NV12 blobs and added the dedicated hello_nv12_classifiction_c sample.

CPU Plugin

CPU Plugin

  • Removed the mkltiny dependency. This allowed to significantly decrease the binary distribution size required for inference on CPU.
  • Added several new operations:
  • GeLU
  • Swish
  • BatchToSpace
  • SpaceToBatch
  • Continued optimizations for the INT8 pipeline that resulted in 1.2x geomean speedup compared with the 2020.1 release on about 140 topologies.
  • Significantly improved load time for INT8 models.

GPU Plugin

GPU Plugin

  • Added support for the following new operations:
    • Swish
    • VariadicSplit
    • Transpose
    • SpaceToDepth
    • ConvertLike
    • Select
  • Added 3D-spatial dimensions support for the Resample primitive.
  • Fixed all known memory leaks.
  • Implemented performance optimizations for grouped convolutions, topk primitive, and models with batch size less than 1.
  • Implemented minor load time improvements for models with huge layers count or weights size.

MYRIAD Plugin

MYRIAD Plugin

  • Aligned VPU firmware with Intel® Movidius™ Myriad™ X Development Kit (MDK) R11 version.
  • Added support for 3D convolutions with restriction on grouped convolution size to 1.
  • Added support for 3D pooling.

HDDL Plugin

HDDL Plugin

  • Linux* kernel version 5.3 (and below) are supported now.
  • Due to the compatibility issue, the ION driver cannot be installed on Ubuntu* with a kernel version higher than 5.0 (included) and CentOS*, fall back to use shared memory.

GNA Plugin

GNA Plugin

  • Added support for the following layers:
    • Abs
    • Exp
    • Log
    • Sign
    • Softsign
    • Unsqueeze
  • Fixed problems with some layer combinations.
  • Fixed export and import of some models.
  • Fixed a problem with configuration key validation.

FPGA Plugin

FPGA Plugin

  • Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Mustang-F100-A10) speed grade 1 bitstreams and Intel® Programmable Acceleration Card (Intel® PAC) with Intel® Arria® 10 GX FPGA are not included in the Intel Distribution of OpenVINO toolkit 2020.2 release.
  • If you use Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA, we recommend continuing to use the Intel Distribution of OpenVINO toolkit 2020.1 release.

nGraph

  • Introduced opset2. This opset contains the following operations that were not included to opset1:
    • BatchToSpace
    • GELU
    • MVN
    • ReorgYolo
    • ROIPooling
    • SpaceToBatch
  • Removed the nGraph Python* API.
  • Enabled building nGraph ONNX Importer low-level API as a part of the nGraph shared library. This is a preview feature, which enables creating the nGraph Function directly from an ONNX model in runtime without running the Model Optimizer.

Post-Training Optimization Tool

Post-Training Optimization Tool

  • Additional models supported:
    • Added 25% more quantized and executed models (145 vs 116, Open Model Zoo scope), all within 1% of accuracy drop.
    • Enabled RetinaNet, Open Pose, Open Model Zoo/Human-pose (Single, 3d-pose), Mobile Face Det, and EAST.
    • Enabled the most of new Open Model Zoo models except the pipelined ones and asl-recognition-0004
  • Usability Improvements:
    • Implemented the annotation-free Default quantization (Simplified mode):
      • The mode allows quantizing a CNN without the Accuracy Checker support.
      • The mode accepts a set of images (unannotated dataset), performs default preprocessing and runs the Default quantization. 
      • The mode is NOT applicable for complex models, which work with many inputs, specific pre- and postprocessing, dataset different from a set of images.
    • Added the Quantization recipe (quantization_parameters section) to IR (*.xml) to show quantization parameters.
  • Implemented the new Bias Correction algorithm to improve the accuracy of quantized models.
    • The algorithm is slower but more precise than the previous version of Bias Correction. For example, it allows to get Mobilenet-v2/CF, mobilenet-v1-0.25/TF, image-retrieval-0001, text-detection-0004, text-recognition-0012 , person-reidentification-retail-0248,  and other models fully quantized with accuracy drop below 1%.
    • Previous Bias Correction algorithm is now called FastBiasCorrection. To activate the new one, set the quantization parameter "use_fast_bias": false.
  • Post-Training Optimization tool generates a compressed Intel Distribution of OpenVINO toolkit intermediate representation suitable for INT8 inference, which takes up to 4 times less disk space than an FP32 one. Use --keep_uncompressed_weights to get an expanded version.

Deep Learning Workbench

Deep Learning Workbench

  • Extended support for the INT8 Post-Training Optimization Tool. Now pretrained models can be calibrated to the INT8 precision with an annotation-free dataset (directory with images), which enables you to calibrate to lower precision without the need for a dataset - inputting images is sufficient. Only the Default (Maximum Performance) algorithm currently supports this capability.
  • Integrated the Deployment Manager, which allows users to get the most minimal package size for deployment after experimentations and fine-tuning using the DL Workbench
  • Added support for the Intel Vision Accelerator Designs target (on Linux only).
  • Separated calibration and validation datasets, which enables you to use different datasets for calibration and accuracy/performance measurements.
  • Implemented multiple usability and experience improvements, including a new look and feel for the comparison mode.
  • Enabled a new authentication mechanism using tokens, which provides additional security protection for users' models and datasets.

OpenCV*

  • Version changed to 4.3.0.
  • nGraph API is used by default in Inference Engine backend API in the dnn module. The NN Builder API is deprecated, but have not been removed yet. You can enable it with the cv::dnn::setInferenceEngineBackendType function.

Examples and Tutorials

  • Added sample to demonstrate nGraph C++ API. The sample demonstrates how to execute an inference using ngraph::Function to create a network. The sample uses the LeNet classifications network as an example.

Open Model Zoo

  • Extended the Open Model Zoo, which includes additional CNN-pretrained models and pregenerated Intermediate Representations (.xml + .bin):
    • [NEW] face-detection-0100
    • [NEW] face-detection-0102
    • [NEW] face-detection-0104
    • [NEW] face-detection-0105
    • [NEW] face-detection-0106
    • [NEW] unet-camvid-onnx-0001
    • [NEW] yolo-v2-ava-0001
    • [NEW] yolo-v2-ava-sparse-35-0001
    • [NEW] yolo-v2-ava-sparse-70-0001
    • [NEW] yolo-v2-tiny-ava-0001
    • [NEW] yolo-v2-tiny-ava-sparse-30-0001
    • [NEW] yolo-v2-tiny-ava-sparse-60-0001
    • [NEW] instance-segmentation-security-1025
    • asl-recognition-0004 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0248 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0249 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0300 - Replaces the 2020 Release 1 model.
    • text-spotting-0002-detector - Replaces the 2020 Release 1 model.
    • text-spotting-0002-recognizer-decoding - Replaces the 2020 Release 1 model.
    • text-spotting-0002-recognizer-encoder - Replaces the 2020 Release 1 model.
    • faster-rcnn-resnet101-coco-sparse-60-0001 - Replaces the 2020 Release 1 model.
    • handwritten-japanese-recognition-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-sparse-30-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-sparse-60-0001 - Replaces the 2020 Release 1 model.
  • Added new demo applications:
    • Colorization
    • Image Inpainting
    • MonoDepth
  • Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow*, MXNet*, and PyTorch* formats:
Model Name Framework
colorization-v2 Caffe*
colorization-v2-norebal Caffe
googlenet-v1-tf TensorFlow
googlenet-v4-tf TensorFlow
gmcnn-places2-tf TensorFlow
i3d-rgb-tf TensorFlow
midasnet PyTorch*
resnet-50-tf TensorFlow
resnet-18-pytorch PyTorch
rfcn-resnet101-coco-tf TensorFlow
yolo-v1-tiny-tf TensorFlow
yolo-v2-tiny-tf TensorFlow
yolo-v2-tf TensorFlow
yolo-v3-tf TensorFlow

Deep Learning Streamer

Introducing the Deep Learning (DL) Streamer, now part of the default installation package of the Intel Distribution of OpenVINO toolkit. DL Streamer enables developers to create and deploy complex media analytics pipelines across Intel® architectures from edge to cloud. It provides optimal pipeline interoperability with familiar developer experience of the GStreamer multimedia framework. While enabling the developers to build a flexible streaming analytics pipeline, it also offers optimized inference across Intel architectures, from CPU and iGPU to Intel® Movidius™ Vision Processing Unit (VPU).

Key GStreamer elements provided by the DL Streamer:

Element Purpose
gvawatermark To overlay the metadata on the video frame to visualize the inference results.
gvatrack To perform object tracking using zero-term or short-term tracking algorithms. Zero-term tracking assigns unique object IDs and requires object detection to run on every frame. Short-term tracking allows to track objects between frames, thereby reducing the need to run object detection on each frame.
gvapython To provide a callback to execute user-defined Python functions on every frame. Can be used for metadata conversion, inference post-processing, and other tasks.
gvametapublish To publish the JSON metadata to MQTT or Kafka message brokers or files.
gvametaconvert To convert the metadata structure to the JSON format.
gvainference To run deep learning inference using any model with an RGB or BGR input.
gvafpscounter To measure frames per second across multiple streams in a single process.
gvadetect To perform object detection using YoloV2-tiny, YoloV3, and SSD-like models. Outputs the region of interest (ROI) for detected objects.
gvaclassify To perform object classification. Accepts the ROI as input and outputs classification results with the ROI metadata.


For more information, please refer to the DL Streamer tutorial, API reference, and samples documentation. at Intel Distribution of OpenVINO toolkit Samples. DL Streamer is an open source project located at GStreamer Video Analytics (GVA) Plugin repository on GitHub .DL Streamer includes Python and C++ APIs that provide access to inference results produced by the GStreamer elements for user applications written in Python and C++.  The sample code for both C++ and Python is available in the samples folder installed by the Intel Distribution of OpenVINO toolkit installer.

New and Changed in the Release 1

Executive Summary

  • Intel Distribution of OpenVINO Toolkit 2020.1 includes security bug fixes. Users should update to the latest version.
  • Post-training Optimization to enable significant acceleration improvement. Little or no degradation in accuracy using a model quantization; while streamlining the development process with no model re-training nor fine-tuning required. 
  • Extends support for custom layers on the Intel® Movidius™ Vision Processing Unit (VPU). This enables greater flexibility and customization for dedicated edge, media and vision inference.  
  • Low-precision runtime for INT8.  Including native support for “fake quantization” and support for 3D convolution layers along with reduced memory footprint. Enabling developers to add more deep learning models to their applications. 
  • Maximized performance optimizations and cross-platform flexibility with the Deep Learning Workbench.  Simplify the development and deployment workflow, with support for the Post-training Optimization tool, COCO dataset, new algorithms and improvements in developer experience and interfaces. For more information, watch the webinar on Deep Learning Workbench
  • Support for new demos and models on Open Model Zoo, which includes: 
    • American Sign Language (ASL) recognition 
    • Character and text detection with Mask R-CNN 
    • 3D human pose estimation
  • New speech libraries and speech recognition demos are included into distributions for Windows* and Ubuntu* 18.04 OSes. The new components enable and demonstrate end-to-end (speech to text) automatic speech recognition scenario.
  • OpenVX* was removed from distribution, please see a separate Intel® Distribution of OpenVX* Implementation page.
  • Switched to Linux Ubuntu* 18.04 kernel 5.3 by default due to performance drop on 9th Generation Intel® Core™ processors (formerly Coffee Lake) integrated graphics with kernel 5.0 compared to 5.3 and 4.15 (latency mode).
  • Added support for AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 is on Windows.
  • Please, review this documentation page to Get a Deep Learning Model Performance Boost with Intel® Platforms.

Backward Incompatible Changes Compared with v.2019 R3.1

  • Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples. Old path to the C++ samples was changed from <install_root>\deployment_tools\inference_engine\samples\ to <install_root>\deployment_tools\inference_engine\samples\cpp\.
  • Old Calibration tool 1.0 that was introduced with 2019 R1 was removed from the package and replaced by a new one.
  • CPU extensions library was removed. Extensions were moved into the plugin.
  • List of Deprecated API

Model Optimizer

Common Changes

  • Model Optimizer generates IR version 10 by default (except for the Kaldi* framework for which IR version 7 is generated) with significantly changed operations semantic. The command line --generate_deprecated_IR_V7 could be used to generate older version of IR. Refer to the documentation for the specification of a new operations set.
  • --tensorflow_use_custom_operations_config has been renamed to --transformations_config. The old command line parameter is deprecated and will be removed in the future releases.
  • Added ability to specify input data type using the --input command line parameter. For example, --input placeholder{i32}[1 300 300 3]. Refer to the documentation for more examples.
  • Added support for FusedBatchNorm operation when mode=’training’
  • Added optimisation transformation to remove useless Concat+Split sub-graphs.
  • A number of graph transformations were moved from the Model Optimizer to the Inference Engine.
  • Fixed networkX 2.4+ compatibility issues.

ONNX*

  • Added support for the following ONNX* operations:
    • Abs
    • Acos
    • Asin
    • Atan
    • Cast
    • Ceil
    • Cos
    • Cosh
    • Div
    • Erf
    • Floor
    • HardSigmoid
    • Log
    • NonMaxSuppression
    • OneHot
    • ReduceMax
    • ReduceProd
    • Resize
    • Sign
    • Sin
    • Sqrt
    • Tan
    • Xor
  • Added support for the following ONNX models:

TensorFlow*

  • Added optimization transformation which detects Mean Value Normalization pattern for 5D input and replaces it with a single MVN layer.
  • Added ability to read TF 1.X models when TF 2.x is installed.
  • Changed command line to convert GNMT model. Refer to the GNMT model conversion article for more information.
  • Deprecated --tensorflow_subgraph_patterns, --tensorflow_operation_patterns command line parameters. The TensorFlow offload feature will be removed from the future releases.
  • Added support for the following TensorFlow models:
  • Added support for the following TensorFlow operations:
    • Bucketize (CPU only)
    • Cast
    • Cos
    • Cosh
    • ExperimentalSparseWeightedSum (CPU only)
    • Log1p
    • NonMaxSuppressionV3, NonMaxSuppressionV4, NonMaxSuppressionV5
    • Sin
    • Sinh
    • SparseReshape (removed when input and output shapes are equal)
    • SparseToDense (CPU only)
    • Tan
    • Tanh

MXNet*

  • Added support for the following MXNet topologies:
  • Added support for the following MXNet operations:
    • UpSampling with bilinear mode
    • Where
    • _arange
    • _contrib_AdaptiveAvgPooling2D
    • div_scalar
    • elementwise_sub
    • exp
    • expand_dims
    • greater_scalar
    • minus_scalar
    • repeat
    • slice
    • slice_like
    • tile

Kaldi*

  • The --remove_output_softmax command line parameter now triggers removal of final LogSoftmax layer in addition to a pure Softmax layer.
  • Added support for the following Kaldi operations:
    • linearcomponent
    • logsoftmax
  • Added support for the following Kaldi models:

Inference Engine

Common Changes

  • Added C API bridge library that allows you to integrate the Inference Engine runtime and plugins into applications not using and depending explicitly on C++.
  • Added support of I420 blobs. Enable preprocessing and use class I420Blob to pass data for inference in I420 format. Inference Engine will convert data automatically to the BGR before inference.
  • Added Remote Blob allowing to address memory on remote hardware to build pipelines not depending on host memory
  • Enabled multi-socket support for Windows OS
  • Re-worked INT8 run-time (fake quantized based int8 flow) so that it provides higher performance and wider topology coverage:
    • Integrated new graph representation
    • Implemented graph transformations
    • Supported both symmetric and asymmetric quantization schemes
    • Extended INT8 support to cover additional use cases: 3d-models, semantic segmentation, instance segmentation
    • Identical functional coverage (features, models) for both CPU and GPU
    • Implemented optimizations needed to get 1.2x performance improvements over approach previously used

CPU Plugin

  • Re-designed (for CPU) fake quantized based int8 flow, which created also the foundation to scale to other int8 capable platforms.
  • Introduced NUMA* support for the Windows, generally re-designed NUMA awareness  (which now relies on the latest Threading Building Blocks (TBB) NUMA awareness feature). 
    • It is recommended to update all Intel products that use the TBB (like Intel® Distribution for Python*) to avoid possible TBB version conflicts when combining with Intel Distribution of OpenVINO toolkit.

GPU Plugin

  • Implemented remote blob for OpenCL and Direct3D* 11 frameworks and  support of memory been described in their terms on GPU

MYRIAD Plugin

  • Extended support for VPU Custom Layers

HDDL Plugin

  • Extended support for VPU Custom Layers
  • Linux kernel versions 5.2 and below are supported.

GNA Plugin

  • Added support for Tensor Iterator layer
  • Added support for Squeeze layer
  • Extended support for Permute Layer
  • Fixed issue with LSTM Cell layer
  • Improved accuracy for TDNN networks

FPGA Plugin

  • Introduced offline compilation for FPGA, HETERO:FPGA devices using compile_tool. Compile tool produces binaries which can be imported via Core::Import to target device.

Model Optimizer and nGraph Integration Into Inference Engine

  • Intel Distribution of OpenVINO toolkit is now powered by nGraph capabilities for Graph construction API, Graph transformation engine and Reshape, that replace former NN Builder API offering. Please refer to the new tutorials and updates for Model Optimizer and Inference Engine Developer's Guides updates on these new run-time capabilities. The NN Builder API is marked as deprecated and will be removed in the later releases.
  • Additionally, we revised operations set used in internal Intermediate Representations and offer an extended set, which covers both former Intel Distribution of OpenVINO toolkit and nGraph operations as well as adding necessary semantics to further speedup models enabling from supported frameworks. The detailed opset specification and new versioning schema for IR-vs-operations are provided in the documentation.
  • As result, Model Optimizer can now generate two types of IRs: a) IR V10 (default option) with this extended semantics and b) IR V7 (under --generate_deprecated_IR_V7 option in MO) which is just an evolution of IR V6 from previous release.
  • We expect that re-converting the existing (and already supported) models into new semantics (IR V10) should be transparent for end users, if for any reason conversion or inference fail, try generating V7 IR (with option above), if this also does not help please report us about your issue.

Post-training Optimization Tool

The main goal of the tool:

  • Transform Deep Learning model into a representation specific for selected type of optimization (e.g. quantization or sparsity) without model retraining
  • After such transformation, the model can be inferred efficiently on hardware supporting such optimizations (e.g. execution in INT8 precision)
  • The tool is extendable in order to support multiple quantization algorithms:

    Post-training Optimization Tool

The tool supports quantization of Intel Distribution of OpenVINO toolkit Intermediate Representation (IR) models for various types of IA hardware:

  • List of algorithms supported:
    • Default algorithm that is essentially a pipeline running three base algorithms:
      • MinMax
      • Bias Correction (runs on top of naive algorithm, based on minimization of per-channel quantization error)
      • Activation Channel Alignment (applied to align activation ranges)
    • Accuracy Aware algorithm that preserves accuracy of the resulting model (keeps accuracy drop below threshold)
  • Provides hardware-specific configs
  • Features per-channel / per-tensor quantization granularity
  • Supports symmetric / asymmetric quantization through presets mechanism
  • Two default presets are introduced for MinMax, Default and Data free algorithms to simplify user experience
    • 'Performance' preset stands for symmetric channel-wise (both for weights and activations) quantization
    • 'Accuracy' preset stands for symmetric weights + asymmetric activations quantization
    • Per-layer quantization tuning is also possible, some layers can be included into 'ignored scope' to skip quantization for those.

Tool is available at <install_root>\deployment_tools\tools\post_training_optimization_toolkit. After installation and running setup.py, the toolkit is available in the command line by the 'pot' alias.

Deep Learning Workbench

  • Added support for Int8 Post-Training Optimization Tool. Two new algorithms (Max performance and Max accuracy) are currently exposed
  • Added support for COCO dataset
  • Multiple UX/UI improvements including new look and feel for per-layer table view

OpenCV*

  • Version changed to 4.2.0.
  • Added experimental support for nGraph API of Inference Engine (can be selected through cv::dnn::setInferenceEngineBackendType() call).
  • Added Inference and Streaming extensions to G-API: now OpenCV G-API can handle full video analytics pipelines including video capture / decoding, image processing, CV, and DL scoring.
  • New tutorials are available for these features: Face Analytics and Face Beautification pipelines.

Examples and Tutorials

  • CPU extensions library was removed.
  • Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples.
  • Added end-to-end demos demonstrating speech recognition pipeline: offline demo for wave file transcription and GUI demo for live speech recognition.
  • Added samples to demonstrate Inference Engine C-API usage

Tools

  • Added feature extractor and decoder tools for speech processing pipelines. Added a tool for conversion of Kaldi* language models to the format used by the decoder tool.

Open Models Zoo

  • Extended the Open Model Zoo, which includes additional CNN-pretrained models and pre-generated Intermediate Representations (.xml + .bin):
    • [NEW] asl-recognition-0003
    • [NEW] person-detection-asl-0001
    • [NEW] person-reidentification-retail-0200
    • [NEW] resnet18-xnor-binary-onnx-0001
    • [NEW] text-spotting-0001-detector
    • [NEW] text-spotting-0001-recognizer-decoder
    • [NEW] text-spotting-0001-recognizer-encoder
    • [NEW] person-reidentification-retail-0103. Replaces the 2019 R3 model.
    • [NEW] person-reidentification-retail-0107. Replaces the 2019 R3 model.
  • Added new demo applications:
    • 3D human pose estimation
    • interactive face detection
    • multi-camera multi-person tracking
    • multi-channel object detection with YOLO v3 support
    • American sign language (ASL) recognition
    • object detection with CenterNet
    • single human pose estimation
    • text spotting

Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow, MXNet, and PyTorch formats:

Model Name Framework
efficientnet-b0 TensorFlow
efficientnet-b0-pytorch PyTorch
efficientnet-b0_auto_aug TensorFlow
efficientnet-b5 TensorFlow
efficientnet-b5-pytorch PyTorch
efficientnet-b7-pytorch PyTorch
efficientnet-b7_auto_aug TensorFlow
face-recognition-mobilefacenet-arcface MXNet
face-recognition-resnet100-arcface MXNet
face-recognition-resnet34-arcface MXNet
face-recognition-resnet50-arcface MXNet
densenet-121-caffe2 Caffe2
resnet-50-caffe2 Caffe2
squeezenet1.1-caffe2 Caffe2
vgg19-caffe2 Caffe2
ctdet_coco_dlav0_384 ONNX
ctdet_coco_dlav0_512 ONNX
brain-tumor-segmentation-0002 ONNX
human-pose-estimation-3d-0001 PyTorch
single-human-pose-estimation-0001 PyTorch
vehicle-license-plate-detection-barrier-0123 TensorFlow

Preview Features Terminology

A preview feature is functionality that is being introduced to gain early developer feedback. Comments, questions, and suggestions related to preview features are encouraged and should be submitted to the forum at Intel Distribution of OpenVINO Toolkit Forums.

The key properties of a preview feature are:

  •     It is intended to have a high quality implementation
  •     There is no guarantee of future existence or compatibility.

Note A preview feature is subject to change in the future. It may be removed or altered in future releases. Changes to a preview feature do NOT require the a deprecation and deletion process. Using a preview feature in a production code base is discouraged.

Fixed Issues

JIRA ID Description Component
26561 Performance degradation for model 'semantic-segmentation-adas-0001' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26560 Performance degradation for model 'deeplabv3' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26353 Accuracy issues on a dataset for deeplab_v3 model for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26384 Performance degradation for model 'refinedet-resnet-101' Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26700 Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) hangs with Linux kernel 5.3 IE HDDL plugin

Known Issues

JIRA ID Description Component Workaround

#1

A number of issues were not addressed yet, please see "Known issue" section in the Release Notes for Intel Distribution of OpenVINO toolkit v.2019

All

N/A

24811

[CPU-VNNI] Segmentation fault on license-plate-recognition-barrier-0007, efficientnet-b0, person-reidentification-retail-003 models. Benchmark app crashes with segmentation fault at the very end of execution.

IE CPU Plugin

Known workarounds for the issue:

1) Build IE with OMP parallelization,

2) Build IE with JIT GEMM implementations instead of MKL

3) Using the full TBB MKL library instead of prebuilt mkltiny-tbb

24517

Specialize_function adds redundant ops for functions with the Split operation

nGraph Core

N/A

25417

Incorrect OL and OH params for the FQ layer

MO TensorFlow

N/A

23985

Incorrect command-line parameters specified in the documentation to convert the LM1B model

MO TensorFlow

N/A

25657

Memory leaks are possible in the GPU plugin in case of multiple network loading/unloading cycles

IE GPU Plugin

Try to avoid constant network load/unload scenarios when using the GPU plugin

25087

Performance degradations are possible in the GPU plugin on MobileNet-like models

IE GPU Plugin

N/A

25358

Some performance degradations are possible in the GPU plugin on GT3e/GT4e/ICL NUC platforms

IE GPU Plugin

N/A

25020

May produce wrong output data in the case when several topologies are loaded into one CPU plugin. The issue requires several conditions:

  • Use IRv10

  • Several models are loaded into one plugin

IE CPU Plugin

Use different plugin instance per each topology.

25940

The TBB binary mismatch is possible. Depending on the version of a wrong loaded TBB module, it may lead to crashes, slowdown, missing symbols on load step.

All

Make sure that the proper version of TBB is loaded from the OpenVINO package.

Check your PATH/LD_LIBRARY_PATH environment variable. OpenVINO path to TBB should be first.

Check that there are no other TBB libraries in system folders like C:\Windows\System32, which the OS uses for search.

If you use some over modules with TBB, make sure that OpenVINO is loaded first. In case of Python, do "import openvino" module before others.

24709

Retrained TensorFlow Object Detection API RFCN model has significant accuracy degradation. Only the pretrained model produces correct inference results.

All

Use Faster-RCNN models instead of RFCN model if re-training of a model is required.

23705

Inference may hang when running the heterogeneous plugin on GNA with fallback on CPU.

IE GNA Plugin

Do not use async API when using CPU/GNA heterogeneous mode.

26585

Speech sample may display incorrect results of comparison with reference scores for models including LSTM cells

Samples

N/A

24523

Low performance of asl-recognition-xxxx model on CPU

IE CPU Plugin

N/A

23198

Inference of OneHot layer in constant subgraphs may be incorrect. As a result, GNMT topology may produce incorrect output.

All

N/A

26129

TF YOLO-v3 model fails on AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 

IE MyriadX plugin

Use other versions of the YOLO network or USB connected device (Neural Compute Stick 2)

22108

Stopping the app during firmware boot might cause device hang for Intel® Neural Compute Stick 2

IE MyriadX plugin

Do not press Ctrl+C too while the device is being booted.

23738

Core::SetConfig and Core::GetConfig methods work incorrectly with the GNA plugin.

IE GNA plugin

Use config parameter of Core::LoadNetwork method.

29082

OpenVINO has thread-safety issues. LoadNetwork and other ie::Core methods simultaneously called from different threads may lead to error or crashes.

All

Avoid scenarios of parallel initialization.

28747

CPU plugin does not work on Windows system with CPUs less then AVX2 instruction set (Intel Atom® processors)

IE CPU Plugin

Manually rebuild the CPU plugin from sources available in the public repository with CMake feature flags ENABLE_AVX2=OFF and ENABLE_AVX512=OFF.

 

nGraph Python API has been removed from the current release due to its incompetence. It will be added back when it is ready with the desired quality.

IE Python API

Use C++ API

28970

TF faster-rcnn and faster-resnet101 topologies accuracy deviation on MYRIAD

IE MyriadX plugin, IE HDDL plugin

For accurate inference on these topologies either use the other HW (i.e. CPU/GPU), or use previous release of OpenVINO on Neural Compute Stick 2.

29361

Performance degradation for model 'text-recognition-0012' and ctpn

IE MyriadX plugin, IE HDDL plugin

For optimal performance use previous release of OpenVINO on Neural Compute Stick 2.

28683

Changing system time on Linux might break watchdog and interrupt the inference

IE HDDL plugin

Do not change system time during inference run.

25723

TF rfcn_resnet101_coco low accuracy on dataset

IE MyriadX plugin, IE HDDL plugin

For accurate inference on this topology, either use the other HW (i.e. CPU/GPU), or use the previous version of OpenVINO™ with Intel® Neural Compute Stick 2.

Included in this Release

The Intel® Distribution of OpenVINO™ toolkit is available in the following versions:

  • Intel Distribution of OpenVINO toolkit for Windows
  • Intel Distribution of OpenVINO toolkit for Linux
  • Intel Distribution of OpenVINO toolkit for Linux with FPGA Support
  • Intel Distribution of OpenVINO toolkit for macOS*
Component License Location Windows* Linux* Linux for FPGA macOS*

Deep Learning Model Optimizer

Model optimization tool for your trained models

Apache 2.0 <install_root>/deployment_tools/model_optimizer/* YES YES YES YES

Deep Learning Inference Engine

Unified API to integrate the inference with application logic

Inference Engine Headers

EULA

 

 

Apache 2.0

<install_root>/deployment_tools/inference_engine/*

 

 

<install_root>/deployment_tools/inference_engine/include/*

YES YES YES YES

OpenCV library

OpenCV Community version
compiled for Intel hardware

BSD <install_root>/opencv/* YES YES YES YES

Intel® Media SDK libraries
(open source version)

Eases the integration
between the Intes Distribution for OpenVINO toolkit
and the Intel Media SDK.

MIT <install_root>/../mediasdk/* NO YES YES NO

Intel® Graphics Compute Runtime for OpenCL 

Improves usability 

MIT

<install_root>/install_dependencies/

    install_NEO_OCL_driver.sh - helps to install OpenCL Runtime, default location /usr/local/lib/

    intel-opencl_*.deb - driver for Ubuntu*

    intel-opencl_*.rpm - driver for CentOS*

    intel-* - driver's dependencies

NO YES YES NO

Intel® FPGA Deep Learning Acceleration Suite (Intel® FPGA DL Acceleration Suite), including pre-compiled bitstreams

Implementations of the most common CNN topologies to enable image classification and ease the adoption of FPGAs for AI developers.

Includes pre-compiled bitstream samples for the Intel Programmable Acceleration Card with Intel Arria 10 GX FPGA and Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2

Intel OBL FPGA SDK 18.1

<install_root>/bitstreams/a10_dcp_bitstreams/*

<install_root>/bitstreams/a10_vision_design_sg1_bitstreams/*

<install_root>/bitstreams/a10_vision_design_sg2_bitstreams/*

 

NO NO YES NO

Intel® FPGA SDK for OpenCL™ software technology

The Intel FPGA RTE for OpenCL provides utilities, host runtime libraries, drivers, and RTE-specific libraries and files

Intel OBL FPGA SDK 18.1

/opt/altera/aocl-pro-rte/*

<user>/intelFPGA_pro/<version>/*

NO NO YES NO

Intel Distribution for OpenVINO toolkit documentation

Developer guides and other documentation

  Available from the OpenVINO™ toolkit product site,
not part of the installer packages.
NO NO NO NO

Open Model Zoo

Documentation for models; Models in binary form can be downloaded using Model Downloader 

Apache 2.0 <install_root>/deployment_tools/open_model_zoo/* YES YES YES YES

Inference Engine Samples

Samples that illustrate Inference Engine API usage and demos that demonstrate how you can use features of Intel® Distribution of OpenVINO™ toolkit in your application

Apache 2.0 <install_root>/deployment_tools/
inference_engine/samples/*
YES YES YES YES

Deep Learning Workbench

Tool that can help developers to run  Deep Learning models through the toolkit's Model Optimizer, convert it to INT8, fine tune it, run inference and measure accuracy.

EULA <install_root>/deployment_tools/tools/workbench/* YES YES NO YES

nGraph

Open source C++ library, compiler and runtime for Deep Learning nGraph

Apache 2.0 <install_root>/deployment_tools/ngraph/* YES YES YES NO

Post-Training Optimization Toolkit

designed to convert a model into a more hardware-friendly representation by applying specific methods that do not require re-training, for example, post-training quantization.

EULA <install_root>/deployment_tools/
tools/post_training_optimization_toolkit/*
YES YES YES YES
Speech Libraries and End-to-End Speech Demos GNA Software License Agreement <install_root>/data_processing/audio/speech_recognition/* YES YES YES NO
DL Streamer EULA <install_root>/data_processing/dl_streamer/* NO YES NO NO

Where to Download this Release

Choose the Best Option

System Requirements

Intel® CPU Processors

Hardware

  • Intel Atom® processor with Intel SSE4.1 support
  • Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
  • 6th - 10th generation Intel® Core™ processors
  • Intel® Xeon® processor E3, E5, and E7 family (formerly Sandy Bridge, Ivy Bridge, Haswell, and Broadwell)
  • Intel® Xeon® Scalable processor (formerly Skylake and Cascade Lake)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Ubuntu 16.04 long-term support (LTS), 64-bit
  • Windows® 10, 64-bit
  • macOS 10.14, 64-bit

Intel® Processor Graphics

Hardware

  • Intel HD Graphics
  • Intel® UHD Graphics
  • Intel® Iris® Pro Graphics

Note This installation requires drivers that are not included in the Intel Distribution of OpenVINO toolkit package

Note A chipset that supports processor graphics is required for Intel®Xeon processors. Processor graphics are not included in all processors. See Product Specifications for information about your processor.

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Windows 10, 64-bit
  • Yocto* 3.0, 64-bit

Intel® Gaussian & Neural Accelerator (Intel® GNA)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Windows 10, 64-bit

Intel® FPGA Processors

Note Only for the Intel Distribution of OpenVINO™ toolkit for Linux with FPGA Support

Hardware

  • Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2
  • Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA (Intel PAC with Intel Arria 10 GX FPGA)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit

Intel® VPU Processors

Hardware with corresponding operating systems

  • Intel Vision Accelerator Design with Intel Movidius Vision Processing Units (VPUs):
    • Ubuntu 18.04 long-term support (LTS), 64-bit (Linux Kernel 5.2 and lower)
    • Windows 10, 64-bit
    • CentOS 7.4, 64-bit 
  • Intel Movidius Neural Compute Stick (NCS) and Intel Neural Compute Stick 2 (Intel NCS2):
    • Ubuntu 18.04 long-term support (LTS), 64-bit
    • CentOS 7.4, 64-bit
    • Windows 10, 64-bit
    • Raspbian* (target only)
  • AI Edge Computing Board with Intel Movidius Myriad X C0 VPU, MYDX x 1:
    • Windows 10, 64-bit

Operating Systems and Frameworks Used In Validation

Operating systems:

  • Ubuntu 16.04.6 with Linux kernel 4.15
  • Ubuntu 18.04.3 with Linux kernel 5.3
  • CentOS 7.4 with Linux kernel 5.3
  • Windows 10 version 1809 (known as Redstone 5)
  • OS X 10.14
  • Raspbian 9

DL frameworks:

  • TensorFlow 1.14.0 and 1.15.2
  • MxNet 1.5.1

Helpful Links

Legal Information

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at http://www.intel.com/ or from the OEM or retailer.

No computer system can be absolutely secure.

Intel, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos

*Other names and brands may be claimed as the property of others.

Copyright © 2020, Intel Corporation. All rights reserved.

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804