Release Notes for Intel® Distribution of OpenVINO™ Toolkit 2020

By Andrey Zaytsev, Anastasiya Ageeva

Last Updated: 05/04/2020

Note For the Release Notes for the 2019 version, refer to Release Notes for Intel® Distribution of OpenVINO™ toolkit 2019

Introduction

The Intel® Distribution of OpenVINO™ toolkit is a comprehensive toolkit for quickly developing applications and solutions that emulate human vision. Based on Convolutional Neural Networks (CNNs), the toolkit extends CV workloads across Intel® hardware, maximizing performance. It accelerates applications with high-performance, AI and deep learning inference deployed from edge to cloud.

The Intel Distribution of OpenVINO toolkit:

  • Enables deep learning inference from the edge to cloud.
  • Supports heterogeneous execution across Intel accelerators, using a common API for the Intel® CPU, Intel® Integrated graphics, Intel® Gaussian & Neural Accelerator (Intel® GNA), Intel® Movidius™ Neural Compute Stick (NCS), Intel® Neural Compute Stick 2 (Intel® NCS2), Intel® Vision Accelerator Design with Intel® Movidius™ Vision Processing Unit (VPU), Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA Speed Grade 2.
  • Speeds time-to-market through an easy-to-use library of CV functions and pre-optimized kernels.
  • Includes optimized calls for CV standards, including OpenCV and OpenCL™.

New and Changed in the Release 4

Executive Summary

  • Improves performance while maintaining accuracy close to full precision (for example, FP32 data type) by introducing support for the Bfloat16 data type for inferencing using the 3rd generation Intel® Xeon® Scalable processor (formerly code-named Cooper Lake).
  • Increases accuracy when layers have varying bit-widths by extending the Post-Training Optimization Tool to support mixed-precision quantization.
  • Allows greater compatibility of models by supporting directly reading Open Neural Network Exchange (ONNX*) model format to the Inference Engine.
    • For users looking to take full advantage of Intel® Distribution of OpenVINO™ toolkit, it is recommended to follow the native workflow of using the Intermediate Representation from the Model Optimizer as input to the Inference Engine.
    • For users looking to more easily take a converted model in ONNX model format (for example, PyTorch to ONNX using torch.onnx), they are now able to input the ONNX format directly to the Inference Engine to run models on Intel architecture.
  • Enables initial support for TensorFlow* 2.2.0 for computer vision use cases.
  • Enables users to connect to and profile multiple remote hosts; collect and store data in one place for further analysis by extending the Deep Learning Workbench with remote profiling capability.

Backward Incompatible Changes Compared with v.2020 Release 3

  • List of Deprecated APIAPI Changes
  • IRv7 is deprecated and support for this version will be removed in the next major version (v.2021.1). It means IRv7 and older cannot be read using Core::ReadNetwork. Users are recommended to migrate to IRv10 to prevent compatibility issues in the next release.
  • Removed Inference Engine NNBuilder API. Users are recommended to use nGraph as a replacement to create a CNN graph from C++ code.
  • The following Inference Engine public API will be removed in the next release (v.2021.1):
    • InferencePlugin, IInferencePlugin and InferencEnginePluginPtr classes. The Core class should be used as a replacement.
    • PluginDispatcher class. The Core class should be used as a replacement.
    • CNNNetReader class. The Core::ReadNetwork class should be used as a replacement.
    • PrimitiveInfo, TensorInfo and ExecutableNetwork::GetMappedTopology. Use ExecutableNetwork::GetExecGraphInfo as a replacement.
    • ICNNNetworkStats, NetworkNodeStats, CNNNetwork::getStats and CNNNetwork::setStat. Use IRv10 with FakeQuantize approach for int8 flow replacement.
    • IShapeInferExtension and CNNNetwork::addExtension. Use IExtension class as a container for nGraph::Nodes which implement shape inference.
    • IEPlugin from Inference Engine Python API. Use Core API instead.
    • Data::getCreatorLayer, Data::getInputTo and CNNLayer. Use CNNNetwork::getFunction to iterate over a graph.

Model Optimizer

Model Optimizer

Common changes

  • Fixed Model Optimizer bugs to generate reshape-able IRs of many models with the command-line parameter "--keep_shape_ops".
  • Fixed Model Optimizer transformations to set operations name in an IR equal to the original framework model operation name.
  • Implemented an optimization transformation to replace a sub-graph with the TanH operation into the GeLU operation.

ONNX*

  • Added support for the following operations:
    • Clip-11
    • CumSum-11
    • NonZero-9
    • Pad-11
    • ReduceL2-11
    • ReverseSequence-10
    • ROIAlign-10
    • Scatter-11
    • ScatterElements-11
    • Softplus-1
    • TopK-10
    • QuantizeLinear-10/DequantizeLinear-10 pair when it can be replaced with FakeQuantize operation
    • Where-9

TensorFlow*

  • Introduced initial support for the TensorFlow* 2.X models, specifically for CNN (or computer vision use cases). RNN and Embedding layers are not yet supported.
  • Added support for the following operations:
    • IdentityN
    • NoOp
    • Where

MXNet*

  • Added support for the following operation:
    • CumSum

Kaldi*

  • Added support for the following operations:
    • fixedbiascomponent
    • fixedscalecomponent

Inference Engine

Inference Engine

Common changes

  • The primary CPU preprocessing pipeline is now fully optimized for AVX2 and AVX512. Previously, only SSE4.2-based optimizations were available.
  • The Core::ReadNetwork method was re-implemented to use readers plugins that are dynamically loaded in runtime:
    • Implemented the inference_engine_onnx_reader  plugin to read native ONNX models. This plugin depends on the ONNX Importer and uses it to read ONNX models.
    • Implemented the inference_engine_ir_reader plugin to read IRs.

Inference Engine Python API

Inference Engine Python API

  • No changes.

Inference Engine C API

Inference Engine C API

  • Added ie_core_read_network_from_memory to ensure encrypted reading.

CPU Plugin

CPU Plugin

  • Introduced support for bfloat16 (BF16) data type for inferencing. This feature improves performance while keeping the same accuracy when compared with FP32 model on targets with native BF16 support (for example, 3rd generation Intel® Xeon® Scalable Processors, formerly code-named Cooper Lake)
  • Several operations from opset3 were implemented: 
    • Bucketize-3
    • CumSum-3
    • EmbeddingBagOffsetsSum-3
    • EmbeddingBagPackedSum-3
    • EmbeddingSegmentsSum-3
    • ExtractImagePatches-3
  • Select, DepthToSpace, SpaceToDepth operations were updated to be conformant with latest opset.
  • INT8 inference performance enhancements for popular use-cases: NLP (BERT), super-resolution, person detection and pose-estimation.

GPU Plugin

GPU Plugin

  • Supported several new operations:
    • CumSum-3
    • EmbeddingBagOffsetsSum-3
    • EmbeddingBagPackedSum-3
    • EmbeddingSegmentsSum-3
    • BatchToSpace
  • Decomposition for the following operations was disabled, so now they are supported directly:
    • ShuffleChannels
    • DepthToSpace

MYRIAD Plugin

MYRIAD Plugin

  • Product Change Notice: Intel® Distribution of OpenVINO™ toolkit
    Change Notice Begins July 2020
    Change Date October 2020
    For support of custom OpenCL layers on Intel® Neural Compute Stick 2, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs, manual dma extension will be removed in favor of automatic-dma.
  • Intel® Movidius™ Neural Compute Stick support is discontinued.
  • 3 LTS is the last one with Myriad-2 HW support.
  • Supported new operations:
    • NonZero
    • ScatterElementsUpdate
    • Broadcast
    • Added I32 data type support in many arithmetic operations
  • Improved performance of existing operations:
    • LSTMCell
    • LSTMSequence
    • Gather
    • TopK
    • OneHot
  • OpenCL compiler, targeting Intel® Neural Compute Stick 2 for the SHAVE* processor only, is redistributed with OpenVINO. OpenCL support is provided by ComputeAorta*, and is distributed under a license agreement between Intel® and Codeplay* Software Ltd.

HDDL Plugin

HDDL Plugin

  • Same new operations and optimizations as in MYRIAD plugin.
  • OpenCL compiler, targeting Intel® Vision Accelerator Design with Intel® Movidius™ VPUs for the SHAVE* processor only, is redistributed with OpenVINO. OpenCL support is provided by ComputeAorta*, and is distributed under a license agreement between Intel® and Codeplay* Software Ltd.

GNA Plugin

GNA Plugin

  • GNA library updated to v2:
    • Limitation on the number of layers is softened
    • Several networks can now be created simultaneously on the GNA device
    • Removed dependency on Intel® C++ Compiler runtime on Linux
    Because of updates to the library, GNA drivers should be updated to GNA v2 compatible versions:
  • Extended support for Permute layer:
    • all variants of the layer that do not require memory reordering ("trivial permutes") are now supported
    • a non-trivial permute is now supported when at least one of the reordered dimensions is <=8
  • Added support for a number of new layer combinations
  • Support for GNA specific parameters in the C++ benchmark app
  • Support for dynamic switching of execution mode (SW vs HW)
  • Up to 100 input layers are now supported instead of 10
  • Memory optimizations

FPGA Plugin

FPGA Plugin

Product Change Notice
Intel® Distribution of OpenVINO™ toolkit for Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA

Change Notice Begins July 2020
Change Date October 2020


Intel will be transitioning to the next-generation programmable deep-learning solution based on FPGAs in order to increase the level of customization possible in FPGA deep-learning. As part of this transition, future standard releases (i.e., non-LTS releases) of Intel® Distribution of OpenVINO™ toolkit will no longer include the Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA.
Intel® Distribution of OpenVINO™ toolkit 2020.3.X LTS release will continue to support Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA. For questions about next-generation programmable deep-learning solutions based on FPGAs, please talk to your sales representative or contact us to get the latest FPGA updates.

nGraph

  • Moved nGraph source code to the OpenVINO™ toolkit repository.
  • Removed dependency on Inference Engine transformations library to the Inference Engine. Now the library only depends on nGraph.
  • Separated ONNX importer from nGraph, so now it is a separate dynamic library which depends on nGraph library.
  • Introduced opset3. The new opset contains the following new operations (not all OpenVINO™ toolkit plugins support new operations):
    • Assign-3
    • Broadcast-3
    • Bucketize-3
    • CumSum-3
    • EmbeddingBagOffsetsSum-3
    • EmbeddingBagPackedSum-3
    • EmbeddingSegmentsSum-3
    • ExtractImagePatches-3
    • GRUCell-3
    • NonMaximumSuppression-3
    • NonZero-3
    • ReadValue-3
    • RNNCell-3
    • ROIAlign-3
    • ScatterElementsUpdate-3
    • ScatterUpdate-3
    • ShapeOf-3
    • ShuffleChannels-3
    • TopK-3

Neural Networks Compression Framework (NNCF)

  • NNCF for PyTorch is completely open-sourced under OpenVINO™ toolkit repository
  • NNCF v1.3.1 for PyTorch has been released:
    • Automatic mixed (INT4+INT8) precision quantization (HAWQ-powered) has been released. Two models have been provided to evaluate performance gain, accuracy metrics for compressed models are as follows:

      Model

      Compression

      Acc1@Imagenet
      (FP32/Compressed)

      MobileNet V2

      46.6% INT8
      53.4% INT4

      71.81%/70.89%

      ResNet-50

      44.8% INT8
      55.2% INT4

      76.13%/76.30%

    • Implemented and integrated to NNCF Soft Filter Pruning and Geometric Median Pruning algorithms. Requested pruning rate (PR) can’t be achieved in some cases as accuracy drop of 1% is preserved. However, performance (inference on CPU) always scales similarly to achieved pruning rate.

      Model name

      Requested PR

      Actual PR

      MACs

      FPS

      resnet18

      30%

      15.8%

      -16.9%

      +18.4%

      resnet34

      30%

      21.7%

      -22.1%

      +25.1%

      resnet50

      30%

      28.3%

      -30%

      +33.1%

      googlenet

      30%

      36.4%

      -41.9%

      +23%

    • HW configuration files support has been added to the tool in order to enable quantization parameters tuning for specific HW targets. The API and format of the configuration files are close to that of the Post-training Optimization Tool - just extending them with NNCF specifics.
    • Release supports standard ONNX opset INT8 export, latest PyTorch, TorchVision and CUDA runtime versions. Apart of that, Docker support and CPU-only training setup are enabled.
  •  NNCF has been used to quantize and fine-tune a number of models from the Transformers-based family: BERT-large and DistilBert. INT8 models all have <1% accuracy drop and use symmetric quantization that should fit well to CPU performance-wise:

    Model

    Dataset

    Accuracy (FP32/INT8)

    Notes

    BERT-large (uncased, WWM)

    SQuAD v1.1

    93.26%/92.60% (F1)

    Fully symmetrically quantized 
    Release to OMZ
    1.5x perf gain vs FP32 on CPU
     

    BERT-large (uncased)

    SQuAD v1.1

    90.87%/90.41% (F1)

    Fully symmetrically quantized
    Model baseline for MLPerf submission
    Quality metrics on par with Nvidia INT8

    DistilBERT (uncased)

    Yelp (SST-2-like)

    95.50%/95.14% 

    Fully symmetrically quantized
    POC for customers (Roblox)

    DistilBERT (uncased)

     

    SST-2

    91.10%/90.30%

    Fully symmetrically quantized

Post-Training Optimization Tool

Post-Training Optimization Tool

  • Added support of quantization (Default, Accuracy Aware) for cascaded models. The quantization path requires support and models configuration through Accuracy Checker (AC) configs:
    • MTCNN: coarse detector (MTCNN-p) to detector/refining (MTCNN-r) to detector/output (MTCNN-o)
    • Action Recognition: encoder (classification backbone, 16 frames) to decoder (multi-head self-attention based)
    • Text Spotting: detection+segmentation+embeddings (mask-RCNN-like) to encoder (conv net on embeddings) to decoder (GRU + attention)
  • Introduced mixed quantization (for example, symmetric quantization for weights and asymmetric quantization for activations), which can be used as a quantization scheme for Default and Accuracy Aware algorithms to get better accuracy. 8 models out of all are quantized using this scheme, default quantization scheme is still symmetric.
  • Improved annotation-free quantization capability
  • Enhanced overall usability, which allows users to integrate the quantization workflow with a model validation pipeline by using an API in order to get the model quantized
  • Added new documentation of Post-Training Optimization Best Practices and Low Precision Optimization Guide
  • The list of layers that are being quantized has been extended to support NormalizeL2. This brings performance benefits for few models like SSD, person-detection-action-recognition (see S-curve below, INT8 2020.4 vs 2020.2 on CLX).

 

Deep Learning Workbench

Deep Learning Workbench

  • Added support for profiling on remote hosts and targets available through SSH protocol
  • Added support for Semantic and Instance segmentation models 
  • Added several usability improvements, including improved group inference configuration, improved history visualization, and a clearer system business indication

OpenCV*

  • SIFT algorithm has been moved from opencv_contrib repository after patent expiration
  • Enabled parallel IPP acceleration for GaussianBlur in OpenVINO builds

Examples and Tutorials

  • Speech libraries and end-to-end demos demonstrating speech recognition pipeline are now included to CentOS distribution.

Open Model Zoo

  • Extended the Open Model Zoo with additional CNN-pretrained models and pregenerated Intermediate Representations (.xml + .bin):
    • [NEW] bert-small-uncased-whole-word-masking-squad-0001
    • [NEW] bert-large-uncased-whole-word-masking-squad-fp32-0001
    • [NEW] bert-large-uncased-whole-word-masking-squad-int8-0001
    • [NEW] person-detection-0100
    • [NEW] person-detection-0101
    • [NEW] person-detection-0102
    • [NEW] person-detection-0106
    • [NEW] person-reidentification-retail-0265
    • [NEW] yolo-v2-tiny-vehicle-detection-0001
    • [NEW] vehicle-attributes-recognition-barrier-0042
    • [NEW] weld-porosity-detection-0001
    • person-reidentification-retail-0267 - Replaces the person-reidentification-retail-0249 model.
    • person-reidentification-retail-0270 - Replaces the person-reidentification-retail-0300 model.
  • Removed models:
    • person-reidentification-retail-0031
    • face-reidentification-retail-0095 - has been temporarily removed
  • The list of public models extended with support of the following models:
    Model Name Framework

    faceboxes-pytorch

    PyTorch
    fast-neural-style-mosaic-onnx PyTorch
    googlenet-v2-tf TensorFlow
    hbonet-1.0 PyTorch
    hbonet-0.5 PyTorch
    hbonet-0.25 PyTorch
    mobilefacedet-v1-mxnet MXNet
    mobilenet-v3-large-1.0-224-tf TensorFlow
    mobilenet-v3-small-1.0-224-tf TensorFlow
    open-closed-eye-0001 Caffe2
    pelee-coco Caffe
    resnet-34-pytorch PyTorch
    retinaface-anti-cov MXNet
    retinaface-resnet50 MXNet
    retinanet-tf TensorFlow
    ssd-resnet34-1200-onnx ONNX
    vehicle-reid-0001 Caffe2
  • Added new demo applications:
    • Classification
    • Bert question answering
    • Object detection faceboxes
    • Object detection retinaface
    • Whiteboard inpainting
  • Open Model Zoo tools
    • Functionality to resume download, in case if the user doesn't have a stable Internet connection, which is added to the Open Model Zoo Model Downloader tool
    • Quantization script extended with support public models trained on COCO and Wider Face datasets

Deep Learning Streamer

  • Added support for post-processing of RCNN, YoloV2 and YoloV3-tiny models in gvadetect
  • Provided an option to compact metadata representation attached to a video frame in order to reduce latency
  • Added a new sample for gvametapublish element. The sample outputs the metadata to file (default=stdout), MQTT or Kafka
  • Added a new property ‘json-indent’ in gvametaconvert to pretty-print json with indentation specified to make the syntax more legible
  • Added two new tutorials in Jupyter Notebook format, for the developers to try DL Streamer on Intel® hardware using the Intel® DevCloud for the Edge: Advanced Tutorials
  • Preview in open source repository: GStreamer element gvaaudiodetect for audio event detection and the corresponding sample pipeline

For more information on DL Streamer, please refer to the DL Streamer tutorial, API reference, and samples documentation at OpenVINO Inference Engine Samples. For the preview feature, please visit DL Streamer open source project located at GStreamer Video Analytics (GVA) Plugin repository on GitHub.

New and Changed in the Release 3 LTS

Executive Summary

New and Changed in the Release 2

Executive Summary

  • Introduces the new Deep Learning (DL) Streamer component, which uses the GStreamer* multimedia framework. This component enables the creation and deployment of analytic pipelines across Intel architecture from edge to cloud.
  • Includes bug fixes and stability fixes in memory leak issues in the Intel® Threading Building Blocks (Intel® TBB) library.
  • Improves support for noise suppression use cases by adding new layers and bug fixes to the Intel Gaussian & Neural Accelerator (Intel GNA) plug-in.
  • Adds support for new FP32 and INT8 models to enable more vision and text use cases:
    • Original models: 3D U-Net, MobileFace
    • Low precision (that is, INT8) models: EAST, OpenPose, RetinaNet, and FaceNet
  • Introduces new pretrained models on Open Model Zoo to reduce time to production, which include:
    • Multi-camera tracking for person reidentification
    • Cascade R-CNN for person detection
  • Published new performance benchmarks.

Backward Incompatible Changes Compared with v.2020 Release 1

  • List of Deprecated APIAPI Changes
  • Part of the Inference Engine shared library has been moved to the Inference Engine Legacy shared library, which contains obsolete routines needed for Intermediate Representation (IR) v7. You should link custom applications against both libraries, which is done automatically if you use the CMake* interface. 

Model Optimizer

Model Optimizer

Common Changes

  • Implemented generation of a compressed OpenVINO IR suitable for INT8 inference, which takes up to 4 times less disk space than an expanded one. Use the --disable_weights_compression Model Optimizer command-line parameter to get an expanded version.
  • Implemented an optimization transformation to replace a sub-graph with the Erf operation into the GeLU operation.
  • Implemented an optimization transformation to replace a sequence of Split-Concat operations to the Interpolate operation.
  • Fixed a number of Model Optimizer bugs to generate reshape-able IRs of many models with the command-line parameter --keep_shape_ops.
  • Fixed a number of Model Optimizer transformations to set operations name in an IR equal to the original framework model operation name.

ONNX*

  • Added support for the MeanVarianceNormalization operation if normalization is performed over spatial dimensions.

TensorFlow*

  • Added support for the TensorFlow* Object Detection models version 1.15.X.
  • Added support for the following operations:
    • BatchToSpaceND
    • SpaceToBatchND
    • Floor

MXNet*

  • Added support for the Reshape operation with input shape values equal to -2, -3, and -4.

Inference Engine

Inference Engine

Common Changes

  • Refactored the Inference Engine Extension API. The new API allows to create Network with custom nGraph operations and load custom CPU execution kernels for custom nGraph operations.
  • Added the Inference Engine Transformation Library, which contains nGraph transformation for the conversion of opset2 operations to opset1, and opset1 operations to the legacy operation set.
  • Added the Inference Engine LP transformations Library, which contains low precision transformation routines. The library is linked directly to plugins and not a dependency of the Inference Engine shared library.
  • Improved the support of AVX2 and AVX512 instruction sets in the CPU preprocessing module.

Inference Engine C API

Inference Engine C API

  • Added support of I420, NV12 blobs and added the dedicated hello_nv12_classifiction_c sample.

CPU Plugin

CPU Plugin

  • Removed the mkltiny dependency. This allowed to significantly decrease the binary distribution size required for inference on CPU.
  • Added several new operations:
  • GeLU
  • Swish
  • BatchToSpace
  • SpaceToBatch
  • Continued optimizations for the INT8 pipeline that resulted in 1.2x geomean speedup compared with the 2020.1 release on about 140 topologies.
  • Significantly improved load time for INT8 models.

GPU Plugin

GPU Plugin

  • Added support for the following new operations:
    • Swish
    • VariadicSplit
    • Transpose
    • SpaceToDepth
    • ConvertLike
    • Select
  • Added 3D-spatial dimensions support for the Resample primitive.
  • Fixed all known memory leaks.
  • Implemented performance optimizations for grouped convolutions, topk primitive, and models with batch size less than 1.
  • Implemented minor load time improvements for models with huge layers count or weights size.

MYRIAD Plugin

MYRIAD Plugin

  • Aligned VPU firmware with Intel® Movidius™ Myriad™ X Development Kit (MDK) R11 version.
  • Added support for 3D convolutions with restriction on grouped convolution size to 1.
  • Added support for 3D pooling.

HDDL Plugin

HDDL Plugin

  • Linux* kernel version 5.3 (and below) are supported now.
  • Due to the compatibility issue, the ION driver cannot be installed on Ubuntu* with a kernel version higher than 5.0 (included) and CentOS*, fall back to use shared memory.

GNA Plugin

GNA Plugin

  • Added support for the following layers:
    • Abs
    • Exp
    • Log
    • Sign
    • Softsign
    • Unsqueeze
  • Fixed problems with some layer combinations.
  • Fixed export and import of some models.
  • Fixed a problem with configuration key validation.

FPGA Plugin

FPGA Plugin

  • Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Mustang-F100-A10) speed grade 1 bitstreams and Intel® Programmable Acceleration Card (Intel® PAC) with Intel® Arria® 10 GX FPGA are not included in the Intel Distribution of OpenVINO toolkit 2020.2 release.
  • If you use Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA, we recommend continuing to use the Intel Distribution of OpenVINO toolkit 2020.1 release.

nGraph

  • Introduced opset2. This opset contains the following operations that were not included to opset1:
    • BatchToSpace
    • GELU
    • MVN
    • ReorgYolo
    • ROIPooling
    • SpaceToBatch
  • Removed the nGraph Python* API.
  • Enabled building nGraph ONNX Importer low-level API as a part of the nGraph shared library. This is a preview feature, which enables creating the nGraph Function directly from an ONNX model in runtime without running the Model Optimizer.

Post-Training Optimization Tool

Post-Training Optimization Tool

  • Additional models supported:
    • Added 25% more quantized and executed models (145 vs 116, Open Model Zoo scope), all within 1% of accuracy drop.
    • Enabled RetinaNet, Open Pose, Open Model Zoo/Human-pose (Single, 3d-pose), Mobile Face Det, and EAST.
    • Enabled the most of new Open Model Zoo models except the pipelined ones and asl-recognition-0004
  • Usability Improvements:
    • Implemented the annotation-free Default quantization (Simplified mode):
      • The mode allows quantizing a CNN without the Accuracy Checker support.
      • The mode accepts a set of images (unannotated dataset), performs default preprocessing and runs the Default quantization. 
      • The mode is NOT applicable for complex models, which work with many inputs, specific pre- and postprocessing, dataset different from a set of images.
    • Added the Quantization recipe (quantization_parameters section) to IR (*.xml) to show quantization parameters.
  • Implemented the new Bias Correction algorithm to improve the accuracy of quantized models.
    • The algorithm is slower but more precise than the previous version of Bias Correction. For example, it allows to get Mobilenet-v2/CF, mobilenet-v1-0.25/TF, image-retrieval-0001, text-detection-0004, text-recognition-0012 , person-reidentification-retail-0248,  and other models fully quantized with accuracy drop below 1%.
    • Previous Bias Correction algorithm is now called FastBiasCorrection. To activate the new one, set the quantization parameter "use_fast_bias": false.
  • Post-Training Optimization tool generates a compressed Intel Distribution of OpenVINO toolkit intermediate representation suitable for INT8 inference, which takes up to 4 times less disk space than an FP32 one. Use --keep_uncompressed_weights to get an expanded version.

Deep Learning Workbench

Deep Learning Workbench

  • Extended support for the INT8 Post-Training Optimization Tool. Now pretrained models can be calibrated to the INT8 precision with an annotation-free dataset (directory with images), which enables you to calibrate to lower precision without the need for a dataset - inputting images is sufficient. Only the Default (Maximum Performance) algorithm currently supports this capability.
  • Integrated the Deployment Manager, which allows users to get the most minimal package size for deployment after experimentations and fine-tuning using the DL Workbench
  • Added support for the Intel Vision Accelerator Designs target (on Linux only).
  • Separated calibration and validation datasets, which enables you to use different datasets for calibration and accuracy/performance measurements.
  • Implemented multiple usability and experience improvements, including a new look and feel for the comparison mode.
  • Enabled a new authentication mechanism using tokens, which provides additional security protection for users' models and datasets.

OpenCV*

  • Version changed to 4.3.0.
  • nGraph API is used by default in Inference Engine backend API in the dnn module. The NN Builder API is deprecated, but have not been removed yet. You can enable it with the cv::dnn::setInferenceEngineBackendType function.

Examples and Tutorials

  • Added sample to demonstrate nGraph C++ API. The sample demonstrates how to execute an inference using ngraph::Function to create a network. The sample uses the LeNet classifications network as an example.

Open Model Zoo

  • Extended the Open Model Zoo, which includes additional CNN-pretrained models and pregenerated Intermediate Representations (.xml + .bin):
    • [NEW] face-detection-0100
    • [NEW] face-detection-0102
    • [NEW] face-detection-0104
    • [NEW] face-detection-0105
    • [NEW] face-detection-0106
    • [NEW] unet-camvid-onnx-0001
    • [NEW] yolo-v2-ava-0001
    • [NEW] yolo-v2-ava-sparse-35-0001
    • [NEW] yolo-v2-ava-sparse-70-0001
    • [NEW] yolo-v2-tiny-ava-0001
    • [NEW] yolo-v2-tiny-ava-sparse-30-0001
    • [NEW] yolo-v2-tiny-ava-sparse-60-0001
    • [NEW] instance-segmentation-security-1025
    • asl-recognition-0004 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0248 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0249 - Replaces the 2020 Release 1 model.
    • person-reidentification-retail-0300 - Replaces the 2020 Release 1 model.
    • text-spotting-0002-detector - Replaces the 2020 Release 1 model.
    • text-spotting-0002-recognizer-decoding - Replaces the 2020 Release 1 model.
    • text-spotting-0002-recognizer-encoder - Replaces the 2020 Release 1 model.
    • faster-rcnn-resnet101-coco-sparse-60-0001 - Replaces the 2020 Release 1 model.
    • handwritten-japanese-recognition-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-sparse-30-0001 - Replaces the 2020 Release 1 model.
    • icnet-camvid-ava-sparse-60-0001 - Replaces the 2020 Release 1 model.
  • Added new demo applications:
    • Colorization
    • Image Inpainting
    • MonoDepth
  • Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow*, MXNet*, and PyTorch* formats:
Model Name Framework
colorization-v2 Caffe*
colorization-v2-norebal Caffe
googlenet-v1-tf TensorFlow
googlenet-v4-tf TensorFlow
gmcnn-places2-tf TensorFlow
i3d-rgb-tf TensorFlow
midasnet PyTorch*
resnet-50-tf TensorFlow
resnet-18-pytorch PyTorch
rfcn-resnet101-coco-tf TensorFlow
yolo-v1-tiny-tf TensorFlow
yolo-v2-tiny-tf TensorFlow
yolo-v2-tf TensorFlow
yolo-v3-tf TensorFlow

Deep Learning Streamer

Introducing the Deep Learning (DL) Streamer, now part of the default installation package of the Intel Distribution of OpenVINO toolkit. DL Streamer enables developers to create and deploy complex media analytics pipelines across Intel® architectures from edge to cloud. It provides optimal pipeline interoperability with familiar developer experience of the GStreamer multimedia framework. While enabling the developers to build a flexible streaming analytics pipeline, it also offers optimized inference across Intel architectures, from CPU and iGPU to Intel® Movidius™ Vision Processing Unit (VPU).

Key GStreamer elements provided by the DL Streamer:

Element Purpose
gvawatermark To overlay the metadata on the video frame to visualize the inference results.
gvatrack To perform object tracking using zero-term or short-term tracking algorithms. Zero-term tracking assigns unique object IDs and requires object detection to run on every frame. Short-term tracking allows to track objects between frames, thereby reducing the need to run object detection on each frame.
gvapython To provide a callback to execute user-defined Python functions on every frame. Can be used for metadata conversion, inference post-processing, and other tasks.
gvametapublish To publish the JSON metadata to MQTT or Kafka message brokers or files.
gvametaconvert To convert the metadata structure to the JSON format.
gvainference To run deep learning inference using any model with an RGB or BGR input.
gvafpscounter To measure frames per second across multiple streams in a single process.
gvadetect To perform object detection using YoloV2-tiny, YoloV3, and SSD-like models. Outputs the region of interest (ROI) for detected objects.
gvaclassify To perform object classification. Accepts the ROI as input and outputs classification results with the ROI metadata.


For more information, please refer to the DL Streamer tutorial, API reference, and samples documentation. at Intel Distribution of OpenVINO toolkit Samples. DL Streamer is an open source project located at GStreamer Video Analytics (GVA) Plugin repository on GitHub .DL Streamer includes Python and C++ APIs that provide access to inference results produced by the GStreamer elements for user applications written in Python and C++.  The sample code for both C++ and Python is available in the samples folder installed by the Intel Distribution of OpenVINO toolkit installer.

New and Changed in the Release 1

Executive Summary

  • Intel Distribution of OpenVINO Toolkit 2020.1 includes security bug fixes. Users should update to the latest version.
  • Post-training Optimization to enable significant acceleration improvement. Little or no degradation in accuracy using a model quantization; while streamlining the development process with no model re-training nor fine-tuning required. 
  • Extends support for custom layers on the Intel® Movidius™ Vision Processing Unit (VPU). This enables greater flexibility and customization for dedicated edge, media and vision inference.  
  • Low-precision runtime for INT8.  Including native support for “fake quantization” and support for 3D convolution layers along with reduced memory footprint. Enabling developers to add more deep learning models to their applications. 
  • Maximized performance optimizations and cross-platform flexibility with the Deep Learning Workbench.  Simplify the development and deployment workflow, with support for the Post-training Optimization tool, COCO dataset, new algorithms and improvements in developer experience and interfaces. For more information, watch the webinar on Deep Learning Workbench
  • Support for new demos and models on Open Model Zoo, which includes: 
    • American Sign Language (ASL) recognition 
    • Character and text detection with Mask R-CNN 
    • 3D human pose estimation
  • New speech libraries and speech recognition demos are included into distributions for Windows* and Ubuntu* 18.04 OSes. The new components enable and demonstrate end-to-end (speech to text) automatic speech recognition scenario.
  • OpenVX* was removed from distribution, please see a separate Intel® Distribution of OpenVX* Implementation page.
  • Switched to Linux Ubuntu* 18.04 kernel 5.3 by default due to performance drop on 9th Generation Intel® Core™ processors (formerly Coffee Lake) integrated graphics with kernel 5.0 compared to 5.3 and 4.15 (latency mode).
  • Added support for AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 is on Windows.
  • Please, review this documentation page to Get a Deep Learning Model Performance Boost with Intel® Platforms.

Backward Incompatible Changes Compared with v.2019 R3.1

  • Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples. Old path to the C++ samples was changed from <install_root>\deployment_tools\inference_engine\samples\ to <install_root>\deployment_tools\inference_engine\samples\cpp\.
  • Old Calibration tool 1.0 that was introduced with 2019 R1 was removed from the package and replaced by a new one.
  • CPU extensions library was removed. Extensions were moved into the plugin.
  • List of Deprecated APIAPI Changes.

Model Optimizer

Common Changes

  • Model Optimizer generates IR version 10 by default (except for the Kaldi* framework for which IR version 7 is generated) with significantly changed operations semantic. The command line --generate_deprecated_IR_V7 could be used to generate older version of IR. Refer to the documentation for the specification of a new operations set.
  • --tensorflow_use_custom_operations_config has been renamed to --transformations_config. The old command line parameter is deprecated and will be removed in the future releases.
  • Added ability to specify input data type using the --input command line parameter. For example, --input placeholder{i32}[1 300 300 3]. Refer to the documentation for more examples.
  • Added support for FusedBatchNorm operation when mode=’training’
  • Added optimisation transformation to remove useless Concat+Split sub-graphs.
  • A number of graph transformations were moved from the Model Optimizer to the Inference Engine.
  • Fixed networkX 2.4+ compatibility issues.

ONNX*

  • Added support for the following ONNX* operations:
    • Abs
    • Acos
    • Asin
    • Atan
    • Cast
    • Ceil
    • Cos
    • Cosh
    • Div
    • Erf
    • Floor
    • HardSigmoid
    • Log
    • NonMaxSuppression
    • OneHot
    • ReduceMax
    • ReduceProd
    • Resize
    • Sign
    • Sin
    • Sqrt
    • Tan
    • Xor
  • Added support for the following ONNX models:

TensorFlow*

  • Added optimization transformation which detects Mean Value Normalization pattern for 5D input and replaces it with a single MVN layer.
  • Added ability to read TF 1.X models when TF 2.x is installed.
  • Changed command line to convert GNMT model. Refer to the GNMT model conversion article for more information.
  • Deprecated --tensorflow_subgraph_patterns, --tensorflow_operation_patterns command line parameters. The TensorFlow offload feature will be removed from the future releases.
  • Added support for the following TensorFlow models:
    • Wide and Deep family of models (CPU only)
    • MobileNetV3 
  • Added support for the following TensorFlow operations:
    • Bucketize (CPU only)
    • Cast
    • Cos
    • Cosh
    • ExperimentalSparseWeightedSum (CPU only)
    • Log1p
    • NonMaxSuppressionV3, NonMaxSuppressionV4, NonMaxSuppressionV5
    • Sin
    • Sinh
    • SparseReshape (removed when input and output shapes are equal)
    • SparseToDense (CPU only)
    • Tan
    • Tanh

MXNet*

  • Added support for the following MXNet topologies:
  • Added support for the following MXNet operations:
    • UpSampling with bilinear mode
    • Where
    • _arange
    • _contrib_AdaptiveAvgPooling2D
    • div_scalar
    • elementwise_sub
    • exp
    • expand_dims
    • greater_scalar
    • minus_scalar
    • repeat
    • slice
    • slice_like
    • tile

Kaldi*

  • The --remove_output_softmax command line parameter now triggers removal of final LogSoftmax layer in addition to a pure Softmax layer.
  • Added support for the following Kaldi operations:
    • linearcomponent
    • logsoftmax
  • Added support for the following Kaldi models:

Inference Engine

Common Changes

  • Added C API bridge library that allows you to integrate the Inference Engine runtime and plugins into applications not using and depending explicitly on C++.
  • Added support of I420 blobs. Enable preprocessing and use class I420Blob to pass data for inference in I420 format. Inference Engine will convert data automatically to the BGR before inference.
  • Added Remote Blob allowing to address memory on remote hardware to build pipelines not depending on host memory
  • Enabled multi-socket support for Windows OS
  • Re-worked INT8 run-time (fake quantized based int8 flow) so that it provides higher performance and wider topology coverage:
    • Integrated new graph representation
    • Implemented graph transformations
    • Supported both symmetric and asymmetric quantization schemes
    • Extended INT8 support to cover additional use cases: 3d-models, semantic segmentation, instance segmentation
    • Identical functional coverage (features, models) for both CPU and GPU
    • Implemented optimizations needed to get 1.2x performance improvements over approach previously used

CPU Plugin

  • Re-designed (for CPU) fake quantized based int8 flow, which created also the foundation to scale to other int8 capable platforms.
  • Introduced NUMA* support for the Windows, generally re-designed NUMA awareness  (which now relies on the latest Threading Building Blocks (TBB) NUMA awareness feature). 
    • It is recommended to update all Intel products that use the TBB (like Intel® Distribution for Python*) to avoid possible TBB version conflicts when combining with Intel Distribution of OpenVINO toolkit.

GPU Plugin

  • Implemented remote blob for OpenCL and Direct3D* 11 frameworks and  support of memory been described in their terms on GPU

MYRIAD Plugin

  • Extended support for VPU Custom Layers

HDDL Plugin

  • Extended support for VPU Custom Layers
  • Linux kernel versions 5.2 and below are supported.

GNA Plugin

  • Added support for Tensor Iterator layer
  • Added support for Squeeze layer
  • Extended support for Permute Layer
  • Fixed issue with LSTM Cell layer
  • Improved accuracy for TDNN networks

FPGA Plugin

  • Introduced offline compilation for FPGA, HETERO:FPGA devices using compile_tool. Compile tool produces binaries which can be imported via Core::Import to target device.

Model Optimizer and nGraph Integration Into Inference Engine

  • Intel Distribution of OpenVINO toolkit is now powered by nGraph capabilities for Graph construction API, Graph transformation engine and Reshape, that replace former NN Builder API offering. Please refer to the new tutorials and updates for Model Optimizer and Inference Engine Developer's Guides updates on these new run-time capabilities. The NN Builder API is marked as deprecated and will be removed in the later releases.
  • Additionally, we revised operations set used in internal Intermediate Representations and offer an extended set, which covers both former Intel Distribution of OpenVINO toolkit and nGraph operations as well as adding necessary semantics to further speedup models enabling from supported frameworks. The detailed opset specification and new versioning schema for IR-vs-operations are provided in the documentation.
  • As result, Model Optimizer can now generate two types of IRs: a) IR V10 (default option) with this extended semantics and b) IR V7 (under --generate_deprecated_IR_V7 option in MO) which is just an evolution of IR V6 from previous release.
  • We expect that re-converting the existing (and already supported) models into new semantics (IR V10) should be transparent for end users, if for any reason conversion or inference fail, try generating V7 IR (with option above), if this also does not help please report us about your issue.

Post-training Optimization Tool

The main goal of the tool:

  • Transform Deep Learning model into a representation specific for selected type of optimization (e.g. quantization or sparsity) without model retraining
  • After such transformation, the model can be inferred efficiently on hardware supporting such optimizations (e.g. execution in INT8 precision)
  • The tool is extendable in order to support multiple quantization algorithms:

    Post-training Optimization Tool

The tool supports quantization of Intel Distribution of OpenVINO toolkit Intermediate Representation (IR) models for various types of IA hardware:

  • List of algorithms supported:
    • Default algorithm that is essentially a pipeline running three base algorithms:
      • MinMax
      • Bias Correction (runs on top of naive algorithm, based on minimization of per-channel quantization error)
      • Activation Channel Alignment (applied to align activation ranges)
    • Accuracy Aware algorithm that preserves accuracy of the resulting model (keeps accuracy drop below threshold)
  • Provides hardware-specific configs
  • Features per-channel / per-tensor quantization granularity
  • Supports symmetric / asymmetric quantization through presets mechanism
  • Two default presets are introduced for MinMax, Default and Data free algorithms to simplify user experience
    • 'Performance' preset stands for symmetric channel-wise (both for weights and activations) quantization
    • 'Accuracy' preset stands for symmetric weights + asymmetric activations quantization
    • Per-layer quantization tuning is also possible, some layers can be included into 'ignored scope' to skip quantization for those.

Tool is available at <install_root>\deployment_tools\tools\post_training_optimization_toolkit. After installation and running setup.py, the toolkit is available in the command line by the 'pot' alias.

Deep Learning Workbench

  • Added support for Int8 Post-Training Optimization Tool. Two new algorithms (Max performance and Max accuracy) are currently exposed
  • Added support for COCO dataset
  • Multiple UX/UI improvements including new look and feel for per-layer table view

OpenCV*

  • Version changed to 4.2.0.
  • Added experimental support for nGraph API of Inference Engine (can be selected through cv::dnn::setInferenceEngineBackendType() call).
  • Added Inference and Streaming extensions to G-API: now OpenCV G-API can handle full video analytics pipelines including video capture / decoding, image processing, CV, and DL scoring.
  • New tutorials are available for these features: Face Analytics and Face Beautification pipelines.

Examples and Tutorials

  • CPU extensions library was removed.
  • Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples.
  • Added end-to-end demos demonstrating speech recognition pipeline: offline demo for wave file transcription and GUI demo for live speech recognition.
  • Added samples to demonstrate Inference Engine C-API usage

Tools

  • Added feature extractor and decoder tools for speech processing pipelines. Added a tool for conversion of Kaldi* language models to the format used by the decoder tool.

Open Models Zoo

  • Extended the Open Model Zoo, which includes additional CNN-pretrained models and pre-generated Intermediate Representations (.xml + .bin):
    • [NEW] asl-recognition-0003
    • [NEW] person-detection-asl-0001
    • [NEW] person-reidentification-retail-0200
    • [NEW] resnet18-xnor-binary-onnx-0001
    • [NEW] text-spotting-0001-detector
    • [NEW] text-spotting-0001-recognizer-decoder
    • [NEW] text-spotting-0001-recognizer-encoder
    • [NEW] person-reidentification-retail-0103. Replaces the 2019 R3 model.
    • [NEW] person-reidentification-retail-0107. Replaces the 2019 R3 model.
  • Added new demo applications:
    • 3D human pose estimation
    • interactive face detection
    • multi-camera multi-person tracking
    • multi-channel object detection with YOLO v3 support
    • American sign language (ASL) recognition
    • object detection with CenterNet
    • single human pose estimation
    • text spotting

Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow, MXNet, and PyTorch formats:

Model Name Framework
efficientnet-b0 TensorFlow
efficientnet-b0-pytorch PyTorch
efficientnet-b0_auto_aug TensorFlow
efficientnet-b5 TensorFlow
efficientnet-b5-pytorch PyTorch
efficientnet-b7-pytorch PyTorch
efficientnet-b7_auto_aug TensorFlow
face-recognition-mobilefacenet-arcface MXNet
face-recognition-resnet100-arcface MXNet
face-recognition-resnet34-arcface MXNet
face-recognition-resnet50-arcface MXNet
densenet-121-caffe2 Caffe2
resnet-50-caffe2 Caffe2
squeezenet1.1-caffe2 Caffe2
vgg19-caffe2 Caffe2
ctdet_coco_dlav0_384 ONNX
ctdet_coco_dlav0_512 ONNX
brain-tumor-segmentation-0002 ONNX
human-pose-estimation-3d-0001 PyTorch
single-human-pose-estimation-0001 PyTorch
vehicle-license-plate-detection-barrier-0123 TensorFlow

Preview Features Terminology

A preview feature is functionality that is being introduced to gain early developer feedback. Comments, questions, and suggestions related to preview features are encouraged and should be submitted to the forum at Intel Distribution of OpenVINO Toolkit Forums.

The key properties of a preview feature are:

  •     It is intended to have a high quality implementation
  •     There is no guarantee of future existence or compatibility.

Note A preview feature is subject to change in the future. It may be removed or altered in future releases. Changes to a preview feature do NOT require the a deprecation and deletion process. Using a preview feature in a production code base is discouraged.

Fixed Issues

JIRA ID Description Component
26561 Performance degradation for model 'semantic-segmentation-adas-0001' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26560 Performance degradation for model 'deeplabv3' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26353 Accuracy issues on a dataset for deeplab_v3 model for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26384 Performance degradation for model 'refinedet-resnet-101' Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) IE MyriadX plugin, IE HDDL plugin
26700 Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) hangs with Linux kernel 5.3 IE HDDL plugin
24811 [CPU-VNNI] Segmentation fault on license-plate-recognition-barrier-0007, efficientnet-b0, person-reidentification-retail-003 models. Benchmark app crashes with segmentation fault at the very end of execution. IE CPU plugin
24517 Specialize_function adds redundant ops for functions with the Split operation nGraph Core
25417 Incorrect OL and OH params for the FQ layer MO TensorFlow
23985 Incorrect command-line parameters specified in the documentation to convert the LM1B model MO TensorFlow
25657 Memory leaks are possible in the GPU plugin in case of multiple network loading/unloading cycles IE GPU Plugin
25087 Performance degradations are possible in the GPU plugin on MobileNet-like models IE GPU Plugin
25020

May produce wrong output data in the case when several topologies are loaded into one CPU plugin. The issue requires several conditions:

  • Use IRv10
  • Several models are loaded into one plugin
IE CPU Plugin
25940

The TBB binary mismatch is possible. Depending on the version of a wrong loaded TBB module, it may lead to crashes, slowdown, missing symbols on load step.

All
23705

Inference may hang when running the heterogeneous plugin on GNA with fallback on CPU.

IE GNA Plugin
26585 Speech sample may display incorrect results of comparison with reference scores for models including LSTM cells Samples
24523 Low performance of asl-recognition-xxxx model on CPU IE CPU Plugin
23198 Inference of OneHot layer in constant subgraphs may be incorrect. As a result, GNMT topology may produce incorrect output. All
22108 Stopping the app during firmware boot might cause device hang for Intel® Neural Compute Stick 2 IE MyriadX plugin
23738 Core::SetConfig and Core::GetConfig methods work incorrectly with the GNA plugin. IE GNA plugin
29082 OpenVINO has thread-safety issues. LoadNetwork and other ie::Core methods simultaneously called from different threads may lead to error or crashes. All
28747 CPU plugin does not work on Windows system with CPUs less then AVX2 instruction set (Intel Atom® processors) IE CPU Plugin
28970 TF faster-rcnn and faster-resnet101 topologies accuracy deviation on MYRIAD IE MyriadX plugin, IE HDDL plugin
29361 Performance degradation for model 'text-recognition-0012' and ctpn IE MyriadX plugin, IE HDDL plugin
28683 Changing system time on Linux might break watchdog and interrupt the inference IE HDDL plugin
25723 TF rfcn_resnet101_coco low accuracy on dataset

IE MyriadX plugin, IE HDDL plugin

Known Issues

JIRA ID Description Component Workaround

#1

A number of issues were not addressed yet, please see "Known issue" section in the Release Notes for Intel® Distribution of OpenVINO™ toolkit 2019

All

N/A

21670 FC layers with bimodal weights distribution are not quantized accurately by the Intel® GNA Plugin when 8-bit quantization is specified. Weights with values near zero are set to zero. IE GNA plugin Use 16-bit weights in these cases.

25358

Some performance degradations are possible in the GPU plugin on GT3e/GT4e/ICL NUC platforms

IE GPU plugin

N/A

24709

Retrained TensorFlow Object Detection API RFCN model has significant accuracy degradation. Only the pretrained model produces correct inference results.

All

Use Faster-RCNN models instead of RFCN model if re-training of a model is required.

26129

TF YOLO-v3 model fails on AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 

IE MyriadX plugin

Use other versions of the YOLO network or USB connected device (Neural Compute Stick 2)

 

nGraph Python API has been removed from the current release due to its incompetence. It will be added back when it is ready with the desired quality.

IE Python API

Use C++ API

26388 Low latency (batch size 1) graphs with LSTMCell do not infer properly due to missing state handling. All Use deprecated IRv7 and manually insert memory layers into the IR graph. Alternatively, add state tensors as extra input and output nodes and associate their blobs given the IR node IDs after loading the graph.
24101 Performance and memory consumption may be bad if layers are not 64 bytes aligned IE GNA plugin Try to avoid the layers that are not 64 bytes aligned to make the model GNA friendly
28006 The result is incorrect for the concat layer having 3 or more input layers. IE GNA plugin Refactor the model so it would not have this layer
34350 When an LSTMCell is the output layer of a graph, the output is all zeros. IE GNA plugin Manually insert an additional non-functional layer (such as Identity activation) at the end of the IR graph.
31164 HDDL Xlink fail to connect more than 32 MYX devices IE HDDL plugin Use up to 4 Intel® Vision Accelerator Design with Intel® Movidius™ VPUs
33146 Some models fails with batch > 1 on Intel® Neural Compute Stick 2 and Intel® Vision Accelerator Design with Intel® Movidius™ VPUs IE MyriadX plugin, IE HDDL plugin Use batch 1 or use other HW (i.e. CPU/GPU)
32934 Models modified with deprecated CNNNetwork API may fail to load to Myriad/HDDL plugins IE MyriadX plugin, IE HDDL plugin Use IRv10 versions of the models
33881 Performance regression on se-resnext and efficientnet models IE MyriadX plugin, IE HDDL plugin Use OpenVINO 2020.3 LTS release version
33810 On Linux, "can't protect" exception may happen when loading large models with large number of CPU streams to the device IE MKL-DNN plugin Add the following line to /etc/sysctl.conf:
vm.max_map_count=2097152
Then reload the config with sysctl -p

Included in this Release

The Intel® Distribution of OpenVINO™ toolkit is available in the following versions:

  • Intel Distribution of OpenVINO toolkit for Windows
  • Intel Distribution of OpenVINO toolkit for Linux
  • Intel Distribution of OpenVINO toolkit for Linux with FPGA Support
  • Intel Distribution of OpenVINO toolkit for macOS*
Component License Location Windows* Linux* Linux for FPGA macOS*

Deep Learning Model Optimizer

Model optimization tool for your trained models

Apache 2.0 <install_root>/deployment_tools/model_optimizer/* YES YES YES YES

Deep Learning Inference Engine

Unified API to integrate the inference with application logic

Inference Engine Headers

EULA

 

 

Apache 2.0

<install_root>/deployment_tools/inference_engine/*

 

 

<install_root>/deployment_tools/inference_engine/include/*

YES YES YES YES

OpenCV library

OpenCV Community version
compiled for Intel hardware

BSD <install_root>/opencv/* YES YES YES YES

Intel® Media SDK libraries
(open source version)

Eases the integration
between the Intes Distribution for OpenVINO toolkit
and the Intel Media SDK.

MIT <install_root>/../mediasdk/* NO YES YES NO

Intel® Graphics Compute Runtime for OpenCL 

Improves usability 

MIT

<install_root>/install_dependencies/

    install_NEO_OCL_driver.sh - helps to install OpenCL Runtime, default location /usr/local/lib/

    intel-opencl_*.deb - driver for Ubuntu*

    intel-opencl_*.rpm - driver for CentOS*

    intel-* - driver's dependencies

NO YES YES NO

Intel® FPGA Deep Learning Acceleration Suite (Intel® FPGA DL Acceleration Suite), including pre-compiled bitstreams

Implementations of the most common CNN topologies to enable image classification and ease the adoption of FPGAs for AI developers.

Includes pre-compiled bitstream samples for the Intel Programmable Acceleration Card with Intel Arria 10 GX FPGA and Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2

Intel OBL FPGA SDK 18.1

<install_root>/bitstreams/a10_dcp_bitstreams/*

<install_root>/bitstreams/a10_vision_design_sg1_bitstreams/*

<install_root>/bitstreams/a10_vision_design_sg2_bitstreams/*

 

NO NO YES NO

Intel® FPGA SDK for OpenCL™ software technology

The Intel FPGA RTE for OpenCL provides utilities, host runtime libraries, drivers, and RTE-specific libraries and files

Intel OBL FPGA SDK 18.1

/opt/altera/aocl-pro-rte/*

<user>/intelFPGA_pro/<version>/*

NO NO YES NO

Intel Distribution for OpenVINO toolkit documentation

Developer guides and other documentation

  Available from the OpenVINO™ toolkit product site,
not part of the installer packages.
NO NO NO NO

Open Model Zoo

Documentation for models; Models in binary form can be downloaded using Model Downloader 

Apache 2.0 <install_root>/deployment_tools/open_model_zoo/* YES YES YES YES

Inference Engine Samples

Samples that illustrate Inference Engine API usage and demos that demonstrate how you can use features of Intel® Distribution of OpenVINO™ toolkit in your application

Apache 2.0 <install_root>/deployment_tools/
inference_engine/samples/*
YES YES YES YES

Deep Learning Workbench

Tool that can help developers to run  Deep Learning models through the toolkit's Model Optimizer, convert it to INT8, fine tune it, run inference and measure accuracy.

EULA <install_root>/deployment_tools/tools/workbench/* YES YES NO YES

nGraph

Open source C++ library, compiler and runtime for Deep Learning nGraph

Apache 2.0 <install_root>/deployment_tools/ngraph/* YES YES YES NO

Post-Training Optimization Toolkit

designed to convert a model into a more hardware-friendly representation by applying specific methods that do not require re-training, for example, post-training quantization.

EULA <install_root>/deployment_tools/
tools/post_training_optimization_toolkit/*
YES YES YES YES
Speech Libraries and End-to-End Speech Demos GNA Software License Agreement <install_root>/data_processing/audio/speech_recognition/* YES YES YES NO
DL Streamer EULA <install_root>/data_processing/dl_streamer/* NO YES NO NO

Where to Download this Release

Choose the Best Option

System Requirements

Intel® CPU Processors

Hardware

  • Intel Atom® processor with Intel SSE4.1 support
  • Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
  • 6th - 10th generation Intel® Core™ processors
  • Intel® Xeon® processor E3, E5, and E7 family (formerly Sandy Bridge, Ivy Bridge, Haswell, and Broadwell)
  • Intel® Xeon® Scalable processor (formerly Skylake and Cascade Lake)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Ubuntu 16.04 long-term support (LTS), 64-bit
  • Windows® 10, 64-bit
  • macOS 10.14, 64-bit

Intel® Processor Graphics

Hardware

  • Intel HD Graphics
  • Intel® UHD Graphics
  • Intel® Iris® Pro Graphics

Note This installation requires drivers that are not included in the Intel Distribution of OpenVINO toolkit package.

Note A chipset that supports processor graphics is required for Intel®Xeon processors. Processor graphics are not included in all processors. See Product Specifications for information about your processor.

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Windows 10, 64-bit
  • Yocto* 3.0, 64-bit

Intel® Gaussian & Neural Accelerator (Intel® GNA)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit
  • Windows 10, 64-bit

Intel® FPGA Processors

Note Only for the Intel Distribution of OpenVINO™ toolkit for Linux with FPGA Support

Hardware

  • Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2
  • Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA (Intel PAC with Intel Arria 10 GX FPGA)

Operating Systems

  • Ubuntu 18.04 long-term support (LTS), 64-bit

Intel® VPU Processors

Hardware with corresponding operating systems

  • Intel Vision Accelerator Design with Intel Movidius Vision Processing Units (VPUs):
    • Ubuntu 18.04 long-term support (LTS), 64-bit (Linux Kernel 5.2 and lower)
    • Windows 10, 64-bit
    • CentOS 7.4, 64-bit 
  • Intel Movidius Neural Compute Stick (NCS) and Intel Neural Compute Stick 2 (Intel NCS2):
    • Ubuntu 18.04 long-term support (LTS), 64-bit
    • CentOS 7.4, 64-bit
    • Windows 10, 64-bit
    • Raspbian* (target only)
  • AI Edge Computing Board with Intel Movidius Myriad X C0 VPU, MYDX x 1:
    • Windows 10, 64-bit

Components Used In Validation

Operating systems:

DL frameworks:

  • TensorFlow 1.14.0, 1.15.2, 2.2.0 (limited support according to product features)
  • MxNet 1.5.1

Helpful Links

Legal Information

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at Intel® or from the OEM or retailer.

No computer system can be absolutely secure.

Intel, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos

*Other names and brands may be claimed as the property of others.

Copyright © 2020, Intel Corporation. All rights reserved.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.