Optimized Analytics Package for Spark* Platform (OAP for Spark* Platform)

Pull Command

docker pull intel/intel-optimized-ml:centos-7-oap-0.9.0-latest


Optimized Analytics Package for Spark* Platform (OAP for Spark* Platform) is a project to optimize Apache Spark* in various aspects including cache, shuffle, execution engine, MLlib and so on. Currently OAP includes the following optimizations:

  • SQL Data Source Cache: Optimize Spark* SQL Data Source using PMem as input data cache.
  • RDD Cache PMem Extension: Optimize Spark* RDD Cache using PMem.
  • Shuffle Remote PMem Extension: Optimize Spark* shuffle using remote PMem and RDMA.
  • Remote Shuffle: Shuffle implementation for writing shuffle to HDFS Filesystem compatible remote storage.
  • Unified Arrow Data Source and Native SQL Engine: Optimize SQL execuiton engine using vectorization, native and columnar data.
  • OAP MLlib: Optimized implementation of part of MLlib agorithms.


Documentation and Sources

Get Started​
Docker Repo
Main Github
Release Notes
Get Started Guide

Code Sources
Report Issue

License Agreement

LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the license file for additional details.


Product and Performance Information


Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804