Optimizing Storage Solutions Using the Intel® Intelligent Storage Acceleration Library

With the growing number of devices connected to the Cloud/Internet, data is being generated from many different sources including smartphones, tablets, and Internet of Things devices. The demand for storage is growing every year.  The combination of the Intel® Xeon® processor family and the Intel® Intelligent Storage Library (Intel® ISA-L) can provide developers with the tools to process data securely and quickly and even reduce storage space requirements.

In China, Intel collaborated with Qihoo 360 Technology Company Ltd to integrate Intel ISA-L into their storage solution. This resulted in a 10x performance increase and two-thirds reduction in required storage space. Read the case study.

Intel® ISA-L provides the tools to help accelerate and optimize storage on Intel® architecture (IA) for everything from small office NAS appliances up to enterprise storage systems.   The functions provided in this library help with storage recoverability, data integrity, data security, and faster data compression mechanisms.  This article provides a high level functional overview of Intel ISA-L.

Intel ISA-L provides the following collection of functions for use in storage applications:

1. RAID (Redundant Array of Inexpensive Disks) functions allow faster parity computation that can be used by a RAID provider.  The RAID functions calculate and operate on XOR and P+Q parity. The mathematics of RAID are based on Galois finite-field arithmetic to find one or two parity bytes for each byte in N sources such that single or dual disk failures (one or two erasures) can be corrected.
    
2. Erasure Code (EC) functions allow breaking up of objects into smaller fragments, storing the fragments in different places, and regenerating the data from any combination of smaller numbers of those fragments.  These EC functions implement a general Reed-Solomon type encoding for blocks of data to protect against erasure of whole blocks. Individual operations can be described in terms of arithmetic in the Galois finite field GF(2⁸) with the particular field-defining primitive or reducing polynomial x⁸+ x⁴+ x³+ x²+1 (0x1d).
    
3. CRC (cyclic redundancy check) functions permit the system to detect accidental changes to raw data during transmission.  The receiver can ask the transmitter to resend the package until the CRC matches. Functions in the CRC section are fast implementations of cyclic redundancy checks using IA specialized instructions such as PCLMULQDQ, carry-less multiplication.
    
4. Multi-buffer Hashing (MbH) functions provide cryptographic hash functions that use the capabilities of IA.  Intel ISA-L supports MD5, SHA1, SHA256, and SHA512. These MbH functions are used to increase the performance of the secure hash algorithms on a single processor core by operating on multiple jobs at once. By buffering jobs, the algorithm can exploit the instruction-level parallelism inherent in modern IA cores to an extent not possible in a serial implementation.
    
5. Encryption functions provide accelerated encryption by using the Intel® AES-NI instruction set.
    
6. Compression functions provide a fast, DEFLATE compatible compression routine. DEFLATE is a widely used binary compression standard that forms the basis of zlib, gzip and zip. The Intel ISA-L implementation of compression is written to be faster than zlib-1 with only a small sacrifice in compression ratio. This is well suited for high-throughput storage applications.

Depending on the platform capability, Intel ISA-L can run on various Intel® processor families.  Improvements are obtained by speeding up the computations through the use of the following instruction sets:

• Intel® AES-NI – Intel® Advanced Encryption Standard - New Instruction
• Intel® SSE – Intel® Streaming SIMD Extensions
• Intel® AVX – Intel® Advanced Vector Extensions
• Intel® AVX2 - Intel® Advanced Vector Extensions 2

Intel® ISA-L also includes unit tests, performance tests and samples written in C which can be used as usage examples.

Additional development information:

• The library supports several generations of Intel® processors by providing multi-binary versions of some functions which developers can compile to deploy as a single binary which will detect and optimize based on the processor in use. Alternatively, developers can reduce code size by calling just one version. 
• The calling convention for most functions in the library is C binding. Individual functions are written to be statically or dynamically linked with an application.
• To build the Intel ISA-L functions, use Yasm* Assembler version v1.2.0 or later.
• Some functions of the Intel ISA-L require the input parameters to be aligned on a 16B or 32B boundary. 

See the Intel ISA-L API reference manual for more details.

For developers who are interested in using the Intel ISA-L, the open source version (limited to Erasure Code functions) is available at 01.org. 

To access the full suite of Intel® Storage Acceleration Library functions, please fill out and submit this request form. We will respond to your query as soon as possible!

Other Related Links and Resources

Intel® Storage Acceleration Library (Open Source Version)

Erasure Code and Intel® Intelligent Storage Acceleration Library

Swift* with Erasure Coding for Storage

Intel and Qihoo 360 Internet Portal Datacenter - Big Data Storage Optimization Case Study

Notices

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to:  http://www.intel.com/design/literature.htm

Intel, the Intel logo, VTune, Cilk, and Xeon Phi are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others

Copyright© 2012 Intel Corporation. All rights reserved.

This sample source code is released under the Intel Sample Source Code License Agreement