for (i = 0; i < 576; i++)

        {

            FLOAT   sum1, sum2;

            sum1 = firbuf[i + 10];

            sum2 = 0.0;

            for (j = 0; j < ((NSFIRLEN - 1) / 2) - 1; j += 2)

            {

                        sum1 += fircoef[j] * (firbuf[i + j] + firbuf[i + NSFIRLEN - j]);

                        sum2 += fircoef[j + 1] * (firbuf[i + j + 1] + firbuf[i + NSFIRLEN - j - 1]);

            }

            ns_hpfsmpl[chn][i] = sum1 + sum2;

        } 

I'm guessing someone tried to optimize the inner loop by unrolling and computing two parallel sums, adding them together for the final sum outside the loop.  The idea may have been to break the dependency chain on a single register which accumulates the sum.  That could help on a machine that can issue two loads per cycle, where the multi-cycle latency of each add in the summing chain would be exposed.  However current Intel CPU’s can only do one load per cycle, so the code will remain bound by the three loads required per result, regardless of the unrolling.

Note that NSFIRLEN is a predefined macro set to the equal 21.  Since the inner loop count is a constant we can unroll it by hand:

        for (i = 0; i < 576; i++)

        {

            FLOAT   sum1;

            sum1 = firbuf[i + 10];

            sum1 += fircoef[0] * (firbuf[i + 0] + firbuf[i + NSFIRLEN - 0]);

            sum1 += fircoef[1] * (firbuf[i + 1] + firbuf[i + NSFIRLEN - 1]);

            sum1 += fircoef[2] * (firbuf[i + 2] + firbuf[i + NSFIRLEN - 2]);

            sum1 += fircoef[3] * (firbuf[i + 3] + firbuf[i + NSFIRLEN - 3]);

            sum1 += fircoef[4] * (firbuf[i + 4] + firbuf[i + NSFIRLEN - 4]);

            sum1 += fircoef[5] * (firbuf[i + 5] + firbuf[i + NSFIRLEN - 5]);

            sum1 += fircoef[6] * (firbuf[i + 6] + firbuf[i + NSFIRLEN - 6]);

            sum1 += fircoef[7] * (firbuf[i + 7] + firbuf[i + NSFIRLEN - 7]);

            sum1 += fircoef[8] * (firbuf[i + 8] + firbuf[i + NSFIRLEN - 8]);

            sum1 += fircoef[9] * (firbuf[i + 9] + firbuf[i + NSFIRLEN - 9]);

            ns_hpfsmpl[chn][i] = sum1;

        }

This allows the Intel Compiler to vectorize with packed SSE instructions around the outer loop and overall leads to a 3% reduction in encode time. 

Since SSE operates on four single-precision float elements held in 128-bit registers, the compiler achieves vectorization by unrolling the loop by four and computing four sums line-by-line.  Vectorization around the original inner loop is possible but not as efficient since it has a small trip count (10) and requires horizontal summing of each result.

The outer-loop approach also takes advantage of the PALIGNR instruction to form operands for each array offset in registers rather than doing a bunch of unaligned 16-byte loads from memory.  It ends up being over 200 lines of assembly code for the whole loop... really makes you appreciate having a smart compiler!

for(i=0; i < l; i++)
   {
      float x0 = xr[i] * istep;
      int rx0 = (int)x0;
      x0 += adj43[rx0];
      ix[i] = (int)x0;
   }

The function consumes about 10% of the run-time for constant bit-rate encoding. I had enabled compiler vectorization using "#pragma vector always" but the resulting SSE2 code didn't make a dent in the overall encode time.

The critical bottleneck with this loop is the "gather" or table-lookup sequence which demands individual reads from the adj43 array for each result. Our instruction set currently does not provide a method to execute such parallel loads, so each rx0 element must be extracted into a general purpose register (eax, ebx, etc.) to set up the loads and likewise each float value retrieved as adj43[rx0] must be packed back into an xmm register. The SSE2 form of this algorithm needs 22 total instructions to produce four results in one register.

My first crack at an SSE 4.1 version used the PEXTRD and INSERTPS instructions to tighten up the sequence in moving the data to and from the GPR's. This shortened the whole loop to 14 instructions, but oddly didn't improve performance by any measure. Breaking the function out into a micro-kernel testing jig actually showed that this version was slightly slower than the SSE2 code on a Nehalem 3.2 Ghz system.

So what happened? I understood this better after digging into the micro-op sequences used in decoding each instruction. Micro-ops are the RISC-like elements that combine to execute each x86 instruction inside the CPU execution pipeline. Naturally, I wanted to choose instructions that decode into shorter micro-op flows so the CPU could get by doing less work. Our CPU definition files showed that PEXTRD and INSERTPS decode to 2 and 3 micro-ops respectively. So, while I reduced instruction count at the externally-visible level, my SSE4.1 code actually increased the number of micro-ops from 25 to 26 for the whole loop.

Curiously, while I was inspecting the micro-op flows I noticed that EXTRACTPS and PINSRD each used one less micro-op than their counterparts noted above. The difference between PEXTRD and EXTRACTPS is mainly conceptual based on the data type, e.g. PEXTRD is for extracting 32-bit integers (DWORD's) and EXTRACTPS is for 32-bit floats (Packed Single-precision data). I intuitively chose PEXTRD and INSERTPS because the algorithm was extracting DWORD array indices and inserting floats. So why the extra micro-ops? In the extract case, I am not sure... However, the INSERTPS instructions does offer the added ability to select the input operand from another xmm register, so that explains why it uses one more shuffle micro-op than PINSRD.

Another factor to consider is port bindings. Each micro-op can only be dispatched on a certain subset of the ports (0-5 on Nehalem) and each port is limited to dispatching one operation per cycle (try Googling "Nehalem block diagram" for a good visual). My initial SSE4.1 loop was bound by having seven micro-ops that map to Port 0, so it had an ideal throughput of 7 cycles. To figure this out I listed all 26 micro-ops and tried to arrange the port mappings so they would be spread as evenly as possible over all ports. It sounds complex but some people here can do it in their head... for almost any piece of code... on any CPU from the 1995 Pentium Pro out to what we are designing for 2013. (In a similar fashion I often proof-read these blog posts over and over trying to figure out how I can reduce the number of words... but I'm not going to do that today ).

After all that I rewrote my algorithm using the combination of EXTRACTPS and PINSRD and saw a 2% speedup in encode time. Not much you might think, but consider I only used two new instructions. The selection went against the data type in each case but nonetheless reduced the micro-op count from 26 to 20. Also the throughput improved to 5 cycles/result, bound on ports 1 and 2.

Here is my intrinsics implementation, a bit ugly for the need to use casts to match the data types:

   int i, t0, t1, t2, t3;
   __m128 x4h, xr4, x4, istep4 = _mm_set1_ps(istep);
   __m128i rx4;

   for(i=0; i < l; i+=4)
   {
      xr4 = _mm_load_ps(&xr[i]);
      xr4 = _mm_mul_ps(xr4, istep4);
      rx4 = _mm_cvttps_epi32(xr4);

      t0 = _mm_cvtsi128_si32(rx4);
      t1 = _mm_extract_ps(_mm_castsi128_ps(rx4), 1);
      t2 = _mm_extract_ps(_mm_castsi128_ps(rx4), 2);
      t3 = _mm_extract_ps(_mm_castsi128_ps(rx4), 3);

      x4 = _mm_load_ss(&adj43[t0]);
      x4 = _mm_castsi128_ps(_mm_insert_epi32(_mm_castps_si128(x4), *(int*)&adj43[t1], 1));
      x4 = _mm_castsi128_ps(_mm_insert_epi32(_mm_castps_si128(x4), *(int*)&adj43[t2], 2));
      x4 = _mm_castsi128_ps(_mm_insert_epi32(_mm_castps_si128(x4), *(int*)&adj43[t3], 3));

      xr4 = _mm_add_ps(xr4, x4);
      rx4 = _mm_cvttps_epi32(xr4);
      _mm_store_si128((__m128i*)&ix[i], rx4);

   }

Alternatively I have put in a request for the Intel Compiler to emit similar code when vectorizing with the /QxS switch (targeting SSE4.1 code generation). This could appear in the 11.0 release but no guarantees as yet.

for (l = -width; l < 0; l++)

            if (xr[j + l] < roundfac)

                  ix[j + l] = 0;

The loop exhibits no data dependencies between iterations and contains no unvectorizable operations.  The trip count (width) is occasionally as small as ‘4’ but often much larger.  However, the compiler by default would not emit SIMD code here, for a few different reasons:

 1)      Because the array arguments ‘xr’ and ‘ix’ are passed into the function, the compiler cannot assume their address ranges don’t overlap.  If they did, vectorizing the code could lead to incorrect results.  This is commonly known as ‘aliasing’ and can be circumvented either by adding “#pragma ivdep” above the for statement, or compiling with /Oa or /Ow switches.

2)      With that fixed, we get a report that the loop was not vectorized due to an “unsupported loop structure”.  Generally, the compiler likes simplicity, so we tried recoding the loop with single-operand indices, as such:

 for (k = j, j += width; k < j; ++k)

            if (xr[k] < roundfac)

                  ix[k] = 0;

3)      That left us with one remaining issue, a bit more esoteric this time – the compiler reporting “loop not vectorized:  condition may protect exception”.  So it fears that the address range dereferenced by ‘ix’ may not be continuous?  That seems unlikely, but fair enough… A SIMD scheme would typically write to every address in the range, either with the existing value or zero, based on a mask generated by the conditional operation.

The solution here required some input from compiler engineering.  Eventually we arrived at the loop below, to alleviate concerns about writing through the range of ‘ix’:

       for (k = j, j += width; k < j; ++k)

            ix[k] = (xr[k] >= roundfac) ? ix[k] : 0;

The compiler sure seems to like that ? operator.  It always worked best for min/max problems on some of my previous projects in financial computing (unfortunately none of those apps advised me to unload my whole portfolio...).

Finally we came out with a concise SSE2 loop:

.B7.32:

        movups    xmm0, XMMWORD PTR [ecx+eax*4]

        movups    xmm2, XMMWORD PTR [16+ecx+eax*4]             

        movaps    xmm1, xmm3                                    

        cmpleps   xmm1, xmm0                                   

        pand      xmm1, XMMWORD PTR [ebx+eax*4]                

        movdqa    XMMWORD PTR [ebx+eax*4], xmm1                

        movaps    xmm4, xmm3                                   

        cmpleps   xmm4, xmm2                                   

        pand      xmm4, XMMWORD PTR [16+ebx+eax*4]             

        movdqa    XMMWORD PTR [16+ebx+eax*4], xmm4             

        add       eax, 8                                       

        cmp       eax, esi                                    

        jb        .B7.32       

Again, this provided a 15% speedup for constant bit-rate encoding, as measured on a 2.9 Ghz Core 2 Duo (Merom) system running WinXP 32-bit.

The loop restructuring was checked into the LAME source tree back in August.  We’d hoped to have the “#pragma ivdep” inserted as well, but the development community prefers to avoid compiler-specific directives.  So, short of adding the #pragma yourself, we recommend using /Oa or /Ow along with your favorite SSE code generation switch (/QxW or better).

The first thing that jumped out from the VTune run was a function called “quantize_xrpow_lines”.  This was one of the top hotspot functions, consuming about 10% of the total run-time.  Here is a link to the latest source file residing on Sourceforge:

http://lame.cvs.sourceforge.net/viewvc/lame/lame/libmp3lame/takehiro.c?revision=1.75&view=markup

The code employs a bit of trickery known as the “Takehiro IEEE754 hack” which uses a sequence of adds to convert a floating point value to its integer counterpart.  The hack was conceived back in 2000, during the era of MSVC++ 6.0 which used an expensive _ftol service routine to convert floats to ints.  Coincidentally I wrote an article about this issue back then that is still on-line at http://software.intel.com/en-us/articles/fast-floating-point-to-integer-conversions/ (note you may need to Google for it, the page has moved around more than transient NBA coach Larry Brown over the years).  The gist of the paper is that prior to the Pentium III, the x86 ISA did not provide instructions that explicitly performed the float-to-int truncation cast required by the ANSI C standard.  The ftol routine had to modify the FP control word to achieve this behavior on the x87 stack.  At that time, the “magic float” hack was a method to improve convert performance without requiring any of the new SSE instructions that were emerging on the latest CPU’s.

Getting back to the present day mindset, SSE has been around for years and the most common compilers will at least generate scalar forms of the instructions.  Also, the hardware implementations of convert-truncate instructions (e.g. CVTTSS2SI) have improved to where they only take a few cycles on Core2 Duo.  Since the hotspot code does two such converts in a tight loop, I wanted to see if the Takehiro hack was still providing the benefits originally intended.

Ultimately I found that disabling the hack and reverting to the original code leads to 30% faster encoding time, under certain compilation conditions.  The best result comes from building with Intel Compiler 10.1 and the –QxT switch which targets Supplemental SSE3 code generation (note, -QxW SSE2 code generation is nearly as fast but I’m sure our compiler group would like to promote the latest switches).  MSVC++ 2005 compilation also chips off significant encoding time as long as you use the switch combination “/arch:SSE2 /fp:fast”.  Those parameters allow the compiler to generate SSE2 code by default and relax precision requirements.  (Without the latter, MSVC will do all calculations in double precision, even if the source code specifies float data types.  Double precision is free on the x87 floating-point stack, but in the SSE context you’ll see many CVTSS2SD and CVTSD2SS instructions throughout the code which will cripple performance.)

Finally, though it only gave a modest performance gain, I came up with a more concise coding of the “quantize_xrpow_lines” loop:

      for(i=0; i < l; i++)       {             float x0 = xr[i] * istep;             int rx0 = (int)x0;             x0 += adj43[rx0];             ix[i] = (int)x0;       }

The adj43 table lookup prevents a straightforward SIMD implementation,but the Intel Compiler can still vectorize this if you specify “#pragma vector always”.  It uses shift and unpack operations to extract the rx0 indices into general purpose registers and gather the array values back into one xmm register.  This measured about 20% faster in a microkernel (aka a test app separate from the full encoder), but didn’t trim off any appreciable encode time.  Nonetheless, restructuring the loop in this fashion leaves it in better position to leverage SIMD hardware improvements down the line. 

        do {
          x1 = 2.0 * random()/RAND_MAX - 1.0;
          x2 = 2.0 * random()/RAND_MAX - 1.0;
          w = x1 * x1 + x2 * x2;
        } while ( w >= 1.0 );

The x1 and x2 computations take a random integer from 0 to RAND_MAX and normalize it into the range -1.0 to 1.0.  While we might expect the compiler to reduce this to a single multiply by the constant (2.0/RAND_MAX) and then subtract 1.0, we can’t assume anything.  Take a look at the assembly listing:

        call      random                                        #55.15
..B2.4:
        cvtsi2sdq %rax, %xmm0                                   #55.15
        addsd     %xmm0, %xmm0                                  #55.15
        divsd     _2il0floatpacket.1(%rip), %xmm0               #55.24
        subsd     _2il0floatpacket.3(%rip), %xmm0               #55.35
        movsd     %xmm0, 24(%rsp)                               #55.35
        call      random                                        #56.15
..B2.5:
        cvtsi2sdq %rax, %xmm4                                   #56.15
        movsd     _2il0floatpacket.3(%rip), %xmm2               #56.35
        addsd     %xmm4, %xmm4                                  #56.15
        divsd     _2il0floatpacket.1(%rip), %xmm4               #56.24
        movsd     24(%rsp), %xmm0                               #57.13
        subsd     %xmm2, %xmm4                                  #56.35

Even without being an assembly wizard, you might detect that the two calls to random() are soon followed with divides by some constant value.  Those are going to chew up a lot of clock cycles.  So we should get a nice gain by explicitly folding this into a multiply:

    const double RMrcp = 2.0/RAND_MAX;

        for (int i = 0; i < LENGTH; i++)
          {
            do
            {
              x1 = random()*RMrcp - 1.0;
              x2 = random()*RMrcp - 1.0;
              w = x1 * x1 + x2 * x2;
            } while ( w >= 1.0 );
            
            _x1[i] = x1;
            _x2[i] = x2;
            _w[i] = w;

          }

This quick code mod pushed the speedup to 1.9x.

 
    for (int i = 0; i < LENGTH; i++)
      {
        double w, x1, x2;

        do {
          x1 = 2.0 * random()/RAND_MAX - 1.0;
          x2 = 2.0 * random()/RAND_MAX - 1.0;
          w = x1 * x1 + x2 * x2;
        } while ( w >= 1.0 );

        w = sqrt( (-2.0 * log( w ) ) / w );
        y1[i] = x1 * w;
        y2[i] = x2 * w;
      }

 
This will produce a stream of Gaussian pseudo-random numbers saved into the arrays y1 and y2.

Going in we know that the Intel Compiler will not vectorize the do/while loop because it has an unpredictable loop count that is dependent on the results of each pass through the code.  The outer for loop won’t vectorize either as the compiler is not yet capable of such magic.

Between the square root, log, and divides we have a lot of expensive stuff going on here.  It would pay to restructure the code into something that the compiler can recognize as a vectorization candidate.  This can be done by splitting the outer loop into two for loops and saving intermediate values into arrays, as such:

double x1, x2, w;
__declspec(align(16)) double _w[LENGTH], _x1[LENGTH], _x2[LENGTH];

for (int i = 0; i < COUNT; i++)
      {
       do
            {
              x1 = 2.0*random()/RAND_MAX - 1.0;
              x2 = 2.0*random()/RAND_MAX - 1.0;
              w = x1 * x1 + x2 * x2;
            } while ( w >= 1.0 );

            _x1[i] = x1;
            _x2[i] = x2;
            _w[i] = w;

    }

#pragma ivdep
#pragma vector aligned
for (int i = 0; i < LENGTH; i++)
          {
            w = _w[i];
            w = sqrt( (-2.0 * log( w ) ) / w );
            y1[i] = _x1[i] * w;
            y2[i] = _x2[i] * w;0
          }

 
With this restructuring we’ve isolated the sqrt/log/divide sequence into a loop simple enough for the compiler to vectorize.  The #pragma’s instruct it to ignore potential vector dependencies, i.e. pointer aliasing, and assume that the arrays exhibit 16-byte alignment (guaranteed by adding “__declspec(align(16))” to the declarations).

 
$ icc -xP -vec_report3  GaussianRand.C
...
GaussianRand.C(85): (col. 21) remark: loop was not vectorized: unsupported loop structure.
GaussianRand.C(95): (col. 2) remark: LOOP WAS VECTORIZED.

This generated a healthy 1.5x speedup on the algorithm.

Note that the compiler employs a “short vector math library” which contains SIMD versions of math calls such as log, exp, sin, cos, etc.  This enables vectorization on loops such as the one above.  You can see how SVML is used in the assembly listing:

..B3.9:                        
        movaps    16032(%rsp,%rbp), %xmm8                       #97.10
        movaps    16032(%rsp,%rbp), %xmm0                       #98.29
        call      __svml_log2                                   #98.29
        mulpd     %xmm9, %xmm0                                  #98.24
        divpd     %xmm8, %xmm0                                  #98.37
        sqrtpd    %xmm0, %xmm2                                  #98.10
        movaps    32(%rsp,%rbp), %xmm1                          #99.24
...

One of the first things we do in working toward a true top-end performance measurement is to rebuild the code with the Intel® Compiler.  Whether comparing against gcc 3.x or Microsoft Visual C++ compilations we often see a considerable performance gain right out of the box.

Beyond that we typically find further improvements by analyzing the source code hotspots and making adjustments to enable even better code generation from the Intel® Compiler.  The biggest gains often come from finding frequently executed computational loops that are not being vectorized, that is, they are not effectively using the SIMD capabilities of the x86 instruction set.  In this entry we’ll look at a couple examples where a simple tweak to a loop allowed it to vectorize and execute much more quickly.

In the first example, we have a “daxpy” loop that should vectorize using packed SSE3 instructions:

44: void DaxpyArray (double *x, double *y, double a, double *r)
45: {
46:
47:      for (unsigned int i = 0; i < LENGTH; i++)
48:           r[i] = a * x[i] + y[i];
49:
50: }

We’ll compile this with icc 10.1, using the ‘-xP’ switch to target processors supporting SSE3 instructions and add ‘-vec_report3’ to get an explanation for loops that did not vectorize:

$ icc -xP -vec_report3 SSE3_example.C

SSE3_example.C(47): (col. 5) remark: loop was not vectorized: existence of vector dependence.
SSE3_example.C(48): (col. 8) remark: vector dependence: proven FLOW dependence between r line 48, and x line 48.
SSE3_example.C(48): (col. 8) remark: vector dependence: proven ANTI dependence between x line 48, and r line 48.
etc.

This loop looks simple enough, so why didn’t it vectorize? The reports can be cryptic but from experience we know that the compiler can be pretty picky about what it will accept. It requires a very specific ‘for’ loop structure and in this case the “unsigned int” loop counter is throwing it off course. The simplest fix is to change the type to a regular signed ‘int’. This will be fine as long as LENGTH does not exceed the range of 32-bit signed integers.

47: for (int i = 0; i < LENGTH; i++)
48: r[i] = a * x[i] + y[i];
This was worth a 1.6x gain as measured on a 3.0 Ghz “Woodcrest” CPU, so a little code change can go a long way. Of course we’d like to see the compiler be less stringent and vectorize with an unsigned loop index. This has already been fixed in the compiler mainline builds and will most likely appear in the 11.0 release. We’re also working on making the vectorization reports more intuitive and useful.

$ icc -xP -vec_report3 SSE3_example.C
...
SSE3_example.C(47): (col. 5) remark: LOOP WAS VECTORIZED.
...

A second example involves the use of STL vector container classes. The compiler doesn’t yet know how to vectorize loops referencing data in that fashion. For example, the following:

void DaxpyVector (const vector& x, const vector& y, const double a, vector& r)
{
for (int i = 0; i < LENGTH; i++)
r[i] = a * x[i] + y[i];
}

must be recoded this way:
void DaxpyVector (const vector& x, const vector& y, const double a, vector& r)
{
double *xP, *yP, *rP;

xP = (double*)&x[0];
yP = (double*)&y[0];
rP = &r[0];

}

... reassigning the vectors to double* pointers so the compiler can emit vectorized code. A fix for this issue is a bit more complex and we are not yet sure when the compilers will be able to handle it optimally.

A few other things that can trip up the vectorizer:

• use of class member variables in the loop, either as array pointers or loop count variables. Either reassign them to local stack variables or experiment with no aliasing options like /Oa, /Qansi-alias, /Ow.
• any manipulation of vector classes, for example, size() and resize() calls, even outside the target loop can disable vectorization because of C++ exception handling anomalies. Either move them outside the function scope or try using the ‘–fno-exceptions’ switch.