Contents

# Processing Data in Blocks

1. compute statistical estimates for out-of-memory datasets, splitting them into blocks
2. analyze in-memory data arrays that become available block by block
3. tune your applications for out-of-memory data support
To compute statistical estimates for out-of-memory datasets, do the following:
1. Set the estimates of your interest to zero, or to any other meaningful value:
for( i = 0; i < p; i++ ) {      Xmean[i] = 0.0;      Raw2Mom[i] = 0.0;      Central2Mom[i] = 0.0;      for(j = 0; j < p; j++)      {          Cov[i][j] = 0.0;      } }
2. Initialize array
W
of size 2 with zero values.
This array holds accumulated weights that are important for correct computation of the estimates:
W[0] = 0.0; W[1] = 0.0;
3. Get the first portion of the dataset into array
X,
and the corresponding weights into array
weights
:
GetNextDataChunk( X, weights );
4. Follow the common usage model of the Summary Statistics algorithms:
Summary statistics domain also enables reading the next data block into a different array. The whole computation scheme remains the same. You just need to provide the address of this data block to the library:
```
double* NextXChunk[N];
estimates = VSL_SS_MEAN | VSL_SS_2C_MOM | VSL_SS_COV | VSL_SS_VARIATION;
for( nchunk = 0; nchunk++; )
{
errcode = vsldSSCompute( task, estimates, VSL_SS_1PASS_METHOD );
If ( nchunk >= N ) break;
GetNextDataChunk( NextXChunk, [nchunk], weights );