Hello

Related topic : efficient way to declare your matrix in parallel.

http://en.wikipedia.org/wiki/Thread-local_storage

It's safe and efficient to share a read only variable or matrix between threads.

It's safe to share a read-write matrix if you write from different threads in different cells of the shared matrix.
But if you do that at high frequency, or high volume, be aware that sharing read-write matrices is not efficient.

RW variables should be declared in each thread to take advantage of thread locality (performance), and avoid potentialrace conditions(keep it safe).

Regards paul

If you are using C++, I would suggest vector of vectors:

vector> allValues

After loading values, you can convert it to some memory allocated "normal" 2D array ( int32_t[][] )

hello

in my case, because I do not want to use auto expandable array structures,
I decided to first read the file in order to find the size of the array,
(quickly parse the first line to find the number of columns, then count the numbers of lines)
then create the array with the right size and read the file again to fill the array.

But I'm not saying it the best solution ...

Tricks like quickly finding the number of line breaks can also work.

regards paul

You may read from file, line by line and dynamically allocate storage for your array. For example : read the first line, retain its size, allocate storage of that size for your array and copy the elements in your array; then the second line and so on...

In all cases, you are going to read the whole file ( if we assume M rows and N columns) then in complexity O(MN). So you can start by reading the first line in the file and counting the number of elemnts to get N the number of columns and then count number of lines in the file to get number of rows M, you will end up with complexity [O(M) (if M>N) or O(N) (if N > M) ] + O(MN) which yields in O(MN). The other solution is to read the elements into a dynamic vector for example, this is more efficient time-wise, but it will need more memory.

Hi,

I would like a clarification from an Intel representative (hint: Paul). In this thread:
http://software.intel.com/fr-fr/forums/showthread.php?t=91909&o=a&s=lr
It has been stated (by Paul) that a better solution should not require the complete data loaded into memory at all times (if I understood correctly), yet in this thread a solution (again by Paul) is clearly loading the whole matrix into the memory. If we want to write a parallel algorithm that doesn't have to keep the whole matrix inside the memory at all times, that algorithm will most probably be slower than the one that doesn't. My question is, should we concentrate on speed in this case or is it worth to concentrate on developing a different algorithm that will process the matrix partially?

It would also be nice if you (Paul) stated if we can consider you an Intel representative, or I'm just being misguided.

Regards,
Nenad

If we want to write a parallel algorithm that doesn't have to keep the whole matrix inside the memory at all times

MTL has some astonishing amount of shared RAM, like 200GB or so (correct me if I am terribly wrong). Is there any reason not to have the file in the memory at once?

In the thread I gave the link to in my previous post, it is said that we should be developing a universal piece of software, that can run in an optimal way on different machines and not specifically for MTL. Please check out the thread, I think you'll understand better what I meant. If anyone has a different/better understanding of this matter, I'd appreciate an explanation.