Hi Dmitry,
I have gathered some more information. The problem only manifests when using the MPI_THREAD_MULTIPLE setting (to allow arbitrarily multi-threaded programs) and not when the program only makes calls from a single thread.
However, I have worked around the problem by creating an MPI thread, initializing with MPI_THREAD_SERIALIZED and going into an infinite loop sending messages back and forth between the two machines as fast as possible, feeding sends from a concurrent queue (sending dummy data is no real data is available) and posting received data back to my application. This way any thread can send data by enqueuing it and incoming messages can be received in any way. The performance is great in my test code, I just have to graft the new code into my production system now...
The F# standard library creates threads indirectly but only using ordinary .NET calls. I was using asynchronous agents to send and receive data over MPI.Here's the code I was using:

let send = let agent = new MailboxProcessor<_>(fun inbox -> async { initialize()

while true do let! (buf : byte []) = inbox.Receive() if buf.Length >= maxSize then printfn "WARNING: %d-byte message is too long for MPI" buf.Length else let nativeArray = NativeInterop.PinnedArray.of_array buf let dst = if rank() = 0 then 1 else 0 Internal.send(nativeArray.Ptr, buf.Length, Internal.MPI_BYTE, dst, 0, Internal.MPI_COMM_WORLD) |> check }) agent.Start() agent.Post let receive = let buf = Array.create maxSize 0uy let nativeBuf = NativeInterop.PinnedArray.of_array buf let status = [|Internal.MPI_Status()|] let nativeStatus = NativeInterop.PinnedArray.of_array status let queue = System.Collections.Concurrent.ConcurrentQueue<_>() async { initialize()

while true do Internal.recv(nativeBuf.Ptr, buf.Length, Internal.MPI_BYTE, Internal.MPI_ANY_SOURCE, Internal.MPI_ANY_TAG, Internal.MPI_COMM_WORLD, nativeStatus.Ptr) |> check let buf = Array.sub buf 0 status.[0].count queue.Enqueue buf } |> Async.Start queue

My "send" is an asynchronous agent that serializes messages posted to it from any thread and sends them over MPI. My "receive" is an infinite loop that sits in a thread on the thread pool waiting for any message to be received. Note that this means there will be a call to "MPI_Recv" blocking one thread while another thread is calling "MPI_Send". Cheers, Jon.