I tried to create a model - Alexnet and Googlenet. For both cases, the creation failed. From caffe.log file in the error log, I see the following:
I0925 18:53:53.413179 222 net.cpp:500] Network initialization done.
I0925 18:53:53.449798 222 solver.cpp:122] Solver scaffolding done.
I0925 18:53:53.454358 222 caffe.cpp:329] Starting Optimization
I0925 18:53:53.454391 222 solver.cpp:495] Solving OneMoreTry
I0925 18:53:53.454398 222 solver.cpp:496] Learning Rate Policy: inv
(222): /localdisk/jenkins/mlsl-build/src/comms_ep.cpp:CommsAlloc:535: ASSERT 'ptr' FAILED: NULL pointer
Attempting to use an MPI routine after finalizing MPI
What could be this assert related to ?