DeepMask Installation Problems and Solutions (3)

Published:08/28/2017   Last Updated:08/28/2017

These are some of the problems that I encountered while installing and training a DeepMask model.

1. module coco not found

Install the lua coco package by following the steps mentioned below:

Clone coco repository:

git clone

Under coco/ run the following command:

luarocks make LuaAPI/rocks/coco-scm-1.rockspec

2. T_END at character 1 when running th train.lua

You need to reinstall torch 7 with lua 5.2 version. There is a step-by-step guide which explains how to install torch with lua 5.2. on this link.

This solution will work fine if you are training DeepMask using resnet model otherwise it could result in another error given below:


3. Failed to load function from bytecode: binary string: not a precompiled chunkWarning: Failed to load function from bytecode

You can resolve this problem by running the train.lua command with luajit instead of lua in following way:

luajit train.lua -dm /path/to/pretrained/deepmask


4. cannot open </pretrained/deepmask/model.t7> in mode r

This error was occurring while I was using the laujit version 2.0.5. You need to install luajit version 2.1.0 to resolve this issue.

You can install this by following the steps given below:

tar zxf LuaJIT-2.1.0-beta3.tar.gz
sudo make install

Add the path for luajit in your .bashrc. Usually Luajit is installed in this path:



5. DataSampler.lua:230: bad argument #3 to 'narrow'

Although I was able to convert my JSON files to t7 files correctly, while I was doing training I was getting this weird error after some initial epochs. After a lot of brainstorming I was able to resolve this problem by simply rounding off numbers in my annotation file.

6. DataSampler.lua:230: bad argument #2 to 'narrow'

This error was occuring because of presence of negative values in my segmentation key for some images. After resolving this error, I was able to run training correctly.


7. Bad argument # 2 to '?'

This was not a problem in my JSON file. I had only one category initially in my dataset, while in DeepMask, the number of categories is hardcoded to be 80. So I had to change this line of code to keep running the training correctly:

local cat,ann = torch.random(80)

Changing 80 to 1 resolved the issue.


8. Getting 'nan' as loss after few iterations, when training with custom dataset

It was because DeepMask uses default learning rate schedule. I solved this problem by changing my learning rate to 1e-06.

Product and Performance Information


Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804