AI Developer Project Part 3: Combating Distracted-Driver Behavior

Training and Evaluation of a Distracted-Driver AI Model

The second Combating Distracted-Driver Behavior article in this five-part series, Experimental Design and Data Preparation for a Distracted-Driver AI Project, covers the dataset, solution, topology and experimental design. This third article goes over implementing the solution, including preprocessing, training, and evaluation of the model.

Solution Implementation

Developers codebook

This section attempts to provide a consolidated set of instructions and commands to run for reproducing the results.

As a prerequisite, TensorFlow* needs to be installed. Please refer to software installation section of the second article of this series, Experimental Design and Data Preparation for a Distracted-Driver AI Project.

Download imgs.zip from Kaggle State Farm competition page. Unzip it. You will get “train” and “test” directory. The “train” directory contains ten folders named after their label names—for example, c0 to c9—and inside each folder are images specific to that label. The “test” directory contains an unlabeled set of around 80,000 images.

Training models using TensorFlow slim models

We referred GitHub* repository.

We followed below mentioned steps:

i. Download the required checkpoint and save to a folder
CHECKPOINT_DIR=<DIRECTORY_TO_SAVE_CHECKPOINT>
mkdir ${CHECKPOINT_DIR}
wget <CHECKPOINT_DOWNLOAD_PATH>
tar -xvf <DOWNLOADED_CHECKPOINT>
mv <EXTRACTED_CHECKPOINT> ${CHECKPOINT_DIR}
rm <DOWNLOADED_CHECKPOINT>

Example

CHECKPOINT_DIR=/tmp/checkpoints
mkdir ${CHECKPOINT_DIR}
wget http://download.tensorflow.org/models/inception_v3_2016_08_28.tar.gz
tar -xvf inception_v3_2016_08_28.tar.gz
mv inception_v3.ckpt ${CHECKPOINT_DIR}
rm inception_v3_2016_08_28.tar.gz
ii. Get code base from GitHub*

We use models directory of TensorFlow for classification using slim models. For this, clone the git repository git clone

This will give us a directory named “models.” The commands mentioned below this step will basically refer this “models” directory.

iii. Creation of validation directory

The competition site specifically mentions the fact that drivers in the training and testing datasets are not the same. The accuracy shown during the training may not be a correct indicator, since the drivers are different for the test data. We also observed that there are multiple images of the same driver for the same class. This gave us a suspicion that the model might overfit to the training data. To check this, we manually labeled a set of 1899 images from the test folder of the unzipped data. We created a validation directory with the help of this. It contained ten folders representing the ten classes from c0 to c9. Each labeled image is put into the folder which it belongs to.

iv. Creation of labels.txt

We also created a labels.txt file which is necessary for tf record generation. labels.txt file contains class labels, one in each line, from c0 to c9.

v. Tf Record generation using build_image_data.py
python models/research/inception/inception/data/build_image_data.py \
--train_directory=<PATH_TO_THE_UNZIPPED_TRAIN_DIRECTORY> \
--validation_directory=<PATH_TO_THE_CREATED_VALIDATION_DIRECTORY> \
--output_directory=<PATH_TO_SAVE_TF_RECORDS> \
--labels_file=<PATH_TO_THE_CREATED_LABELS_FILE>

Example

python models/research/inception/inception/data/build_image_data.py \
--train_directory=/home/user1/train \
--validation_directory=/home/user1/ \
--output_directory=/home/ user1/TfRecords \
--labels_file=/home/user1/labels.txt
vi. Model training/fine-tuning using transfer learning

Model training or fine-tuning is done with the help of train_image_classifier.py from models/research/slim directory. It give us an option to do transfer learning by accepting checkpoint. We could give checkpoint corresponding to the model that we want to train it on. This will define and initialize the network model from the pre-trained checkpoint of known topologies like inception v3, inception v4, vgg 16, vgg 19 and the like.

python models/research/slim/train_image_classifier.py \
--train_dir=<DIRECTORY_WHERE_CHECKPOINTS_&_TRAIN_LOGS_SAVED> \
--dataset_dir=<PATH_TO_TF_RECORDS_DIRECTORY> \
--dataset_split_name=train \
--model_name=<MODEL/TOPOLOGY_NAME> \
--checkpoint_path=<PATH_TO_PRE-TRAINED_MODEL_CHECKPOINT> \
--clone_on_cpu=True \
--max_number_of_steps=<NUMBER_OF_ITERATIONS> \
--batch_size=<BATCH_SIZE> \
--learning_rate=<LEARNING_RATE> \
--optimizer=<NAME_OF_OPTIMIZER> \
--checkpoint_exclude_scopes=<LAYERS_EXCLUDED_FROM_CHECKPOINT> \
--trainable_scopes=<LAYERS_TRAINED_ON_NEW_DATA>

Example

python models/research/slim/train_image_classifier.py \
--train_dir=/home/user1/trainedModelDir/ \
--dataset_dir=/home/user1/TfRecords \
--dataset_split_name=train \
--model_name=inception_v3 \
--checkpoint_path=/home/user1/ckpts/inception_v3.ckpt \
--clone_on_cpu=True \
--max_number_of_steps=50000 \
--batch_size=24 \
--learning_rate=0.001 \
--optimizer=rmsprop \
--checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits \
--trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits
vii. Model evaluation using eval_image_classifier.py

Once the model is trained, we need to evaluate the model to check how accurately it predicts the label of the test data. For this, we use eval_image_classifier.py from models/research/slim directory.

python models/research/slim/eval_image_classifier.py \
--alsologtostderr \
--checkpoint_path=<PATH_TO_SAVED_CHECKPOINT_FROM_TRAIN_LOGS_DIR> \
--dataset_dir=<PATH_TO_TF_RECORDS_FILE> \
--dataset_split_name=validation \
--dataset_name=<DATASET_NAME> \
--model_name=<MODE/TOPLOGY_NAME> \
--eval_dir=<DIR_TO_SAVE_EVAL_LOGS> \
--max_num_batches=<MAXIMUM_NUM_OF_BATCHES_OF_EVALUATION> \
--batch_size=<NUM_OF_VALIDATION_IMAGE_SAMPLES_TAKEN_FOR_EACH_BATCH>

Example

python models/research/slim/eval_image_classifier1.py \
--alsologtostderr \
--checkpoint_path=/home/user1/trainedModelDir/model.ckpt-5 \
--dataset_dir=/home/u5662/TfRecords \
--dataset_split_name=validation \
--dataset_name=imagenet \
--model_name=inception_v4 \
--eval_dir=/home/u5662/eval_dir \
--max_num_batches=10 \
--batch_size=200

Model training using retrain.py for inception V3

We referred the link How to Retrain an Image Classifier for New Categories for retraining the final layer of inception v3 for our data. The steps that we followed are given below:

i. Get the code set up from GitHub.

We used the code already available from GitHub. For this, we cloned the TensorFlow directory from GitHub using the command below.

git clone

This will give us a directory named TensorFlow. All the commands given below assumes a relative path w.r.t this directory.

ii. Retrain the model using transfer learning with the help of retrain.py.
time python tensorflow/tensorflow/examples/image_retraining/retrain.py \
--image_dir=<PATH_TO_UNZIPPED_TRAIN_DIRECTORY> \
--bottleneck_dir=<PATH_TO_DIR_WHERE_BOTTENECK_FILES_ARE_SAVED> \
--how_many_training_steps=<NUM_OF_ITERATIONS> \
--train_batch_size=<BATCH_SIZE> \
--learning_rate=<LEARNING_RATE> \
--output_graph=<PATH_WHERE_TRAINED_GRAPH_IS_SAVED> \
--intermediate_output_graphs_dir=<PATH_TO_SAVE_INTERMEDIATE_GRAPHS> \
--output_labels=<PATH_TO_SAVE_LABELS_FILE> \
--summaries_dir=<PATH_TO_SAVE_RETRAIN_LOGS> \
--model_dir=<PATH_TO_SAVE_PRETRAINED_MODEL_FOR_TRANSFER_LEARNING>

Example

time python tensorflow/tensorflow/examples/image_retraining/retrain.py \
--image_dir=/home/user1/train \
--bottleneck_dir=/home/user1/bottleneck_dir \
--how_many_training_steps=50000 \
--train_batch_size=24 \
--learning_rate=0.001 \
--output_graph=/home/user1/output_graph.pb \
--intermediate_output_graphs_dir=/home/user1/inter_graph \
--output_labels=/home/user1/output_labels_24.txt \
--summaries_dir=/home/user1/retrain_logs \
--model_dir=/home/user1/imagenet
iii. Creation of test directory

As pointed out earlier, we manually labelled a set of 1899 images from the test folder of the unzipped data. Those images were put into a folder. The true labels of the images in this folder are recorded in a CSV file. Please note that the validation set for TF slim model used had subfolders from c0 to c9, while in this case, we have no such subfolders.

iv. Evaluate the model using a set of test images

label_image.py in the tensorflow/tensorflow/examples/label _image directory evaluates only one image at a time. As pointed out earlier, we wanted to check if the model overfits to the training data. For this, we created a test directory as mentioned in the previous step. We wanted to get the accuracy of the model over this entire test directory. Hence we modified label_image.py file in such a way that it accepts “image_dir” instead of “image.” It is also modified in such a way that it predicts the labels for all images in the test folder and also prints the accuracy. (The modified label_image.py called label_image_modified.py file will be part of the deliverable.) Copy label_image_modified.py file to the tensorflow/tensorflow/examples/image_retraining directory. In addition to the arguments already part of the original file, we added an extra argument called test_labels_file to incorporate the path to the CSV file where true labels of the test images are recoded. This CSV file has the format (image name, true label).

Python tensorflow/tensorflow/examples/image_retraining/label_image_modified.py \
--graph=<PATH_WHERE_TRAINED_MODEL/GRAPH_IS_SAVED> \
--labels=<PATH_WHERE_LABELS_FILE_IS_SAVED> \
--output_layer=final_result:0 \
--image_dir=<PATH_TO_MANUALLY_LABELLED_TEST_DATA>
--test_labels_file=<PATH_TO_CSV_FILE_CONTAINING_TRUE_LABEL_IMAGE_MAPPING>

Example

python tensorflow/tensorflow/examples/image_retraining/label_image_modified.py \
--graph=/home/user1/output_graph.pb \
--labels=/home/user1/output_labels.txt \
--output_layer=final_result:0 \
--image_dir=/home/user1/test_dir \
--test_labels_file=/home/user1/test_labels.csv

Model evaluation using ensemble

i. Get the code set up from GitHub.

Same as section b

ii. Gather the different checkpoints trained with different topologies.

To create an ensemble, we gathered 3 checkpoints trained on inception-resnet-v2, inception v4, inception v3 with help of TensorFlow slim. These models had an accuracy of 58, 56 and 53 respectively.

iii. Convert checkpoints to .pb files.

We wanted graph (.pb) files to evaluate the models using label_image.py file. Hence we converted the checkpoint (.ckpt) files, that we gathered in the previous step, to graph files with the help of a 2 stage conversion. The first stage is done with the help of export_inference_graph.py file from models/research/slim directory. It creates an inference graph for the topology mentioned from TF Record file. The second stage is done with the help of freeze_graph.py file from tensorflow/tensorflow/python/tools directory. It creates the .pb file for evaluation using label_image.py.

python models/research/slim/export_inference_graph.py \
--model_name=<MODEL/TOPOLOGY_NAME> \
--output_file=<PATH_TO_SAVE_INFERENCE_GRAPH> \
--dataset_dir=<PATH_TO_TF_RECORDS_FILE>

python tensorflow/tensorflow/python/tools/freeze_graph.py \
--input_graph=<PATH_TO_INFERENCE_GRAPH_FILE> \
--input_checkpoint=<PATH_TO_CHECKPOINT_FILE> \
--input_binary=true \
--output_graph=<PATH_TO_SAVE_OUTPUT_FROZEN_GRAPH> \
--output_node_names=<OUTPUT_NODE_NAME_FOR_THE_CONSIDERED_TOPOLOGY>

Example

python models/research/slim/export_inference_graph.py \
--model_name=inception_v3 \
--output_file=/home/user1/inception_v3_inf_graph.pb \
--dataset_dir=/home/user1/tfRecords_ensemble

python tensorflow/tensorflow/python/tools/freeze_graph.py \
--input_graph=/home/user1/inception_v3_inf_graph.pb \
--input_checkpoint=/home/user1/trainedModelDir/model.ckpt-50000 \
--input_binary=true \
--output_graph=/home/user1/frozen_inception_v3.pb \
--output_node_names=InceptionV3/Predictions/Reshape_1
iv. Repeat the above step for all checkpoints gathered.
v. Evaluate the ensemble model.

As already discussed, label_image.py in the tensorflow/tensorflow/examples/label_image directory evaluates only one image, for a single model, at a time. We modified label_image.py file to accept “image_dir” instead of “image”. Modifications were also done to predict the labels for all the images in test directory for all models in the ensemble. Once we get the predicted labels for each image for different models, we applied different logics to get the correct prediction. This is compared against the true label and accuracy is printed. (The modified label_image.py called label_image_ensemble.py file will be part of the deliverable.) Copy the label_image_ensemble.py file to tensorflow/tensorflow/examples/label_image directory. In addition to the arguments already part of the original file, we added an extra argument called test_labels_file to incorporate path to the CSV file where true labels of the test images is recoded. This CSV file has the format (image name, true label).

python tensorflow/tensorflow/examples/label_image/label_image_ensemble.py \
--image_dir=<PATH_TO_MANUALLY_LABELLED_TEST_DIRECTORY> \
--input_layer=input \
--output_layer=<COLON_SEPARATED_SET_OF_OUTPUT_NODE_NAMES_OF_MODELS> \
--graph=<COLON_SEPARATED_SET_OF_GRAPH_FILES_OF_MODELS> \
--labels=<PATH_WHERE_LABELS_FILE_IS_SAVED> \
--input_mean=0 \
--input_std=255 \
--test_labels_file=<PATH_TO_CSV_FILE_CONTAINING_TRUE_LABEL_IMAGE_MAPPING>

Example

python tensorflow/tensorflow/examples/label_image/label_image_ensemble.py \
--image_dir=/home/user1/test_labeled \
--input_layer=input \
--output_layer=InceptionResnetV2/Logits/Predictions:InceptionV4/Logits/Predictions:InceptionV3/Predictions/Reshape_1  \
--graph=/home/user1/frozen_inception_resnet_v2.pb:/home/user1/frozen_inception_v4.pb:/home/user1/frozen_inception_v3.pb \
--labels=/home/user1/labels.txt \
--input_mean=0 \
--input_std=255 \
--test_labels_file=/home/user1/test_labels.csv

Preprocessing

i. Augmentation

We referred the link GitHub for augmentation. It provides a set of features of which we mainly used rotation, random cropping, zooming, random distortion, random erasing, skewing, shearing, and greyscale conversion.

To parallelize the above process for all classes created a Python* file with Augmentor. The file img_augment.py is part of deliverable.

ii. Black mask/random pixel mask

This kind of preprocessing identifies the approximate coordinates where driver apparel appears in the image and replace it with either black or random pixels.

Img_black_mask.py and img_random_mask.py is part of the deliverable.

iii. Color-to-grayscale conversion

We believed that greyscale.py is part of deliverable.

iv. Pad-slice-merge

For training images, we pad the image to make it square and then slice it and merge with another image. img_slice_merge.py does this. img_resize_pad.py does the padding for test images. Both of these files are part of the deliverable.

v. Color quantization

Color quantization reduces the number of distinct colors used in an image. We created four-color quantized replica images with different numbers of colors preserved. The Python file (color_quantization.py) used to create such preprocessed images is part of the deliverable.

vi. Person detection using TensorFlow object-detection API

We used common objects in context (COCO) model for detecting the driver in the image using TensorFlow object-detection API. We followed these steps:

  1. Get the code for TensorFlow  API from GitHub.

    TensorFlow models

    (TensorFlow object-detection)

  2. Edit visualization_utils.py.

    Open the Python file models/research/object_detection/utils/ visualization_utils.py and edit the function visualize_boxes_and_labels_on_image_array() in such a way that it returns the coordinate values of the object detected from the image. In the case of distracted-driver detection, only one object (the driver) will be detected for each image.

  3. Convert object_detection_tutorial.ipynb to object_detection_tutorial.py.\

    models/research/object_detection/object_detection_tutorial.ipynb is converted to object_detection_tutorial.py and modified to generate the object detected cropped image dataset.

    In object_detection_tutorial.py, the image and its coordinate values are returned from the function visualize_boxes_and_labels_on_image_array(). These coordinate values are used to crop the object detected from the image. Further the cropped image is saved and used as the input dataset for training and testing.

  4. In the Python file (object_detection_tutorial.py), there are mainly four fields to be given.

    PATH_TO_CKPT:

    Here specify the path to the frozen_inference_graph.pb of

    ssd_mobilenet_v1_coco_11_06_2017. This can be downloaded from: GitHub repository.

    PATH_TO_LABELS

    Specify the path to label map (mscoco_label_map.pbtxt). It will be here:

    models/research/object_detection/data/

    PATH_TO_IMAGES_DIR

    Specify the input directory path which contains the input images.

    OUTPUT_DIR

    Also specify the output directory path to which the cropped images will be saved.

    Once the fields are updated, run this command: python object_detection_tutorial.py

  5. Run the Python file (object_detection_tutorial.py) for both training and testing images separately.
k-Nearest neighbors algorithm

The machine learning classification implementation of the problem, with the help of k—nearest neighbors algorithm (k-NN), is wrapped in the file knn_distracted.py (part of deliverable).

Quick start to Intel® Movidius™ Neural Compute Stick

Follow the below step to quick start the Intel® Movidius™ Neural Compute Stick.

  1. The following environment set-up is required:
    • Ubuntu* 16.04 system, RPI3 Model B or Ubuntu* VirtualBox instance.
    • Intel Movidius Neural Compute Stick
    • Internet connection to download and install Intel® Movidius™ Software Development Kit (SDK).
  2. Install NCS SDK and run the below commands on a terminal window.
    mkdir -p ~/workspace
    cd ~/workspace
    git clone https://github.com/movidius/ncsdk.git
    cd ~/workspace/ncsdk
    make install
    
  3. Generate the NCS SDK acceptable meta file from the trained ckpt.

    Python 3 Inception-v3.py

  4. Compile for TensorFlow to generate the graph. The command below shows how to compile the saved session from the above code sample.
    mvNCCompile <META_FILE_PATH> -in=<INPUT_LAYER_NAME> -on=<OUTPUT_LAYER_NAME> -s12

    Example

    mvNCCompile output/inception-v1.meta -in=input -on=InceptionV1/Logits/Predictions/Reshape_1 -s12
  5. If successfully executed, a graph file will get generated.

Setup of Django* webserver

  1. System specification
    NameDescription
    Operating SystemUbuntu (VDI)
    Version

    16.04.3 LTS

    Release16.04
    CodenameXenial
  2. Get dependencies installed using the below commands:

    a. sudo apt get install redis server
    b. sudo apt get install python3 tk
    c. sudo apt-get install ffmpeg
    d. pip3 install django
    e. pip3 install channels
    f. pip3 install multiprocessing
    g. pip3 install moviepy
    h. pip3 install asgi_redis
    i. pip3 install pandas
    j. pip3 install natsort
    k. pip3 install numpy

How to run distracted-driver web application

Open the 3 terminals.

  • Terminal 1

    Start the redis-server using the below command.

    redis-server

  • Terminal 2
    1. Go to project Home folder.
    2. Start the Django server using the below command:
      python3 manage.py runserver 10.0.2.15:9876		
      open any browser					
      https://10.0.2.15:9876
      
      username : intel					
      password : intel
      
  • Terminal 3
    1. Go to the project Home folder.
    2. Start the background worker.

      python3 manage.py stream_image_reader

Next Steps

The fourth article of this Combating Distracted-Driver Behavior series, Designing and Fine-tuning a Distracted-Driver AI Model details designing and fine-tuning the project based on experimental design outcomes. Refer back to the second article for more information on Experimental Design and Data Preparation for a Distracted-Driver AI Project.

For reference on AI Developer Project: Combating Distracted-Driver Behavior ›

Join the Intel® AI Academy

Sign up for the Intel® AI Academy and access essential learning materials, community, tools and technology to boost your AI development. Apply to become an Intel® Student Ambassador and share your expertise with other student data scientists and developers.

Resources

For more complete information about compiler optimizations, see our Optimization Notice.