Real-time Sensor Fusion for Loss Detection Reference Implementation

Overview

Real-time Sensor Fusion for Loss Detection allows you to deploy sensor fusion technology for loss detection at self-checkout and enables a more seamless experience. Use machine learning to connect different sensors, such as point-of-sale systems, weight scale sensors, cameras and RFIDs to accurately detect checkout items.

Real-time Sensor Fusion for Loss Detection and the software listed in table 1 are installed when selected as part of the Edge Insights for Retail package.

Table 1

Time to Complete Approximately 2 hours
Programming Language Google Go* version 1.12 or greater
Software

Target System Requirements

  • Ubuntu* 18.04.3 LTS (64 bit)
  • 6th to 10th Generation Intel® core™ processors with Iris® Pro graphics or Intel® HD Graphics
  • Optional (Sensors):
    • Point-of-sale hybrid scanner or scale
    • Barcode scanner
    • Point-of-sale bagging scale
    • Four USB cameras
    • RFID mat
    • Touch display

How It Works

Sensor fusion, also called basket reconciliation, is the root of the detection system. It works as follows:

  1. Sensor devices trigger events, such as detecting an RFID tag or food type.

  1. The EdgeX framework publishes events to the main application as an event message through the microservices, assembles the information to correlate events to the detected products.

  1. The data is then reconciled to make sure that it matches what is being purchased.

Irreconcilable objects might indicate suspicious purchase. A point-of-sale system integrates with either the EdgeX REST or MQTT device service to send the events to the main application. The Scaled Devices Service provided is specifically for a CAS USB scale and is a good starting point for integrating other USB scales. Alternatively, the point-of-sale bagging scale events can be sent to the EdgeX REST or MQTT device service.

For a robust, production-ready solution, you must provide an improved object detection model. The object detection model (based on the Intel® Distribution of OpenVINO™ toolkit) used by the Video Analytics Serving is an example for object detection.

After the reference implementation core components are up and running you can use Postman or the Intel-provided Event Simulator to simulate data. Postman is a collaborative platform for API development that sends RestAPI requests for simulated data.


Figure 1. Architecture Diagram

The high-level architecture diagram in Figure 1 shows the sensors and services used with the Real-time Sensor Fusion for Loss Detection reference implementation. The diagram shows the sensors and services, and how they communicate through EdgeX.

Get Started

Step 1: Install the Reference Implementation

Follow the steps below to install the Reference Implementation.

NOTE: If the host system already has Docker images and containers, you might encounter errors while building the reference implementation packages. If you do encounter errors, refer to the Troubleshooting section at the end of this document before starting the reference implementation installation.

1. Open a new terminal, go to downloaded folder and unzip the RI package

unzip real_time_sensor_fusion_for_loss_prevention.zip

2. Go to real_time_sensor_fusion_for_loss_prevention/ directory.

cd real_time_sensor_fusion_for_loss_prevention/

3. Change permission of the executable edgesoftware file.

chmod 755 edgesoftware

4. Run the command below to install the Reference Implementation

./edgesoftware install

6. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.

6. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module.

NOTE: If you encounter any issues, please refer to the Troubleshooting section at the end of this document. Installation failure logs will be available at path - /var/log/esb-cli/Real_Time_Sensor_Fusion_for_Loss_Detection_<version>/<Component_name>/install.log 

7. Go to the working directory:

cd Real_Time_Sensor_Fusion_for_Loss_Detection_<version>/Real_Time_Sensor_Fusion_for_Loss_Detection/rtsf-at-checkout-reference-design

 

NOTE: <version> is the Intel® Distribution of OpenVINO™ toolkit version downloaded.

Step 2: Build the Application

Build the real time sensor fusion for loss detection Docker images: 

sudo make docker

NOTE: This command may take a while to run depending on your internet connection and machine specifications. 

NOTE: If network calls fail during image build on a corporate network, e.g,, ‘apt-get update’ error, please refer to the Troubleshooting section at the end of this document.

Check for Success

Check for the built Docker images by run command below: 

sudo docker images

If it was successful, the results will be:

rtsf-at-checkout/event-reconciler
rtsf-at-checkout/device-scale
rtsf-at-checkout/product-lookup
rtsf-at-checkout/loss-detector
rtsf-at-checkout/rsp-controller-event-handler
rtsf-at-checkout/cv-region-of-interest

Check for Failure

If you do not see all of the above Docker image files:

  1. Look through the console output for errors as sometimes dependencies fail to resolve and must be run again.
  2. Address any obvious issues.
  3. To check the Docker images again, run following command:
    sudo docker images
  4. Return to the beginning of step 1 to try again.

Step 3: Start the Application

Run Docker Compose to start the EdgeX Device Services and the Loss Detection Services. 

sudo make run-base

NOTE: If you encounter any issues, please refer to the Troubleshooting section at the end of this document.

List the Docker images that are built successfully:

sudo docker ps --format 'table{{.Image}}\t{{.Status}}'

 

Check for Success

Your output is as follows:

Check for Failure

Only edgexfoundry/docker-core-config-seed-go:1.1.0 should display a status of Exited.

If any other service displays Exited, then use the following steps to fix the proxy config issue.

  1. Stop Docker Containers:
    sudo make down
  2. Remove Docker proxy config file:
    mv ~/.docker/config.json ~/.docker/config.json.backup
  3. Return to the beginning of Step 3: Start the Application.

Step 4: Simulate Data

All of the core components of RTSF are up and running, and you are ready to begin data simulation using an Intel-provided event simulator.

Run the Application

1. Open the Docker logs in a terminal window. This lets you make sure checkout events are processed correctly. To open the Docker logs, run:

sudo docker logs -f event-reconciler

 

Check for Success

If the above command is successful, you will see the screen below:

Check for Failure

If any connection error occurs,

  • Stop the services by running the following command:
make down

2. Install Postman: Postman will be used to send the RestAPI request for the simulated data.

3. After successful Installation, open Postman and send an HTTP GET request to http://localhost:49986/api/v1/ping to test the EdgeX Foundry rest operation. This makes sure the service is online.

Check for success

If it was successful, you see a message returned. If not, refer to the Troubleshooting section at the end of this document.

 

 

4. Initiate a transaction. For basket-open, send a POST request to http://localhost:49986/api/v1/resource/device-pos-rest/basket-open. Select raw body, enter the text below, and press send.

{
    "lane_id":"1",
    "basket_id": "abc-012345-def",
    "customer_id": "joe5",
    "employee_id": "mary1",
    "event_time":15736013930000
}

Check for success 

You will get a Response Status as “1”.  If not, refer to the Troubleshooting section at the end of this document. 

5. Scan an item. For scanned-item, send a POST request to http://localhost:49986/api/v1/resource/device-pos-rest/scanned-item with body. Select raw body, enter the text below, and press send

{
    "lane_id":"1",
    "basket_id": "abc-012345-def",
    "product_id": "00000000571111",
    "product_id_type": "UPC",
    "product_name": "Trail Mix",
    "quantity": 1,
    "quantity_unit": "EA",
    "unit_price": 5.99,
    "customer_id": "joe5",
    "employee_id": "mary1",
    "event_time":15736013940000
}

Check for success 

You will get a Response Status as “1”.  If not, refer to the Troubleshooting section at the end of this document. 

6. Prepare for payment. For payment-start, send a POST request to http://localhost:49986/api/v1/resource/device-pos-rest/payment-start with body. Select raw body, enter the text below, and press send.

{
    "lane_id":"1",
    "basket_id": "abc-012345-def",
    "customer_id": "joe5",
    "employee_id": "mary1",
    "event_time":15736013950000
}

Check for success 

You will get a Response Status as “1”.  If not, refer to the Troubleshooting section at the end of this document. 

7. Payment succeeded. For payment-success, send a POST request to http://localhost:49986/api/v1/resource/device-pos-rest/payment-success with body. Select raw body, enter the text below, and press send.

{
    "lane_id":"1",
    "basket_id": "abc-012345-def",
    "customer_id": "joe5",
    "employee_id": "mary1",
    "event_time":15736013960000
}

Check for success 

You will get a Response Status as “1”.  If not, refer to the Troubleshooting section at the end of this document. 

8. Transaction is closed. For basket-close, send a POST request to http://localhost:49986/api/v1/resource/device-pos-rest/basket-close with body. Select raw body, the text below, and press send.

{
    "lane_id":"1",
    "basket_id": "abc-012345-def",
    "customer_id": "joe5",
    "employee_id": "mary1",
    "event_time":15736013970000
}

Check for success 

You will get a Response Status as “1”.  If not, refer to the Troubleshooting section at the end of this document. 

Summary and Next Steps

You have successfully created a simulated scenario with the POS. Next, try the section, Components to build a solution that is based on this reference design, in the Real-time Sensor Fusion at Checkout Get Started Guide.

Learn More

To continue learning, see the following guides and software resources:

Troubleshooting 

Installation Failure

If host system already has Docker images and its containers running, you will have issues during the RI installation. You must stop/force stop existing containers and Images.

  • To remove all stopped containers, dangling images, and unused networks:

sudo docker system prune --volumes
  • To stop Docker containers:

sudo docker stop $(sudo docker ps -aq)
  • To remove Docker containers:

sudo docker rm $(sudo docker ps -aq)
  • To remove all Docker images:
sudo docker rmi -f $(sudo docker images -aq)

 

Run the Application Failure

During Step 3: Start the Application, 'make run' fails with error 'address already in use' due to same container running in different instances.

 

To resolve this, get the PID of the processes using the required addresses and force stop them using the commands below:

sudo netstat -tulpn | grep <address in use>
sudo kill -9 <PID of the process>

MQTT Address Error

If you get MQTT address ‘already in use’ error, get the PID of the processes using the required addresses and force stop them using the commands below:

sudo lsof -i -P -n | grep <port number>
sudo kill -9 <process id>
  • If you get 'address already in use' error, get the PID of the processes using the required addresses and force stop them using the commands below: 

sudo lsof -i -P -n | grep <port number>
sudo kill -9 <process id>
  •  Address any obvious issues, e.g., Network issues, Docker proxy issues, etc. 

Device-Rest Error

If you get a Response Status Code 404 or get no response, make sure you correctly built and ran device-rest. 

sudo netstat -tulpn | grep <address in use>
sudo kill -9 <PID of the process>
  • Stop the services by running the following command: $ make down

  • Return to Step 3: Start the Application. Do not continue until you successfully receive a message. 

Docker Image Build Failure

If Docker image build on corporate network fails, follow the steps below.

1. Get DNS server using the command:

nmcli dev show | grep 'IP4.DNS'

2. Configure Docker to use the server. Paste the line below in  /etc/docker/daemon.json file:

{
    "dns": ["<dns-server-from-above-command>"]
}

3. Restart Docker:

sudo systemctl daemon-reload && sudo systemctl restart docker

 

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804