Automated Checkout Reference Implementation

Overview

Create a solution that ties together proprietary software stacks and sensor fusion technology for an automated grab-and-go experience during heightened social distancing.

Table 1

Time to Complete	Approximately 4 hours
Programming Language	Go*
Software	EdgeX Foundry* Docker* Container Portainer* Docker* Compose Google Go* programming language 1.12+ Git* GNU* make cURL* or Postman* Intel® Distribution of OpenVINO™ toolkit

 

Target System Requirements

• Ubuntu* 18.04.3 LTS
• 6th to 10th Generation Intel® Core™ processors with Iris® Pro Graphics or Intel® HD Graphics
• USB webcam

Recommended Development Kits

• IEI* TANK AI/IoT Developer Kit (Intel® Core™ Processor)
• Intel® Vision Accelerator Design
• Intel® Neural Compute Stick 2

How It Works

Learn to create a contactless shopping experience during a time of heightened social distancing due to the Covid19 pandemic. With this reference solution, explore how to use a common open middleware framework, EdgeX Foundry*, to enable multiple IoT sensors to work together and deliver an automated checkout solution that can ultimately scale for an entire store. This reference solution detects changes in inventory as well as machine telemetry, such as temperature or humidity. Easily insert your own deep learning computer vision inference models and other sensors to create your own solution.

The Automated Checkout reference implementation simulates the following three scenarios:

1. An employee stocking the cooler  

2. A customer purchasing an item 

3. The cooler requires maintenance  

Scenario 1: An Employee Stocking the Cooler

Simulation of the stocker’s activity requires these substeps:

1. Swipe badge
2. Open the cooler door
3. Close the cooler door
4. Verify that the cooler's inventory has been populated
5. Verify that the cooler's audit log shows the transaction

 

 

Scenario 2: A Customer Purchasing an Item

Simulation of a  purchase activity requires these substeps: 

1. Swipe badge
2. Open the cooler door
3. Close the cooler door
4. Verify that the cooler's inventory has been altered
5. Verify that the cooler's audit log shows the transaction
6. Verify that the cooler's ledger shows the transaction

 

 

Scenario 3: The Cooler Requires Maintenance

Simulation of the maintenance resolution requires these substeps:

1. Set the temperature to a value above the default maximum temperature threshold
2. Continue to set the temperature until the time-averaged temperature is above the default maximum temperature threshold
3. Simulate a maintenance worker swiping their badge to maintain the cooler
4. Reset the temperature back to normal
5. Verify that maintenance mode is no longer active

 

 

 

Get Started

Install the Reference Implementation

Follow the steps below to install the Reference Implementation.

NOTE: If the host system already has Docker images and containers, you might encounter errors while building the reference implementation packages. If you do encounter errors, refer to the Troubleshooting section at the end of this document before starting the reference implementation installation.

1. Open a new terminal, go to the downloaded folder and unzip the downloaded RI package.

   unzip automated_checkout.zip

    

2. Go the to automated_checkout/ directory.

   cd automated_checkout/

    

3. Change permission of the executable edgesoftware file.

   chmod 755 edgesoftware

    

4. Run the command below to install the Reference Implementation:

   ./edgesoftware install

    

5. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.
6. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module.

 

NOTE: If you encounter any issues, please refer to the Troubleshooting section at the end of this document. Installation failure logs will be available at path - /var/log/esb-cli/Fleet_Driver_Assistance_<version>/<Component_name>/install.log

Run the Application

1. Go to the working directory:

cd Automated_Checkout_<version>/Automated_Checkout/automated-checkout

NOTE: <version> is the Intel® Distribution of OpenVINO™ toolkit version downloaded.

2. Run the command below to build the automated checkout services and create local Docker images:

make build

NOTE: Add ‘sudo’ for all Docker and Docker-compose commands if user group is not added in docker. This command may take a while to run depending on your internet connection and machine specifications.

Check for Failure

If network calls fail during image build on corporate network, e.g, ‘apt-get update’ error, please refer to the Troubleshooting section at the end of this document.

Check for Success

List the Docker images that are built successfully:

docker images | grep automated-checkout

If it was successful, the results will be:

REPOSITORY

automated-checkout/as-controller-board-status automated-checkout/as-vending automated-checkout/build (latest tag) automated-checkout/ds-card-reader automated-checkout/ds-controller-board automated-checkout/ds-cv-inference automated-checkout/ms-authentication automated-checkout/ms-inventory automated-checkout/ms-ledger

 

Check for Failure

If you do not see all of the above Docker images, look through the console output for errors. Sometimes dependencies fail to resolve and must be run again. Address obvious issues and to try again, repeat step 2.

3. ​Run the command below to start the EdgeX Foundry device services and then the reference implementation suite.

make run

 

NOTE: If you encounter any issues, please refer to the Troubleshooting section at the end of this document.

​4. Make sure all the containers are all up and running, using the Docker ps following command:

docker ps --format 'table{{.Image}}\t{{.Status}}'

NOTE: Add ‘sudo’ for all Docker and Docker-compose commands if user group is not added in docker.

Table 2

IMAGE	STATUS
automated-checkout/ms-ledger:dev	Up 53 seconds
eclipse-mosquitto:1.6.3	Up 52 seconds
automated-checkout/as-vending:dev	Up 52 seconds
automated-checkout/ms-inventory:dev	Up 52 seconds
automated-checkout/ds-controller-board:dev	Up 52 seconds
automated-checkout/ms-authentication:dev	Up 55 seconds
edgexfoundry/docker-device-mqtt-go:1.2.0	Up 53 seconds
automated-checkout/ds-card-reader:dev	Up 53 seconds
automated-checkout/as-controller-board-status:dev	Up 52 seconds
edgexfoundry/docker-core-command-go:1.2.0	Up About a minute
edgexfoundry/docker-core-data-go:1.2.0	Up About a minute
edgexfoundry/docker-core-metadata-go:1.2.0	Up About a minute
edgexfoundry/docker-support-notifications-go:1.2.0	Up About a minute
edgexfoundry/docker-edgex-consul:1.2.0	Up About a minute
automated-checkout/ds-cv-inference:dev	Up 51 seconds
redis:5.0.8-alpine	Up About a minute

 

Check for failure

If any of the service displays Exited, then use the following steps to fix the proxy config issue.

• Stop Docker Containers:

make down

• Remove Docker proxy config file:

mv ~/.docker/config.json ~/.docker/config.json.backup

• Re-run the Docker image by repeating step 3.

5. Check that the services are not running in maintenance mode:

curl -X GET http://localhost:48099/maintenanceMode

6. Verify the output and check that maintenanceMode value is set to false, as shown below:

{
  "content": "{\"maintenanceMode\":false}",
  "contentType": "json",
  "statusCode": 200,
  "error": false
}

7. If maintenanceMode is set to true, run the following command to reset maintenance mode back to false:

docker-compose restart as-vending ds-controller-board as-controller-board-status

8. In a separate terminal window, watch the logs for a few Automated Checkout services, so that incoming events can be seen:

docker-compose logs -f ds-card-reader ds-controller-board ms-authentication as-vending as-controller-board-status device-mqtt

9. Each section below contains specific steps and expected output for each of the scenarios mentioned above.The following sections present the specific steps for each of the scenarios described above along with expected outputs.

NOTE: For visualization purposes, the CV inference service serves the post-processed images over http. Open http://127.0.0.1:9005 on a web browser to observe how products are being added or removed from the cooler.

 

Scenario 1: An Employee Stocking the Cooler

The sequence of events and calls to REST APIs in this section and the following sections are time sensitive. Once started, continue the sequence of commands as described, observing wait periods as indicated. Use the curl command to make REST API calls to various services.

Execute the software calls with these timing assumptions:

Table 3: Time Sensitive Calls to REST API

Action	Time Sensitive	Description	Note
Card Swipe	Yes	After a user has swiped a card, the cooler door unlocks. The user has roughly 20 seconds (configurable) to open the door before it locks again. Prepare to check the lock status and open the door commands soon after.
Check Lock	Yes	Execute immediately after card swipe.	Value of 0 indicates unlocked.
Open/Close Door	Yes	Wait three to four seconds between opening and closing the cooler door.
Inventory Discovery	Yes	It may take up to 20 to 30 seconds for inventory discovery to finish.
Inventory Check	No
Set Temperature			Run repeatedly to get an average.

 

1. Simulate the specific sequence of events performed by stock worker with REST API call to the ds-card-reader service (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"card-reader-event":"0003293374"}' http://localhost:48098/api/v1/device/name/ds-card-reader/card-reader-event

NOTE: There should not be any response message when running this EdgeX Foundry command successfully.

2. By default, the card number 0003293374 corresponds to a card in the ms-authentication/cards.json file that has the "stocker" role associated to it. JSON object for cardId 0003293374.

{
  "cardId": "0003293374",
  "roleId": 2,
  "isValid": true,
  "personId": 1,
  "createdAt": "1560815799",
  "updatedAt": "1560815799"
}

 

3. Immediately use the as-controller-board-status /status API endpoint to verify that the lock is unlocked (time sensitive):

curl -X GET http://localhost:48094/status

 

WARNING: The lock must have a reading of 0 (unlocked) before proceeding to the next steps. If a user attempts to open the door while the lock is engaged, the reference implementation assumes an error occurred and transitions to maintenance mode.

4. Open the door and close it as two separate vents. Wait three to four seconds between each event. Use the curl command to make REST API calls to the ds-controller-bard service:

5. Open the door: Make a REST API call to the ds-controller-board service to open the door (no response body expected) (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"setDoorClosed":"0"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setDoorClosed

 

6. Wait 3.75 seconds:

sleep 3.75

NOTE: Waiting around three to four seconds is necessary as the frequency of "auto-events" that relay readings between some services is set to three seconds by default.

 

7. Close the door: Make a REST API call to the ds-controller-board service to close the door (no response body expected) (time sensitive):

 

curl -X PUT -H "Content-Type: application/json" -d '{"setDoorClosed":"1"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setDoorClosed

 

8. Wait about 20-30 seconds for the inventory to be discovered by the CV inference service, and for background processing of events to occur. The time-sensitive sequence has completed.

 

9. After the completion of inventory discovery, check the inventory, audit log, and ledger (not time sensitive):

curl -X GET http://localhost:48095/inventory
curl -X GET http://localhost:48095/auditlog
curl -X GET http://localhost:48093/ledger

 

Scenario 2: A Customer Purchasing an Item

Now that the cooler's inventory has been stocked, simulate a customer swiping a card and removing one or more items from the cooler for purchase. The same time-sensitive assumptions from the stocking simulation apply here as well.

1. Use the REST command to simulate a customer swiping a badge to open the cooler (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"card-reader-event":"0003278380"}' http://localhost:48098/api/v1/device/name/ds-card-reader/card-reader-event

NOTE: There should not be any response message when running this EdgeX Foundry command successfully.

2. Use the as-controller-board-status /status API endpoint to verify that the lock is unlocked (time sensitive):

curl -X GET http://localhost:48094/status

 

3. Open and close the door as two separate events. Wait approximately three to four seconds between each event. Open the door: make a REST API call to the ds-controller-board service to open the door (no response body expected) (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"setDoorClosed":"0"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setDoorClosed

 

4. Wait 3.75 seconds:

sleep 3.75

5. Close the door: make a REST API call to the ds-controller-board service to open the door (no response body expected) (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"setDoorClosed":"1"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setDoorClosed

6. Customer simulation is complete.
7. Obtain the Inventory, ledger, and audit logs, and verify that they all show consistent information (not time sensitive, but may need to wait 20-30 seconds for background processing):

curl -X GET http://localhost:48095/inventory

 

8. In this example, the customer purchased 1 item of Gatorade and 1 item of Pringles. You can compare the unitsOnHand of each sku from the previous inventory data in step 1.

curl -X GET http://localhost:48095/auditlog
curl -X GET http://localhost:48093/ledger

 

Scenario 3: The Cooler Requires Maintenance

1. Set the temperature of the cooler to 99.00 degrees Fahrenheit. The following command will make a REST API call to the ds-controller-board service (time sensitive):

NOTE: There should not be any response message when running this EdgeX Foundry command successfully.

curl -X PUT -H "Content-Type: application/json" -d '{"setTemperature":"99.00"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setTemperature

2. Repeat this command once or twice, over a span of 15 seconds to establish an average.

3. During this waiting period, periodically check the status of maintenance mode, a REST API call to the as-vending service (no longer time sensitive):

curl -X GET http://localhost:48099/maintenanceMode

4. Expected output: The value of maintenanceMode will switch to true from false once the time-averaged temperature exceeds the maximum temperature threshold (default 83 degrees Fahrenheit).

5. A maintenance worker swipes a badge to resolve the issue. When a maintenance worker swipes the badge, maintenance mode is reset.

6. The maintenance worker has to swipe the badge twice because maintenance mode will be set to false immediately but will be immediately set to true once the next temperature reading arrives. To avoid swiping the badge twice, the temperature will be reset before swiping.

7. Set the temperature to a normal value (e.g., 45 degrees) a few times over the span of 15 seconds (minimum) (time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"setTemperature":"45.00"}' http://localhost:48082/api/v1/device/name/ds-controller-board/command/setTemperature

NOTE: There should not be any response message when running this EdgeX Foundry command successfully.

8. The maintenance worker can proceed to swipe the card, fix the cooler, and set the maintenance mode back to false. To do this, use a card that has been assigned to a maintenance worker role (not time sensitive):

curl -X PUT -H "Content-Type: application/json" -d '{"card-reader-event":"0003278385"}' http://localhost:48098/api/v1/device/name/ds-card-reader/card-reader-event

INFO: The card 0003278385 is assigned to the maintenance worker role, and a person with ID 1. Additionally, if the maintenance worker opens and closes the door, there will be a corresponding audit log entry for that door open/close event.

9. Check that maintenance mode has been reset using a familiar command (not time sensitive):

curl -X GET http://localhost:48099/maintenanceMode

Summary and Next Steps

This application successfully implements Intel® Distribution of OpenVINO^TM toolkit and provides the base components for creating a framework to run a computer vision-powered Automated Checkout. Extend the application further to provide support for different sensors and devices such as an RFID card reader or microcontroller board.

Learn More

To continue your learning, see the following guides and software resources:

• Automated Checkout Get Started Guide

• Intel® Distribution of OpenVINO^TM toolkit documentation

• EdgeX Foundry Documentation

 

Troubleshooting

Installation Failure

If host system already has Docker images and its containers running, you will have issues during the RI installation. You must stop/force stop existing containers and Images.

• To remove all stopped containers, dangling images, and unused networks:

sudo docker system prune --volumes

• To stop Docker containers:

sudo docker stop $(sudo docker ps -aq)

• To remove Docker containers:

sudo docker rm $(sudo docker ps -aq)

 

• To remove all Docker images:

sudo docker rmi -f $(sudo docker images -aq)

 

Run the Application Failure

During Step 3 of Run the application, 'make run' fails with error 'address already in use' due to same container running in different instances.

To resolve this, get the PID of the processes using the required addresses and force stop them using the commands below:

sudo netstat -tulpn | grep <address in use>
sudo kill -9 <PID of the process>

Docker Image Build Failure

If Docker image build on corporate network fails, follow the steps below.

1. Get DNS server using the command:

nmcli dev show | grep 'IP4.DNS'

2. Configure Docker to use the server. Paste the line below in   /etc/docker/daemon.json file:

{
    "dns": ["<dns-server-from-above-command>"]
}

3. Restart Docker:

sudo systemctl daemon-reload && sudo systemctl restart docker

 

*Other names and brands on this website may be claimed as the property of others. Intel is not endorsing or promoting any non-Intel brands mentioned on this website or the documentation.