Edge Data Processing POC with the UP Squared* Board

Intro

With the amount of continuously generated data on the rise, the cost to upload and store that data in the cloud is increasing. Data is being gathered faster than it is stored and immediate action is often required. Sending all the data to the cloud can result in latency and presents risks when internet connectivity is intermittent. Edge computing involves processing data locally for immediate analysis and decisions. Hence it can help reduce network load by sending only critical, processed data to the cloud. This proof of concept (POC) will explore edge computing in a digital signage scenario where the display is recording the number of people viewing the display throughout the day. OpenCV will be used to search the camera feed for faces and Dash* by Plotly* to graph out the results in real time without the cloud. 

Up Squared* Board

Figure 1. UP Squared* Board

Set-up

Hardware

The POC uses the UP Squared* board, a high performance maker board with an Apollo Lake processor, with Ubuntu* 16.04 LTS as the OS. For more information on the UP Squared board, please visit: http://www.up-board.org/upsquared/.

The camera used is the Logitech* C270 Webcam with HD 720P which connects to the board by USB.

Software

OpenCV needs to be installed on the UP Squared board to process the images. Reference here for instructions on how to install it on Ubuntu 16.04.

The processed data will be graphed with Dash by Plotly to create a web application in Python*. For more information on Dash: https://plot.ly/dash/

To install Plotly, Dash, and the needed dependencies:

pip install dash==0.19.0  # The core dash backend
pip install dash-renderer==0.11.1  # The dash front-end
pip install dash-html-components==0.8.0  # HTML components
pip install dash-core-components==0.14.0  # Supercharged components
pip install plotly --upgrade  # Plotly graphing library used in examples

#dependencies
pip install pandas
pip install flask
apt get install squlite3 libsqlite3-dev

Face Detection

Face detection is done in Python with OpenCV using the Haar Cascades front face alt algorithm. Each found face is individually tracked in the frame so that each viewer can be counted. This data is recorded to a sqlite database with their face ID and timestamps. Possible expansions to this would be doing facial recognition or using other algorithms to detect demographics of viewers.

The code, seen below, will create and connect to sqlite3 to initialize the database faces.db. The tables inside the database can only be created the first time the code is run, hence it is inside a try clause. OpenCV will then connect to the camera and loop over the feed looking for faces. When it finds a face or faces, it will write send the array of face rectangles to the face_tracking method which will which give each face an ID and then track it based on position. Each face class object stores its current x and y location as well as the time it was last seen and its face ID. If the face disappears for more than 2 seconds, it is aged out of the array and assumed the person moved on or looked away for too long; this will also help with any in-between frames where OpenCV might fail to detect the face where there really is one. This data is then written to the database as the face ID and the timestamp it was seen and to another table is inputted the total number of viewer faces at that timestamp. 

Data in the visitorFace table with time stamps and ID

Figure 2. Data in the visitorFace table with time stamps and ID

#!/usr/bin/env python

from __future__ import print_function

# Import Python modules
import numpy as np
import cv2
import Face
import sys
import os
import pandas as pd
import sqlite3
from datetime import datetime
import math

#array to store visitor faces in
facesArray = []

#connect to sqlite database
conn = sqlite3.connect('/home/upsquare/Desktop/dash-wind-streaming-master/Data/faces.db')
c = conn.cursor()

# Create tables
try:
    c.execute('''CREATE TABLE faces
                (date timestamp, time timestamp, date_time timestamp, numFaces INT)''')
    c.execute('''CREATE TABLE visitorFace
                (date_time timestamp, faceID INT)''')
except:
	print('tables already exist')

#find the last visitor faceID to start counting from
df = pd.read_sql_query('select MAX(faceID) FROM visitorFace', conn)
lastFid = df.iloc[0]['MAX(faceID)']
if lastFid is None:
	fid=1
else:
	fid = 1 + int(df.iloc[0]['MAX(faceID)'])

try:
    # Checks to see if OpenCV can be found
    ocv = os.getenv("OPENCV_DIR")
    print(ocv)
except KeyError:
    print('Cannot find OpenCV')


# Setup Classifiers
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

#draw rectangles around faces
def draw_detections(img, rects, thickness = 2):
    for x, y, w, h in rects:
        pad_w, pad_h = int(0.15*w), int(0.05*h)
        cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), thickness)

#track visitor faces based on their location
def face_tracking(rects):
	global fid
	global facesArray
	numFaces= len(rects)
	#age faces out when no faces are detected
	if len(rects) == 0:
		for index, f in enumerate(facesArray):
			if((datetime.now()-f.getLastSeen()).seconds > 2):
				print("[INFO] face removed " + str(f.getId()))
				facesArray.pop(index)
	for x, y, w, h in rects:
		new = True
		xCenter = x + w/2
           	yCenter = y + h/2
		for index, f in enumerate(facesArray):
			#age the face out
			if((datetime.now()-f.getLastSeen()).seconds > 2):
				print("[INFO] face removed " + str(f.getId()))
				facesArray.pop(index)
				new = False
				break
			dist = math.sqrt((xCenter - f.getX())**2 + (yCenter - f.getY())**2)
			#found an update to existing face
			if dist <= w/4 and dist <= h/4:
				new = False
				f.updateCoords(xCenter,yCenter,datetime.now())
				c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId())+")")	
				break		
		#add a new face	
		if new == True:
			print("[INFO] new face " + str(fid))
			f = Face.Face(fid, xCenter, yCenter, datetime.now())
			facesArray.append(f)
			fid += 1
			c.execute("INSERT INTO visitorFace VALUES (strftime('%Y-%m-%d %H:%M:%f'),"+ str(f.getId()) +")")	
	print(len(facesArray))
	c.execute("INSERT INTO faces VALUES (DATE('now'),strftime('%H:%M:%f'),strftime('%Y-%m-%d %H:%M:%f'), "+ str(len(facesArray))+")")

try:
    # Initialize Default Camera
    webcam = cv2.VideoCapture(0)
    # Check if Camera initialized correctly
    success = webcam.isOpened()
    if success == True:
        print('Grabbing Camera ..')
    elif success == False:
        print('Error: Camera could not be opened')

    while(True):
        # Read each frame in video stream
        ret, frame = webcam.read()
        # Perform operations on the frame here
        # First convert to Grayscale 
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        # Next run filters
        gray = cv2.equalizeHist(gray)
        faces = face_cascade.detectMultiScale(gray, 1.3, 5, minSize=(30, 30), flags=cv2.CASCADE_SCALE_IMAGE)
        print('Number of faces detected: ' + str(len(faces)))
	#send the faces to be tracked for visitorFace faces
	face_tracking(faces)

	#draw on the cv viewing window
        out = frame.copy()
	draw_detections(out,faces)
        cv2.imshow('Facetracker', out)

	#commit the sqlite insertions to the database
        conn.commit()

        # Wait for Esc Key to quit
        if cv2.waitKey(5) == 27:
            break
    # Release all resources used
    webcam.release()
    cv2.destroyAllWindows()
    
except cv2.error as e:
    print('Please correct OpenCV Error')

Code 1. faceCounting.py code for detecting faces

Below is the Face class for each visitor face in the facesArray.

class Face:
    path = []
    def __init__(self, id, x, y, lastSeen):
        self.id = id
        self.x = x
        self.y = y
        self.lastSeen = lastSeen	
    def getId(self):
        return self.id
    def getX(self):
        return self.x
    def getY(self):
        return self.y
    def getLastSeen(self):
        return self.lastSeen
    def updateCoords(self, newX, newY, newLastSeen):
        self.x = newX
        self.y = newY
	self.lastSeen = newLastSeen

Code 2. face.py for Face Class

Processing and Graphing

To visualize the data after doing some data processing on it, Dash will be used to create a graph that will update itself as new data comes in. The graph will also be in a web application running at localhost:8050. This way insights and information can be seen directly at the edge.

In addition to graphing the number of faces data, the data will be smoothed using Pandas exponentially weighted moving average (EWMA) method to create a less jagged line. 

    #query the datetime and the number of faces seen at that time
    df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #do a EWMA mean to smooth the data
    df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()

Code 3. Code snippet to query the number of faces and do EWMA smoothing

Face Counting Data graph and EWMA smoothed data

Figure 3. Face Counting Data graph and EWMA smoothed data

From the visitorFace table, the face IDs and timestamps will be used to calculate the time someone looked at the display, sometimes referred to as dwelling time. This type of data could be monetized to drive revenue of ads at peak traffic times and report ad impact/reach. The dwelling time is calculated by getting the first and last timestamp for each face ID and then finding the difference between them. 

#query the datetime and the number of faces seen at that time, we need this for the next query
    dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #calculate the viewing time of each visitor face
    #query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
    #taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
		, con)
    #calculate the seconds a visitor face viewed the display
    df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.seconds

Code 4. Code snippet to query the viewing time for each visitor face

Scatter plot of each visitor face viewing time

Figure 4. Scatter plot of each visitor face viewing time

The Dash graph application itself has a few key pieces: the page layout, the callback for the update interval, and the graph components. The layout for this application has four main graphs with four headers: Face Counting Data, Viewing Time Data, Daily Face Count, and Hourly Face Count. The interval will call the callback every second to pull the last 1000 data entries from the sqlite database to graph, hence the graph will update itself as new data is added to database. For the graph components, each line or scatter dot is called a trace and the overall graph’s axis’s values are defined.

from __future__ import print_function
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, Event
import plotly.plotly as py
from plotly.graph_objs import *
from flask import Flask
import numpy as np
import pandas as pd
import os
import sqlite3
import datetime as dt
from datetime import timedelta

#number of most recent datapoints to grab
tail = 1000

app = dash.Dash()

#dash page layout
app.layout = html.Div([
    html.Div([
        html.H2("Face Counting Data")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces'),
        ]),
        dcc.Interval(id='face-count-update', interval=1000),
    ]),
    html.Div([
        html.H2("Viewing Time Data")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='viewing-data'),
        ]),
        dcc.Interval(id='viewing-data-update', interval=1000),
    ]),
    html.Div([
        html.H2("Daily and Hourly Face Count")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces-daily'),
        ]),
        dcc.Interval(id='face-count-daily-update', interval=1000),
    ]),
    html.Div([
        html.H2("Hourly Face Count")
    ]),
    html.Div([
        html.Div([
            dcc.Graph(id='num-faces-hourly'),
        ]),
        dcc.Interval(id='face-count-hourly-update', interval=1000),
    ]),
], style={'padding': '0px 10px 15px 10px',
          'marginLeft': 'auto', 'marginRight': 'auto', "width": "1200px",
          'boxShadow': '0px 0px 5px 5px rgba(204,204,204,0.4)'})



#callback method for face counting data graph interval
@app.callback(Output('num-faces', 'figure'), [],
              [],
              [Event('face-count-update', 'interval')])
def gen_num_faces():
    con = sqlite3.connect("./Data/faces.db")
    #query the datetime and the number of faces seen at that time
    df = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #do a EWMA mean to smooth the data
    df['EWMA']= df['numFaces'].tail(tail+100).ewm(span=200).mean()

    trace = Scatter(
	x=df['date_time'].tail(tail),
        y=df['numFaces'].tail(tail),
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines',
        showlegend=False,
        name= '# Faces'
    )

    trace2 = Scatter(
	x=df['date_time'].tail(tail),
        y=df['EWMA'].tail(tail),
        line=Line(
            color='#FF4500'
        ),
        hoverinfo='x+y',
        mode='lines',
        showlegend=False,
        name= 'EWMA'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(df['date_time'].tail(tail)),
                   max(df['date_time'].tail(tail))],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(8,10),
            title='Date Time'
        ),
        yaxis=dict(
            range=[0,
                   max(df['numFaces'].tail(tail))],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,max(df['numFaces'].tail(tail))),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace,trace2], layout=layout)


#callback method for viewing time data graph interval
@app.callback(Output('viewing-data', 'figure'), [],
              [],
              [Event('viewing-data-update', 'interval')])
def gen_viewing_data():
    con = sqlite3.connect("./Data/faces.db")
    #query the datetime and the number of faces seen at that time, we need this for the next query
    dfFaces = pd.read_sql_query('SELECT numFaces, date_time from faces;'
                            , con)
    #calculate the viewing time of each visitor face
    #query the first date time and last date time that a visitor face was seen from the tail of the dfFaces data
    #taking the data from the tail will make the Viewing Time Data graph and the Face Counting Data Graph line up
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from (select * from visitorFace Where date_time > "%s") group by faceID;""" %(min(dfFaces['date_time'].tail(tail)))
		, con)
    #calculate the seconds a visitor face viewed the display
    df['VD']= (pd.to_datetime(df['Last_Seen']) - pd.to_datetime(df['First_Seen'])).dt.seconds

    #declare the traces array to hold a trace for each visitor face
    traces = []
    for i in range(len(df.index)):
        traces.append(Scatter(
            x= [df['First_Seen'][i],df['Last_Seen'][i]],
            y= [i],           
            hoverinfo='name',
            mode='markers',
            opacity=0.7,
            marker={
                'size':df['VD'][i],
                'line': {'width':0.5,'color':'white'}
            },
            showlegend=False,
            name= df['VD'][i]
        ))
    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfFaces['date_time'].tail(tail)),
                   max(dfFaces['date_time'].tail(tail))],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(8,10),
            title='Date Time'
        ),
        yaxis=dict(
            range=[0,len(df.index)],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,len(df.index)/4),
	    title='Viewer'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=traces, layout=layout)


#callback method for daily face count graph interval
@app.callback(Output('num-faces-daily', 'figure'), [],
              [],
              [Event('face-count-daily-update', 'interval')])
def gen_num_faces_daily():
    weekAgo = dt.datetime.today() - timedelta(days=6)

    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per day
    dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')

    trace = Scatter(
	x=dfDaily['First_Seen'],
        y=dfDaily['counts'],
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines',
        name= 'Daily'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfDaily['First_Seen']),
                   max(dfDaily['First_Seen'])],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=max(2,len(dfDaily.index)),
            title='Date Time'
        ),
        yaxis=dict(
            range=[min(0, min(dfDaily['counts'])),
                   max(dfDaily['counts'])],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(4,len(dfDaily.index)),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace], layout=layout)

#callback method for hourly face count graph interval
@app.callback(Output('num-faces-hourly', 'figure'), [],
              [],
              [Event('face-count-hourly-update', 'interval')])
def gen_num_faces_hourly():
    weekAgo = dt.datetime.today() - timedelta(days=6)
    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per hour
    dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')


    trace = Scatter(
	x=dfHourly['First_Seen'],
        y=dfHourly['counts'],
        line=Line(
            color='#42C4F7'
        ),
        hoverinfo='x+y',
        mode='lines'
    )

    layout = Layout(
        height=600,
        xaxis=dict(
            range=[min(dfHourly['First_Seen']),
                   max(dfHourly['First_Seen'])],
            showgrid=True,
            showline=True,
            zeroline=False,
            fixedrange=True,
            nticks=14,
            title='Date Time'
        ),
        yaxis=dict(
            range=[min(0, min(dfHourly['counts'])),
                   max(dfHourly['counts'])],
            showline=True,
            fixedrange=True,
            zeroline=False,
            nticks=max(2,max(dfHourly['counts'])/10),
	    title='Faces'
        ),
        margin=Margin(
            t=45,
            l=50,
            r=50,
            b=100
        )
    )

    return Figure(data=[trace], layout=layout)

external_css = ["https://cdnjs.cloudflare.com/ajax/libs/skeleton/2.0.4/skeleton.min.css",
                "https://fonts.googleapis.com/css?family=Raleway:400,400i,700,700i",
                "https://fonts.googleapis.com/css?family=Product+Sans:400,400i,700,700i"]


for css in external_css:
    app.css.append_css({"external_url": css})

if __name__ == '__main__':
    app.run_server()

Code 5. app.py code for Dash by Plotly web graph

The above graphs give a very close up view of the traffic in front of the display. To get a higher level picture of what is going on, let’s arrange the data and look at how many visitor faces per day and per hour from the last week. 

weekAgo = dt.datetime.today() - timedelta(days=6)

    con = sqlite3.connect("./Data/faces.db")
    #query the first date time and last date time that a visitor face was seen
    df = pd.read_sql_query("""select faceID, MIN(date_time) as First_Seen , MAX(date_time) as Last_Seen 
		from visitorFace where date_time >= "%s" group by faceID;""" %weekAgo , con)
    #reset the index to do a groupby
    df= df.set_index(['First_Seen'])
    df.index = pd.to_datetime(df.index)
    #group the visitor faces by day and record the number of faces per day
    dfDaily = df.groupby(pd.TimeGrouper('D')).size().reset_index(name='counts')
   dfHourly = df.groupby(pd.TimeGrouper('H')).size().reset_index(name='counts')

Code 6. Code snippet to query face count daily and hourly

Daily Face Count graph

Figure 5. Daily Face Count graph

Hourly Face Count Graph

Figure 6. Hourly Face Count Graph

Summary

So concludes this POC of gathering and analyzing data at the edge. Running the face tracking application and the Dash application at the same time will show data graphing in real time. From here any of the processed data is ready to send to the cloud to store for long term.

About the author

Whitney Foster is a software engineer at Intel in the Software Solutions Group working on scale enabling projects for Internet of Things.

UP Squared*

Notices

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.

Intel, the Intel logo, and Intel RealSense are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others

© 2017 Intel Corporation.

Для получения подробной информации о возможностях оптимизации компилятора обратитесь к нашему Уведомлению об оптимизации.
Возможность комментирования русскоязычного контента была отключена. Узнать подробнее.