Moving mouse cursor using hand gesture using C#

Moving mouse cursor using hand gesture using C#

Hello Everyone,

I am trying to do something like, my fingers or hand will be acting like mouse. Hand will control mouse cursor and hand openness state to thumb movement will act like a click of left mouse button.

I kind of done it but the code is very jittery, its not having smooth movement. Mouse cursor will not move smooth.

var gesture = QueryGesture();
PXCMGesture.GeoNode[] nodeData = new PXCMGesture.GeoNode[6];
PXCMGesture.GeoNode singleNode;
searchPattern = PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY;
var status = gesture.QueryNodeData(0, GetSearchPattern(), out singleNode);
if (status >= pxcmStatus.PXCM_STATUS_NO_ERROR)
{
if (OnHandMoved != null)
{
OnHandMoved(singleNode.positionImage.x, singleNode.positionImage.y);
}

and below is my event hook up

Dispatcher.Invoke((Action)delegate()
{
double width=System.Windows.SystemParameters.PrimaryScreenWidth;
double height = System.Windows.SystemParameters.PrimaryScreenHeight;
double tranform = width / 320;
double newwidth = x * tranform;
double newheight = y * tranform;

SetPosition((int)newwidth, (int)newheight);

}

private void SetPosition(int a, int b)
{
SetCursorPos(a, b);
}

[DllImport("User32.dll")]
private static extern bool SetCursorPos(int X, int Y);

Have anyone done this before, how to make it more smooth and provide perfect control.

How to simultanously captual thumb movement for notifying click.

Thanks in advance.

18 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hi Shyam,

noise on image processing algorithms is very common causing the values to change rapidly, a possible solution is to average the values you get from the SDK, this can be done in the following way:

First we would need to create an int array

            int[] xPosition = new int[10];

Then create a method to populate the array

            if (index > 10) index = 0;

            xPosition[index] = a;
            index++;

Then create a second method to get the average

            int average = 0;

            foreach (int item in xPosition)
            {
                average += item.x;
            }

            average = average / 10;

In this example the values are smoothed by 10, I'm sure that movement will be smoother (although more slower too)

Hope it helps!

But I seen lots of intel demo apps, where mouse cursor is moving so smoothly, what is different they are doing. I will keep trying for it.

kalman filter can really smooth your points
I have tried it and it really works
But I am using emguCV (has kalman filter library) and not Intel SDK

Thanks for information, can you give more detials about it how you did it and details about emguCV library.

What I wanted is to use intel sdk and use hand as mouse pointer. In all intel demo they have shown that.

Hi, I have used emguCV too, Intel SDK and emguCV can work together fine. You would use Intel SDK for the Gesture recognition and emguCV to smooth the points with the Kalman Filter.

Look at this wiki (http://www.emgu.com/wiki/index.php/Kalman_Filter) ,it explains how to use the function. There is a Mouse Live Data example, if I'm not wrong it is about smoothing the cursor position so it will help you a lot.

I tried it again today, and noticed one thing here the value retured by sdk will be within range of 320*240. I think it need to be converted in actual coordinate system plus an filter like kalman which alex have specified is required to get it smoothly moving.

Anyone tried it?

-Shyam

Han anyone gave a shot to this? Seems like couple of guys trying but no one found any solution yet.

Hi Shyam,

Did you manage to get any usable values for openness?  All i got is ZERO.

Thanks!

Hi

I am using a similar approach for moving the mouse cursor, but in C++. For converting to screen coordinates you need to get the screen size (W, H), then convert to screen coordinates using the frame size (fW = 320, fH = 240):

x = x*W/fW

y = y*H/fH

However, there is also an offset that you need to correct since the frame upper left corner is not at screen coordinates (0,0). I do this correction manually, with some predefined constants.

However, I am facing the problem of variable frame rate, sometimes the rate at which the frames come is too slow and also the mouse pointer moves much slower than usually. This does not happes all the times, sometimes I get smooth mouse movements (the mouse position is recomputed each time a frame becomes available). Does anyone noticed a similar behavior ?

regards

Bogdan

Hello Bogdan,

I had figured it out long back. But still I have not added any costant to fine tune it, may be that's why it does not move to corners of screen properly. What constants you have used?

Yes the same issue I have also faced even I have also tried to smooth the values.

-Shyam

I have found those constants empirically, by moving the cursor to the upper left corner.

Is there any workaround for the slowness in the PC SDK ? I notice the same behavior with the provided examples.

Something like this:

static int scrW = Screen.PrimaryScreen.Bounds.Width;

static int scrH = Screen.PrimaryScreen.Bounds.Height;

private static double xPos = 0;

private static double yPos = 0;

...

PXCMImage[] images = new PXCMImage[PXCMCapture.VideoStream.STREAM_LIMIT];

...

PXCMGesture.GeoNode data_mouse;

sts = gesture.QueryNodeData(0, PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY | PXCMGesture.GeoNode.Label.LABEL_HAND_FINGERTIP, out data_mouse);

if (sts >= pxcmStatus.PXCM_STATUS_NO_ERROR)

{

uint camH = images[0].imageInfo.height;

uint camW = images[0].imageInfo.width;

yPos = data_mouse.positionImage.y * (scrH / camH);

xPos = scrW - (data_mouse.positionImage.x) * (scrW / camW);

}

And then values xPos, yPos use in Kalman Filter.

Thanks Evgeniy L for code. I did similar just the difference is I applied filter then I convert into another coordinate system but yours is first conversion then filter which is making it more efficient.

I tested the code after applying filter to it. But still cursor is not moving to all the 4 corners of screen. I think as @Bogdan already guided us we need to improve it using some constants.

Hi, @Shyam

here is a recomendation that may help to reach the 4 points of the screen: Once that you have the position of the hand in the 320, 240 coordinate system compute the difference of this point to the central point (160, 120), lets call this difference x_difference and y_difference so that:

x_difference = data_mouse.positionImage.x - 160;
y_difference = data_mouse.positionImage.y - 120;

This value has to be multiplied by a constant like the one you already have so that:

x_difference = x_difference * X_MULTIPLIER;
y_difference = y_difference * Y_MULTIPLIER;

And then make the cursor position equal to the center of the screen plus this difference so that:

Cursor.Position.X = (Screen.PrimaryScreen.Bounds.Width / 2) + x_difference;
Cursor.Position.Y = (Screen.PrimaryScreen.Bounds.Height / 2) + y_difference;

With this formula, which based on the center of the screen it is granted that you will reach the 4 points of the screens just by playing a little with the values X_MULTIPLIER and Y_MULTIPLIER. This is very different from the old approach since the "center" is on 0, 0 and you may never reach the 4 points unless you patch it with an offset.

Let me know if it worked.

Little comment about that:

Citazione:

del Rio ha scritto:

x_difference = data_mouse.positionImage.x - 160;

y_difference = data_mouse.positionImage.y - 120;

I think that you must not use constants. All values must be callculated. It will be useful if your application use camera with another resolution

Like this(from my code above):

Citazione:

PXCMImage[] images = new PXCMImage[PXCMCapture.VideoStream.STREAM_LIMIT];

uint camH = images[0].imageInfo.height;
uint camW = images[0].imageInfo.width;

x_difference = data_mouse.positionImage.x - (camW/2);

y_difference = data_mouse.positionImage.y - (camH/2);

I hope this will be helpful.

Hello,

When i run the gesture example, i am trying to move the mouse using hand gestures, but the mouse only moves in the top left portion of my screen although i have selected scale (radio button) so that it scales across the screen.

Every time i query node data i set Cursor postion as node[5].positionImage.x,node[5].positionImage.y

Can anyone please help me!! ??

Thanks in advance

Hello,

When i run the gesture example, i am trying to move the mouse using hand gestures, but the mouse only moves in the top left portion of my screen although i have selected scale (radio button) so that it scales across the screen.

Every time i query node data i set Cursor postion as node[5].positionImage.x,node[5].positionImage.y

Can anyone please help me!! ??

Thanks in advance

Leave a Comment

Please sign in to add a comment. Not a member? Join today