Ultimate Coder - A zig and a zag

Ziggy plays guitar.

Well here we are,  nearing the end of the Perceptual Challenge. There's not much longer to go with the competition, so it's time to start locking things down. and really starting to test the application. To that end, I've been concentrating on adding some gestures in to trigger the gesture creation, and doing some of the standard application housekeeping work such as creating an installer project.

The big news this week is that I haven't actually changed the interface around much. It's fairly locked down now, but I have added that little bit of extra Ultrabook support in. A little while back, I wrote some code for a previous Windows 8 desktop application, and I leveraged the WinRT support for Ultrabook features. I've brought features of that code base in to Huda, and I'm using it to identify when the Ultrabook orientation is flipped. This gives me the ability to switch the UI around so that the application is maximised in tablet mode. This is incredibly simple to achieve by hooking in to the DisplayProperties.OrientationChanged event. When the orientation changes, the UI reacts to this by identifying whether or not the tablet is a desktop or tablet like this:

var orientation = DisplayProperties.CurrentOrientation;
TabletMode mode = TabletMode.Desktop;
if (orientation == DisplayOrientations.LandscapeFlipped || 
  orientation == DisplayOrientations.PortraitFlipped)
{
  mode = TabletMode.Tablet;
}

When I change the UI around, if it's changing to a tablet application, I store the bounds of the application and then maximise the UI; I also set the WindowState so that it can only maximised or minimised. These settings are reapplied to the interface when the user flips it back from tablet to desktop mode.

Okay, I lied, there is one big change

Lee threw down the challenge, and I decided to resurrect the code. Yup. Voice recognition is back in. Fortunately, adding it back was as simple as adding this class

using AForge.Imaging.Filters;
using GalaSoft.MvvmLight.Ioc;
using Goldlight.Perceptual.Sdk;
using Huda.Transformations;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using Huda.Messages;
using GalaSoft.MvvmLight.Messaging;
using System.Collections.Concurrent;
 
namespace Huda.Model
{
    /// <summary>
    /// This class is used to manage the interactions between different commands and the operations.
    /// </summary>
    public sealed class VoiceRecognitionManager
    {
        private ConcurrentDictionary<string, StandardVoiceCommand> commands = new ConcurrentDictionary<string, StandardVoiceCommand>();
        private AppliedFiltersModel model;
        private readonly object SyncLock = new object();
        private string photo;
 
        public VoiceRecognitionManager()
        {
            Messenger.Default.Register<PhotoSelectedMessage>(this, PhotoSelected);
        }
 
        private void PhotoSelected(PhotoSelectedMessage msg)
        {
            photo = msg.Path;
        }
 
        void Voice_VoiceRecognized(object sender, Goldlight.Perceptual.Sdk.Events.VoiceEventArgs e)
        {
            string phrase = e.RecognizedPhrase;
            foreach (var command in commands)
            {
                foreach (string say in command.Value.Commands)
                {
                    if (string.Compare(phrase, say, true) == 0)
                    {
                        Application.Current.Dispatcher.Invoke(command.Value.Task, System.Windows.Threading.DispatcherPriority.Normal);
                    }
                }
            }
        }
 
        internal void EnableVoiceRecognition()
        {
 
            List<string> grammar = new List<string>();
            foreach (var command in commands)
            {
                foreach (string say in command.Value.Commands)
                {
                    if (string.IsNullOrWhiteSpace(say)) continue;
                    grammar.Add(say);
                }
            }
 
            PipelineManager.Instance.RunVoicePipeline(grammar);
            PipelineManager.Instance.Voice.VoiceRecognized += Voice_VoiceRecognized;
        }
 
        internal void AddFilters()
        {
            model = SimpleIoc.Default.GetInstance<AppliedFiltersModel>();
 
            foreach (var filter in model.AvailableFilters)
            {
                Add(string.Format("Add the {0} filter", filter.Name), filter.Name, filter);
                Add(string.Format("Add the {0} filter", filter.Name), string.Format("Add {0}", filter.Name), filter);
            }
        }
 
        private void Add(string name, string command, IImageTransform xform)
        {
            Add(name, command, () =>
            {
                model.ApplyFilter(xform);
            });
        }
 
        private void Add(string name, string command, Action task)
        {
            if (!commands.ContainsKey(name))
            {
                StandardVoiceCommand voice = new StandardVoiceCommand();
                voice.Task = task;
                voice.Name = name;
                commands.TryAdd(name, voice);
            }
 
            commands[name].Commands.Add(command);
        }
    }
 
    public sealed class StandardVoiceCommand
    {
        public StandardVoiceCommand()
        {
            Commands = new List<string>();
        }
 
        public Action Task { get; set; }
        public string Name { get; set; }
        public List<string> Commands { get; set; }
    } 
}

This gave me the ability to add filters in, so that you can say "Sepia" to add the Sepia filter (or "Add Sepia" if you like). All I needed to do, then, was add these commands into the voice pipeline:

using Goldlight.Perceptual.Sdk.Events;
using Goldlight.Windows8.Mvvm;
using System;
using System.Collections.Generic;
using System.Diagnostics;
namespace Goldlight.Perceptual.Sdk
{
    /// <summary>
    /// Manages the whole voice pipeline.
    /// </summary>
    public class VoicePipeline : AsyncPipelineBase
    {
        private WeakEvent<VoiceEventArgs> voiceRecognized = new WeakEvent<VoiceEventArgs>();
        private List<string> commands;
 
        /// <summary>
        /// Event raised when the voice data has been recognized.
        /// </summary>
        public event EventHandler<VoiceEventArgs> VoiceRecognized
        {
            add { voiceRecognized.Add(value); }
            remove { voiceRecognized.Remove(value); }
        }
 
        /// <summary>
        /// Instantiates a new instance of <see cref="VoicePipeline"/>.
        /// </summary>
        public VoicePipeline()
            : base()
        {
            EnableVoiceRecognition();
        }
 
        public VoicePipeline(List<string> commands) : base()
        {
            EnableVoiceRecognition();
 
            this.commands = commands;
            SetVoiceDictation();
 
            string[] cmd = commands.ToArray();
 
            this.SetVoiceCommands(cmd);
        }
 
        public override void OnRecognized(ref PXCMVoiceRecognition.Recognition data)
        {
            var handler = voiceRecognized;
 
            if (data.label >= 0)
            {
                if (handler != null)
                {
                    handler.Invoke(new VoiceEventArgs(commands[data.label]));
                }
            }
            base.OnRecognized(ref data);
        }
    } 
}

I'm shaking all over

I decided that I'd like to use a variation of the shake the Ultrabook to trigger the blur effect. To that end, I'd create a gesture that was a swipe of the hand to the right, and then a swipe of the hand back to the left. As long as the gesture takes less than two seconds, it will be added in. I'm going to show the code below and then explain how it all works.

using Goldlight.Windows8.Mvvm;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
namespace Goldlight.Perceptual.Sdk
{
    public class ShakeBlurGesture : GestureBase
    {
        private WeakEvent<EventArgs> shakeEvent = new WeakEvent<EventArgs>();
 
        public event EventHandler<EventArgs> ShakeEvent
        {
            add { shakeEvent.Add(value); }
            remove { shakeEvent.Remove(value); }
        }
        public override void EvaluateGesture(GestureItem gesture)
        {
            gestures.Add(gesture);
 
            var allGestures = (from p in gestures
                               where p.GestureCreated > DateTime.Now.Subtract(TimeSpan.FromSeconds(2))
                               select p).ToList();
            // RIght, move over the list
            IterateGestures(allGestures);
 
            // Finally, remove older entries the list.
            RemoveGesturesCreatedBefore(allGestures.First());
        }
 
        private void IterateGestures(IEnumerable<GestureItem> gestures)
        {
            // Right, let's find the point at which the gesture turns
            var maxGesture = (from p in gestures
                              where p.X == gestures.Max(g => g.X)
                              select p).First();
            var minGesture = (from p in gestures
                              where p.GestureCreated > maxGesture.GestureCreated
                              && p.X == gestures.Min(g => g.X)
                              select p).FirstOrDefault();
            if (minGesture != null && minGesture.X < maxGesture.X - 100)
            {
                // I've picked 100 as a reasonable shake gesture.
                // Tidy down the gestures so that they don't get picked again.
                this.gestures.Clear();
                var handler = shakeEvent;
                if (handler != null)
                {
                    handler.Invoke(EventArgs.Empty);
                }
            }
        }
    } 
}
As you can see, I'm storing the data in a list - I could have used a Queue here but I just need a list. Now, what happens is that the EvaluateGesture method gets the gestures that have been created in the last 2 seconds (if the gesture is completed quicker than 2 seconds, then that's fine - the 2 seconds is merely the maximum time I've allowed for the gesture). Once it has this list, we use a couple of simple Linq queries to identify the swipe to the right, and then to check if the swipe to the left has happened - it's worth noting that we have applied an offset of 100 here to make sure that it's a deliberate movement to the left, and not just a little judder of the hand. The first query identifies the maximum X coordinate that's been reached, and the second query uses this as the starting point to search for the minimum X point reached after this.

You may be asking why I've put in a time limit of 2 seconds. There are many reasons, but the main ones are because we don't want the user to accidentally trigger the gesture just because they lazily moved their hands about. Plus, we don't want to let this list grow too big - this technique allows us to constrain the size of the list, preserving memory.

So far, for the gestures, I have chosen to add the following gesture support.

  1. Smooth the picture. Extend your fingers and swipe your hand from the right to the left.
  2. Flip horizontal. Extend your fingers and swipe your hand from the left to the right.
  3. Flip vertical. Extend your fingers and swipe your hand from bottom to top.
  4. Smooth. Extend your fingers and swipe your hand from top to bottom.
  5. Blur. Swipe your hand from left to right, and then back to the left.
  6. Red filter. Close your fingers and perform a thumbs up.
  7. Green filter. Close your fingers and perform a thumbs down.
  8. Blue filter. Stretch your thumb and fingers as far apart as you can.

Like Eskil, I have found the gesture support to be unsuitable for use as a form of pointer. I have left the code in place, and the pointer moves around the screen, but it's just not accurate enough to use as a valid selection mechanism.

But Pete, how do you know what files you can choose?

I realised that I haven't told you what the file types are that you can open up? Well, I decided that Huda wouldn't just be limited to the standard JPEG types that you get from your camera. You can also open up PNG, BMP, GIF files as well. Now, there's a handy little LINQ trick that you can use to select multiple file types using the standard Directory.GetFiles call:

IEnumerable<string> files = extensions.SelectMany(filter =>
    Directory.GetFiles(currentFolder, filter, SearchOption.TopDirectoryOnly));

You set me up

Yup, I'm even adding in a setup project. Microsoft dropped support for setup projects in Visual Studio 2012, so I need to use an alternative mechanism for creating setups. There are many options available, and I have decided to use InstallShield to create the installer. I could have used WiX, but it's quicker and easier for me to use InstallShield, so that's what I'll stick with.

But Pete, there are no pictures. Where are the screenshots?

In lieu of new screenshots, I thought you might like to see gesture support in Huda:

This weeks music

  • AC/DC - For Those About To Rock
  • AC/DC - Back In Black
  • Bon Jovi - What About Now
  • Cait Lin - The Show Must Go On
  • Twisted Sister - You Can't Stop Rock And Roll
  • Avenged Sevenfold - Nightmare

Status Update

So, by now we have the UI pretty much locked up in its entirety. All I'll be doing with it now is bug fixing and performance enhancements. We have gesture support for adding filters, and the application even tells you that which filter it's applying. Finally, you can even tell the application which filter you want to apply and it will do so.

I'm afraid, that from here on in, there won't be much new code for me to show as I'm not really going to be adding much in the way of new features. Don't worry, I will continue to share code, but you may some repeated code as I show you the code that I've fixed.

An answer to the question that you never asked.

"So Pete, why weren't you at GDC?" Unfortunately, due to family considerations, my attention hasn't fully been on Huda this week. This also meant that I was unable to attend the GDC conference, which was a major disappointment for me. I hope that the other contestants and the judges had a great time there, and I really hope that we can meet up at some point in the future.

Well, that's another weeks post over. Until next time, I bid you adieu.

Para obtener más información sobre las optimizaciones del compilador, consulte el aviso sobre la optimización.