Ultimate Coder Challenge: Sixense Studios - Week 4

Hey everyone!  It’s chip here once again to tell you all about our week’s progress! Meet one of our little pigs:

Art & Design

He is the first modeling stage and we are going after a more cartoonish silhouette with a bigger head, smaller body...etc. We already have the pig in Unity and can be driven with our new control system.

After a source control snafu in the way we originally set up our project hierarchy, Dan produced a basic scene with some temporary assets to form a “white box” of our set.  With it we’ve begun to visualize the general placement and density of assets in the scene such as it’s multi-plane depth and our approach to background planes for distant scenery and sky.

Dan also had some time to explore the Unity3d tree creator tool. This behaves much like a user friendly version of some of the open source solutions he’s used in the past and seems very promising.  The Unity3d tree creator is a fully procedural and hierarchical platform for creating fauna (not just trees). With values allowing us to set level of detail, breakage, twist, crinkle, frequency, and many other attributes; creating a realistic random tree is much easier than modeling and texturing from scratch.  This was especially exciting due to how tree heavy our concept was and the fact that this tool specifically lends itself to the two main types of trees (pine and palm) that we require.  I'm now starting to populate the scene and I’m looking forward to seeing Dan’s final art assets start to fill out the scene as he checks them in.

Tools & Systems

Danny has been hard at work on our tools.  He added a custom “Director” script to Unity that allows us to author the story scene by scene.  It essentially exposes camera settings and provides a simple method for authoring transitions such as tweening (ease in, ease out, etc), transition time, look at targets, fov...etc.  He also hooked up a way to author puppets so that an artist can wire up the puppet controls to inputs.  For instance you can connect the head orientation to the hand orientation, and the mouth joints to the hand openness state, all without needing to dive into the code or bug Ali (our engineer) to do it for you.  The joint controls can also be constrained, inverted, parented etc.  On top of that awesomeness, because we are using Unity, all of this can be done live which allows for very fast iteration times.

For our project, it became clear that basing rotations on joint orientations was not intuitive when the joints were not aligned with the camera. This would cause the puppet to rotate in local space so that left was always “left relative to his forward vector”. This wasn’t working because you had to think about the controls instead of doing what just felt natural, and it tended to kill the whole performance.The better approach was to have the joint controls be camera relative so that pitch and yaw is always relative to the user. 

Danny also ran into a snag while trying to hook up voice recognition with the Unity plugin. The Unity Editor would crash every time we tried to add recognizable strings using  pxcupipeline.SetVoiceCommand( string[] voiceStrings ). We reached out to Intel and were given the following suggestion which solved our problem, replacing two lines in their plugin in pxcupipeline.cs:

[DllImport("libpxcupipeline", EntryPoint = "PXCUPipeline_SetVoiceCommand")]
private static extern bool SetVoiceCommandC(IntPtr pp, [MarshalAs(UnmanagedType.LPWStr)] String cmd);

Mouth Control with Pinch

Ali worked on creating an algorithm to interpret a traditional “Pinch” gesture for controlling the puppet’s mouth openness.  The easiest way to explain how this works is to show you, so take it away Ali!:

If you want all the details, here is the C# Unity code Ali wrote for our “Pinch” gesture mouth control:

void UpdateHandPinchOpenness()
{
 int[] nDepthMapSize = new int[2];
 if ( pxcPipeline.QueryDepthMapSize( nDepthMapSize ) )
 {
  if ( ( nDepthMapSize[0] > 0 ) && ( nDepthMapSize[1] > 0 ) )
  {
   int nDepthMapWidth = Mathf.Abs( nDepthMapSize[0] );
   int nDepthMapHeight = Mathf.Abs( nDepthMapSize[1] );
   short[] nDepthMap = new short[nDepthMapWidth * nDepthMapHeight];
   if ( pxcPipeline.QueryDepthMap( nDepthMap ) )
   {
    // find min position
    short nMinDepth = PinchOpennessDepthBand.m_nMaxPossibleDepth;
    int nMinDepthHeight = 0;
    int nMinDepthWidth = 0;
    for ( int i = 0; i < nDepthMapHeight; i++ )
    {
     for ( int j = 0; j < nDepthMapWidth; j++ )
     {
      int nIndex = ( i * nDepthMapWidth ) + j;
      if ( nDepthMap[nIndex] < nMinDepth )
      {
       nMinDepth = nDepthMap[nIndex];
       nMinDepthHeight = i;
       nMinDepthWidth = j;
      }
     }
    }
    
    // find horizontal range in row
    const float fHorizontalRange = 50.0f;
    int nRowRange = ( int )( ( fHorizontalRange / m_fFocalLengthHorizontal ) * ( float )( nMinDepth ) );
    
    // clear depths with values more than range from min, and find row bands
    const short nDepthRange = 60;
    const int nMaxDepthBands = 10;
    const int nMinBandHeight = 3;
    PinchOpennessDepthBand[] depthBands = new PinchOpennessDepthBand[nMaxDepthBands];
    for ( int i = 0; i < nMaxDepthBands; i++ )
    {
     depthBands[i] = new PinchOpennessDepthBand();
    }
    int nCurrentDepthBand = 0;
    for ( int i = 0; i < nDepthMapHeight; i++ )
    {
     // process row
     bool bRowEmpty = true;
     int nDepthColumnsInRow = 0;
     short nMinRowDepth = PinchOpennessDepthBand.m_nMaxPossibleDepth;
     for ( int j = 0; j < nDepthMapWidth; j++ )
     {
      int nIndex = ( i * nDepthMapWidth ) + j;
      if ( ( nDepthMap[nIndex] >= ( nMinDepth + nDepthRange ) ) || ( Mathf.Abs( nMinDepthWidth - j ) > nRowRange ) )
      {
       nDepthMap[nIndex] = PinchOpennessDepthBand.m_nMaxPossibleDepth;
      }
      else
      {
       bRowEmpty = false;
       nDepthColumnsInRow++;
       if ( nDepthMap[nIndex] < nMinRowDepth )
       {
        nMinRowDepth = nDepthMap[nIndex];
       }
      }
     }
     // clear out rows with too few depth points
     const int nMinDepthColumnsInRow = 4;
     if ( !bRowEmpty && ( nDepthColumnsInRow < nMinDepthColumnsInRow ) )
     {
      for ( int j = 0; j < nDepthMapWidth; j++ )
      {
       int nIndex = ( i * nDepthMapWidth ) + j;
       nDepthMap[nIndex] = PinchOpennessDepthBand.m_nMaxPossibleDepth;
       bRowEmpty = true;
       nDepthColumnsInRow = 0;
       nMinRowDepth = PinchOpennessDepthBand.m_nMaxPossibleDepth;
      }
     }
     // process band
     if ( nCurrentDepthBand < nMaxDepthBands )
     {
      // update min band depth
      if ( ( -1 != depthBands[nCurrentDepthBand].m_nRowStart ) &&
        !depthBands[nCurrentDepthBand].m_bRowEmpty &&
        ( nMinRowDepth < depthBands[nCurrentDepthBand].m_nMinDepth ) )
      {
       depthBands[nCurrentDepthBand].m_nMinDepth = nMinRowDepth;
       depthBands[nCurrentDepthBand].m_nMinDepthRow = i;
      }
      
      // start new band
      if ( -1 == depthBands[nCurrentDepthBand].m_nRowStart )
      {
       depthBands[nCurrentDepthBand].m_bRowEmpty = bRowEmpty;
       depthBands[nCurrentDepthBand].m_nRowStart = i;
       depthBands[nCurrentDepthBand].m_nMinDepth = nMinRowDepth;
       depthBands[nCurrentDepthBand].m_nMinDepthRow = i;
      }
      else if ( depthBands[nCurrentDepthBand].m_bRowEmpty != bRowEmpty )
      {
       // is band tall enough
       if ( ( ( i - 1 ) - depthBands[nCurrentDepthBand].m_nRowStart ) < nMinBandHeight )
       {
        // skip short bands
        if ( nCurrentDepthBand > 0 )
        {
         depthBands[nCurrentDepthBand].m_nRowStart = -1;
         nCurrentDepthBand--;
        }
        else
        {
         if ( nCurrentDepthBand < nMaxDepthBands )
         {
          depthBands[nCurrentDepthBand].m_bRowEmpty = bRowEmpty;
          depthBands[nCurrentDepthBand].m_nRowStart = i;
          depthBands[nCurrentDepthBand].m_nMinDepth = nMinRowDepth;
          depthBands[nCurrentDepthBand].m_nMinDepthRow = i;
         }
        }
       }
       else
       {
        // finish band
        depthBands[nCurrentDepthBand].m_nRowEnd = i - 1;
        nCurrentDepthBand++;
        // start new band
        if ( nCurrentDepthBand < nMaxDepthBands )
        {
         depthBands[nCurrentDepthBand].m_bRowEmpty = bRowEmpty;
         depthBands[nCurrentDepthBand].m_nRowStart = i;
         depthBands[nCurrentDepthBand].m_nMinDepth = nMinRowDepth;
         depthBands[nCurrentDepthBand].m_nMinDepthRow = i;
        }
       }
      }
     }
    }
    // finish last band
    int nNumBands = 0;
    if ( ( nCurrentDepthBand < nMaxDepthBands ) && ( -1 != depthBands[nCurrentDepthBand].m_nRowStart ) )
    {
     depthBands[nCurrentDepthBand].m_nRowEnd = nDepthMapHeight - 1;
     // is band tall enough
     if ( ( depthBands[nCurrentDepthBand].m_nRowEnd - depthBands[nCurrentDepthBand].m_nRowStart) < nMinBandHeight )
     {
      // skip short bands
      if ( nCurrentDepthBand > 0 )
      {
       depthBands[nCurrentDepthBand].m_nRowStart = -1;
       nCurrentDepthBand--;
      }
     }
     nNumBands = nCurrentDepthBand + 1;
    }
    // find hand top/bottom
    float fTopY = 0.0f;
    float fBottomY = 0.0f;
    // open fingers (two blobs w/ empty above, between, and below)
    if ( ( 5 <= nNumBands ) && depthBands[0].m_bRowEmpty )
    {
     fTopY = ( ( ( float )( depthBands[1].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[1].m_nMinDepth;
     fBottomY = ( ( ( float )( depthBands[nNumBands - 2].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[nNumBands - 2].m_nMinDepth;
    }
    else if ( ( 4 == nNumBands ) && depthBands[0].m_bRowEmpty )
    {
     fTopY = ( ( ( float )( depthBands[1].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[1].m_nMinDepth;
     fBottomY = ( ( ( float )( depthBands[3].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[3].m_nMinDepth;
    }
    else if ( ( 4 == nNumBands ) && !depthBands[0].m_bRowEmpty )
    {
     fTopY = ( ( ( float )( depthBands[0].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[0].m_nMinDepth;
     fBottomY = ( ( ( float )( depthBands[2].m_nMinDepthRow - ( nDepthMapHeight / 2 ) ) ) / m_fFocalLengthVertical ) * depthBands[2].m_nMinDepth;
    }
    // openness
    const float fMinHeight = 20.0f;
    const float fMaxHeight = 160.0f;
    const float fMaxAdjustedHeight = fMaxHeight - fMinHeight;
    float fHeight = fBottomY - fTopY - fMinHeight;
    fHeight = ( fHeight < 0.0f ) ? 0.0f : ( fHeight > fMaxAdjustedHeight ) ? fMaxAdjustedHeight : fHeight;
    pxcPinchOpenness = fHeight / fMaxAdjustedHeight;
   }
  }
 }
}

Ali also recorded a comparison of the current implementations of our gesture control modes.  The first one he implemented is “Openness”, which uses the hand normal, position, and openness provided by the SDK to control the corresponding view direction, position, and mouth openness of the puppet directly.  The second one he just added this week is the “Pinch” gesture, which is a new direction we are exploring to try to achieve a more natural control.  As you can see in this next video, it’s a tradeoff between power and usability. In the end our vision is to eventually be able to drive the puppets through pure simulation and have the puppet respond just as if you were wearing it over your hand.

Next Week

In addition to continuing to work on improving the gesture controls, Ali will be looking at adding multiplayer functionality. We still had a large number of features we were hoping to come online based on our original vision for the contest. As many other teams are likely doing, we will be having a meeting to finalize the scope of our project in terms of what we can deliver for the contest based on where we are now.

The good news is that by the end of the week we should also have the first playable version of our story!

 

 

Einzelheiten zur Compiler-Optimierung finden Sie in unserem Optimierungshinweis.