Threading Challenge 2011 -Maze of Life, Apprentice Problem 1: Judging & Scoring Criteria and Methods
As of June 21, 2011, we announced the winners forApprentice Problem 1, Maze of Life. Our group of Judges used the following judging and scoring criteria and methods for selecting the winners. In addition, you will find a link to the testing results and the scores further below.
Apprentice Level Problem Set 1 (P1A1) Maze of LifeKey Scoring Principles
Basic scoring principles used for the contest entries judging are described at the official rules page. Here is a short summary: each contest entry was scored according to the following criteria: 1) up to 100 points for solutions performance (speed); 2) a maximum of 25 bonus points for a contestants activity in the forum, calculated as 5 bonus points for each valid forum post/reply.
There may be multiple solutions to a given puzzle. A bonus of 5 points per puzzle awarded to those entries that find the shortest path. Sometimes bonus was awarded to multiple entries outputting the same length path, even if those entries discovered different solutions. Please find the detailed problem set description at the Apprentice Level contest page.
Input Data Sets Used for Performance Scoring
Ten different input data sets were used to compute the execution score for this problem. The simplest one is a 7x7 Maze of Life grid given as an example in the problem set description. The hardest is a large 300x300 grid. Full archive of input data sets can be downloaded here.
Points in Performance Scoring
Each input data set was judged individually based on a ranking scheme. The weights of data sets used in performance scoring were equivalent. The overall performance score was calculated as a sum of all ten input data sets individual performance and bonus points.
We allowed a total of 120 seconds execution (2 minutes) maximum for each input set; for those runs that took longer than 120 seconds or had runtime errors during execution, zero performance points were awarded. Some entries that could not be built on the MTL got zero points as well.
Successful contest entries that found smart cell path in less than 2 minutes were ranked based on their execution time and got performance points according a reciprocal rank scale. For example, the fastest solution[s] of a data set got 5 points, next solution got 2.5 points, then 1.67 points and so on and so forth.
On top of that all the solutions that found a shortest path in particular input data set got 5 bonus points. Thus a successful solution could get a maximum of 10 points per input data set if it finds the shortest path and demonstrates the best performance.
Execution Results and Point Spread
Weve received 28 contest entries in the Apprentice Level Maze of Life problem set.
Six entries successfully solved all 10 grids. One entry solved 9 out of 10. Two entries solved 8 grids, four solved 7 Unfortunately, nine entries were incomplete and therefore unable to solve any test data sets.
All the timings and performance points are available in the final Maze of Life scoring table below.
Bonus Points for shortest paths found
As for the results in terms of shortest paths, they vary. Only one participant provided a solution that always found a shortest path on the grid. Interestingly enough, none of the entries that solved all 10 grids were able to find more than 2 shortest paths. Therefore, they did not get significant bonus points. All the bonus points are available in the final Maze of Life scoring table below.
Forum Activity and Bonus Points
Additional bonus points were given for contestants forum posts made before the problem entries were closed. Five points per post (maximum 25 points possible) were awarded.
Entry points and penalties.
Each contest entry got 100 entry points. A penalty of 50 points was taken off in case the entry is not able to solve simple data grid given as an example in the problem set description.
The problem winners based on highest point total are:
These three contestants provided the solutions which resolved the maximum number of our test grids. They also had the fastest overall code execution and a fair amount of bonus points for the shortest paths.