Threading Challenge 2011 - Masyu Puzzle, Master Problems 1: Judging & Scoring Criteria and Methods
As of June 21, 2011, we announced the winners for Master Problem 1, Masyu Puzzle. Our group of Judges used the following judging and scoring criteria and methods for selecting the winners. In addition, you will find a link to the testing results and the scores further below.
Master Level Problem Set 1 (P1:M1) Masyu PuzzleKey Scoring Principles
Basic scoring principles used for the contest entries judging are described at the official rules page. Here is a short summary: each contest entry was scored according to the following criteria: 1) up to 100 points for solutions performance (speed); 2) up to 50 points for the solutions write-up using the 10-30-10 breakdown of points for a serial algorithm description, parallel algorithm description and performance analysis respectively; 3) a maximum of 25 bonus points for a contestants activity in the forum, calculated as 5 bonus points for each valid forum post/reply.
Input Data Sets Used for Performance Scoring
Ten different input data sets were used to compute the execution score for this problem. The simplest one is a 7x7 Masyu puzzle given as an example in the problem set description. The hardiest is a dense 40x36 puzzle. Full archive of input data sets can be downloaded here.
Points in Performance Scoring
Each input data set was weighted equally and was judged individually based on a ranking scheme. The overall performance score was calculated as a sum of all ten input data sets individual points.
We allowed a total of 120 seconds execution (2 minutes) maximum for each input set; for those runs that took longer than 120 seconds or had runtime errors during execution, zero performance points were awarded. Some entries that could not be built on the MTL got zero points as well.
Successful contest entries that solved test puzzles in less than 2 minutes were ranked based on their execution time and got performance points according to a reciprocal rank-weight scale. For example, the fastest solution[s] of a data set got 10 points, next solution got 5 points, then 3.33 points and so on and so forth.
Execution Results and Point Spread
We received 16 contest entries in the Master Level Masyu Puzzle problem set.
Only one entry successfully solved all 10 puzzles. Another entry solved 9 out of 10. Two entries solved 6 and 5 puzzles each. Four entries were able to solve 3 puzzles each; three entries 2 puzzles. Finally, three entries were incomplete and therefore unable to solve any puzzle.
All the timings and performance points are available in the final Masyu scoring table below.
The write-up portion of each entry was read and scored by several judges. Each judge used the 10-30-10 breakdown of points for serial algorithm description, parallel algorithm description, and performance, respectively. One important component to the judging was to determine how close the submission was for publication on ISN. The assigned score is available in the final Masyu scoring table below.
Forum Activity and Bonus Points
Bonus points were given for contestants forum posts made before the problem entries were closed. Five points per post (maximum 25 points possible) were awarded.
Entry points and penalties.
Each contest entry got 100 entry points. A penalty of 50 points was taken off in case the entry is not able to solve simple Masyu puzzle given as an example in the problem set description.
The Masyu puzzle winners based on highest point total are the following:
2) Rui Diao
These three contestants provided the solutions which resolved the maximum number of our test puzzles. They also had the fastest overall code execution and well-written write-ups.
Access Judging Score Card for Masyu Puzzle, Master Problem 1 Updated July 9, 2011