Forum Jump

Select Group :
Select Forum :
Sorted By :
Sort Order :
From The :
 
Thread Tools  Search this thread 
Clay Breshears (Intel)
Total Points:
15,225
Status Points:
15,225
Black Belt
May 13, 2009 11:28 AM PDT
Scoring Criteria

There were 5 input files used to test the code solution portion of the Radix Sort problem. The first had 94 keys that each started with a different printable character and contained text that made it easy to determine if the sorting was done correctly by visual inspection. This file was only used to determine if the executables could accept the file names from the command line and handle the range of characters to be found in keys. The other four input files contained 1 million, 10 million, 20 million, and 50 million keys. As long as the data was not altered in any way during input, the execution time for the sorting only was recorded and used to compute the execution time score for this problem. Output was checked via program due to the large size of the files.

A maximum of one minute was set for each input data file. For those runs that aborted or generated some other runtime error, this maximum runtime was entered. The total execution time of the four timed data sets was added together. This total was used to compute a score between 0 and 100 based on where the total fell between the minimum total and the maximum (240 seconds) in a linear fashion. (That is, if a total execution time was 60 seconds, the execution score would be around 75 points.)

There were 25 submissions of code solutions, 7 on Linux and 18 on Windows. There was one entry written in C#, one in Java, and the rest in C/C++. One entry using MPI was unable to be tested since we don’t have the facilities to handle this form of parallelism. The approaches used were split about 50-50 for using the MSD (Radix Exchange Sort) algorithm or the LSD (Straight Radix Sort) algorithm.

Point spread:
100
99 99 98 98 98 98 97 96 96 96 95 95 92 91
89
75 73
69
25
0 0 0 0 0

The write-up portion of each entry was read and scored by two judges. Each judge used the 10-30-10 breakdown of points for serial algorithm description, parallel algorithm description, and performance, respectively. One important component to the judging was to determine how close the submission was for publication on ISN. The assigned score was the average of the two judges scores.

Point spread:
49 47 46 45 44 41
38 37 36 33 32 32 31
29 27 27 
18 11
2

Bonus points were given for contestant’s forum posts made before the problem entries were closed. Five points per post (maximum 25 points possible) were awarded.

The overall winner was denghui0815. The fastest code execution was submitted by emacswu; the highest scoring write-up submission was from calebe.

(Now that we have gotten through the first problem and worked through some of the bugs of the contest pages, we'll be able to execute the judging and scoring of future problems faster than we did for this one.)

--clay

dweeberlyloom
Total Points:
6,630
Status Points:
6,130
Brown Belt
May 13, 2009 12:19 PM PDT
Rate
 
#1

There were 5 input files used to test the code solution portion of the Radix Sort problem. The first had 94 keys that each started with a different printable character and contained text that made it easy



...

(Now that we have gotten through the first problem and worked through some of the bugs of the contest pages, we'll be able to execute the judging and scoring of future problems faster than we did for this one.)

--clay


How do we find out well we did (or didn't do) ?

Oh, and times ... what are the times ... metrics man, metrics!  ... how do we know who's got the smallest one if we can't measure against each other ... wait I don't think that came out right ... but you know what I mean :-)

Last contest there use to be a link for this, but I can't seem to find it for this one.
Thanks

akki
Total Points:
2,720
Status Points:
2,220
Brown Belt
May 13, 2009 10:46 PM PDT
Rate
 
#2

There were 5 input files used to test the code solution portion of the Radix Sort problem. The first had 94 keys that each started with a different printable character and contained text that made it easy to determine if the sorting was done correctly by visual inspection. This file was only used to determine if the executables could accept the file names from the command line and handle the range of characters to be found in keys. The other four input files contained 1 million, 10 million, 20 million, and 50 million keys. As long as the data was not altered in any way during input, the execution time for the sorting only was recorded and used to compute the execution time score for this problem. Output was checked via program due to the large size of the files.

A maximum of one minute was set for each input data file. For those runs that aborted or generated some other runtime error, this maximum runtime was entered. The total execution time of the four timed data sets was added together. This total was used to compute a score between 0 and 100 based on where the total fell between the minimum total and the maximum (240 seconds) in a linear fashion. (That is, if a total execution time was 60 seconds, the execution score would be around 75 points.)

There were 25 submissions of code solutions, 7 on Linux and 18 on Windows. There was one entry written in C#, one in Java, and the rest in C/C++. One entry using MPI was unable to be tested since we don’t have the facilities to handle this form of parallelism. The approaches used were split about 50-50 for using the MSD (Radix Exchange Sort) algorithm or the LSD (Straight Radix Sort) algorithm.

Point spread:
100
99 99 98 98 98 98 97 96 96 96 95 95 92 91
89
75 73
69
25
0 0 0 0 0

The write-up portion of each entry was read and scored by two judges. Each judge used the 10-30-10 breakdown of points for serial algorithm description, parallel algorithm description, and performance, respectively. One important component to the judging was to determine how close the submission was for publication on ISN. The assigned score was the average of the two judges scores.

Point spread:
49 47 46 45 44 41
38 37 36 33 32 32 31
29 27 27 
18 11
2

Bonus points were given for contestant’s forum posts made before the problem entries were closed. Five points per post (maximum 25 points possible) were awarded.

The overall winner was denghui0815. The fastest code execution was submitted by emacswu; the highest scoring write-up submission was from calebe.

(Now that we have gotten through the first problem and worked through some of the bugs of the contest pages, we'll be able to execute the judging and scoring of future problems faster than we did for this one.)

--clay



The big picture is pretty clear here but it says nothing about the individual... :(
What I hoped to see, was something like:

User         OS          Code score           Write-up score          Total



Am I the only one or do people agree with me on this?


邓辉
Total Points:
3,850
Status Points:
3,350
Brown Belt
May 14, 2009 9:49 AM PDT
Rate
 
#3
Perhaps we need to see own points :)

--------
写字楼里写字间,写字间里程序员
程序人员写程序,又拿程序换酒钱
酒醒只在网上坐,酒醉还来网下眠
酒醉酒醒日复日,网上网下年复年


Dmitriy Vyukov
Total Points:
25,462
Status Points:
25,462
Black Belt
May 14, 2009 10:02 AM PDT
Rate
 
#4 Reply to #2
Quoting - akki
he big picture is pretty clear here but it says nothing about the individual... :(
What I hoped to see, was something like:

User         OS          Code score           Write-up score          Total



Am I the only one or do people agree with me on this?


Agree. I would like to see timings for each individual input data size separately.



haojn
Total Points:
1,760
Status Points:
1,260
Brown Belt
May 15, 2009 3:06 AM PDT
Rate
 
#5
Maybe everyone can select a unique number to replace the user name in order to protect privacy.


akki
Total Points:
2,720
Status Points:
2,220
Brown Belt
May 15, 2009 4:47 AM PDT
Rate
 
#6 Reply to #5
Quoting - haojn
Maybe everyone can select a unique number to replace the user name in order to protect privacy.

I didn't realize people would be concerned about privacy here. It is a competition after all. Competitions have scoreboards... ;)

I'm 10th out of 25 and not ashamed to admit it. :D

It's just that I really would like to know how my code performs against the others'. I know my code score must be between 89 and 97 but that's too large a range to derive any conclusions. And, since there were only 25 entries, it shouldn't be too difficult to compile a table of scores...


@clay: Just re-read your post. Sure looks like you've put a lot of effort into putting it together. Thanks.


kenw2
Total Points:
140
Status Points:
90
Green Belt
May 20, 2009 1:22 PM PDT
Rate
 
#7 Reply to #4
As a novice in these competitions, I am interested knowing how well I did.  It would also be beneficial to see what the best code and write-up looked like.  Somebody noted that they were 10th out of 25.  How do you find out this information?



akki
Total Points:
2,720
Status Points:
2,220
Brown Belt
May 20, 2009 11:38 PM PDT
Rate
 
#8 Reply to #7
Quoting - kenw2
As a novice in these competitions, I am interested knowing how well I did.  It would also be beneficial to see what the best code and write-up looked like.  Somebody noted that they were 10th out of 25.  How do you find out this information?


Take a look at this thread. How you interpret it is upto you... ;)
http://software.intel.com/en-us/forums/general-contest-questions/topic/65529/


feanorgem
Total Points:
40
Registered User
May 26, 2009 7:13 PM PDT
Rate
 
#9
Actually, I'm interested in seeing the various code with the scores to see what things worked well and what didn't.




Intel Software Network Forums Statistics

8491 users have contributed to 31629 threads and 100764 posts to date.
In the past 24 hours, we have 32 new thread(s) 141 new posts(s), and 200 new user(s).

In the past 3 days, the most popular thread for everyone has been gemm(A,A,A) like possible? The most posts were made to Crash when loading skeleton The post with the most views is Dear Steve, excuse me for a d

Please welcome our newest member shadowwolf99