Overall it went smoothly, but it was still alot of work. I was intent on offering full critiques (from a designer perspective) of the ones I was reviewing, so every competitor would be able to get valuable feedback to improve their future games. It was very exhausting.

I also made a Google spreadsheet to crunch totals and remind judges of how many points were permitted in each category (feel free to steal the spreadsheet). IIRC, the green checkmarks help indicate which judges have fully entered their scoring for which entries. In practice, most judges just did their scoring offline and entered the results in one swoop when finished.

Last year's judging guidelines is here.

Breaking the games across all judges was something that was done out of necessity, but it also makes it much less "fair". I can remember going through something similar in high school where we had 2 math teachers, and the two of them had a drastic differences in their exams. Teacher A had a relatively easier exam, whereas Teacher B was more demanding. To the outsider, it felt like class B (associated with Teacher B) had on average poorer results, which would've indicated that they were not as good, but as it turned out, this was incorrect.

Last year, we had enough judges that every game was reviewed by at least three judges + slicer reviewed every one. To further mitigate variations in judging standards, we made it so no entry was judged by the same three judges, by shifting the rows of judges down:

I'm sure you could get by with just three judges judging every game.

Interesting.

Your spreadsheet would indicate there were only 6 full entries? I imagine this to be a sample of the full entry?

Do you have the full count?

Here's the spreadsheet - see for yourself. Should be public - we made the entire sheet public after the final results were revealed.

We had 24 entries and 6 judges. Each judge judged 14 or 15 entries. Slicer judged all 24.

Each entry was judged by 4 judges, except one which happened to get judged by 5 - but since scores are averaged between all judges (instead of merely added), one extra reviewer doesn't give an entry any innate benefit or detriment.

The more reviews that get averaged, the less accidental bias creeps into the scoring.

I'd personally limit the judging to 3 reviews per game instead of 4, to reduce judge workload (depending on the number of judges and finished entries).

Having clear examples of the expected quality for each judging dimension could help out - I felt like I spent a lot of time re-reviewing earlier games, after later games caused my scoring criteria to move (i.e. I'd score a game highly for graphics early on, then have to re-evaluate it down as later entries exceeded expectations).

The crux of that problem, I think, comes down to whether you are using an absolute or relative scoring system. Either:

a) Your scoring system is absolute - so playing a later game has no impact on the score of a game you played earlier. It's not like the graphics for that first game you judged have actually gotten any worse just because you played another game, right?

b) Your scoring system is relative - so you couldn't possibly come up with a numerical score for a game *before* you have compared it to the games you are scoring it relative to.

In theory, for either case, you wouldn't need to go back and re-address entries you have already judged.

It actually sounds like you were using a relative system but you were using the scores to encode a relative rank, which meant that whenever a game moved up in rank you had to go back and decrement the numbers of all games that have fallen below it. At that point I guess you probably also took on the additional mental burden of trying to decide "how much" has a particular game fallen below. Probably you would have found it easier to just rank entries in a list and reorder the list as needed, then work out their numeric scores at the end (based on their position in the list).


Some people last year felt that the judging criteria should be so well defined that participants could accurately gauge how well their game ticks off points and all judges would line up and generate nearly the same scores for the same games.
Personally I don't like that - I believe judges should be able to express their different views through the scoring system. Something like a ranking system (per judging dimension) allows for that I think.

One of the problems that I suspect arose in previous years is that if different judges are using different scoring systems then the average score (which is used to decide the winner) is a meaningless value really. This is worse if (as seems to be the case) the games are not all judged by all judges! E.g. If judge A is using a linear absolute system, judge B is using some logarithmic system, judge C is using a relative ranking system, and so on, then Game1's score might be an average of 3 such systems while Game2's score might be an average of 3 other such systems - there is no sane/fair way to compare the resultant number to determine which game won.

tl;dr: Judging against the same criteria is not enough, all judges need to be scoring under the same system, but that system shouldn't be inherently opinionated IMO, leave the opinions to the judges! :D

I've actually been thinking about the competition of late, and do intend to take part if it takes place this year. ^_^

I've actually been thinking about the competition of late, and do intend to take part if it takes place this year. ^_^

It will take place even if I have to return to reality and shovel this competition into everyone's infile :)

@Orymus I will send you a mail once I am much less sleep deprived, which given my current schedule will be in two days time.

Orymus3 / Slicer4Ever - contact me if this is something you want to move forward with. We can make this an official GameDev.net virtual event and use our past experience with contests / judging.

p.s. anyone still remember the 4 Elements Contest?

Hey their folks, so I'm still uncertain if I want to run this years, while last years ran mostly fine, a few folks took issue with the scores they received, and the number of people talking about being disappointed in how they did really left a sour feeling for how things turned out. Even so if i don't run this years, i figured i should outline the changes I had in plan for this years.

First off the prize pool, I got a feeling that too many folks were putting alot more stock into winning than they should have. I have no issue with having prize money but I don't think I want it to be as big as it was last year, I certainly have no plans to contribute as much this year to it as I have last. I will also likely remove the prizes for feedback, and funnel that into the main pool, as it didn't seem many people were interested in that prize.

Secondly, the theme was going to branched out to have more options. a big thing throwing people off was the fact that the theme was too strict to ideas, i do agree with this. I was planning to offer 4 themes this year, and 2 of which must be implemented(basically 5 points for each theme you work into your game). this would likely make the theme simpler, being 1 or 2 words, and likely split between two gameplay, and two graphical themes.

Thirdly would be how judging is handled, i was toying with the idea that having additional judges is still good, but to only take the top 3 scores in each category. meaning if we have a total of 4 judges on each game, that game will not be pulled down because one judge didn't see it as being as good as the others.

Finally would be removing the sponsors concept, I don't think it was very good, and basically if you want to contribute to the pot, then you should feel free to do so, and they will be recognized for doing so, but not in anyway that's in each game.

I seem to remember there being some discussion even among the judges as to what the actual scores should represent.

I think having something written up-front as to what the different scores mean for each category might be beneficial, both for judges and contestants. I can't actually remember if this was done last time or not...

At least it might help negate some disappointment concerning expectations and results.

I also think that any disappointment was, for the most part, short-lived -- personally I would definitely have liked to end up higher placed (and thought I would), but given a day or two to let things settle I didn't have any strong issues with the results :)

---

For me, I might possibly want to participate and make something, but I don't think I have the resources to offer anything more than that this time around.

Orymus3 / Slicer4Ever - contact me if this is something you want to move forward with. We can make this an official GameDev.net virtual event and use our past experience with contests / judging.

p.s. anyone still remember the 4 Elements Contest?

Got'cha, still giving this a bit more time to see where folks stand regarding the competition.

So far, looks to me like there is enough interest to hold it, but that some changes would be welcomed.

Trying to get a feel of what this summer will be like for me too.

4 Elements Contest? Can't say this rings a bell :(

Hey their folks, so I'm still uncertain if I want to run this years, while last years ran mostly fine, a few folks took issue with the scores they received, and the number of people talking about being disappointed in how they did really left a sour feeling for how things turned out. Even so if i don't run this years, i figured i should outline the changes I had in plan for this years.

First off the prize pool, I got a feeling that too many folks were putting alot more stock into winning than they should have. I have no issue with having prize money but I don't think I want it to be as big as it was last year, I certainly have no plans to contribute as much this year to it as I have last. I will also likely remove the prizes for feedback, and funnel that into the main pool, as it didn't seem many people were interested in that prize.

Secondly, the theme was going to branched out to have more options. a big thing throwing people off was the fact that the theme was too strict to ideas, i do agree with this. I was planning to offer 4 themes this year, and 2 of which must be implemented(basically 5 points for each theme you work into your game). this would likely make the theme simpler, being 1 or 2 words, and likely split between two gameplay, and two graphical themes.

Thirdly would be how judging is handled, i was toying with the idea that having additional judges is still good, but to only take the top 3 scores in each category. meaning if we have a total of 4 judges on each game, that game will not be pulled down because one judge didn't see it as being as good as the others.

Finally would be removing the sponsors concept, I don't think it was very good, and basically if you want to contribute to the pot, then you should feel free to do so, and they will be recognized for doing so, but not in anyway that's in each game.

My feeling from 2015 was that the increased prize pool made it a lot more competitive and a bit less 'friendly' than 2014, but that was just my impression. I tend to agree that a smaller prize pool would probably be desirable, and it seems like this is in line with the poll results we currently have as far as funding is concerned.

You Theme idea isn't bad, but I believe it should have some form of main theme. That being said, I've taken part in a gamejam years ago where you had bonus points for including elements. Some of them were related to community insider jokes, but some of them were outright challenges, and it ws up to the judge to determine whether you had succeeded at making it 'work' (how easy is it to talk about underwears in a high fantasy game without falling straight into trash comic territory!)

Judging should work. Question still remains to see how many we end up with...

I was not opposed to sponsering per se, but it seems you've had a bad experience with that, so I'll defer to your judgment here.

I seem to remember there being some discussion even among the judges as to what the actual scores should represent.

I think having something written up-front as to what the different scores mean for each category might be beneficial, both for judges and contestants. I can't actually remember if this was done last time or not...

At least it might help negate some disappointment concerning expectations and results.

I also think that any disappointment was, for the most part, short-lived -- personally I would definitely have liked to end up higher placed (and thought I would), but given a day or two to let things settle I didn't have any strong issues with the results :)

---

For me, I might possibly want to participate and make something, but I don't think I have the resources to offer anything more than that this time around.

I was leaning that direction too. I feel that if each category was 1-5 or something similar (and THEN weighted for finals coring) it would create consistency and we could describe 1-5 as items such as '3 - Passable: the element worked, but it did not carry the game to be a compelling experience', etc.

I've actually been thinking about the competition of late, and do intend to take part if it takes place this year. ^_^

It will take place even if I have to return to reality and shovel this competition into everyone's infile :)

@Orymus I will send you a mail once I am much less sleep deprived, which given my current schedule will be in two days time.

Will await your PM :)

At this point, I'm unclear whether I'll be organizing this, as I'm still unsure where my availability will land, but I'd really like to get to a point where whoever ends up organizing this has an easy task. Slicer has had to do a LOT of work the past two years to make this happen, and I'm sure I can help streamline the process a bit, if not more.