My first year as an ENnies judge I wrote down all on my thoughts about books on post-it notes and littered my books with them. I created stacks of "yes", "no", and "maybe". I printed up sheets with the names of every book and every category and used different colored highlighters to show which ones I had read, liked, and needed to revisit. Finally I broke down and used lined paper with a category name at the top and went through each book one by one and wrote its name on the paper with the category I thought it belonged in.
Somehow it actually worked, and it worked well. I'm proud of the lists we produced that year, and I'm proud of the work I did to make it happen. I'm a closer you see. I listen and I discuss, but in the end, as time winds down the part of me that was a small business owner, the part that handled sales, marketing, and operations all by myself for 12 years kicks in. At that point I organize, gather everyone's opinion, then push through to consensus.
In 2007 we used a message board for that. Each award had its own thread, and each judge published their lists for what would be in the category. I then went through an organized them into a single list, ranked in order of what games had appeared the most on various lists. From that point discussion resumed with a focus on explaining our choices and wheeling and dealing our way to consensus.
In 2009 I updated my personal process to the use of a spreadsheet and everything got better. It was faster, easier to read, and I didn't have to look through the same books over and over again to compare my thoughts. I listed each book in one column, then created columns for every award category. As I read each book I scored it with a number in every column, followed by commentary on why it got that score. If the book didn't qualify I highlighted it red and ranked it "0". Books that had no chance got orange and a "1". A "2" was yellow and indicated a good quality submission that was lees than likely to make the final cut. "3" was green, and often the hardest category; these are books that deserved recognition in the area, but not instant winners. Finally there was "4"; I tried to be very stingy with these, 2 or three at most in every category. They were the books I read, and knew had to make the short list.
When time came to discuss what was getting the award I simply sorted each column by score, then compared the notes that followed until I had narrowed my choices to a reasonable number. this worked extremely well, and I still use it to help me narrow down my own thoughts. Then again, I still have my highlighters and paper too, so maybe I'm just old-fashioned.
The point of this whole thing is that this year we moved EVERYTHING to spreadsheets. Thanks to the easy and ubiquity of Google Docs we were able to make a spreadsheet with a column listing every category, and another for each judge. If a judge wanted to discuss a product for possible nomination he or she would add it under the appropriate category and then start an email (through Google Groups of course) discussing the product, what he/she liked about it, and asking for thoughts of the others. At the end this turned into emails about each category, and each judge went in and put an "X" in their column next to any products they supported.
These weren't votes precisely, more an idea of where each judge stood - a starting point for discussion. From there we talked at length in each category's email, while adding and taking away "X"'s on the spreadsheet.
I think that this system is the best we've ever used. There was no last minute scramble, no going-along to get-along; we came up with a solid list determined by mutual assent.
Did everything that deserved an award make it on to the list? No. That'll never happen. But I am firmly of the mind that everything that is on the list deserves to be there.
This method gave us the most diverse group of nominations I've ever seen in the awards, and I'm proud of it.
So, why waste so much of your time going over a long list of the "hows" of judging? I know it doesn't much effect anyone who is not a judge, but it lays the groundwork for my next post: "Why your product doesn't suck."
Do me a favor. Go vote for me before you read that one, okay?
No comments:
Post a Comment