MetaFic

The Metafic/Cracked Muggle administrators invite people to be judges based on one of two key criteria:  (a) they are an author of note in our opinions, or (b) they are a reviewer that goes above and beyond by at least an order of magnitude.

We’ve asked people to participate as a judge in general, knowing that for specific challenges they might either be unable to judge due to lack of interest (or time), or else they might be barred due to having a submission for that particular challenge.

The judging process uses a points-based system, much like the peer-review process for scientific publications.  Unlike those publications, however, we have some hard and fast break points that will be evaluated.   The scale of judging per entry is based on the following set of discrete choices:

1. Poor entry; Recommend a strong rejection of this work
2. Weak entry; Recommend a rejection of this work
3. Neutral entry; The work has good and bad qualities, but am ultimately indifferent
4. Good entry; Recommend this for recognition
5. Outstanding entry; This is a defining work in the area

Each story is rated by each judge in whatever categories are requested.  Judges are not required to give any particular scores: there may be no fics worthy of a 4 or 5 score, or there may be several.  This metric is a score, not a rank in any way.  Most stories could be considered as starting at the value 3, then increasing or decreasing as needed.  Stories with a few errors might receive a 2, while unreadable stories might receive a 1.  Stories that are excellent but not perfect might receive a 4.  Only a small percentage of fanfic stories should receive a score of 5, and they should represent the very best of the genre in question.  Stories scored at 5 should be technically superior with excellent prose, solid characterizations, and a very engaging plot and/or premise.

In the case of a tie among entries, there is one “recused judge” who is normally non-voting and is also not allowed to submit an entry for judging (any judge may submit an entry and receive reviews, but they cannot be voted on).  This recused judge is there for moderation in the case of questions about entries following the rules, as well as casting any tie-breaking vote.  The only time the recused judge may vote is in the case of a tie: in that case the recused judge votes only to break the tie and does not vote on the other stories in the challenge.  When any voting judge has a question or concern, their first point of contact should always be the recused judge.

Any score of “1” automatically renders the entry ineligible to win anything.  If any judge truly feels that strongly about a work, there must be something wrong with it.  Upon receipt of any “1” vote, the mediating judge may opt to instigate a dialog if it seems particularly out of alignment with other votes.   The objective is not to change the vote but to ensure that sound reasoning lies behind it and no bias or mistake was made.

In terms of the criteria, judges currently rate each challenge entry on two metrics:

1. Overall merit based on technical skills, following the rules, quality of work, and “compliance to the spirit” of the challenge
2. Novelty and Dynamic Range of the author in their entry

The key aspect to the second category is that it cannot exist outside of the first category.  Any entry that fails on the technical aspects cannot win in the second category.  That said, from a collection of many entries that do conform to the expectations, there may be one story that is best “overall” in quality of work, while a separate story may stand out for particular novelty or insight on the part of the author.

The purpose of the CM challenges is only partially about a given theme or topic; for the most part, the challenges exist to encourage the author to do something they would not otherwise do.  This translates to advancing their mechanics, grammar, plot elements, ability to characterize, and so forth.  

To that end, we ask all judges to do two things:

1. Carefully read each entry and leave detailed reviews for each one.  These reviews should address not only the skill in writing mechanics displayed by the author, but should also comment on those things which were done well or poorly – characterization, use of cliché, unnecessary or abrupt plot points, etc.
2. After reading and reviewing all entries, submit a private email to the mediating judge containing ratings for each story in each category.  

The mediating judge tallies all votes and announces the final numeric score results to all judges.  In the case of a tie, the mediating judge may ask for a dialog of consensus or simply cast a tie-breaking vote without discussion.  Upon request, the mediating judge must email to all the judges the anonymized scores received along with their point tallies from each judge.  If there is concern remaining over the scores or final results, the mediating judge turns over the original emails to the Metafic site owner for verification purposes only.

When each challenge has been fully judged, the mediating judge or a non-competing administrator will announce the results as well as the list of judges who participated in that particular challenge evaluation.