Judging Process: It's complicated!

Participating clubs sometimes wonder why the judging process takes several months. The bottom line is that the process is 1) a lot of work and 2) many complicated steps.

The process has evolved over the years from being a completely manual, paper-driven process to one that is now fully electronic. Still, there are many manual checks along the way.

We thought we'd share the process of preparing and judging entries.

Checking for Accuracy

We try to be accommodating to clubs who are missing items from their entries. After receiving all of the entries from clubs, they are checked for accuracy. 

We make sure that we receive an Excel Entry Manifest from each club. Each year, some clubs forget to send this along or send it in the wrong format.

We also check to see that the correct version of the entry form is used for each entry. It is not uncommon to find that the entry form is missing or incorrect. For example, in 2013, one club did not include their reprint policy on the submission form.

Next, we check the entries against the manifest to ensure that all of the entries are present. If any are missing or extra entries are included, we contact the club. 

We generally uncover some mistakes here. One common error is submitting separate entries in the Best Publication categories, instead of combining all issues together. FAAS generally makes minor corrections for mistakes such as these. 

Global Manifest Creation

The Excel manifests from all participating clubs are combined to make a single manifest for judging. There are usually some issues in this process. Not all clubs use Excel and sometimes clubs make modifications to the template which can cause issues when cutting and pasting.

The result, when completed, a list of all of the entries submitted, by category. For proper sorting, FAAS adds leading zeros to the category number.

Entry Preparation Process for Judging

In order to share the entries with the judges, each category is combined into a single PDF before it is sent for judging. At this point, you need a good understanding of the PDF file format as this process is a bit complicated.

The first step is stamp each entry with its file name. We do this so that we can immediately identify the file name associated with an entry from any page of the entry. We use an Acrobat Action for this.

Since each entry contains a fillable form, combining them will corrupt the form data. For this reason, all of the PDFs are first flattened. The Flattening process makes the forms non-editable, moving the text in the form into the non-editable layer of the PDF. An Adobe Acrobat Pro action is used to batch process the files.

Once the entries are flattened, all the entries for each category are combined. Since the files can get very large, we use Acrobat's Reduce File Size command to make them more manageable. The combined PDFs are sent to the judges for their review.

Recruiting Judges and the Rules

Most years we have three to five judges who participate in reviewing entries. Fairness is important.

FAAS rules state that no judge may review a category which contains entries from a club in which he/she is a member. Thus, we carefully screen judges and only assign them to eligible categories.

Judging the Entries

The judges review the Category PDFs in either Adobe Acrobat or Adobe Reader. Judges affix an Acrobat sticky note on top of the submisson form denoting either First, Second, Third or Honorable Mention. A few thing to consider for clubs about the process:

While judging is subjective, the main criteria considered are:

Gathering the Results

Judges may elect to send the entire, marked category file back to FAAS, but most often they simply report the winners via email such as:

Hopefully, you can see why naming the entries is so ciritcal to the accuracy of this process.

Finalizing the Winners in Excel

Once judging is completed, the winners are indicated on the combined manifest in Excel. We place 1, 2, 3 or HM in the Place column.

From there, we create several other tabs in the spreadsheet:

Generating the Certificates

We use Microsoft Word to mail merge the spreadhseet data into the certificates, We preview the data in Word and occasionally find issues, so at that point it is back to Excel to correct any errors.

Once we are happy with the preview, we generate a merged Word file from the data. We always have to make a few adjustments to the text size for very long article or author names which cause line breaks. Once complete, we generate a PDF of the certificates. In any given year, there are usually between 90 and 120 certificates created.

FAAS institued a change for the 2013 Publication Year. Instead of printing the certificates and mailing them, we now generate them electronically. We use Adobe Acrobat's Background function to add a fancy border to each page of the PDF. 

In Acrobat, we bookmark the first certificate for each club, then use Acrobat's Split by Bookmark function to create each society's certificates.

Sending the Certificates to Clubs

In Excel, we create a spreadsheet for each club's contact first and last name, society name and email address. In Word, we create a mail merge from this data and mail merge to Outlook in draft mode. Each clubs certificates are then attached to the draft message and sent out.

Winning Entry Preparation for Website

The next step is to post the winning entries on the website. One challenge we have is that the maximum upload size for our document database is 3MB and some entries exceed that signficantly. We use a variety of PDF compression processes to get around this. Sometimes, we have to save the document as an image to reduce its size.

Since the entry forms contain some personal information, we use Acrobat's redaction features to remove it. This includes email addresses for contacts, etc.

Due to file size and complexity, we only post the first place winners. We use a file renaming tool to rename the winners to year_category number format. So, the 2013 category six winner would be: 2013_006.pdf.

Posting to the Website

We create a new article on our website which lists the winning entries in a table. Some fiddling is involved so that the table displays well in our Joomla Content Management System.

We use the Phoca Download component to manage downloads on our website. We create a new category in Phoca (e.g. 2013 Winners), then bulk upload the winning files. Each upload them must be renamed (e.g. 2013 Category: 006) so that it will sort correctly. 

We then add a link to the Winners page so that folks can download the winining samples.

Final Thoughts

All in all, it takes 20 to 40 hours of work just to manage the judging process. This does not include the time that judges take to review entries. But, it's all worth it. Each year we continue to be impressed by the impressive output of writers and editors whose work enhances our hobby.