Chenin Challenge scores just don't add up 26 January 2006

From Cassuis:

Working through my treasured Wine magazine in the ridiculous heat we are all trapped in I stumbled onto the Chenin Challenge. First things first, heartfelt congrats to both Tania and Ken for what I assume are AMAZING wines.

Secondly a even more heartfelt thought to Ken....

"Following the debate among the panellist, eight wines emerged as contenders to win the competition and these were re-tasted, with the panellists asked to rank the wines in terms of preferance to determine final placings and to agree on their final ratings."

Now I must state Wine magazine also mentions "star ratings were awarded on the basis of discussion rather than solely on arithmetical averages."

So five people are judging the Chenin Challenge, the arithmetical results according to the scores given in the magazine are as follows:

KWV Val du Chêne: average score:17.5
FMC Chenin : average score: 17.9

Guess FMC won that one ( but it doesn't count! )

So let's imagine the wines were discussed, I would guess if I rated a certain wine higher than a other, without knowing anything else about said wine (it was a blind tasting??) I will back the wine I rated the highest regardless of others’ points of view. (That is, if I rated my tasting skills, which I wouldn't really back!) From that point of view this is what should (could) have happened. Two panel members gave the wines identical scores ( including Martin Meinaart, the M in the FMC) One member gave the KWV the win, and TWO gave the FMC the win....

Please for the love of all that is holy explain this to me: on point averages the FMC wins, on simple preferance of a favourite the FMC got it again – or should we focus on the final part of the sentence ‘to agree on their final ratings’. Whose final rating is this we are talking about? After retasting the same wine all of four times, I would hope that the scores given in the Wine mag would be their final rating, on which I based all this senseless rambling.

[Wine magazine have been strangely silent on all this - and see also Neil Pendock's discussion of the issue on Winecoza – but fortunately Michael Fridjhon was more forthcoming and sent the following explanation to us as well as to Winecoza:]

From Michael Fridjhon:

The conclusion arises from a selective reading of the Challenge tasting report.

The Challenge finalists had been scored through two rounds – a screening and a second round elimination - where tasters were asked to use the 20 point system. The final round ignores all previous scores and is based on a ranking (where the judges in fact usually use a 100 point system to separate their preferences). The ranking is used to minimise loading which results from higher scoring judges.

The higher a wine ranks, the lower its final tally. If all the judges scored it first, this would result in a number of 5 – if last, a number of 40.

The Val du Chêne's final ranking tally was two lower than the FMC, and hence it won.

From Mark:
Perhaps I am missing the point, but ultimately how can these two wines be judged in the same light?One a super serious wine, one a good example of how to make 20 000 + cases and have consistency. I don't buy this, mainly because I am by nature a sceptic.

From Gert:
Mark, I believe a wine on a competition is like a photograph. Take the same photo an hour later - the light will be different etc. At least, a competition has a panel of judges tasting blind – unlike Platter ratings that retailers force down the throats of consumers these days. By the way, the sighted tastings in the Platter scored the KWV 3 stars and Ken Forrester 4 and a half. Perhaps some bias for reputation, it can be argued.

And a note from the editor:
Interestingly, of course, the KWV Val du Chêne 2004 scored a whole star less in last year's Challenge – also a very good score, of course. More interesting is that the price they quoted a year back was a mere R36 – it has gone up by a third since then!

From Christian Eedes [Deputy editor of Wine magazine:]

It is entirely legitimate that there were three rounds of tasting in the Chenin Blanc Challenge, with this merely attesting to the thoroughness of the process.

It is stated quite unequivocally in the tasting report that the places of the final eight were done solely according to a ranking with this supplied in detail. Therefore the KWV wine got a combined ranking of (1+1+3+4+5) = 14 while The FMC got (1+2+2+4+7) = 16. [The lower score reflecting higher rankings.]

Rightly or wrongly, final scores for the last eight wines were done after the ranking. On that basis, Michael Fridjhon and Martin Meinert had the wines equal, Dave Hughes and Marilyn Cooper had The FMC ahead of the KWV and I had the KWV ahead of The FMC. This was consistent with rankings even if the aggregate scores apparently puts The FMC ahead of the KWV (the two judges that scored the wines equal were previously obliged to separate them in terms of rankings).

By giving primacy to rankings instead of score, this meant that the wine that elicited the strongest opinion from the individual judges took the honours. In addition, it eliminated the impact of extreme scoring which results in undue loading one way or another. I concede that Wine magazine could have made this thinking much clearer.

It is understandable that there is some concern about Martin Meinert’s participation in the judging given that he is a high-profile producer of Chenin Blanc. Nevertheless, it is difficult enough to obtain qualified personnel to sit on panels without excluding winemakers.

I suspect that a lot of the controversy to do with this year’s Challenge results stems from a prejudice against the KWV and in favour of Ken Forrester and Martin Meinert as respected individuals in the local wine industry. However, I would suggest that it is long overdue that our major producer-wholesalers started making competition-winning wines and therefore it is something to be applauded that the KWV won.

To argue that the Val du Chêne’s large volumes (27 750 cases) somehow detracts from its quality is nonsensical: one of the world’s most iconic wines, namely Penfolds Grange, is a multi-regional blend made in quantities rumoured to be far greater than that of the KWV wine.

It is also intriguing that the aesthetics of the two wines have elicited no discussion: the KWV wine has an alcohol by volume of 13.06% and a residual sugar of 2.2g/l compared to 14.5% and 9.7g/l respectively in the case of The FMC. Who says panels only reward extravagant styles?

Lastly, the KWV Val du Chêne costs only R45 a bottle, a price which will allow the sceptics to open bottle after bottle while bemoaning just how wrong the Wine magazine panel got it…

From the editor:
Thanks, Christian. That does help clarify the 'how', if not entirely clarifying the 'why'. We are left wondering why this extremely elaborate procedure (with its apparently unfair elements] was necessary in this particular tasting, while it is not felt to be necessary in all the other panel tastings conducted for your magazine (as far as I know). Surely if you believe in the value of averaging scores you should believe in it consistently – or was there some particular need to mistrust the result of averaged scores here? And whether you are correct in suspecting an unreasonable bias in favour of small producers, I am not sure; but it is disingenuous to suggest that the concern about Meinert's participation is simply that he is a high-profile producer of chenin - although it could fairly well have been predicted that one of his wines, at least, would reach the final round. The problem is, surely, that his scores for his own wines were not discounted, despite the fact that he would (certainly should) have recognised his own wine in a line-up of eight: makes something of a mockery of your much-vaunted 'blind tasting' procedure, I'd have thought.

From Mark:
Finally. Thanks for the explanation, it now makes sense. Not sure if I am being referred to as one of the 'sceptics', but to clarify my point, I mentioned that KWV produces 20 000+ cases of this wine and its quality was consistent; perhaps I should have added good. Secondly, Christian mentions that the wines are completely different, sugar levels, alcohol – hence my point that these wines are poles apart, how in fairness can they be matched against each other?

From Jean-Pierre Rossouw:
Yes, I agree that the 'why' of the 'new' system is still blowing in the wind. As for Christian's surprise that there have been no comments on the results in relation to the wine styles, I offer the following. As someone who generally likes drinking a more elegant wine (especially when tackling more than a sip), I am happy to see such a style do well. I do not, however, think that this one result augurs a sea-change in wine competitions and the wines they reward. Furthermore, the questionable circumstances around this result render much debate on stylistics secondary.