Introduction
Although I aspire to be connected to a social world of sharing, sometimes a lot of the time I am out of the loop.
I am very grateful to everyone who nudges me back to lived reality.
Earlier this week, I spent a lot of time researching predictive models for the 2014 FIFA World Cup. I wrote a blog post about my discoveries. David Dormagen (2014) started me off and his paper coincided with my reading of Goldman Sachs’ predictions.
Simon Gleave tweeted:
@520507 Very interesting Keith. Infostrada has a model too. Goldman Sachs getting a lot of criticism in the analytical blogging world.
— Simon Gleave (@SimonGleave) May 29, 2014
I admire Simon’s work immensely and was mortified to think I was so disconnected. Simon is connected and his leads took me to James Grayson, Calcio Cassini and Steve Fenn in addition to Infostrada.
I have included links to their thoughts about the Goldman Sachs’ report as a Postscript to my post.
My reading of Dominic Wilson and Jan Hatzius’s analysis for the Goldman Sachs’ report remains twofold: I liked the transparency of their methodology; and I looked at the macro outcome of their predictions. I have no comment to make about the granularity of their predictions. Their macro view is shared by other models.
I appreciate the criticisms made of the report and this set me off thinking about a within and post-World Cup collaboration.
How will the community of practice of football analysts account for granularity of within game performance that leads to counter-intuitive (counter-probabilistic) outcomes?
In addition to a fascinating insight to collective wisdom, this collaboration has the potential to provide a wonderful opportunity to discuss negative evidence and critical incidents.
My own analysis of the World Cup will be triggered by a very basic question: did the higher Elo ranked team score the first goal in the game?
The answer to that question will enable a closer look at what happened if the higher ranked team loses. My expectation is that a higher ranked team that scores first will not lose (it will win or draw). I hope to explore any negative evidence and critical incidents that lead to a counter-predictive outcome.
In 2010 I used FIFA Rankings to ascribe status to teams. 51 of the 64 games in the 2010 World Cup were won by the higher FIFA ranked team (the exceptions were: Serbia (v Ghana and v Australia); Cameroon (v Japan and v Denmark); France (v Mexico and v South Africa); Greece (v Korea); Spain (v Switzerland); Germany (v Serbia); Italy (v Slovakia); Denmark (v Japan); USA (v Ghana); and Brazil (v Netherlands). Nine of these eleven defeats were in group games. USA lost to Ghana in the Round of 16 and Brazil lost to the Netherlands in the Quarter Final.
Negative Evidence
I have had a fascination with negative evidence ever since one of my tutors at the LSE introduced me to George Lewis and Jonathan Lewis’s (1980) paper The dog in the night-time: negative evidence in social research. The title of the paper was inspired by a Sherlock Holmes’ story, Silver Blaze. In their introduction they note:
In social research, there is an overwhelming emphasis upon collecting positive data, whether it be in the form of statistically significant attitudes, important documents, or observer descriptions of unique settings. This emphasis, while responsible for shedding much light in previously dark areas, none the less has had the important and dangerous side-effect of minimizing the worth of negative evidence, that is, the significance of a thing’s absence.
George and Jonathan have a typology of negative evidence. I think four of their seven types have direct relevance to us:
- Type 1: Events do not occur (non-occurrence of events which were either expected to occur or not to occur)
- Type 5: The effects of the researcher’s idea set (preconceived notions of where to look and what sorts of data to look for can create blind spots)
- Type 6: Unconscious non-reportage (a researcher honestly feeling that deviant cases as observed are insignificant and not worthy of being reported)
- Type 7: Conscious non-reportage (researchers leave out themselves as part of the research context)
They conclude their paper with this summary:
negative evidence is data that either is (1) the non-occurrence of events, (2) an occurrence that is not reacted to or not reported (because it is outside the frame of reference of the population or of the researcher, or (3) although noted in its raw form, distorted in its interpretation or withheld from analysis and report.
Critical incidents
Back in 1954, John Flanagan wrote a definitive paper about ‘critical incident technique’. He clarified the concept thus:
By an incident is meant any observable human activity that is sufficiently complete in itself to permit inferences and predictions to be made about the person performing the act. To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.
When I first read the paper, I was struck by the research evidence used to develop the technique:
- Analyses of failure to learn to fly in the US Aviation Program (1941)
- Failures of bombing missions (1943-44)
- Combat leadership (1944)
- Disorientation in flying (1946)
Critical incident technique includes measurement of typical performance and the observation of operational procedures.
The paper concludes with this suggestion:
It should be emphasized that critical incidents represent only raw data and do not automatically provide solutions to problems. However, a procedure which assists in collecting representative samples of data that are directly relevant to important problems such as establishing standards, determining requirements, or evaluating results should have wide applicability.
Conclusion
I think two academic papers, one from 1954 and the other from 1980, have some great insights to share today. For many years, performance analysts have engaged with a process that makes a permanent record of performance that enables description, analysis, modelling and prediction. The final part of this process for me is the transformation of performance by coaches and athletes.
We have an increasing amount of insight into performance. The 2014 World Cup is a great focus for analytical effort and elegance.
It is fascinating what we could achieve together as we reflect in and on the World Cup.
Photo Credits
Train Crash at Montparnasse (Robert Wade, CC BY-NC-SA 2.0)
Gol/Goal (Antonio Martinez, CC BY-NC-SA 2.0)
Ready for the heaven (Cristian, CC BY ND-NC 2.0)
