Complexity and Models

Mark Upton shared two links recently that set me off looking at complexity and models.

John Launer (2018) observes “the study of complex adaptive systems (CAS), also known as complexity science, is burgeoning”. In his article, he proposes “the idea that the fundamentals of complexity are in fact extremely simple” and suggests that “complicated descriptions of complexity may fail to capture its most important qualities, and that simple ones, especially those that use metaphor and appeal to intuition, may be better ways of doing so”.

John asks “in a world where prediction can never be certain, are there nevertheless some general rules that can reduce uncertainty, so that our actions stand a better chance of achieving their intended results?”.

John adapts Jeffrey Braithwaite and his colleagues’ (2017) consideration of complexity science in healthcare to suggest ways to promote complexity thinking. These include:

  • Resisting the temptation to focus on an isolated problem and looking for interconnections within the system.
  • Things happen when you least expect them.
  • Looking for patterns in the behaviour of a system, not just at events.
  • Keeping in mind the system is dynamic, and it does not necessarily respond to intended change as predicted
  • Drawing up a model of the system surrounding a problem.

(Note: in their paper, Jeffrey Braithwaite et al (2017:3) observe “Complexity refers to the density of interactions between different components (agents, parts, elements, artefacts) in a system or a model representing a system, and which produce roles and behaviours that emerge from those interactions. Complex Systems are rich in collective behaviour”. They make a distinction between  complicated (“a lot going on with all the components”) and complex (interrelatedness, emergent behaviour, self-organisation and dynamics).

Jeffrey and his colleagues make use of and adapt Thomas Kannampallil and colleagues’ (2011) visualisation of complex and complicated:

I liked John’s view of interrelatedness: “Human groups engaged in an endless dance of mutually responsive interactions, in which everyone including yourself plays a part”.

The second link from mark introduced me to Joshua Epstein’s (2008) lecture Why Model? The lecture distinguishes between “explanation and prediction as modeling goals, and offers sixteen reasons other than prediction to build a model”.

Joshua notes “the choice, then, is not whether to build models; it’s whether to build explicit ones. In explicit models, assumptions are laid out in detail, so we can study exactly what they entail. On these assumptions, this sort of thing happens. When you alter the assumptions that is what happens. By writing explicit models, you let others replicate your results” (2008: 1.5).

16 Reasons

  • Explain (very distinct from predict)
  • Guide data collection
  • Illuminate core dynamics
  • Suggest dynamical analogies
  • Discover new questions
  • Promote a scientific habit of mind
  • Bound (bracket) outcomes to plausible ranges
  • Illuminate core uncertainties.
  • Offer crisis options in near-real time
  • Demonstrate tradeoffs / suggest efficiencies
  • Challenge the robustness of prevailing theory through perturbations
  • Expose prevailing wisdom as incompatible with available data
  • Train practitioners
  • Discipline the policy dialogue
  • Educate the general public
  • Reveal the apparently simple (complex) to be complex (simple)

These encapsulate the freedom to doubt (Feynman, 1955). Joshua concludes:

the most important contribution of the modeling enterprise—as distinct from any particular model, or modeling technique—is that it enforces a scientific habit of mind, which I would characterize as one of militant ignorance—an iron commitment to “I don’t know.” That is, all scientific knowledge is uncertain, contingent, subject to revision, and falsifiable in principle. (This, of course, does not mean readily falsified. It means that one can in principle specify observations that, if made, would falsify it). One does not base beliefs on authority, but ultimately on evidence. This, of course, is a very dangerous idea. (2008: 1.16)

Both of the references shared by Mark are excellent prompts to reflect on how we address the interrelatedness of sport behaviours. I thought Joshua’s help in sharing openly what we do juxtaposed two fascinating second order conversations: ‘freedom to doubt’ and ‘militant ignorance’.

Photo Credit

Robin Poke’s PhD Submission: A Narrative History of Australian Rowing

Robin Poke holding his PhD Submission

Robing Poke submitted his PhD thesis for examination today at the University of Canberra. It is the culmination of six years assiduous research that is titled A Narrative History of Australian Rowing 1770-2016.

I have been fortunate to be Robin’s primary supervisor.

I believe it to be a magnum opus in the history of rowing. It extends to two volumes and shares some remarkable primary sources to build the narrative.

The abstract is:

This thesis describes in detail the beginnings, development and progress of rowing in Australia through fifteen chapters that set out chronologically how the sport transitioned from the days of settlement, the early watermen, and to the 19th century and the onset of professional sculling. Then came, in the 20th century, the era of pure amateurism before, given the massive funding in contemporary sport, it reverted at the very least to the semi- professional level.

The initial chapters describe the early use of boats by settlers and the exploits of the earliest professional scullers, who captured the imagination not just of the citizens of New South Wales but of all the colonies. Then comes the rapid expansion of rowing and sculling at all levels: club, colonial and national, and the onset of the amateur ideology. The transition from inter-colonial to inter-state competition is described, as is the emergence of women’s rowing. Then comes Australia’s growing involvement at the international level between the two world wars. The retirement of professional sculler Bobby Pearce and the eventual decline of professional sculling are discussed.

A continuing swing away from amateurism towards at least semi-professionalism is seen. Also described is the improvement in the administration of national rowing, at the hands, initially, of John Coates, assisted by John Boultbee. Australia’s first professional Director of Coaching, Reinhold Batschi is introduced.

An extraordinary decade in the history of Australian rowing arrives, during which the sport experiences hitherto unforeseen success and at the end of which hosts an Olympic Regatta. At the heart of this success are the stunning results obtained by a crew that had become known as the Oarsome Foursome.

The period between the celebrating of a successful ‘home’ Olympic Games in 2000 and the London Olympic Games in 2012 is described. In the interim were the Athens 2004 and Beijing 2008 Games. The thesis ends with a discussion about Rowing Australia’s high performance plans for the future of rowing and contemplation about the process of writing a narrative history of rowing.

Robin at the Graduate Office at UC handing in his thesis

We await with great interest the external examiners’ responses in 2019.

Photo Credits

Robin Poke (Keith Lyons, CC BY 4.0)

Forecasting, predicting, classifying and uncertainty

This is a post to share my bumping into work by Glenn Brier, Frank Harrell and David Spiegelhalter. It coincides with an email exchange I had with Tony Corke about how to share posterior outcomes in the context of prior statements of probability.

I saw a reference to a Brier Score in David Glidden’s (2018) post on forecasting American football results (link) and followed up David’s reference to a Brier score in Wikipedia (link). This encouraged me to seek out some of Glenn Brier‘s papers to find the origin of the score named after him.

An early paper was written in 1944 for the United States Weather Bureau. It was titled Verification of a forecaster’s confidence and the use of probability statements in weather forecasting (link). In the introduction to that paper, Glenn observed “one of the factors that has contributed to the difficulties and controversies of forecast verification is the failure to distinguish carefully between the scientific and practical objectives of forecasting” (1944:1). He proposed that:

  • The value of forecasts can be enhanced by increased use of probability statements.
  • The verification problem can be simplified if forecasts are stated in terms of probabilities.

He added “there is an inherent danger in any forecast if the user does not make use of (or is not provided with) the pertinent information regarding the reliability of the forecast” (1944:7). The sharing of information about the reliability of the forecast makes it possible to provide recommendations for action. Glenn concluded “the forecaster’s duty ends with providing accurate and unbiased estimates of the probabilities of different weather situations” (1944:10).

In a paper written in 1950, Verification of forecasts expressed in terms of probability, Glenn provided more detail about his work on probability statements and presented the details of his verification formula (link). He proposed ” perfect forecasting is defined as correctly forecasting the event to occur with a probability of unity” with 100 percent confidence (1950:2).

The 1950 paper raised the question of skill in forecasting. A decade later, Herbert Appleman (1959) initiated a discussion about how to quantify the skill of a forecaster (link). Glenn’s 1950 paper prompted Allan Murphy (1973), amongst others, to look closely at vector partitions in probability scores (link). Some time later, Tilmann Gneiting and Adrian Raftery (2007) considered scoring rules, prediction and estimation (link).

Frank Harrell fits into this kind of conversation in a thought-provoking way. Earlier this year he wrote to distinguish between classification and prediction (link). He proposes:

Whether engaging in credit risk scoring, weather forecasting, climate forecasting, marketing, diagnosis a patient’s disease, or estimating a patient’s prognosis, I do not want to use a classification method. I want risk estimates with credible intervals or confidence intervals. My opinion is that machine learning classifiers are best used in mechanistic high signal:noise ratio situations, and that probability models should be used in most other situations.

He concludes:

One of the key elements in choosing a method is having a sensitive accuracy scoring rule with the correct statistical properties. Experts in machine classification seldom have the background to understand this enormously important issue, and choosing an improper accuracy score such as proportion classified correctly will result in a bogus model.

There is a reference in Frank’s paper to risk (“by not thinking probabilistically, machine learning advocates frequently utilize classifiers instead of using risk prediction models”) and a link to a David Spiegelhalter paper written in 1986, Probabilistic prediction in patient management and clinical trials (link). In that paper, David argued for “the provision of accurate and useful probabilistic assessments of future events” as a fundamental task for biostatisticians when collaborating in clinical or experimental medicine. Thirty-two years later, David is Winton Professor for the Public Understanding of Risk (link).

In 2011, David and his colleagues (Mike Pearson and Ian Short) discussed visualising uncertainty about the future (link). They describe probabilities best treated “as reasonable betting odds constructed from available knowledge and information”. They identified three key concepts that can be used for evaluating techniques to display probabilistic predictions:

  • Common sense and accompanying biases
  • Risk perception and the role of personality and numeracy
  • Type of graphic presentation used

David and his colleagues summarise their thoughts about visualising uncertainty in this box:

David returned to the theme of uncertainty in 2014 and suggested “it will be vital to understand and promote uncertainty through the appropriate use of statistical methods rooted in probability theory” (link). Much of David’s recent work has focussed on the communication of risk. At the Winton Centre:

All too often, numbers are used to try to bolster an argument or persuade people to make a decision one way or the other. We want to ensure that both risks and benefits of any decision are presented equally and fairly, and the numbers are made clear and put in an appropriate context. We are here ‘to inform and not persuade’. (Link)

All these thoughts were running through my head when I decided to contact Tony Corke. I admire his work immensely. A couple of rapid emails helped me with a Brier Score issue for priors and posteriors from my Women’s T20 cricket data. I am not blessed with mathematical intelligence and Tony was very reassuring.

I am now off to research Solomon Kullback and Richard Leibler who were writing a year after Glenn Brier’s 1950 paper (link).

Photo Credits

Foggy mountain by Eberhard Grossgasteiger on Unsplash

White clouds and blue sky by Paul Csogi on Unsplash

Sailing on the ocean by Andrew Neel on Unsplash