In the last post we set out the problem of defining analytical 'confidence'. In this post we'll define what 'analytical confidence statements' are, with some examples.
Probabilistic Judgements
Statements of analytical confidence attach levels of confidence to probabilistic judgements. A 'probabilistic judgement' is a probability attached to a hypothesis, such as:
"There is a 90% chance that it will rain tomorrow."
Probabilistic judgements sometimes use words to express probabilities, but they are probabilistic judgements nonetheless:
"It is extremely likely that human influence has been the dominant cause of the observed warming since the mid-20th century." (IPCC (2013), Climate Change: the Physical Science Basis)
An assessment that something is certain or certainly false is still a probabilistic judgement, albeit one with probability 1 or 0:
"What I believe the assessed intelligence has established beyond doubt is that Saddam has continued to produce chemical and biological weapons." (UK Government (2002), Iraq's Weapons of Mass Destruction)
So a 'probabilistic judgement' is a proposition, or statement, along with some measure of the probability which is assigned to it.
Confidence Statements
The assignment of confidence levels to probabilistic judgements is thus a further step down the road of qualifying a proposition. It involves attaching a confidence level to a probabilistic judgement, with some intended effect on its force. For example:
Photo: Luca Galuzzi "The frequency of heavy precipitation events (or proportion of total rainfall from heavy falls) has increased over most areas (likely). Globally, the area of land classified as very dry has more than doubled since the 1970s (likely). There have been significant decreases in water storage in mountain glaciers and Northern Hemisphere snow cover. Shifts in the amplitude and timing of runoff in glacier- and snowmelt-fed rivers, and in ice-related phenomena in rivers and lakes, have been observed (high confidence)." (https://www.ipcc.ch/pdf/technical-papers/ccw/executive-summary.pdf)
In case it isn't clear here, the 'high confidence' at the end is intended to apply to the paragraph as a whole, including the statements flagged as 'likely'. (The IPCC's guidance on confidence levels is here.)
Here is an example from a US National Intelligence Estimate:
"We continue to assess with low confidence that Iran probably has imported at least some weapons-usable fissile material, but still judge with moderate-to-high confidence it has not obtained enough for a nuclear weapon."
or, from a declassified paper released by the UK's Joint Intelligence Committee (JIC):
"Against that background, the JIC concluded that it is highly likely that the regime was responsible for the CW attacks on 21 August. The JIC had high confidence in all of its assessments except in relation to the regime’s precise motivation for carrying out an attack of this scale at this time – though intelligence may increase our confidence in the future."
Things that Look like Confidence Statements but Aren't what We're Talking About
There are a number of constructions which look like confidence statements, but which are not what we're talking about here. Sometimes the language of confidence is used to make straightforward probabilistic judgements. This kind of thing is quite common and includes statements like "I am confident it will rain" when this merely means something like "it is likely to rain". Problematically, some analytical organisations actively recommend using confidence terms to express probability. The extent to which probability and confidence are intertwined is something we'll look at shortly, but for clarity we are not thinking here about statements of probability disguised as confidence statements.
Confidence language is also sometimes used to express the credibility of a source of information. This is the kind of confidence used to express reliability of or doubt about some evidence. Ratings of this kind are used routinely in intelligence to express qualitative judgements about the nature of sources, but the idea also has a correlate in the information-theoretic concept of channel capacity. There is a tempting symmetry in the following statements which might make us think that 'source confidence' is related to the kind of confidence we are discussing here:
"I am reasonably confident that there is an 80% chance of rain tomorrow."
versus
"I am reasonably confident in Michael's judgement that there is an 80% chance of rain tomorrow."
However, we should not give in to this temptation. The two statements are very different. The second statement lends itself to a straightforward probabilistic interpretation: it suggests that there is some probability (your 'reasonable confidence') that Michael is in good command of his evidence and that his assessment of an 80% chance of rain is well-calibrated, but that there is also a probability that his assessment is poor and that some other probability (e.g. a base rate) would be more appropriate for decision-making purposes. In theory the probabilities could then be combined (by you) into a final probability that would take account of Michael's judgement (i.e. as a piece of evidence) but not perhaps coincide with it.
But this is not an interpretation we can meaningfully use for:
"I am reasonably confident that there is a 80% chance of rain tomorrow."
unless the speaker is being particularly obtuse. It is tempting to think that by making the statement above, you are specifying (part of) a probability distribution over a set of probabilities, and that you might clarify and continue in the following sort of manner:
"By 'reasonably confident', I mean that there is a 60% probability that that judgement is correct. So what I mean is that there is a 60% chance that rain is 80% likely, and I also think there is a 40% chance that rain is instead 20% likely. There is therefore a 60% x 80% + 40% x 20% = 56% chance of rain tomorrow."
This all sounds plausible. However, for reasons we will set out later on, it cannot be given a coherent interpretation as meaning anything other than:
"There is a 56% probability of rain tomorrow."
The point here is that while there is a meaningful (but straightforward) interpretation for statements about confidence in third-party probability judgements (like Michael's) this interpretation cannot be made meaningful for statements about one's own judgements.
In Summary
The kind of statement we are interested in is one in which a confidence level is attached to a probability that is attached to a hypothesis, or, generically:
"I have confidence C that hypothesis H has probability P."
In the next post, we'll take a detour into statistical confidence intervals, primarily to clarify why the kind of confidence we're talking about here can't straightforwardly be interpreted using them.