Methods for the Analysis of Choices and Evaluations

Notes on uncertainty, probability, risk and Options Analysis

  1. Models of appeal and choice

  2. Options Analysis methods

  3. Testing predictions of choice and share and the role of uncertainty

  4. The nature of appeal uncertainty

  5. How are risk and uncertainty related?

  6. Sources of appeal uncertainty

  7. Relations between an individual's option appeal distributions

  8. Measuring probabilistic decision-making

  9. The ambiguity of probability and the need for appeal diagnostics

  10. Options Analysis - current applications and development prospects

Models of appeal and choice

These notes deal with the methods currently applied in Options Analysis and how they relate to:

  1. uncertainty in evaluations,

  2. option risk, and

  3. probability of individual choices.

Related issues involve the links between measurement models and research objectives, precision requirements, and measurement costs.

There is some discussion of analysis of a single individual's decision-making as distinct from the analysis of individuals composing a group. Links related this discussion are:


Options Analysis methods

Options Analysis methods are capable of analysing complex and intuitively unpredictable situations involving option advantage and share within groups.

The general rule for inferring individual choice involves:

  1. Collection of point-measures of appeal from each individual for each option.

  2. Comparison of appeal levels amongst the options in each individual's selection set.

  3. Selection of an individual's most appealing option as his or her option of choice.

Methods for measuring appeal for Options Analysis application are comparatively refined. Yet, at times, more than single-point measures of option appeal are needed to fully understand the factors underlying choice and share.

The main reason for obtaining multiple measures of each option's appeal by an indvidual is to study the effects of uncertainty on individual decision-making. These effects might then be aggregated to evaluate impacts on share of choice at the group level.

By their nature, studies including uncertainty are more demanding, costly and complex than studies limited to single-point measures of appeal.

Options Analysis does not currently include a method that incorporates the effects of uncertainty on share of choice within groups except insofar as uncertainty affects the general evaluation of each option. (Uncertainty may be seen as a self-referential criterion for degrading or enhancing the global evaluation of an option's appeal.)

Nevertheless, it is possible to analyse the impacts of uncertainty for single individuals. (Using this approach, it may be practical to aggregate individual results for groups of up to, say, 10 people. So, small group studies are possible that include explicit consideration of uncertainty.)

Disregarding the practical problems, it is of some importance to appreciate the role of appeal uncertainty in individual decision-making and how this might affect group results.


Testing predictions of choice and share and the role of uncertainty

A validation test involving a comparison over all respondents of each inferred choice with each corresponding most likely choice intention, will show to what extent current Options Analysis methods can provide an acceptable basis for further data analysis.

For example, comparing the 1996 US NES presidential ratings with voting intentions, about 95% of the concurrently stated choices are predicted based on the inferred choices. (This analysis is included in the Options Analysis download package.) These results would seem to provide a reasonable basis for proceeding to interpretation of Options Analysis results without being overly concerned with the effects of uncertainty.

With a simple extension of this validation procedure, by including an additional survey questionnaire item it is possible to assess the extent to which the choice outcomes, specifically share of choice results, are affected by uncertainty. This involves separate measurement of immediate choice intention and the most likely choice intention. Differences between the shares of choice based on these two measures will suggest the degree of appeal uncertainty. For more details click here.


The nature of appeal uncertainty

Appeal uncertainty refers to variability in attractiveness of an option to an individual. We know from experience that an option's choice may result in variety of outcomes and that each may have a different level of appeal. Taken together the appeals of all outcomes associated with an option by an individual compose his or her appeal distribution for that option.

We refer to options associated with differing levels of variation as "conditional", "unreliable", "dependable", "unpredictable", "uncertain", "trustworthy", "creative", "novel", "risky", "exciting", and so on, as applicable to the nature of the option.

The full appeal distribution for an individual summarises the range and incidence of possible option appeals associated with an option. The distribution may be specified by its moments (mean, standard deviation, skewness and kurtosis). Its correlations with other option uncertainty distributions can also be important in determining choice.

The single measure of appeal for each option for each respondent, which is currently input to Options Analysis, may be viewed as the mean of each individual's appeal distribution.

The uncertainty distribution reflects previous experience and current perceptions relevant to an option. It is conceptually analagous to generalisation effects found in the psychology of learning.

When an individual makes a choice selection the uncertainty distribution collapses to a single appeal scale point. The point selected will vary from occasion to occasion depending on perceptions of the specific circumstances.

The individual option appeal selection process may be better understood by considering the uncertainty distribution as a population of option scenarios. From amongst these the currently most appropriate choice scenario, with its associated appeal, is identified by the individual.

For example, consider an option of your visiting a doctor. You may readily assign an appeal value to such an option. You could also be able to add a range of appeal values reflecting health visits for a variety of health conditions and in differing circumstances with varying degrees of unpleasantness or otherwise. These appeal evaluations may form the basis of your visit uncertainty distribution. The procedure described above suggests that the appeal of your next visit, would more reflect the appeal of a similar scenario than any other point on the distribution. Depending on the appeal of alternative options, this variation from the mean appeal of the general option of visiting a doctor, may or may not make any difference to whether you do actually choose to go to a doctor. Sometimes appeal uncertainty will make a difference to choice and at others it will not.


How are risk and uncertainty related?

The concept of risk has no formal role in Options Analysis. However, it may be seen by individuals as a noteworthy attribute of specific options. For example, some options have all-or-nothing or strongly categorised outcomes. A gamble based on a coin toss will have two well defined results. A lotto ticket delivers a list of formal outcome categories. Equally, risk may be associated with, say, getting a good meal at a restaurant, or, enjoying a night out at the movies. Such events may turn out poorly or well. Along with other perceived option attributes, risk can clearly influence appeal.

Evaluations of appeal uncertainty incorporate perceptions of risk. In addition, however, perceptions of risk will also affect an individual's evaluation of an option's general level of appeal. When considering buying a house, perceptions of significant risk are likely to reduce purchase appeal as well as increase uncertainty. On the other hand, the option of an "adventure sport" holiday with perceptions of possible danger may be seen as more attractive than a holiday seen as 100% safe. Here, increased risk may be additionally associated with an increase in general level of appeal.

While a risk may be objectively agreed as the same for all, the effects of risk on evaluations of both appeal and appeal uncertainty may be expected to differ from individual to individual. In particular, uncertainty perceptions attributable to risk will be affected by individuals' sense of control over an option. This, in turn, will be affected by such factors as previous experience, relevant skills, and self-confidence.


Sources of appeal uncertainty

From the view of suppliers of products, services, and so on, uncertainty should be regarded as an additional parameter to those analysed in the appeal point-measurement model. Correspondingly, it is amenable to analysis of its sources and subject to consideration for management. Suppliers of goods and services are important sources of manageable uncertainty.

Here are some classes of individual uncertainty:

Here are a few factors, by way of example, which affect appeal uncertainty. Of course, uncertainty may equally stem from factors or aspects which are very option specific or shared between sub-sets of options. Common influencing factors can affect different options differently.

Distance from choice action: When the time between choice intention and choice action is small there will be more certainty than for the same option some distance in the future. Proximity of choice intention and choice action will usually mean that the circumstances surrounding the choice situation will be better defined.

Familiarity with options: Familiarity involving experience of an option may markedly reduce uncertainty. But increased familiarity can also increase uncertainty, for example, in the case of an unreliable service. Advertising along with other forms of information supply are obvious means for managing uncertainty associated with unfamiliarity.

Inherent character of option context: Any option perceived as affected by variable uncontrolled conditions will have variable appeal. Here are a couple of examples. The potential variability of weather will generate a range of appeals for a picnic. Fine weather will make a picnic more appealing and wet weather less. Consider the appeal of shares as an investment and the general influence of varying states of the economy on their performance.

Fluctuations in taste: The most obvious associations here are with food but the principle applies in all areas of human activity due to such factors as drive satiation, curiosity and the desire for novelty. Switching between tea and coffee, and different styles within these are examples which can be elaborated across many foods and beverages. The onset of such fluctuations can be difficult to predict and generate uncertainty as a result.

Difficulty of option performance: Here there is doubt about the level of realisation due partly to human skill factors. The varying prospects of a fish catch can be expected to generate a range of appeal uncertainties for a fishing expedition. In considering employment opportunities, the varying levels of job challenge introduce uncertainty, with more challenging jobs offering more variable chances for success and failure.

Personality and background of individual: Different individuals may be expected to vary in their sensitivity to uncertainty and reflect this in their appeal assignments. This will systematically affect the role of uncertainty mainly through individual differences in appeal disparities. Its effect is most likely to be evident by comparison amongst individuals since its impact will be mainly through interaction with other sources of uncertainty.


Relations between an individual's option appeal distributions

As is clear from description of factors influencing uncertainty, there is no automatic expectation that the relations between each individual's uncertainty distributions will be random. Sources of appeal uncertainty may be common to a selection set. For example, weather may be a common factor in an option selection set consisting of outdoor sports. If sources of uncertainty impact options in a uniform fashion, then those option appeal distributions will be positively correlated. (In such cases the role of uncertainty in selecting between options is reduced.)

In practice, an individual's appeal distributions will be influenced by general, shared and unique factors. Consequently, standard deviations, skewness, kurtosis of options and correlations between an individual's option evaluations can all be expected to vary.

Just as the uncertainty distribution for each option is seen as composed of appeal scenarios for that option, so the joint distributions across the options may be viewed as composed of selection sets of scenarios. Selecting the option with the highest appeal within each selection set, summing the choices across the sets, and calculating proportions of the total number of scenarios, will yield probabilities of selection for each of the options for an individual. The implication is that for a given group of possible choice occasions a given proportion of selections would be made of one option, a given proportion of another, and so on.

Choice probability is a statistical device of the analyst for summarising all choices amongst the scenario sets. It is used to derive a picture of relative selection frequencies over choice replications. It is not suggested that probability is calculated by a chooser in the course of making a choice. That would lead the chooser, logically, to select the most probable appropriate choice over all replications. That would in turn result in answers identical to those from using standard Options Analysis without uncertainty data. (The reason for this result is that the usual single-value measure of appeal may be regarded as the mean of an underlying uncertainty distribution. The highest option mean appeal will be almost always be associated with the highest probability of selection.)

Any specific choice of an individual cannot inferred from the appeal uncertainty distributions (except where there is unanimity of choice over all scenario sets). Rather the choice process under uncertainty would seem to involve collapsing each uncertainty distribution to the particular option appeal scenario seen as most relevant at the point of choice. Appeal amongst the options in this reduced set can then be compared in the usual way. In this process no computation involving probabilities is needed or relevant.

Given sufficient choice replications under a given individual's appeal uncertainty distribution, option selections will be proportional to the number of selection sets of option scenarios where that option has maximum appeal. Such results would be consistent with the Matching Law.

One might expect a probabilistic approach incorporating uncertainty to give markedly superior results to those which would be obtained using the standard Options Analysis methods which do not incorporate uncertainty distributions. The answer depends to some extent on the purposes of the research and the nature of the specific option uncertainties.

Even where uncertainty is an important determinant of choice, validity assessments, using the current quasi-deterministic model may commonly yield satisfactorily high levels validity. Options Analysis inferred choice is invariably the most probable choice. So, when a respondent's most likely stated choice is used as the test criterion, high levels of validity will usually be expected. One may then reasonably proceed to use the range of current Options Analysis methods to evaluate inter-option relationships and their consequences for advantage and share of most likely choices.

If, however, an important objective is the accurate estimation of shares of choice a probabilistic estimate will be more accurate. In focusing on "most probable next choice", secondary choices will usually be understated relative to all choices. Thus, asking for a respondent's immediate preference is more likely to elicit an option other than the most probable. (Even here some caution is required since some respondents may assume that "Which one would you choose right now?" means most common choice simply because of the questionnaire context of the question. Some question qualification may be needed for emphasis.)

It is common for discussion on decision-making to focus on uncertainty as the key consideration in choice. The present framework provides no general basis for this position. In particular, such a view ignores the potentially critical role of differences between the average appeal of options. If the differences between means of the most preferred options are large, then even large uncertainties may not matter. Furthermore, positive correlations between uncertainty distributions of options in a selection set also minimise the effects of uncertainty on choice. On the other hand, small levels of uncertainty may play a large role if differences in mean appeal of options are small. This makes it clear that statements of choice probability are diagnostically ambiguous. Useful insights will require supporting information about at least the dispersion size of appeal uncertainty distributions, plus the mean option appeal separations and correlations.

Furthermore, it must be said that uncertainty may merit consideration on account of its direct effects on reducing or enhancing individual levels of appeal. This may quite possibly be the more important aspect of uncertainty. This effect of uncertainty is automatically incorporated into the current standard Options Analysis.

The relations between an individual's set of appeal uncertainty distributions are analagous to the option appeal distributions for a population. For a population group, the outcomes are stated in terms of share or choice. For individuals the outcomes become probability of option choice. The only difference is that the individual measures are from a single source while the population measures are from multiple sources.

In addition to providing a basis for understanding how appeal uncertainty generates choice probabilities, this approach also reveals how Options Analysis could be used to analyse and simulate a single individual's decision-making process.


Measuring probabilistic decision-making

As already suggested, it may well be that people do not commonly apply a probabilistic approach to decision-making amongst uncertain options. That does not mean that respondents are not capable of providing reasonable estimates of probabilities of selection. This can be a useful means for gathering data to validate the results of an analysis of option appeal which incorporates uncertainty and where option shares are based on individual probabilities. It also provides another method for evaluating the effects of uncertainty on share through comparison with Options Analysis share outputs.

It is not uncommon to ask survey respondents to provide answers in terms of probability, likelihood or certainty. Sometimes this is simply implied, such as in a follow-up to a person who can't decide on a choice, asking: "Are you leaning towards A or B?"

More explicitly, respondents may be asked for their probability assessment in terms of percentage points, a simple scale of likelihood categories, or a semantic differential scale ranging from "certain to happen" to "certain not to happen" with uncertain in the middle. An example of a formal and standardised approach is the Juster purchase probability scale consisting of eleven points evenly spaced from 0 in 100 chances to 99 chances in 100.

Where respondents are asked to use such scales to assign probabilities to each option within a selection set, it is common for the assigned "probabilities" to add to more than unity. The estimates would be better regarded as "impressions of likelihood" having only ordinal metric qualities.

If the intention is to use respondent probabilities as a basis for calculating choice intentions, a constrained method should be applied. Here an exact probability assessment can be obtained by controlling, for example, the total allocation of tokens, points or votes. If ten tokens are allocated amongst selection set options to reflect likelihood of choice in a selection set, we end up with a constrained 11-point probability scale. Eric Marder of EMA Associates has successfully used just such an approach for many years in his STEP methodology.


The ambiguity of probability and the need for appeal diagnostics

While the methods just described may be suitable for obtaining a criterion measure for validation testing, probability data is of limited diagnostic value. This is especially so if one is concerned with revealing the changeability of choices at the individual level, and, the role of uncertainty in changing shares, advantage or disadvantage at the group level.

Discussion elsewhere on the nature of share indicated that share sizes have no necessary relationship with the changeability of share. The same applies to an individual's probabilities of option selection. In both cases the levels, dispersions, shapes and inter-alignments of appeal need to be examined. A change in appeal by one unit may yield no change in choice probability, or a change from p=0 to p=1, or anything in between, depending on the uncertainty parameter configuration. Measurement of both appeal and appeal uncertainty would enable such analysis.

Often there seems to be a lack of clarity concerning the respective meanings of such terms as "uncertainty" and "probability" which can be seen here to have quite distinct meanings and roles. Uncertainty refers to variability in evaluation while probability is a description of choice likelihood outputs.


Options Analysis - current applications and development prospects

The Options Analysis methods currently available are substantially simpler than those incorporating variability in individual appeal due to uncertainty to be analysed across a sample. The methods minimise data collection requirements, costs and analytical complexity. Built-in application of the suggested validation test should provide confidence in applying the full range of Options Analysis tools where the obtained results from appeal evaluation and direct choice are substantially the same.

An approach incorporating option appeal uncertainty with variable choice probability outcomes offers avenues for refinement in understanding the evaluation and decision-making process. In particular, it permits analysis of within-individual choice volatility and provides more accurate estimates of share of choice. But it challenges survey data collection processes, is analytically more complex, and may be substantially more costly.


Collecting individual uncertainty data

Creation of data distributions for Options Analysis of an individual

Notes index


Copyright 2012 Corporate R & D Ltd
All rights reserved. Email: