Methods for the Analysis of Choices and Evaluations

Collecting individual uncertainty data

  1. Conventional data collection

  2. Collecting uncertainty data

  3. Data on individual uncertainty parameters

  4. Creating option distributions for Options Analysis of an individual

Conventional data collection

In the collection of field data it is usual to obtain, insofar as possible, single data points as answers. This is done to avoid ambiguity and increase simplicity. Often, this process requires respondents to draw fine lines when they think they might fall around and between two categories.

Researchers are well aware of these problems but assume that through a process of statistical "swings-and-roundabouts" results will work out to be representative of the sampled population even when the underlying scale is continuous.


Collecting uncertainty data

Paradoxically, adding uncertainty data can further reduce ambiguity and improve accuracy. It's use can avoid invidious category or point selections or qualify them in ways which provide a truer measurement of the characteristic or behaviour of interest.

Requesting such data may be seen as somewhat different to the data most people have learned to supply. Because of its less constrained nature some may find it difficult to provide. As a result, framing questions, training interviewers, and so on, to collect data on the character of their uncertainty (which will often be indistinguishable from "vagueness"), represents something of a challenge.

The relevant questions need to be readily understandable by the full range of respondents and facilitate accurate responses. Pre-testing of questionnaires is likely to be very important for successful application.

It may be possible (and potentially useful) to research the general nature of appeal distributions for individuals in the psychological laboratory. For data collection through social surveys this is impractical. The more obvious course involves the collection of parameter data which can then be used to create a given distribution of an assumed general character (e.g. a normal distribution adjusted to fit the parameters).

An additional issue is the level of detail and precision required. The amount of data collected is a significant factor in survey costs. There may also be added complexity in analysis. Attention to survey objectives, the particular nature of the subject matter, and potential substantive implications of research findings, should provide guidelines on data requirements. The type of distribution selected will also have some effect on the results.

Using the standard deviation to assess appeal dispersion is the best single measure of uncertainty.

The impact of uncertainty on choice will be determined primarily by the relationships between each individual's:

  1. differences between option means,
  2. size of option uncertainty standard deviations,
  3. shape differences in option uncertainty distributions, and,
  4. inter-option correlations between option uncertainties,

amongst the individual's most preferred options.

Uncertainty amongst each individual's lesser preferred options will commonly have no impact on choice.

Simulations are likely to be a useful form of analysis for evaluating sensitivity to variations in any uncollected parameters and possible changes in collected parameters. For example, uncertainty effects stemming from correlation might be most easily be assessed through sensitivity tests using simulation of concordant, random and contrasting relationships between options. (Note that simulation of fully concordant relationships will generally eliminate any uncertainty impacts.)


Data on individual uncertainty parameters

Mean: For the current Options Analysis methods, a single appeal rating per option is sufficient. This may be assumed to be the mean.

Dispersion: Collection of data on dispersion about the mean will provide basic uncertainty data. The standard deviation can be estimated from data in the form of an appeal range containing a given percentage of an individual's ratings. For example, following introduction of the questioning, "Between which points on this scale [SHOW CARD] would you rate the attractiveness of X to you?", there is a need for followup questioning to determine what the range means to the respondent in terms of likelihood of containment - 99%, 95%, 90%, etc. probability. Alternatively, establishing a range of uncertainty may be preceded by the provision of a guiding rule establishing a containment probability applying to all range-setting answers e.g. "90% confident".

If it is considered reasonable to assume a symmetric distribution, mean appeal may be inferred as the mid-point of the range. If not, it will be desirable to collect the mean as a separate data point. The comments on skewness suggest an approach.

Skewness: Sometimes it may be considered important to collect data on the skewness of appeal uncertainty within individuals. For example, in a study of lotto gambling as an option compared to more everyday options, such as purchase of household items, it would probably be important to consider skewness. One would expect purchasing a lotto ticket to have high positive skewness i.e. a limited "downside" appeal and a highly extended "upside" appeal. Consideration of skewness requires collection of at least three data points: one for the mean and two more for a range. This approach may be combined with that for obtaining the standard deviation so that the mean, standard deviation and skewness are collected together.

The simplest approach would probably be to ask first for the range as described for the option standard deviation and then to ask for the "most representative single point" for the mean.

Kurtosis: Collecting data for kurtosis, with any precision, will commonly exceed the boundaries of survey practicality. It should still be feasible within a laboratory setting.

Nevertheless, when a range of uncertainty has been requested for determining the standard deviation, it may be feasible to frame questions which determine the general shape of the distribution in terms of concentrations of likelihood. Such questions would focus on whether the individual's appeal likelihood distribution was uniform ("could be anywhere across the range"), peaked ("most probably in the middle"), or bimodal ("could be at one end or the other").

Correlation: Obtaining impressionistic information from respondents on inter-option relationships may be the most useful single avenue for gather data on inter-option correlations within individual cases.

Reviewing the nature of the options should suggest some of the more likely possible relationships. Options which are very similar in character and which may be expected to be influenced in a similar way by uncertainty factors common to them all, may frequently have positive correlations. For example, the appeal of active outdoor summer sports may be uniformly affected by uncertainties about the effects on their appeal from variations in the weather.

Performing simulations based on differing correlational assumptions is likely to be useful to evaluate the sensitivity of the results to their impacts.


Creating option distributions for Options Analysis of an individual

With parameter information for an individual's uncertainty distributions for a set of options, you can create the distributions in a form suitable for input to Options Analysis. Here are the steps to do this.

You do not need all the parameters. The means and standard deviations set a useful minimum. As discussed above, simulation may provide a guide to sensitivities to variation in skewness, kurtosis or correlation in addition to variation in means and standard deviations.

The rationale for analysis of individual appeal uncertainty is set out in the webpage on uncertainty, probability, risk and Options Analysis.


Creation of option distributions for Options Analysis of an individual

Notes on uncertainty, probability, risk and Options Analysis

Notes index


Copyright 2012 Corporate R & D Ltd
All rights reserved. Email: