Conservation Technology Assessment (CTA) 1 Introduction
Last Updated: October 02, 2017; First Released: January 09, 2015
Author: Kevin Boyle, President, DevTreks
Version: DevTreks 2.1.0
A. Introduction
This reference introduces techniques for analyzing the risk and uncertainty associated with Resource Stock and Monitoring and Evaluation (M&E) Indicators and communicating these results in terms of likelihoods and probabilities. These techniques will be used to analyze the risk and uncertainty associated with the Resource Stock and M&E data explained in the Resource Stock Calculation and Analysis and the Monitoring and Evaluation references. The Stock references define CTA as follows:
Conservation Technology Assessment is the analysis of resource stock flows and balances, and conservation technologies that are designed to prevent or correct imbalances in the stocks.
B. Risk and Uncertainty Introduction
When NASA launches a space explorer to Mars, they carefully assess the likelihood that the space ship will accomplish its mission (see US, NASA 2011). When a country’s defense agency builds a plane or ship, they carefully assess the riskiness of the plane crashing, or ship sinking (see US GAO, 2009). When many countries assess the effectiveness of a new health care technology, they carry out a Health Technology Assessment that carefully assesses the probability of the medical treatment working (see WHO, 2011). What do all these “high priced” assessments have in common? They use formal risk analysis to quantify probable outcomes. The risk analysis plays a central decision support role that determines how large sums of money get allocated, and lives saved, based on the quantified results.
When a local extension agent travels to a local dairy to assess a problem with milk production, they make recommendations for improvements based on the dairies they have worked with in the past and on recent professional dairy publications they have read. When a local low income housing agency builds a new low income housing project, they assess the likelihood of success based on previous experience building similar types of projects. When a local family doctor makes a recommendation for a prescription, they base the prescription on the guidance offered by their professional associations and on what they have found to work in the past. What do all of these “low priced” services have in common? Much less reliance on quantitative decision support tools, and much more reliance on tacit knowledge gained from experience and study (see Kahneman, 2011, for further explanation about these types of decisions).
What do “low and high priced” risk analysis share in common? Most of the data used to support decisions has been lost or can’t be accessed in a way that easily supports similar decisions that will be faced by future decision makers. This reference introduces a set of risk analysis tools that can help “high or low priced” services and projects quantify the likelihood of work outcomes. That structured data can be added to both online, and tacit, knowledge banks. Clubs that desire quantified tools will have concrete, online, tools and knowledge banks available to help guide their decisions. Clubs that rely on their experience will have easier access to additional background data that goes into their tacit knowledge banks. Future decision makers will have easier access to historical data that can assist them with current problems.
C. Statistical and Probabilistic Risk Analysis
This tutorial relies on the references for fuller explanation of the mathematics and analytic techniques used in statistical and probabilistic risk analysis (PRA). The references do a professional job defining key risk assessment concepts (Probability), formulas (Lognormal Distributions), methodologies (Monte-Carlo), and reporting techniques (Probability Density Functions). Appendix A summarizes their major points. Most of the references recommend that PRAs should be reported using confidence intervals for uncertain numbers, along with probability density functions. Sibling references include examples demonstrating how these recommendations can be carried out.
PRAs can generate completely wrong numbers when models used to analyze data are defined badly, the data used in the models is spurious, or wrong assumptions are made about the distribution of indicators. The likelihood of conducting a bad PRA can be reduced by closely following the recommendations found in the major references (i.e. WHO, IPCC, NASA).
In additional to PRAs, a variety of statistical risk analysis techniques can be used to determine the uncertainty of resource stock data. The following list describes some of these assessment techniques (US OMB 2006, World Bank 2012). Many of these techniques overlap. References and algorithms that demonstrate these techniques will be included in future releases.
Personal, Social, and Managerial Decision Support: Data associated with personal preferences, societal biases, and subjective probabilities, is collected and analyzed to determine how to improve personal, social, and managerial decisions. For example, researchers have found that investors weigh losses twice as high as equivalent gains and tend to overestimate low probability events while underestimating high probability events.
Damage Assessments and Disaster Risk Reductions: Historical and simulated data is used to assess the probable damages associated with various intensity levels of natural disasters. For example, flood damage assessments show the probable damages associated with various depths of flooding. The associated CTA-Prevention (CTAP) reference in the Technology Assessment 2 tutorial has examples of algorithms that demonstrate these techniques.
Actuarial Analysis of real-world data: Historical data is analyzed using standard statistical methods. For example, historic data on car crashes has been analyzed to determine the likelihood of safety features working.
Dose-Response Analysis using experimental data: Experiments are conducted on humans or animals to determine how they react to various doses of substances. For example, experiments on rodents have been conducted to determine the likelihood that threshold amounts of substances lead to the development of cancer tumors.
Infectious Disease and Epidemic Modeling: The occurrence and severity of epidemics are simulated using computer algorithms. For example, the spread of the HIV virus has been modeled to understand transmission pathways among select groups of people.
Failure Analysis of Physical Structures: The failure of physical structures is commonly analyzed using low risk and high consequence statistical techniques. For example, concrete floors are commonly analyzed to determine whether they can withstand earthquakes.
Appendix A presents a framework for conducting risk analysis. Appendix B contains a list of additional analytic techniques, such as Sensitivity Analysis, that can be used to support, or replace, PRAs. Appendix C introduces a framework for conducting probabilistic statistics analysis.
D. Cost and Benefit Uncertainty (1*)
A key requirement of CTA is to tie base element economic cost and benefit amounts to indicator amounts. That allows common CTA Performance Analysis techniques, such as cost effectiveness analysis, to be supported. The NIST (1988) and World Bank (2013) reference introduces traditional approaches to account for risk when conducting economic evaluations. Those references discuss how risk preferences influence decision making. The World Bank and United Nations (2010) reference explains how to tie physical science indicators associated with climate change, such as projected flood depths, to economic damages. The NASA (2008) and GAO (2009) references cover cost risk analysis at greater depth.
The Resource Stock Calculation reference demonstrates how to use indicators to calculate uncertain costs and benefits. Several examples in this reference also demonstrate how to calculate uncertain costs and benefits. As uncertainty is incorporated into more calculators, additional techniques for analyzing uncertain costs and benefits will be explained. For example, The Monitoring and Evaluation tutorials demonstrate how to use Indicators to measure risk and uncertainty for each, separate, base element.
E. Base Element Multiplier Uncertainty
The IPCC 2006 reference discusses the importance of distinguishing between the uncertainty associated with individual emission indicators and the uncertainty associated with their aggregated mean and time period trends. The examples in this reference demonstrate how to determine the uncertainty associated with individual indicators (i.e. their QTM properties) and the uncertainty associated with their combined aggregated means (i.e. their ScoreM properties). The associated Resource Stock Analysis reference provides additional guidance for accounting for Input and Output multiplier uncertainty. The M&E and NPV references demonstrate that calculations can be run for additional base elements, including Operation/Component/Outcomes (i.e. categories) and Time Periods (i.e. time trends). The Version 2.0.4 upgrade of the M&E tools began allowing uncertain base element risk calculations to be run for all base elements.
F. Base Element Mathematical Calculations
Besides the uncertainty of base element multipliers, base elements are often used in mathematical calculations. For example, constrained optimization algorithms might have an objective to find the combinations of operations, components, outcomes, time periods, and budgets with the highest returns. Future upgrades of related calculators, such as the M&E tools upgraded in Version 2.0.4, will address this need.
G. CTA Analysis (2*)
All CTA algorithms use Resource Stock Calculators and M&E Calculators and therefore their calculations can be analyzed further using Resource Stock Analyzers and M&E Analyzers. When the data being analyzed is observational data stored in Data URL datasets, these analyzers produce automated metadata analysis (analysis of analyses). Metadata analysis of randomized control trial data is the primary technique employed in Health Technology Assessments (HTAs) (see Example 1m below and Appendix C, DevPacks Stock Analysis, in the Resource Stock Analysis reference for examples).
Keep in mind the conventions used by those analyzers. For example, Example 1J demonstrates that Operations and Operating Budgets use Input.OCAmount as a multiplier for Input Stock and M&E calculations. The example also demonstrates that Components and Capital Budgets use Input.CAPAmount as the multiplier. Even though most algorithms are run asynchronously and simultaneously, keep in mind that this technology has limits on what data can be realistically processed (and full tests have not been run to find out yet). This version does not support cancelling an analysis and does not show numeric progress during analyses.
H. Online Models
DevTreks recommends limiting the actual analyses that can be accessed from online production servers to proven models. We recommend using development servers, whether online or offline, for testing models prior to putting them “into production”. When possible, use validated models, containing known indicator and score analysis properties, including vetted Math Expression and scripts. For example, Algorithm 1 includes a basic regression analysis algorithm that can be investigated using standard statistical packages. Algorithms 2 and 3 use mathematical and statistical libraries that can be tested on a variety of platforms before being made available through DevTreks. After the analyses have been satisfactorily tested on the development server, see if the final model can be run, or at least displayed, on the production server. If not, make recommendations for improvements to the algorithms and online tools.
I. Mathematical and Statistical Libraries (3*)
Algorithms that use statistical libraries, such as the R Project and Python, began appearing in version 1.8.2. Virtual machine-based libraries showed up in version 2.0.2. The current version runs most of these libraries as command line executables or through web apis (Application Programming Interfaces) and web services. That introduces some latency, or slowness, as executables load associated libraries and urls get routed to and from Internet destinations. The related CTA references demonstrate that some of the statistical packages, such as R Server, are starting to address scalability.
J. Communicating CTA Results
Appendix A demonstrates that the analytic results of CTA can be explained using techniques that vary from simple tables to complicated graphics. URLs referenced with R Project and Python algorithms demonstrate the power of using graphics to explain statistical results. The risk assessment framework introduced in that Appendix explains the importance of basing the CTA analysis on the objectives of stakeholders and then communicating the results of the CTA in a manner that meets stakeholder needs. Most of the climate change reference demonstrate the value of analyzing and communicating some types of risks using GIS techniques. Future releases will improve the tools available for communicating the results of CTA.
K. CTA Algorithms (4*)
All CTAs are completed using automated, quantitative, algorithms. The algorithms are defined using Indicator or Score’s Math Type (algorithm1) and Sub Math Type (subalgorithm1) properties. The Math Type property must select an option identifying the statistical library to use, using a generic name such as “algorithm1” or “algorithm2”. The Sub Math Type property must identify the name of the subalgorithm within the library to use, using a generic name such as “subalgorithm1” or “subalgorithm2”.
Not every algorithm supports running the algorithm for both indicators and scores. For example, algorithms that run regression analysis require TEXT datasets, but Scores for Resource Stock calculations do not currently use datasets and therefore can’t use that algorithm. The Resource Stock Calculation reference explains properties of CTA-related calculators. The data standards defined in that reference, particularly data column names and Math Expression terms, must be followed exactly for these algorithms to work. In addition, most algorithms rely on text data stored in URLs. Those URLs must contain the data expected by the algorithm. The convention of many of those TEXT files is to use the last 3 rows of data as scoring datasets (i.e. used to generate a sensitivity analysis of confidence intervals) and the rest of the rows as the training dataset.
Datasets must be cleaned up prior to using them with any CTA algorithm. Unless an algorithm shows a different example, all data matrixes must have column vectors that have the same number of rows (positive definite) –if not, zeros will be inserted to make them that way. The manipulation of very large matrixes presents technical software challenges that will continue being addressed (i.e. by using algorithms that don’t rely on matrix manipulation or that use multi-threaded mathematical libraries, such as algorithm2’s Intel Math Kernel library).
The following algorithms are currently available.
algorithm1 (MathNet): The sibling reference, CTA Algorithm 1, gives examples of subalgorithms used with this algorithm. The examples use probabilistic statistics that employ the open source MathNet mathematical library.
algorithm2 (R): The sibling reference, CTA Algorithm 2, gives examples of subalgorithms used with this algorithm. The examples use probabilistic statistics that employ the open source R statistical library.
algorithm3 (Python): The sibling reference, CTA Algorithm 3, gives examples of subalgorithms used with this algorithm. The examples use probabilistic statistics that employ the open source Python statistical library.
algorithm4 (AML): The sibling reference, CTA Algorithm 4, gives examples of subalgorithms used with this algorithm. The examples use probabilistic statistics that employ the Azure Machine Learning statistical library.
algorithm5 (Display): The sibling reference, CTA Algorithm 5, gives examples of subalgorithms used with this algorithm. The examples display probabilistic statistics generated by any statistical library. Metadata analysis of those results can be carried out using Resource Stock Analyzers.
Algorithm6 (Julia): Version 2.0.2 stubbed out most of the source code to support this open source statistical library, but it hasn’t been tested yet. The main point is to reinforce the importance of economies of scope in this type of IT product. A future release will add a CTA 6 reference to demonstrate this algorithm.
Future releases will include additional algorithms, examples, and statistical libraries.
Summary
This reference introduces CTAs. CTAs may help Indian smallholders to adapt effectively to changing monsoons, Somalian health care administrators to deliver more cost effective medical treatments, Philippine city managers to mitigate rising typhoon intensity more efficiently, California vegetable growers to manage water wisely, Egyptian municipalities to manage energy distribution better, Chilean mariners to adopt sustainable fish stock management practices faster, and people to improve their lives and livelihoods.
Footnotes
1. Version 1.7.6 started using mathematical parsers to calculate mathematical expressions and mathematical libraries to carry out analysis. Version 1.8.0 changed the default software pattern used in calculators and analyzers to accommodate heavier traffic loads and larger datasets, but this pattern still requires full testing with large datasets. Version 1.8.2 added statistical libraries. Version 2.0.2 added the option of running the libraries from virtual machines. Versions 2.0.4 and 2.0.6 allowed all of the CTA algorithms to be used with the Resource Stock calculators and all of the Monitoring and Evaluation calculators. The full capability of these types of tools extends well beyond the examples in this reference. These issues will continue to be addressed in future releases. Version 2.1.0’s Social Performance algorithms began relying primarily on TEXT datasets as the primary input and output for algorithms.
2. Many algorithms generate results that are not intuitive. It’s hard to be confident of their accuracy without third party, vetted, reference datasets. The point is that algorithms cannot simply be used as black boxes that magically spit out answers. All of the algorithms that are used in this reference will continue to be researched, improved, or replaced, as more knowledge is gained about their workings.
3. We investigated using additional open source, or free, statistical libraries. Two algorithms were built using a Bayesian statistical software library (Minka et al, 2014) prior to developing algorithms based on scripting languages such R and Python. We decided not to release those algorithms for now because the scripting languages are popular. Related open source statistical software libraries, such as OpenBugs and STAN, were investigated and may be supported in future releases.
4. Several tutorials point out that, while it may be the role of a software development company like DevTreks to build and document basic algorithms, that full-time role is the responsibility of DevTreks’ social networks and their clubs. Users of this source code are encouraged to build and document their own algorithms targeting the needs of their own networks, clubs, and members.
5. A performance commitment defines the acceptable risk tolerance as defined by the decision makers. NASA (2011) describes this amount as follows: “A performance commitment is a performance measure value set at a particular percentile of the performance measure’s pdf, so as to anchor the decision-maker’s perspective to that performance measure value as if it would be his/her commitment, were he/she to select that alternative. For a given performance measure, the performance commitment is set at the same percentile for all decision alternatives, so that the probability of failing to meet the different alternative commitment values is the same across alternatives.” The media used with the URLs in Examples 1a, 1b, and 1k include a graphic of the data’s cumulative density function with a potential performance commitment shown on the graphs.
6. Most experts in regression analysis recommend researching the model and data thoroughly prior to running a regression. At least that is what the author was taught. They do not advocate the commonly used practice of running multiple regressions with the goal of finding a “good model”. That approach is consistent with the recommendations made by experts in PRA who advocate thoroughly understanding what is being modeled prior to running any analysis. At this stage of development, these Appendixes can rightly be viewed as no more than placeholders. Refer to Footnote 4.
References
Ireland, Health Information and Quality Authority. Guidelines for the Economic Evaluation of Health Technologies in Ireland. 2010 (www.hiqa.ie)
Lurie, Philip, Goldberg Matthew and Robinson, Mitchell. US Institute for Defense Analyses. A Handbook of Cost Risk Analysis Methods. April, 1993
Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus, and Giroux. New York, USA. 2011
T. Minka, J. Winn, J. Guiver, S. Webster, Y. Zaykov, B. Yangel, A. Spengler, and J. Bronskill. Infer.NET 2.6, Microsoft Research Cambridge, 2014. http://research.microsoft.com/infernet
U.S. Air Force. Cost Risk and Uncertainty Analysis Handbook. 2007
U.S. Department of Commerce. National Institute of Standards and Technology. Engineering Statistics Handbook, http://www.itl.nist.gov/div898/handbook/index.htm (last accessed on the web in May, 2015)
U.S. Department of Commerce. National Institute of Standards and Technology. Techniques for Treating Uncertainty and Risk in the Economic Evaluation of Building Investments. NIST Special Publication 757. 1988
US Government Accountability Office. Applied Research and Methods. GAO Cost Estimating and Assessment Guide. Best Practices for Developing and Managing Capital Program Costs. March, 2009.
U.S. National Aeronautical and Space Administration. NASA Risk Management Handbook, NASA/SP-2011-3422, Version 1.0, November, 2011.
U.S. National Aeronautical and Space Administration. Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners, NASA/SP-2011-3421, Version 2.0, December, 2011.
U.S. National Aeronautical and Space Administration. 2008 NASA Cost Estimating Handbook, 2008.
U.S. Office of Management and Budget. Proposed Risk Assessment Bulletin. 2006
Westen, Alkema, Damen, Kerle, and Kingma. Multi-hazard risk assessment, Distance Education Course, Guide Book. United Nations University. ITC School on Disaster Geoinformation Management. 2011
World Bank. Risk Analysis Course Manual. Global Facility for Disaster Reduction and Recovery. 2011
World Bank and United Nations. Natural hazards, unnatural disasters: the economics of effective prevention. 2010
World Health Organization. Health Technology Assessment of Medical Devices. 2011
References Note
We try to use references that are open access or that do not charge fees.
Improvements, Errors, and New Features
Please notify DevTreks (devtrekkers@gmail.com) if you find errors in these references. Also please let us know about suggested improvements or recommended new features.
A video tutorial explaining this reference can be found at:
https://www.devtreks.org/commontreks/preview/commons/resourcepack/Technology Assessment 1/1526/none
Appendix A. PRA Introduction
The following three images introduce a framework (NASA, 2011) for analyzing risk and uncertainty in a project, program, or technology:
The US, GAO (2009) points out that “Point estimates (i.e. the Performance Parameters displayed in the previous image) alone are insufficient for good decisions. … Quantitative risk and uncertainty (R&U) analysis provide a way to assess the variability in the point estimate”. The following images (GAO, 2009) demonstrate that the variability of the alternative point estimates (i.e. Performance Parameters and Measures) can be modeled and summed using probability (or sampling) distributions:
The following image demonstrates that US NASA (2011) uses “sampled” Performance Parameter (i.e. point estimates) to derive distributions of Performance Measures:
The following image demonstrates that the US GAO (2009) concurs with using simulated point estimates (i.e. in this case, Cost) to analyze risk and uncertainty:
The following image (US GAO 2009) demonstrates that some types of risk analysis can be satisfactorily completed using simple qualitative techniques:
The following image (Ireland, HIQA 2010) demonstrates that some types of risk can be understood better by using more comprehensive analytic techniques (in this image, three health care treatment alternatives are shown using scatter plots with incremental costs on the y-axis and incremental benefits, or QALYs, on the x-axis):
The following image (US NASA, 2011) demonstrates that the results of risk analysis are often reported in terms of imposed constraints such as budget constraints or minimal acceptable levels of safety (5*):
The following image (NASA, 2011) demonstrates that some types of probabilities can be estimated better by supplementing prior beliefs about model parameters with subsequent observations of real data – a technique known as Bayesian statistics.
US NASA (2011) identifies the following best practices for carrying out probabilistic risk analysis. Substitute CTA for the NASA space mission project context.
Conduct the analysis in the context of a full Risk Analysis Framework (i.e. NASA’s CRM) that continually assesses risk and makes appropriate implementation adjustments that ensure the accomplishment of a project’s objectives. Complete the actual risk analysis in a manner that can fully support the decision making (i.e. NASA’s RIDM) needed in the encompassing framework. Conduct an actual risk analysis by following these steps:
1. Use Stakeholder Expectations to Identify Alternatives that meet their Performance Objectives: Describe the Performance Objectives required by Stakeholder’s. Identify Alternatives that can meet those objectives. In terms of CTA, Alternatives will be alternative technologies, practices, or projects. Risk analysts who specialize in health or safety, as contrasted to building and machine risk assessments, point out the importance of incorporating the correct target populations being affected by risk (i.e. Thompson, 2011, HIQA, 2010). These targeted populations are important stakeholders for most social improvement-related technologies, practices, and projects,
2. Identify Performance Measures and Constraints: Determine the most important Performance Measures or Risk Factors (i.e. Cost, QALY, Data Collection Labor), and Constraints (i.e. Safety Allowance), for each Performance Objective.
3. Identify Important Scenarios, Likelihoods, and Consequences. Evaluate each Performance Measure in terms of potential scenarios that can lead to likely positive and negative consequences. Not every Performance Measure will need to be evaluated in terms of scenarios. Scenarios are equivalent to “comparators” in DevTreks Change and Progress Analyzers.
4. Identify Performance Parameters: Identify initial, but uncertain, Performance Parameters that can be used to measure each Performance Measure. The Resource Stock and M&E references demonstrate that Indicators can be used as Performance Parameters.
5. Specify the Uncertainty of each Performance Parameter: Use sampling or ranking distributions (i.e. min-max, scenario likelihood, triangular, normal, ranking matrix) to model the uncertainty of the initial Performance Parameters.
6. Account for Correlated Performance Parameters: Consider appropriate correlations for each Performance Parameter.
7. Generate Performance Measures: Use a modeling technique (i.e. weighted average, algebraic, Monte Carlo, Latin Cube, Markov, Bayes) to generate sampled point estimates, or Performance Measures, from running calculations that use the uncertain Performance Parameters.
8. Analyze the Performance Measures: Display and analyze the simulated point estimates, or Performance Measures (i.e. scatter plots, histograms, probability distribution graphs) with any imposed constraint levels. Eliminate or add Performance Measures.
9. Use Performance Commitments to Present the Risk Analysis to Decision Makers (5*): Use risk tolerance levels, or Performance Commitments, for each Performance Measure. Make sure these levels meet stakeholder expectations. Report the Performance Measures and Performance Commitment for each alternative to the decision makers. If necessary, examine additional alternatives.
10. Report the Selected Alternatives: Report the selected Alternatives using a Risk-Informed Selection Report that expresses the final Performance Measure results in terms of their Performance Commitments (i.e. Alternative 1, Performance Measure Cost 1 = $10,000 with probability). As discussed in item 1, social improvement-related risk assessments need to communicate risks in terms of effects on targeted populations.
Not every risk analysis needs every “best practice”. For example, some types of risk analysis (i.e. rank matrices) use limited, or no, sampling distributions of point estimates. Some types of cost estimates or benefit estimates do not need to compare alternatives. A simple sensitivity analysis of how different fuel prices may change tillage and planting costs does not need every step.
The following items demonstrate general software features that support carrying out PRAs of CTA data. Substitute CTA for the NASA space mission project context (and no, this is not rocket science).
A. Domain-Specific (i.e. Safety and Radiological Contamination) Models: These types of custom models can be built in DevTreks using standard Linked Views (i.e. custom calculator and analyzers that carry out numerical analysis in technical domains such as Agriculture, Food Nutrition, Health Care, Natural Resources, Disaster Prevention/Recovery, or in the NASA example, Safety and Radiological Contamination).
B. Technical Data Collection Models: DevTreks itself is an online data collection model. Multimedia is stored in the Resources application. Stories, calculators, and analyzers, are stored in the Linked Views application. The DevPacks application supports a wide variety of hierarchical content management techniques (i.e. objective hierarchies, trade trees, risk taxonomies).
C. Schedule Models: DevTreks’ Monitoring and Evaluation (M&E) calculators and analyzers can replicate, in part, this model. The full replication of this model will require additional types of M&E calculators and analyzers (i.e. Scheduling Risk Analyzers, Least Cost Scheduling Analyzers).
D. Benefit and Cost Models: This model can be replicated using DevTreks’ standard economic evaluation calculators and analyzers (i.e. Net Present Value and Life Cycle calculators and analyzers). DevTreks treats benefit estimation just as importantly as cost estimation.
E. Risk Analysis of Models: This reference introduces the first set of CTA tools available for conducting risk analysis. These generic indicators, mathematics, and statistics, support a wide assortment of risk measurement. They can also be used to set the initial costs and benefits found in the Benefit and Cost models.
PRAs of modeled data can still generate completely wrong numbers when models are defined badly, the data used in the models is spurious, or wrong assumptions are made about the distribution of indicators. Most of the tutorials refer to scientific publications (i.e. IPCC) that, if used properly, can improve the models, the assumptions, and the data.
Several references (i.e Lurie, Goldberg, and Robinson, 1993) further clarify the limitations of PRAs as follows: “Although a formal analysis of cost [and benefit] risk will not reduce the risk inherent in a program, it will help program managers understand the nature of the risks involved, and quantify and display the uncertainty associated with cost [and benefit] estimates”.
Appendix B. Supplemental Analyses
Numerous related analyses can be conducted in support of a risk analyses. Some of these include:
Sensitivity Analysis examines changes in one risk indicator while keeping all other indicators constant. This procedure is used to identify (and manage) the most important drivers of project and program results. Some sensitivity indicators compare the percentage change in one risk indicator with percent changes in other indicators (i.e. a 1% change in yields results in a 10% change in net profit). The usual steps in a sensitivity analysis include (US GAO 2009):
1. identify key cost drivers, ground rules, and assumptions for sensitivity testing;
2. reestimate the total cost by choosing one of these cost drivers to vary between two set amounts —for example, maximum and minimum or performance thresholds;
3. document the results;
4. repeat 2 and 3 until all factors identified in step 1 have been tested independently;
5. evaluate the results to determine which drivers affect the cost estimate most.
The following image (US GAO, 2009) demonstrates a typical sensitivity analysis of gross project costs. Examples of sensitivity analysis of Benefit Cost Ratios and Cost Effectiveness Ratios can be found in the CTAP reference.
Incremental Analysis is a type of cost effectiveness analysis that measures how an incremental change in one indicator affects another indicator. Some Health Technology Assessments (Ireland HIQA, 2010) use measurements of incremental costs and incremental benefits as the initial point estimates that go into a risk analysis. The following image (US, NIH, HTA Course 101) displays a typical incremental analysis of costs vs. number of medical tests:
Weighted Average Analysis relies on simple subjective assessments of risk using weighted risk scores such as 1 for low levels, 2 for medium levels, and 3 for high levels. The following image (US GAO, 2009) displays an example:
Scenario Analysis (US Air Force, 2007; or ADB, 2002’s Simplified Probabilistic Analysis) is a variant of sensitivity analysis that defines a group of indicators (a scenario) that change together and that represent a logical scenario for a project. For example, the Fukushima nuclear reactor disaster may be related to a failure to adequately consider feasible scenarios, such as a large tsunami, in its design. Algebraic formulas are used to generate these types of probability distributions:
Decision Tree Analysis depicts risk using branches to display the probability of alternative actions. The following image (US, NIST, 1988) displays a building construction investment which can use alternative insulation (R1, R3 …). Each alternative can be quantified using a probable (i.e. minimum, most likely, maximum) net investment return ($3 and $2 in the top node):
Event Tree and Fault Tree Analysis: (NASA 2011) risk is modeled as a series of events that collectively lead to the likelihood of a consequence, such as an accident.
Behavioral and Psychological Decision Analysis (see page 14 in US NASA, 2011 or Kahnemann, 2011) uses techniques developed by psychological and behavioral scientists.
Constrained Choice, or Optimization, Analysis (refer to economics texts) measures an optimal level of one or more outcomes (i.e. net returns) given constrained input choices, such as cost, acres of land, or minimal level of safety.
Schedule Analysis (see GAO, 2009) assesses the probability of falling behind schedule and incurring additional project costs.
Markov Analyses (see Ireland HIQA, 2010) measures R&U indicators that change over time, such as when a series of health care treatments change health care states over time.
Parametric Analysis (see NASA, 2011 and GAO, 2009) uses regression analysis techniques to develop probabilistic models that predict a dependent variable based on datasets of observed dependent and independent variable data.
Computer Algorithms (see the McCaffrey reference) use a variety of numeric analyses techniques (i.e. neural network, bayes, matrix decomposition) to assess the probability of outcomes.
Risk Scoring and Risk Cubes: (see the GAO 2009 reference) Factors that have large potential impacts on results are identified and then ranked and scored, using both qualitative and quantitative means.
Appendix C. Probabilistic Statistics using Statistical Models
Risk analysis plays an integral role in most CTAs. Yet several algorithms in this reference do not directly employ risk analysis techniques. Instead they employ optimization, forecasting, prediction, classification, and machine learning, techniques. These algorithms must still be used in an appropriate scientific framework. Most serious references that explain these techniques also introduce appropriate frameworks. Two examples follow.
Example 1. Basic Randomized Experimental Design
Basic randomized experiments use the following terms to define the data used to conduct an analysis:
Factors: independent variables that are related to a response variable y
Levels: intensity setting of a factor
Treatment: particular combinations of levels of factors involved in an experiment.
Basic experiments are then designed using the following steps:
1. Select the factors to be included in the experiment and identify the parameters that are the object of the study. Usually, the target parameters are the population means associated with the factor-level combinations (treatments).
2. Choose the treatments (factor-level combinations)
3. Choose the number of observations for each treatment.
4. Decide how the treatments will be assigned to the experimental units.
Example 2. Multivariate Data Analysis
The following steps are often used as a framework for conducing multivariate data analysis (Studeman, 1992):
Step 1. Review the literature
Step 2. Specify the model: select the independent variables and the functional form
Step 3. Hypothesize the expected signs of the coefficients (regression) or quantify the prior beliefs about the independent variables (bayes)
Step 4. Collect the data
Step 5. Estimate and evaluate the model (6*)
Step 6. Document the results
Step 7. Communicate the results to decision makers
DevTreks –social budgeting that improves lives and livelihoods
1