Monitoring and Evaluation Analysis 1 Last Updated: August 13, 2014, Original Release: March 28, 2013 Author: Kevin Boyle, President, DevTreks Version: DevTreks 1.6.8 A. Introduction This reference documents how to complete Monitoring and Evaluation (M&E) analyses using M&E Calculator 1. The DevTreks reference, Monitoring and Evaluation 1: Food Nutrition introduces the background logic for M&E analysis in DevTreks. The Monitoring and Evaluation 1 reference explains how this calculator works. B. Data Fictitious data for Malnutrition Improvement projects were used in this reference (real data couldn’t be found). The Calculators and Analyzers explained in this reference can be found at the following URIs: * M&E Calculators https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/M and E Calculators/53/none/ * M&E 1 Analyzers https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/M and E 1 Analyzers/52/none/ C. Work Breakdown Structure (WBS) DevTreks recommends using standard Work Breakdown Structures to structure data. The WBS labels are used to aggregate and analyze data for all clubs throughout a network. Although some malnutrition codes can be found in the World Health Organization’s International Classification of Diseases (ICD10), a complete WBS for malnutrition interventions does not exist (to the author’s knowledge). Each of the M&E element sections below include the WBS used with the data. All of the data used in this reference, including the indicators, were aggregated using these labels. Although the examples below show the same labels being used by both an M&E element and its associated indicators, the two labels can be different. D. Analyzers Separate M&E analyzers are available for each M&E element, including Inputs, Outputs, Operations, Components, Outcomes, Operating Budgets, and Capital Budgets. The Calculator and Analyzer 1 reference documents how all DevTreks’ Analyzers work. The Analysis Type property of M&E Analyzers is used to specify the type of analysis to run. Each analysis will display up to 10 aggregated indicators for each separate M&E element. The number of displayed indicators is somewhat arbitrary and may be changed in future releases. E. M&E Indicator Analyses M&E analyses are carried out in two stages. The first stage aggregates base elements using standard Analyzer aggregators, such as Label or Group Id (3*). The second stage then aggregates the indicators associated with each base element. Unlike NPV and LCA analysis, the number of observations is not based on the number of aggregated base elements, but the number of distinct indicators in the aggregated base elements. Analysis Result Properties The results of running analyses are displayed using the following basic properties for all base elements (3*): M and E Type: The stage of the monitoring and evaluation analysis. Options include baseline, actual, realtime, post. Each M&E Analyzer includes a selection list for setting this property. Total Name: name of the total indicator Total Label: WBS indicator label used to aggregate indicators Total Q1: total indicator quantity 1 Total Q1 Unit: unit of the Q1 indicator quantity Total Q2: total quantity indicator quantity 2 Total Q2 Unit: unit of the Q2 indicator Total: total of the Q1 (+, -, *, /) Q2 results Total Unit: unit of the total indicator F. Multipliers Base element multipliers, such an Operation Amount, Time Period Amount, or Input Times, do not change the quantity of indicators in any analysis. G. M&E Analyzers The current version supports the following analyses: 1. Statistics 1 Analysis This type of analysis uses the Totals calculations to measure basic statistical properties of aggregated indicators. Total, Median, Mean, Variance, and Standard Deviation statistics are generated for all of the indicators that have the same Label property. Only indicators with an Indicator Type property set to Actual are used in the aggregation. The following Output Statistics Analysis displays basic statistics associated with output indicators. These statistics measure the proportion of families with malnourished children receiving food nutrient packages. Q1 is the number of families receiving a minimal quantity of food nutrient packages. Q2 is the total number of families in the targeted group. The statistics for the output series aggregates 8 observations consisting of 2 separate Output Series with 4 quarterly indicators. All 8 indicators have the same Label. Year 2012 included 1 Benchmark indicator and 4 Actual quarterly indicators. 2013 included 4 Actual quarterly indicators. Note that the analysis takes certain indicator properties, such as name and description, come from the first indicator only (admittedly a little misleading, but a standard convention used throughout DevTreks). The 2013 Output Series (and 2014, 2015 …) name will be shown in the analysis, even though its indicators were added to the 2012 results, to make it clear that its indicators are included in the analysis. The following Capital Budget Statistical analysis combines the Inputs, Components, Outputs, Outcomes, and Time Periods for 1 Capital Budget containing 2 Time Periods. A Labels aggregator was chosen for the aggregation and the base elements in both years contained the same Labels. The 2013 indicators are being included in the 2012 statistical analysis. If the Labels aggregator had not been chosen, the subsequent analysis would have been harder to interpret. 2. Change 1 Analysis This type of analysis uses indicators that support Change by Year and Change by Id calculations to measure total and percent changes in aggregated indicators. Benchmark Total is the initial amount of the indicator being measured that has an Indicator Type property of Benchmark (regardless of the year). Actual Total (or Q1, Q2) is a summation of all of the indicators that have an Indicator Type set to Actual for the same year except for the Benchmark. Total Change (or Q1, Q2) is measured as Actual Total Year x – Actual Total Year x -1 (with the first calculation subtracting zero). Percent Total Change (or Q1, Q2) is measured (Total Change Year x / Actual Total Year x-1) * 100. Benchmark Percent Change (or Q1, Q2) is measured as ((Actual Total Year x – Benchmark Total) / Benchmark Total) * 100. Average Cost is measured as Actual Total / Q1 Total (i.e. Total Cost / Total Quantity). Marginal Cost is measured as Total Change / Q1 Change (i.e. Incremental Change in Total Cost / Incremental Change in Total Quantity). Price Elasticity is measured as ((xminus1.TotalQ1 - current. TotalQ1) / (xminus1. TotalQ1 + current. TotalQ1) / 2) / ((xminus1. TotalQ2 - current. TotalQ2) / (xminus1. TotalQ2 + current. TotalQ2) / 2). Q1 is interpreted as a Quantity and Q2 is interpreted as a Price. If any divisor is zero, the calculated number will be zero as well. Although the terms “Cost” and “Price” are used in these properties because those are common measurements for these types of numbers, the measurements are useful for numbers that are not prices or costs as well. The following image uses the same exact data as the Statistics 1 analysis but now analyzes annual indicator changes. The 2012 Output Series data is measuring changes between 2012 Benchmark and 2012 aggregated Actual Indicators. The 2013 Output Series data is measuring changes between the 2012 Benchmark and aggregated Actual Indicators and the 2013 aggregated Actual Indicators. Note that although up to four indicators are being aggregated in each series, properties such as name and description come from the first indicator only. Also note that the 2013 decreased Percent Total Change in the proportion of families receiving food nutrition packages could mean that malnutrition conditions are improving. The only way to know is to read the story that must accompany these analyses and that explains their content. Operating Budget and Capital Budget analysis will display Input and Output elements as shown above (Q1 to Q4 in an aggregated 2012 element and Q5 to Q8 in an aggregated 2013 element. The following image shows that other elements, such as Outcome and Components, are displayed in the same element: The following Input Change by Id Analysis carries out legitimate Marginal Cost Analysis because the Q1 and Q2 indicator properties were entered using a standard Quantity (Q1) and Price (Q2) relation (refer to the M&E 1 reference for further details about this type of analysis). 3. Progress 1 Analysis This type of analysis uses indicators that support Progress calculations to measure the actual progress achieved for Full Target and Partial Target indicators. Actual indicators must have a date that is less than the corresponding partial target date, and greater than or equal to a previous partial target date. The properties in this analysis include: Actual Total is a summation of all of the indicators within a partial period that have an Indicator Type property of Actual. Benchmark Total is the initial amount of the indicator being measured (using a Benchmark option for the Indicator Type property). Benchmark Percent equals Actual Total / Benchmark Total. Partial Target Total is the partial target amount of the indicator being measured for a partial target period. Partial Target Percent equals Actual Total / Partial Target Total. Full Target Total is the full target amount of the indicator being measured. Full Target Percent equals Actual Total / Full Target Total. If any divisor is zero, the calculated number will be zero as well. The following Progress 1 Outcome Analysis displays the progress achieved for two M&E element indicators over two quarterly periods: The following Operating Budget Progress Analysis examines the progress achieved for 2 quarters in 2 Time Periods: 4. Other Analyses: Future references will include additional types of analyses. H. M&E Analysis and Economic Analysis Although M&E Analyses can be completed independently of their encompassing economic evaluation numbers, we recommend that both M&E analyses and economic analyses (refer to the General Analyzers section of DevTreks’ home page) be used together. The Performance Analysis 1 reference demonstrates how to use various Performance Measures, such as Incremental Cost Effectiveness Ratio and Cost per Unit Indicator, to support decisions that combine monitoring and evaluation data with benefit and cost data. I. Comparative Analysis DevTreks supports basic M&E comparative analysis. Each indicator being compared in an analysis must have the Indicator.Alternative option set to an appropriate option (one, two, three, four, or five). The exact same type of Analyses explained in previous sections will be carried out, but, before running the analysis, the Indicators will be subdivided further by the Alternative. This generation of analyses displays the results of each Alternative but does not carry out any mathematical operations between the Alternatives. Even though the math carried out in comparative analysis is simple (addition, subtraction, multiplication, or division), the amount of data generated can make interpretation difficult. A good strategy is to always include a summary story with each analysis. The comparative analyses supported in DevTreks include: 1. Statistics 1 Comparative Analysis The following image displays the results of running a comparative analysis for a Statistics 1 Analysis. The analysis compares three alternatives, each consisting of 8 indicators that have been aggregated from two different Output Series (2012 and 2013). 2. Change 1 Comparative Analysis The following image displays the same data as found in the comparative Statistics 1 Comparative Analysis, but an Annual Change 1 Analysis has been run. The analysis compares three alternatives over two years, each consisting of 4 indicators found in separate Output Series (2012 and 2013). The 2012 Output Series are measuring changes between a 2012 Benchmark Indicator and aggregated 2012 Actual Indicators. The 2013 Output Series are measuring changes between their related 2012 Output Series Alternative and 2013 aggregated Actual Indicators. 3. Progress 1 Comparative Analysis A future update of this reference will include a sample data set demonstrating the use of this analysis. J. Sample Data Sets This section demonstrates how a data set for an M&E analysis might be structured. The data set uses a Progress 1 analysis. Examples of Statistical, Annual Change, and Marginal Cost analyses can be found in nearby M&E elements (i.e. a Project 02 Malnourishment Capital Budget is a sibling of the Malnutrition Project 1 Capital Budget URI below). Data This section contains links to sample data sets that display the results of running these M&E analyzers. Keep in mind that these data sets were structured and used to test the analyzers. The data is fictitious and no weight should be assigned to the absolute numbers –pay attention to the aggregation techniques only. Many of these data sets were not all upgraded to their latest version on the cloud site. As testing takes place on some elements, changes are made to indicators to further test an analysis (i.e. Change by Alt). Those changes may not be reflected in analyses that were run before the changes were made (i.e. Totals). The data can be examined at the following URIs (1*): * Input Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Inputs/2651/none/ * Output Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outputs/2656/none/ * Operation Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Operations/2654/none/ * Component Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Components/2650/none/ * Outcome Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outcomes/2655/none/ * Operating Budget Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Op Budgets/2653/none/ * Capital Budget Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Investments/2652/none/ WBS Examples The following WBS is an example of how to classify Inputs: Type: T120. Food Nutrition Group: IG120. M and E Malnutrition Education Group Input: I120. 2013 Nutrition Training Manual Development Input Series: I120. 2013 Nutrition Training Manual Development Indicator 1: I120. Training Material Development Labor Input: I121. 2013 Nutrition Training Workshop Input Series: I121. 2013 Nutrition Training Workshop Indicator 1: I121. Training Labor The following WBS is an example of how to classify Outputs: Type: T120. Food Nutrition Group: 0G120. M and E Nutrition Training and Practice Output: O121. 2013 Nutrition Training Workshops Output Series: O121. 2013 Nutrition Training Workshops Indicator 1: O121. Passing Nutrition Grades Output: O120. 2013 Nutritious Meals Consumed Output Series: O120. 2013 Nutritious Meals Consumed Indicator 1: O120. Daily Nutrition Target Days per Month The following WBS is an example of how to classify Operations or Components: Type: T120. Food Nutrition Group: OPG120. M and E Malnutrition Training Operation 1: OP121. 2013 Conduct Training Workshops Indicator 1: OP121. Workshops Held Input 1: I121. 2013 Nutrition Training Workshop Indicator 1: I121. Training Labor Operation 2: OP120. 2013 Develop Nutrient Training Materials Indicator 1: OP120. Training Materials Completed Input 1: I120 2013 Nutrition Training Manual Development Indicator 1: I120. Training Material Development Labor The following WBS is an example of how to classify Outcomes: Type: T120. Food Nutrition Group: OCG120. Nutrition Education and Practices Outcome 1: OC120. Consumption of Nutritious Meals Indicator 1: OC120. Percent Population Eating Healthy Output 1: O121. 2013 Nutrition Training Workshops Indicator 1: O121. Passing Nutrition Grades Output 2: O120. 2013 Nutritious Meals Consumed Indicator 1: O120. Daily Nutrition Target Days per Month The following WBS is an example of how to classify Operating or Capital Budgets: Type: T120. Food Nutrition Group: BG120. M and E Malnutrition Projects Budget 1: B120. Malnutrition Project 1. The goal of this project is to reduce malnutrition in a targeted population. Children in the targeted population suffer from malnutrition. Time Period 1: TP120. 2013 Malnutrition Improvement Progress Indicator 1: TP120. Nutrition-Related Health Improvements Indicator 2: 120A. Female Children Indicator 3: 120B. Male Children Outcome 1: OC120. Consumption of Nutritious Meals Indicator 1: OC120. Percent Population Eating Healthy Output 1: O121. 2013 Nutrition Training Workshops Indicator 1: O121. Passing Nutrition Grades Output 2: O120. 2013 Nutritious Meals Consumed Indicator 1: O120. Daily Nutrition Target Days per Month Operation 1: OP121. 2013 Conduct Training Workshops Indicator 1: OP121. Workshops Held Input 1: I121. 2013 Nutrition Training Workshop Indicator 1: I121. Training Labor Operation 2: OP120. 2013 Develop Nutrient Training Materials Indicator 1: OP120. Training Materials Completed Input 1: I120 2013 Nutrition Training Manual Development Indicator 1: I120. Training Material Development Labor Summary Clubs using DevTreks can carry out the basic monitoring and evaluation of projects, programs, and technologies. Clubs can solicit help with projects that are struggling and share structured knowledge explaining success and failure. Networks can build knowledge banks that explain why projects, programs, and technologies succeed or fail and pass that knowledge down to future generations. The result may be more effective social improvement programs and people who improve their lives and livelihoods. Footnotes 1. DevTreks focuses on software development, not content development. We don’t run calculations for every URI in a data set, just enough to carry out software tests. The monitoring and evaluation numbers generated by these analyzers were tested within the limitations of existing data sets. Those data sets have limited M&E data, but were successfully tested using Version 1.5.0’s 88 M&E 1 and 2 calculators and analyzers. They’ll continue to be tested using new data sets and combinations of base elements, prices, amounts, dates, amortization periods, and multipliers. We recommend running all analyses using the Full, rather than Mobile, view. The Full view generates html faster than the Mobile view. Once the Full view has been saved, click on the Mobile view to save that view. References References can be found in the Monitoring and Evaluation 1: Food Nutrition reference. References Note We try to use references that are open access or that do not charge fees. Improvements, Errors, and New Features Please notify DevTreks (devtrekkers@gmail.com) if you find errors or can recommend improvements. Video tutorials explaining this reference can be found at: https://www.devtreks.org/commontreks/preview/commons/resourcepack/M and E Analysis 1/518/none/ DevTreks –social budgeting that improves lives and livelihoods 1