Monitoring and Evaluation Analysis 2 Last Updated: August 03, 2018; First Released: February 14, 2014 Author: Kevin Boyle, President, DevTreks Version: DevTreks 2.1.4 A. Introduction This reference explains how to start to collect, measure, and analyze, basic monitoring and evaluation data (1*). DevTreks believes that every project, program, and technology, from malnutrition improvement projects to garment factory safety programs to new climate change abatement technologies, has a story to tell and lessons to teach. Those lessons can only be learned when monitoring and evaluation data about work progress and performance is collected, measured, aggregated, analyzed, explained, and saved in online knowledge banks. A full, uniform, and accurate accounting of work progress and performance for investments made in malnutrition improvements, medical treatments, conservation practices, flood prevention technologies, factory safety programs, and public infrastructure, should be one or two links away for everyone. If a business owner, lender, nonprofit member, government official, parent, or citizen, needs to make a decision involving project, program, and technology progress and performance, they should have ready access to the best data available. This reference introduces another DevTreks way to build these knowledge banks. Section Page Data URLs 2 Base Element M&E Calculations 5 M&E Analyses 6 Multipliers 10 Multimedia Support 11 Complementary Analyses 12 Knowledge Banks and Conclusions 14 Appendix A. Multipliers 18 Appendix B. M&E Analysis Examples 30 Appendix C. DevPacks M&E Analysis 58 The analyses documented in this reference derive from the M&E frameworks and calculators documented in the M&E Calculation tutorial. That tutorial should be completed prior to this tutorial. B. Data URLs (2*) The Construction Analysis 1, Health Care Analysis 1, Ag Production Analysis 1, Malnutrition Analysis 1, and Work Breakdown Structures, tutorials demonstrate how basic monitoring and evaluation data can be structured to support the analyses shown in this reference. The Calculators and Analyzers explained in this reference can be found at the following URIs (3*). Appendix B includes examples that explain the analytic results. M&E Calculators https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/Monitoring and Evaluation Calculators/53/none/ M&E 2 Analyzers https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/ Monitoring and Evaluation 2 Analyzers/61/none/ Sample datasets, containing M&E analyses, can be found at the following URLs. Appendix B includes examples that explain the analytic results. These datasets are mainly used to test whether or not the calculators and analyzers work as documented and that all of their styles are correct. They are by no means examples of really good M&E analysis (refer to Footnote 2 in the M&E Introduction reference). Version 2.1.0 tests are documented in the M&E Calculation reference. Input Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Inputs/2651/none/ Output Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outputs/2656/none/ Operation Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Operations/2654/none/ Component Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Components/2650/none/ Outcome Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outcomes/2655/none/ Operating Budget Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Op Budgets/2653/none/ Capital Budget Service URI https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Investments/2652/none/ Multimedia URI: https://www.devtreks.org/hometreks/select/farmworkers/resourcegroup/M and E Stories/144/none/ Story URI: https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/Monitoring and Evaluation Malnutrition Stories/54/none Localhost URIs The Food Nutrition Club in the HomeTreks Network Group owns this data (i.e. if needed, switch clubs). http://localhost/hometreks/preview/farmworkers/outcomegroup/ME2 Food Consumed/40/none/ http://localhost/hometreks/preview/farmworkers/investmentgroup/ME2 Malnutrition Projects/275505679/none/ http://localhost/hometreks/preview/farmworkers/budget/ME2 Project 01/273083907/none/ Additional examples of Progress Analysis data sets can be found in the Earned Value Management 1 reference. Additional examples of Change Analysis data sets can be found in the Change Analysis 1 reference. Additional examples of Input and Output data sets can be found in the Price Analysis 1 reference. Examples of additional M&E datasets, dealing with climate change, can be found in the Technology Assessment 2 tutorial. Examples of using custom algorithms, Indicators, and TEXT datasets, for conducting more advanced M&E can be found in the Social Performance Analysis tutorial. C. Base Element M&E Calculations Analyses use the results of base element calculations. The M&E 2 Calculation tutorial explains that each calculation is specific to a specific type of base element. Output calculations are only pertinent to Output elements, Input calculations to Input elements, Time Period calculations to Time Period elements, and so on (4*). Make sure to understand the references in that tutorial prior to running M&E analyses. The following image displays a typical Indicator calculation. D. M&E Analyses Separate M&E analyzers are available for all base elements. The Calculator and Analyzer 1 reference documents how all DevTreks’ Analyzers work. The Analysis Type property of M&E Analyzers is used to specify the type of analysis to run. Although these analyzers use similar patterns to the analyzers documented in the Resource Stock Analysis tutorial, some differences require explanation. * No 10 Indicator Limit: These analyzers do not implement the 10 aggregated Indicator limit found in the Resource Stock Analyzers because descendent Indicators are not aggregated into ancestors. Even so, the M&E Introduction reference points out that only the most important Indicators should be calculated and analyzed –online, html, data has limitations. * Indicator.Labels: The Resource Stock Calculator requires unique Indicator.Labels. That calculator references Indicators in TEXT datasets by their Indicator.Label. The M&E 2 Calculator does not require that Indicators have unique Labels. Instead, the calculator references Indicators in TEXT datasets by their index position (i.e. Score = index position 0, Indicator 1 = index position 1, Indicator 15 = index position 15), rather than their Label. * Scores: Version 2.0.4 began allowing a Score to be used in M&E calculations. Unlike the Resource Stock Analyzers, the Score is actually just another Indicator stored in the zero index position of collections of Indicators. In most cases, care must be taken not to aggregate the Score with its sibling Indicators. In most cases, Scores should be analyzed as standalone Indicators (i.e. calculated as an aggregation of the sibling Indicators) by setting their Labels to different values than their sibling Indicators’ Labels. The Social Performance tutorial includes an example of a Score being used to carry out Multi Criteria Analysis, with the sibling Indicators being the Criteria. M&E analyses are carried out in two stages. The first stage aggregates base elements using standard Analyzer aggregators, such as Year or Label. The second stage then aggregates the index-defined Indicators that have been aggregated in each base element. Unlike Net Present Value (NPV) and Life Cycle (LCA) analysis, the number of observations is not based on the number of aggregated based elements, but the number of distinct Indicators in the aggregated base elements. With the exception of Input and Output analysis, NPV calculators must be run prior to running an analysis –the NPV calculator is used to pull fresh database data together. Another difference from Resource Stock, LCA, and NPV analysis is that children elements are not aggregated into ancestors. Input Series Indicators are treated as pertinent only to Input Series, Inputs to Inputs, Operations to Operations, Outcomes to Outcomes, Time Periods to Time Periods, and so on (4*). Analysis Result Properties The results of running analyses are displayed using the following basic properties for all base elements: M and E Stage: The stage of the monitoring and evaluation analysis. Options include baseline, realtime, midterm, final, and expost. Each M&E Analyzer includes a selection list for setting this property. Total Name: name of the total Indicator (5*) Total Label: WBS Indicator label used to aggregate Indicators (5*) Total Date: Date that the Indicator was measured Most Likely Estimate Total: total of the Indicator.IndTMAmount, or Indicator Most Likely Estimate, calculations Most Likely Unit: corresponds to Indicator.TMUnit Lower Estimate Total: total of the Indicator.IndTLAmount, or Indicator Lower Estimate, calculations Lower Estimate Unit: corresponds to Indicator.TLUnit Upper Estimate Total: total of the Indicator.IndTUAmount, or Indicator Upper Estimate, calculations Upper Estimate Unit: corresponds to Indicator.TUUnit The following image displays a typical view of an M&E analysis. E. Multipliers Base element multipliers, such an Operation Amount, Time Period Amount, or Input Times, are used to change the quantity of Indicators for each base element. Version 1.9.4 simplified the use of multipliers by multiplying the quantitative analyzer properties (i.e. Most Likely Estimate, Lower Estimate, and Upper Estimate) by these multipliers. Multipliers come from before-aggregated base elements. Appendix A gives examples demonstrating the use of multipliers. In hindsight, the author probably would have preferred that the multipliers also change the calculations, rather than just the analytic results. The standard protocol for this type of decision is for Network administrators to inform their information technologists about their preferences. F. Multimedia Support Version 2.0.2 began displaying the first image or video contained in an analyzer’s Media URL property on the Preview panel. That allows good contextual information about the analytic results to be evaluated prior to loading the analysis. The following image, from the Preview panel of an Input base element, shows that multimedia is not being used properly. The same image is being displayed for all analyses and the description field of the first Analyzer has been left blank. Decision makers might conclude that these analyses do not supply serious decision support. Professional M&E analysis requires effort (and staff) to complete. Serious content development is not the role of a software development company, but it is the role of members and clubs in social networks. G. Performance Analysis The Performance Analysis and Social Performance tutorials demonstrates how to use various Performance Measures, such as Incremental Cost Effectiveness Ratio, to support decisions that combine monitoring and evaluation data with benefit and cost data. Indicators can include prices, quantities, costs, or benefits, which means they can be used in Performance Measures such as Cost per Unit Output or Output per Unit Input (6*). H. M&E Analysis and Net Present Value Analysis (NPV), Life Cycle Analysis (LCA), and Resource Stock Analysis These analyzers match the same set of analyzers found in the Benefit Cost Analysis 1, Life Cycle Analysis 2, and Resource Stock Analysis 1 references. Section M’s Sample Data Sets contain M&E, NPV, and LCA Analyzers that demonstrate how these techniques relate to one another. The tools are often used together to tie monetary benefits and costs to nonmonetary Indicators of performance, outcomes, and impacts. Cost effectiveness analyses are conducted using both sets of data (6*). I. Custom Analysis This reference explains how to analyze base element M&E data. The structure of base element data may not support many types of analyses, such as CTAs that use randomized control trial data or comparisons of base elements that are not siblings. Three options for custom analysis can be used to overcome this shortfall: 1. DevPacks: The DevPacks tutorial introduces base elements that allow analysis of arbitrary structures of hierarchical data, including randomized control trial data and non-sibling base elements. Appendix C, DevPacks M&E Analysis begins to demonstrate how to use them to analyze M&E data. 2. Data URLs: The introductory reference introduces the use of Indicator.URL and Score.JointDataURL properties of calculators and analyzers that allow custom datasets to be linked directly to calculators and analyzers. 3. Packages: The Calculators and Analyzers and Performance Analysis references explain that datasets can be packaged, downloaded as zip files, cleaned up, imported into statistical packages, and further analyzed. The Conservation Technology Assessment and Social Performance Analysis tutorials explains these techniques further. Future releases and references will also explain these techniques further. J. Knowledge Bank Standards All monitoring and evaluation analyses should be entered into online knowledge banks (i.e. production servers as contrasted to development servers) that can be used to monitor and evaluate projects, programs, and technologies. That structured evidence must be passed down to future generations. These knowledge banks aggregate and analyze all of the data in a network. Future references will demonstrate how these knowledge banks will evolve (i.e. semantic data, forecasts, national decision support systems) to support future decision making needs. The flexibility offered by DevTreks in documenting M&E means that networks need to develop “rules” explaining the “standards” that should be followed by clubs in their network. The “standards” make it possible to build knowledge banks. Summary and Conclusions Clubs using DevTreks can start to carry out the basic monitoring and evaluation analysis of projects, programs, and technologies. Clubs can solicit help when projects falter, programs fail, or technology fumbles. They can share structured evidence explaining how to increase accountability, stay on budget, improve performance, and eliminate waste. Networks can build knowledge banks that explain why projects, programs, and technologies, succeed or fail and pass that knowledge down to future generations. The result may be malnutrition projects that improve children’s long term health, agricultural development programs that reduce farmers’ risks, factory safety programs that save factory workers’ lives, hospitals that treat patients more affordably, schools that educate children more effectively, governments that mitigate climate change more convincingly, and people who improve their lives and livelihoods. Footnotes 1. Monitoring and Evaluation Analysis includes more advanced analytic techniques than those used in this reference. Future releases, such as Version 2.1.4’s Social Performance Analysis algorithms, include some of these techniques. 2. The August, 2014 upgrade found a lot of gaps in these data sets on the cloud site, but found that the binaries work as documented. The December, 2015 upgrade found several bugs or flaws, including mistakes with how multipliers were used and how the Indicator.QTotal property was used. That release was a CTAP release and the Technology Assessment 2 tutorial further demonstrates best M and E practices. The August, 2016 refactor deprecated the M&E 1 Calculator and Analyzers to focus on the M&E 2 Tools. Version 2.0.4 upgraded the M&E 2 calculator to similar patterns as the Resource Stock Calculators, thereby allowing the nascent CTA algorithms documented in the Technology Assessment tutorials to be used in M&E calculation and analysis. Version 2.14 upgraded the calculator patterns to support the more advanced M&E demonstrated in the Social Performance Analysis tutorial. 3. The Monitoring and Evaluation Calculation tutorial documents that the M&E 1 tools found in some of these URIs have been deprecated in favor of the M&E 2 tools. Some of the datasets that were built for the deprecated M&E Calculator 1 may have been updated to test state management for the M&E 2 Calculator. State management should not be confused with content management. 4. The M&E references, found in the Monitoring and Evaluation Calculation 1 reference, explain why descendant Indicators are not aggregated into ancestors. Those references treat the Indicators associated with different base elements as serving distinctly different purposes in an M&E analysis. An Operation element is not a simple aggregation of children Input Indicators (as would be the case for the Inputs’ costs). Instead, the Input Indicators track the resources used in a project, while Operations track the actual activities being carried out. The Resource Stock tutorials demonstrate an alternative set of Indicators and analytic techniques where the Indicators are aggregated in the same manner as costs and benefits. 5. The Totals are generated by aggregating each individual Indicator by its WBS Label. The names and units derive from the first Indicator being aggregated. 6. The current calculator and analyzer patterns do not allow mathematical operations that use combinations of different base elements, such as Cost per Unit Output, Output per Unit Input, or Optimal Farm Activities given known Input and Output constraints. Instead, the calculated and analytic results must be manually manipulated to produce those types of calculations. The Version 2.14 release includes examples of algorithms that address this need. 7. Indicators for Investment, Budget, and Time Period base elements measure M&E Impacts. That is, what evidence exists that money has been, or is being, spent well? Have lives and livelihoods actually improved? A useful experiment is to try to find this evidence for just about any government expenditure, anywhere. As another example of “doing it right”, interpret the following question: “Other than institutional factors, is there any technical reason that this data can’t be completed online, stored uniformly online, and easily accessed online by people and machines, for every major government expenditure in the world?” 8. These types of analyses are based on the general principle “is it better to give a community another ‘expert’ analysis, or is it better to give them the tools needed to build the analyses and make the decisions themselves about consequent courses of action?”. References References can be found in the Monitoring and Evaluation 1: Food Nutrition reference. References Note We try to use references that are open access or that do not charge fees. Improvements, Errors, and New Features Please notify DevTreks (devtrekkers@gmail.com) if you find errors or can recommend improvements. Video tutorials explaining this reference can be found at: https://www.devtreks.org/commontreks/preview/commons/resourcepack/M and E Analysis 2/519/none/ Appendix A. Multipliers All analyses use the multiplier-adjusted Most Likely Estimate, Lower Estimate, and Upper Estimate, properties. Ancestor multipliers are not used. For example, the Operation Amount property will not be used to change children Input Indicators (but they will change the Operation’s Indicators). Inputs that are children of Operations use the Input.OCAmount, as a multiplier (along with the Times multiplier). As of 1.9.4, Inputs that are children of Components use the Input.CAPAmount, as a multiplier (along with the Times multiplier). The Input.AOHAmount is not used as a multiplier. Careful use of multipliers makes it easier to use generic “unit Indicators” rather than Indicators that are always tied to one specific project or technology. That’s the same way that base element unit Input and Output prices work. Rather than being useful in only one particular project or technology, the Indicators can then be used in any project or technology. The following images display the initial M&E calculations in one year of an Operating Budget that has not been adjusted by any multipliers (default values = 1). The bottom image of each base element shows the before and after results of a Totals Analysis after changing base element multipliers. The calculations don’t change, but the analytic results, specifically the base Totals used in all analyses, change by the multiplier. Some of the Scores have Indicator.RandomSeed = 0 and will vary each time the Monte Carlo algorithm, algorithm1 and subalgorithm1, is run. Time Periods Totals Analysis TimePeriod.Amount = 1 Totals Analysis TimePeriod.Amount = 2 Outcomes Totals Analysis Outcome.Amount = 1 Totals Analysis Outcome.Amount = 2 Outputs Totals Analysis Output.Amount = 1, Output.CompositionAmount = 1, Output.Times = 1 Totals Analysis Output.Amount = 2, Output.CompositionAmount = 2, Output.Times = 2 (2*2*2 = 8 is the multiplier) Operations Totals Analysis Operation.Amount = 1 Totals Analysis Operation.Amount = 2 Inputs Totals Analysis Input.OCAmount = 1, Input.Times = 1 Totals Analysis Input.OCAmount = 2, Input.Times = 2 (Investments and Components use Input.CAPAmount as the multiplier) (2*2 = 4 is the multiplier) Appendix B. Monitoring and Evaluation Analysis Examples Although the following examples show that the current crop of analyzers use simple mathematical operations, such as addition and subtraction, when their underlying calculations derive from the algorithms documented in the Technology Assessment and Social Performance tutorials, the resultant metadata analysis can be quite powerful. In most cases, the comparative analyses shown in the following images do not display Inputs and Outputs because the quantity of data becomes difficult to interpret. Descendent base elements, such as Inputs and Outputs, can still be evaluated by running a Totals Analysis, and in some cases, by setting the “Compare Using” option to “none”. All of these datasets were strictly used to test the M&E tools. No attempt was made to ensure that the underlying economic content, found in the base elements and evaluated using NPV calculators, was meaningful. In general, equal attention must be given to base element economic content and Indicator content. Decision making is enhanced when benefit and cost, or economic, content is used together with performance, or Indicator, content. 1. Totals Analysis A Totals Analysis sums Indicators for every base element in an analysis. All analyzers run this analysis for each aggregated base element before carrying out additional calculations. All totals derive from initial base element M&E calculations. This is the only analysis that includes Inputs and Outputs in Operation, Component, Outcome, Budgets, and Investment analyses. The remaining analyses do not display Inputs and Outputs because the quantity of data becomes difficult to interpret. Version 2.0.4 stopped rerunning M&E base element calculations prior to summing the calculations because the original base element calculations aren’t changed during analyses. Appendix A explains how to use multipliers to adjust the original base element calculations. So what is the difference between the results of running M&E calculators and this particular analysis? This analysis aggregates M&E calculations using each Indicator’s WBS label (4*). For example, the following Input Totals Analysis has aggregated 4 quarterly Indicators in the 2013 Series element and 4 quarterly Indicators in the 2014 Series element. Although not shown, 2015 and 2016 Input Series Indicators were similarly aggregated. The Indicators in both series were aggregated because they had the same WBS label. The Score Indicator has 1 Observation because each set of base elements has 1 Score. The following image shows a Totals Analysis that has set the Aggregate Using Option to Labels. The previous Totals Analysis set this option to None. Although the Totals Analyzers may display the same “Compare Using” option as the remaining Analyzers, it’s not supported –aligning too much data does not necessarily enhance decision making. The following image displays the results of this analysis. In this case, the 4 Input Series base elements had the same WBS Label resulting in the final aggregation of the 16 Indicators, or 4 Indicators * 4 Series. The Score Indicator has 4 Observation because each set of the 4 base elements has 1 Score. The base elements were first aggregated and then the Indicators within the base elements were aggregated. When the equivalent analyses were run on the cloud dataset, the parent Input’s Total Amounts all showed zeros. The investigation revealed that the Input.OCAmount, a multiplier used to set Totals, had been set to zero. The error with the cloud dataset was left in place to make this point. 2. Statistics 1 Analysis A Statistics Analysis uses the Totals calculations to measure basic statistical properties of aggregated Indicators. Total, Median, Mean, Variance, and Standard Deviation statistics are generated for all of the Indicators in aggregated base elements. A Statistical Analysis of the quarterly data shown for the first of the previous Totals Analysis, or Aggregate Using = None, confirms that 4 separate quarterly Indicators are being aggregated, that is Observations = 4. A Statistical Analysis of the quarterly data shown for the second of the previous Totals Analysis, or Aggregate Using = Labels, confirms that 16 separate quarterly Indicators are being aggregated, that is Observations = 16. The base elements were first aggregated by their InputSeries.Labels, and then their Indicators were aggregated by their Indicators.Labels. The Indicators will be aggregated by Label regardless of the type of aggregator being used in the analysis. The following Operation Statistical Analysis displays basic statistics associated with malnutrition Operations. The Operation elements track Indicators that are different than the Input elements. Note that base elements, such as Inputs, that don’t have Indicators will still be displayed, but no analysis can take place of base elements that don’t have M&E calculations. Notice the use of 3 different Labels for the Operation’s Indicators. The following Statistical Analysis for Capital Budgets demonstrates the use of the Compare Only option (7*). Scores will have the same number of observations as the number of base elements that have been aggregated, while sibling Indicators will have the same number of observations as the number of base element Indicators that have been aggregated. 3. Change 1 Analyses The Change 1 Analyses use the Totals calculations to measure incremental changes in aggregated Indicators. A Change by Year Analysis measures incremental changes between aggregated Indicators that have different Years. A Change by Id Analysis measures incremental changes between Indicators that have different Ids. A Change by AlternativeType Analysis measures incremental changes between aggregated Indicators that have different Alternative Types. Changes are analyzed in ascending order (Id = 1,2,3; Year = 2000, 2001, 2002; AlternativeType = A, B, C). The first member of the sequence will be used as a “Base” element to make comparisons. The sibling sequence member immediately before the current sequence member will be used as an “x-1” element to make comparisons. Gaps in the sequence, such as a missing Year, will be ignored. Further documentation about these analyses can be found in the Change Analysis 1 reference. A Change by Year Analysis of the quarterly data shown for the first of the previous Totals Analysis, or Aggregate Using = None, confirms that 4 separate quarterly Indicators are being aggregated, that is Observations = 4 and then compared between the 4 years of Input Series. In this analysis the second year, 2013, is being compared to the first year, 2012, and only shows Base changes. The third year, 2014, is being compared to the first and second years, 2012 and 2013, and shows Base changes and Amount, or x-1, changes. Given that the Score’s Math Expression aggregates its 4 sibling Indicators, both the Score and Indicator changes are equal. The following image displays an Outcome Change by Alternative Analysis. The Indicators in two alternative Outcomes, A and B, are being aggregated and then compared. The Alternative Types and Labels had to be set correctly in the Outcomes being compared. For example, if Outcome A’s Indicators mistakenly had an Alternative Type = B, no changes would occur. The following Operating Budget Change by Year Analysis shows the results of comparing a 2013 Time Period to a 2012 Time Period (7*). The analysis compares Indicators that have different base element years, but that have the same Indicator Labels. The 2013 Time Period found the corresponding 2012 Time Period, Outcome, and Operation Indicators because the Indicator Labels were the same. Inputs and Outputs are not included in Budget Change Analysis because too much data needs interpretation. No Amount Changes took place because no xminus1 Time Periods were present. Note the mistake with the Time Period base element Label. Not setting this properly (i.e. to none) doesn’t affect this particular analysis but it will affect other types of analyses. To successfully compare Time Periods in Budgets, the Time Periods must have Indicators with the correct Labels, Years, and Alternative Types (7*). To successfully compare Budgets in Budget or Investment Groups, the Budgets and Time Periods must have Indicators with the correct Labels, Years, and Alternative Types. The following analysis is comparing Investment A to Investment B. No aggregators were chosen when the analysis was run. Each Budget has two time periods. All of the elements in Budget A have Alternative Type = A, while all of the elements in Budget B have Alternative Type = B. This image displays Budget A’s Time Periods and 1 Outcome. Note that 2013 Time Period is an aggregation of the Indicators in two Time Periods because they all had Alternative Type = A. This dataset can be found on localhost. The equivalent cloud dataset is missing the second Investment. The original 98 M&E tools, which included the ME1 Calculators and Analyzers, were hard to maintain and test. 4. Progress 1 Analysis A Progress 1 Analysis uses the Totals calculations to measure actual versus planned progress for aggregated Indicators. The planned Indicators use a Target Type property of Benchmark. The actual Indicators use a Target Type property of Actual. The U.S. GAO (2009) emphasizes using Earned Value Management (EVM) best practices to ensure the cost of work completed aligns with the value of work performed. A key requirement of EVM is to measure budget variances and scheduling variances. Budget variances measure the costs (and benefits) of work planned versus actual work completed. Scheduling variances measure the amount, quality, and timeliness of work planned versus actual work completed. EVM uses both variances to measure changes in the value of work planned versus actual work completed. A Progress 1 Analysis measures all of these variances. DevTreks’ best practices extend EVM to include Outputs (work progress), Outcomes (technical performance), Benefits (earned value), and M&E Indicators (performance effectiveness). Further documentation about these analyses can be found in the Earned Value Management Analysis 1 reference. A Progress Analysis of the quarterly data shown for the first of the previous Totals Analysis, or Aggregate Using = None, shows that, the Target Types for the 2013, 2014, 2015, and 2016, years have been set, respectively, to benchmark, actual, benchmark, and actual. Progress is being compared between the 2013 benchmark and 2014 actual, and the 2015 benchmark and 2016 actual. This image displays the Score analysis only. A real Progress Analysis should compare 4 benchmark years with 4 actual years. The following image of the Indicator Progress Analysis shows that the 4 separate quarterly Indicators have been aggregated in each year prior to the final analysis, that is Observations = 4. The following image displays an Operation Progress Analysis. The Indicators in two Operations with different Target types (benchmark and actual) and their children Inputs (also benchmark and actual) are being aggregated and then compared. The Target Types and Labels had to be set correctly in the Operation and Input Indicators being compared. For example, if the Benchmark Operation’s Input Indicators mistakenly had a Target Type = none, no Input Indicator changes would occur. The Score has Upper and Lower numbers because it uses CTA algorithms to set those numbers while the sibling Indicators do not use any algorithms. To successfully compare Time Periods in Budgets, the Time Periods must have Indicators with the correct Labels, Years, and Target Types (7*). To successfully compare Budgets in Budget Groups, the Budgets and Time Periods must have Indicators with the correct Labels, Years, and Target Types. The current version does not display Input or Output Indicators in Capital or Operating Budget Analysis. The quantity of data displayed can make interpretation of the data difficult. If demand warrants, they can be included in future releases. The following Progress Analysis for Operating Budgets shows how the Compare Only option works: The following image displays a Progress Analysis of 2 Malnutrition projects. Note how the Target Type properties have been properly set. Also note that the Investment.Labels for the 2 Investments are the same (BUD01), as well as the Investment.Indicator.Labels (I122). The second Investment uses the Investment.Labels to find the base element comparator, ME Project 01. It uses Investment.Indicators.Labels to find the Indicator comparator, Food Security Score. It’s useful to experiment by changing Labels in selected base elements to understand how Labels affect analyses (i.e. take a close look at the Labels used for the Investment Statistical and Change Analyses in the examples above). Appendix C. DevPacks M&E Analysis (8*) DevPacks support the analysis of arbitrary hierarchies of data, such as randomized control trial data. They also support a variety of custom analyses, such as the comparisons of base elements that are not siblings. When the data being analyzed is observational M&E data stored in Data URL TEXT files, the resultant metadata analysis can provide scalable and powerful decision support. This appendix contains examples demonstrating different ways to use DevPacks to conduct M&E Analysis. The DevPacks tutorial should be read prior to this Appendix. Example 1. Change by Alternative (non-sibling base element analysis) Appendix B displays images of comparative M&E analysis of 2 Capital Investments in 2 alternative malnutrition projects. Although the following localhost URL shows that these are 2 sibling Investments in 1 Investment Group, what if they were not siblings? How could the same Appendix B analyses be conducted? http://localhost:5000/hometreks/preview/farmworkers/investmentgroup/ME2 Malnutrition Projects/275505679/none [The following cloud URL shows why cloud URLs aren’t being used. Labor constraints resulted in incomplete data for the 2nd investment. https://www.devtreks.org/hometreks/preview/farmworkers/investmentgroup/M and E 2 Malnutrition Projects/275505679/none] Example 1 in Appendix C of the Resource Stock Analysis tutorial demonstrates how to use DevPacks data services to compare 3 non-sibling Operating Budgets. The following URL demonstrates how to use the same techniques with the M&E tools to compare these two non-sibling Capital Budgets. http://localhost:5000/hometreks/preview/smallholders/devpackgroup/M and E RCT Tests/78/none The following image shows the resultant Change by Alternative analysis being completed using the more flexible DevPacks data services. This analysis used 2 of the 4 possible levels of DevPacks base elements. The Resource Stock Analysis tutorial discusses current limitations with this approach. DevTreks –social budgeting that improves lives and livelihood 1