Monitoring and Evaluation Calculations
Last Updated: September 18, 2018. Original Release: March 28, 2013
Author: Kevin Boyle, President, DevTreks 
Version: DevTreks 2.1.6
A. Introduction
The DevTreks sibling reference, Monitoring and Evaluation (M&E) Introduction: Food Nutrition introduces the background logic for M&E calculation and analysis in DevTreks. This reference documents how to use DevTreks’ M&E calculators. These calculators support the basic monitoring and evaluation of projects, programs, and technologies (1*).
Section
Page
M&E Calculator Properties
1
M&E Base Element Calculators
12
Data URLs
13
M&E Analysis
15
Summary and Conclusion
16
Appendix A. Deprecated M&E1 Calculators and Analyzers
18

B. M&E Calculator Properties (2*)
Version 2.0.4 upgraded this calculator to similar properties and patterns as the Resource Stock Calculator, referenced in the Resource Stock Analysis tutorial. The Stock patterns promote consistency in the use of indicators and accommodate risk and uncertainty in indicator measurement and valuation. Importantly, all of the custom algorithms, referenced in the Technology Assessment and Social Performance tutorials, can also be used with this calculator.
The following URLs and images display simple indicators in the M&E Calculator 2. These properties coincide closely with the Resource Stock Calculator. For that reason, the properties will not be defined in this reference. Instead, refer to the Resource Stock Calculation tutorial for their definitions. This data is owned by the Family Budgeting and Food Nutrition club in HomeTreks.
https://www.devtreks.org/hometreks/preview/farmworkers/output/2013 Number of children examined for malnutrition/2141223448/none
Indicator Example 1 – no risk and uncertainty calculations (the Indicator.MathExpression fills in the Indicator.QT and Indicator.QTM properties)

Indicator Example 2 – risk and uncertainty calculated using CTA algorithms (the Indicator.Algorithm and Indicator.Subalgorithm properties use the remaining Indicator properties to fill in the Indicator.MathResult)

Version 2.1.4 and Version 2.1.6 deprecated the use of the Score.DataURL property explained in the next example in favor of the Indicator.URL and Score.JointDataURL properties. The Social Performance Analysis tutorial demonstrates how the upgraded pattern does a better job with advanced analyses (i.e. R and Python algorithms). 
Indicator Example 3 (deprecated pattern) – risk and uncertainty calculated using CTA algorithms with Data URL TEXT datasets (this data is owned by the Natural Resource Stocks club)
https://devtreks1.blob.core.windows.net/resources/network_carbon/resourcepack_1548/resource_9105/MandEDataURL2.csv

https://www.devtreks.org/greentreks/preview/carbon/output/CTA and M and E Example 1/2141223473/none



The following image demonstrates that the Version 2.1.6 upgraded patterns emphasize the use of the Indicator.URL property (i.e. the same pattern used in the DevTreksStatsApi app).
 

A major difference from the calculator patterns in the Resource Stock Calculator is that the Score is now technically just another indicator, rather than a completely different type of Indicator. The Score properties are set in a zero-based indexed indicator. 
The following image of Score properties show that, for the sake of consistency, they appear similar to the Resource Stock Calculator’s Score properties. 

The following properties differ from the Resource Stock Calculator and require further explanation. 
Score Label, Related Label, and Date: These properties can also be set for the Score. In effect, the Score will be treated like any other indicator when analyses are run. In general, these properties should be set separately for the Score –they should not match sibling indicators, or the Score will be aggregated together with the siblings.
Score Iterations, Score Confidence Interval, and Score Random Seed: Each indicator, including the Score, now contain these properties. For the sake of consistency, this version only displays and uses these properties from the Score. This version uses the Score properties with sibling Indicators as well.
Score Joint Data URL, URL, and Data URL: This calculator does not include the Stock Calculator’s Joint Data URL property. Instead the Indicators[0].URL property is used to hold the Joint Data URLs. Unlike the Stock calculator, Calculator.DataURL and Indicator.URL TEXT datasets can also be used to set the Score calculations. 
Score Math Expression: This calculator can use the zero indexed indicator or Score, I0, in the Indicator.MathExpression (I0.QT*I1.QTM). Scores do not display an Indicator’s Q1 to Q5 properties because those properties are not believed essential for calculating Scores (until proven otherwise). The Score Q1 to Q5 properties should not be referenced in the Score Math Expression.
The following list explains the differences from the Resource Stock Calculator Indicator properties. 
Resource Stock Indicator.Label and TEXT datasets: The Resource Stock Calculator requires that each Indicator.Label in a base element is unique. That calculator uses Labels as unique keys in the collections of indicators used to carry out analyses. TEXT datasets, such as Indicator.DataURL and Indicator.URL, also reference and update Indicators by their unique Labels.
M&E 2 Indicator.Label and TEXT datasets: This calculator does not require unique Labels. In fact, most M&E calculations may prefer using collections of Indicators with the same Labels. For example, many examples in the M&E 2 Analysis reference demonstrate using temporal Indicators, such as quarterly or semiannual Indicators, that have the same Label. The calculator uses an Indicator’s indexed position, rather than Label, in calculations and analyses (i.e. Score = index position 0, Indicator 1 = index position 1, Indicator 15 = index position 15). The Label is only used to aggregate Indicators prior to conducting any analyses. TEXT datasets, such as Calculator.DataURL and Indicator.URL, also reference and update Indicators by their indexed position, rather than their Label.
Further examples of the use of TEXT datasets with M&E calculations can be found in the Technology Assessment 1, CTA 01, tutorial. The Social Performance Analysis tutorial demonstrates that existing TEXT datasets used with Stock calculators do not need to be changed when an algorithm, rather than the M&E calculators, manipulates the data.
Indicator.BaseIO and Score.BaseIO: The Resource Stock calculators use this property to update the underlying Input or Output’s properties, such as price or quantity. The M&E calculators currently do not. Current thinking is that the two sets of calculators complement one another and should be used jointly. In other words, legitimate analysis can be run by using the Resource Stock calculators to document Inputs and Outputs and to use Stock Analyzers to automatically analyze those aggregated Indicators in the remaining base elements. It may not always be necessary to run M&E calculators for every base element and it probably shouldn’t be done for base element Indicators that can be automatically calculated using the Stock analyzers. 
The Technology Assessment references have concrete examples that make this point in greater detail. The CTA 01 reference demonstrates that economic performance Indicators, in particular, may be analyzed better using Stock Indicators –the cost and benefit Indicators can be aggregated in the same manner as the base element costs and benefits.
The following image of the Score Totals for Example 3 demonstrates that Score analytic results are displayed as Indicator 0, based upon a zero-based indicator index. Scores do not display an Indicator’s Q1 to Q5 properties.

The final calculations and analyses look cleaner and more professional when all properties are filled out for each indicator (if I1.Q2 is not being used, set I1.Q2Amount=0 and I1.Q2Unit=none). The following sections explain how each M&E calculator uses these properties to support specific types of analyses.
Each calculator supports up to 15 indicators plus the Score. The same indicator, such as I121 Labor for Training, can be added to more than one M&E element. For example, a two year project may use two separate Input Series to keep track of the same indicators. When first adding indicators, good practice is to add the indicators needed into a parent base element and then use the Relations properties to copy the indicators into children base elements (i.e. Input to Input Series, Operation Group to Operations). Once they have been copied, change the children as needed, return to the parent and update its views by making a base document, setting Use in Descendants= true and Overwrite Descendants=false, rerun the calculator, and save the results.
Individual indicators can’t be inserted or deleted, but indicators that have their Indicator.Name and Indicator.Label set to empty or “none” won’t be calculated or displayed. Indicator measurement should be carefully planned out, preferably using the results of previous M&E analyses, before indicators are entered by any club.
C. M and E Base Element Calculators
Separate M&E calculators are available for each M&E element, including Inputs, Outputs, Operations, Components, Outcomes, Operating Budgets, and Capital Budgets. Indicators can be added to each element within each application. For example, an Input M&E element can have indicators for the Input Group, Input, and Input Series elements. Operating Budgets can have indicators for the Budget Group, Budget, and Time Period base elements. 
Each calculation is specific to a specific type of base element. Output calculations are only pertinent to Output elements, Input calculations to Input elements, Time Period calculations to Time Period elements, and so on. This rule also holds for all descendent elements –Output Series calculations are only pertinent to Output Series, Input Series calculations to Input series, and so on. The result is that, unlike most other calculators and analyzers in DevTreks, calculations are never aggregated from descendants to ancestors. Output calculations never include Output Series data, Operations never contain Input data, and so on. The Monitoring and Evaluation 1: Food Nutrition reference explains the reason for this type of aggregation –Output indicators are not the same as Outcome indicators, Time Period indicators are not the same as Outcome or Component Indicators.
Use the Resource Stock Calculators when descendent indicators need to be aggregated into ancestors. 
Although NPV calculators do not need to be run prior to running the M&E Calculations, good practice is to update the base elements costs and benefits at the same time as the M&E calculations. The M&E Indicators add a performance dimension to the cost and benefits dimensions.
D. Data URLs
These calculators and sample data sets can be found at the following URLs (3*). Additional M&E URLs can be found in the CTAP reference in the Social Performance tutorial. Section F explains why the M&E 1 calculators and analyzers in some of these URLs are no longer used. Most sample datasets referenced in tutorials throughout DevTreks were rerun during Version 2.1.0 tests. Only a subset of the following datasets were rerun for Version 2.1.0. In addition, localhost:5000 datasets were rerun and their URLs can be found in other tutorials.
Calculators URI: 
https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/Monitoring and Evaluation Calculators/53/none/ 
Inputs URI:
https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Inputs/2651/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/input/2013 Nutrition Training Manual Development/2147397488/none
Outputs URI:
https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outputs/2656/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/output/2013 Number of food nutrient packages distributed/2141223449/none
Components URI: 
https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Components/2650/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/componentgroup/M and E Food Delivery/656/none
Operations URI:
https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Operations/2654/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/operationgroup/M and E 2 Food Delivery/758/none
Outcomes URI: 
https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Outcomes/2655/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/outcomegroup/Nutrition Delivery/36/none
Capital Budgets URI:

https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Investments/2652/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/investment/M and E Malnutrition 2 Project A/426/none

Operating Budgets URI:

https://www.devtreks.org/hometreks/select/farmworkers/servicebase/M and E Malnutrition Op Budgets/2653/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/budget/M and E 2 Operating Budget/273083904/none

Multimedia URI: 

https://www.devtreks.org/hometreks/select/farmworkers/resourcegroup/M and E Stories/144/none/

Story URI: 

https://www.devtreks.org/hometreks/select/farmworkers/linkedviewgroup/M and E Malnutrition Stories/54/none/
Version 2.10 tests
https://www.devtreks.org/hometreks/preview/farmworkers/linkedviewgroup/Monitoring and Evaluation Malnutrition Stories/54/none

E. ME 2 Analysis
The M&E Calculator 2 supports Totals, Statistical, Incremental Change, and Progress, analysis. The Monitoring and Analysis 2 tutorial documents theses analyses. 
F. Other Calculators (4*)
Version 1.9.4 moved documentation for the M&E Calculator 1 to Appendix A. It’s been deprecated in favor of M&E Calculator 2 simply because 49 calculators and analyzers can be maintained, improved, and tested, better than 98. Future releases may include additional calculators and types of calculations, such as those documented for M&E Calculator 1 (if they can be maintained well).
Summary and Conclusions
Clubs using DevTreks can use M&E calculators to track generic indicators that support the basic monitoring and evaluation of projects, programs, and technologies. Better monitoring and evaluation of projects, programs, and technologies may help people to improve their lives and livelihoods.
Footnotes
1. Analysts have developed a wide assortment of indicators for M&E analysis. For example, the US, CMS and the World Health Organization use Performance, or Outcome, Indicators that have advanced properties relative to the generic indicators used with the calculators in this reference (refer to the Health Care Analysis 1 reference). The Resource Stock Calculation 1 tutorial explains how to use both sets of indicators together.
2. Although the M&E Calculator 2 used prior to Version 2.0.4 had much simpler indicator properties, it turns out that several of those properties including Name, Label, Description, Date, Q1 and Q2, are the same in the upgraded calculator. It’s not particularly difficult to upgrade the old calculations to the new calculations (i.e. by using the Indicator.MathExpression to define the mathematical relation between Q1 and Q2). No real effort was made to ensure compatibility between calculator versions because the upgraded calculator has more to offer.
3. DevTreks doesn’t run calculations and analyses for every base element in a data set. Our primary role is software, rather than content, development. We run enough calculations and analyses on the localhost and cloud host to test how the calculators and analyzers work.
4.  Further examples of M&E Progress calculators and calculations can be found in Table 33, US, GAO 2009.

References
References for M&E calculation and analysis can be found in the Monitoring and Evaluation 1: Food Nutrition reference.
References Note
We try to use references that are open access or that do not charge fees.
Improvements, Errors, and New Features
Please notify DevTreks (devtrekkers@gmail.com) if you find errors or can recommend improvements.
Video tutorials explaining this reference can be found at:
https://www.devtreks.org/commontreks/preview/commons/resourcepack/Monitoring and Evaluation 1/476/none


Appendix A. Deprecated M&E Calculator 1 and M&E 1 Analyzers
This calculator was deprecated in favor of M&E Calculator 2 simply because 49 calculators and analyzers can be maintained and improved better than 98. It’s also useful in demonstrating that, without labor constraints, any number of M&E calculators can be developed and supported.
The following image displays an indicator in the M&E Calculator 1. Note that these indicators include separate Alternative Type or Target Type properties.

The only property differences from M&E calculator 1 are:
Target Type: Used with progress analyzers to measure actual progress achieved versus benchmark goals and targets. Options include benchmark, full target, partial target, and actual. The partial target option can be used when thresholds must be defined.
The M and E 2 Calculator documentation holds for this calculator as well. Separate calculators can be added to separate base elements and the calculations will not be aggregated into ancestor elements.
G. M&E 1 Analyzers
The calculations run from the M&E Calculator 1 supports Totals, Statistical, Incremental Change, and Incremental Progress, analysis (see the Monitoring and Evaluation Analysis 1 reference). M&E Calculator 2 is meant to be used together with the calculators and analyzers explained in the Net Present Value 1, Benefit Cost Analysis 1, Life Cycle Analysis 1 and 2, and Resource Stock, references. This calculator places greater emphasis on aggregating base elements prior to aggregating indicators compared to the M&E Calculator 1. Multiple different indicators can be measured in the same base element by using different labels for each indicator.Indicator properties, such as Date, Label, Target Type, and Alternative Type, must be set a specific way for each type of analysis, as follows:
1. Totals Analysis
A Totals analysis is the initial calculation run for any calculation or analysis that uses this calculator. The final Total is derived from the mathematical operation carried out of the Q1, Q2 and Weight properties. All analyses run this calculation prior to running any subsequent analysis. Section D. M&E Calculator Properties, displays a typical indicator. The base element’s Multipliers, such as Input.OCAmount, Output.Times, Operation.Amount, TimePeriod.Amount, do not change the final calculated, or analyzed, totals.
2. Statistics Analysis
A Statistics analysis supports the statistical measurement of indicators. This calculation is appropriate when indicators need gross numeric measurement rather than goal measurement. It’s also appropriate when alternatives need simple statistical comparisons. Aggregation takes place using the Label property. Only indicators with an Indicator Type property set to Actual are used in the analysis. Practical examples include tracking average drug prices, median farmland values, the probability of health care costs, or the variance of carbon pollution levels.  
3. Change Analysis
A Change analysis supports the analysis of incremental changes among indicators. A Change by Year calculation supports analysis of annual changes of indicators, such as 2012, 2013, and 2014. A Change by Id (or Label) calculation supports analysis of changes between indicators that have different labels, such as IN100, IN101, and IN102. 
A Change by Year analysis is appropriate when changes in indicators need to be analyzed yearly. It’s also appropriate when alternatives need annual comparisons. The measurement uses the Label and Date (i.e. year) properties to aggregate indicators. The first indicator entered is generally a Benchmark Indicator that sets initial amounts and totals. If the Benchmark is left out, the initial amounts and totals for calculating changes will be zero. Subsequent indicators that have an Indicator Type property set to Actual are used to measure yearly changes. Indicators within the same year will be summed together (sum Q1, sum Q2, sum Total) and the totals will be used to calculate incremental changes between periods (keep this in mind when setting Benchmarks). Practical examples include tracking changes in drug prices, farmland values, health care costs, or carbon pollution levels. 
The following image displays what an Output Indicator for one year looks like (notice that only Benchmark and Actual indicators are used):

A Change by Id analysis is appropriate when changes need to be analyzed between different indicators. It’s also appropriate when alternatives need regular comparisons. Only indicators that have an Indicator Type property set to Actual are used in the analysis. The first indicator entered is generally a Benchmark Indicator that sets initial amounts and totals. If the Benchmark is left out, the initial amounts and totals for calculating changes will be zero. Subsequent indicators that have an Indicator Type property set to Actual are used to measure changes. Practical examples include tracking changes in drug prices, farmland values, health care costs, or carbon pollution levels. 
A Change by Id calculations can support other types of incremental change analyses, such as Marginal Cost per Unit Output, by carefully setting the Q1, Q2, Weight and Math type properties. This calculation is appropriate when decision support must be based on cost effectiveness criteria, such as cost per unit of benefit. For example, examine the following basic marginal cost analysis:

This analysis can be carried out using the Change by Id calculation. The following image shows that the Indicator.Q1Amount property must correspond to the Quantity of Cancers Detected column in the image above, while the Indicator.Q2Amount property must equal the Price per Diagnostic Test. The Indicator.Weight property is a general multiplier that is used to derive the final Total Cost. The Total is the Total Cost (Q1Amount * Q2Amount * Weight) of the indicator. This particular calculator is useful for quick, summary, marginal cost analysis. 

The following image displays the corresponding Change by Id Analysis. Note that this analysis uses the Marginal Cost property to measure the incremental change between indicators. Future references are expected to expand the number of tools for carrying out marginal analysis.

4. Progress Analysis
Progress analyses support the analysis of incremental progress in planned versus actual indicators. The planned indicators use a Target Type property of Benchmark, Partial Target, or Full Target. The actual indicators use a Target Type property of Actual. The actual indicators are always compared to the planned indicators. The indicators being compared must have the same Label.  This calculation is appropriate when indicators need to measure the progress being made in the accomplishment of goals and targets. The targets are defined using Target Type properties of Full Target or Partial Target. It’s also appropriate when alternatives need goal-dependent comparisons.
Most M&E calculations will have multiple Partial Target indicators for partial target periods, such as monthly, quarterly, or semiannually. Each of these partial target periods will include one or more Actual indicators that sum the actual accomplishments for the period. The summations are carried out by aggregating actual indicators that have a date that is less than their corresponding partial target date, and greater than or equal to a previous partial target date. 
The following two images display six Input indicators that measure progress over two quarterly periods.


Although not displayed, additional base elements are used to store the actual results for additional quarters. Note the conventions for dates - actual indicators must have a date that is less than their corresponding partial target date, and greater than or equal to a previous partial target date. The following image displays a Progress Analysis of these calculations:

5. Comparative Analyses
DevTreks supports basic M&E comparative analysis. Each indicator being compared in an analysis must have the Alternative Type property set to an appropriate option (A, B, C, …). The exact same type of Analyses explained in previous sections will be carried out, but, before running the analysis, the Indicators will be subdivided further by the Alternative. In effect, this is similar to a Change by Alternative Type calculation, except the analyses are carried out using the Compare Only property of analyzers. Further details about can be found in the Monitoring and Evaluation Analysis 1 reference.

  DevTreks –social budgeting that improves lives and livelihoods

1