This is a primer to help Rhode Island state agencies develop and improve the performance measures that are required by Rhode Island State Law. The development of performance measures are part of a state strategic planning process which assesses where agencies are, what they have to work with, what their goals are and how they will attain those goals. Agencies are required to include performance measures for each of their budget programs with their annual budget proposals which they submit to the State Budget Office.
Performance measures offer the taxpayer an opportunity to see not only what programs they pay for but also the impact of those programs. They can also provide an evaluative tool for program managers and public-policy makers and can help improve program services and service delivery. The performance measures are also useful in the budget process as they can measure the programmatic impact of budget adjustments.
State government resources support activities that produce goods and services intended to affect societal results whether directly or indirectly. State resources include personnel and funding in support of activities conducted by the state itself, local and federal agencies, quasi-public agencies, private persons and companies or nonprofit agencies with contracts with the state. Performance measures indicate the quantity and quality of those resources (input measures), how well goods and services are delivered (process measures), the quantity and quality of goods and services (output measures) and the societal effect of these goods and services (outcome measures).
There are, of course, many influences affecting societal outcomes besides state supported activities. Outcomes tied directly to state agency support are measured by program or agency level outcome measures. Outcomes may also be tied to the efforts of more than one program whether it is another state-supported program or a program supported by other governmental entities such as a city fire department or a federal law enforcement agency. Such outcomes would be measured by cross-agency outcome measures. Some societal outcomes may be only partially affected by state programs and result, to one degree or another, from factors outside the purview of state programs. Such outcomes are measured by population level outcome measures. The state’s unemployment rate, for example, may be higher than it would be without the state’s training programs and economic development efforts, but it is obviously affected significantly by the national economy as well.
There are some population level outcomes that are so significantly affected by state supported programs that their performance measures are considered both agency level outcome measures and population level outcome measures.
The first thing to be done in this primer is to give you, the reader, a test. Don’t panic. It is will show how well you already recognize the different types of performance measures but more importantly it will serve as a learning device especially as we review the answers with explanations. Reread Don’t peek at the answers till you’ve completed the test!
After the test, the process of developing performance measures will be described and some criteria for good ones will be listed. You will also be provided with examples of input, process, output and outcome measures. After that you will be challenged to come up with some performance measures on your own after reading a case study.

Test Your Performance Measure Quotient
(Pretest)

Reread the second and third paragraphs of the introduction before taking this test. Feel free to refer to it during the test. Place the letter next to the performance measures below which best describes the type of performance measure it is. Remember the measures that the Budget Office wants are outcome measures tied to the performance of state supported programs. For certain types of programs, process measures are accepted. This test is designed to assist you in identifying differences between outcome measures and the other types of measures.

• Input Measure (Agency Level)
• Process Measure (Agency Level)
• Output Measure (Agency Level)
• Outcome Measure (Agency Level)
• Outcome Measure (Cross Agency)
• Outcome Measure (Population Level)

 

1	____	Percentage of invoice vouchers processed within 30 business days
2	____	Percentage of biennial audits required by law that are completed.
3	____	Percentage of Unemployment Insurance Claims paid promptly.
4	____	Attorney caseload at the Office of the Public Defender.
5	____	Percentage of summons disposed.
6	____	Number of food borne illnesses per 100,000 population
7	____	Percentage of Sheriffs completing formal training in court security
8	____	Percentage of eighth graders testing proficient in math skills.
9	____	Number of prison escapes annually.
10	____	Medication error incidents per 100 hospital beds.

 

Answers with Explanations

1. B. The percentage of invoice vouchers processed within 30 business days is a process measure. It is not an input measure because it is not a measure of the elements or ingredients of the process such as the percentage of staff trained on the state accounting system. It is a measure of the quality (namely, the timeliness) of the process itself. It is not an output measure because it does not measure the quantity or quality of a direct good or service. It may measure how well payment was made for a good or service but does not indicate the quality of a good or service itself. It is not an outcome measure because it does not measure a societal result.
2. C. The percentage of biennial audits required by law that are completed is an output measure. It is not an input measure because it does not measure the elements or ingredients that make the audits possible such as auditor training. It is not a process measure because it does not measure how well the audits were conducted as much as how many were conducted. It is an output measure because it is a measure of a good or a service namely audit reports. It is not an outcome measure because it does not measure a societal result. Conceivably, a significant audit finding could contribute to a societal result such as, for example, taxpayer savings, but the audit report would be a means to that end not the end itself.
3. B. The percentage of unemployment insurance claims paid promptly is a process measure. It is not an input measure because it is not a measure of elements of the process such as a well-trained clerical staff and well-designed computer systems. It is not an output measure because it does not measure how many claim payments were made but the how promptly the payments were made. It is not an outcome measure because it does not measure a societal result.
4. A. Attorney caseload at the Office of the Public Defender is an input measure. It is a measure of the quality of an element contributing to the process of legal defense of indigents. It assumes that smaller caseloads allow for more defense preparation time. It is not a process measure because it does not measure a quality of a process at the Office of the Public Defender such as timeliness. It is not an output measure because it does not measure the quantity or quality of a good or service provided by the agency. It is not an outcome measure because it does not measure a societal result such as the number of innocent persons proven innocent with the use of DNA evidence.
5. C. The percentage of summons disposed is an output measure. It is not an input measure because it does not measure the elements that contribute to the summons process. It does not measure how quickly or how well the summons were prepared so it is not a process measure. It is not an outcome measure because it does not measure a societal result.
6. D and F. The number of food borne illnesses per 100,000 population due to salmonella is a population level outcome measure but it is also so closely tied to the inspections of the Health Department that it can also be considered an agency level outcome measure. It is not an input measure because it measures a result rather than the program elements that contribute to that result. It is not a process measure because it does not measure the quality of the process leading to the results but the results themselves. It is not an output measure since it does not measure goods and services produced by the Health Department such as food inspections.
7. A. The percentage of Sheriffs completing formal training in court security is an input measure. It is not a process measure because it does not measure the quality of the Sheriffs’ court security procedures. It is not an output measure because it does not measure what the Sheriffs do as much as that they have been trained to do it. It is not an outcome measure because it does not measure the court security as an outcome such as the number of prisoner escapes or prisoner suicides while in the custody of the Sheriffs.
8. E and F. The percentage of eighth graders testing proficient in math skills is a cross agency outcome measure since it involves the effects of state aid to education and the educational programs at local schools. It is also a population level outcome measure since math skills proficiency may result from a variety of social and cultural values from such sources outside the school system such as the family. It is not an input measure because it does not measure the elements contributing to the educational result such as teacher training and parental support. It is not a process measure because it does not measure the quality of the educational techniques used to affect the educational outcome. It is not an output measure since it does not measure goods and services produced by the school and financed in part, by state aid, such as total annual hours spent on math skills.
9. D. The number of prison escapes annually is an agency level outcome measure because it is tied to agency program efforts to deter escapes and it does measure the results of those efforts. It is not an input measure because it is not a measure of the elements of the prison system contributing to deterring escapes such as prison guards and electronic prison security systems. It is not a process measure because it is not a measure of the quality of prison security procedures designed to reduce escapes. It is not an output measure because it does not measure the good and services produced by the Department of Corrections such as the number of inmates served.
10. D. Medication error incidents per 100 hospital beds is an agency level outcome measure because it measures a result tied to state hospital efforts to reduce errors. It is not an input measure since it does not measure the resources contributing to error reduction such as the number of staff trained in how to avoid medication errors. It is a process measure of sorts because it measures the accuracy of medication distribution but it is a measure of a direct impact on patients and therefore, qualifies as an outcome measure. It is not an output measure because it does not measure goods and services such as the number of medications distributed.

Scoring

 

Developing Performance Measures

Performance measure may be developed using the following steps:

1. Review present goals and objectives for each agency program.
2. Develop goals and measurable objectives for each agency program which specifies
3. outcomes desired.
4. Select performance measures consistent with program objectives.
5. Determine the source of data for each indicator.
6. Define a standard against which to compare actual performance.
7. Report results to program managers, policy makers and in budget document.
8. Undertake periodic performance measure review. Update if necessary.
9. Take results into account in budget decisions and policy making.

Criteria of Good Performance Indicators
Good performance measures are:

·  Clear. The measures should be designed and described so that the average member of the public should understand it. They should be as simple as possible and avoid or explain terms that only experienced state personnel would understand.
·  Cost Effective. The data associated with the performance measure should be worth collecting. The purpose of performance measures is not to add to the bureaucracy in order to collect data
·  Relevant. The performance measures should be logically related to the program objectives stated in the Budget document.
·  Significant. Agencies should attempt to measure results that are central to the purpose of its programs.
·  Consistent An agency’s measures should be consistent over time so that the data presented is easy to compare from year to year. If a definition is changed from one fiscal year to the next it should be clearly noted if the performance measure is to be continued.
· Practical Performance measures should be managerially useful to the agency. They should not only be informative to the taxpayers but also be helpful gauges for program managers and other decision-makers. They should be accepted by the agency.
· Verifiable Data for performance indicators should accurate, on file and auditable.
· Honest Performance measure data should be reported from year to year even if data doesn’t look as good as it once did.
· Linked to Funding Performance measures should be linked to funding levels for program activities. Outcomes may differ with funding levels.
·  Result-based. There are different types of performance measures but much emphasis should be placed on those that measure outcome. The focus is on ends more than means even though means are important too.

Examples of Performance Measures

Input Measures Examples
· Number of full time equivalent positions
· Number of courtrooms
· Student-teacher ratio at the University of Rhode Island
· Number of food inspectors at the Department of Health
· The appropriation for the Boards for Design Professionals
·  Number of training programs funded by the Department of Labor and Training

Process Measure Examples
·  Percentage of tax refunds mailed within 30 days
·  Percentage of wage information transferred to other states within five days
· Business days needed to stop payment on a state check
· Investigation completed within 180 days of filing.

Output Measure Examples
· Number of consumers receiving Interpreter services for the deaf annually.
· Number of persons receiving information from the RI Developmental Disabilities Council
· Number of discrimination case investigations.
· Number of Supreme Court cases disposed annually.

Outcome Measure Examples
·   Employment rate for thirteen weeks following training program
· Annual return on short term investments
· Teenage birth rate
· Percentage of consumers indicating consumer satisfaction with psychiatric hospital services.
· Percentage of training school residents taking the GED test who pass it.
· Percentage of welfare recipients receiving job training that are employed three months after receiving job training
· Elder abuse recidivism rate

Post-test or can you tell an output from an outcome?

The following is a case study regarding the meningitis vaccination program organized by the Rhode Island Health Department in 1998. The information was taken from articles appearing in the Providence Journal. Read the case study and develop a list of measurable objectives for the vaccine program on the next page. Then make a list of input, process, output and outcome performance measures related to the program. It is permissible to make plausible assumptions about information not provided in the case study. Just make sure you state them.

Case Study
Back in 1996, the City of Woonsocket had four cases of meningitis in one week including one death of a 16-year-old. The state vaccinated about 17,000 schoolchildren. The outbreak died. In 1996 there were 23 cases statewide including nine that resulted in death. In 1997, there were 24 cases statewide including nine that resulted in death. At around the time the Department started mobilizing the vaccination program in February of 1999, there were about eleven cases including three resulting in death. These cases were not considered an outbreak and said to be unconnected. However, highly publicized cases led to increased parental demand for vaccines.
The Health Department proceeded to mobilize a statewide program to inoculate 225,000 young persons 2 to 22 year olds with the meningitis vaccine. Additional telephones were set up to answer questions from the public. A Health Department working group was organized to deal with the problem including the Director of Health, two public relations consultants, a microbiologist, three pediatricians, three public heath experts and several employees shifted from other assignments at the Health Department. Some 64 clinics in 25 communities were set up. Some Health Department employees worked seven days per week on the program. Volunteers were sought such as National Guard emergency medical personnel, doctors and nurses to work in the clinics. As of March 4, 1998, 225,000 doses of the vaccine were on hand for distribution. Public health officials including those speaking Spanish and Portuguese were being lined up for appearances on radio and television programs to inform the public about meningitis in Rhode Island.
Insurance company reimbursement was sought for the vaccine. A possible musical concert was suggested (not implemented as of this writing) to entice the post-high school age group to get inoculated.
A few points should be made about the vaccine. About half the meningitis cases occur in children under two but the vaccine doesn’t work with them. The vaccine doesn’t work in about 15 to 20 percent of older people and it loses its effectiveness over three years. Children are more likely to be killed in a car crash on their way to be inoculated than they are to die from meningitis. Inoculating 225,000 young people statewide was estimated to cost $7,000,000. An estimated 8 to 10 meningitis cases and one death would be prevented.

Program Objectives:
Input Performance Measures:
Process Performance Measures:
Output Performance Measures:
Outcome Performance Measures: