Planning the review
A governmental entity that decides to conduct a performance review may choose among several different approaches.
TPR uses a functional approach for its major studies of state government. State operations are reviewed by teams working in functional categories such as education, public safety and criminal justice, transportation, health and human services, natural resources, general government, employee benefits and cross-governmental issues, which affect all aspects of government.
However, reviewers may wish to evaluate a specific agency's procedures and operations, perhaps to determine whether the agency's basic mission needs rethinking. This would require an organizational approach. Depending on the topics being studied, such a performance review could concentrate on one or several agencies.
Perhaps management wants to focus on the effectiveness of a specific program within the organization that serves a large client population and affects several other groups, agencies or operations. In this case, a program approach would be applied.
A fourth review approach is by theme. This approach links a series of related activities or functions that may be carried out by a variety of organizations. For example, a city government may believe it can achieve cost-efficiencies by consolidating certain administrative functions currently dispersed among several city offices. This "theme" of centralizing administrative functions could include a review of printing, mail, computer and other functions across many city offices.
These approaches are all valid; others might work equally well. No matter which approach is chosen, TPR methods can help ensure that the review is effective.
As with any complex project, careful planning is essential to the success of a
performance review. The project plan should be a flexible document that
evolves as the review unfolds. Executive management and the project leader
must work together to establish the basic review plan. Each has specific
duties as well.
Executive management probably will not be involved in the actual execution of a performance review. However, managers play a crucial role in planning the review because they must decide its general purpose and objectives. Management also must continually demonstrate a commitment to the review and make it clear to the organization that cooperation with the review team is a necessity.
A volume issued by the Urban Institute, Program Analysis for State and Local Governments, lists some key duties for management.1 Executive managers should:
- actively participate in selecting program and policy issues.
- assign responsibility for analysis to a unit that can conduct the study
- ensure that relevant departments or agencies cooperate.
- provide adequate staff to meet a timely reporting schedule.
- insist that objectives, evaluation criteria, client groups and program alternatives considered in the analysis include those of prime importance.
Comptroller Sharp is briefed on all TPR proposals and approves all
recommendations. Before recommendations reach his desk, however, they have
been thoroughly researched and have passed through a series of "tollgates" in
which senior staff members examine their policy implications, fiscal impacts
The project leader's role
The project leader's efforts can make or break the review. The project leader must manage the project's day-to-day operations and ensure that each phase of the review is completed on time.
Early in the review, the project leader should meet frequently with management
to discuss issue development, staffing requirements and timelines. As new
issues arise and others fail to yield useful results, the leader may seek
management approval to drop or add topics to the review. Effective
communication between the review team and its managers will save time and make
the review process easier. Unless the project leader facilitates such
communication, the project may be crippled from the start.
The review team
This guidebook is designed to show a governmental entity how to conduct a review using its own personnel or those borrowed from other areas. Some agencies may draw on staff expertise from internal audit divisions. Such skills are useful, but auditing backgrounds are not a prerequisite for performance review work. As stated previously, performance reviews are not traditional financial audits, and involve significant amounts of management analysis.
Team members must be disciplined self-starters, with strong analytical, organizational and research skills. They should possess good oral and written communication skills and be able to interact with a variety of personalities. They should have, or be able to develop, sufficient expertise to examine the review topic at hand. Usually, this doesn't require special training; it doesn't always take an engineer to review a function that engineers perform. But a review analyst should be a quick study who can acquire a basic understanding of complex issues and processes.
Executives assembling a review team probably will wish to consider using some analysts from outside the department or program under review. These members can help balance the views of others who may be too close to a problem or issue. Many authorities feel that a performance review's objectivity and credibility can be enhanced by including team members from outside government--from universities, research organizations, consulting firms and the business world. Many reviews in other states have used such expertise.
TPR recommends including outside personnel in the review if an agency has the
resources to do so, particularly if the agency questions whether its staff has
enough expertise to carry out the review alone. But before seeking such help,
consider whether the benefits are likely to outweigh the time and money needed
to educate outside specialists about the program or organization in question,
as well as the risk of draft recommendations falling into the wrong hands prior
to the release of the report.
Defining a purpose and objectives
The first major step of a performance review is to determine its purpose and objectives. According to one source, the "purpose is a general statement of why the [performance review] is being done. The objectives are more specific statements of exactly what the [review] must accomplish in order to achieve this purpose."2 More simply, the purpose is the "why"; the objectives are the "how."
To begin the review, the project leader and executive management must reach a tentative agreement on its purpose. Next, the project team should discuss review activities and objectives, assigning specific responsibilities to team members. The team then should translate the purpose and objectives into a series of five major review phases. These phases form a basic planning framework for the entire project and act as milestones. Each marks a significant event in the review process.
For instance, the purpose of TPR's performance review of the Texas Department of Criminal Justice (TDCJ), Behind the Walls, was to reduce the operating costs of the state's criminal justice system. The objective was to keep the recommendations within the limits of the guidelines issued in the Ruiz court cases. Review team members were assigned to each aspect of TDCJ's operations, such as construction management, information systems, prison industries, parole and health care services.
Phase I--Background research Once the review's general subject is chosen, the team conducts the research needed to become familiar with the operations, procedures and practices of the agency or area under review. Background research also helps narrow the field of interest and identifies the specific areas that promise the greatest rewards. This research phase may involve a series of overview meetings with the personnel of the agency or program under review.
It is extremely important that the review team begin maintaining careful and detailed records of its research from this point on, including lists of sources, contacts and comments. Such records will prove important in preparing recommendations and essential in defending them.
Phase II--Focusing and issue identification Next, analysts should focus on the specific areas and questions identified and any potential significance in terms of savings, increased efficiency and other improvements. This will help refine goals and objectives and determine the scope of the review.
To develop the best and most inclusive list of issues, TPR maintains a "docket" of potential issues. Proposals find their way onto the docket in a variety of ways: issues have been developed from constituent letters, newspaper articles and conversations with officials from other states. The docket is maintained by TPR staff using an off-the-shelf software package.
More detailed work plans can be developed at this time in close consultation with managers. As a useful working tool for this phase, analysts should create briefing sheets for each issue considered for review. Each sheet should contain background information, a statement of the problem, the desired benefit and the expected results. These sheets can help managers make the decisions that set the stage for in-depth research.
Next, clearly define the review scope--decide which areas will and will not be covered. Questions on the scope of the review will come up time and again throughout the project. Early in the project, it is useful to weigh which topics are open for consideration.
To set review limits, first make sure the scope is consistent with any
directives or charges the team has been given. Second, consider available
time, resources and the team's expertise. Given these limitations, some
subjects will be eliminated because they are simply too large or unwieldy to
tackle. Third, weigh the available information on the proposed review topic.
Will the team be able to glean valuable ideas from existing articles and
reports, or must it conduct original
Fourth, reexamine the initial assessment of where the real problems lie within the topic. This will help decide how to direct the first in-depth review efforts. An example cited in Program Analysis for State and Local Government concerns a review of a local emergency ambulance service. This review was criticized for concentrating excessively on ambulances' response time while largely ignoring the broader and more important question of how response times affected care delivery.3
Finally, when setting the review scope, consider the political climate in which the team will operate. Some innovations, however valid, may not be politically "do-able" despite the best analysis; others may require a degree of public education before their release. Review teams should concentrate on areas that offer the best possibility of success. Studying the track record of any previous related efforts should provide valuable information on this question. This effort also may allow the team to repackage a previously unsuccessful idea or proposal.
Phase III--Issue development During this phase, the review team will conduct on-site reviews and in-depth research and analysis on the specific issues selected for study. The goal of this phase is to collect the documentation necessary to complete the analysis and to develop and support a recommendation.
This effort should include complete background information and an analysis of the fiscal implications of the recommendation, as well as a list of its other advantages and disadvantages. The results should be scrutinized by a peer group or management to ensure that all necessary information has been included. By the end of this phase, the review team should have essentially all the data needed to write a final recommendation.
Phase IV--Recommendation writing; final review Now, the team should put its written recommendations into a final format for publication. This report should detail how the recommendation should be implemented. For instance, recommendations that would affect legislation should cite specific statutes that need amending. The team also may offer several alternative recommendations.
Draft recommendations should be forwarded to the team's managers for review and editing; this process should include a careful review for accuracy and consistency of tone.
Phase V--Report production; preparation of legislation request After the editing process is complete, the final report is sent to production. If the recommendations call for new legislation, the team should prepare a request to be used as a guide by bill drafters and policy makers for each applicable recommendation. Requests for legislation should cite the statutes to be amended, suggest language and comment on the intent of the recommended language.
Note: During any phase of the review, be prepared to alter the review's initial boundaries, if necessary. The analysis process often will uncover opportunities and approaches that weren't originally anticipated. Be prepared to modify the project's scope to accommodate these new ideas.
As a practical matter, before expending too much effort in any one review area,
be sure that the appropriate individuals approve the scope of the project.
It's all too easy to become engrossed in developing possible approaches, only
to find that management interest in them is minimal.
Developing the plan
Using the information discussed so far (purpose, objectives, scope, review phases and deadlines), the review team is now ready to develop a detailed project plan. If possible, use an appropriate project management software package to track the progress of the review.
The project plan is vital to keeping a review on course. The plan should:
- describe the purpose and objectives of the review;
- list all personnel assigned to the review;
- outline, in detail, each significant activity in the review, the personnel
responsible for the activity and the projected timeline, and
- state when each deliverable or product of the review can be expected.
- Organization/agency/department and team names
- Project identification (name or number)
- Starting date
- Completion date
- Review phases:
- Background research
- Focusing and issue development
- Recommendation writing; final review
- Report production; preparation of legislation request (if any)
- Tasks or activities
- Personnel assigned to each
- Start and finish dates
- Comments or notes
- Deliverables or expected products
Of course, this basic structure can be tailored in whatever way works best for the analysts and management. Appendix 1 of this manual provides a sample TPR work plan.
To create a plan on these lines, it's best to start by setting the planned deadline or final completion date and then working backwards through the plan to the first day of the review. This will help the review team assign realistic deadlines for each step in the review plan and decide how much time to allow for each step.
Always make sure that the project plan is updated to reflect any changes that may affect the progress of the review. When team members are assigned to other projects, or when research reveals a new twist to the project at hand, the plan should reflect these changes.
Once a team has been assembled, the purpose and phases of the review have been determined and activities within each review phase have been listed, the next step should be to assign specific responsibilities to each team member. Be careful to make the objectives and expectations of the review clear to team members. They should understand their place in, and responsibility to, the overall project as well as their individual tasks and timelines. To perform their assignments effectively, reviewers need a feel for the context of the topic under review.
Our plan outline mentions deliverables, but what are they? Inevitably, in the course of the performance review, the team will produce background reports, documentation and recommendations that may or may not be distributed beyond the organization. The writing, editing and, if applicable, the publication and promotion of these products should receive as much attention as any other significant activity in the review.
|Texas Comptroller of Public Accounts|| Window on State Government|
Privacy and Security Policy