The review team's research and analysis probably will culminate with the
publication of a report to decision makers, interest groups and the public.
Therefore, the process of writing and disseminating the report deserves as much
attention as the work that preceded it.
A report allows others to review results and makes it less likely that conclusions will be misunderstood. The report also serves as a permanent record of the review and a guide for the reviewed organization--and it can be used later, during a follow-up review, to determine if recommendations were implemented.
A well-written performance review report should be persuasive, objective and clear. This chapter discusses how to prepare such a report and provides a "recipe" for a standard format that can be followed easily by writers and readers alike. The chapter also will address some of the executive and administrative decisions necessary to complete the report.
This chapter also addresses the media as a factor critical to the success of a
review. One factor in TPR's success is that its recommendations are relatively
easy to understand, and have actually been read by a variety of people,
from reporters to academicians.
Determining the audience
The review team should first determine the report's audience. Some analysts may think that their only audience is the organization being evaluated. This impression can result in a document laden with the specialized jargon and acronyms of the organization under review, based on the assumption that any failure to use their precise terminology may make the report appear naive or superficial. This approach might be appropriate for a confidential report for the organization under review. In most cases, however, TPR strongly suggests that the team assume a broad target audience including the media and the general public.
Report writers inevitably will experience conflict between the goal of simple
and direct communication and the desire to present findings in a detailed and
scholarly manner. But analytical reports ultimately have greater value when
they translate technical issues into plain language. When a report
simplifies or explains complicated information, its readers are more likely to
be persuaded by its contents.
Style and tone
Once the audience is defined, the writers should determine the style and tone of the report. "Audit" reports traditionally have been written in a stiff and formal style, probably because auditors and analysts wish to seem impartial and credible. In recent years, however, many have realized that reports can be simple without losing professional credibility. A more informal style reflects the fact that many government documents now have a broad readership. Report authors should write persuasively for general readers or their reports will not have the greatest possible influence.
Analysts often find it difficult to judge how they should voice criticisms of an agency. Avoid presenting an agency's shortcomings in a fashion that implies the agency always fails to do something or never performs some important activity. The truth may be that the analyst has found some examples of a weakness within the agency. A better approach is to point out that weaknesses expose the agency to certain dangers. This shifts the focus from the agency's wrongdoing to the agency's exposure to risk, something the agency's staff is much more likely to accept positively.
Keep in mind the ultimate purpose of the review: to bring about positive change. To accomplish this goal, analysts should write the report in a way that encourages the agency to adopt a positive, proactive stance rather than a defensive posture. Shortcomings should be discussed to justify a proposed change, but there are many different ways to frame such remarks without resorting to pointed criticism that may be perceived as an attack. While writing the report, try to see the draft through the eyes of the organization under review and structure the approach accordingly.
In determining a style and tone, decide how to report the organization's strengths. TPR believes the report should always acknowledge the extent to which an organization's current activities approach recommended changes. It's not necessary to spend too much time inventorying all the activities the organization performs well. Attention is better focused on strengths and weaknesses that exist in the areas needing change.
In sum, the report must describe the current situation clearly to convince
decision makers of the need for change. Nonetheless, the overall emphasis in
the report should be on the organization's potential and its opportunities for
improvement, rather than on flaws and shortcomings.
Determining a format for the report before actually writing it will help accomplish several objectives. First, if the report is to be prepared by more than one author, a common format will help to ensure consistency among writers. TPR's major reports reflect the work of dozens of analysts. Even a performance review of a single organization is likely to combine a number of individual write-ups. Each one should follow a similar format regardless of its author, or readers will be forced to struggle with confusing shifts in style and tone.
The overall format for a performance review report should contain several major elements. First among these is a background piece that describes the organization or program in question and discusses its mission, its statutory authority and its budget and staffing. This background also will discuss significant events, such as budget cuts, federal mandates or changing legislation, that have affected the organization or program. The background piece precedes and sets the stage for separate writeups on recommendations.
In addition to a general background piece and specific recommendations, a final report also may include appendices featuring the results of any surveys performed in the course of the review; organizational and staffing charts; a glossary of relevant terms, and brief discussions of any issues that failed to generate recommendations but are felt to warrant further study. Every report should feature an acknowledgments list thanking all persons outside the review team that contributed to the project, a list of review team members and a complete bibliography.
Reports should contain a detailed discussion for each specific recommendation. A consistent and specific format for these discussions will help guarantee that analysts include all the pertinent elements needed to explain and justify the recommendation. Moreover, the format should force writers to group important concepts in logical order.
The elements in a recommendation format should include:
Background. TPR write-ups begin by summarizing the background of the issue or program under review as it relates to the specific recommendation. This background gives the reader the context and history needed to understand the findings and recommendations that follow. (Material of a more general nature belongs in the report's opening background piece.)
Analysts often provide either too much background or too little in recommendation writeups. The background should include just enough information so that the reader is not confused by the findings and recommendations. For example, if an analyst recommends consolidating similar activities currently performed by two agencies, there is no need to describe all the agencies' activities, but there should be a description of the similar activities to be consolidated.
Whenever possible, the background should state how the program or activity should function in an ideal situation. The standard or objective could be specific, as in cases where a national association or a federal law has established specific requirements. On the other hand, the analyst may simply set out a general standard against which to compare the activity under review at a later date.
In the earlier example concerning similar functions in two different agencies, a writer might establish a general standard that the state should provide services in an efficient and effective manner, without unnecessarily duplicating functions or expenditures. This is a standard that a reader can understand and support, and thus it can serve as a suitable "anchor" for the recommendations that follow. The goal is to make recommendations that bring the current situation closer to the standard.
The background also should include your relevant findings, a description of conditions existing in contrast to the standard. Remember that writers can unnecessarily complicate matters for the reader by insisting on including every finding in the report. Some findings will not be relevant to the recommendation; others will be so insignificant that they can be omitted without detracting from the recommendations or the integrity of the report. TPR believes that findings should involve strong conclusions. If you are reluctant to state a conclusion boldly, consider whether this is due to a lack of confidence in the finding itself or in the research and analysis supporting it. If the analysis is accurate and free of bias or unjustified opinions, one should not hesitate to be straightforward.
To support your recommendations, help the reader understand specifically how the current situation falls short of the standard or objective. It may be that the organization lacks a particular procedure or that it fails to meet its statutory directives. To describe a problem, state not only what the deficiency is, but also what the effect of the deficiency is. What is the agency's risk or exposure resulting from the current situation? How does the current situation affect the users of the agency's services? What is the effect on the agency itself?
Performance review teams quickly become aware of their opponents' tendency to cry, "if it ain't broke, don't fix it." Critics may argue that unless a problem is clearly documented, there is no need for a recommendation. Analysts can gain credibility with those resistant to change by making every effort to clearly explain problems uncovered and illustrate how their recommendations can improve an agency's performance.
Try to couch problem statements so that no reasonable person can oppose them. For instance, if a problem statement notes that "the agency's permit process forces applicants to wait three months for permits," opponents to the recommendation must show either that the current time frame is acceptable or that the problem statement is inaccurate.
Recommendations. A recommendation is a blueprint for change. Make sure it is specific enough to be implemented properly, without providing overly detailed directions that might limit management's flexibility. For example, an analyst might determine that the organization should set a standard for processing its claims vouchers. It may not be appropriate to tell management exactly what that standard should be, especially if the recommendation would be codified in law or ordinance. Any specific standard could quickly become obsolete if improvements in technology allow future workloads to be greatly increased. An alternative to specific requirements might be to establish a range within which the organization should operate.
If a recommendation would require a change in existing law, indicate which law should be amended and provide detail on any specific provisions new legislation should include. For example, TPR recommended requiring, by statute, that each state agency adopt rules jointly with the Office of the Attorney General describing their process for collecting delinquent obligations. TPR's recommendation added that these rules should establish a reasonable period for agency collection and that the law should mandate a period after which agencies must report the obligation to the Attorney General's Office for further collection efforts. By requiring a time period but not dictating what that time period should be, the recommendation provides flexibility but makes the agency responsible for devising its own methods for carrying out the recommendation in an effective and timely manner.
In presenting recommendations, analysts also must address the implications of the proposed changes. The first important implication should be that the change will bring the current situation closer to a standard established earlier. Explain in general terms who would benefit from the recommendation. Describe how the problem is addressed by the proposed change. Will the organization or process under review become more accurate, fair or efficient? Will it produce better service or better information?
Also, explain precisely what steps must be taken to implement the recommendation. If new equipment or additional personnel would be needed, mention these costs even if exact figures cannot be provided.
To eliminate any appearance of bias, acknowledge any disadvantages or possible adverse effects resulting from the recommendation. In doing so, however, explain how such disadvantages can be corrected.
After outlining the recommendations, the analyst should review previous sections of the report. Does the background include everything necessary for the reader to understand the recommendations? Do findings adequately describe the conditions that justify the changes proposed? Do recommendations clearly explain how the resulting change will more closely reflect the pre-determined standard? Perhaps most importantly, is all evidence used to prepare and justify recommendations carefully and completely documented and cited?
Fiscal impact. The fiscal impact is one of the most important and most scrutinized elements of a recommendation. Fiscal implications are researched by team members and reviewed by team leaders and senior management.
The fiscal impact section should state whether the recommendation saves money,
increases revenues, costs money or avoids certain costs. TPR typically
presents fiscal implications for each recommendation over a five-year period.
(See Appendix 2 for an example of a TPR recommendation.) The fiscal impact
section should include a statement of the exact number of employees affected,
the year in which the change is anticipated, and the salary levels and fringe
benefits of the employees affected. In addition to "hard-dollar" savings such
as eliminated salaries and benefits, TPR reviews sometimes cite "productivity
savings," such as employee time freed up by a streamlined process.
Internal and external review
The team leader should work out a system for conducting an internal review of the draft report. TPR has found it useful for members of a team to conduct "peer reviews" of one another's work. In addition, it is wise to develop a series of internal approval steps beyond the team level. However, while careful review is essential to the production of a top-quality report, it should be done quickly and efficiently. An inefficient approval process can delay the release of the report unnecessarily and may result in the report becoming outdated.
The team may wish to have representatives of the organization under review preview the draft report. This course of action has benefits and disadvantages. The agency under review obviously is better positioned than anyone else to examine the report for accuracy. In addition, agency representatives are most likely to detect statements that appear to be opinions rather than fact, or that lack substantiation, because of their natural sensitivity to such issues. If the team decides to share a draft of the report with the reviewed organization, TPR suggests asking them to point out any inaccuracies and to suggest words or phrases that more clearly fit the intent. Furthermore, explain to the staff that the team wants to be advised of any misunderstandings or oversights in the report.
TPR's school performance review team typically shares the background and findings section of a report with representatives of the school district under review. The review team's recommendations, however, are not shared. This allows school representatives to review the bulk of the report prior to its release, and gives the review team a chance to correct any serious errors identified by representatives of the district, while maintaining an element of secrecy until the report is officially released.
If data are shared with the organization under review, it should be made clear that the review team does not wish to debate the merits of the recommendations. Establishing this up front should prevent the reviewed organization from spending time writing rebuttals that are unlikely to be incorporated into the final draft.
Even so, be receptive to suggestions made by the agency and make whatever changes are necessary to improve the report's content or message. This will communicate a willingness to understand the organization's position. Overall, it has been TPR's experience that comments from the reviewed organization improve the report and provide additional assurance that the final product is reliable.
Occasionally, the team may find it necessary to release a report without a
prior external review. The risks and benefits of such a move must be weighed
on a case-by-case basis. If it is standard policy to permit outsiders to
review the drafts and the team then departs from that policy, be ready to
provide a clear explanation of the reasons.
Timing the release
Obviously, the report's recipients should be given adequate time to digest the substance of your recommendations. Make sure that the report isn't delayed until policy makers or the organization under review can no longer benefit from the recommendations. On the other hand, be aware that if your report is issued too far ahead of any contemplated actions to implement its proposals, you are giving special-interest groups more time to mount opposition to your work.
Working with the media
The news media can help explain your review findings and recommendations to as broad an audience as possible. In TPR's experience, members of the media are actively interested in the results of performance reviews. You may find it beneficial to work with the media by holding a press conference to release the report; drafting and submitting editorials or letters to the editor regarding report findings, preparing report summaries that explain key issues and recommendations in a simple format, and issuing press releases as recommendations are implemented or addressed by governing bodies.
A few cautionary notes: You should not release information to the media until your recommendations are finalized and you are able to fully explain and defend them. As the media reports on your findings and recommendations, you may encounter individuals or groups opposed to your work. You will have to explain and document your work and should be ready to do so upon short notice. Always be prepared to defend your recommendations against detractors to maintain their significance and credibility. Be sure that the news stories you generate are about the substance of your recommendations.
Second, you should decide what topics are "hot." The recommendations that propose the most savings or the most radical organizational changes are not always the ones that generate the biggest news stories. If possible, you should consider consulting with experienced media professionals to ensure that you know what stories are likely to be generated by your report. In its school district reviews, for instance, TPR reviewers work closely with Comptroller media personnel who maintain strong outreach efforts with local officials and understand local issues.
Finally, stay current on your recommendations as they are considered and
implemented. As surely as your review team had compelling reasons to
formulate a particular recommendation, another group will find good reasons to
alter or disregard it. Such events are inevitable, and make it critical for
you to follow your recommendations through whatever implementation processes
occur to ensure you are fully informed about the outcome of your
Review teams should have some idea of the persons or groups that will receive the report before beginning a review. Of course, as the review progresses and people become aware of it, the list of persons interested in the results of the report may well grow. The team must have a clear idea of who is entitled to receive copies, and establish a system for distribution. TPR creates a mailing list at the outset of a review, and adds to it throughout the review. A policy for charging for reports will be necessary if unlimited copies cannot be provided free of charge. TPR typically tries to establish a balance between recovering its costs and ensuring that persons directly affected by the report have ready access to it.
TPR publishes its results in a variety of formats. In the case of lengthy
reports, a brief overview often is made available free of charge. In addition,
major reports to the Legislature always feature a slim first volume that acts
as an executive summary. The Comptroller's office is currently transferring
some TPR reports to the Internet as well.
Backup materials and work papers
The report probably will not detail all of the evidence and background information. But retain all the information used to substantiate findings or make conclusions in an easily retrievable format, whether on hard copies, computer discs or microfiche. This helps to ensure that all of the conclusions can be verified if they come into question.
The team's organization already may have standards for retaining such records. If not, establish work paper standards and ensure that analysts follow them. Organize papers by issue area or any other logical method, using a notebook or expandable folder. Number and cross-reference backup materials to items in the report. In this way the team will guarantee that, should it need to provide more support for its conclusions, it can always access the appropriate materials quickly and easily.
|Texas Comptroller of Public Accounts|| Window on State Government|
Privacy and Security Policy