Best Practice
NOTES

Outcome Measures for Title IIIB Legal Assistance Programs: An Introduction

By: Matthew G. Batista, J.D.*

Although the concept of outcome measures, which seek to gain information on the "impact" or "benefits" in the lives of individuals receiving a particular service, has been around for years, it has become more popular lately in the provision of legal assistance to low income older persons. While much of this attention is due to the Government Performance and Results Act (GPRA), discussed below, developments such as the 1995 cuts in Legal Services Corporation funding led many providers, who were forced to "do more with less," to look for better ways to demonstrate the enormous impact legal assistance programs have in the lives of low income individuals.

I. Introduction

This article presents an overview of the concept of outcomes measures for legal assistance programs, specifically as it applies to programs serving older persons. As an overview, it describes the impetus for the increased significance of outcome measures, explains the measures themselves, how they can be used, and how and where they "fit" with traditional reporting systems.

Note: It is important to understand that outcome measures reporting is not a replacement for more traditional reporting which provides information on numbers of and demographic data on clients served, types of cases handled, types of legal service provided, and related information. Outcome measures are a complement to this traditional reporting. Further, the development of outcome measures should simultaneously be a time for re-evaluating the existing traditional reporting system to determine what information is truly needed to achieve the overall goals and purposes of reporting. This combined examination of the reporting system for legal services should seek to reduce or, at a minimum, not increase the reporting burden. This also suggests, as it should, that outcome measures are not an end in themselves, nor a stand-alone reporting system; they are to be a part of the overall reporting system, and as such they should not duplicate information already being collected. Together, traditional and outcome measures reporting will provide a more complete picture of what legal providers are achieving for their clients.

II. Outcome Measures: The Impetus

A. The Government Performance and Results Act

As mentioned above, a key factor in the renewed importance of outcome measures in the provision of legal assistance to older persons is the Government Performance and Results Act (GPRA). In 1993 Congress enacted the GPRA as a means to "improve confidence in the Federal Government, hold agencies accountable for achieving results, and improve effectiveness and accountability." In pursuit of this accountability, GPRA requires Federal agencies (e.g., the Administration on Aging, hereafter "AoA") to submit to the Office of Management and Budget (OMB) and Congress a strategic plan and annual performance plans and reports to detail the goals and outcomes of agency activities. Each of these is described briefly below.

Strategic plan. Federal agencies submitted their strategic plans to the Office of Management and Budget and Congress in September 1997. Each agency was required to submit a plan that included (1) a mission statement of the agency, (2) the "general goals and objectives, including outcome-related goals and objectives" of agency activities, (3) a description of how the goals will be achieved, including a description of the resources required, (4) a description of how the goals of the performance plans (described below) relate to the general goals of the strategic plan, (5) a list of the "key factors external to the agency and beyond its control" that could affect the achievement of the stated goals, and (6) a description of the evaluations used in establishing general goals, and a schedule for future evaluations. These strategic plans must be updated at least every three years.

Performance plan. Each Federal agency must submit annually a "performance plan" that is to be consistent with the strategic plan. Performance plans must state each budgeted activity, set performance goals for those activities in "an objective quantifiable, and measurable form", and describe performance indicators, resource needs for those activities, a basis for comparing actual program results with the established goals, and the means to be used to "verify and validate measured values." AoAÕs FY 1999 Performance plan can be found on their world wide web site at: http://www.aoa.dhhs.gov/network/Gpra99.html Agencies are given some flexibility in these requirements in that they may aggregate or disaggregate program activities in their plan, as long as major activities of the agency are not omitted or minimized.

Performance reports. Annually beginning in March 2000, agencies must report to the President and Congress on the previous yearÕs performance. These annual reports must compare the yearÕs performance outcomes with the stated performance goals, describe successes and explain shortcomings, the latter of which must include why the goal was not met and what plans and schedules have been established to shore up the shortcomings.

B. The GPRA and Legal Assistance Programs For Older Persons

AoAÕs FY 1999 Performance plan does not directly address legal assistance programs. Further, the Administration on Aging, National Association of Area Agencies on Aging, and the National Association of State Units on Aging are currently funding a development and pilot program with eight State Units on Aging and an Area Agency in Aging in each state, the focus of which is not likely to include legal assistance. Nevertheless, they will eventually have to be developed and will likely be included in the aggregation of "Community-Based Access Services."

The GPRA requires performance outcome measures to be in "an objective, quantifiable, and measurable form." Although it is discussed in detail below, it is worth noting here that under more "traditional" reporting systems, "quantifiable" might translate to amount of funds spent on each service, number of individuals served, etc. While these measures are valuable, and may play some role in outcome measures under GPRA, they do not capture the entire picture, particularly in the area of legal services, where the variety and complexity of cases makes standard outcome measures difficult. The goal with performance outcomes is to get the most meaningful information about the beneficial effect the service has had on the recipient.

GPRA recognizes that it is difficult to quantify outcomes for some performance goals. It therefore allows for an agency to consult with the Director of the OMB to obtain authorization for an alternative form of measurement. The statute requires that the alternative form include a descriptive statement of either a minimally effective program and a successful program, or an alternative form "with sufficient precision and in such terms that would allow for an accurate, independent determination of whether the program activityÕs performance meets the criteria of the description." However, even with this possibility for a more qualitative measurement written into the statute, it is difficult to speculate the degree to which it will be allowed by the OMB. Therefore, it is critical to put time and effort into developing some type of quantifiable measurement system for legal assistance outcomes.

As discussed throughout this article, developing good measurement systems not only benefits AoA in its fulfillment of GPRA requirements -- it also benefits Legal Services Developers, Area Agencies on Aging (AAAs), legal providers, and ultimately older persons themselves. Performance outcomes have the potential to get meaningful information about the impact and effectiveness of legal services, whether priority is being given to the appropriate designated populations and case types, etc. They also have the potential to stress the importance of legal services to older persons, which is of particular import considering the debate regarding legal assistance remaining a priority under the Older Americans Act. Outcome measures can show in black and white not only the number of people who benefit from Title IIIB-funded legal assistance, but more importantly, the impact that assistance has on their lives.

C. Fundraising and Good Management Practice

Although a mandate from the federal government carries significant weight, outcome measures are also being used more and more by legal assistance providers because they make good management sense. In an era that has seen government funding reduced, providers have been forced to broaden their funding searches. In doing so, many providers have found that outcome measures are effective tools in providing evidence that resources are actually producing results for the clients. Funders and potential funders can easily see what a provider can do with the resources they're given. In fact, some funders, such as the United Way, have acknowledged the usefulness of outcome measures and are beginning to require legal providers to utilize outcome measures in their reporting systems.

Aside from the fundraising benefits, outcome measures affect a provider's staff. Studies have shown that achievement and recognition are the two most important employee motivators. Gathering information via outcome measures attends to both -- staff achievements, not simply numbers of cases closed, etc., are documented, compiled and used to show to the benefits provided to clients. In fact directors of legal assistance providers have commented that their employees were "pleased" and "relieved" that the real impact of their work was coming to light when outcome measures were incorporated into reporting systems.

III. Outcome Measures

A. The Concept

So just what are "outcome measures?" Before answering that question, some definitions would be useful. In simple terms, "outcomes" are the benefits that participants (i.e. clients) receive from the legal assistance during and after the provision of service. They are more than just the "results" or "products," i.e. number of cases closed, etc.; outcomes represent improvements in a client's life -- financial and personal independence/stability, new knowledge or skills; etc.

To determine what the outcomes are, or rather if a provider's desired outcomes are being achieved, there must be a system to measure the outcomes. First, a provider must choose the indicators they would like to use to measure the success of the program. Indicators are items of information used to track the program's progress toward achieving the outcomes. For example, a program could have a desired outcome of financial stability for clients, and could define "stability" as securing new income source for an individual; preventing a loss of income; reducing Medicaid spenddown; etc. The number of individuals who accomplish such things is an indicator of how successful the program is in achieving the desired outcome. To complete the picture, a program must set some benchmark for performance in the outcome area. This source of the benchmark data can be previous year's work or from the work other programs do/have done in the same types of matters, however attention should be paid to the reliability and comparability of the specific outside data used.

To distinguish what outcomes are and what they are not, the following "hints" provided by the United Way explain some general guidelines:

o Recruiting and training staff and volunteers, purchasing or upgrading equipment, and various support and activities.

These are internal program operations intended to improve the quality of program resources. The number of staff recruited, amount of equipment purchased, etc. indicate the volume of these internal operations. However, the operations do not represent benefits of changes in participants and therefore, are not outcomes.

o Number of participants served

This information relates to the volumes of work accomplished. In most cases, volume of service is an output. It tells nothing about whether participants benefited from the service and therefore is usually not an outcome.

o Participant satisfaction

Most often, whether a participant is satisfied or not with various aspects of a program (e.g., courteousness of staff, timeliness of follow-up) does not indicate whether the client's condition improved as a result of the service. Satisfaction data are by definition subjective. Thus, participant satisfaction generally is not an outcome.

The above examples all emphasize that outcomes must be looked at from the client's perspective. Some traditional reporting systems, look only from the provider's perspective, i.e. what service did we, the provider, provide -- Counsel & advice? Brief service? Negotiated settlement? Represented in court? Outcome measures focus instead on how the client has benefited from that service -- how did the client benefit or improve her or his situation as a result of our assistance?

B. The Process

Before addressing specific outcomes and indicators for legal assistance programs, the general process of developing a system should be noted. Indeed, anyone who has been involved in the huge task of designing a reporting system knows how important a "game plan" is. Without careful thought and planning of every part of the system, the chances of successful implementation decrease significantly. The following steps are key to developing and implementing outcome measures into a reporting system.

1. Get Buy-In

As with any reporting issue, getting the buy-in of all interested parties -- the providers themselves if the system is being implemented on a state level; the staff, etc. of providers if implemented on a provider level --is crucial to a system's success. Generating ownership in the system and its goals is necessary to ensure that those individuals "on the front lines," who presumably already have an interest in the success of the program itself, will also have an interest in effectively and accurately demonstrating the impact their office has in the lives of the clients. Ownership is generated by involving the providers and/or staff in setting program goals and choosing outcomes to measure, and listening to their input -- provider and staff's day to day experience and insight can be invaluable, especially in choosing indicators and designing the actual forms. An important part of getting buy-in is keeping interested parties informed from the beginning. Providers and/or staff and others are much less likely to accept additions or changes to the reporting system if requirements are thrust upon them.

2. Choose Outcomes

Choosing which outcomes to measure depends on the goals of the program. For example, a program with the general goal of improving the quality of life for the low income residents in a particular community through legal representation might choose outcomes in the areas of financial stability, housing security, and personal independence. Fundamental questions one must ask when choosing outcomes are: "What are we trying to accomplish?" and "How will we know if we are succeeding?" While these questions may have intuitive answers, verbalizing these goals in anything other than general terms can be difficult. Nevertheless, they must be addressed to effectively collect the information desired.

3. Specify Indicators

After choosing which outcomes will be tracked, the indicators to measure them must be determined. The indicators are the results, in the common understanding of the term, of the legal assistance. In the example described in section 2. above, indicators used to measure housing stability might be "prevented repossession of home" or "prevented eviction." Providers and/or staff should have significant input into the selection and wording or these indicators.

4. Coordinate Logistics

Coordinating logistics is just as it sounds -- preparing forms, setting up the collection system, and ensuring that staff (attorneys and support staff) have been completely informed of the system. As with all parts of the process, maintaining open lines of communication between the state office and providers, if that is the level on which the program is being established, is important.

5. Test the System

When a system is implemented, it should be assumed that some changes will need to be made. If the four previous steps are planned and carried out carefully, the necessary changes should be minor. Nevertheless, some amount of modification will undoubtedly be necessary. Therefore, the initial implementation should be thought of as a test. Once again, the individual or individuals in charge of the implementation should diligently seek the input of those using the system regularly -- the providers.

6. Refine the System

This part of the process is self-explanatory. The goal is to rectify the problems found in the initial implementation. In most cases, refining the system should require only a few minor changes. At this point, major tasks such as the drafting of the outcome statement should not be an area that needs to be addressed.

An important factor in developing an outcome measures system that a program must consider is the extent to which outcome measures are being implemented system-wide. In other words, when designing an outcome measures system, a program must take into account whether their measures will (or should) fit into a broad measurement system, such as the one required by the GPRA. While gaining information on the impact of all types of services is an important and admirable goal, the practicality of accomplishing such a task is tricky at best. The difficulty in creating system-wide measures is that they often require the indicators to be standardized across services to ease summarizing and working with the data. In many cases this standardization, while completely well-intentioned, results in measures such as "units of service," which fail to get at the real impact in the lives of the clients.

Although each situation is different, programs must be cognizant of the possible need to translate their measures to some sort of broad measurement system, whether it be GPRA or on the state level. However, this does not mean that the initial efforts to design an outcome measures system should be dictated by a system-wide approach. Rather, a program should focus on their own goals and their own clients. As discussed above, an outcome measurement system has many uses; for that reason, a program should start out to develop a system that meets its needs and accurately demonstrates the impact the legal assistance has in the lives of the older individuals. The translation, if necessary, can come later.

C. The Examples

As mentioned above, general legal assistance programs, especially those receiving United Way funds, are increasingly using outcome measures in their reporting systems. However, Title III-B legal assistance programs are just beginning to see the advantages of outcome measures.

One state office on aging which has devoted substantial effort in developing outcome measures for Title IIIB legal assistance programs is the Tennessee Commission on Aging (TCA). TCA 's work with outcome measures in the area of public benefits provides a good example of the issues that arise in developing outcome statements. Over a period of several months and meetings, the following outcome statement was developed:

Client(s) (persons age 60 years of age or older) residing in our serving area who come in contact with the Program (Title III Legal Assistance Program) will maintain, or change in a positive way (increase, obtain, or restore) their receipt of public benefits -- Social Security Disability and Retirement, Supplemental Security Income, Food Stamps, Veterans Benefits, Unemployment Compensation, AFDC, and Tax Relief -- as indicated by the:

*(###) Title III clients obtained information about public benefits (Social Security Disability and Retirement, Supplemental Security Income, Food Stamps, Veterans Benefits, Unemployment Compensation, AFDC, and Tax Relief).

*(###) clients received, maintained or restored their public benefits (Social Security Disability and Retirement, Supplemental Security Income, Food Stamps, Veterans Benefits, Unemployment Compensation, AFDC, and Tax Relief).

* Each program will establish this number for their program.

In this example, the first paragraph is the outcome objective: to assist in maintaining or increasing in some way eligible individuals' income from public benefits. The outcome or benefit in this example is the maintenance or increase of income through public benefits. This follows a programmatic goal of improving the economic stability of individuals receiving legal assistance.

In developing the outcome statement, the individuals responsible for drafting the language came upon several issues that demonstrate the importance of both clarity and communication. First, some drafters questioned the use of the word "positive" in the outcome statement. In particular, the word "positive" was thought to connote a "winning vs. losing" view of cases, and some questioned whether that was the most meaningful way to think about outcomes in legal assistance programs. Although further discussion resulted in a consensus feeling that "positive" was meant to describe the clientsÕ lives not the outcome of the cases, it nevertheless demonstrates the importance of all interested parties "being on the same page."

A second issue in the drafting of the statement had perhaps an even greater potential to effect the accuracy of the measure. In discussion it became clear that different providers were defining "clients" differently. Specifically, one program interpreted the term broadly to include persons who had heard community education presentations in addition to individuals who received advice and/or information specific to their situation. Another program interpreted "client" more narrowly to include only individuals who received specific information and would be considered to be "clients" in the traditional sense of the word. Obviously, this difference in interpretation would have a huge impact on the data collected on the number of clients served. After discussion, a consensus was reached that the term would have to be interpreted narrowly. Any other interpretation would provide an inaccurate measure; the statement calls for a count of clients who obtain information, and there is no way to accurately gauge individual members of an audience as to how much, if any, information they received from a community education presentation. Once again, careful drafting and communication between interested parties are shown to be crucial to obtaining accurate, useful information.

Ideally, a provider will track and measure outcomes for all of its overall programmatic goals. The example from Tennessee addresses one -- public benefits. Other possible areas for which outcome measures can be developed include "quality of life," "monetary," "broad impact," and "public education" benefits such as the following:

Quality of Life Benefits

o Housing A possible outcome objective could be:

To maintain or improve the stability and quality of housing for low income older persons.

Indicators for such an objective could be those cases in which clients come to a provider with a housing problem and are able to: keep their current housing, preserve utility service, or improve the condition of the housing (especially in rental situations), or gain an opportunity to purchase a Habitat for Humanity home.

A program could set a benchmark of a certain percentage of the clients with housing cases that come in to a provider have the opportunity to receive one of the above benefits.

o Domestic Violence/Physical Elder Abuse

A possible outcome objective could be:

To reduce instances of domestic violence and increase physical security in [a given geographic area] though legal assistance.

An indicator for such an objective could be the number of protective orders received by clients. Depending on staffing and funding of providers, a survey could be mailed to client at a later date (e.g. 30 days) to gauge the long-term effect of the order.

A program could set a benchmark of a specific number of orders from the previous year and require that number or more.

o Personal Independence

A possible outcome objective could be:

To preserve personal independence of older individuals through legal representation.

Indicators could be the number of individuals who request assistance in defending against a petition for guardianship or conservatorship and are successful.

oFinancial Independence

A possible outcome objective could be:

To maintain or increase the income of older individuals in [a given geographic area].

Indicators could be the number of individuals who seek assistance from a provider and are successful in maintaining or increasing Social Security, SSI, veterans' benefits, or avoiding/declaring bankruptcy.

A benchmark could be the number of successful cases in a given year compared to a prior year.

Monetary Benefits

A particularly effective measurement tool, one that is relatively easy to implement, is the monetary benefit. This measure can either be incorporated into a stated outcome measure or calculated for all benefits received over a set period of time, or both. For example, a measure of the financial independence outcome above could be the dollar amounts awarded to a client. Monetary benefits could also be calculated and aggregated for all clients assisted in a reporting period. In both cases, the monetary benefit measure is an effective tool -- a dollar figure is striking and immediately understandable.

A second monetary measurement tool is the amount that would have been billed by the provider. To measure this, a provider could assign to each staff attorney and paralegal an hourly billing rate in accordance with that person's level of experience and expertise and in accordance with the going rate for that level of experience in the community. The levels chosen might actually be on the low side (e.g., $50/hr for a recent law school graduate, $20/hr for paralegals) of the rates private law firms charge. At the end of each quarter all cases closed during that time can be examined and the number of hours each attorney and paralegal worked on those cases can be multiplied by his or her hourly billing rate. The figure arrived at gives an estimate of what the cost of those same services would have been if they had been provided by a private law firm rather than a provider of legal assistance to low income individuals.

The "what it would have cost" measurement tool can be applied in public benefits situations as well. For example, in most unemployment insurance benefits cases that are eligible for legal assistance, it can be assumed that the individuals who are seeking benefits, if unsuccessful, will be forced to seek public assistance. If those individuals are assisted by a provider and are successful in hearings, the saving to taxpayers is the cost of public assistance for the time that the client receives the unemployment insurance benefits. These numbers can be calculated quarterly and presented as total savings to taxpayers. This methodology can be applied to all types of income-producing cases where a provider reasonably believes that the individual and/or her family would be eligible for public assistance but for the provider's efforts.

Public Education Benefits

Most legal assistance programs provide some level of public education, whether it be lectures, workshops or, in some cases, pro se clinics. In some programs, a significant amount of resources are devoted to such educational efforts. Therefore, it is understandable that programs would want to include information on public education in their reporting.

However, the real outcome of the educational efforts -- the impact it had on the participants -- can be difficult to measure, both because it is difficult to determine how much information was understood and retained, and the benefits may not occur until much later. Nevertheless, the information is important to gain a full understanding of a program's work. Programs may want to account for public education efforts through "traditional" reporting methods -- compiling information such as numbers of participants, and types, subject matter, and location of presentations. This information could be supplemented by a follow-up survey, either immediately following the educational activity or at a later designated time, which could be drafted to elicit information on the effect of the presentation will have on the participants' lives. However, because such surveys can be costly and time-consuming, programs, especially those that do not devote significant resources to public education efforts, would have to consider its cost effectiveness.

D. The Benefit

Rather than discussing the benefits of outcome measures as a separate system, it is perhaps more valuable to look at the benefits outcome measures provide from inside the framework of reporting generally.

A carefully developed, meaningful reporting system for legal assistance serves a number of useful purposes. These purposes include: demonstrating legal compliance with reporting and targeting requirements; accountability; performance evaluation; and demonstrating the value and impact of legal services. State and area agencies on aging and legal services providers should keep these purposes in mind as they develop reporting systems in compliance with the Older Americans Act requirements set forth below. As with any reporting system, it is vital that the content and format of information to be reported be linked back to the purposes of the reporting system. This ensures that time and resources are not wasted collecting information that is neither needed nor useful. Each of the five major purposes of reporting is described below.

1. Demonstrating Legal Compliance with OAA Reporting Requirements

Reporting fulfills its first purpose -- demonstrating legal compliance with reporting requirements -- by supplying information that is required by law and that is necessary to determine whether the program meets requirements imposed by federal and state authorities. The OAA contains a number of provisions that describe the data that must be collected by the Administration on Aging (AoA) on Title III support services, including legal assistance. As discussed above in Section II., currently AoA reporting requirements do not include outcome measures. However, they most certainly will in the near future as the Department of Health and Human Services fully implements its plan in compliance with the Government Performance and Results Act.

2. Demonstrating Targeting of Services

The second, and one of the most important purposes of reporting, is to indicate the extent to which legal services are being targeted. A good reporting system can help providers, area agencies on aging (AAAs), state units or offices on aging (SUAs), and the AoA determine if limited legal assistance dollars are being used to provide services to those older individuals in greatest social or economic need. Because providers are still feeling the effect of the 90's cuts in LSC dollars, which reduced the availability of legal services for low-income elderly, the need for targeting is extremely important.

The Older Americans Act explicitly requires that legal assistance be targeted to those older persons in greatest social or economic need. Individuals in "greatest economic need" are those with an income level at or below the poverty line. Individuals in "greatest social need" are those with need caused by noneconomic factors such as physical and mental disabilities; language barriers; and cultural, social, or geographic isolation. Of those individuals in greatest need, the Act directs that special attention be paid to low-income minorities. More specifically, the Act requires that AAAs include in each agreement made with a Title III provider of any service, a requirement that the provider will, to the maximum extent feasible, provide needed services to low-income minority individuals.

To help ensure that legal services are targeted to those in greatest social and economic need, while the prohibition on means testing is observed, the OAA requires that priority be given to legal assistance related to income, health care, long-term care, nutrition, housing, utilities, protective services, defense of guardianship, abuse, neglect, and age discrimination. The Act also places specific restrictions on the role of legal assistance in guardianship cases, to ensure that services go to those in greatest social need -- those who have lost or are at risk of losing their autonomy through guardianship. The Act allows representation of individuals who are wards, or are allegedly incapacitated, but limits representation of older individuals who seek to become guardians to cases where other adequate representation is unavailable.

A reporting system should be able to capture information about efforts to target individuals in greatest social and economic need and the success of those efforts. Because means testing is prohibited, careful design of the system is needed to capture this type of information. Statistical information that can be useful in determining the extent and success of targeting efforts includes:

o number of clients served, by age -- distinguishing between the old (e.g., 65-79) and the "old-old" (e.g., 80+);

o number of female clients versus number of male clients;

o number of clients living in rural areas;

o ethnic/racial status of clients;

o number of clients who are frail and/or socially needy;

o listing of community education topics/audiences and outreach efforts.

Implementing an outcome measures system enhances targeting efforts by gathering data specific to the priority areas set by the AoA. Also, in addition to the above demographic information collected, outcome measures show how those individuals in the greatest social and economic need benefit from the assistance. This adds important meaningful information to the demographic information.

3. Accountability

Another purpose of reporting is to demonstrate the extent of a Title IIIB legal provider's compliance with contractual obligations to its funding source. Because specifications in provider-funder contracts vary, information required to show compliance also varies. One basic measure of compliance -- also required by federal law -- is the "units of service" provided. This statistic is only the starting point; showing accountability to a funding source means much more than simply measuring units of service completed.

Another statistic in which most AAAs are interested is the number of clients served during any given reporting period. Both the AAA and the provider may find this statistic useful because (1) it serves as an indication of the legal program's progress in meeting its contractual responsibilities, and (2) it may provide a measure of the program's cost-efficiency.

Accountability also includes a determination of whether priority legal problems are being addressed, as well as the identification of target populations represented among the clients. For example, is the provider offering services in the priority areas most crucial to the target populations? Is the legal services provider serving clients with priority needs? Is the provider capable of addressing client needs? Note that many of the statistics which could be used to answer these questions are the same as those which would demonstrate compliance with the Act's targeting requirements.

Outcome measures enhance these numbers by breaking them into more meaningful units. While the overall numbers can be useful, numbers from the outcome measures allow funders to see with more specificity what types of cases are being handled, what the outcome of these cases are, etc.

Accountability may also include a determination of whether unmet legal needs exist within the service area. To determine unmet needs, a reporting system may require information about clients who requested services but were not served, the kinds of legal problems they faced, and the reasons they were not served.

Again, because the specifications in provider-funder contracts vary, and therefore information required to show compliance varies, local needs must be considered when developing a reporting system.

4. Performance Evaluation

A funding agency also can use a reporting system as a creative tool for gathering information for evaluation of the quality of legal services, and providers can use it for self-evaluation. The AAA may want answers to questions such as: Is the program doing good legal work? What is the quality of advice and services given? Are the program's outreach efforts effective? These are the very questions outcome measures, when well-designed, allow providers to answer. The outcomes, when compared to the benchmarks, give a good indication how well a program is accomplishing its stated goals. In other words, the positive impact on the lives of clients give an indication as to the quality and effectiveness of the work provided.

In addition to the outcome measures, certain types of statistics can be used to investigate a variety of hard-to-measure attributes of a program, while open-ended and less structured data collection methods can be used to examine other program dimensions. For example, statistics on numbers and types of cases, time spent on cases, as well as results gained and practices followed, will help a provider or funding source examine the quality of legal assistance services given. In addition, statistics on the number of new clients contacting the program, and the source of new clients' information about the program (e.g., public service announcements; referrals from social agencies; talks by program staff at such sites as nursing homes, senior housing projects) can measure a program's success in publicizing the availability of its services among the elderly. Statistical data on numbers of cases opened and closed each quarter may also indicate whether the program maintains a constant, well-controlled case flow.

Also, narrative reports in which programs describe certain types of important non-statistical information also can be useful supplements to outcome data. Narratives allow the provider to transmit, for example, anecdotal information that can be of great value in "putting a face" on the impact of the legal assistance. Information that can be collected through a narrative report includes: case examples; descriptions of significant litigation; self-evaluations; outreach methods used; community education efforts and training events; new publications developed and distributed by the provider (e.g., training materials or community education pamphlets); changes in program staff; and newspaper articles, press releases or public service announcements regarding the legal assistance program. This information can be collected from narrative space provided on case closing forms, or through the use of a check-box (or separate field on an electronic system) that allows the provider to mark a certain case as factually favorable to a narrative description. Again, it should be noted that many of the numbers and other information collected for performance evaluation can also be used in reporting for accountability and other purposes.

Finally, as another evaluation mechanism, an area agency can hold periodic meetings between legal program staff, the legal services developer, and AAA staff and/or boards of directors. These meetings provide an excellent opportunity for staff to provide updates on the program's progress toward achieving its outcome objectives. Unlike written reports, meetings allow for a detailed discussion with follow-up questions about the legal program's activities. A related evaluation tool used by a number of agencies is the site visit, during which agency staff personally observe the functioning of the program's offices.

5. Demonstrating the Value and Impact of Title IIIB Legal Assistance

A valuable aspect of reporting is the excellent opportunity it presents to gather data on the impact and importance of legal assistance to older persons, that can be used to "sell" legal assistance to current and potential funders. This data can also be used to significantly increase the visibility and recognition of the value of legal assistance programs and to strengthen community support for these programs. Outcome measures are by far the best way to gather this data and present it to both funders and the general public. Because outcome data focus on the benefit to the client, less interpretation is required. In other words, the system is designed to collect and measure data on how the services effect the lives of clients. These data don't require a reader to speculate how a stated number of individuals served benefited; the data themselves are measures of the benefit.

Although legal assistance has long been a priority area under the OAA, those outside the legal services community, including some AAAs, on occasion have difficulty in understanding the value and impact of legal assistance. Unlike the clearly recognizable need for, and value of, a home-delivered meal or a ride to a doctor's office, recognition of the need for, and value of, legal services is much more difficult. What this means is that it is much more important for legal services providers to communicate with AAAs, and provide reliable and persuasive information about what they do, than it is for providers of other non-legal services. Outcome measures give providers the tools to accomplish this.

A reporting system that includes outcome measures can help a provider, AAA, SUA or even AoA to explain and justify the expense and provision of legal assistance. Outcome measures are a crucial piece in effectively using reporting for this purpose. To justify the expense of legal assistance, the report must include data on the impact on, or value to, the clients. Outcome measures do just that. Further, while outcome measures give a more complete picture of the impact, they can be supplemented with narratives to dramatically demonstrate the importance of the program.

Integrating outcome measures into an existing system can enhance all of the above purposes. Using outcome measures allows a program to collect information on how well it is accomplishing its self-stated goals. In other words, programs can determine what kind of positive impact their assistance is having in the lives of clients. This information isnÕt intended to simply demonstrate a use of resources. Rather, it can be used by the program to improve the legal assistance it provides.

Outcome measures can have other benefits as well. For example, the ability to demonstrate the program's impact on the lives of individuals and the community can aid programs to:

o Recruit and retain talented staff

o Enlist and motivate able volunteers

o Attract new participants

o Engage collaborators

o Garner support for innovative efforts

o Win designation as a model or demonstration site

IV. Conclusion

Outcome measures have a tremendous amount of potential -- the information they provide is persuasive to funders, satisfying to employees, and simply part of a good management system. Through well-designed outcomes measures, providers, funders, and the general public can see the true impact legal assistance has in the lives of older persons. However, the development process must not be an exercise for the sake of exercise; nor should it be undertaken simply because "it's required" or "it's supposed to be good." The process of developing them requires time and concerted efforts of providers and funding sources, and many, programs with well-developed outcome measures work months designing and refining them.

To develop truly useful and effective measures, providers or funders must go into the process because they want to answer the questions mentioned above: "What are we trying to accomplish?" and "How will we know if we are succeeding?" By starting with these questions, providers will have the opportunity to step back and focus on the overall purpose of the program. To get meaningful information, programs must think about the service provided from the point of view of the client. In the end, it is the client -- the older person -- who needs and benefits from the assistance.

Back to TCSG Home 

---

The Center for Social Gerontology, Inc.
A National Support Center in Law and Aging
2307 Shelby Avenue  Ann Arbor, MI  48103
Tel: (734) 665-1126  Fax: (734) 665-2071
Email:  tcsg@tcsg.org