Success Stories as Hard Data can be ordered from: Rose Barrett, Potomac Press, 3288 Oakmont Lane, Conway, SC 29526, Phone: (843)365-7005 Fax: (843)369-1011 or from Amazon.com  - A preview chapter appears below

Success Stories As Hard Data Barry M. Kibel, Ph.D.  

This abridgement of Success Stories As Hard Data has been prepared by the author to satisfy the immediate demand for materials on Results Mapping. The book was published by Plenum Publications in 1999 and is available in  hard and soft cover versions.

Introduction

Many health and social programs and virtually all community development efforts share a common dilemma. The best of what they do--the transformations and healing they help catalyze, as well as their short-term contributions to longer-term outcomes--cannot be easily measured. This makes it difficult to demonstrate their successes and full worth to funders, Boards, and others.

The best illustration of a program's work is frequently found in stories that relate its most dramatic successes with clients. These are the stories that program staff present to their Boards, share at conferences, or pull out when soliciting funds, and are the ones that some times reach the media. But funders and others who seek proof of program benefits are suspicious of such stories and with good reason. A few anecdotes about remarkable client turnarounds generally represent the exceptions to the rule and do not offer a true picture of what a program does on a day-to-day basis with most of its clients. Faced with a choice between receiving colorful narratives on a select few clients or colorless data on all or most program activities, funders have invariably opted for the latter. "Show us the numbers!"

Toward this end, a program focused on healing or client transformations will typically report body counts of those they have served, results from client satisfaction surveys, and an assortment of numeric measures of client or community outcomes, while engaging in a qualitatively different enterprise with its clients than these quantitative indicators reveal. Such a program will likely deliver more than the routine services indicated by such counts; it will work with its clients in unique and complex ways, frequently beyond standard work hours and over extended periods, helping them to gain control over difficult life situations and move beyond current crises. In place of remedies for all that ails its clients, the program will offer the resources and support necessary for its clients to exercise increased self-determination--hopefully leading, over time, to improved health and life quality.

Program staff are the first to admit that the numeric data they provide just skim the surface of program activity and relate little about the wondrous transformations that may have occurred with specific clients or target populations. They recognize that the parts of stories that give goose bumps and show the program in its best light, as well as key descriptive data, are missing from their reports.

The program's numbers tell little about the clients who have progressed beyond dependency on the service delivery system to heal themselves and become healing supports to others. They also reveal little regarding the innovative work of staff with difficult-to-serve clients, even when no dramatic outcomes have yet occurred; or of pioneering efforts of the program to partner with other service providers and community agents to serve clients who heretofore were not reached or who could not afford the type and range of services they desperately needed.

This book has been written for these programs. It presents a new form of program evaluation, called Results Mapping. While based on stories, the approach is not anthropological or merely descriptive. And while the evaluation field has been searching for a creative blend of quantitative and qualitative methods and this new form does this well, we did not set out with the intention of creating a blend. What we were aiming at was building a creative bridge between process and outcome evaluation, something which the evaluation field has also been lacking and demanding.

The past two years on the lecture circuit, coupled with several dozen small and medium-sized consulting contracts, have provided a priceless opportunity to refine the approach. The rules and conventions of Results Mapping have undergone multiple adjustments and outright changes during these two years as the methodology has evolved and matured. We have not been whimsical. Our aim throughout has been to increase the method's reliability. This twofold aim has involved (1) creating and applying mapping rules that translate narrative accounts of a program's best work into structured forms (i.e., mapped stories) without distortion, while (2) devising and applying coding and scoring conventions to these mapped stories that lead to fair ratings and accurate measures of program contributions toward long-term client success. The originality and complexity of the most outstanding work of programs kept driving us back to the drawing board to determine how best to capture their achievements using maps and scores. Of late, the changes we have made in the methodology have been relatively modest even as the volume of stories we have reviewed has increased manyfold. This provides some measure of hope that only modest tweaking may be needed in the months ahead and that Success Stories As Hard Data will continue to serve as the definitive text book for the approach.

The programs we have been asked to evaluate do not fit a mold. They do not produce the same outcome, nor even a small set of outcomes, again and again. In fact, they are not outcome producers. They offer no magic bullets or panaceas for the problems besetting individuals, families, neighborhoods, communities, and systems. They can, however, be exceedingly helpful to those they serve. At their best, they focus on client assets, not deficits. They raise spirits. They offer hope. They untangle knots. They set people on the path to new life possibilities. They help make difficult outcomes possible.

These programs need to be evaluated on their terms.

A Different Kind Of Evaluation

The types of programs best positioned to demonstrate that they are outcome producers are what I refer to as "fix it/cure it" programs. In the extreme case, one such program will offer a single intervention (e.g., a flu shot or brief intervention) in expectation of some favorable future outcome (e.g., continued or enhanced health). Clients are passive recipients of the program's service. Since each client gets the same service with expectations of a similar result, the program's body counts and success rates alone provide good and near complete proof of program performance. A fix it/cure it program of this type would not typically tell client stories; however, should it agree to partake in story telling, one such anecdote might read:

Contrast the relative simplicity of that story with the one following from a residential center for homeless youth:

With a story like Mary's, we have moved from the realm of simple attribution into the realm of complex attribution. Linking services to outcomes is no longer a simple exercise. The following chart contrasts the differences, at the extreme, between these two realms.
 
 

There is not a sharp boundary between the two realms, but rather a transition to ever increasing complexity. With a simple story, a program provides a single service to a client in hopes of causing a single outcome in the not-too-distant future. As programs provide more and varied services, as well as network with other agencies to support client needs, stories become more complex and thus also does attribution. The program's multiple efforts each contribute in varying degrees toward the attainment by the client of a range of diverse outcomes. Some of these occur soon after services are delivered while others may not occur until months or years following the services.

The logic underlying simple attribution is causal. The contention is that the single program service is responsible for the single outcome; stated in brief, it has caused it. Other possible causes of that outcome are considered either non-existent or insignificant. In contrast, the logic underlying complex attribution is synchronistic. Events are linked in time and space through connections and associations that overlap and mutually influence one another. Outside factors beyond the reach of the program may influence outcomes as much or more than the program's services. Further, in some cases, the client may as likely influence the program and the services it provides as the program is to influence the client. And, even more dramatic, the client may heal or grow from within.

In fact, the essential feature of complex systems--be these molecules, cellular structures, persons, families, teams, neighborhoods, communities, organizations, or institutions--is that, in them, new and previously undetected and unpredicted properties can and do emerge. One day, my daughter is a young girl; the next, she is suddenly a woman. What happened during the night to make the difference? For years, America was waging the cold war against the Russians. Suddenly, it was over. We did not "nuke" the Russians into submission. What happened was that our world transformed, emerging with new properties and possibilities. One can look for causes in the global economy, the information revolution, the inability of the Russians to match our defense spending, the mounting pressures of the culturally and nationally diverse elements of the soviet system, and the incompatibility of a "closed political-social system" within an open, dynamic, modern world society. But suddenly, seemingly overnight, there was and remains a post-cold war reality.

There are programs whose essential challenge is to fix and cure. There are other programs whose essential challenge is to help their clients to grow and/or heal. These two types of programs operate differently. They need to be evaluated differently. For both types of programs, outcomes may result. For the first type, the test of success is the ability to produce these outcomes. For the second type, the test of success is the ability to contribute to changes in the behaviors and status of their clients. As these changes occur, outcomes will result. But they are not produced by the program and it is misleading and a disservice to these programs to think solely or primarily in these terms.

Why Success Stories?

Effective program evaluation is information-rich. The primary use of this information is to assure the program's supporters that desirable outcomes are happening through the actions of the program and in sufficient quantity and with enough quality. In addition, the information provided should validate, if not celebrate, the hard and sometimes outstanding work of program staff in making these outcomes more likely. Further, the information should detect program shortfalls that ought to be corrected and pinpoint program strengths and emerging opportunities that can be exploited to increase program contributions toward these outcomes.

For "fix it/cure it" programs, the best form of information is numeric. Numbers are relatively easy to gather, combine, analyze, and use to draw conclusions regarding program performance. Further, the link between services and outcomes (the zone where stories might best be told) is simple, undramatic, and short.

For programs engaged in healing, transformation, and prevention, the best source and form of information are client stories. It is through these stories that we discover how program staff interact with clients, with other service providers, and with family and friends of their clients, to contribute to outcomes; and how the clients, themselves, grow and change in response to program inputs and other forces and factors in their lives. There is a richness here that numbers alone cannot capture. It is only for a story not worth telling, due to its inherent simplicity, that numbers will suffice.

There are, however, some inherent difficulties in relying on stories as the principal source of program evaluation information. First among these is the time and effort it takes to capture a story. Most programs do not spend much time following up with their clients to find out what happened to them after interacting with the program. While there has been a dramatic shift from process toward outcome measurement across the nation, adequate compensation for time needed to determine what these outcomes have been has not been forthcoming. Somehow, programs are expected to measure their outcomes but with little or no new money or staff time devoted to this venture. So most programs simply muddle through, capturing outcomes in ad hoc manner to meet funder requirements or to demonstrate that their promised productivity has been reached or surpassed. Consequently, the stories they can relate about their work with clients are incomplete. This is unfortunate. A lot of learning can occur if time is taken to find out what really worked and what worked less well or not at all in the months that followed the interaction period. Did short-term gains persist? Did what looked like a failure actually turn into a success?

A second difficulty is more fundamental. Most people lack the training and skills needed to be good story tellers for evaluation purposes. They tend to ramble, skip key points while dwelling on incidentals, and get the logic and order of the story mixed up. Related to this is a third difficulty. The person telling the story--most often the program staffer who has had most contact with the client featured in that story--may not know the whole story. Other staff may have been involved at the beginning who have moved on to other jobs. Further, the client will likely have details or know of factors influencing the outcomes that no one in the program has heard about.

Even when the stories are complete and accurate, the styles of relating them will vary greatly across story tellers. This is a fourth difficulty. And finally, assuming we can achieve accuracy, completeness, and consistency in a data base of stories, how can the varied information within these stories be combined to draw findings and conclusions useful to funders, program staff, and others?

These are some of the major issues that we have been struggling with in the development of Results Mapping as an alternative or complementary form of evaluation for programs engaged in healing, transformation, and prevention. And, I am pleased to report, we have arrived at reasonable solutions to each of them. The remaining materials in this book present and illustrate these solutions. In this section, as a way of introducing the approach, partial answers will be offered to each difficulty listed above.

Before proceeding with these answers, however, I should first explain what we mean by a "story." The following is a narrative account of a story included in an evaluation of a senior center.

Note that this is not a life history of Frances. The narrative focuses on her relationship with the program, a senior center in Cincinnati. Included is the first contact with the program from which she benefitted, followed by other interactions leading to contributions to her life. Also included are milestones reached by her and actions she has taken to benefit others (that are directly or indirectly traceable back to the program encounters). This narrative would be translated into a set of six results maps, each map documenting some major program action or related activity key to Frances' growth or well being.

Following are the six maps that document the program's contributions to her health and life quality. They reflect these six elements of the narrative account:

No reference was made to outcomes in either the narrative or mapped accounts of Frances' story. Were these to be articulated, the outcomes associated with her on-going story might include enhanced quality of life, continued independent living, and, should the need again arise, access to quality healthy care. As referenced earlier, in Results Mapping, we focus on contributions toward these outcomes rather than on their attainment. Each map in the story was such a contribution, some more modest than others. The levels assigned to the maps (e.g., ACT3/LEV3 or MLS4) reflected the relative value of each contribution. Included were program contributions to the life of the client, as well as actions by the client herself that built on and were a follow-through to the program contributions.

Time and Effort. Once getting accustomed to the mapping procedure, it takes about ten minutes to map a story like the one just illustrated. What might take far more effort and time is finding the details of the story. In the illustration, this was relatively easy and quickly accomplished. Frances is a regular at the center and the fact that she attends regularly and has recently begun helping out in the kitchen is well known to the director and key staff at the senior center who provided the story. Furthermore, Frances' story did not involve a lot of twists and turns and the center was the only service provider. It gets far tougher to research a story where contact with the client has been lost and has to be re-established.

One might wonder how a program with one hundred or more active or recently served clients could possibly capture all the information needed for Results Mapping. Well the good news is that the program does not have to relate one story for each client. In fact, data from 12-15 stories are usually more than enough to gain a good sense for how the entire program operates to produce client growth--even where hundreds of clients have been served in recent years. But these cannot be just any dozen stories or a random sample of stories. These need to be the very best stories that the program can provide (i.e., the stories, that when scored, receive the most points for program contributions and follow-through client actions).

It is quite common for programs to showcase their one or two most outstanding client success stories. It is much rarer for programs to array and analyze the dozen or so stories that feature their best work with clients. Yet this set of top stories, when arrayed and analyzed, affords invaluable information regarding the practices of the programs. Further, should these programs be dedicated to continual improvement, there is perhaps no better data on which to base such improvement than that provided through study of their top stories.

For fix it/cure it type of programs, where all cases are supposed to be alike, the average case is a good place to start--since all cases ought to look like that case. However, for programs engaged in healing, transformation, and prevention, the average case offers little that the programs should want to emulate that is not also included in their better cases. But the reverse is not true. The average case lacks much that the better cases include. By drawing attention to a program's best work, it is our intention to prod that program to make the necessary adjustments so that the exceptional becomes the norm.

There is a second compelling reason why the best stories are enough for Results Mapping purposes. Because there has not been a focus by most programs on being exceptional, and because staff resources often have had to be spread thin among many clients, there simply are not that many exceptional stories that can be told. Once a program gets five or ten stories down in its story base, all the remaining stories tend to be rather "average." Put aside the few top stories, and the vast majority of programs engaged in healing and transformation take on the look of "fix it/cure it" programs. In short, they provide near identical services to clients and all their remaining stories sound pretty much the same.

So, in conclusion, programs are not required to devote excessive time and energy to story telling and related research. A dozen or so complete stories are adequate--in most cases--for an end-of-the-year assessment of overall program performance. For the rare programs that have mastered the art of healing and transformation, and further have learned how to maximize local resources to serve lots of clients at exceptional levels, more stories (say, a total of 25-30) will be needed to derive a more complete picture of the best work of each such program. But these are invariably the types of programs that keep in touch with their clients, know the stories, and--if need be--can get their clients to write their own stories and submit these for editing and analysis.

Story Telling Technique. The thorniest problem associated with Results Mapping is getting programs to report their work with clients clearly and completely. Once the details of a story are available, the mapping and associated scoring are easily accomplished by us or by program staff that have mastered the technique. When we first started asking programs to present their stories, we had little to offer them by way of examples or to serve as guides. Today, with several years of experience, reports, and story submissions from across the country to build upon, we can provide programs with clear examples of what a story needs to contain to be suitable for analysis. Many examples of this type appear in this book.

The rules governing mapping and scoring of stories are also helpful in guiding story telling. The rules encourage a story teller to explain simply and clearly how the program first got involved with the client, what actions it subsequently initiated to promote client health and growth, and how the client responded to these actions. When we return partially completed maps to programs with questions, and point out the rules that either have been violated or cannot be applied, they learn what it is we need and what is excessive or redundant reporting.

Getting the Story Straight. Story telling is a new endeavor for most programs. The information available in case notes, where these exist, can be very helpful in piecing together a story. For community development stories, there probably are no detailed case notes, and programs need to rely on memory, in-house monthly or quarterly reports, and the like as aids to story construction. Where networking with other service providers has occurred, these agencies should be contacted. They can help fill in details and add new information that the program may not have on record or otherwise know about.

Where clients can be contacted, programs are encouraged to do so. Programs need to stress to clients that: "They are being featured in a report on the very best work of the program in recent times. The program is proud of this work, as well as the growth that has taken place in the client, and wants to share this good news with others who may benefit from it." We know of no situations, so far, where clients did not want to have their story told when presented in this light. In many cases, the process of having their stories told is self-validating and contributes further to the clients' transformations. They may ask to have identifying details removed or altered; and this can almost always be done to protect confidentiality and still produce stories that are close enough to the facts so that learning is not distorted and reported results remain valid.

Consistency Across Stories. The narrative story represents an informal, somewhat unstructured account of the interactions between the program and the client (what might be classified as a "right brain dominant, left brain subordinate" reporting of the program-client interchange). The mapped story represents a formal, structured retelling of the account (i.e., a "left brain dominant, right brain subordinate" re-play of that same interchange). It is as though a clever computer digests the narrative and reports it back emotion-free, logically, and linearly. We do not require consistency in the narrative accounts, provided there is sufficient detail to accomplish the mapping.

We do, however, expect consistency in the mapped versions. Two individuals trained by us--or who have read and mastered the materials included in this book--

when presented with a detailed narrative account, should map and score it in virtually the same way. There is some room for interpretation during mapping, but very little. A particular map, for example, might be assigned a raw score of 2 by one mapper and a score of 3 by the second; but this will have little bearing on the analysis that follows or on the conclusions drawn--unless there are many maps of this type in the program's story set. Where significant mapping differences have arisen in the past, we have invariably modified a mapping rule or convention to eliminate these differences and regain consistency across mappers.

Aggregating the Information. The coding and related scoring system associated with Results Mapping is what sets it apart from other forms of program evaluation of which we are aware. Each program action that directly or indirectly encourages growth of the client featured in a story receives a code and score. Linked actions by other service providers or family/community members that benefit that client are also coded and scored, as is each action taken by the client for self-help or to benefit someone(s) with similar needs. As scores for a story accumulate, these are subtotaled as service, networking, village building, and self-determination points.

Among these codes, a special set are called "milestones". These are akin to short-term and intermediate-range outcomes associated with more traditional evaluation approaches. In the story illustrated earlier, Frances reached a short-term outcome (coded as MLS4) when she began participating in activities at the senior center on a regular basis. This was significant in her life, since she was new to the city and had few acquaintances in her own age range. The new habit of going to the center certainly added to her quality of life, as reflected in subsequent maps of the story. If the program continues to map Frances's story, at some point soon she will be credited with reaching MLS6, a sustained or intermediate-range outcome.

The combination of codes, milestones reached, and points--within and across stories--provides a comprehensive picture of the best work of a program. A very effective total quality improvement system emerges when a program begins to track these data from one evaluation period to the next, while taking steps to increase the levels of activity, numbers of client milestones reached, and point productivity in its top stories.

With due caution, agencies that operate multiple programs--each aimed at different target populations pursuing varied and different outcomes--can contrast the action levels, client milestones, and point productivity associated with each program and draw conclusions regarding their relative effectiveness as agents of healing and transformation. Similarly, funders providing resources to varied programs can begin to see where they are getting most impact--via program contributions toward short-term and longer-term client outcomes--for their dollars. Care must be taken to relate findings of program differences to factors such as client readiness, contextual variables (e.g., culture and political climate), program experience, adequacy of resources, and existence of best practices and well-researched strategies that can be adapted locally. Still, it is important to begin to make these comparisons so that reasonable pressure can be placed on programs to make the most of the resources with which they have been entrusted.

The Results Ladder

There appears to be a hierarchy of levels that individuals, groups, or communities invariably pass through on their path toward transformation and healing. We refer to it as The Results Ladder. This Ladder is the spine around which Results Mapping has been constructed.

The Results Ladder. Included are seven levels, three of which are designated as milestones because they represent sustained behavior adjustments toward longer-term outcomes.
 
 
 
 
 
 
 

Although The Results Ladder is an original template for describing stages of progress toward healing and transformation, not surprisingly it somewhat resembles other hierarchies that have been developed to capture change and growth processes. To illustrate:

As can be seen, the expanded Ladder includes the same seven levels and milestones. However, it provides for the introduction of an additional actor, the change agent and his/her/their role in assisting the client (or other recipients included in the client's story) to reach Levels 1,2, 3, and 5. The use of a dual system of coding (ACTn and LEVn/MLSn) is key to Results Mapping.

Allow me to discuss the four action levels, after which I will return to the three milestones:

The easiest way to distinguish between ACT1>LEV1 and ACT2>LEV2 is through the directness of the relationship between the change agent and the client. At Level 1, the change agent is targeting efforts at the general public and not at any specific client. At Level 2, the change agent is focused directly on a specific client's progress or health.

The primary distinction between ACT2>LEV2 and ACT3>LEV3 is in the role the change agent plays for the client. At Level 2, that role is one of prodder and referral source. At Level 3, it is to actually mend, educate, train, or otherwise cause short-term improvements in the client's status.

The main difference between ACT3>LEV3 and ACT5>LEV5 is in the nature of the relationship between the client and the change agent. At Level 3, the change agent leads and the client follows. It is a teacher-student, parent-child, or doctor-patient relationship. At Level 5, the relationship transforms to adult-adult (to employ the terminology of Eric Berne's transactional analysis) or from an I-it to an I-Thou relationship (to use Martin Buber's paradigm) and the role of the change agent is gradually reduced to allow the client to move forward to MLS6. If the client is an institution or a system, Level 3 activity tends to be preparatory work (e.g., a series of planning meetings), whereas Level 5 activity relates to full program implementation (e.g., carrying out the plan).

At Milestones 4, 6, and 7, there are no external change agents (and hence no ACT codes). The client is acting as a self-change agent. Let me review each of the three:

The main difference between MLS4 and MLS6 is the intensity of commitment of the client and the extent to which changes have been integrated into the client's life. This is typically measured by the length of time that the client has sustained these changes. At MLS4, the client is testing the waters with no long-term commitment. At MLS6, the client has a multi-month or multi-year history of personal engagement in the healing or transformation process and can point to sustained gains in health, wellness, or life quality.

The main difference between MLS6 and MLS7 is that the latter represents a total integration of the changes the client has been seeking and inducing. The work is complete, no further advances are contemplated. If further change does occur, it will be along an entirely new growth path. The caterpillar has become a butterfly.

These three milestones are the outcome levels that funders and programs are most interested in seeing clients reach. MLS4 can be viewed as a short-term or intermediate outcome, MLS6 as a longer-term outcome, and MLS7 as an ultimate or ideal outcome. The earlier levels (particularly gains at LEV3) can be viewed as interim outcomes that often are necessary precursors to higher-level development. The integration of these milestones with ACTs and LEVs allows a full picture to emerge of the actions and responses needed to bring clients toward and to these desired outcome levels.

Terminology and Key Mapping Concepts

Results Mapping is used to map, score, analyze, and provide feedback to improve the best work that a program does with its clients, be these individuals, families, teams, groups, communities, organizations, or systems. Each story features some of that best work. It is not the client's life story that is being presented. Nor is it only the interface of the program with the client. Rather, it is a story that begins with the first interaction between the program and the client and extends to further program-client interactions, to program interactions with the client's support system, to client interchanges with others mobilized by the program to assist the client, and to personal client achievements in support of self or to benefit others.

Excluded from the story are services provided to the client by other agencies that are not linked to earlier program actions to benefit the client. Also excluded are client activities and achievements that are not linked to the program's objectives or are well beyond the contributions of the program to these achievements (note: a discussion of leveraging appears in the book but not in this abridgement).

Cl-I-ent Not Client. Rather than produce outcomes, the programs whose work is best suited for study through Results Mapping are helping their clients grow out of the circumstances that diminish their lives and into new life contexts. To keep reminding us of this, we began using the term cl-I-ent. The capital "I" emphasizes that the program is working with subjects, not objects, to foster self-determination, growth, health, and emergence of creative potential. This convention is followed for all types of cl-I-ents, be they indiv-I-duals, fam-I-lies, un-I-ts, ne-I-borhoods, commun-I-ties, organ-I-zations, or inst-I-tutions.

Types of Maps. A story relates how the program being evaluated has contributed to near-term and longer-term cl-I-ent successes both directly and through leveraging the resources of others (including those of the cl-I-ent). That story is told (by a program representative, the cl-I-ent, or both) in narrative form and then mapped and scored using a formalized method. Each element of the mapped story is referred to as a mapped sentence or simply as a map.

There are two types of maps used to document a story. The more common type is a transactional map. A transactional map has this form:

        [CHANGE AGENT] [TOOK THIS ACTION] resulting in [RECIPIENT][GAINING THIS BENEFIT]

Here there is both a change agent and a recipient. The change agent may be a staff member of the program, but could also be staff from another program, a volunteer, a member of the cl-I-ent's family, the cl-I-ent (in support of others), or any one else taking actions to benefit a recipient. The recipient may be the cl-I-ent, a family member, or another community member benefitting from the actions of the change agent; or a future change agent that is being mobilized to action.

The second type of map is a self-referential map. A self-referential map has this form:
 
                        [CHANGE AGENT] [TOOK THIS ACTION FOR SELF-BENEFIT]

Here there is a change agent but no recipient. In effect, the change agent is the recipient, taking action for self-benefit. This type of map is also used when the change agent is taking action that ultimately is meant to benefit another, but when that benefit will only accrue after subsequent action is taken.

Map Sequence. Maps are presented in roughly chronological order to capture all significant program contributions to current and future successes of the cl-I-ent. Again, these include contributions where program staff are the change agent but also contributions made through the efforts of others that can be linked back to earlier, related program efforts.

Map Codes. Each change agent action is coded (with an ACTn) as is each recipient response (with a LEVn). For self-referential maps, where there is no recipient, only the change agent action is coded (as an MLSn or ACTn). For transactional maps, the results level (n) is the same for both the ACTn and the LEVn codes. Thus, for example, if the action taken is coded as ACT1, then the gain to the recipient must be LEV1. The values of n range from 1 through 7, corresponding to the seven results levels. They are used to categorize the type of action taken by the change agent and the matching benefit to the recipient. As outlined above, three of these levels only appear on self-referential maps (and are distinguished from the others by changing the ACTn code to a MLSn code--for personal "milestone"). The remaining four levels may appear on both transactional and self-referential maps, although they are far more common in transactional maps.

Starting A Story. As noted earlier, a story is mapped in roughly chronological order beginning with the time that the program first interacted with the cl-I-ent featured in that story. The first map of the story (Map 1) is always a transactional map with program staff (or a staff volunteer) as the change agent. Should there be relevant background information that helps to explain the story, particularly if it justifies the claim of a subsequent cl-I-ent milestone, this material is presented as Map 0 and not scored (i.e., no ACTn or LEVn codes are affixed to the map).

Population Sizes. Each map also contains the number of change agents and recipients featured in that map.  When the change agent or recipient is a collective (i.e., a team, organization, or institution) acting or reacting as a single entity and not as separate individuals, a population value of 2 is used, by convention, rather than the actual number of people in the group.

Population Types. Each map also indicates the type of change agent and recipient. The following six codes are used to indicate the type of change agent or recipient:

        S     Program staff

        C     Cl-I-ent

        F     Family member of the cl-I-ent

        P     Individual provider or professional (not staff)

        G     Group (team, committee, organization, institution, or system)

        X     Other community member

When a map's change agent has performed the action as a volunteer, a "V" is placed before the code. For example, a volunteer organization would be coded as "VG", a doctor providing free medical care would be coded as "VP", and a citizen serving as a volunteer would be coded as "VX".

A volunteer from the community who provided a one-time service or a service that is short-lived is coded as "VX". However, if the service was ongoing (e.g., serving as a mentor or care giver) and the program being evaluated provided logistical or other support for the volunteer, then the volunteer is considered surrogate staff and is coded as "VS". This has important implications for leveraging (as discussed below).  Furthermore, an individual coded with "VS" can kick off a story (i.e., there is no need to show that person being recruited by the program to serve the cl-I-ent unless this is key to the story).

Mapping Personal Milestones. Each personal milestone is mapped separately (i.e., if three cl-I-ents each reached a personal milestone, these would not be clumped together on one map but would be documented on three separate maps).

Maps Cover Three-Month Periods. When a change agent provides repeated services during the same three-month period directed at the same cl-I-ent objective, it is mapped only once (with the dates in the Date field that follows the Map Code indicating the time span and the text under Did What noting the frequency as well as type of service provided). However, if the service continued beyond three months, for each additional three-month segment, a new map is added to the story. Thus, for example, a service that is provided continually for a year would result in four maps, one for each three-month period.

When services are directed at different cl-I-ent objectives, then multiple maps are used even when these services occur within the same three-month period. Thus, for example, if program staff provided services to a cl-I-ent aimed at improving the latter's reading skills while also providing services to that ind-I-vidual dealing with some health issue, each set of services would be mapped separately. For either set of services, the three-month rule would apply.

Handoffs. In many of a program's top stories, it is likely that services will be provided by others in addition to program staff. When the program refers a cl-I-ent or family member to another service provider, the map showing this referral is called a handoff and has the program staff as the change agent and the service provider as the recipient. This holds true even when direct communications between the two did not occur. So, for example, if the program told a cl-I-ent about a provider, and the cl-I-ent made that contact and received the service, the map describing the referral is still shown as a handoff from the program to the provider.

Only Activities During the Past Two Years Are Scored. Evaluations typically run on an annual cycle, established by the funder. Stories unfold according to their own natural rhythm and may take months or years to play out. Thus, when mapping a story, it may be necessary to go back two years or longer to capture the full extent of program involvement with the cl-I-ent. Consequently, it is common for stories to run into a second, third, or even fourth evaluation cycle.

By convention, when scoring stories of long duration, only maps with dates of two years or less from the cutoff date for the evaluation report are scored. The earlier maps are zeroed out (i.e., included in the story to provide context but not scored). This allows the entire story to be told, but avoids crediting a program for work that was done well before the current evaluation cycle. So, for example, if an evaluation began in January 1998 and the first annual report was due a year later (January 1999), the program stories in that report might go back as far as the early 1990's, but only maps with dates after January 1997 would be scored and included in the analysis of the program.

Results Mapping can be applied at any time to ongoing programs to capture how successful they have been with their top stories during the most recent years. One does not have to design an evaluation plan; one just starts mapping. For this reason, Results Mapping is ideal for programs with little or no evaluation budgets which still want to benefit from the rich feedback that a comprehensive evaluation can provide. Mapping can also be appended to ongoing evaluations using more conventional methods (e.g., multi-year, quasi-experimental evaluation designs) to supplement and enrich these efforts.

For start-up programs, it may take six months or more before the program has sufficient impact on its cl-I-ents so that its best work can be distinguished from more "average" activity. We recommend that a start-up program begin thinking from the outset about the types of information that will be needed to relate and map its best stories, and set up an information system or case notes format that will facilitate later story mapping.

As suggested above, a first-year evaluation based on Results Mapping should include a program's 12-15 top cl-I-ent stories. In the second and subsequent years of the evaluation, we suggest increasing that number to 25-30 stories. At any time, new maps can be added to a story. Therefore, a story featuring the same cl-I-ent may have a different point value (referred to as its story score) from one evaluation to the next. Since only the last two years worth of point productivity will be included in the analysis, some stories likely will be dropped each round and replaced with others where the program has recently  made greater contributions.

Comparable Cl-I-ents. One final note. Only comparable stories can be included in an analysis. If the program has two or more distinctly different types of cl-I-ents, then requirements for Results Mapping are 12-15 top stories per cl-I-ent type. To illustrate, in our current evaluation of family resource centers in Cincinnati, each of the five centers included in the study provides services to individuals and families but also spearheads community development projects. For the baseline for this evaluation, each center provided its 15 top service-based stories and its 15 top community development stories. In the second year of the evaluation, these numbers were increased to 30 top service-based stories and 30 top community development stories. As a variation on this last point, should program management want to contrast the work of different staff or teams that are working with similar cl-I-ents, it will also need 12-15 top stories per staff member/team to allow useful analysis.

What the Mapped Data Show

Mapped stories afford a wealth of information regarding what a program does best. The stories themselves are illuminating. It is remarkable how few persons actually know how a program works to get its top results. Few Board members, program administrators, funders, or even co-workers can relate in any detail the twists and turns of stories involving a program's most successful cl-I-ents--beyond perhaps its first two "super success" stories. The 15-30 stories compiled each mapping cycle for analysis often represent the first comprehensive picture of the day-to-day performance of the program ever captured. The data these stories yield make it clear how the program works to achieve its successes and where it needs to work harder or smarter to achieve more of these.

Few social, health, and prevention programs are as potent and impactful as they can be. The best programs can get better and the more average programs have far to go in optimizing their resources and services on behalf of their cl-I-ents. And one key to such dramatic improvement involves learning from their best work: making today's positive exceptions, tomorrow's norms.

Through Results Mapping, we can address a fundamental evaluation question: To what degree is the program living up to its potential? This is a question not often asked. Most evaluations tend to focus on a less ambitious question: Is the program meeting its promised targets? Although a good question, by tracking a program's contributions toward cl-I-ent outcomes, we are able to answer the latter question while probing deeper.

Strong programs appreciate this deeper level of inquiry because they know they are good at what they do, would like others to recognize this, and want to get even better. These programs have no problems in meeting targets and typically learn little from evaluation findings that simply report this truth. Their administrators and staff tend to view evaluation data as something to be prepared for others but of little relevance for day-to-day operations.

We are discovering that weaker programs, while suspicious and somewhat skeptical at first, also quickly learn to like this deeper question. The answers provided through Results Mapping make it easy to pinpoint program shortfalls (e.g., the absence of timely follow-through with cl-I-ents or lack of attention to the role that seniors might play as volunteers) and direct managers and staff to actions that strengthen what they do. With a focus on continual program improvement, programs more easily meet stated targets, look good to their funders, and move up in the rankings toward the performance levels of strong programs. Although I frequently hear programs complain that they shouldn't be compared with other programs--since they are unique--they do recognize the value in comparing their own performance patterns from one evaluation period to the next.

To improve quality and advance toward excellence, one must unsettle those who maintain the status quo. This holds for science, for art, for success in business, for shared beliefs and prejudices, and for programs. Head counts, satisfaction survey results, and bottom-line measures of outcome and efficiency--the standard feedback to date of program evaluations--are rarely unsettling. These data do not often provoke or promote fundamental change. When positive, they merely reinforce the status quo; when negative, they become the points from which to attack the logic or practices of the evaluators.

Results Mapping provides feedback with an unsettling punch. In a very commonsensical way, the approach is used to uncover for scrutiny the very best work of a program--work of which line staff and managers ought to be proud. If that work is truly superior, it scores high and graphs well. If there are gaps or inconsistencies in performance, these are reflected through the data generated. "Here are your scores," we say. "What do you think of them? Are there findings that are disturbing or a source of concern? If so, let's talk about what might be changed in the program to move the program closer to where you wish it to be!"

To be unsettling, we document through Results Mapping how much the program is contributing toward outcomes. We examine its top stories and ask:

To be unsettling, we focus on how effective the program has been in networking with other service providers in the community. We examine the networking points from story to story and see where these partnerships have been most successful in moving cl-I-ents to milestones. And ask:

To be unsettling, we focus on activities with volunteers. We array the points earned by different types of volunteers. We determine the extent to which programs have made creative use of their cl-I-ents as helpers for others or made it more likely that they would be positioned to be helpful to others in the future. We then ask:

We are particularly unsettling when we provide scores to programs and especially to funders. There is something about a score that provokes an emotional reaction--likely a throwback to school days. And, unfortunately, a score is still viewed as "harder evidence" than a story. While physicists and other leading edge scientists now recognize that quality is more substantial than quantity, most of the rest of us still believe the opposite. And I, for one, while promoting the use of a mix of evaluation data, am not opposed to scores being part of that mix. I enjoy following sports and recognize the critical function that scores and other performance measures play in pushing athletes to their current limits and beyond. I am also a fan of total quality management and its guiding principle: Only what gets measured gets attention; only what gets attention, gets fixed.

The blending of stories, outcome and impact data (where available and relevant), and scores yields the "hard data" that are needed for fair and comprehensive evaluations of programs engaged in healing, transformation, and prevention. "Hard data," to me, are those that (a) are accurate, (b) provide the kinds of evidence that evoke confidence among decision-makers and (c) are consistent with the best current science from whatever fields the latest truths and insights regarding human nature are being generated.

The "hard data" provided through Results Mapping is useful to funders engaged in outcome-based funding. With clear documentation in hand of what programs have been able to accomplish with their best cl-I-ents, and with data to estimate how much the overall program has been contributing toward making future cl-I-ent outcomes more likely, funders are better positioned to make realistic demands on these and similar programs. It becomes possible, for example, to negotiate performance-based contracts where, under terms acceptable to both funder and program, reimbursements can be linked to progress of cl-I-ents (e.g., numbers reaching MLS4 and MLS6) and associated point productivity (to reward, where appropriate, increases in village building and networking activity).

When similar programs are funded in different sites, Results Mapping data from each site can be contrasted. Since programs may be serving different population mixes and have differing conditions and constraints under which they operate, contrasts and comparisons must be made with caution. Still, much can be learned by contrasting the top stories and associated practices of different programs. Funders and others can use the stories and associated data to explore with programs why there are cross-program differences. And, having accounted for site-to-site variations, funders might reasonably ask why they should continue to fund sites that are far less productive or that are less committed than others in contributing to cl-I-ent progress.

We urge programs using Results Mapping to join our informal learning network. We continue to learn new things about this relatively new approach that we are happy to share with others. This book is a first step in this direction. We are anxious for the approach to be applied and gain acceptance across the country. But we are also concerned about quality control. We would hate for Results Mapping to gain a bad reputation through misuse. For this reason, we protect the term legally and require active permission or formal licensing agreements for the proprietary aspects of the product. Naturally also, we want to be kept informed of Results Mapping applications and be called on to answer questions or to troubleshoot where needed. And we are, of course, pleased to be invited to participate in new adventures. So keep in touch.

For further information contact:

The Results Mapping Laboratory
Pacific Institute for Research and Evaluation
1229 E. Franklin St., 2nd Fl.
Chapel Hill, North Carolina 27514-3307

Phone: (919) 967-8998 ex: 15
Fax:      (919) 968-1498
E-mail Journey@pire.org

Home