Print bookPrint book

Unit 11 – Designing and Evaluating Systems and Processes

Unit 11 – Designing and Evaluating Systems and Processes

Site: Athabasca University: Open Courseware
Course: LGST 489: Alternative Dispute Resolution (OCW)
Book: Unit 11 – Designing and Evaluating Systems and Processes
Printed by: Guest user
Date: Friday, 22 June 2018, 12:46 PM MDT

Table of contents

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes

Overview, Learning Objectives, and Reading Assignments

The mediation was ... - cartoon

Reprinted with permission from

Unit 11 focuses on what has been learned about the design of suitable ADR processes and systems. It also explores the issue of how to evaluate the success of ADR initiatives, and how to determine which ADR method is appropriate in a given dispute and its context. Within the discussion of the pros and cons of ADR, issues of culture and gender are again raised with the concern about whether a process can be designed to accommodate everyone. You will find yourself needing to answer the question of whether alternative dispute resolution is a good alternative.

Learning Objectives

When you have completed Unit 11, you should be able to achieve the following learning objectives.

  1. Identify and describe the elements to be considered in designing an appropriate dispute resolution process.
  2. Describe specific problems and concerns relating to culture and gender in attempts to design appropriate dispute resolution processes.
  3. Describe some methods of evaluating dispute resolution processes and systems.
  4. Describe some measures of the effectiveness of alternative dispute resolution processes and their effects on the legal system.

Reading Assignment

Amsler (formerly Bingham), Lisa Blomgren. (2009). Designing justice: Legal institutions and other systems for managing conflict. Ohio State Journal on Dispute Resolution, Forthcoming.

Brahm, Eric, & Ouellet, Julian. (September 2003) Designing new dispute resolution systems. Beyond Intractability. Guy Burgess & Heidi Burgess (Eds.). Conflict Information Consortium, University of Colorado, Boulder.

British Columbia Ministry of Attorney General. (June 2003). Reaching resolution: A guide to designing public sector dispute resolution systems.

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes


We're here to litigate - cartoon

Reprinted with permission from

Dispute resolution (DR) design, or conflict management system design, refers to creating initiatives that use mediation and other DR mechanisms in an organized way to prevent or resolve disputes (Chornenki & Hart, 2001, p. 185). DR design is a customized process for a specific dispute, such as a sexual harassment policy that will use mediation as the main DR tool. These programs could exist in any industry or organization or for any group. They could be designed to deal with an absence within an existing program, the development of a new approach, or an addition to a dispute resolution strategy. They typically deal with organizational change by moving from one method to another, such as litigation to interest-based dispute resolution.

“Process” design tends to focus on the development of a single process for a particular conflict, such as mediation. Dispute “system” design (DSD) normally deals with multiple disputes within an organization, which may require multiple processes for different stages and types; for example, a multi-step grievance procedure in a human resources system of a local government. System design and evaluation is complex and involves policy considerations. A single process can become a system, built over time, in stages or for different cases.

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes

Orientation Questions

In preparation for the material studied in Unit 11, write out your answers to the following questions. You are not expected to give detailed answers at this time, but your responses should reflect your current knowledge, experiences, and beliefs. Keep these orientation questions in mind as you proceed through the unit, then review them again after completing the unit material, adding the information you have learned.

  1. What considerations must be kept in mind when designing a dispute resolution system?
  2. How might a system deal with issues of gender and culture?
  3. How have dispute resolution processes changed the legal system?

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes


The Stages of Design

Daddy has the power ... - cartoon

Reprinted with permission from

Alternative dispute resolution design occurs in four stages: needs assessment, program design, implementation, and evaluation.

Needs Assessment: The organization must consider both its own interests and those of its stakeholders. It must assess how to handle complaints or potential disputes, costs, and negative feedback. The company must be motivated to make change and must be committed to seeing those changes through. During this stage, the organization will look at other alternative dispute resolution practices to determine fit. Benefits and objectives are identified and considered at this stage.

Program Design: Methodology is chosen and a design team is selected at this stage. Decisions are made about internal or external resources, with the goal of maximizing party satisfaction.

Program design considers the following:

  • what gets designed
  • DR methods to be used, and why
  • rules for governing the DR program
  • who will perform the services
  • what records will be kept
  • funding resources
  • administration
  • promotion
  • implementation
  • information to be saved

Implementation: This stage determines the steps that will be followed, including information and education sessions for users, appointment of ADR personnel, and planning the tasks and timeframes. Pilot project and ADR training are considered at this stage.

Evaluation: Program goals are revisited at the evaluation stage. Information is elicited and synthesized to determine whether the goals were met. Were the disputes resolved? Were these resolutions less costly and less time-consuming? Will the program continue or be modified? Were data collected? Were surveys done on an ongoing basis? What immediate follow-up needs to take place?

Since the objectives of a program will determine how it is structured, each program will be unique.

Since the objectives of a program will determine how it is structured, each program will be unique. Some may be single-process programs, such as the Canadian Motor Vehicle Arbitration Plan (CAMVAP), which is primarily an arbitration program for disputes between vehicle manufacturers and customers. Others may have multilevel dispute resolution, such as Canada Customs and Revenue Agency’s (CCRA) “Independent Third-Party Review” (ITPR) process, which deals with staffing issues.

The connection between program objectives and program design is well illustrated by the example of human resources issues with respect to filling positions within the Canada Customs and Revenue Agency. A CCRA employee has recourse to three separate levels of review if that employee believes he or she was qualified for a position but did not get it. The first level of recourse is that of “individual feedback,” where the employee is entitled to ask for additional information from the person who made the decision about the position. If recourse at this level is not deemed by the employee to be sufficient, then he or she may request the second level of recourse, known as “decision review,” where a manager facilitates a review of the decision with the employee. At the third level of recourse, the employee may apply for review of the decision by an independent third party. This review involves a hearing by a roster arbitrator who is an independent contractor. The parties expected to participate in the hearing would be those involved in the original decision, which typically would include the staff member and any witnesses, human resources staff who have had input in the decision-making process, and perhaps union personnel if their contribution was necessary. The reviewer conducts the hearing and then provides a written decision that is final and binding.


This nationwide program underwent all the stages of ADR program design. A needs assessment was conducted because of the significant number of grievances and disputes that needed to be managed within this large, national agency. Feedback from employees indicated the need for a new and more independent system. Multiple human resource associations were consulted, alternative dispute resolution processes were reviewed and considered, and a range of objectives, goals, and desired outcomes were identified. The resulting program was designed to make use of multilevel methodology to satisfy the organization’s many needs. The program was then implemented, starting with the selection of roster arbitrators and administrative staff, followed by information sessions for those involved. Cases were heard and procedures were modified to improve the system. Evaluation was done over a two-year period, which allowed for changes and improvements on an ongoing basis. The final product is being administered with the benefit of the information provided from the reviews.

Outcome Objectives

Defining outcome objectives is a crucial step in any process design. Knowing where one is headed—and how to read the map—will help one recognize the desired destination. Despite the stated objectives, some outcomes may be unanticipated.

I know you normally ... - cartoon

Reprinted with permission from

Not all objectives will relate to processes or costs. Some may involve relationships or facilitating dialogue where there has been acrimony, as in many divorce cases. Other objectives may seek to improve systems that no longer meet the needs of those they serve. Still others may be designed to achieve better outcomes than can be provided by an adjudicative system, where win-lose is the standard and win-win is the objective. Robert Baruch Bush (1989) suggests that dispute resolution outcomes may be categorized into six separate categories:

  1. individual satisfaction (needs met)
  2. individual autonomy (own resolve)
  3. social control (strengthen social standards)
  4. social justice (neutralize social inequities)
  5. social solidarity (provide common texts)
  6. personal transformation (encourage personal change)

Client-Centered Design Process

Clients (those for whom a system is being designed) know best what they are looking for since they know the group they aim to serve. They have experienced what has not worked and therefore seek a solution that will make things work. Certain questions need to be considered. Who is the group to be served? What are their particular needs? What cultural implications might there be? Any intervener who sets out to design a dispute resolution system must consider the demographic, status, and affiliation variables involved. As Silbey and Merry (1984) state, “disputes are cultural events, evolving within a framework of rules about what is worth fighting for, what is the normal or moral way to fight, what kinds of wrongs warrant action, and what kinds of remedies are acceptable” (p. 157).

Herbert Kelman (2005) emphasizes the importance of building trust between members of different cultures and the role of an intervenor in facilitating that development as the first step in a conflict resolution process.

Recognizing and understanding the cultural implications of conflict is demonstrated by police departments that appoint officers of a particular cultural background, or have familiarity with a certain culture, to participate in dispute resolution in districts where they can use their knowledge.

Applying cultural considerations to designing a conflict resolution process would include paying attention to language issues among the participants, or asking how the participants interpret the conflict. Other questions of importance relate to how the parties typically communicate with one another, which parties ought to be involved, and what rituals, if any, might be observed. Recognizing and understanding the cultural implications of conflict is demonstrated by police departments that appoint officers of a particular cultural background, or have familiarity with a certain culture, to participate in dispute resolution in districts where they can use their knowledge. Through an understanding and association with that particular culture, these officers are usually better informed and may be more readily accepted as dispute referees, or interveners, when there is a need.

Involving the stakeholders in the design of any process is a good thing. In this way, the necessary information is shared, processes are defined, and misunderstandings can be cleared up. Self-design allows the parties to grow with the process and to learn about it as they participate. It also provides an opportunity for the participants to select the approach or process that will be used to tackle the problem and to determine the rules of participation that best apply to the situation at hand. In this way, the participants gain greater respect for the rules and will be more likely to apply them when resolving their differences. Involving the stakeholders might be achieved through training programs or planning committees at various stages of the process.

Interests, Rights, and Power

Costs generally include time and money, emotional energy, resources consumed or destroyed, destruction of a relationship, and opportunities lost. Benefits generally include the parties’ mutual satisfaction, perceived fairness, and meeting the interests of a party in a suitable manner.

Interests, rights, and power are the three basic elements of any dispute. In choosing one of the elements, the parties will be deciding whether to reconcile or to learn who is right or the most powerful. Deciding which approach to pursue will generate different costs and benefits. Costs generally include time and money, emotional energy, resources consumed or destroyed, destruction of a relationship, and opportunities lost. Benefits generally include the parties’ mutual satisfaction, perceived fairness, and meeting the interests of a party in a suitable manner.

Ury, Brett, and Goldberg (1988) believe there are six critical principles for dispute system design:
  1. Put the focus on interests;
  2. Build in “loop-backs” to negotiation;
  3. Provide low-cost rights and power back-ups;
  4. Build in consultation before, feedback after;
  5. Arrange procedures in a low-to-high cost sequence, and
  6. Provide the necessary motivation, skills and resources. (p. 415)

These authors believe that the focus on reconciling the parties’ interests, rather than a focus on power, is less costly and more rewarding for the participants. They suggest four major ways to accomplish this reconciliation: designing procedures, strengthening motivation, enhancing skills, and providing resources. They also encourage designing procedures that can be implemented early and that will provide for effective, perhaps even mandatory, negotiation to avoid build-ups that are more serious and difficult to resolve.

Ury, Brett, and Goldberg (1988) also suggest that procedures be built into any design that allows for disputants to turn back from a power-contest to a negotiation. The two-fold purpose of this “loop-back procedure” is to gain information about what the rights of the parties are, and to provide that information to the relevant groups or parties so they might then use it in negotiation. Another form of the loop-back procedure is advisory, non-binding arbitration, where information is used to encourage a negotiated resolution. Other methods could include mini-trials, cooling off periods, crisis negotiation, and intervention.

Our common and preferred ... - cartoon

Reprinted with permission from

Power-based resolutions are generally a low-cost method of resolving disputes. Examples include private adjudication (arbitration), med/arb, or final offer arbitration. Where a power contest is unavoidable, having low-cost procedures available—such as voting by affected employees, limited strikes, or rules of prudence—may limit the use of weapons or tactics.

Another alternative dispute resolution method is that of preventing unnecessary conflict and heading off future disputes by notification and post-dispute analysis and feedback. This process could involve collecting complaints or establishing forums to prevent future disputes.

Ury, Brett, and Goldberg’s (1988) concept allows for a sequence of procedures that permits the parties to move along a continuum. The sequence is designed to assist in resolving disputes early with interest-based methods, then moves through loop-back procedures, and, ultimately, allows for back-up measures, using the least costly methods first.

Alberta Justice provides a glossary of some of the distinct dispute resolution processes now available. This list is not exhaustive, but it reminds a designer of the variety of processes there are to choose from. Moreover, these process formats may be modified wherever appropriate to better meet the needs of the parties and the context of the dispute.

Another important element in designing a DR system is to consider the education and training needs of those who will be involved. Mary Rowe (2001) notes:

Various skills, including problem solving, direct negotiation, coaching, mediation, recognition and appreciation of diversity, and harassment prevention, need to be taught to participants. This training is a tool for cultural change and fosters individual responsibility and accountability at all levels. (p. 15).

People want different things, and having choices can empower people. Further, different methods are more suitable than others to specific types of complaints, and some methods that are costly are not necessary for all complaints.

Rowe (2001) also suggests that the process should provide a range of options for those involved, and that the design of the process needs to pay attention to this aspect. People want different things, and having choices can empower people. Further, different methods are more suitable than others to specific types of complaints, and some methods that are costly are not necessary for all complaints. Employees who feel they have had a choice in a dispute resolution matter tend to feel more satisfied with the outcome, which may in turn help the employer. Rowe also notes that any dispute resolution system must provide options for complaint handling within the wide range of real workplace disputes. Everyone within the organization should have recourse available to them. The dispute handlers should both represent and reflect the workforce in such areas as race, gender, culture, position, status, skill, experience, and so on.

Don't forget to include the ... - cartoon

Reprinted with permission from

Process design must also be, and appear to be, fair and consistent for everyone. Confidentiality and accountability must be balanced against the issues of public interest and access to information. Ongoing evaluation or supervision of the working model is necessary to ensure that it is meeting its objectives.


The first task in evaluation is to identify the characteristics of the process or system that is to be evaluated. This task can be difficult in ADR due to the wide variety of processes within the industry. Perhaps, as some have suggested, the entire dispute resolution system needs to be evaluated rather than just ADR processes.

With respect to ADR processes, evaluation must determine how well they are working. Evaluation must also take into account how well the courts are performing. When that step has been completed, a second evaluation must consider the criteria being used to determine the success or failure of the program. This step should include the objectives of the program: cost, time savings, satisfaction, systemic changes, and so on.

Baruch Bush (1989) suggests that determining what makes a “good” process can be aided by use of the six categories for determining quality, which were noted in the earlier discussion of process outcomes. These are: individual satisfaction, individual autonomy, social control, social justice, social solidarity, and personal transformation. Individual satisfaction can be interpreted at least three ways: great pleasure, individual autonomy, or preserving the status quo. Social justice might be interpreted as the value of reducing suffering wherever it exists, or preserving the autonomy and dignity of all individuals. Given the potential for different interpretations, any evaluation would have to be considered within the definition and understanding of outcomes that was provided from the start of a new program or process.

Any evaluation of settlements depends on the context of what the parties valued at the outset.

Galanter and Cahill (1994) use parallel reasoning when looking at settlements. Any evaluation of settlements depends on the context of what the parties valued at the outset. If a party was partial to cost savings, then a settlement that resulted in money being saved would be perceived as successful. If the party was looking for an order that directed deterrence, then the settlement may not provide the satisfaction that was being sought. Evaluation requires comparison to an alternative, which may produce skewed results if it compares significantly different alternatives. Typically, an adjudication process is the “control” measure, making ADR the “experiment” that is being compared to it. Since the characteristics of adjudication and ADR differ, the results will also differ.

Rolph and Moller (1995) advocate using a true control group where the evaluation is to provide information on the effects of the program. Everything else must remain constant. Using this method, cases are assigned randomly to each dispute resolution environment that is being compared. These environments must operate simultaneously and smoothly, and they must be able to receive randomly assigned disputes. When these conditions are met, the data measurements will provide the truest picture of any differences in effect between the alternatives.

A non-equivalent control group compares two similar groups that are not randomly assigned. This design depends on finding similar populations of cases, gathering information from both groups on characteristics that may affect their performance, and measuring the outcomes and dispute characteristics of interest for both groups in a standardized way.

The objectives of the evaluation will determine the methods to be used. Quantitative analysis will assist in determining costs or saving money. Where qualitative methods—such as interviews or open-ended questions on surveys—are used, then the information can demonstrate consequences and relationships that may have been unintended, but nevertheless welcome.

Other considerations to be noted when evaluating include causality and biases in the selection of the participants. Causality involves the relationship between the new system and the desired outcomes. The evaluator must be able to screen out factors outside the evaluated system that might cause or bring about the same result, such as voluntary participants whose situations might have improved in any case, with or without the services of the program.

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes

Study Questions

When you have completed the assigned readings for Unit 11, test your understanding of the material by answering the following study questions. You may find it helpful to write out your answers. If you have difficulty with a question, reread the relevant material. If you still cannot answer a question, contact your tutor for assistance.

  1. What are the four main steps of systems design?
  2. Identify and discuss five factors to be considered in assessing intercultural disputes.
  3. What is a loop-back? What does it do? What are the benefits?
  4. How does Client-Centered Design address issues of organizational culture and diversity of individuals in the organization?
  5. Why are interests, rights, and power important in designing a dispute resolution process?
  6. Examine the concerns in evaluating these systems.

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes

Practice Exercises

Test your learning with the following practice exercises. No credit is attached to these exercises, but they will help you determine whether you need to re-study any part of the unit before proceeding.

  1. Examine a policy against discrimination or sexual harassment (or both) in your work or academic setting. Does this policy meet the requirements of client-centered design? Give reasons for your answer.
  2. What important cultural factors would you consider in designing a dispute resolution system in an Aboriginal community?

Self-Study Quiz for Unit 11

Before continuing on to Unit 12, please conduct the self-study quiz for Unit 11.

Study Guide

Unit 11 – Designing and Evaluating Systems and Processes

References and Optional Readings

Baruch Bush, Robert A. (1989). Defining quality in dispute resolution: Taxonomies and anti-taxonomies of quality arguments. Denver University Law Review, 66, 335–380.

Chornenki, G. A., & Hart, C. (2001). Bypass court (2nd ed.). Toronto: Butterworths.

Galanter, Marc, & Cahill, Mia. (1994). “Most cases settle”: Judicial promotion and regulation of settlements. Stanford Law Review, 46(6), 1339–1391.

Greenberg, Elayne E. (2015). Fitting the forum to the pernicious fuss: A dispute resolution design to address implicit bias and ‘isms in the workplace. St. John’s School of Law Legal Studies Research Paper No. 15–0023.

Kelman, Herbert C. (2005). Building trust among enemies: The central challenge for international conflict resolution. International Journal of Intercultural Relations, 29, 639–650.

Lynch, Jennifer F. (2001). Beyond ADR: A systems approach to conflict management. Negotiation Journal, 17, 206–216.

Menkel-Meadow, Carrie. (2009). Are there systemic ethics issues in dispute system design? And what we should [not] do about it: Lessons from international and domestic fronts. Harvard Negotiation Law Review, 14, 195–231.

Rabinovich-Einy, Orna, & Katsh, Ethan. (2012). Technology and the future of dispute systems design. Harvard Negotiation Law Review, 17, 151–199.

Reuben, Richard C. (2005). Democracy and dispute resolution: Systems design and the new workplace. Harvard Negotiation Law Review, 11, 1–342.

Rolph, Elizabeth, & Moller, Erik. (1995). Evaluating agency alternative dispute resolution programs: A users’ guide to data collection and use. Santa Monica, CA: Rand Corporation.

Rowe, Mary. (2001). Designing integrated conflict management systems: Guidelines for practitioners and decision makers in organizations. Society of Professionals in Dispute Resolution.

Silbey, Susan S., & Merry, Sally Engle. (1984). What do plaintiffs want? Reexamining the concept of dispute. Justice System Journal, 9, 151–178.

Ury, William L., Brett, Jeanne M., & Goldberg, Stephen B. (1988). Designing an effective dispute resolution system. Negotiation Journal, 4, 413–431.

United States Postal Service. “Redress.”

Various. (2009). Dispute System Design Issue of Harvard Negotiation Law Review, 14, 1–342.