Measuring Legal Service Value, Part 2

Slaw.ca Legal Ethics Column, Tuesday June 1, 2018.

Online: http://www.slaw.ca/2018/06/01/measuring-legal-service-value-part-2/.

What makes a great law firm? How can one quantify just how great a firm is, and compare it to its competitors? Last time in this space I suggested that legal service value has four elements (full paper here):

  • To the extent that a firm gets good legal results for its clients, it has effectiveness value.
  • To the extent that the firm’s fees are low and easy to pay, it has affordability value.
  • The more the firm’s practices minimize clients’ time and stress costs, the more client experience value it has.
  • Finally, if the firm’s work has many benefits and few costs for people other than its clients, it has high third party value.

In principle, a firm’s performance on these four elements of value can be quantified. If we could actually create accurate charts like these for legal services providers, we would have more empowered consumers, better self-awareness of strengths and weaknesses within law firms, and more evidence-based regulation. This project can also move us toward a more meritocratic legal profession, in which individual success is less dependent on racial and socioeconomic privilege.

An accurate quantification of value will require multiple metrics, and the array of metrics will depend on the legal niche in question. This column outlines three basic types of metric for quantifying legal service value: output metrics, internal metrics, and input metrics. Output metrics measure what comes out of the legal service provider. Internal metrics examine what happens inside the firm. Finally, input metrics analyze the people who walk in (or log in) to the firm each morning.

Output Metrics

Output metrics may analyze the outcomes created by law firms, or their work products. A simple example is win/loss rate, which can be informative in some administrative litigation niches if properly designed. Judges in England & Wales are now evaluating the advocacy of all lawyers who appear before them in criminal matters.

The best proof of a pudding is in the eating. Output metrics are, in principle, the bestway to evaluate legal service value. If we can tell from its outputs that a firm consistently produces great, highly affordable results for its clients, while leaving them happy and doing good in the world, we can be confident that it is a great law firm.

Unfortunately, output metrics for legal services are methodologically troublesome. They must be valid — they must measure what we are actually interested in instead of something else. Measuring affordability value by comparing the average bills of three law firms over the last 10 cases handled by each would not work, because any differences would be just as likely to reflect the complexity of the cases they handled. Especially in contested matters, the outcome depends on a wide range of factors other than the inherent value of the legal service provided.

One egregious example of an invalid output metric is the personal injury “Litigator Awards” handed out by the “Trial Lawyers’ Board of Regents” in the United States. This metric is based on whether or not a firm has settled cases for more than certain amounts within a set period. However no account is taken of the inherent strength of the cases in which these results were obtained. Thus, receiving a “Litigator Award” may well speak more to the prowess of a personal injury firm in attracting high-value cases than it does to the effectiveness of its advocacy for those clients.

Even if valid, an output metric may still lack reliability if the sample size is too small. For example, surveying past clients can be a good way to assess client experience value. However, because clients have idiosyncratic expectations of their lawyers, and some client evaluations are unreasonable or biased, the results cannot be relied upon unless a sufficient number of clients were surveyed.

Internal Metrics

Looking at what happens inside law firms is an an alternative to scrutinizing outputs in the effort to identify value. Most consumers aren’t interested in what goes on inside — they care about the outputs. However certain practices and structures demonstrably increase the chance that high-value services will be delivered. Checking for these practices and structures can be a helpful way to identify the firms likely to consistently produce great value, especially when output measures are methodologically impossible.

Some internal metrics focus on processes — what lawyers, and other people involved in legal service provision, actually do when working for a client. Common sense practices such as actively listening to clients and giving them a single point of contact within the firm have been empirically demonstrated to produce client satisfaction.

Structural attributes can also be the basis for good internal metrics. A firm that is free from harassment, racism, sexism, conflict, and excessive turnover is likely to produce better value.

The methodological challenge of internal metrics is that the information is private. Self-reporting is one way to get at this data. Peer file review and audit are somewhat more forceful techniques sometimes used by regulators and legal aid funders.

Input Metrics

How many years of experience does the lawyer providing the service have? What were her bar exam scores? How many practice hours have the firm’s staff accumulated in the niche? These are examples of input metrics, focused on the attributes of the people who do the work.

The advantage of such input metrics is that the data are relatively easy to gather and compare. The disadvantage is that this data seems to have a weak relationship at best with meaningful elements of value. Potentially more fruitful are surveys identifying the knowledge or skills that are most important for other lawyers to possess. If a variety of substantive knowledge (e.g. of the Criminal Code) or a personal skill (e.g. persuasive writing) demonstrably helps practitioners provide high quality services in a certain niche. If we have data about the extent to which different practitioners have these attributes, that can help us quantify and compare the value they offer.

A Steep Path, Worth Climbing

The value of a legal service is a complex aggregate of its effectiveness, affordability, client experience, and third party effects. Measuring it accurately requires gathering data from multiple sources, using a variety of methodologies. A “tyranny of metrics” that are misleading or counterproductive might be even worse than the status quo, in which the true relative value offered by different firms is mysterious, even to insiders.

However the author’s view is that it is entirely possible – and entirely worth the effort — to create accurate, objective methods to quantify the value propositions of different firms in different legal niches.

For a more detailed account, please see https://ssrn.com/abstract=3144771

Measuring Legal Service Value, Part 1

Slaw.ca Legal Ethics Column, Tuesday April 5, 2018.

Online: http://www.slaw.ca/2018/04/05/measuring-legal-service-value-part-1/.

If you work at a law firm, how good is that firm? If you’re a client or potential client, how good are the different legal services providers that you might choose to patronize?

It’s too difficult, at present, to answer these questions in an objective and reliable way. This is most obviously true for individual people with legal needs. They generally confront a mysterious landscape populated with apparently indistinguishable law firms, as well as proliferating alternative sources of legal services.

However, even experienced corporate clients, and lawyers themselves, lack solid information about the respective merits of different legal service providers. To evaluate quality, they must often rely on opaque, methodologically suspect rankings or else anecdotal impressions of firms.

We need better ways to quantify and compare the value propositions offered by different providers of legal services. This column proposes a definition of legal service value, and next time in this space I’ll propose and categorize metrics for quantifying it. A detailed paper on these topics is available here.

Elements of Legal Service Value

There are four types of value offered by a law firm:

  1. Effectiveness Value means accomplishing clients’ legal goals and protecting clients’ legal interests. In litigation, this can mean maximizing (or minimizing) the award or settlement, or minimizing custodial sentence in a criminal defence matter. In uncontested matters, effectiveness means accomplishing the client’s legal goals, and minimizing all associated risks, to the greatest extent permitted by law and the client’s instructions.
  2. Affordability Value: More affordable legal services cost less. The way a firm structures its prices— for example by offering price certainty or deferred billing — also affects the affordability value the firm offers.
  3. Client Experience Value: Legal services are more valuable when they have a more favourable impact on the client’s time and the client’s mental state. Timeliness, communication, and understanding of the client’s life and/or business are valuable in and of themselves. If two firms are equally effective, and equally affordable, the firm that creates a better experience for its clients offers better value.
  4. Third Party Value: In addition to the client, there are various others who can be affected positively or negatively by a legal service and the way it is performed. Firms that create access to justice through pro bono work, or contribute to social goals like diversity and the rule of law, offer better value than those that do not.

The aggregate value offered by a service provider is represented by the space occupied inside this figure. The greater the area, the greater the total value offered by that firm.

The Value Profiles of by Different Firms

Different providers of a certain legal service (e.g. an initial public offering, or representation in an employment law dispute) have strengths and weaknesses in different elements of value. Quantifying elements of value separately, as this model seeks to do, allows consumers to make well-informed decisions, based on differing individual sensitivity to price and different aspects of value. Unlike rankings, this method can help consumers who are hunting for bargains, or those who are interested in results above all else.

The “All About Results” firm is the best of the four at getting the job done (high effectiveness value). The “No Frills” firm offers great affordability but modest scores on the other three elements of value. The third firm is “Service First” – its effectiveness and affordability are mediocre but it offers an outstanding client experience. Finally, the “Model Citizen” firm offers wonderful contributions to social goals backed by solid, if unspectacular results on the other three elements of value.

How to Quantify Value? 

This framework is of little use unless it can be filled in with real numbers. What data would quantify the extent to which competing legal service providers provide different elements of value? Next time in this space, I will consider output metrics, internal metrics, and input metrics – practical ways to measure and compare the value offerings of different legal services providers.

This blog is based on a working paper entitled “Measuring Legal Service Value,” which is available at http://ssrn.com/abstract=3144771.

Shady Billing: Closing the Hall of Shame

Slaw.ca Legal Ethics Column, Jan. 30 2018

Shady Billing: Closing the Hall of Shame

Only “fair and reasonable” fees and disbursements can be charged by lawyers to their clients. This rule is uncontroversial, and applies across the country. Nevertheless, the following billing practices are used by some Canadian firms, and not clearly forbidden by regulation:[1]

Continue reading Shady Billing: Closing the Hall of Shame

Access to Justice Levies for Lawyers: Putting Our Money Where Our Mouths Are

Slaw.ca Legal Ethics Column, Dec. 12 2017

http://www.slaw.ca/2017/12/12/access-to-justice-levies-for-lawyers-putting-our-money-where-our-mouths-are/

Tyrell Moodie, accused of drug offences and facing several years in prison, was denied a Legal Aid Ontario certificate because his income of $16,211 per year exceeded the cut-off threshold. Legal aid services for refugees in B.C. and Ontario were threatened with drastic cuts in 2017. Self-represented litigants are now the majority in many family courts, mostly because people cannot afford the legal assistance that they would love to have, and legal aid won’t pay for it.

Every media story about a legal aid shortfall includes a quote from a lawyer, pointing the finger at the government for inadequate funding. However, every time the legal profession points its finger at the state, three fingers are pointing back at the legal profession. As trustees and beneficiaries of the legal system, lawyers should make a more tangible contribution to ensuring its accessibility.

Continue reading Access to Justice Levies for Lawyers: Putting Our Money Where Our Mouths Are

Generalism and Access to Justice: Jack of All Trades, Master of None?

Slaw.ca Legal Ethics Column, October 6 2017

Generalism and Access to Justice: Jack of All Trades, Master of None?

The rise of specialization is among the biggest changes in the practice of law over the past hundred years. Most lawyers and paralegals are increasingly able to focus on a smaller number of legal niches. That is good news, for practitioners and also for clients. However, I will suggest here that generalist legal professionalism has an enduring role in fostering access to justice.

Continue reading Generalism and Access to Justice: Jack of All Trades, Master of None?

Personal Plight: Mending the Market

Slaw.ca Legal Ethics Column, Aug. 11 2017

Personal Plight: Mending the Market

“Personal plight” legal services are those provided to individual clients whose legal needs arise from disputes. Personal plight areas such as family law, refugee law, and human rights are the site of Canada’s worst access to justice problems.

The market for personal plight legal services functions poorly, as Malcolm Mercer and Amy Salyzyn have shown in this space. A key problem, I suggest here, is that it is too difficult for consumers to shop intelligently. This undermines healthy competition and legal professionalism, in addition to access to justice. Regulators can and should mend the market for personal legal services.

Continue reading Personal Plight: Mending the Market