Editor’s Note: The views expressed in this article and the panel discussion it references are the panelists’ own and not representative of their respective organizations.
As more organizations invest in in-house e-discovery programs, understanding how to track and measure performance effectively is critical for optimizing workflows, identifying areas for improvement, and demonstrating internal value.
At Relativity Fest 2023, a star-studded panel of in-house experts shared with attendees their insights on the metrics corporate legal teams are using, and how to use those metrics to drive better outcomes.
These folks had some deep insights to share on which metrics matter, how to build (or rebuild) a data-driven in-house e-discovery practice, and why measurement is absolutely essential to getting this discipline right. Our panelists included:
- Adam Rouse, Director and Sr. Counsel, e-Discovery Operations, Walgreens
- Briordy Meyers, Discovery Counsel, Google
- Michallynn Demiter, e-Discovery Expert, Bayer (moderator)
- Whitney Fogerty, Managing Director, Legal, Federal Express Corporation
- Phyllis Strader, Manager Legal Technology and Electronic Discovery Operations, Constellation Energy
Read on for some of the most impactful takeaways from their conversation.
Tip #1: You have to start somewhere—and justifying your team’s existence is as good a place as any.
Though legal departments have traditionally been seen as cost centers, the advent of modern legal technologies has helped justify in-house e-discovery work as an opportunity for efficiency and cost effectiveness, and an overall value-add for today’s biggest and most litigious industries.
But it does take some investment, and many executives may be hesitant to approve initiatives to build out these teams without a clear picture of how those investments will pay off in the near term.
Whitney Fogerty, managing director in legal at FedEx, told Relativity Fest attendees that she’s walked this path and they can, too: “When we moved in-house five or six years ago, we used metrics to justify the ROI on tech, additional headcount, and so on, to get the program up and running.”
Having run the numbers, she said that those numbers continue to show leadership how her department is helping the broader organization over time.
“Now we use the numbers to measure our path toward our goals: decrease costs, increase efficiency, increase consistency. The more repeatable a process is, the more defensible it is.”
Briordy Meyers, discovery counsel at Google, recognized that initially intimidating feeling—and how good measurement can empower a team to push forward.
“It can be tough when you’re starting out, but the approach I would suggest is looking at what you have from the ground up and top-down. Develop a better sense of what you care about as an organization, and use those metrics to create KPIs,” he advised Fest attendees. “It’s tough to build that, but it helps.”
He had some helpful practical advice on where to begin, too:
Start with legal billing platforms analyzing invoices from vendors and outside counsel, pulling sources into a single dashboard—covering review platforms, processing engines, legal hold tools, and all of that. Do some data mapping to figure out what metrics you have and cultivate the ones you care about, then drive those basic metrics into combinations and pivots that can become valuable KPIs. Normalizing information is a lot of the work; the same basic metrics across different platforms might be reported differently, so standardizing those is helpful. And working with partners to get on same page in how they deliver invoices or reports to you—seeing if they’re able to format accordingly and deliver on the cadence you need.
Essentially, getting your ducks in a row here can help make the path forward clearer. And doing this work lays the foundation for ongoing KPI measurement to help your team succeed well into the future.
Tip #2: Lean on your partners (and peers) for advice.
All of that data-gathering and number-normalizing sounds intimidating, but our panelists repeatedly emphasized that no in-house professional is in this alone. There’s help to be found, whether it’s in your formal partnerships or among peers in your broader network.
For instance, as Briordy mentioned, vendors are often very willing to adapt the metrics and reports they deliver to your needs and advise on which numbers stand out most. Software vendors can also provide a lot of great advice and support here—Relativity included, of course.
“Customer success at Relativity gives me metrics every month to make it easy to understand what’s happening in our environment and take action on it,” Michallynn Demiter, e-discovery expert at Bayer, shared. “That’s a really good place to start.”
For those using RelativityOne, Adam Rouse—director and senior counsel in e-discovery operations at Walgreens—had some specific tools to recommend.
“One of the nice things we’ve found using RelativityOne are the built-in tools, like Cost Explorer. The other place we’ve found particularly helpful is the script library. There are several scripts you can run on a workspace or instance level to give insight into things like database sizes, billing sizes, looking across matters or workspaces on a granular level,” he explained to a packed crowd in the Fest session.
Plus, as with so many things in the e-discovery community, he emphasized the value of networking and simply striking up a conversation with your peers to see what you can learn from those in the field along with you.
“Phone a friend works really well. It’s the easiest way to get started. e-Discovery, legal ops—we’re a very collaborative part of a very adversarial system,” Adam said. “We’re all willing to talk to each other; we see the same people at conferences, and the community is strong. So pull someone aside, show them what you’re considering, and ask them for advice. More often than not, you’ll get someone willing to collaborate with you.”
Tip #3: Always, always keep proportionality in mind.
One way good measurement of legal metrics that matter can help teams prevent excess spend and save company resources is by bolstering proportionality arguments. Legal teams can make stronger arguments around the scale and scope of discovery during a given matter when armed with the data to back up their claims on costs and timelines.
“We use measurement … when running technology-assisted review or email threading to have data that shows if terms were extreme, if analytics are effective, and so on,” Phyllis Strader, manager of legal technology and electronic discovery operations at Constellation Energy, said during the panel. “That way, we can go in and say: ‘If you’re requesting that we pull this many custodians and run them against this date range, we need to utilize something like TAR or threading to make those make sense.’”
Adam agreed.
“By plugging in these metrics, we can be self-enabled to make a proportionality argument: dig in, get a nice-looking report, arm outside counsel with the best information we have to take into court, and help get terms reduced, custodians dropped, or whatever we need to contain proportionality,” he said.
There are lots of different ways to approach this, including benchmarking across workspaces, matters, EDRM stages, and more. Bringing this sort of data to a legal argument will bolster your perspectives with real-world facts, and improve your ability to defend your organization from unnecessarily bloated discovery projects.
“e-Discovery is one of the few areas of the law I can think of where numbers can help you make beneficial legal arguments, whether at a meet and confer to justify your position to the other side, or when you want outside counsel to be prepared to make scoping arguments, or making a motion for protective order,” Briordy observed. “Judges want more evidence; we can’t just say ‘this is hard to measure.’”
Tip #4: Measurement is best done at a regular cadence, at a reasonable scale.
Once you have your preferred numbers in mind, revisitng them regularly—not just throughout each matter, but over time at a department-level—can help you continue to drive home the value your team contributes to your organization.
It can also help you spotlight opportunities for improvement. As we all know, the field of e-discovery is always evolving. Just as data types, sources, and volumes change, so do the legal technologies teams use to confront each project. And your team needs to stay abreast of the challenges that plague your own processes in order to see how those technological innovations might help you move the needle year over year.
“There are metrics we look at monthly to determine whether we’re meeting our goals. Everything from where our data is in Relativity, to which platform we’re utilizing most each month, how frequently we move data between platforms, and usage stats like new users,” Whitney explained at Fest.
“I also look at metrics that help me measure our efficiency. We have set goals for each step in EDRM, so if, for example, our collection goal is 15 days, we want IT to collect data within 15 days of when legal team submits a ticket,” she continued. “If we’re running longer than that, we look at metrics for why—custodian issues, complicated mobile collection issues—then use that data to justify investment in different collection tools.”
Tip #5: Believe it or not, numbers can tell pretty compelling stories.
Many attorneys got into this profession because they live for storytelling: the kind of storytelling that wins cases and helps right wrongs. So, for many of us in this field, all this talk of numbers is a little … dull? Deflating? Something like that.
But each of our panelists at Fest last year had emphasized that numbers aren’t wonderful for their own sake; they’re wonderful for the stories they help legal teams tell to their peers, outside counsel, and judges.
“It’s interesting how we tend to measure the same things year over year: cost, custodial compliance with legal holds, growth of data sources and volumes. If you’re just starting out or reorganizing an e-discovery program, it’s overwhelming—so we start off rudimentary, and where is that but cost? Justify your existence, calculate ROIs on tools,” Adam said. But that’s just the beginning.
“It starts as crude proof that these programs need to exist. Once you get past that, and have told that story, the metrics themselves don’t change much—but how we use them gets more elegant. Figure out what metrics work for your team in your industry in your particular situation,” he suggested to Fest attendees.
What does that look like?
“Taylor what you measure—or how you present it—to whatever goalsetting you’d like to do. Is it budget? Career growth? Legal arguments? The numbers are there. The data will support you. So the nuanced ability to turn that data into a story is the skill you want to work on,” he explained. “Use those metrics to weave stories and take those to the business and become a more integral process to the business. Lift and heighten e-discovery and legal across the organization and build relationships, increase your budget and visibility, and increase compliance. I don’t want a call from legal to instill dread; I want to build those relationships.”
That sounds like a win-win to me.
Briordy shared some similar thoughts on the role measurement plays in the successful practice of law—and e-discovery.
“Never forget the context in which e-discovery happens. It’s procedural; it’s workflow-driven. Ultimately, you have to be able to connect what you’re doing and how with your legal arguments. I don’t know how you do it if you’re not measuring what you do,” he said. “I don’t know how to have good relationships with vendors and outside counsel if you’re not doing this; it’s better for them to have you articulate your needs, it’s better for you to articulate how you’re saving money and managing risk and being defensible.”
And he went on to share my favorite insight from the session: “Do your best with what you have, and you’ll start to see the storylines pop up. This is partly what makes e-discovery fun: how to draw meaning from patterns to help manage risk in the courtroom.”
Sam Bock is a member of the marketing team at Relativity, and serves as editor of The Relativity Blog.
