The 2022 ILTA Tech Survey found that nearly two-thirds of law firm IT departments reported using no metrics to measure their success. But there are many ways to start collecting that data
The data explosion within law firms has seen the transformation of both external functions like litigation and contracting, as well as internal functions like financial and document management. Technology-enabled services have let legal teams not only better serve their clients, but collect metrics to determine what’s working and what’s not as well.
However, there remains one area within law firms that has seemed more reluctant to embrace metrics to measure department success, and it’s perhaps a surprising one: IT departments themselves.
The International Legal Technology Association’s (ILTA) 2022 Tech Survey revealed that 63% of surveyed law firm IT departments reported using “none/non-applicable” metrics to measure IT department performance. That figure was lower, but only slightly, from 2021’s 67%. The most popular available metrics used were downtime per month (37% of surveyed firms), Help Desk closed ticket feedback (36%), and Help Desk ticket resolution times (32%).
The ILTA report did note that “[t]he larger the firm, the more likely they are to actually poll users on satisfaction with regard to the performance of the IT department.” Regardless of size, however, the fact that nearly two-thirds of firms weren’t using any metrics to measure IT department performance doesn’t surprise Kenneth Jones, chief technologist at law firm Tanenbaum Keale and ILTA committee member.
Jones notes that in the corporate world, technology projects tend to be tied directly to return-on-investment, with collecting metrics baked directly into employment. At his previous work at pharmaceutical giant Bristol Myers-Squibb, this attitude was prevalent, Jones says. “One of the measurements was, try not to put 40% of your time into 1% of your business,” he adds. “Conversely, in the legal field, I think sometimes… whoever screams the loudest tends to get the most attention, and it may not be the project that’s delivering the most revenue or profit to a firm.”
Indeed, in the ILTA survey, less than 30% of surveyed firms reported measuring law firm success via IT capital costs per lawyer (29%), IT operating costs per capital lawyer (25%), or IT spending as a percentage of revenue (22%). Jones views this as a missed opportunity. The goal of any IT department, he says, should be to make sure as many dollars as possible are actively being utilized.
“How many of us have Hulu accounts or Netflix accounts, or this subscription or that subscription, and we don’t use it anymore?” he asks. “And so, that’s probably a useful metric also, just to be sure that as many dollars as possible are truly being allocated productively to business.”
For those who haven’t built out metrics programs for the IT department, Jones says that “a good first step is having the data collection and measurement in place.” He separates the data needed for this type of program into two buckets: i) what technologies are actually being used and how effectively; and ii) where spend is going in an organization. The latter, he adds, can be tricky — spend shouldn’t just include official IT department projects, but also what individual attorneys are purchasing for themselves to augment their daily work (such as subscription licenses, discovery costs, technology consultants, and more).
Building out the metrics program
Consultant and legal survey expert Rees Morrison says that to actually begin collecting IT department data, a firm’s options can be broken down into four different buckets. The first is rather simple: focus groups. “It’s a way of not only collecting some numbers, but also subjective views and opinions,” Morrison says. “And so a focus group, it’s a flexible tool, but it’s also somewhat expensive and you have to have people who know how to run them.”
Morrison’s second choice is perhaps the way most firms operate today: short email surveys, which he called a “crude but simple way of collecting some data.” The most effective email surveys are short — at about two or three questions — in order to encourage a higher response rate. However, he adds, these are tough to do en masse, as IT department leaders would also then be responsible for collecting, sorting, and displaying the data.
His third option is one that is perhaps more underutilized for law firm IT departments: formalized surveys. In fact, the ILTA survey found that just 16% of firms utilized user-satisfaction surveys to measure success, and 20% incorporated user feedback in general. While surveys may require more work on the front end, they also reveal greater insights on the back end, Morrison explains. “You can ask more questions. It’s easier for people to answer, because a lot of them, it’s ‘check a box’ or ‘fill in a number’ as compared to an email, where it’s a little less formatted,” he says. “And the huge advantage is, it then brings everything together, often in a spreadsheet so you can analyze it how you want.”
Finally, IT departments can utilize what Morrison calls exhaust data: logs of how often a product is being used. This type of data can provide insights that the other types can’t, he says, because it’s agnostic of attorney or legal professional biases. “They’re not thinking probably about how much time they’re online doing that or how many times they do it,” he adds. “But somebody else can start drawing some patterns or making decisions whether a subscription is worthwhile or not, or get more of the sorts of ancillary data that flows from it.”
Morrison notes that collecting these sorts of metrics can have multiple positive effects. There is, of course, the increased insight into firm technology usage and better ROI that comes from increased efficiency. but there is also positive firm engagement that can come from IT metrics projects, he says.
“I do believe that folks in general like their opinion being asked,” he explains. “Otherwise they feel they’re just serfs and nobody cares — I don’t happen to like this contract management system, but what can I say? But if they come and ask you on a scale of 1-to-10, how do you like it and a few questions related to that, I get a voice.”
The work does not stop here, however. Law firm IT metrics are not a “set it and forget it” proposition. Law firms looking to collect this data also need to plan how they’re going to incorporate feedback around the data they’ve collected — and make sure those they’ve collected data from are heard, Morrison notes.
“If you gather metrics from people, I think you owe it to get back to them something,” he says. “It shouldn’t go into the black box of the firm and the associates never find out what happened. You give me some time and your thoughts and estimates and numbers, and I’m going to return with some findings and describe what we’re going to do about it.”
