Courts can learn from the experience of Orange County Superior Court, which recently developed and implemented EVA, an AI-powered chatbot that provides employees with quick access to court procedures, policies, and forms, ultimately improving efficiency and support
In recent years, the California Superior Court of Orange County experienced high vacancy rates and turnover in staff resulting in the loss of expertise and knowledge, in part due to retirements. Knowing that the current situation was unsustainable, the Court took an innovative approach to solving this challenge by pursuing automated solutions involving AI, according to Darren Dang, Chief Financial & Administrative Officer at the Orange County Superior Court.
One of the main achievements in the Superior Court’s pursuit of innovation was its creation of EVA (Employee Virtual Assistant), an AI-powered chatbot used internally by the Court. The tool helps staff, especially those with limited experience, to quickly and accurately access information about court procedures, policies, and forms that are essential to perform their duties. It also provides quick access across various case types and departments; for example, civil case processing alone has 150 procedures.
To use the chatbot, staff members just type in questions in their natural language. EVA then searches a large database of documents, such as court procedures and policies, and returns a concise summary of relevant information to the user, including a link to the source document so that users can conduct further research easily. The link to the reference material is not only a great resource to validate and verify the accuracy of the tool, but it’s also an excellent mechanism to build trust and comfort with it, thereby increasing adoption and utilization. EVA is currently being used by staff in several different court divisions.
Although EVA is a work in progress, the Orange County team expects this tool to reduce training time, improve procedural accuracy, and serve as the centerpiece in creating a more supportive working environment for staff and enhancing timely service to the public.
The journey to developing and implementing EVA
The development of EVA began with the creation of a knowledge base, comprised by uploading more than 150 civil procedures and automatically indexing them. One of the initial challenges faced by the EVA development team was managing the multiple contexts within these procedural documents in which terms like complaint could have varied meanings depending on the situation.
To overcome this, the team shifted their tactics by developing hyper-specific bots tailored to individual departments and case types. They also used advanced embedding techniques (AET), which is a special way to help computers understand what words mean by giving each word a special set of numbers that are like a secret code that only the computer can understand. This tactic breaks down the knowledge into contextually relevant pieces, and the combination of using tailored bots and AET can go a long way to improve accuracy.
Another challenge during the development of EVA was outdated procedures within the knowledge base, leading to inaccurate or irrelevant responses. To remove this barrier, the team implemented procedural reviews.
In the testing phase, subject matter experts refined EVA’s responses and identified outdated or inconsistent procedures for revision. This collaborative effort with the operations training teams led to significant improvements in the chatbot’s accuracy.
After fine tuning EVA’s capabilities, a soft launch was conducted with a select group of 15 users, who provided valuable feedback to identify and fix system bugs. This iterative feedback process ensured that EVA was robust and ready for full deployment. Additionally, user engagement was crucial in improving EVA’s functionality; and iterative testing and feedback sessions with end-users, such as legal processing specialists, enhanced the chatbot’s effectiveness.
During the implementation phase, comprehensive training materials, including videos and reference guides, were developed. In-person demos also were conducted to facilitate a seamless transition for all users. These resources ensured staff were well-equipped to integrate EVA into their workflows effectively.
Lessons learned and guidance for courts
The development and implementation of EVA yielded valuable lessons that will guide future innovations and improvements in court operations. Ryan Gee, Strategic Innovations Group Operations Analyst, and Bryan Hernandez, Strategic Innovations Group Data Analyst, both at Orange County, California Superior Court, described some of these lessons:
-
-
- First, direct engagement with end-users during the development process is essential. They provide critical insights into their needs and ensure the solution meets their requirements.
- Second, consolidating and updating procedures was a time-consuming effort, but it had a significant impact on the accuracy and effectiveness of EVA.
- Finally, tailoring AI solutions to specific departments and case types ensured the solution meets the unique needs and improves the workflows of each department.
-
To help successfully navigate the implementation of this transformative technology further, Hernandez and Gee offered ways that other courts can get started, based on their firsthand experience:
-
-
- Engage stakeholders early — Court tech teams should involve key stakeholders from the beginning to ensure their buy-in and support throughout the implementation process. This also helps in aligning the tool with the needs and expectations of its users.
- Start with low-risk proof of concepts — Teams should begin with small-scale, low-risk proof of concepts to test and familiarize themselves with the technology. Then, use these initial trials to gather feedback and make iterative improvements.
- Future-proof solutions — Teams should design the tool with flexibility in mind to accommodate future advancements, including making the solution easily upgradable and adaptable to new developments, such as integration with the latest large language models.
-
By leveraging AI to streamline access to critical information, EVA aids in mitigating the effects of limited training due to high turnover and demonstrates the Court’s commitment to embracing technology as a means to enhance operational efficiency and staff support, said Nora Sanchez, Chief Operating Officer at the Orange County Superior Court. And as courts continue to face evolving challenges, solutions like EVA showcase the potential of AI to transform traditional systems, upskill staff, and improve the overall administration of justice.
You can find out more about the use of chatbots to improve operationals efficiency here
