Document analysis 

Documents are a complex web of references (explicit and implicit), expansions of ideas, and resolutions of conflicting ideas. Some lines of reasoning are grounded in precedent, while others are proposed or suggested. Humans give significant clues to the big ideas and their purposes through the structure and layout of a document. As a liminal task, document analysis tools have to accurately extract those structural clues. This includes capturing such information as document titles, sections and subsections and their headings, as well as paragraph and sentence boundaries. It also includes elements of document zoning (determining the purpose of each section of a document) which is often domain dependent.

A starting principle is that a document analysis product must be agnostic to the source of a document and the formatting choices that the author may choose. However, formatting conventions exhibit wide variance, and the link between structure and content cannot be ignored. For example, are the sections of a document compact and mostly independent, or are there a couple of long, hierarchical sections? Finding segmentation algorithms that are aware of both structure and content is an area of active research.

Entity and relation extraction is another well-known NLP task that is used extensively in document analysis products. While off-the-shelf approaches to entity extraction are generally adequate, relation extraction is highly specialized in tax, legal, and regulatory domains. Relevant relations are generally more fine-grained than the classical NLP factoid relations (e.g., residence, employer, or date-of-birth), and as such are easily confused with other similar relations, or are subject to variation in language if a high-level taxonomy is used. Relation extraction would be critical if one, for example, might wish to extract the claims between each party, assuming there are many parties, in a legal action.

While the traditional field of entity extraction attempts to find people, locations, organizations, and geopolitical entities in text, a machine may have to detect and localize occurrences of named abstract ideas in text. For example, legal principles often come with tests to distinguish acceptable and unacceptable activity; one such principle is equal protection under the law, a constitutional right. The courts have found that a law will be invalid if it defines a class of individuals without any connection to the matter being regulated; a law prohibiting red-headed people from driving between the hours of 1 am and 3 am would be invalid under this principle. One might wish to detect if some paragraph of a document is discussing the legal principle of irrational classification.

A common approach to this problem would be to create a taxonomy, to collect and annotate instances of each node of the taxonomy, and finally to develop models. The first two steps are considerable investments of time and personnel, even before the beneficial task of model development can begin. Establishing an area of research is one important way to lower the cost of creating and maintaining the taxonomy and getting labeled examples. Research areas around this problem include key-phrase extraction, active learning, distant supervision, and hybrid (ML/expert) systems.

Domain-specific concepts have many use cases. For example, they can be used as navigational markers (of the underlying semantic space) or as wayfinding tools in a search engine (e.g., see “Concept Markers” in Checkpoint Edge).

One of the most common tasks in document review is to identify deviations from the norm. In contracts, for example, clauses often follow specific language constructs because these language norms withstood the tests of time and litigations. Material deviations from the norm — while permitted — is risky and must be done carefully. Similarly, in filings with the U.S. Securities and Exchange Commission, significant deviations from the norm in unstructured sections (e.g., Management Discussion & Analysis, or Notes to Financial Statements) can also be used as a predictor of risk. Other types of deviation include any changes in terms and conditions, for example in the context of contract review.

At the surface, language deviation is can be viewed as constructing language models for the norms and then defining domain-specific functions to measure material deviation. These functions are often aided by a dictionary of concepts and terms of art. Such an approach can be effective for some use cases, but we don’t believe it is capable of capturing structural deviations that are expressed in natural language. This remains an open and active research problem.

We have discussed several tasks that lie at the threshold of creating a document analysis product; however, most of these tasks are trivial for a human, at least at a small scale, and would not make a beneficial product by themselves. Instead, a product in this space needs to be designed according to a deep understanding of the best way to help a user by synthesizing these “atomic” inputs into something useful.

The Thomson Reuters Quick Check product grew from a study of the process of writing, reviewing, and updating trial and appellate briefs. Developers found out that a common workflow process is for experienced attorneys to review briefs written by less-experienced attorneys or to update an old brief. The objective here is to ensure that the brief cites relevant caselaw authorities (ideally relevant to the pertaining facts and the legal issues) that also reach a similar conclusion. Another task involves reviewing briefs filed by the opposing party, looking for relevant authorities that do not support the opposing party’s conclusion.

These use cases gave rise to a host of technical challenges; for instance, creating accurate document segmentation to capture the various issues that are raised in the brief. Further, each issue consists of four elements: i) the facts; ii) the legal principles raised; iii) the authorities that are cited; and iv) the explanatory text. Given segmented documents, one must then find relevant authorities and determine the relative weights of the four elements above.

Often, one can’t find an authority with the same set of facts and legal issues, which is why attorneys often think by analogy. While our algorithms are incapable of ‘thinking’ by analogy (e.g., a bus driver is similar to a machine-operator), we attempt to approximate it in a statistical sense.

From a usability perspective, attorneys may choose not to cite relevant authorities for many reasons and the last thing they want — especially in a time-crunch — is to review a list of these authorities suggested by an algorithm. So, it is prudent that we recommend highly relevant (and if possible) controlling authorities. To further reduce the burden of the review, the product must also provide the user with an explanation of why an authority might be relevant.

The product vision gave rise to a host of secondary questions: How can relevance in this context be approximated so that we can find relevant documents and create explanations? As a constructive answer to this question, we made hypotheses based on atomic tasks like the ones described above. Of course, not all of these were proven, nor did we think of the best hypothesis at first. Thus, there is a nexus of interaction between the product vision and the atomic tasks that are needed to implement it. One can envision developing a product as the process of forming hypotheses and constructively validating them.