Best Practices to Configure Coding Suggestions

The Everlaw AI Assistant can provide coding suggestions to reviewers based on code criteria configured by a Project Admin in Project Settings. 

The goal of this article is to support Admins in writing effective code criteria. 

Introduction

The code criteria describes when a certain code should be applied. You can write these descriptions in plain English, without any technical formatting or jargon. When a reviewer generates coding suggestions, whether for an individual document or in a batch, the Everlaw AI Assistant examines the document to evaluate whether any codes should be applied, and make a suggestion (Yes, Soft Yes, Soft No, or No). For each code, Everlaw will provide a rationale for its suggestion and, if available, a link to a potentially relevant area of the document. For codes with actionable suggestions, reviewers will have the option to accept each suggested action with a single click.

Everlaw offers maximum flexibility in allowing you to choose which codes you want to configure for Coding Suggestions. You can select which categories and codes to configure and may configure your entire coding sheet or just a single code. 

Enable the coding assistant

Required permissions: Project Admin

Everlaw AI Assistant live on the Project Settings > Everlaw AI > Assistant page. Here, you will configure the code criteria used to generate coding suggestions for documents.

1. Go to Project Management > Project Settings.
2. Select Everlaw AI. Then select Assistant.
   This opens the AI Assistant settings page.
3. Switch on Enable Coding Suggestions.
   
   A list of your project's existing coding categories appears.
4. Click Configure in the Case description section and write out the case background 
5. Click Configure for coding categories for which you'd like suggestions.
   
6. In the Describe category field, write a category description by clearly describing the type of decision that needs to be made.
7. In the Configure codes section, write a code description any code by clearly describing the criteria by which that specific code might apply to a document.
   Alternatively, you can also deselect codes for which you do not want the Everlaw AI Assistant to generate suggestions.
8. Select Save.
   This closes the model. Your category and code descriptions become visible.

Note: You can revisit and edit your descriptions, at any time, for both the category and codes at any time by selecting Edit configuration next to the category name on the AI Assistant settings page.

To learn more about enabling and configuring coding suggestions, visit Coding Suggestions.

Make the most of Everlaw AI Assistant for coding suggestions

The Everlaw AI Assistant uses three descriptions when generating coding suggestions — the case description, coding category description, and code description. You can think of these descriptions as instructions that the AI uses to generate suggestions. Writing clear and detailed instructions at each level will produce the best results.

Description	Content to include
Case	Summarize the overall case, including the main issues and arguments at play. Define the key entities and their roles, and any other information that is important context for making coding decisions. This is a good space to provide  any important information about the case that is likely relevant across multiple code categories.
Category	Include information that is relevant for coding decisions to be made for most codes under this category, but not relevant to other coding categories. There is no need to repeat information from the case description here.
Code	This is the space to write out the criteria indicating that this code should be applied to the document. Describe information, attributes, or qualities found in document text that would suggest the application of this specific code. Include extra-textual information (information that is unlikely to appear directly in the text) that could be important to evaluating a code.

 

Provide sufficient background and context 

Remember that generative AI (genAI) models are trained on human language and can understand nuance similar to how a person would. It is helpful to imagine the AI Assistant as a new reviewer without context for the case or the codes. When prompting the Everlaw AI Assistant, be as explicit as possible and define connections between people, places and events, rather than assuming things are implied. This context can include:

• The history of the underlying dispute
• The legal claims at issue
• Jargon or technical terms
• Entities involved (including alternative ways an entity may be referred to in the text because of name changes or abbreviations)

In general, the idea is to provide information from outside of the document text that is important for understanding and analyzing textual information found in the documents. 

You can also think about the right scope of where to include this background and context.

• If the information is relevant for the case as a whole and across all categories of codes, then it should be included in the case description
• If the information is relevant only to a particular category, then it should be included in the category description. If the information is only relevant to a particular code, then it should be included in that code’s criteria.

How to write strong case, category, and code descriptions

Here are some tips and tricks to writing effective coding criteria:

Adjust your code criteria based on your goals for the code

If the code you are configuring is more extractive in nature (ie. you can clearly point to or “extract” a piece of textual evidence that supports the code’s criteria with no additional context or explanation needed), then you should specify exactly what features or information in the text the system should be looking for in order to decide whether the code should be applied or not. 

For example, if you have a code called “Meetings with FDA regulators” meant to capture documents that evidence such meetings, your code criteria can be:

• Any direct evidence of interactions between employees of Company A with Federal Drug Administration (FDA) regulators, including, but not limited to evidence or mentions of email communication, phone communication, in–person meetings, etc. Use any title, contact information (like email domains), or contextual information in the document to determine if the individuals involved are employees of Company A or the FDA.

If the code you are configuring is more analytical in nature (i.e. additional context is required to explain how or why a piece of text relates to the concept you’re trying to capture with a code) then you may need to include more information and guidance on how the system should apply aspects of the text to the concept you are trying to capture with the code.

For example, if you have a “Breach of contract” code meant to capture documents relevant to analyzing the extent to which a breach has occurred, then you may want to describe the clause at issue, specify the parties relevant to the analysis, and give examples of things that would evidence a breach. Your coding criteria could resemble the following: 

• "Any direct or circumstantial evidence relevant to analyzing whether Company A failed to meet its contractual obligations to Company B to deliver a functional software program that also has ‘an intuitive and high quality UI/UX’. Of particular issue is whether the delivered product met the “intuitive and high quality UI/UX” standard set out in the contract. Relevant evidence can include, but is not limited to, discussions or instructions about the UI/UX between employees of Company A and B, exchange of intermediate prototypes or wireframes, and representations made about the state and status of development. In general, look for anything that may shed light on whether and how the parties discussed the UI/UX of the product, and implicit or explicit evidence of the understanding that the parties held around the concept of intuitiveness or quality, even if not described in those exact terms.” 

Tips and Tricks

Tip	Example
Ambiguity and Context  Be mindful of when your language may be ambiguous or subjective. Define or provide additional context on words or expressions that may be subject to interpretation.	Examples of ambiguous/subjective terms: misleading, unusual, suspicious, reckless, effective, generous Additional context to include: “A misleading statement is characterized by a projected profit” or “A suspicious order may exceed 10,000 units”
Modulating the Scope  Modulate how broad or narrow your scope should be. To capture more documents that may be potentially relevant, with higher the risk of false positives, use language to broaden the scope. Conversely, to be more conservative, use language to narrow the scope.	Broaden the scope: use language like “explicitly or implicitly related to…” or “...including similar or related information even if described in different terms” Narrow the scope: use modifiers like “directly”, "specifically", "exactly", or "only"
Specify Features or Information Specify the features or information in the text, providing examples if known. Indicate whether examples provided are exhaustive.	List non-exhaustive examples: “...reports on earnings, including but not limited to internal profit and loss  reports, minutes from earning calls, or SEC Form 10-K filings"
Alternative Entity References  Spell out alternative ways an entity may be referred to in the text by including details on name changes, abbreviations, or relationships.	Entities with multiple names: “Everlaw was formerly known as EasyESI” or “Company ABC consists of a joint venture between AB Inc. and C Corp”
Defining Technical Terms Define any technical terms, jargon, and acronyms.	Acronyms: “CDR refers to the the company’s central data repository, which is …” or “DOI = date of incident”
Affirmative Examples  In general, be affirmative with examples and instructions. Instead of saying “don’t,” say “do”; give examples of x, instead of examples of NOT x.	Avoid: "Do not look for internal communications between employees of Company A" Rephrase with positive example: "Focus only on external communications involving an employee of Company A and at least one member of another organization"
Be Positive in Sentiment and Tone Describe the sentiment, or tone, that suggests a specific code applies in that circumstance.	Sentiment adjectives: angry, upset, negative, positive, optimistic

A full example prompt

Category: Issues 

Category Prompt: These are codes capturing documents relevant to various sub-issues in the matter. For these codes, documents from and by employees of Company X and Company Y can be particularly important. Company X may also appear as ‘ABC Corp’, and Company Y was formerly known as ‘M Enterprises’. 

Code: Collusion

Code Prompt: Identify documents containing language that explicitly or implicitly suggests collusion between Company X and Company Y. This includes but is not limited to communications that reference ‘strategic partnerships’, ‘coordinated efforts’, or ‘joint ventures’. In the context of this inquiry, ‘collusion’ is defined as any agreement or cooperative effort, whether formal or informal, aiming to manipulate market conditions or pricing strategies. Make sure to capture both direct communications and any indirect references or inferences made in internal memos, emails, or meeting minutes. Communications that have a tone of secrecy or concealment about these topics may be particularly interesting. Financial information indirectly evidencing collusion or business conditions can also be relevant, such as internal profit and loss reports, projection models, or pricing calculations. 

How this prompt fits the criteria

Ambiguity and context: The prompt clearly defines "collusion’" and gives specific examples of expressions that might indicate it, like "strategic partnerships."

Modulating the Scope: The prompt uses phrases like “explicitly or implicitly suggests” and “includes but is not limited to” to widen the scope.

Specifying Features or Information: The prompt includes a section on financial information, providing specific, non-exhaustive examples of relevant documents (e.g. “internal profit and loss reports, projection models…”)

Alternate entity references: The prompt accounts for different names of the companies involved (e.g. “Company X” may also appear as “ABC Corp”).

Defining Technical Terms: The prompt defines the main term of interest, ‘collusion’, in the context of the investigation.

Affirmative examples: The prompt specifies what to focus on (e.g. “identify any documents containing language…”) rather than what to exclude.

Sentiment or Tone: The prompt requests to focus on documents that exhibit a tone of secrecy or concealment, linking the sentiment to the investigation's context.