Coding Suggestions (Everlaw AI Assistant)

This article is a deep dive on Coding Suggestions, which is a feature available as part of Everlaw AI Assistant. If you are new to Coding Suggestions, we recommend you start with this companion article that covers key concepts and recommended workflows. 

• If you're looking for a starter guide for all Everlaw AI Assistant features, see this article.
• If you're looking for a general overview of Everlaw AI Assistant and information about our privacy and security standards, please see our AI Assistant FAQ. 
• If you have feedback or questions, feel free to email us at feedback@everlaw.com. We love hearing from our users.

Coding Suggestions automatically categorizes documents based on codes on your coding sheet. For each code configured for use in Coding Suggestions, Everlaw will suggest whether the code should be applied and provide a rationale for its suggestion. 

Coding suggestions can be used to speed up review of new documents, QC existing review work, and enable useful search/filtering options.  

Configuring coding suggestions

Before users can generate coding suggestions, a project admin must first enable the feature and write coding criteria on the Project Settings > Everlaw AI > Assistant page. Coding criteria provides background, context, and guidance about what codes are meant to capture and how to evaluate codes against a document and consists of three description fields:

1. Your description of the case
2. Your description of the category
3. Your description of the code

Adding a case description

To add a case description, enable coding suggestions on the Assistant settings page and click Configure or Edit in the Case description field. 

Similar to how you would describe the case to a reviewer unfamiliar with the matter, provide a concise summary describing the nature of the dispute. Include context and information that is generally relevant across all codes in your matter. This can include the legal claims at issue, jargon or technical terms, and key events and entities involved (including alternative ways an entity may be referred to in the text ).

Information that is relevant primarily to individual categories or codes do not need to be included in the case description and instead should be included in the category and code descriptions as described below.

Selecting codes to configure

Everlaw offers maximum flexibility in choosing which codes you want to configure for use in Coding Suggestions: you can configure your entire coding sheet or just a single code. To configure codes:

1. First, ensure Coding Suggestions is enabled in Project Settings > Everlaw AI > Assistant 
2. A table will be shown displaying all code categories in your project and a summary of configured codes. Click Configure or Edit to make changes to a particular code category.
   • 
3. A pop-up will appear where you can add or edit the code criteria for the category and the individual codes. Unselect the checkbox next to a code's name to exclude it from your configuration. 
   • 

Any changes made to the configuration will only affect suggestions generated from that point forward; existing suggestions will not be affected, unless suggestions are explicitly rerun on documents with outdated suggestions. 

Writing coding criteria

The quality and accuracy of Coding Suggestions hinges on the quality of your code criteria. Here are some general tips and tricks to writing effective code criteria:

• Provide sufficient background and context: Include extra-textual information (information outside of the document text) that is important for understanding and analyzing textual information found in the documents. This can include relevant acronyms, key events, jargon or technical terms, entities involved (including alternative names and email domains), etc.
  • You can think about the right scope of where to include this background and context. If the information is relevant for the case as a whole and across all categories of codes, then it should be included in the case description. If the information is relevant only to a particular category, then it should be included in the category description. If the information is only relevant to a particular code, then it should be included in that code’s criteria. 
• Adjust your code criteria based on your goals for the code: 
  • If the code you are configuring is more extractive in nature (ie. you can clearly point to or “extract” a piece of textual evidence that supports the code’s criteria with no additional context or explanation needed), then you should specify exactly what features or information in the text the system should be looking for in order to decide whether the code should be applied or not. 
    • For example, if you have a code called “Meetings with FDA regulators” meant to capture documents that evidence such meetings, your code criteria can be:
      • “Any direct evidence of interactions between employees of Company A with Federal Drug Administration (FDA) regulators, including, but not limited to evidence or mentions of email communication, phone communication, in–person meetings, etc. Use any title, contact information (like email domains), or contextual information in the document to determine if the individuals involved are employees of Company A or the FDA.”
  • If the code you are configuring is more analytical in nature (i.e. additional context is required to explain how or why a piece of text relates to the concept you’re trying to capture with a code) then you may need to include more information and guidance on how the system should apply aspects of the text to the concept you are trying to capture with the code
    • For example, if you have a “Breach of contract” code meant to capture documents relevant to analyzing the extent to which a breach has occurred, then you may want to describe the clause at issue, specify the parties relevant to the analysis, and give examples of things that would evidence a breach. Your coding criteria could resemble the following: 
      • "Any direct or circumstantial evidence relevant to analyzing whether Company A failed to meet its contractual obligations to Company B to deliver a functional software program that also has ‘an intuitive and high quality UI/UX’. Of particular issue is whether the delivered product met the “intuitive and high quality UI/UX” standard set out in the contract. Relevant evidence can include, but is not limited to, discussions or instructions about the UI/UX between employees of Company A and B, exchange of intermediate prototypes or wireframes, and representations made about the state and status of development. In general, look for anything that may shed light on whether and how the parties discussed the UI/UX of the product, and implicit or explicit evidence of the understanding that the parties held around the concept of intuitiveness or quality, even if not described in those exact terms.” 

For more guidance and tips on creating effective coding criteria, please refer to the Coding Suggestions Prompting Guide, found here.

Viewing code criteria history

Everlaw saves a history of updates and edits to the code criteria. This allows you to track and compare changes that may have affected performance. To view this history, go to Project Settings > Everlaw AI > Assistant and click View history in the upper right corner of the coding suggestions configuration table. 

The panel that appears allows you to see a timeline of edits and access previous versions of the configurations.

Understanding suggestions

Suggestion categories

For each configured code, Everlaw will provide one of four possible suggestions:

• Yes: The document directly matches the configured criteria
• Soft Yes: Although the document is not directly relevant, there is a strong plausible link
• Soft No: The document is at best only weakly relevant
• No: The document has no relevance given the configuration

For more information and guidance on how to use and interpret these suggestions, please see this article. 

Out-of-date suggestions

Because code criteria can change over time, a suggestion may become out-of-date. A suggestion is out-of-date if any one of the following is modified after the suggestion is generated:

• The code prompt 
• The category prompt 
• The case description

Out-of-date suggestions are indicated with a dotted outline; you have full control over whether to rerun these suggestions. 

Actionable suggestions

When searching or filtering by suggestions, you can narrow down to only actionable suggestions. Actionable suggestions are:

• Codes where the suggestion is Yes or Soft Yes, but the code is not applied to the document
• Codes where the suggestion is No or Soft No, but the code is applied to the document

Generating coding suggestions

Coding suggestions can be generated from the Review Window or in batch from the Results Table. 

• AI credits are consumed when suggestions are generated. The number of credits required is shown by the generate button. You will be prevented from generating new suggestions if there are insufficient credits or if you lack the proper permissions. To learn more about AI credits and permissions, see this article. 
• There is a limit of 10 codes per request. If you need to generate suggestions for more than 10 codes per document, you'll need to break out the task over multiple generation requests.

To generate coding suggestions for a document in the Review Window, open the AI context window. Then navigate to the Coding Suggestions tab and click Generate. 

By default, all configured coded are selected for inclusion in the request, subject to the 10 code limit, and excluding codes that already have up-to-date suggestions on the document. To modify the codes included in the request, click Select codes to suggest... and make your selection on the resulting dialog prior to clicking Generate. 

For each of the selected codes, Everlaw will return a suggestion of whether the code should be applied to the document and a rationale for the suggestion. If Everlaw identifies a potentially relevant area in the document, a link to it will also be shown as part of the rationale. Coding suggestions are grouped by category for easy identification. 

You can easily apply, remove, or replace a code directly in the Review Assistant based on the suggestion. Note that there will not be any indication that the action was taken from coding suggestions, so be sure to verify that you agree with the suggestion before taking an action.  

To see the underlying coding criteria, click View configuration in the upper right of the Coding Suggestions tab. Project admins can edit the existing criteria directly from this dialog, and can add new codes from Project Settings. 

 

Generating Coding Suggestions in batch

Coding suggestions can be generated in batches of up to 20,000 documents. Before batch generating coding suggestions for large sets of documents, be sure to test, iterate, and evaluate your code criteria according to the recommended workflow described in this article. This will help you avoid costly mistakes or reruns! 

To initiate a batch action for coding suggestions:

1. Open a results table.
2. Ensure that there are no more than 20,000 documents selected on the table
3. Click Batch in the toolbar and select Coding Suggestions

A confirmation dialog will appear where you can:

• Select the codes to include in the request
  • For each document in the batch, Everlaw will auto-exclude any selected codes for which the document already has an up-to-date suggestion
• Optionally add the coding suggestions column to your results table. 

From this dialog, you can also view and verify configuration by clicking View configuration. 

Once confirmed, the task is initialized and the request will be submitted to Everlaw's AI task queue. Depending on the size of the batch request and the number of other documents in the queue, a batch of coding suggestions tasks can take anywhere from a few seconds to many hours to complete. 

Viewing the results

As individual documents are completed, the Coding Suggestions column will populate with the results. 

The column displays all codes that the document has suggestions for, with the actionable suggestions listed first followed by the non-actionable suggestions in a lower opacity. The rationale for each suggestion is viewable only from the Review Window.  

You can filter the documents by suggestion from the filter panel accessible from the Coding Suggestions column header. 

If the Coding Suggestions column is not visible on your results table, you can add it from the dialog that appears if you click View > Add or remove columns.

 

Searching and exporting Coding Suggestions

You can search for documents by coding suggestions using the “Suggested code” search term. This term allows you to search by:

• The code
• The suggestion
• The datetime the suggestion was generated
• Whether the suggestion is actionable based on the current coding status of the document

Finally, you can configure CSV exports to include coding suggestions.