Document summaries, topic analysis, custom extractions, and Q&A

This article is a deep dive on the summaries, topic analysis, custom extractions, and document Q&A features, which are available as part of Everlaw AI Assistant. 

• If you're looking for a starter guide for all Everlaw AI Assistant features, see this article.
• If you're looking for a general overview of Everlaw AI assistant and information about our privacy and security standards, please see our AI Assistant FAQ. 
• If you have feedback or questions, feel free to email us at feedback@everlaw.com. We love hearing from our users.

Table of contents

• Overview and background
  • Accessing features
  • Feature availability and scope of results
  • Deleting results and regenerating
  • Document text and limits
  • AI credits
• Document summaries
  • Chunked summaries and shorter descriptions
  • Using summaries in the Review Window
  • Generating summaries in batch and exporting summaries
  • Searching based on summaries
• Topics extraction, with integrated entity and sentiment analysis
  • Generating and viewing topics
  • Navigating topic snippets
  • Searching/filtering topics in the Review Window
  • Additional options in the Review Window
  • Generating topics in batch
  • Searching documents by topics
• Custom extractions
  • Important workflow considerations and limitations
  • Creating an extraction request
  • Using extractions
• Document Q&A

Overview and background

Accessing features

Document summaries, topic analysis, custom extractions, and document Q&A can be accessed from the Everlaw AI context panel in the Review Window:

• The summaries and topic tasks are located in the Overview tab
• Custom extractions is located in the Extractions tab
• Document Q&A can be accessed from the Ask questions about this document button in the footer of the panel

In addition, the batch versions of the summary, topics, and extractions tasks can be accessed from the results table via the Batch icon in the toolbar. 

Feature availability and scope of results

Summaries, topics, extractions, and document Q&A are available in both review and ECA projects. 

• Summaries and topics: Summaries and topics are saved per document, per database. This means that summaries and topics generated for a document in one project will also be available for that document in other projects in the same database. 
• Extractions: Extractions are saved per document, per project. Extractions are not shared across projects in a database. 
• Document Q&A: Results are saved per user, per project. You cannot see questions asked by other users in the project. 

Deleting results and regenerating

From the Review Window, you can discard a saved result for a task if you have the right permissions. This means the next time the task is run against the document, a new generation request will be sent to the LLM. The result for a given task on a given document should rarely change in substance across requests, so you should only delete and regenerate tasks if there are clear errors. 

Document text and limits

Text: If the PDF image for a document has extractable text, Everlaw uses that as the basis of analysis. If not, Everlaw falls back on the OCR text file. For batch tasks, Everlaw will always use the OCR text file. Documents that do not have extractable text or an OCR file cannot be analyzed by the Everlaw AI Assistant. In addition, documents with insufficient text will also be excluded from analysis. 

Document length limits: Everlaw currently limits the length of supportable document text to approximately 300 “pages” (a page being defined as ~500 words). If you run a task against a document that exceeds this threshold, we will let you know, via a banner in the Everlaw AI context panel, that the response is based on only a subset of the available content.

Variability of output: Because LLMs are not deterministic systems (ie. there is randomness in their outputs, even with the same input), you should expect some level of variability in the output, even when re-running the same task against the same document.

AI credits

AI credits are required to run AI Assistant tasks. Tasks consume a variable number of credits depending on the feature and the scope of the action (ex. length or number of documents analyzed). The number of credits consumed by a task is always shown on the generate button. Your ability to generate tasks depend on the available credits and various permissions. For more information about credits and AI permissions, see this article. 

 📄 Document summaries

Chunked summaries and shorter descriptions

This task generates a narrative summary of the document. In order to preserve sufficient information about the document's content, documents are summarized in chunks, with the size of the chunk scaling with the size of the document. For example, some documents' summary chunks may cover roughly every 5 pages of content, while others may cover every 10 pages.  

If the document is long enough to be summarized over multiple chunks, a shorter description of the document will also be generated. Short documents will only have descriptions. 

Document summaries/descriptions can be viewed:

• In the Review Window, via the Everlaw AI tab in the context panel
  • By default, only the description is shown. Click on View summary by section to expand the view to include the longer, chunked summary, if available.  
• On the results table, via the Description column
  • Only the descriptions are shown
• In the Description field for a document on the Storybuilder timeline, if there is no human-created description available
  • Only the descriptions are shown. You can learn more about AI descriptions in Storybuilder in this article. 

Using summaries in the Review Window

When viewing the chunked summaries in the Review Window, you can click on a page range marker to jump to the first page in the range. This allows you to navigate the document you're viewing by the summary. For longer documents, you can imagine a workflow where you first read the summaries to get the gist of the document, and then focus in on the most pertinent sections utilizing the page range markers. 

The following actions are available for summaries in the Review Window, depending on your permissions:

• Copy to clipboard: copy the contents of the summary or summary chunk to your clipboard
• Copy to note: copy the contents of the summary or summary chunk to the notes field in the Review Window
• Delete: delete the existing summary/description for the document

Generating summaries in batch and exporting summaries

Summaries can be generated in batches of up to 20,000 documents at a time. To generate a batch summary:

1. Open the results table, whether associated with a search, binder, upload, etc.
2. Ensure that fewer than 20,000 documents are selected on the table
3. Click Batch on the toolbar and select the Descriptions and summaries option
4. Confirm the task on the dialog that appears, and optionally add the associated Description column to your table

After the task is confirmed, the documents will be sent to the Everlaw AI task queue for processing. Depending on (1) the number of documents submitted and (2) the size of the queue, the batch task can take anywhere from a couple of seconds to many minutes to complete. The description column will populate in real-time as individual summary jobs are completed. 

Once descriptions are generated, they can be exported as part of a CSV export. When configuring your CSV export from a results table, ensure that the Description column is checked. 

Searching based on summaries

You can search for documents based on whether they have AI descriptions/summaries. To search by this status, add the Has Description/Summary search term to your search builder. 

Negate this search to find documents in your project that do not have AI descriptions/summaries.

 

🔖 Topic extraction, with integrated entity and sentiment analysis

This task analyzes the document for topics. Depending on the document's length, there can be anywhere from 1 to over a dozen identified topics.

Note: Sometimes, the topic names may make it appear like there are duplicate topics; in such cases, we encourage you to read the summaries of the topics as that will provide information to help you distinguish topics. 

Generating and viewing topics

The topics task is found on the Overview tab in the Everlaw AI context panel. To generate topics, simply click Generate in the topics section. 

 

Once generated, topics will be appear as a list of expandable cards.

Clicking on a collapsed topic card will expand it. 

A topic card has up to 5 components:

• Topic name: The high-level topic. 
• Topic summary: A bullet-point summary for the topic based on the document text.  
• Relevant area: Navigation between document snippets relevant to the topic.  
• Sentiment: The sentiment rating for the topic and an analysis of the rating. The rating will be one of “positive”, “negative”, “harmful and offensive”, or “neutral”. Everlaw will not display any sentiment information if the rating is neutral. 
• Entities: The entities associated with the topic, if any, categorized into organizations and natural persons. Clicking on an entity’s badge will open up a pop-over with additional information about the entity and the option to add the entity name as a hit highlight. 

Navigating topic snippets

For each topic, Everlaw will identify a set of document snippets relevant to the topic. To navigate through these snippets, first open the desired topic, then click the “Show relevant area” component. This will:

1. Assign the topic a highlight color, which will appear on the left edge of the topic card
2. Highlight the related document snippets in that color
3. Replace the “Show relevant area” component with arrows that can be used to navigate between the highlighted document snippets. Only one topic’s snippets can be highlighted and navigated at a time. 

Searching/filtering topics in the Review Window

There are three ways to navigate, search, or filter topics:

• You can use the search option to filter topic cards by keyword. This option only searches/filters by the topic name. 
  • 

• You can jump to and auto-expand a topic card by clicking on it in the Topic menu

  • 

• You can filter topic cards by sentiment rating or entities via the filter option. Multiple selections within the sentiment or entities category are ORed together. Sentiment and entity filters are ANDed together. For example, if a user filters by “negative” and “harmful or offensive”  sentiment and “John Doe” and “Jane Doe”, the filtering logic would be: (sentiment is “negative” OR “harmful or offensive”) AND (entity is “John Doe” OR “Jane Doe”). 

  • 

Additional options in the Review Window

Depending on your permissions, you can copy the information for individual topics to a note or clipboard via the icons in the lower right of the topic card. The ability to copy the information for all topics to a note or clipboard, and to delete the generated topics, can be found via the menu icon for the topic task. 

 

 

Generating topics in batch

You can batch generate topics for up to 20,00 documents at a time. To batch generate topics:

1. Open a set of documents. Ensure that there are 20,000 documents or fewer selected on the results table.

2. Click Batch in the toolbar and select Topics. A pop-up will appear for you to confirm the action.

3. Optionally, choose to add the Topics column to your results table, if it is not already visible.

4. Submit the batch job.

As topics are generated for documents, topic titles and sentiment ratings are populated in the Topics column. Depending on the number and size of documents, a batch task can take anywhere from a few seconds to many minutes to complete.

You can open a document to review the full topic generation. You can also include the topics column as part of a CSV export. To learn more about exporting, see this article. 

Searching documents by topics

You can search for documents by topic title and sentiment rating. To search by topics:

1. Add the Topics search term to your query

2. If no values are specified in the Title and Sentiment fields, the search will find all documents with generated topics. Negate this search to find documents without any generated topics. 

   • You can search the Title field by keyword or key-phrase, or select a topic from the auto-complete dropdown. 

     • 

   • You can search the Sentiment field by any mixture of sentiment ratings. The sentiment search returns documents that have any topics with that sentiment rating. 

     • 

   • Searching the Title and Sentiment fields together will return documents with one or more topics that match both the topic and sentiment search criteria. 

✄ Custom extractions

Custom extractions allow you to extract out desired pieces of information from documents. For example, if patient records are part of your corpus, you can extract out values from those records – drugs and dosages, medical care providers, etc. – that are important to capture. Custom extractions are useful in cases when keyword or regular expression searches are too ineffectual or cumbersome.

Extractions are based on Fields you configure. A field is a category of information you wish to extract out of the document; there can be multiple extractions per field. For example, if your field is "medications", Everlaw AI will extract out all potential mentions of medications in the document. 

Important workflow considerations and limitations

Test your extraction configurations on a few documents before scaling use: Testing your extraction instructions will help you fine-tune the language to avoid under-extracting or over-extracting. Based on your testing, you may find that, for example, you need to use more precise language or describe what you are not looking for in addition to what you are looking for.

Extractions may be wrong: Extractions may be wrong in four ways:

1. The extraction does not exist in the text
2. The extraction exists in the text, but shouldn't have been extracted based on your instructions
3. There's relevant text that should have been extracted, but wasn't
4. The extraction is generally correct, but may not exactly match the document text (this is more of an issue for longer extractions)

You can check for Type 1 errors by adding extractions as hit-highlights in the document viewer. This will highlight extractions that exist in the document text and display a count of hits. Because extraction errors require various degrees of manual verification at this stage, we recommend using extractions more as a workflow adjunct or prioritization aid. If extractions are used in reporting situations where the primary source won't be otherwise checked or referenced, you should definitely plan perform a manual verification. 

Creating an extraction request

To create an extraction:

1. Open a document in the Review Window
2. Open the Review Assistant context panel on the left sidebar
3. Toggle to the Extractions tab

4. Click Configure and generate extractions to open the configuration view

5. Enter Field, Type, and Description information. Up to 5 fields can be configured for extraction at a time.                                       

• • Field: The name or category of the extraction (ex. contract date, patient name, indemnity language)

  • Type: Choose from:

    • Text: The extractions are expected to be multiple words or sentences in length

    • Number: The extractions are expected to be numbers or references to numerical information

    • DateTime: The extractions are expected to be in datetime format or are references to dates and times

    • Entity: The extractions are expected to be names of entities, such as persons, organizations, or locations. 

  • Description: A concise description of the field

6. Click Generate

7. Wait as the results populate in the extractions table

Extractions are additive: users can configure multiple extractions for a given document, and new results will be added to existing results.

Using extractions

Once extractions are generated, you can:

• Search the table of extractions via the search option on the top right of the table
• Add extractions as hit highlights by clicking the hit highlight icon
• Copy extraction results to your clipboard by clicking the copy to clipboard icon
• Copy extraction results to a note by clicking the note icon

Generating extractions in batch

To generate extractions in batch:

1. Identify the set of documents you wish to extract information from using any combination of Everlaw’s search, filter, and results table document selection tools.

2. Ensure that there are fewer than 20,000 documents selected on the table

3. On the toolbar, select Batch > Extractions.

4. In the dialog that appears, select Configure Extractions and create up to 5 fields to extract. Then click Configure

   • 

5. [Optional] Add the Extractions Column to your table.

6. Click Generate

As extractions complete, they populate in the extractions column and are viewable in the document review window.

🧐 Document Q&A

This task allows you to ask your own question of a document/create your own task. It can be accessed from the "Ask questions about this document" button in the footer of the AI context panel.

If the document is sufficiently long, the answer will first be returned by document chunk ("detailed" answer), with a final synthesized answer generated at the end ("condensed" answer).

When reviewing the detailed answer, you can click on a page range for a chunk to navigate to the first page in the range. You can also click on the note icon to populate a note field with the content of a generated response, or the clipboard icon to add the generated answer to your clipboard. 

Unlike other review window tasks, the response to this task is private to you. Everlaw will preserve the last 20 questions asked for a given document.