Use Deep Dive and Clustering to Seed Predictive Coding – Knowledge Base

Workflow Guidance

Who: Project Admins and review leads planning a first-pass review
What: Guidance for leveraging Deep Dive, Clustering and Predictive cCding to prioritize documents for review
When: After your documents are uploaded and your project is enrolled in Deep Dive, but before you assign a large first-pass review
Why: Deep Dive can find your most important documents without requiring you to know the right keywords first. Using those documents as a starting point for Clustering and Predictive Coding lets you direct human review at the documents most likely to matter, instead of reviewing a large set in an arbitrary order.

When you receive a large volume of documents, the hardest part of planning review is often deciding where to start. Keyword searches only work when you already know what to search for, and they return every document that matches whether you want them all or not. That frequently leaves you with a set that is too large to review linearly and too undifferentiated to prioritize.

The flow in this article solves that problem by pairing tools that are good at different things:

Deep Dive is built to surface relevant documents in response to a natural-language question. This lets you ask about the concepts that matter, rather than define keywords that cast a too-wide net.
Predictive Coding and Clustering work across your entire corpus at scale

Used together, a small set of documents that Deep Dive surfaces becomes the a prioritized set of documents across the whole project, so that the review work your team does by hand is aimed at the right documents.

The flow helps teams prioritize human review in large, unfamiliar, or low-richness document sets while preserving human validation and reducing reliance on arbitrary review order.

This article is designed for admins and those planning review for large document sets.

In this article we'll cover:

How to write Deep Dive questions that identify important documents
How to collect and confirm those documents
How to expand the initial set through Clustering
How to use a Predictive Coding model to turn your initial set into a prioritized review set across your project

You'll use the following Everlaw tools:

Important

This article describes how to use tools to prioritize human review. The documents returned by Deep Dive are a starting point for human judgment, not a substitute for it.

Requirements

To ask questions in Deep Dive, your project must be enrolled in Deep Dive and you must be in a permissions group with the Ask or View and Ask permission for Deep Dive. Enrollment is handled by an Organization Admin or Everlaw AI Admin. For more information, see our Enroll and Get Started with Deep Dive article.
To create a predictive coding model, you must have Create or Admin permission for Predictive Coding
To view the Clustering visualization, you must have at least Clustering View permission
To add documents to a binder, you must have Edit or Full access permission on that binder
To create an Assignment Group, you must have Create or Admin permission for Assignments

Before you start, we recommend that a Project Admin:

Write a case description on the Case Description page in Project Settings. Deep Dive uses this background for every question asked in the project. For instructions, see our Everlaw AI Case Description guidance.
Enable Clustering for all documents in the project. For instructions, see our Clustering article.
Set up the codes you plan to use for review and Predictive Coding, so that the review work you apply to your seed documents can also define a predictive coding model. For more information, see our Administer Codes for Review article.

Example scenarios

This workflow is useful whenever you need to prioritize review of a large document set. For example:

You received a large data dump and keyword searches aren't narrowing it down. Even the obvious search terms leave you with tens of thousands of documents.
In this scenario, asking Deep Dive questions can help you get started with a core set of relevant documents. You use the documents behind those answers, along with the clusters they belong to, to build a training set for a predictive coding model. Then, use the model's prediction scores to decide what gets reviewed first and what can wait.
This is the flow outlined in this article.
You inherited an unfamiliar dataset with many custodians. You don't yet know which people or events matter. You ask Deep Dive targeted questions about specific individuals and events, then use Clustering to find the documents that are conceptually similar to the ones Deep Dive surfaced, and organize review by theme.

When not to use this workflow

This workflow is worth the setup on large, unfamiliar, or low-richness document sets. It is the wrong choice when:

Your set is already highly responsive: If you expect most documents to be relevant, targeted seeding adds a step without reducing volume
You need every matching document: Deep Dive prioritizes the most relevant documents rather than complete recall. When your obligation is exhaustive, rely on search and Search Term Reports, or consider Coding Suggestions for more comprehensive categorization.
Your set is small enough to review directly: If your team can read the whole set in the time available, read it
Your documents don't have usable text: Documents without extractable or OCR text can't be analyzed by Deep Dive, Clustering, or Predictive Coding

Main scenario

The example below follows a complex cryptocurrency case. One of the main issues you're tracking is the timeline of decision making around misappropriation of customer funds. "Funds" "allocate" and other related terms return more documents than your team can review.

The steps below can be used as general guidance for seeding any first-pass review.

Important: Seeding a Predictive Coding model from Deep Dive and Clustering is a form of targeted training, which means your training set is deliberately unrepresentative of your corpus. That is useful for getting a model started quickly on data with low richness, and it is a known source of bias. We recommend that you supplement targeted training with randomly sampled training sets, deliberately supply negative examples, and use the coverage map on the model page to find documents whose features aren't yet represented in your training. For more on the tradeoffs, see the training section of our Beginner's Guide to Predictive Coding.

Step 1: Set up a Predictive Coding model

Predictive Coding learns from your team's coding decisions to start predicting additional relevant documents. We recommend setting up a model at the beginning of your case for two reasons:

Setting up a model early helps you define the exact issue(s) that you consider relevant for review. Defining your relevancy criteria within a Predictive Coding model helps you focus on the questions you ask Deep Dive and concepts to look at in clustering.
Your model will start generating scores as soon as you hit the thresholds for reviewed documents. This can save you time and let you jump into prioritized review faster.

To learn more about setting up a Predictive Coding model, visit Create and Edit a Predictive Coding Model and A Beginner's Guide to Predictive Coding.

Step 2: Use Deep Dive to find a set of relevant documents

When you start your investigation with Deep Dive, you can find documents that are semantically related to your question. This means that they don't just hit on specific terms, but conceptually match the meaning behind what you're looking for. For large datasets with lots of irrelevant documents, Deep Dive gets you to a relevant set without extra sifting.

To get started, think of questions that pull from the criteria you already have: your requests for production, your issue list, or your privilege criteria.
In your question, tell Deep Dive what your goal with the response is. This context will help it generate a response that cites a broader set of documents than one trying to narrowly answer a specific question.
As an example:
1. Instead of: Who approved using customer funds? What is the timing of how customer funds were routed for specific uses?
2. Try: I need to find a set of relevant documents to train a predictive coding model. I'm interested in who approved using customer funds and the timing of how customer funds were routed for specific uses. Rather than answer the question narrowly, can you answer with a response that cites the highest number of relevant documents possible to help me get started?
If the Deep Dive response looks good, then the documents it used in its response will probably be relevant to your overall goals. To get to them, select the Sources button under the response, then select the number next to the document.

Tip

To get a larger set of documents, expand Deep Dive's thinking. Each thinking step identifies the number of documents it found for that step, and you can use the number to access a results table of those documents.

This opens a results table of all the documents cited in the response.
Add the documents into a binder named after the question or issue that led you to them. This will help you stay organized for the next steps.
[Optional] Go back to Deep Dive to ask more questions to get more possibly relevant documents. Make sure you add them to the same binder as in step 3.

Learn more about asking questions in Deep Dive in our Use Deep Dive article.

Step 2a: Validate Deep Dive documents

Before moving on, validate that these documents actually are relevant for your case. Open and code at least a few of them (with the codes you're using for your Predictive Coding model). If you're not finding any that feel relevant, try asking Deep Dive the question in a different way.

Step 3: Move on to Clustering to expand your seed set

Once you have an initial set of documents from Deep Dive, you can jump into Clustering to broaden your seed set. When you focus in on the clusters that contain your Deep Dive documents, you get just the right aperture to finish identifying the initial set of documents for training your model. In this step, you'll find the clusters that your initial Deep Dive documents fall into.

Go to Document Analytics > Clustering.
Select Filter in the toolbar, and build a search for the binder you created in Step 2. The visualization narrows to just those documents, showing you which clusters they fall into. This will help you focus on which clusters are your review priorities.
Enter Multi-select mode to select relevant clusters. In your selection, prioritize clusters that hold several documents from your Deep Dive binder, and don't select clusters that look off-topic.
Remove the filter. Now, you have selected all the documents in the selected clusters.
Select the unique document count at the top of the panel to open a results table of the selected clusters.
In the results table, add those documents to the binder for review.

Note

Make sure that your binder has at least 200 documents in it. This is the minimum number of documents that must meet the review criteria before a Predictive Coding model can start generating scores. If you don't yet have 200 documents in your binder, go back to Deep Dive to ask another related question to identify more possibly relevant documents. Depending on the richness of your binder, you may need to review more documents before the model can start generating scores.

Learn more about Clustering in our Clustering article.

Step 4: Confirm and code your seeds

The combination of Clustering and Deep Dive has identified a core set of documents that might be relevant for your case. Now, have someone you trust review the documents. Once they're done, your model can start generating scores to help you prioritize the rest of review.

The reviewer should use the coding criteria you used for your predictive coding model:

Code the documents that are genuinely responsive under the code you intend to use as the Relevant code for your model
Code the documents that turned out not to be responsive with the Reviewed (but not relevant) criteria. Documents that look relevant but aren't are valuable negative examples, and will train the model with better precision.
Note anything Deep Dive missed. If your reviewer expected to see a document or an issue that didn't surface in this specific response, that gap tells you which additional questions to ask to find more.

Your model generates prediction scores once 200 qualified documents have been reviewed under the model's criteria, with at least 50 relevant and 50 irrelevant.

Step 5: Route the documents to human review

Once your model generates scores, you have a prioritized set for review rather than an undifferentiated one.

Use an assignment group to batch them out for review.

Tip

Make sure you sort your assignment group byPrediction score, in descending order. This ensures that the documents with the highest prediction scores get reviewed first.

For guidance on creating an assignment group, see our Create Assignment Groups article.

Conclusion and next steps

Deep Dive tells you what your important documents look like. Clustering tells you what sits next to them. Predictive Coding shows you how many more of them there are and points you to them. Running the tools in that order means your reviewers spend their time on the documents most likely to matter, and you have prediction scores and performance metrics to support how you scoped the review.

As next steps, we recommend that you:

Prepare your reviewers before the first assignment goes out. Send them our Receiving an Assignment and How to Code a Document in the Review Window and Use Coding Presets articles.
Keep asking questions as review progresses. Deep Dive is as useful for confirming that your team found what it should have as it is for finding it in the first place.
Revisit your model as coding accumulates. Everlaw updates prediction scores approximately every 24 to 48 hours, so a model seeded early keeps improving throughout review.