To view all of Everlaw's predictive coding-related content, please see our predictive coding section.
Table of Contents
For an introduction to predictive coding, feel free to reference our beginner’s guide to predictive coding.
For a guide to predictive coding-related terms and commonly-asked questions about Everlaw’s predictive coding feature, see our Predictive Coding Terms and FAQs.
Accessing Predictive Coding
Users must have proper permissions to access prediction models (i.e., at least Receive permissions on Prediction Models). Below are the necessary permissions for accessing prediction models:
- Receive: Users in this group can receive prediction models shared by others, but cannot create new prediction models themselves. When a prediction model is shared with a user in this group, the user can be given View, Edit, or Full access permissions on the individual prediction model.
- Create: Users in this group will be able to create prediction models as well as edit and delete the prediction models they have created. Permission levels on Everlaw are additive, so groups with the Create permission are able to to receive prediction models shared by others.
- Admin: This is the highest permission a group can have on prediction models. Users in groups with Admin permissions on prediction models are able to view, edit, share and delete all prediction models in the project, regardless of whether the prediction model was shared with them or not.
If you are a Project Administrator, you have access to predictive coding by default. Project Administrators can also give Prediction Model access to specific groups in the Permissions page.
Learn more about configuring project permissions.
Creating a Predictive Coding model
To start building a predictive coding model, click on the Document Analytics icon on your top toolbar and select Predictive Coding.
Then, click “+Create New Model” from the left side menu.
In the first step of the predictive coding wizard, you’ll be provided with an introduction to predictive coding, as well as a link to the Everlaw predictive coding beginner’s guide. Click Next to begin building your model.
Reviewed documents
First, specify which documents the model should learn from. These documents are considered “reviewed” for the purposes of the model. As an example, let’s say your team is reviewing documents for responsiveness, using the codes Responsive and Not Responsive under the coding category Responsiveness. Now, you want to build a predictive coding model that will find other responsive documents. To teach the model which types of documents are responsive and which are not, the model needs to be pointed towards documents your team has already reviewed for responsiveness. To do this, you would set your criteria for reviewed documents to be "Coded: Responsive OR Coded: Not Responsive" or “Coded under Responsiveness” (assuming the codes Responsive and Not Responsive are the only codes under the category Responsiveness). The model will therefore look at all documents that have been coded Responsive or Not Responsive to help it understand which types of documents are responsive.
In our example, it’s important that the "reviewed" criteria captures documents that are not responsive, as well as those which are responsive. In other words, we wouldn't want the "reviewed" criteria to only be "Coded: Responsive." This is because the model needs to learn what both responsive and non-responsive documents look like in order to make accurate predictions.
Relevant documents
Here, specify which types of documents you want the model to find. These documents are considered "relevant" to the model. For our responsiveness model, responsive documents are relevant to the model. In other words, we want the model to find responsive documents.
To specify relevant documents, build a query that captures only those documents that you want to find more of. In our responsiveness example, we would build a query that captures documents coded Responsive.
Now, the model knows what types of documents you want to see more of.
Excluding documents
The exclusion step allows you to specify documents you want to exclude from your model, even if they have been reviewed in a way that matches the model’s criteria. The model will not use excluded documents for training and evaluation purposes. Nor will model predictions be generated for excluded documents. By default, documents produced in Everlaw are excluded from models. This is meant to prevent redundant training and duplicative document suggestions. However, you may choose to remove this default exclusion criteria, if desired.
One common reason to exclude documents is if you know certain classes of documents have either atypical content or insufficient text for reliable predictions. For example, you may want to exclude documents primarily in a foreign language, as well as documents with little textual content, like spreadsheets and audio files. If you do not exclude any documents, all documents with adequate text (including transcribed audio and video files) will receive prediction scores.
Finalizing your model
On the next page, enter a name for your model. This name will be visible to everyone who uses the model, and will also be the name you use to search for the model’s predictions. By default, the name of your model will be the relevance criteria for your model, but you can rename it to whatever you like.
Finally, submit your model. Generally, your model will begin making predictions once you have met the review threshold by reviewing at least 400 unique documents (no duplicates). At least 100 of those unique documents must be relevant and at least 100 must be irrelevant. Additionally, irrelevant documents that are near duplicates or emails in the same thread as relevant documents will be considered "conflicts' and removed from consideration in your training set. In other words, these irrelevant documents will not count towards the 100 irrelevant documents needed to meet the review threshold. If it seems that you have hit the review thresholds but are not seeing an update to your model, it's likely that you need to review more unique documents in your training set or that you have too many "conflicts" that are irrelevant.
You can share your model before the model has generated any predictions. Click the share button in the top right of the model's page.
You can edit your model's name and criteria by clicking the edit icon.
You can also delete the model by clicking the trash can icon.
Editing a predictive coding model
At any point after you create a predictive coding model, you can edit your model’s name and criteria for reviewed, relevant, and excluded documents. Your ability to edit predictive coding models will depend on your project permissions. See the above section for more information about permissions on prediction models. You’ll be able to tell that you have edit permissions on a model if you see a pencil icon towards the top right corner of the model’s page.
Edit model name
Users can edit a predictive coding model’s name on the model’s page. To quickly edit a predictive coding model, click the name of the model at the top of the model’s page which highlights the name in blue and start making edits. After you make your edits, press Enter on your keyboard. Your model’s name should immediately update at the top of the model page and on the left hand menu.
You can also edit your model’s name along with your model’s criteria in the predictive coding wizard as described in the next section.
View and edit model criteria
To view and edit the criteria of an existing model, click the pencil icon on the model’s page to open the predictive coding wizard. In the predictive coding wizard, you can view your model’s current criteria and name. To edit a predictive coding model’s criteria from the predictive coding wizard, make changes to reviewed, relevant, and excluded documents in the EQL builder located on each corresponding step. Once all your desired edits have been made, click Submit to confirm edits and update your model. This will remove any prediction scores and historical performance associated with the model’s previous criteria. Note that it may take up to an hour for the model update to complete and implement changes to your model’s criteria.
Once the model update is complete, new prediction scores will be generated according to your model’s updated criteria if the review threshold has been met. Also, any existing workflows referencing the edited model using the Predictive search term (e.g. inclusion criteria for assignment groups) will be updated accordingly.
View model edit history
Users with at least Receive permissions can view a model’s edit history by clicking the activity icon located at the top right of the model’s page. All edits to a model will be captured in the model's edit history.
To read more about analyzing your model’s results, see the Predictive Coding Model Interpretation article.
0 Comments