Multi-matter Models

Multi-matter Models enable organizations on Everlaw to leverage previously trained predictive coding (PC) models to find relevant documents in brand-new, similar matters. Since these models are already trained, they can generate prediction scores on documents in new matters immediately. Once initial prediction scores are generated, teams can prioritize reviewing documents that the multi-matter model predicts to be relevant. 

Important: This article covers functionality that is only available to users with an annual, organization wide contract. Please contact your account representative or support@everlaw.com if you are not sure if you qualify. 

Table of contents

• Overview of multi-matter models
  • What is Predictive Coding?
• Access and Permissions
• Multi-matter Models page
• Creating a multi-matter model
  • Step 1: Open the New Multi-matter Model wizard
  • Step 2: Select PC Models
  • Step 3: Define Details
• Accessing the Multi-matter Model library
• Editing selected models and details for multi-matter models
• Using a multi-matter model in a project
• Deleting multi-matter models
• Multi-matter Models FAQs

Overview of Multi-matter Models

Organization Admins can create a multi-matter model for their organization by selecting one or more PC model(s) previously trained to find the same type of relevant documents in similar matters (i.e., datasets involving the same area of law or subject matter). Once created, models will be available for use in all projects across the organization. Organization members can use any of their organization’s available multi-matter models to generate prediction scores on new datasets without needing to train a new model from scratch. 

Over time, your organization on Everlaw can leverage a library of your own tailored multi-matter models to find important documents in specific legal matters, enabling your review teams to start reviewing documents predicted to be relevant sooner.

Back to table of contents

What is Predictive Coding?

Understanding predictive coding is essential for effectively using multi-matter models. 

At its core, predictive coding is a form of technology-assisted review (TAR) that leverages supervised machine learning to streamline the document review process. This technology allows computers to learn from human input and make informed predictions when classifying documents. Everlaw’s Predictive Coding system utilizes a regression-based learning algorithm, which describes how the system learns from review work in the project.

At a high level, a predictive coding model analyzes the text and some metadata of documents you’ve reviewed, measures the importance of each term, and then uses these measurements to learn which terms are good indicators of relevant documents. This learning is captured in the form of “weights” that represent how strongly each term is associated with documents being relevant or not. The model then uses these weights to generate prediction scores between 0 to 100 for each document to indicate how likely it is that a given document is relevant to your case.

Multi-matter models use the weights gathered from the selected PC models to generate prediction scores on documents in new matters, even if no documents have been reviewed. Once initial prediction scores are generated, you can immediately jump into a prioritized review of documents that are highly predicted to be relevant based on the multi-matter model in use. As more documents are reviewed in the new project, prediction scores are updated based on both the multi-matter model in use and the documents reviewed within the new project. 

Note: Multi-matter models do not contain any actual documents used to train the selected PCs models in their associated projects.

For an introduction to predictive coding, reference our Beginner’s Guide to Predictive Coding.

For a guide to predictive coding-related terms and commonly-asked questions about Everlaw’s predictive coding feature, see our Predictive Coding Terms and FAQs.

Back to table of contents

Access and Permissions

• To create multi-matter models:
  • You must be an Organization Admin.
  • Your organization must have at least one existing predictive coding model that has generated prediction scores. 

• To use available multi-matter models in a project: 
  • You must be a member of the organization and have Admin permissions on prediction models in the given project to select from your organization’s available multi-matter models. 
  • Once a multi-matter model is selected for use in a project, the page associated with the model is shared with all project users with Admin permissions on prediction models by default.
• Access to your organization's multi-matter models:
  • Multi-matter models are only available for use in projects owned by your organization. 
  • Multi-matter models created in your organization cannot be transferred to another organization or exported out of the Everlaw platform. 
  • If you are part of a parent organization, any multi-matter models created within the parent organization cannot be accessed or used from a sub-organization. 
  • If you are part of a sub-organization, any multi-matter models created within the sub-organization cannot be accessed or used from the parent organization.

Learn more about access and permissions for Predictive Coding and Organization Admin.

Back to table of contents

Multi-matter Models page

Organization Admins are able to access the Multi-matter Models page from the “Multi-matter Models” tab on the left-side menu of the Organization Admin (OA) page: 

1. Go to the Organization Admin (OA) page. 
2. Select the “Multi-matter Models” tab to open the Multi-matter models page.

On this page, Organization Admins can create new multi-matter models and manage their organization’s multi-matter model library (Model Library). Learn more about managing your organization’s Model Library.

Back to table of contents

Creating a multi-matter model

You must be an Organization Admin to create a multi-matter model. Your current organization must also have at least one existing PC model with generated prediction scores. 

To create a multi-matter model:

• Step 1: Open the New Multi-matter Model wizard
• Step 2: Select PC Model(s) as a baseline for your new multi-matter-model.
• Step 3: Define Details about these models and the matters they were trained on. This will help organization members determine when to use the multi-matter model in future projects. 

Step 1: Open the New Multi-matter Model wizard

To open the New Multi-matter model wizard:

1. Go to the Organization Admin (OA) page and select “Multi-matter Models” to open the Multi-matter models page. 
2. Select “+ New Multi-matter Models.”
   This will bring you to the Intro page for creating a new multi-matter model.
   
3. Review the information. Then select “Next”.

In the next step, you’ll select one or more PC model(s) to use as a baseline for training your new multi-matter-model.

Back to table of contents

Step 2: Select PC Model(s)

Once on the “Select model(s)” section of the New Multi-matter Model wizard, you will see a table of all existing PC models that have generated predictions in projects across your organization. If you are not seeing a specific PC model in your table, it is likely because either (a) the PC model has not generated prediction scores or (b) the model is contained within a database where OA access is disabled. 

For each available PC model, the table provides:

•  The name of the project that the model was trained in (i.e., source project)
•  and the criteria used to train it (i.e., reviewed and relevant criteria). 

Note: Some of your organization’s existing PC models may not be immediately available for selection upon release of this feature but will eventually be available and appear in the “Select model(s)” described above.

To start creating your multi-matter model by selecting from existing PC models:

1. [optional] Narrow down your organization's existing PC models in the table by filtering on model name or project name.
2. Select one or more existing PC models to form the new multi-matter model. Continue reading to learn about approaches for selecting PC models.

Approaches for selecting PC model(s)

There are two main approaches to selecting PC models for your multi-matter model: 

• (A) Select one PC model trained to find a specific type of relevant documents in a previous legal matter. This is an excellent option if your organization does not have more than one PC model trained in similar matters.
  Or
• (B) Select multiple PC models previously trained to find the same type of relevant documents in similar matters (i.e., datasets involving the same area of law or subject matter). 

You have the option to add PC models that are trained on similar matters in the future to an existing multi-matter model.

We recommend selecting PC models from projects where review is complete or near completion, as they are generally more effective at identifying relevant documents in a given context because they are trained on more documents. However, you can select PC models from projects where review is ongoing. As more documents are reviewed in a source project and the selected PC model is further trained, the multi-matter model it is selected to be part of will be updated accordingly. New prediction scores are then generated during the next scheduled update or you can click the "Update model" button in the project to manually update sooner if you have Admin permissions on prediction models.

To get a sense of a PC model’s current training, you can click on a model in the table. This opens a panel that shows more information about the model, including the number of relevant/irrelevant documents used to train the model and any available performance metrics.

After at least one existing PC model has been selected, click “Next” to proceed to the next step: defining details about your model.

Back to table of contents

Step 3: Define Details

Once in the Details section of the New Multi-matter Model wizard, you can enter details to organize your multi-matter model and help members of the organization determine when to use it in future projects. 

Note: Aside from the name of the multi-matter model, information entered in this step will only be visible to Org Admins and organization members that have “Admin” permissions on prediction models in a given project.

The following fields can be defined:

• Name (Required)
  • The name of the multi-matter model should briefly describe what the selected model(s) are trained to find. 
  • Examples: Harassment detector, LIRR Co Privilege, Financial Documents re Bankruptcy model, Spam finder
• Type (Required) 
  • The type of relevant documents the model(s) selected for this multi-matter model is/are trained to find. 
  • Examples: Harassment, Privilege, Financial Documents, Spam
• Area of law 
  • This field can be used if model(s) selected for this multi-matter model were trained in matters within a specific area of law.
  • Examples: Employment, Corporate, Bankruptcy, General 
• Description 
  • This field can be used to provide any additional details that could help organization members understand the purpose of the multi-matter model and when to use it. 
  • Example: This model was trained to find documents relating to workplace harassment.

After filling out the  required fields,  click “Submit." You’ll be taken to your organization’s Model Library, where you can find your newly created multi-matter model. Once a multi-matter model is created, it becomes available for use in projects across the organization.

Learn more about managing your multi-matter model library. 

Back to table of contents

Accessing the Multi-matter Model library 

You must be an Organization Admin to access the Multi-matter Model library.

To open the Multi-matter Model library:

1. Go to the Organization Admin (OA) page. 
2. Select “Multi-matter Models” to open the Multi-matter Models page.

Each multi-matter model is represented by a card that indicates the name of the multi-matter model, type, area of law, and description. 

To view and manage a given multi-matter model: click the “View model” button on the associated card. This opens the individual model’s page.

The page for a given multi-matter model has the following sections:

• Usage:  Displays all the target projects in the organization where the multi-matter model is being used. Available review progress and basic performance metrics in a given target project will also be displayed.
• Selected models:  Identifies all the PC models that are currently selected for the multi-matter model. Org Admins will also be able to edit the selected PC models included as part of the multi-matter model.
• Update history:  Tracks all activities on PC models selected for the multi-matter model that have caused the multi-matter model to update (e.g., Org Admin has edited the current selection of PC models forming the multi-matter model or one of the selected PC models has been updated in its source project). When a multi-matter model is updated, all existing uses of the multi-matter model in target projects will also be updated.
• Details:  Displays all information inputted by Organization Admins to describe the multi-matter model upon creation. Org Admins will be able to edit the multi-matter model details here.

Back to table of contents

Editing selected models and details for multi-matter models

At any point after creating a multi-matter model, Organization Admins can edit the current selection of PC models and details about the multi-matter model. 

Once a multi-matter model has been edited, the changes are reflected in the next use of a model in a project. 

Selecting or deselecting models will cause a model update in all target projects where this multi-matter model is used.

If an Organization Admin selects (or deselects) an additional PC model for an existing multi-matter model used in a project, the use of the multi-matter model in that project will be updated. New prediction scores will be generated during the next scheduled update. You can also click the "Update model" button in the project to manually update sooner if you have Admin permissions on prediction models.

To edit an individual multi-matter model:

1. Go to your Multi-matter Model library.
2. Identify the multi-matter model that you want to edit, and click its "View Model" button.
3. Click the pencil icon in the top right corner.
4. Select (or deselect) additional PC models for your multi-matter model from the table. Then click “Next.”
5. Add or edit details about the multi-matter model (e.g., name, type, area of law, description). Then click “Confirm.”

Back to table of contents

Using a multi-matter model in a project 

After an Organization Admin creates a multi-matter model, it is available for use in projects within the organization. If you are a member of the organization and have “Admin” permissions on prediction models (project level permissions) in a new project, you can use the multi-matter model to generate prediction scores on documents in that project.  

To use a multi-matter model in a project:

1. From your Everlaw homepage, click on the Document Analytics icon on your top toolbar and select Predictive Coding.
   This brings you to the default PC Rating model page.
2. In the page’s left side menu, click “+New model” to open the New model wizard. 
3. Select “Use multi-matter model.”
   Note: The “Use multi-matter model” option will not appear until the organization has created at least one multi-matter model. If you see the “Use multi-matter model” option but it is grayed out and not clickable, a tooltip will indicate that you either do not have permissions to access multi-matter models in this project or or all existing multi-matter models have already been used in this project. 
4. Click “Confirm.” Your organization’s Model Library will appear containing cards that represent each created multi-matter model. Each card indicates the name and type of the multi-matter model, as well as any additional details that your Organization Admin provided. You can use this information to select the multi-matter model that is most applicable to the type of documents that you are looking for in the current project. Note: Any existing multi-matter models that have already been used in the same project will appear but are not available for selection because each multi-matter model can only be used once within the same project. 
5. After selecting a model, click “Next”.

6. Now, use the query builder to configure the criteria for documents that will be considered as reviewed, relevant, and excluded in the project. Here, you are determining what documents will be used to further train this use of the multi-matter model in the target project. Learn more about setting up criteria in this article.
7. After setting up your criteria for additional training, click “Submit.”
   You will automatically be taken to a model page that is dedicated to the use of the selected multi-matter model in your project, where the use of the multi-matter model will be queued for update and generate initial prediction scores even if no documents have been reviewed in the project.

Once initial prediction scores are generated, you can immediately jump into a prioritized review of documents that are highly predicted to be relevant based on the multi-matter model in use. 

Here, you can open a results table to review documents that are highly predicted to be relevant. You can do this one of three ways:

• Click the “Prioritize” action item to find unreviewed documents with high prediction scores. Note: Initially the “Prioritize” action item will identify documents that have prediction scores of 90 or above.
• Select documents from the distribution graph by moving the green flag to the desired cutoff and click the blue number of documents to the right of the green flag.
• Use the “Predicted” search term, select the multi-matter model from the dropdown and determine the range of prediction scores

We recommend sorting the selected documents by prediction scores in descending order before reviewing. To do this: 

1. Go to the Results Table, using one of the methods outlined above.
2. In the toolbar, select “View”
3. Then select “Add or Remove column” 
4. You’ll find all of your models listed under the Prediction Columns category. Select the relevant multi-matter model from this list. 
5. Once this column has been added to your results table, you can sort by prediction score by selecting to modify the table sort. This enables you to view your documents by predicted relevance and prioritize document review. 

After at least 200 qualified documents are reviewed (containing at least 50 relevant and 50 irrelevant) and the model update runs, prediction scores are updated based on both the multi-matter model in use and the documents reviewed within the target project. At this point, training coverage will also be generated. 

Note: Training coverage will be generated based only on documents reviewed from the target project. 

As more documents are reviewed in the target project, prediction scores will continue to improve and initial performance metrics will be generated. 

Note: Documents reviewed in a target project where a multi-matter model is being used will not affect the multi-matter model at the organization level. 

Back to table of contents

Deleting multi-matter models

Organization Admins are able to delete a multi-matter model. When a multi-matter model is deleted, it will also be deleted in all target projects where it is being used. For example, if you delete a multi-matter model (MM1) that is being used in Project A and Project B, it will also be deleted from both projects. This means that upon deletion, the model page associated with the use of the multi-matter model and any prediction scores generated by the use of the multi-matter model will also be deleted. 

To delete a multi-matter model:

1. Go to your Multi-matter model library.

2. Identify the multi-matter model that you want to delete, and click its "View Model" button.

3. Click the trashcan icon in the top right corner.

4. Review the list of target projects that the multi-matter model is currently being used in that will be deleted and confirm deletion by selecting “Delete.”

Back to table of contents

Multi-matter Models FAQs

What happens to my multi-matter model if I delete the source project or database for a selected PC model being used?

If a selected PC model is part of a multi-matter model and its source project or database is submitted for deletion, a copy of the selected model as of the last model update will be saved. This means the selected model will still be available for use in new and existing multi-matter models.

Note: documents used to train the selected PC model in its source project will not be saved.

Can I use a multi-matter model to preserve an existing PC model?

Yes! To preserve a PC model for future use, ensure that it is selected to be part of a multi-matter model before deleting the associated source project or database. If a PC model is not part of any multi-matter model when its source project or database is deleted, the model will be permanently deleted and not recoverable. 

I have a PC model in my organization, but it is not available for selection when creating a new multi-matter model. What is going on? 

If you are not seeing a specific PC model in your model selection table, it is likely because either (a) the PC model has not generated prediction scores or (b) the model is contained within a database where OA access is disabled. Note: Some of your organization’s existing PC models may not be immediately available for selection upon release of this feature but will eventually be available.

Can I create a Multi-matter Model from a target model in a suspended project?

Yes.

Back to table of contents