Uploading Processed Data with a Load File

Table of Contents


Uploading structured, processed data such as productions into Everlaw can be a complex process with many potential sources of error. This article outlines how to perform your own uploads of productions and processed data with a load file. 

If you want to upload PDFs without a load file as processed data, you can check out this article. In addition, for more complex uploads the Everlaw Customer Success team can upload your files for you. For information on how to transfer files to the Everlaw team, consult this article.

Common reasons to use the processed uploader include:

  • You received a production from another party, including a load file and images, natives, and text files. You want to upload these documents into Everlaw.
  • You already have documents on the Everlaw platform, and you want to replace images, text, natives, and/or metadata, or add additional metadata, or fill in missing files (e.g., if some single page TIFFs were initially missing in a document). You want to use the processed uploader to perform an overlay of this new or updated information.
  • You exported native data off of Everlaw to archive your case materials. The matter re-opens and you need to re-upload the data.

Please note that uploading processed data only works in the Chrome, Edge, and Firefox web browsers.

Accessing the uploader

To begin a processed upload from within a project, select the data transfer icon in the navigation bar and choose Uploads. Organization administrators can find this on the Processed Uploads tab in the Organization Administrator page. From the "+ New upload" section, you can select "Processed" and then "Document sets with load file".

The Processed Data page is divided into three tabs: ‘In progress’, ‘Complete’, and ‘Deletion in progress’. Each tab is populated with upload cards.


Each card represents a single upload to a single database. In the image below you can see the upload name highlighted in red, the database name in blue, and the type of upload in green. Please note that when viewing processed upload cards from within a project, the database name will not be listed.


At the top of the page are various filters you can use to look at uploads by database, upload name, status, and date. Filters only apply to uploads that have been loaded on the page.  You can use the “Load more” button at the top or bottom of the cards to load more cards.


Return to table of contents

Required format

The required format for uploads is relatively inflexible, so make sure that your documents conform to this standard before uploading. Documents uploaded via the processed uploader must have a load file. Please read our article on standard formats for processed data for additional information. 

Return to table of contents

Uploading processed data

Initializing the upload

If you have with database upload permission, you can begin a new processed upload by selecting the data transfer icon in the navigation bar and choosing Uploads. Next, select “Processed” and then “Document sets with load file”.


Alternatively, you can start an upload from the Processed Data page. Clicking “+ New Upload” will open a drop down menu where you can select “Upload document sets with load file.”


Next, drag and drop the unzipped folder containing the load file and image/text/native files into the uploader. This will open a dialog where you can edit your upload name, if you would like to use a different name than the name of the folder you dragged in.

If you are an Organization Admin, you can click “+ New Upload” and select "Upload document sets with load file" on the organization admin Processed Uploads tab. Next, drag and drop the unzipped folder containing the load file and image/text/native files into the uploader. This will open a dialog where you will configure your upload database, upload name, and authorizing user. This step determines which database your data gets uploaded into, so take care that you are selecting the right one. If you would like to give the upload a different name than the name of the folder you dragged in, you can change the name now. Then, select the name of the user authorizing the upload. This is a safety check to make sure that the documents are being uploaded to the correct database, and can also be used as an audit trail to see who has provided and/or approved various uploads. If there is no user who has requested the upload, you may override this step by providing an authorizing organization or a free form note. 



Load file selection

After initiating the upload, select the load file. This will often be listed under a folder called “Data.” If there are multiple load files present, it tends to be the case that the .DAT file most closely conforms to the uploader’s requirements. Otherwise, you can select a .CSV or a .TXT file. .OPT files will not work to ingest data into Everlaw. Then, select Upload Load file.


Everlaw will report errors in parsing your load file and you will have the option to proceed with your upload without uploading these problematic documents. Please note that selecting “Skip errors and continue” will NOT upload all of the documents in your processed upload, and will skip documents associated with the skipped lines.


To ensure that all of your documents get uploaded, you can troubleshoot your load file using the error reports that can be downloaded by clicking  “Download error report and load files.”


You can proceed in one of the following ways: 

Download the error report by clicking “Download error report and load files,” proceed with uploading the documents that were able to be processed, fix the “load-error-lines.DAT” load file found in the error report, and upload the rest of your documents that were skipped during the first upload, 


Download the error report, fix any errors in your original load file, click "Replace load file," and upload all of your documents in one upload.

If you would like information on troubleshooting syntax errors in your load file, read this article about processed data errors.

Return to table of contents

Identity fields

The next step involves selecting identity metadata fields. These are the special fields in your load file that Everlaw will use to identify where documents begin and end, as well as find the native documents. The uploader will make an attempt to automatically detect these fields for you. Please verify the fields that are selected to ensure that your files are uploaded correctly. To assign an identifying field to a different header, use the dropdowns on the left. 


By default, the uploader will assume that all documents have a Bates prefix, and will attempt to automatically list all detected prefixes. If your upload contains documents with no Bates prefix, or if you need to specify Bates prefixes which occur in the Begin Bates column, click the pencil icon under “Bates prefixes.” A pop-up menu will appear. To enable uploading of documents with an empty prefix, check the “UseAllow Bates with empty prefix” box in the “Edit Bates Prefixes” configuration -tool. You can additionally enter any Bates prefixes which occur in the loadfile in the data-entry box of this configuration pop-up. In instances where Everlaw is unable to detect prefixes automatically, specifying Bates prefixes in this way will allow the uploader to correctly identify the begin Bates and end Bates columns of the loadfile.


The identifying fields are listed below:

  • Begin Bates
    • This field is always required.
    • The identifier for the first page of your documents.
    • This is designated by selecting “B” next to the metadata field that will be used.
    • It may be called “BeginBates,” “BegDoc,” “ProdBegBates,” or any of several other potential options.
    • The uploader may accidentally pick up a different Bates value (such as ParentBates or Beginfamily), which must be corrected.
  • End Bates
    • The identifier for the last page of your documents.
    • Either this field or “Page Count” are required so that the system knows where the end of the document is. End Bates is preferred over Page Count because it is less error prone.
    • This field may have multiple potential names, like “EndBates” or “EndDoc.”
  • Page Count
    • The length, in pages, for the image of your document.
    • If there is no End Bates value, Page Count can be used to determine the end of documents. Use the drop down under Page Count to select the correct header. You can also select both page count and End Bates. If the uploader prompts you for an .OPT file, that is an indication that the two values do not match up.
  • Native Path
    • The location of the file within the directory provided.
    • This field will often be called "Native Link" or "Doc Link" and will contain text that looks like a file path and includes the word "Native", e.g., ENRON PROD009\ENRON PROD009\VOL0001\NATIVES
    • This field is necessary if you would like to upload native files, even if those files have identical names to the image.

In this step, you can also select to ignore certain metadata fields, like “Page Count” as mentioned above. To ignore a field, deselect its entry in the table. The uploader will automatically ignore all fields that have no values for any of the documents. If you do choose to include a field with no values, the uploader will assume you are performing an overlay and want to delete metadata (all values will be deleted from that field for documents overlaid by upload). For more information on deleting metadata, read the Metadata deletion section of this article.

Make sure that you always ignore the text path field, which could be called “Text Link” or “OCR Path.” This is because Everlaw doesn’t use text path to upload text, and the value on its own has little purpose.


If you are not able to assign the fields you want to in this step, it likely is because there is something wrong with the formatting in the load file. You can find out what the problem is by hovering over the header to see what percent of the column could not be parsed as the desired field.


If your upload contains a Bates prefix that does not exist on your database, you will need to confirm the addition of the new prefix. This acts as a safeguard to ensure that you are uploading to the correct database.


If your upload contains a # prefix, you will be prompted to change the # prefix to a valid Bates prefix. Click "Assign Bates prefix" in the prompt.


In the Assign Bates prefix dialog, choose an existing prefix, or enter a new one. Then click Assign.


Once you have assigned a Bates prefix, the document numbering is retained. For example, if the load file specifies a Begin Bates of '#123.1' and you assign the prefix 'ABC-', then the Bates number of the uploaded document will be 'ABC-123.1'. (Note that when uploading an Everlaw export, your work product CSV file will still reference # prefixes; you can update your work product file to use the newly assigned prefix. Contact support with any questions on updating your work product file before transferring work product.)

If you choose to proceed with a # prefix, the uploader will assume you are uploading an overlay, and will error in the following step if it cannot find an existing document with the same control number to overlay.

Return to table of contents

Load file errors

Everlaw will report inconsistencies in your Identity fields, and you will have the option to proceed with your upload without uploading these problematic documents. Please note that selecting “Skip errors and continue” will NOT upload all of the documents in your processed upload, and will skip documents associated with the skipped lines.


To ensure that all of your documents get uploaded, you can troubleshoot your load file using the error reports that can be downloaded by clicking “Download error report and load files.”


Again, you can proceed in one of the following ways: 

Download the error report by clicking “Download error report and load files,” proceed with uploading the documents that were able to be processed, fix the “load-error-lines.DAT” load file found in the error report, and upload the rest of your documents that were skipped during the first upload, 


Download the error report, fix any errors in your original load file, click "Replace load file," and upload all of your documents in one upload. 

If you would like information on troubleshooting malformed information errors in your load file, read this article about processed data errors.

With respect to documents with overlapping Bates numbers identified in this step, you can proceed in one of the following ways:

Proceed with uploading non-overlapping documents, rename the skipped documents so that their Bates numbers no longer overlap, and upload the renamed documents that were previously skipped during the first upload,


Navigate to the Database Settings page, enable the “Overlapping Bates” setting within the “General” tab, return to the Uploads page, click “Processed Data” from the left-side menu, find the relevant card for your upload in progress, click “Configure upload,” and click the “Try again” button appearing in the yellow banner. Because you enabled the Overlapping Bates setting, you will be able to ignore the warnings relating to overlapping Bates ranges and continue to upload these documents.

If you would like more information on overlapping Bates, please see this article on the Overlapping Bates setting and this article explaining different overlapping Bates errors.

Return to table of contents

Metadata deletion

There is one operation under defining Identity fields that is only relevant when performing an overlay: deleting metadata values. If you would like to remove a metadata value from all the documents listed in your load file, you can use a load file with no values in that field. Though the platform defaults to ignoring empty fields, you can tell the platform that you’d rather delete existing values from that field by including the empty field in the Identify fields table. It will then show up under “Delete values from” on the left-hand side of the uploader.


If you have chosen to delete a field’s values, you will be taken to the Metadata deletion step. You will be asked to map the load file field to a metadata field in your database and to confirm the deletion.


Please note that if you elect to delete metadata values, the previous entries will not be available on the site or recoverable by Everlaw. Only select this option if you are sure you would like to remove all metadata values in a given field for the documents in your load file.

Return to table of contents

Overlays and mixed uploads

After you set the identity fields, Everlaw will then read the load file and determine the documents in your upload.

If your upload contains documents that already exist in your database, you will be prompted to perform an overlay. An overlay lets you replace images, text, natives, and/or metadata, or add additional metadata, or fill in missing files. Performing an overlay maintains any review work applied to the documents (such as codes, notes, and ratings). Use caution, however: overlays cannot be deleted or undone.

In this screen you can choose only to upload the new documents found in the load file, only overlay the existing documents found, or to do both. If you choose to overlay documents that already exist on the platform, you will be asked what formats of the documents you would like to overlay. Natives will only be an available option if a Native Path column was identified in the previous step.


If your load file references both new and existing documents, it is considered a mixed upload. Everlaw will automatically split the new and existing documents from your load file into two new load files which you can download from the “Download split load files” button associated with the upload. These downloads will also be available under the three-dot icon once the upload is complete.

Note on overlapping documents: If you get an error at this step letting you know there are overlapping documents, it means the upload contains entries that would overwrite several documents. You can export a CSV of the conflicts by clicking the three-dot menu on the upload card and clicking “Download current stage error report”. Please note when the “Overlapping Bates” setting is enabled and you choose to overlay images, any single-page image files (TIFF, JPEG, etc.) in the selected image folders may be assigned to the wrong documents because overlapping Bates are allowed for the database. Learn more about the “Overlapping Bates” setting.

Overlays can change the page length of documents, but they must continue to have a unique Bates number for each page. Please review the overlay to confirm that this is the case before attempting the upload again.


Return to table of contents

Metadata type and format

In this step, you determine which data type the uploader should use to parse each metadata field. For example, if there is a “File Size” field with contents that are just numbers, you can choose whether the uploader should understand it as a text string or a numerical value. If you select the latter, you will be able to do searches on number ranges (e.g., a search for file size larger than 100). The uploader will only allow you to select a field as numerical if there are only numbers and no letters in any of the values. For instance, a value of “100kb” would prevent the uploader from allowing you to select the field as numerical. Of particular importance in this step are

1. Making sure that the uploader recognizes date fields as dates, not as text. This will enable you to perform date range searches.

2. Listing the To, From, and CC fields as AddressList type and the From field as AddressFrom. This will enable you to use advanced conversational search functionality.


3. Listing any Bates fields, including Begin Family, End Family, Other Bates, Parent ID, and Produced From, as Bates type. This will enable you to search over these Bates fields by prefix and range.


Each metadata field has an icon that will reflect whether the uploader is confident with the designation of field type.

  • Green check mark: The uploader is confident that the field is parsed correctly. You should verify that Everlaw did in fact make the proper assessment.
  • Yellow exclamation point: This icon specifically means that the field has more than one non-text format option, such as number and datetime. Usually these are correctly parsed, but it’s advisable to review them to make sure the data seems to be in the proper format. Fields you have not reviewed will appear as white. Once you have clicked on a field, its color will change to a muted grey hue. Sometimes fields containing numerical values, e.g., "Attachment Count,” may need to be changed from text to number format.UFUmetadataYellow.png
  • Red X: This means there has been an error. There is at least one value that cannot be parsed. To select this format, you will either have to create a new date field as described below, or mark the values that do not conform to the specified format as “Null.” This will ignore that particular value and will allow the uploader to move to the next step.

Parsing Datetime Fields

Everlaw will automatically detect and parse various datetime formats contained in your load file. Parsing the field takes the values and ensures that Everlaw correctly recognizes the date. For example, a date given like 2001/12/30 would be parsed such that the system recognizes the year as 2001, the month as December, and the day as the 30th. Since dates can be ambiguous, based on your login region Everlaw will prefer either a Month-Day (US) or Day-Month (Non-US) ordering. This preference can be changed for a given upload by clicking the gear icon in the top right of the metadata type selector and selecting “Change date order to DMY” or “Change date order to MDY”.


You can also view and edit the datetime formats Everlaw used to parse your datetime values by clicking on the gear icon and selecting “Preferred formats.” This will present you with a dialog showing samples from your load file, the current format being used by Everlaw, and a sample result for what that returns. If you notice a date is incorrectly interpreted, you can select a new format in the ‘Choose format’ column. If you select a new format, clicking on the ‘Test format whitelist’ button will populate the ‘New result’ and ‘Error’ columns with the expected results. All samples must have valid results in order to confirm and reanalyze the entire load file. If the datetime format you would like to select does not appear in the list, please contact Everlaw Support to have it added to the list.


Return to table of contents

Combining DateTime metadata

In this step, you can combine date and time fields so that they are present as one field instead of two. For example “Date Created” and “Time Created” could be combined into one field. The uploader generally does this automatically, in which case the combined fields are shown with a green check mark. If you would like to split them anyway, select “Split” after clicking on the values. If the uploader is not confident, they will appear with a yellow exclamation point.


If a field cannot be combined with anything else, Everlaw will let you know that this column cannot be combined with any others. If fields that seem like they should be combined are displaying this message, you may need to return to the previous section, and choose different data types.

Return to table of contents

Metadata mapping

In this step, you match the metadata fields in the load file (present in the second column of the table) with the metadata field that will be displayed in Everlaw. There are two types of metadata fields on the Everlaw platform: standard and custom fields.


Standard fields are the curated list of fields we have enumerated to try to limit the proliferation of redundant metadata fields. For example, if multiple parties are producing documents that are present in your database, you could have fields called “Begin Family,” “BeginAttach,” “Begin Attachment,” and “BegFamily” that all actually reference the same underlying field. To facilitate searching all the fields at once, as well as making the data in your database easier to read, you should pick Everlaw’s standard “Begin Family” field if any of those are present. You can tell if a field is standard because it will have a green check mark to the left of the field’s name.


Custom fields are fields that are not present in Everlaw’s standard field list. In general, you want to map all fields to standard fields to the degree possible. In other words, you want as many rows as possible to have a green check mark. This ensures that your database remains easy to search and metadata fields are easy to understand.

There are some fields that do not have a good standard field equivalent, however, such as “Request number” or “Req. No.,” though you will still want to preserve them in the database. In those cases, you can select a custom field. Metadata that is being mapped to a custom field will have a yellow exclamation point icon to the left of their name. 


You can change what metadata field you are mapping to in the Everlaw display name dropdown. To avoid all of the problems with using multiple custom fields, you should make sure that you are using the same custom field each time. This means using “Request number” consistently instead of alternating between that and “Req. No.” Custom fields that have been previously used on this database will have a grey band next to them, whereas new custom fields that have not been used on this database will have a yellow band.


One important thing to remember about this section is that though it does display example metadata values in rows and columns, the documents aren’t necessarily the same across rows. That means that the entry in the third column of the “Custodian” field is not necessarily the  same document as the one in the third column of the “Duplicate Custodian” field, even though they are right on top of each other. The best way to verify what metadata values correspond to a particular document is always to check the load file.

Return to table of contents

Resolving conflicting values

In this step, you resolve inconsistencies among the metadata fields you mapped in the previous step. For example, if you have mapped multiple load file fields to the same metadata field in Everlaw, but the load file document does not have all identical values, a conflict will appear and the box will be red. You can choose to go back to the previous step to map them to different metadata fields, save both values, or delete the conflicting values.


If you choose to save the conflicting values, the values on the left will be the ones shown on the document. If the leftmost column is missing a value, the value in the column to its right will be displayed on the document. The value that will be displayed on the document will have a green check mark next to it. You can rearrange these columns by dragging them into the order you prefer.


In the example shown above, 45.eml will be displayed on the document. 'eml' will be saved and will also be searchable. 'eml' will be shown if you hover over the metadata field while looking at the document.

Values that do not have a green check mark will be present when hovering your mouse over the field you ultimately choose to save in the metadata panel in the review window, and when doing searches on that value. If you choose to delete the conflicting value, it will not be present at all in the database.

If the fields have many of the same values, but some values are only present in one field but not the other, the field will have a yellow exclamation point icon. In many cases, this means you can combine them, as they reflect the same data. In other cases, it could represent that one field is a superset of another - for example, “Custodian” and “All Custodians” will have a good deal of overlap, but should not necessarily be combined.

For overlays and mixed uploads

For overlays and mixed uploads, the uploader will provide you with the previous mapping of each field. Importantly, mapping a load file metadata field to a different metadata field on Everlaw will delete the values in the old field for documents in current upload to migrate the values over. This is because the system does not want to store extraneous data, and believes that if you have remapped it, there is a good reason to have done so. If you do not want the old values to be removed, go to the previous step and re-map the metadata to the same field that it was previously mapped to.

You will only be able to move on to the next step once you have resolved all issues.

Return to table of contents


This step gives you a summary of your configuration. This includes any conflicts, new fields that are being added, or custom (non-standard) fields being added to the database. If you would like to change any of these fields, go back to the previous steps and adjust them.

Select folders

In this step, you select the folders that will be used to upload each type of document: image, text, and, if a native link is present, native. You can click a folder in the file navigator on the right to open it. This allows you to navigate your folders. You will first be prompted for the Images folder. You can select multiple folders at once and they will be added to the panel on the left. If you would like to clear your folder selection, click Clear all on the bottom left of the panel.

When you are satisfied with your selected folder, click “Select folders” to proceed. Select your text folders in the same way. If you have natives, click “Select folders” to specify those folders as well. When you are done specifying your folders, click “Verify files” to continue with your upload.


Please note that the uploader will give you an error if you point it to two files with the same name. This could happen if there are native PDFs or other image files in the native directory and you pick a top level directory that includes both the images and the native directory. The solution to this is to make sure that images and native documents are in separate folders and to select only those folders during this step.

Return to table of contents

Verify files

In this step, the uploader will first go through all of the files and verify that they are present.  If any are missing, you will receive the following error screen and you will be presented with a number of options: reselect your folders, continue anyway, download a missing files report, or generate the missing files.




If you choose Reselect folders, you will be taken to the previous step, where you can change your folder selection. If you choose Continue anyway, you will proceed with your upload despite the missing files. Should you want to start your upload anew with all of the necessary files, you can select “Download missing files report” to export a CSV summary of files that you are missing. Finally, you have the option to generate missing files. If you select “Generate missing files”, you can specify which files you would like to generate and what documents you would like to create PDFs for. These files will be generated after the documents in your load file have finished uploading. To generate text, Everlaw will first try to extract the text. If no text is found, the natives will be OCRed to generate text files. Please note that this will upload and then overwrite any existing images/PDFs, potentially discarding Bates stamps and other features. Please read this article for more information on the files generated and Everlaw’s reprocessing.


After this step, the upload will occur, and you can watch as the system goes through all of the files. As the upload goes on, documents are indexed as they are ingested into the system. That means you will begin to see documents in the database before the upload is complete. While the upload continues, do not navigate away from the Processed Data tab as that will interrupt the file transfers.

Once the upload has completed, the upload card will be available in the Complete tab of the Processed Data page. You can click on the number on the upload card to open a search of the documents in the results table. 


You can click the three-dot menu on an upload card to download the load files that were used to configure this upload, as well as download any error reports generated during the upload. You can also delete the upload by clicking on the trashcan.

Deleting the upload will remove the documents from the database and all its associated projects, not just delete the uploads table entry. Deleting a mixed upload will only delete the new documents that were created by that upload. Modifications to documents will remain.


Return to table of contents

[Organization Administrator] Associate documents with source card

When users transfer zip files of processed documents to the platform via the File Transfers page, a file transfer card is created for the source.

UFUFileTransferForUpload.pngOrganization admins can associate processed data cards with their original file transfer source(s). To do this, navigate to the completed upload card and click “View results.” Here, you will see a summary of the upload as well as a “Processed Sources” drop down. Choose the file transfer source from the dropdown and click “Update Processed Sources”. The selected processed source(s) will appear under “Sources to Assign.”


Associating uploads with their file transfer source will automatically mark the upload complete on the file transfer card. Additionally, the card will have a document icon linking users to the uploaded documents.


Return to table of contents

OPT Files

An OPT file is an explicit listing of the name for each page of a document, including the filename of the image for that page. It is used to determine the names of image files when the name cannot be determined implicitly from the document’s Bates range and page count.

An OPT file is required whenever there is an inconsistency between the number of pages indicated by the Page Count column and the implicit number of pages described by the Bates range. For example, load file record ABCD001-ABCD010 has a page count of 12 whereas the count implied by the Bates range is 10. An OPT file will be required to rectify the inconsistency. There are several possible reasons for the apparent or actual inconsistency:

  • The Page Count column is inaccurate (rather than provide an OPT file you could ignore the Page Count column).
  • Some pages have non-obvious names, e.g., the pages go ABCD001, ABCD002, ABCD002.2, ABCD002.3, ABCD003, ABCD004, … ABCD010. In this case the Bates range is incorrect and the page count of 12 is correct. The OPT file is needed to determine the name for each image. This problem may not be obvious without inspecting the images.

If the Page Count is correct but the Bates range is incorrect, an OPT file won’t fix the problem. You will need to correct the Bates range in the loadfile.

Another case requiring an OPT file is whenever the names of a document’s images cannot be inferred from the Bates range. For example:

  • ABCD001-ABCD004.4 (some pages use a page separator, others don’t)

OPT files are currently not supported for uploads using PDFs as the image format.

For common errors when uploading documents, please see our help documentation.


Return to table of contents

Have more questions? Submit a request


Article is closed for comments.