Table of Contents
- Overview
- Accessing the uploader
- Required format
- Uploading processed data
- OPT files
Overview
Uploading structured, processed data such as productions into Everlaw can be a complex process with many potential sources of error. This article outlines how to perform your own uploads of productions and processed data with a load file.
If you want to upload PDFs without a load file as processed data, you can check out this article. In addition, for more complex uploads the Everlaw Customer Success team can upload your files for you. For information on how to transfer files to the Everlaw team, consult this article.
Common reasons to use the processed uploader include:
- You received a production from another party, including a load file and images, natives, and text files. You want to upload these documents into Everlaw.
- You already have documents on the Everlaw platform, and you want to replace images, text, natives, and/or metadata, or add additional metadata, or fill in missing files (e.g., if some single page TIFFs were initially missing in a document). You want to use the processed uploader to perform an overlay of this new or updated information.
- You exported native data off of Everlaw to archive your case materials. The matter re-opens and you need to re-upload the data.
Please note that uploading processed data only works in the Chrome, Edge, and Firefox web browsers.
Accessing the uploader
To begin a processed upload from within a project, select the data transfer icon in the navigation bar and select Uploads from the dropdown menu. Organization administrators can upload processed data through the Processed Uploads tab on the Organization Administrator page.
The Processed Data page is divided into three tabs: ‘In progress’, ‘Complete’, and ‘Deletion in progress’. Each tab is populated with upload cards.
Each card represents a single upload to a single database. In the image below you can see the upload name highlighted in red, the database name in blue, the type of processed data in orange, and the type of upload in green. Please note that when viewing processed upload cards from within a project, the database name will not be listed.
At the top of the page are various filters you can use to look at uploads by database, upload name, status, and date. Filters only apply to uploads that have been loaded on the page. You can use the “Load more” button at the top or bottom of the cards to load more cards.
Required format
The required format for uploads is relatively inflexible, so make sure that your documents conform to this standard before uploading. Documents uploaded via the processed uploader must have a load file. Please read our article on standard formats for processed data for additional information.
Uploading processed data
Initializing the upload
If you have database upload permissions, you can begin a new processed upload by selecting the data transfer icon in the navigation bar and choosing Uploads from the dropdown menu. Next, select “Processed” and then “Document sets with load file.”
Alternatively, you can start an upload from the Processed Data page. Clicking “+ New Upload” will open a drop down menu where you can select “Upload document sets with load file.”
Next, drag and drop the unzipped folder containing the load file and image/text/native files into the uploader. This will open a dialog where you can edit your upload name, if you would like to use a different name than the name of the folder you dragged in.
If you are an Organization Admin, you can start an upload from the upload page or the Organization Admin Processed Uploads tab. On the Organization Admin page from the Processed Uploads tab, clicking “+ New Upload” will open a dropdown menu where you can select “Upload document sets with load file." Next, drag and drop the unzipped folder containing the load file and image/text/native files into the uploader. This will open a dialog where you will configure your upload database, upload name, and authorizing user. This step determines which database your data gets uploaded into, so take care that you are selecting the right one. If you would like to give the upload a different name than the name of the folder you dragged in, you can change the name now. Then, select the name of the user authorizing the upload. This is a safety check to make sure that the documents are being uploaded to the correct database, and can also be used as an audit trail to see who has provided and/or approved various uploads. If there is no user who has requested the upload, you may override this step by providing an authorizing organization or a free form note.
When creating a processed upload from the project upload page instead of the Organization Admin page, Organization Admins have the additional option of making the upload visible or not on the homepage. Additionally, Org Admins can choose to merge the new upload with an existing upload.
Load file selection
After initiating the upload, select the load file. This will often be listed under a folder called “Data.” If there are multiple load files present, it tends to be the case that the .DAT file most closely conforms to the uploader’s requirements. Otherwise, you can select a .CSV or a .TXT file. .OPT files will not work to ingest data into Everlaw. Then, select Upload Load file.
Everlaw will report errors in parsing your load file and you will have the option to proceed with your upload without uploading these problematic documents. Please note that selecting “Skip errors and continue” will NOT upload all of the documents in your processed upload, and will skip documents associated with the skipped lines.
To ensure that all of your documents get uploaded, you can troubleshoot your load file using the error reports that can be downloaded by clicking “Download error report and load files.”
You can proceed in one of the following ways:
Download the error report by clicking “Download error report and load files,” proceed with uploading the documents that were able to be processed, fix the “load-error-lines.DAT” load file found in the error report, and upload the rest of your documents that were skipped during the first upload,
or,
Download the error report, fix any errors in your original load file, click "Replace load file," and upload all of your documents in one upload.
If you would like information on troubleshooting syntax errors in your load file, read this article about processed data errors.
Identity fields
The next step involves selecting identity metadata fields. These are the special fields in your load file that Everlaw will use to identify where documents begin and end, as well as find the native documents. The uploader will make an attempt to automatically detect these fields for you. Please verify the fields that are selected to ensure that your files are uploaded correctly. To assign an identifying field to a different header, use the dropdown menus on the left.
By default, the uploader will assume that all documents have a Bates prefix, and will attempt to automatically list all detected prefixes. If your upload contains documents with no Bates prefix, or if you need to specify Bates prefixes which occur in the Begin Bates column, click the pencil icon under “Bates prefixes.” A pop-up menu will appear. To enable uploading of documents with an empty prefix, check the “Allow Bates with empty prefix” box in the “Edit Bates Prefixes” configuration tool and click "Reanalyze." You can additionally enter any Bates prefixes which occur in the loadfile in the data-entry box of this configuration pop-up. In instances where Everlaw is unable to detect prefixes automatically, specifying Bates prefixes in this way will allow the uploader to correctly identify the begin Bates and end Bates columns of the loadfile.
The identifying fields are listed below:
- Begin Bates
- This field is always required.
- The identifier for the first page of your documents.
- This is designated by selecting “B” next to the metadata field that will be used.
- It may be called “BeginBates,” “BegDoc,” “ProdBegBates,” or any of several other potential options.
- The uploader may accidentally pick up a different Bates value (such as ParentBates or Beginfamily), which must be corrected.
- End Bates
- The identifier for the last page of your documents.
- Either this field or “Page Count” are required so that the system knows where the end of the document is. End Bates is preferred over Page Count because it is less error prone.
- This field may have multiple potential names, like “EndBates” or “EndDoc.”
- Page Count
- The length, in pages, for the image of your document.
- If there is no End Bates value, Page Count can be used to determine the end of documents. Use the drop down under Page Count to select the correct header. You can also select both page count and End Bates. If the uploader prompts you for an .OPT file, that is an indication that the two values do not match up.
- Native Path
- The location of the file within the directory provided.
- This field will often be called "Native Link" or "Doc Link" and will contain text that looks like a file path and includes the word "Native", e.g., ENRON PROD009\ENRON PROD009\VOL0001\NATIVES
- This field is necessary if you would like to upload native files, even if those files have identical names to the image.
In this step, you can also select to ignore certain metadata fields. To ignore a field, deselect its entry in the table.
The uploader will automatically ignore all fields that have no values for any of the documents. If you do choose to include a field with no values, the uploader will assume you are performing an overlay and want to delete metadata (all values will be deleted from that field for documents overlaid by upload). For more information on deleting metadata, read the Metadata deletion section of this article.
Make sure that you always ignore the text path field (Textpath), which could be called “Text Link” or “OCR Path.” This is because Everlaw doesn’t use text path to upload text, and the value on its own has little purpose.
If you are not able to assign the fields you want to in this step, it likely is because there is something wrong with the formatting in the load file. You can find out what the problem is by hovering over the header to see what percent of the column could not be parsed as the desired field.
On the Identity fields page, the Bates prefixes detected in your upload will be listed under “Bates prefixes” on the left. New Bates prefixes (prefixes that do not exist on your database) are identified with a preceding (!) icon.
If your upload contains a new Bates prefix, you will need to confirm the addition of the new prefix before you proceed with your upload. Here, a dialog box will appear that identifies any new prefixes contained within the upload and the name of the current database.
Click the checkbox within the dialog box to confirm that this new prefix should be added to the current database. Once the checkbox is clicked, the “Add prefixes” button is no longer grayed out and you can click it to proceed with your upload. This process acts as a safeguard to ensure that you are uploading to the correct database.
Note: In general, the dialog box to confirm new prefixes will not appear on the first upload to a new database. However, when uploading processed data with new prefixes through the Organization Admin page, the dialog box prompting approval for new prefixes appears for all uploads (including the first upload).
If your upload contains a # prefix, you will be prompted to change the # prefix to a valid Bates prefix. Click "Assign Bates prefix" in the prompt.
In the Assign Bates prefix dialog, choose an existing prefix, or enter a new one. Then click Assign.
Once you have assigned a Bates prefix, the document numbering is retained. For example, if the load file specifies a Begin Bates of '#123.1' and you assign the prefix 'ABC-', then the Bates number of the uploaded document will be 'ABC-123.1'. (Note that when uploading an Everlaw export, your work product CSV file will still reference # prefixes; you can update your work product file to use the newly assigned prefix. Contact support with any questions on updating your work product file before transferring work product.)
If you choose to proceed with a # prefix, the uploader will assume you are uploading an overlay, and will error in the following step if it cannot find an existing document with the same control number to overlay.
Load file errors
Everlaw will report inconsistencies in your Identity fields, and you will have the option to proceed with your upload without uploading these problematic documents. Please note that selecting “Skip errors and continue” will NOT upload all of the documents in your processed upload, and will skip documents associated with the skipped lines.
To ensure that all of your documents get uploaded, you can troubleshoot your load file using the error reports that can be downloaded by clicking “Download error report and load files.”
Again, you can proceed in one of the following ways:
Download the error report by clicking “Download error report and load files,” proceed with uploading the documents that were able to be processed, fix the “load-error-lines.DAT” load file found in the error report, and upload the rest of your documents that were skipped during the first upload,
or,
Download the error report, fix any errors in your original load file, click "Replace load file," and upload all of your documents in one upload.
If you would like information on troubleshooting malformed information errors in your load file, read this article about processed data errors.
With respect to documents with overlapping Bates numbers identified in this step, you can proceed in one of the following ways:
Proceed with uploading non-overlapping documents, rename the skipped documents so that their Bates numbers no longer overlap, and upload the renamed documents that were previously skipped during the first upload,
or,
Navigate to the Database Settings page, enable the “Overlapping Bates” setting within the “General” tab, return to the Uploads page, click “Processed Data” from the left-side menu, find the relevant card for your upload in progress, click “Configure upload,” and click the “Try again” button appearing in the yellow banner. Because you enabled the Overlapping Bates setting, you will be able to ignore the warnings relating to overlapping Bates ranges and continue to upload these documents.
If you would like more information on overlapping Bates, please see this article on the Overlapping Bates setting and this article explaining different overlapping Bates errors.
Metadata deletion
There is one operation under defining Identity fields that is only relevant when performing an overlay: deleting metadata values. If you would like to remove a metadata value from all the documents listed in your load file, you can use a load file with no values in that field. Though the platform defaults to ignoring empty fields, you can tell the platform that you’d rather delete existing values from that field by including the empty field in the Identify fields table. It will then show up under “Delete values from” on the left-hand side of the uploader.
If you have chosen to delete a field’s values, you will be taken to the Metadata deletion step. You will be asked to map the load file field to a metadata field in your database and to confirm the deletion.
Please note that if you elect to delete metadata values, the previous entries will not be available on the site or recoverable by Everlaw. Only select this option if you are sure you would like to remove all metadata values in a given field for the documents in your load file.
Overlays and mixed uploads
After you set the identity fields, Everlaw will then read the load file and determine the documents in your upload.
If your load file references both new and existing documents, it is considered a mixed upload. Everlaw will automatically identify and split out new documents from existing documents, allowing you to act on each set separately during the "Found existing documents" step of the upload wizard.
At this step, you can choose to perform any combination of the following actions:
- Download split load files associated with the new and existing set of documents. These split load files are also available for download on the upload card once the upload is complete.
- Proceed with the upload of the new documents
- Overlay any combination of text, image, and native files or metadata onto the existing documents. An overlay allows you to replace images, texts, natives, or metadata for existing documents in your database while maintaining any review product applied to those documents. You can also use overlays to fill in missing data. For more information, see this article about overlays.
Note on overlapping documents: If you get an error at this step letting you know there are overlapping documents, it means the upload contains entries that would overwrite several documents. You can export a CSV of the conflicts by clicking the three-dot menu on the upload card and clicking “Download current stage error report.” Please note when the “Overlapping Bates” setting is enabled and you choose to overlay images, any single-page image files (TIFF, JPEG, etc.) in the selected image folders may be assigned to the wrong documents because overlapping Bates are allowed for the database. Learn more about the “Overlapping Bates” setting.
Overlays can change the page length of documents, but they must continue to have a unique Bates number for each page. Please review the overlay to confirm that this is the case before attempting the upload again.
Metadata type and format
In this step, you determine which data type the uploader should use to parse each metadata field. For example, if there is a “File Size” field with contents that are just numbers, you can choose whether the uploader should understand it as a text string or a numerical value. If you select the latter, you will be able to do searches on number ranges (e.g., a search for file size larger than 100). The uploader will only allow you to select a field as numerical if there are only numbers and no letters in any of the values. For instance, a value of “100kb” would prevent the uploader from allowing you to select the field as numerical. Of particular importance in this step are
1. Making sure that the uploader recognizes date fields as dates, not as text. This will enable you to perform date range searches.
2. Listing the To, CC, and BCC fields as AddressList type and the From field as AddressFrom. This will enable you to use advanced conversational search functionality.
3. Listing any Bates fields, including Begin Family, End Family, Other Bates, Parent ID, and Produced From, as Bates type. This will enable you to search over these Bates fields by prefix and range.
Each metadata field has an icon that will reflect whether the uploader is confident with the designation of field type.
- Green check mark: The uploader is confident that the field is parsed correctly. You should verify that Everlaw did in fact make the proper assessment.
- Yellow exclamation point: This icon specifically means that the field has more than one non-text format option, such as number and datetime. Usually these are correctly parsed, but it’s advisable to review them to make sure the data seems to be in the proper format. Fields you have not reviewed will appear as white. Once you have clicked on a field, its color will change to a muted grey hue. Sometimes fields containing numerical values, e.g., "Attachment Count,” may need to be changed from text to number format.
- Red X: This means there has been an error. There is at least one value that cannot be parsed. To select this format, you will either have to create a new date field as described below, or mark the values that do not conform to the specified format as “Null.” This will ignore that particular value and will allow the uploader to move to the next step.
Parsing Datetime Fields
Everlaw will automatically detect and parse various datetime formats contained in your load file. Parsing the field takes the values and ensures that Everlaw correctly recognizes the date. For example, a date given like 2001/12/30 would be parsed such that the system recognizes the year as 2001, the month as December, and the day as the 30th. Since dates can be ambiguous, based on your login region Everlaw will prefer either a Month-Day (US) or Day-Month (Non-US) ordering. This preference can be changed for a given upload by clicking the gear icon in the top right of the metadata type selector and selecting “Change date order to DMY” or “Change date order to MDY”.
You can also view and edit the datetime formats Everlaw used to parse your datetime values by clicking on the gear icon and selecting “Preferred formats.” This will present you with a dialog showing samples from your load file, the current format being used by Everlaw, and a sample result for what that returns. If you notice a date is incorrectly interpreted, you can select a new format in the ‘Choose format’ column. If you select a new format, clicking on the ‘Test format whitelist’ button will populate the ‘New result’ and ‘Error’ columns with the expected results. All samples must have valid results in order to confirm and reanalyze the entire load file. If the datetime format you would like to select does not appear in the list, please contact Everlaw Support to have it added to the list.
Combining DateTime metadata
In this step, you can combine date and time fields so that they are present as one field instead of two. For example “Date Created” and “Time Created” could be combined into one field. The uploader generally does this automatically, in which case the combined fields are shown with a green check mark. If you would like to split them anyway, select “Split” after clicking on the values. If the uploader is not confident, they will appear with a yellow exclamation point.
If a field cannot be combined with anything else, Everlaw will let you know that this column cannot be combined with any others. If fields that seem like they should be combined are displaying this message, you may need to return to the previous section, and choose different data types.
Metadata mapping
In this step, you match the metadata fields in the load file (present in the second column of the table) with the metadata field that will be displayed in Everlaw. There are two types of metadata fields on the Everlaw platform: standard and custom fields.
Standard fields are the curated list of fields we have enumerated to try to limit the proliferation of redundant metadata fields. For example, if multiple parties are producing documents that are present in your database, you could have fields called “Begin Family,” “BeginAttach,” “Begin Attachment,” and “BegFamily” that all actually reference the same underlying field. To facilitate searching all the fields at once, as well as making the data in your database easier to read, you should pick Everlaw’s standard “Begin Family” field if any of those are present. You can tell if you are mapping to an Everlaw standard metadata field because the field will have a blue band next to it in the display name dropdown.
A green check mark will appear to the left of the field's name to indicate that you are either mapping to a standard field or a custom field that already exists.
Custom fields are fields that are not present in Everlaw’s standard field list. In general, you want to map all fields to standard fields to the degree possible. In other words, you want as many rows as possible to have a green check mark. This ensures that your database remains easy to search and metadata fields are easy to understand.
There are some fields that do not have a good standard field equivalent, however, such as “Request number” or “Req. No.,” though you will still want to preserve them in the database. In those cases, you can select or create a custom field. Metadata that is being mapped to a new custom field will have a yellow exclamation point icon to the left of their name.
You can change what metadata field you are mapping to in the Everlaw display name dropdown. To avoid all of the problems with using multiple custom fields, you should make sure that you are using the same custom field each time. This means using “Request number” consistently instead of alternating between that and “Req. No.” Custom fields that have been previously used on this database will have a grey band next to them, whereas new custom fields that have not been used on this database will have a yellow band.
One thing to keep in mind is that the randomly sampled results for each field are independent as between fields. In other words, just because a value lives in the same column as another value in the "Random samples from load file" section of the tables does not mean the values are from the same document.
For example, you cannot assume that the values in the highlighted column are all from the same document. Given the nature of random sampling at the field level, they are, in fact, highly likely to be from different documents.
Resolving conflicting values
In this step, you resolve inconsistencies among the metadata fields you mapped in the previous step. For example, if you have mapped multiple load file fields to the same metadata field in Everlaw, but the load file document does not have all identical values, a conflict will appear and the box will be red. You can choose to go back to the previous step to map them to different metadata fields, save both values, or delete the conflicting values.
If you choose to save the conflicting values, the values on the left will be the ones shown on the document. If the leftmost column is missing a value, the value in the column to its right will be displayed on the document. The value that will be displayed on the document will have a green check mark next to it. You can rearrange these columns by dragging them into the order you prefer.
In the example shown above, 45.eml will be displayed on the document. 'eml' will be saved and will also be searchable. 'eml' will be shown if you hover over the metadata field while looking at the document.
Values that do not have a green check mark will be present when hovering your mouse over the field you ultimately choose to save in the metadata panel in the review window, and when doing searches on that value. If you choose to delete the conflicting value, it will not be present at all in the database.
If the fields have many of the same values, but some values are only present in one field but not the other, the field will have a yellow exclamation point icon. In many cases, this means you can combine them, as they reflect the same data. In other cases, it could represent that one field is a superset of another - for example, “Custodian” and “All Custodians” will have a good deal of overlap, but should not necessarily be combined.
For overlays and mixed uploads
For overlays and mixed uploads, the uploader will provide you with the previous mapping of each field. Importantly, mapping a load file metadata field to a different metadata field on Everlaw will delete the values in the old field for documents in the current upload to migrate the values over. This is because the system does not want to store extraneous data, and believes that if you have remapped it, there is a good reason to have done so. If you do not want the old values to be removed, go to the previous step and re-map the metadata to the same field that it was previously mapped to.
You will only be able to move on to the next step once you have resolved all issues.
Review
This step gives you a summary of your configuration. This includes any conflicts, new fields that are being added, or custom (non-standard) fields being added to the database. If you would like to change any of these fields, go back to the previous steps and adjust them.
Select folders
In this step, you select the folders that will be used to upload each type of document: image, text, and, if a native link is present, native. You can click a folder in the file navigator on the right to open it. This allows you to navigate your folders. You will first be prompted for the Images folder. You can select multiple folders at once and they will be added to the panel on the left. If you would like to clear your folder selection, click Clear all on the bottom left of the panel.
When you are satisfied with your selected folder, click “Select folders” to proceed. Select your text folders in the same way. If you have natives, click “Select folders” to specify those folders as well. When you are done specifying your folders, click “Verify files” to continue with your upload.
Please note that the uploader will give you an error if you point it to two files with the same name. This could happen if there are native PDFs or other image files in the native directory and you pick a top level directory that includes both the images and the native directory. The solution to this is to make sure that images and native documents are in separate folders and to select only those folders during this step.
Verify files
In this step, the uploader will first go through all of the files and verify that they are present. If any are missing, you will receive the following error screen and you will be presented with a number of options: reselect your folders, continue anyway, download a missing files report, or generate the missing files.
If you choose Reselect folders, you will be taken to the previous step, where you can change your folder selection. If you choose Continue anyway, you will proceed with your upload despite the missing files. Should you want to start your upload anew with all of the necessary files, you can select “Download missing files report” to export a CSV summary of files that you are missing. Finally, you have the option to generate missing files. If you select “Generate missing files”, you can specify which files you would like to generate and what documents you would like to create PDFs for. These files will be generated after the documents in your load file have finished uploading. To generate text, Everlaw will first try to extract the text. If no text is found, the natives will be OCRed to generate text files. Please note that this will upload and then overwrite any existing images/PDFs, potentially discarding Bates stamps and other features. Please read this article for more information on the files generated and Everlaw’s reprocessing.
After this step, the upload will occur, and you can watch as the system goes through all of the files. As the upload goes on, documents are indexed as they are ingested into the system. That means you will begin to see documents in the database before the upload is complete. While the upload continues, do not navigate away from the Processed Data tab as that will interrupt the file transfers.
Once the upload has completed, the upload card will be available in the Complete tab of the Processed Data page. You can click on the number on the upload card to open a search of the documents in the results table. 
You can click the three-dot menu on an upload card to download the load files that were used to configure this upload, as well as download any error reports generated during the upload. You can also delete the upload by clicking on the trashcan.
Deleting the upload will remove the documents from the database and all its associated projects, not just delete the uploads table entry. Deleting a mixed upload will only delete the new documents that were created by that upload. Modifications to documents will remain.
[Organization Administrator] Associate documents with source card
When users transfer zip files of processed documents to the platform via the File Transfers page, a file transfer card is created for the source.
Organization admins can associate processed data cards with their original file transfer source(s). To do this, navigate to the completed upload card and click “View results.” Here, you will see a summary of the upload as well as a “Processed Sources” drop down. Choose the file transfer source from the dropdown and click “Update Processed Sources”. The selected processed source(s) will appear under “Sources to Assign.”
Associating uploads with their file transfer source will automatically mark the upload complete on the file transfer card. Additionally, the card will have a document icon linking users to the uploaded documents.
OPT Files
An OPT file is an explicit listing of the name for each page of a document, including the filename of the image for that page. It is used to determine the names of image files when the name cannot be determined implicitly from the document’s Bates range and page count.
An OPT file is required whenever there is an inconsistency between the number of pages indicated by the Page Count column and the implicit number of pages described by the Bates range. For example, load file record ABCD001-ABCD010 has a page count of 12 whereas the count implied by the Bates range is 10. An OPT file will be required to rectify the inconsistency. There are several possible reasons for the apparent or actual inconsistency:
- The Page Count column is inaccurate (rather than provide an OPT file you could ignore the Page Count column).
- Some pages have non-obvious names, e.g., the pages go ABCD001, ABCD002, ABCD002.2, ABCD002.3, ABCD003, ABCD004, … ABCD010. In this case the Bates range is incorrect and the page count of 12 is correct. The OPT file is needed to determine the name for each image. This problem may not be obvious without inspecting the images.
If the Page Count is correct but the Bates range is incorrect, an OPT file won’t fix the problem. You will need to correct the Bates range in the loadfile.
Another case requiring an OPT file is whenever the names of a document’s images cannot be inferred from the Bates range. For example:
- ABCD001-ABCD004.4 (some pages use a page separator, others don’t)
OPT files are currently not supported for uploads using PDFs as the image format.
For common errors when uploading documents, please see our help documentation.
0 Comments