Table of Contents
- What is the uploads feature?
- Do I have access to upload documents?
- Accepted native data types
- Accepted processed data types
- Preparing your native data for upload
- Preparing split archive for upload
- Migrating data from another platform into Everlaw
- Creating a native data upload
- Managing native uploads
- Updating an existing native upload
- Creating and managing processed data uploads
- Standard format for processed data
- Searching for uploaded documents
- Troubleshooting upload errors
- How Everlaw handles metadata
- Notes on processing defaults
What is the uploads feature?
Everlaw’s uploads feature allows you to add new data to your database at any time of your choosing. If you are a case admin and part of the ‘uploaders’ group, you can access the uploads page via the admin icon in the navigation bar.
The uploads page accepts:
- Native Files: Everlaw has a cloud-processing system that will automatically process native files for ingestion into the platform. This includes de-NISTing, deduping, and generating text, metadata, and images for your documents.
- Processed Files: You can also use the upload page to transfer processed files directly to Everlaw. Processed files can be either Bates-stamped or non-Bates stamped. Unlike native files, Everlaw support staff will need to put processed files through an intermediary review stage before uploading them into the case.
Once documents are successfully uploaded, they will be searchable and viewable in your case. You can also easily find documents by upload on the homepage under the Document Sets column, by searching for them, or by using the history cards in the uploads page. Please note the following:
- For native uploads: A native upload can contain one file, or multiple files. You can also retroactively upload new documents to an existing upload (This is useful if you want to group related documents together. See the “Preparing your native data to upload” section for more information). Uploaded documents go through three stages of processing before being added to your case: examine, PDF, and text. For more information about native uploads, see the native uploads section.
- For processed uploads: A processed data upload goes through three phases: (i) the successful transmission of the data to Everlaw, (ii) the completed processing of the data on the backend, and (iii) the uploading of the transmitted data into the platform. You can keep tabs on what stage an upload is at via the appropriate card on the “Processed Data” page. For more information about processed data uploads, see the processed uploads section.
Do I have access to upload documents?
You must gain access to the "Uploader" permissions group to upload documents. Admins can request access by reaching out an Everlaw support member, by emailing email@example.com, or by calling 1-844-383-7529 during normal support hours. You will not have upload permissions by default.
To add users to the "Uploaders" group:
- Navigate to the users section of the case settings page
- Click the gear icon in the "Groups" column of the user you want to give uploading privileges to.
- Select the "Uploaders" group.
Remember, the user must also be part of a user group with administrative privileges in order to have access to the uploads feature.
If you are in the Uploaders user groups, you are able to upload documents into any partial project that you are part of. In addition, uploads will automatically be added to all complete projects from the same primary database, even if you are not on that project.
If you are considering whether or not to authorize uploading privileges, be aware that anyone in the Uploaders user group will be able to see the names of all complete projects originating from a parent project, even those they are not part of. However, they will only be able to access projects they are on.
Accepted native data types
Native data are copies of the original data in the format generated by the originating application. For example, Microsoft word creates .doc and .docx files. Everlaw supports automatic native uploading and processing for the following file formats:
|Email and Chat Formats||Web Formats||Microsoft Office Formats||Image Formats||Container Files|
EML, Gchat, iChat, Lotus Notes (NSF), MBox, MSG, OST, PST
|HTML, MHTML, SVG||All modern formats (.docx, etc), and all original formats (.doc, etc)||Apple graphics, BMP, CAL, DCX, DIB, EMF, GIF, JPG, MIL, PDC, PCT, PCX, PNG, PSD, RAS, TGA, TIFF (all compression types), WMF, and many more||.7Zip, AD1, E01, Gzip, L01, PST, Rar, Tar, TrueCrypt, Zip, .Zip.001|
|Medical Image Files||CAD Formats||Audio and Video||Other Formats|
|DCM||DGN (V7, V8), DWG, DXF||Most common audio and video formats||ICS, PList, PostScript, TXT, Source Code, SQLite databases, WPD|
Unsupported files might return an ‘error’ during the examine stage of the upload process. If so, these files can be accessed via the upload card by clicking on the ‘examine’ stage. You can review the files to see what the filetype is, and reach out to Everlaw support (firstname.lastname@example.org) for troubleshooting help. If you upload an unsupported file, one of two things can happen: If you are trying to upload file types not included in the box above, such as SQL databases, please reach out to email@example.com.
- Unsupported files could also be categorized as ‘binary‘ files. In this case, these documents will not return any errors during processing, but will not display properly during review. Please see the “Troubleshooting Upload Errors” section for more information about how to find and troubleshoot these documents. Once identified, you can reach out to Everlaw support (firstname.lastname@example.org) for help.
Accepted processed data types
You can upload two types of processed data:
- Pre-produced: The documents in the file have not been Bates stamped, and may or may not come with a load file.
- Produced: The documents in the file have been Bates stamped. These documents usually come with an accompanying load file.*
Everlaw can process and convert any combination of standard and non-standard data formats. However, some data formats are easier to process than others, potentially affecting wait times between when documents are received and when they become available for review on the platform.
* Often, the details and specifications for a production (ex. What formats to produce? Which metadata fields to include?) are negotiated at the beginning of a case. If you have questions about negotiating a production protocol, please reach out to email@example.com.
Preparing your native data to upload
In order to make the native upload process as smooth as possible, please keep the following in mind:
- For documents without embedded text, Everlaw will automatically generate text for .pdf and .tiff images using optical character recognition (OCR). Text embedded in .pdf or .tiff images is automatically extracted, and the image is updated to allow for hit highlighting and text selection within the image view of the document. If you need to have text generated for files in a different image format, please contact Everlaw support.
- Be aware that CGM and Shockwave files do not work well with our platform. If you want to add these files to your case, please reach out to firstname.lastname@example.org for help.
- If you have documents with multiple custodians, make sure documents of the same custodian are organized in their own sub-folders. You can create one top-level folder with any number and depth of sub-folders, but make sure that all documents within each sub-folder are from the same custodian. This is useful for preserving document path information, and will save you from having to specify the proper custodian at the document-level during the native upload process.
- If you are uploading a large folder, or a top-level folder that contains many subfolders, use a program like 7zip or iZip to compress the top-level folder before uploading. The uploads page will not accept folders.
- If any of your files or folders are password-protected, make sure that you know the password(s) as you will need to provide them during the upload process. Inaccessible files and folders will not be processed or uploaded into the platform.
- If your upload includes multiple documents, make sure that you are comfortable propagating the same time zone to documents that do not have a time zone specified in the metadata. If you want to propagate different time zones to documents in an upload that do not have a time zone specified, you should break up your upload in multiple uploads.
All documents in an upload will be added to an uploads card on the homepage. Some review teams like using these cards to organize uploaded documents. For example, you might want to add all documents from a particular custodian to a single upload card. In that case, you would want to ensure that all documents in a particular upload are from the same custodian, and that subsequent data from the custodian are added to the original upload as opposed to a new upload.
Please keep your preferences in mind as you prepare and package your documents for upload, and as you decide whether to create a new upload or add documents to an existing upload.
Preparing split archives for upload
If you are uploading a forensic image archive that is split up into several parts (e.g. parts example.E01, example.E02, and example.E03), there are special preparations to undertake to make sure that our system can handle them correctly.
First, files in a split archive must be uploaded together in a single container (zip, tar, etc). Second, inside this archive you should have all the parts for the archive, and nothing else aside from summary/information files generated by the software that produced the E01/L01 file. No other types of files, like PDF's or emails, should be included.
All the files in the container should be in a single directory. Here are examples of good and bad container files and their contents:
Migrating data from another platform into Everlaw
You can migrate data from another platform into Everlaw through a processed, non-Bates stamped data upload. By default, we import review data as metadata fields, which will allow you to search by those values. If you instead want to import the review data as codes in Everlaw, please contact email@example.com - these types of imports will incur additional support charges.
For more information about migrating data, see our migration workflow guide.
Creating a native data upload
To access the uploads page:
- Click the admin icon in the top right-hand corner of the navigation bar
- Select “Uploads” from the dropdown menu
- Select “New Upload” on the left-hand sidebar
Here, you can add the data you want to upload. Please see the “preparing your native data to upload” section for information on how to prepare your files.
Once you select your file (either by dragging-and-dropping or using the file selector), you will be asked to specify whether you’re uploading native files or processed files. Choose ‘native’.
Once you’ve made your selection, a wizard will appear where you can specify settings for the upload:
Step 1: Dataset details
Dataset Name: The name you give here will appear as the name of the associated uploads card on the homepage.
Display Timezone: The timezone you select here will be used as the assumed timezone for metadata fields that lack an explicit timezone value. If you do not select a timezone, then the timezone provided as raw text from the PDF's or emails being uploaded will be displayed.
Create PDFs: By default, Everlaw creates PDF images for all files in an upload, and placeholder images for file types that don’t image well (like spreadsheets). If desired, you can choose to image the file types that don’t image well, or choose to not image any of the files in an upload.
Deduplication: Upon upload, you can choose to:
- Deduplicate against all of the existing documents in your database
- Deduplicate against all of the existing documents by custodian
- Not deduplicate the documents in an upload
Duplicates of documents other than emails are identified using the hash value. Email duplicates are identified through the content, headers and attachments. All attributes must be exactly the same for an email to be deemed a duplicate.
Passwords: If any of your files/folders are password-protected, input the password(s) into the password box (one password per line). Inaccessible files will not be processed.
Step 2: Select custodians
The custodians step allows you to specify what custodian value to associate with the documents you’re uploading. You can specify a default custodian for all documents in an upload and/or set custom custodian values for particular files or folders.
To set a default custodian, input the custodian name into the “default custodian” box at the top of the table.
To set custom custodian values for particular files or folders, find the file/folder on the table and input the custodian name into the custodian box on the right. If there is a default custodian, it’ll be overridden for that particular file/folder. Files that have a black caret symbol in the far left can be expanded to display the individual sub-folders/files they contain. Click on the caret icon to expand or collapse.
Step 3: Uploading into children and partial projects
Aside from uploading the documents into the current database you’re on, you can also add the uploaded documents to any partial project you have access to. In addition, documents you upload will automatically be added to all complete projects associated with the primary database (ie. projects that share all of the documents from the primary database). The upload will be added to any project that has been selected in the table. To select or deselect a project, click on the checkbox. You cannot deselect compete projects.
Once you click ‘upload’, your data will be transferred to our servers. An overlay will appear to show you the status of the transfer. From the overlay, you can add additional documents to the upload by clicking on the “+Add files” button. If you don’t want to add files, close out of the overlay and you’ll be able to see your full native upload history.
A status card will be added to the native data page corresponding to your upload. A time estimate will appear on this status card to indicate approximately how long it will be until processing is complete. As your upload progresses, you can start reviewing completed docs. You do not need to stay on this page for the upload to continue processing. Once all your files are successfully processed, you will see a document icon with a green checkmark in the status card. Clicking the icon will take you to a results table of your processed documents.
Note that uploaded native files will not be assigned EVER numbers; instead they will have a “#” value as its prefix. Documents already on the platform will keep their assigned EVER numbers.
To learn how to view upload status, rerun, delete, rename, and take other actions, please see the “Managing Native Uploads” section.
Managing Native Uploads:
Your native upload history will appear in the “Native Data” section on the upload and productions page.
You can take the following actions on an upload status card:
1. Rename the upload
2. Click the three-dot menu icon to:
- Manage source files
- View configuration
- Delete the upload
3. Click on the document count to see your uploaded documents
4. View a report of your uploads
Here are some additional details about these actions:
Manage source files: When you select this option, a dialogue box will appear. You can see if previous files have been successfully transferred to Everlaw’s servers for processing. You can also add more files by clicking “add files” in the bottom right. Adding files to an existing upload is a good way to keep related files, such as files from the same custodian, together in a single upload.
View configuration: View the timezone propagated to your documents and projects the data was uploaded to.
Delete an upload: The documents in the upload, including the files generate during processing, will be removed from your database.
View report: The native uploads reporting tool will analyze your upload, including the upload errors and billable size.
The pie chart in the middle of the report shows a breakdown of the file types that have been processed and uploaded. You can also see the absolute number documents for each file type using the list on the left. At the top, you can see the number and size of documents that were deduplicated or de-NISTed. At the bottom, you can view the number of documents that were OCR’d and imaged, as well as the billable size of the upload. Finally, the right panel shows the number of documents that registered errors during processing, broken down by processing stage. Any number in blue is clickable, and will bring you to a results table with the appropriate documents. Print the report by clicking “print report” in the bottom right-hand corner.
If you are on the homepage, you can access documents from a completed native upload from the Document Sets column. The card’s name will be “Natives:” followed up the name of the upload (the name of the title given upon upload). Notice the distinction between "Native" uploads and "Processed" uploads: native upload cards have a dark orange band, while processed upload cards have a light orange band.
Updating an existing native upload
You can retroactively re-configure native documents that have already been uploaded. For example, you can change the timezone information, create image files that weren’t initially imaged, or re-run documents that weren’t initially processed because of a missing or incorrect password.
To update documents, first run a search to fetch the ones you want to alter. You can use the uploaded search term to quickly pull up documents from a particular upload. Then, use additional terms to get a subset of those documents, if desired.
Next, click ‘begin review’ to pull up the results table. Finally, click export, navigate to the ‘Process’ tab, select any re-configurations you need, and hit ‘Process’.
Creating and managing processed data uploads
In addition to processing your native data, Everlaw accepts processed and produced files for upload.
If uploading multiple files, you need to compress them in a .ZIP file first. Once you have created your zip file, drag-and-drop it into Everlaw, or use the file selector. You will then be asked to specify whether your data is native or processed. Choose "Processed."
Then, further specify whether or not your document have already been Bates-stamped.
Once you click submit, you'll be asked to provide any passwords for the files you uploaded. After confirming, you’ll be taken to the “Processed Data” page. Your upload will be represented as a status card. A progress bar will show an estimate for when your files will finish transferring to Everlaw. You can leave this page at any time.
When you first submit your data for upload, the status card for that upload should say “Received.” When the upload has been completed by our Everlaw support team, that card’s status will change to “Completed.”
You can access documents from a completed processed upload via the appropriate uploads card on the homepage. Processed uploads cards are located under the Document Sets column and have a light orange band. The card’s name will be “Processed:” followed by the name of the upload. Everlaw support staff will also notify you when your documents are ready for use in the platform.
If you want to delete the documents from an upload, locate the appropriate card, click the three-dot menu icon, and select "Delete." All documents from the given upload will be removed from your database.
Standard format for processed data
The produced data should have four main components:
- A directory containing images of every document in the production. Each image file named after its Bates number.
- A directory containing native files for a subset of the documents. Each file must be named with the same extension as the original native file type of the document (e.g. xlsx, .pptx, etc.), and preferably after the Bates number of the first page of the document it represents.
- A directory containing OCR or extracted text files, one file per document, each file named with the Bates number of the first page of the document it represents, followed by “.txt”
- A load file containing Bates range and metadata for each document
At minimum, the load file should contain:
- The begin Bates
- Either the number of pages or the end Bates
- If there are native documents, a native link/path pinpoint to the native files
If any other metadata fields are important to your review, make sure to request the producing party to include them in the load file.
- If you are migrating data from another platform, please follow Migration Workflow Guide and Best Practices
- Documents must have Bates prefix. A space (“ “) is not a valid Bates prefix.
- The begin Bates number is the unique identifier of a document and each page of a document is assigned a sequential Bates value. Documents cannot have overlapping Bates numbers. For example, if document ABC001 has three pages, with the end Bates ABC003, no other document can use Bates ABC001, ABC002, or ABC003. However, numeric page separators are supported. For example, the same three-page document can have the Bates range ABC001.001 - ABC001.003. And the next document can have the begin Bates ABC002.
- Everlaw does not support using alphabetical Bates suffixes to distinguish two documents with the same Bates prefix and number. For example, ABC001 and ABC001_Translation are considered the same and cannot both be uploaded to the platform.
- A parent email and its attachment(s) should be separate documents and the relationship between the parent and child(ren) represented via a metadata field such as “begin family” or “family id”; every document in the same family should have the same value and each family should have a unique value. For example, a parent email ABC001 and its two attachments ABC004 and ABC005 can be associated by having the same “begin family” value “ABC001.”
How to search for uploads
You can use the Bates or uploaded search terms to easily pull up documents from particular uploads.
Searching by bates
Native documents uploaded by Everlaw will be given a control number. You can find these documents by selecting the Bates search term and choosing “#” for the Bates prefix. If you know the specific number or number range you want to search across, you can input that into the second parameter.
You can search for processed documents in the same manner, though you would select one of the Bates prefixes instead of the "#."
Searching by the uploaded search term
To search for documents by their processing status, use the “uploaded” search term. You can select the name of the dataset that you wish you search for, or select “Any Dataset.”
Using the second parameter, you can find documents by their current processing state. There are three states in the uploads process: examine, text, PDF. A document can fail on any one of the three stages.
In the third parameter, you can select any additional flags (or properties) that you want to find documents by. Here is a list of the parameters, and their basic definitions:
- Container Document: The file was originally a top-level folder, and not an actual document
- Empty Text: Document does not have a text file
- Encrypted: The document is encrypted
- Has OCR: The document has been OCRed
- Has Placeholder PDF: The document has a placeholder image
- NIST Duplicate: The document is on the NIST National Software Reference Library list of known, traceable software files
- Opened with Password: The document was in a password protected folder or was a password protected file
- Unknown Document Type: The file is not a known document type
Using these flags, you can find documents that have certain properties, or that might need troubleshooting.
Troubleshooting Native Upload Errors
Native files go through three stages during processing: examine, PDF conversion, and text conversion. Your data may experience issues during any of these stages. For example, upon examine, the system may be unable to open the content of a container file (such as a zip file), which could contain thousands of important documents. It is important to examine these errors, and reach out to firstname.lastname@example.org if you experience these issues.
To view issues, click any of the stages, displayed as red icons in each status card (if they exist). Clicking on the number of issues within a stage will open a results table with the affected documents.
You can also locate documents based on potential issues through the reporting tool.
Please refer to the “accepted native data types” section to learn more about what could happen if you uploaded an unsupported filetype. You can also look at the uploads and productions FAQ page for answers to questions we commonly see.
Sometimes errors can be fixed if you re-run the upload. You can re-run documents from the results table view. See the 'updating an existing upload' section for more details.
Reach out to email@example.com if you need help troubleshooting upload errors or to re-run uploads.
How Everlaw handles metadata
Upon upload, Everlaw automatically detects metadata fields that refer to the same underlying concept, and groups them together. For example, all variations of the custodian field (“Custodian”, “Custdn”, “CUST”, etc.) will be grouped into the “Custodian” field. Though this automatic grouping is performed for all fields in the case, particular attention is paid to approximately 40 of the most common metadata fields, determined through an empirical analysis of the types of metadata fields that appear in document sets. These 40 or so fields are referred to as canonical fields. Raw metadata fields will therefore be mapped to these grouped and canonical fields upon upload.
You can view the ‘raw’, unmapped metadata fields for a document in the review window. If you expand the metadata panel and scroll to the bottom, you can expand the ‘raw metadata’ field category. All of the provided fields for a document will be visible. You can click again to hide.
Because of the mapping, the default name of a metadata field might not be original field name. If you hover over the value of a metadata field in the panel, you’ll be able to see both the original and mapped field name.
For more information on canonical fields, visit our help article about the topic.
Advanced processing defaults
The list below outlines some default processing configurations. Though it is not exhaustive, it will hopefully set proper expectations for you and your review team.
- The orientation of documents is preserved from its native version (a document that is in landscape orientation will remain that way upon upload)
- Embedded files (ex - a spreadsheet within a Word document) will not be extracted. If they are visible in the document when opened, they will remain displayed in Everlaw.
- The children of container files are extracted with no limitation on depth. As an exaggerated example, a Word doc attached to an email that’s attached to another email that’s in a Zip file that’s in another Zip file will be extracted.
- Hidden columns in Excel are displayed.
- Notes will be extracted and presented in the image view (for Word docs) and the native view (for spreadsheets)
Sometimes, you may try to upload an entire hard drive or a folder with personal files mixed in with system/software files. Some of these files have no user-specific content and can be removed upon processing. This process is called deNIST (removing NIST files). Any files that are on the NIST list (http://www.nsrl.nist.gov/) will qualify for deNISTing automatically upon upload. Binary files, and virtually all containers, are not part of this list and will not be removed.