The Electronic Discovery Reference Model (EDRM), a global advisory body seeking to improve the practice and provision of data and legal discovery, has published the EDRM Cross Platform Email Duplicate Identification Specification.

This specification provides a framework for identifying duplicates across multiple email platforms, allowing organizations to identify duplicate emails efficiently and effectively in a defensible and cost-effective manner.

Currently no means of cross platform email duplicate identification exists, except to reprocess the data using a single vendor platform, often expending significant time and cost.

The solution is a simple, but effective approach which involves the use of the hash value of an email Message ID metadata field which is the EDRM Message Identification Hash (“MIH”). This new approach will not replace current email deduplication methods but will enable cross platform email duplicate identification.

“Our global, multidisciplinary EDRM Committee, comprising legal technologists, forensic professionals, product developers, consultants, c-suite business leaders, data scientists and lawyers, is really excited to bring this solution for email cross platform duplicate identification to the market. We believe it has numerous applications that will save significant time and cost," said Beth Patterson, director at ESPConnect and project trustee.

"We encourage product vendors to implement the generation of the MIH in their products and lawyers, eDiscovery & forensic professionals, service providers, regulators and courts to request the MIH in productions. Like any eDiscovery tool, expertise is required to determine how best to apply the MIH for a particular case."

The global EDRM Email Duplicate Identification Specification project team has developed the EDRM Email Duplicate Identification Toolkit as a resource to support the implementation of the EDRM MIH and use of it for cross platform email duplicate identification. The Toolkit outlines several considerations which should be taken into account when deciding how to utilize the MIH. Comments from the public are welcomed until March 15, 2023.

The project team anticipates that the use of EDRM MIH will lead to significant time and cost benefits. For example, during discovery, disclosure or an investigation, it is often useful to identify duplicate emails in data exchanged between parties. This can deliver many benefits, including the ability for legal teams to rapidly triage emails already reviewed that also reside in data received from others.

“Anyone who has been active in e-discovery for a while has almost certainly run into the situation where they want to de-duplicate data across platforms,” said David Cohen, Chair of the EDRM Project Trustees and Practice Group Leader of Reed Smith’s Records & E-Discovery Practice Group.

“For example, that occurs where clients or providers want or need to change processing and review platforms or wish to use multiple tools that have different duplication identification standards. Prior to development of the MIH hash, there was no economical way to deduplicate across platforms, so parties ended up having to pay for duplicative review and analysis of the same documents. Now, thanks to Beth Patterson and her dedicated team, there is a solution that can save parties significant time and money.

“The whole development team deserves credit, including the volunteers from companies that offer some of the leading platforms that can now implement the MIH hash, like Relativity, Reveal-Brainspace, EDT and Nuix.”

"The strength of the EDRM MIH springs from its simplicity and economy,” said Craig Ball, EDRM General Counsel, Special Master and Texas litigator. “Pairing decades-old technologies in an innovative way, the EDRM MIH enables cross-platform duplicate identification--an unprecedented, disruptive capability that seamlessly integrates with existing tools and workflows. The EDRM MIH is more than simply a great idea that's been rigorously developed and tested; it's an open-source, collaborative effort of leading companies and technologists backed by the integrity and influence of the EDRM. I'm proud to be part of the EDRM MIH development team."

The project team includes (organizations listed for identification purposes):

• Murali Baddula, Chief Digital Officer, Law In Order (Sydney, Australia)
• Craig Ball, Attorney, Certified Computer Forensic Examiner, Adjunct Professor, University of Texas School of Law (Austin, Texas USA)
• Klaus-Peter Finke-Härkönen, Director Strategic Initiatives, Cyber Diligence LLC, (Syosset NY USA)
• Ian Folkman, Vice President – Forensic Technology at Deloitte Tohmatsu Financial Advisory (Japan)
• Scott Foster, Senior Managing Director at FTI Consulting–Technology Lead (Australia)
• Matthew Golab, Director Legal Informatics and R+D, Gilbert + Tobin (Sydney, Australia)
• Phil Haselden, Technical Fellow, EDT (Australia)
• Greg Houston, Workflow Enablement, Relativity (Chicago, Illinois USA)
• Dr Paul Hunter, Chief Data Scientist, EDT (Australia)
• Dinesh Karamchandani, Senior Data Scientist, Reveal-Brainspace (Chicago, Illinois USA)
• Lisa Kozaris, Chief Innovation & Legal Solutions Officer, Allens (Melbourne, Australia)
• Enzo Lisciotto, Director of Customer Success at Reveal-Brainspace & Co-Membership Director, ACEDS-ANZ (Sydney, Australia)
• James MacGregor, Partner, FORCYD (London, UK)
• Karan Mehta, Head of Legal Technology, Allens (Sydney, Australia)
• Rachi Messing, Co-Founder, Altorney (Israel)
• Elizabeth Miller, Senior Director, Law in Order (Australia)
• Beth Patterson, Director, ESPconnect & Adjunct Professor, University of Technology Sydney Law School (Australia)
• Alexander Poelma, Product Manager Nuix Workstation & Engine, Nuix (California, USA)
• Jo Sherman, CEO & Founder, EDT (Australia/USA)
• Paul Sirkis, Division Director, Regional Head of Litigation, Americas at Macquarie Group (New York, New York USA)
• George Socha, Senior Vice President of Brand Awareness, Reveal-Brainspace (Chicago, Illinois USA)
• Stephen Stewart, CTO, Nuix (Philadelphia, PA USA)
• Gavin Wingfield, Director Applied Legal Technology, King & Wood Mallesons
• Emma Young, Head of eDiscovery Delivery, Sky Discovery (Brisbane, Australia)