After the development and deployment of OpenPGP and S/MIME, it was commonly agreed that the secure messaging problem was solved, and that public key cryptography provides a viable solution: Digital signatures for authentication (and nonrepudiation) and digital envelopes for confidentiality protection. It was argued that the slow deployment of OpenPGP and S/MIME in the field was only due to poor usability, rather than technical inadequacy. This changed in 2004, when Nikita Borisov, Ian Goldberg, and Eric Brewer published a research paper, in which they challenged common wisdom and questioned the adequacy of existing technologies for secure messaging. In particular, they stressed the fact that these technologies do neither provide forward secrecy nor deniable authentication, and that these shortcomings limit their usefulness in the field. Note what happens if a long-term private key gets compromised. In this case, all messages that have been or will ever be enveloped with this key are compromised, as well. The respective damage in terms of confidentiality loss is as large as possible. Furthermore, many of these messages will be digitally signed, and hence carry a cryptographic proof of their origin. This, in turn, means that the originator of such a message cannot meaningfully deny having sent it. There are certainly cases, in which this undeniability does not pose any problem. But there are also cases, in which it may pose a huge problem to the originator of such a message. Think about whistle-blowers, dissidents, and political activists.
Against this background, the above-mentioned paper argued that people sometimes want to hold a casual conversation that is private, informal, and unofficial. It is like a conversation held in a backyard without witnesses. In real life, we attribute the term “off-the-record” to this type of conversation; it does neither leave traces nor records that may prove that the conversation ever took place. The term “off-the-record” can also be used in the realm of secure messaging, and hence OTR messaging refers to this type of message exchange. It is as private as possible, and it provides repudiation, meaning that it can be denied by all participants. For the reasons mentioned above, OTR messaging cannot be implemented with digital envelopes and digital signatures. Instead, technologies are needed that provide forward secrecy and deniability – or even plausible deniability. Forward secrecy is usually achieved by using ephemeral Diffie-Hellman key exchangesand it can be even further improved by restricting the lifetime of the keys in use. OTR messaging uses a Diffie-Hellman ratcheting mechanism to refresh the key as often as possible. Plausible deniability, in turn, is achieved by using MACs instead of digital signatures. Note that a MAC is symmetric in nature, meaning that a MAC can be generated and verified by either side of a communication, i.e., the sender and the recipient of a message. This, in turn, allows the originator of a message to deny having sent it. There are even some complementary techniques that can be used to further strengthen deniability.
From today’s perspective, OTR messaging has paved the way to end-to-end encrypted (E2EE) messengers that use similar techniques. Most importantly, the Signal protocol (that is also used in WhatsApp and the Facebook Messenger) combines the Diffie-Hellman ratchet with some other technologies that can be used in an asynchronous setting. Keep in mind that OTR messaging has been developed for instant messaging, where the participants need to be online. This need not be the case in an asynchronous setting, like the one used for e-mail. This complicates things a little bit, but OTR messaging is still a milestone in the development of contemporary secure and E2EE messaging solutions. It will be explained in detail in a new and upcoming book on E2EE messaging on the Internet that is scheduled for 2020.