
End-to-end encryption represents the foundation for 
security, privacy, trust, and compliance for all services 
on the internet. Several protocols supporting end-to-end encryption 
exist, all serving different purposes. 
Transport Layer Security (TLS) \cite{rfc8446} and 
Quick UDP Internet Connections (QUIC) \cite{rfc9369} 
secure web traffic; Wireguard \cite{Donenfeld_2017}, 
OpenVPN \cite{openvpn} secure point-to-point tunneling, and Tor \cite{Dingledine_Mathewson_Syverson_2004} secures multi-hop routing. 
One such use case --- instant messaging --- has become particularly 
ubiquitous in today's society. 

While secure instant messaging protocols strive for the same 
high-level cryptographic properties as other standard protocols 
such as TLS --- secrecy, authentication, and integrity --- instant 
messaging carries several subtleties that lead to differences in
overall protocol design. First, messaging is \textit{asynchronous}:
an \textit{online} peer must be able to send messages to an 
\textit{offline} peer, and the offline peer must then receive
the messages upon coming online. Therefore, parties must rely 
on a potentially untrusted central server to initiate 
authentication, key exchange, and message coordination. Second,
conversations are \textit{long-lived}; unlike TLS connections, 
which typically last a few seconds, instant messaging conversations 
may go on for years and carry both sensitive and non-sensitive 
messages. Thus, it is likely an endpoint will be compromised 
during a conversation lifetime, necessitating measures protecting 
conversation contents in long-term key compromise scenarios. 
Third, conversations are ideally \textit{deniable}, allowing 
participants to protect their own privacy by plausibly denying 
the authorship of a given message or transcript. Fourth, message
transcripts are ideally \textit{restorable}: users expect to be
able to restore their message conversation history from a server 
given the right credentials, thereby introducing additional 
challenges for the integrity, authentication, and confidentiality 
of stored messages on untrusted servers.

For our properties of interest, it is unquestionable that the 
Signal protocol has fully emerged as the best-in-class solution 
for peer-to-peer messaging. The Signal protocol offers secrecy, 
authentication, integrity, deniability, and asynchronicity
through the Extended Triple Diffie-Hellman handshake \cite{Marlinspike_Perrin_X3DH}, long-lived
conversation security through the post-compromise and perfect 
forward secrecy guarantees of the double ratchet \cite{Moxie_DoubleRatchet}, restorability 
through Signal's sub-protocol, Sesame \cite{Moxie_Sesame}, and post-quantum security
through the advent of the PQXDH key agreement protocol \cite{Kret_Schmidt_PQXDH}. Signal 
has been the subject of ample formal analysis, rigorously proving 
both on-paper and with proof assistants that each of the 
aforementioned 
properties hold \cite{Bhargavan_PQXDH, cremers_signal, alwen_doubleratchet, VatandasDeny, bhargavan_dy}. 

Signal, today, is extremely dominant in secure messaging. It is a reasonable statement to say that every modern end-to-end encrypted messaging protocol is, in fact, a derivative of the Signal protocol. However, the Signal protocol has a single critical caveat: its excellent guarantees fall apart for group messaging. The group messaging scenario carries similar subtleties and desired high-level properties, with the obvious caveat of a shared transcript history. However, at the time of writing, no universal solution akin to what Signal is for peer-to-peer messaging exists for secure group messaging. The current state of secure group messaging is fragmented between tree-based group key agreement schemes such as Messaging Layer Security \cite{rfc9420}, and protocols that \textit{compose} a group fan-out layer with secure peer-to-peer channels such as Sender Keys and Megolm. While there has been excellent recent work done on Messaging Layer Security \cite{Wallez_TreeSync, Wallez_TreeKEM}, the composite-type protocols remain dominant for group messaging in-practice. Sender Keys is directly deployed by WhatsApp \cite{WhatsAppSecurity2024}, Facebook Messenger \cite{MetaMessengerE2EE2023}, Signal \cite{SignalSenderKeysRust}, and Session \cite{Jefferys2020SessionProtocol}; Megolm is deployed by Matrix,\footnote{See \href{https://matrix.org/docs/matrix-concepts/end-to-end-encryption/}{matrix.org/docs/matrix-concepts/end-to-end-encryption} for more details.} which is in turn widely deployed by both governemnts and the private sector \cite{Albrecht_Dowling_Jones}. 

Thus, to analyze this flavor of group communication protocol, namely Sender Keys and Megolm, within a single security model, we define the generalized \textit{nested ratchet protocol} primitive. We define the nested ratchet protocol as a protocol that constitutes two main features: (1) a double ratchet-based peer-to-peer channel such as Signal, which securely communicates (2) a \textit{session} containing a signing keypair and symmetric key, which is then ratcheted forward to encrypt and decrypt fanned out group communications. Informally, we name these two composite features the \textit{peer-to-peer (P2P)} layer and the \textit{fan-out} layer respectively.

While there has been previous work studying nested ratchet protocols, namely Sender Keys \cite{Balbas_SK} and Megolm \cite{Albrecht_Dowling_Jones} in isolation, several questions are left unanswered by the literature:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]

  \item \noindent\textbf{Nested ratchet protocols generally assume the underlying peer-to-peer channel is secure. What happens if that assumption is broken?} 
    Broadly, nested ratchet protocols treat the underlying peer-to-peer channels as a black box and assume their security \cite{matrixorg_megolm_doc}. If this assumption fails, how is the broader fan-out layer protocol affected? If peer-to-peer channels have post-compromise security guarantees (i.e. double ratchet is used), what do recovery scenarios look like? The failure and recovery scenarios of nested ratchet protocols have yet to be thoroughly studied.

\item \noindent\textbf{What are the deniability guarantees of nested ratchet protocols?}
  The deniability of Triple Diffie Hellman, Extended Triple Diffie Hellman, PQXDH, and Authenticated Key Exchange --- the peer-to-peer channel protocols --- are well-studied \cite{VatandasDeny, FiedlerPQXDHdeny}. However, it remains an open question whether or not nested ratchet protocols such as Sender Keys and Megolm preserve the deniability guarantees of their peer-to-peer channels.

\item \noindent\textbf{Do nested ratchet protocols hold up under the scrutiny of formal verification?}
  In recent years it has become the standard to incorporate formal verification techniques into the design and evaluation of both novel and historical protocols, both inside and outside the cryptographic sphere \cite{SoK_CAC}. However, nested ratchet protocols, including Sender Keys and Megolm, have not received such a treatment. Previous work is hand-written and has not been mechanized \cite{Balbas_SK, Albrecht_Dowling_Jones}. Until this work, it has remained an open question whether nested ratchet protocols hold up under the scrutiny of formal verification. 

\end{itemize}

A rigorous investigation into these unresolved issues is necessary to either validate or reconsider the widespread trust currently placed in nested ratchet protocols.

\textbf{Our contribution.} Our work seeks to round off and tie together the previous literature on nested ratchet protocols. To do so, we take an approach rooted in \textit{formal methods} to study the security of nested ratchet protocols, allowing us to construct computer-verified proofs or provable and explicit counterexamples. We make the following contributions.

\textbf{Models}. We present the first mechanization of nested ratchet protocols, primarily using the state-of-the-art symbolic cryptanalysis tool \pv \cite{Blanchet_2016}. Specifically, we present canonical models of the nested ratchet protocol; we also model Megolm, and Sender Keys according to their specifications. To support our larger nested ratchet protocol models, we also model 3DH, X3DH, Signal, and Olm (the secure point-to-point protocol employed by Megolm). 

\textbf{Formal Analysis}. Using \pv's automated analysis suite, we prove secrecy, authentication, integrity, post-compromise security, and perfect forward secrecy for the nested ratchet protocol, including Megolm and Sender Keys. Using our models, we reason about offline deniability and the various failure cases of the nested ratchet protocol. To support our analysis, we present the first mechanizations of offline deniability, in addition to standard secrecy, authentication, integrity, post-compromise security, and perfect forward secrecy results for 3DH, X3DH, Signal, and Olm. 

\textbf{Comparison}. Using our \pv models, we precisely compare the cryptographic properties and failure cases of the nested ratchet protocol. We quantify how differences in design between both the peer-to-peer and fan-out channels propagate to the overall high-level guarantees of hthe protocols.

\textbf{Code}. Our models are entirely open-source, and our results, environment, and dependencies are readily reproducible via a nix flake: \href{https://zenodo.org/records/16959099}{zenodo.org/records/16959099}
