synchronous/usenix-2026-nested
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
Jake Ginesin
2025-10-25 03:54:21 -04:00
commit da9a2906c3
43 changed files with 19617 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
sections/abstract.tex Normal file
View File

@@ -0,0 +1,11 @@
\begin{abstract}
Nested ratchet protocols—such as Sender Keys and
Megolm—combine pairwise peer-to-peer
double-ratchet channels with a server-assisted fan-out layer to scale
end-to-end encrypted group messaging. Despite the widespread deployment
of nested ratchet protocols, including in WhatsApp, Signal, Matrix, and Facebook Messenger, their security properties are typically analyzed
piecemeal rather than in a single unified model. Thus, we define the unified nested ratchet protocol primitive to capture the security guarantees of Sender Keys and Megolm. We present a symbolic, mechanized model of the nested ratchet protocol in ProVerif, and instantiate it with canonical designs and faithful encodings of Sender Keys and Megolm. We formalize and evaluate core properties, including message secrecy, mutual authentication, perfect forward secrecy, post-compromise security, and offline deniability. Using our models, we systematize compromise scenarios across the pairwise and fan-out layers, quantify how attacks propagate, and identify recovery conditions induced by
session (re)generation. We analyze two design tradeoffs,
including signed vs. unsigned pre-keys at the peer-to-peer layer (server trust vs. mutual deniability), and signatures vs. MACs at the fan-out layer (non-reputation vs. deniability).
Our analysis also surfaces actionable recommendations for protocol implementors, including the use of forward-secure signatures, peer-to-peer layer pre-key signing, and fan-out layer encryption usage for two-party channels. We release our ProVerif models and scripts as reproducible artifacts to facilitate verification and comparison of future designs.
\end{abstract}

93
sections/analysis.tex Normal file
View File

@@ -0,0 +1,93 @@
We describe our analysis of the base nested ratchet protocol Megolm (as described by Matrix), and Sender Keys (as described by Signal, Whatsapp, and Facebook Messenger).
\subsection{Sub-Protocol Properties}
\label{sec:section label}
Building up to our models of the nested ratchet protocol, Megolm, and Sender Keys, we construct simpler models of 3DH, X3DH, Olm, and Signal in isolation. For 3DH and X3DH, we re-prove the classic results on message secrecy and mutual authentication. For Olm and Signal, we re-prove message secrecy, mutual authentication, perfect forward secrecy, and post-compromise security. Notably, we also offer the first mechanization of offline initiator deniability and offline responder undeniability results for 3DH, X3DH, Olm, and Signal.
\subsection{Symbolic Analysis Results}
\label{sec:section label}
We completely model the nested ratchet protocol, Megolm, and Sender Keys (encompassing Signal, Whatsapp, and Facebook Messenger) under X3DH within \textsc{ProVerif}. To inform and validate our models, we closely referenced the computational constructions of Sender Keys, Megolm, and Signal, referenced the relevant open-source implementations and documentation, and interfaced with the specification authors when necessary. We formalized and automatically proved properties P1-P6 with respect to the models of the nested ratchet protocol, Megolm, and Sender Keys. We also formalized P7 and P8 for use in our case studies.
%and Signal, Whatsapp, and Facebook Messenger (
%We prove properties P1-P6 hold
%- secrecy
%- mutual authentication
%- initiator deniability for megolm sessions
%- responder undeniability for megolm sessions
%- perfect forward secrecy over megolm msgs
\subsection{Failure Case Analysis}
\label{sec:section label}
Using our models, we analyze the various failure cases of the nested ratchet protocol, Megolm, and Sender Keys. For conciseness, we jointly refer to the aforementioned protocols cumulatively, as the results are very similar, unless otherwise stated. We also refer to offline deniability simply as ``deniability'' unless otherwise stated. A stratification of our failure case analysis is shown in Table \ref{tab:failure-comparison}.
\subsubsection{Failure Taxonomy}
\capbox{\underline{\textbf{C1}}: Compromise of a single fan-out layer session ratchet key. \\
\textbf{Properties violated}: P2}
Message secrecy (P2) is violated until the compromised peer completes session regeneration and re-transmission. Note, perfect forward secrecy within a single fan-out session (P5) is maintained because session symmetric keys are ratcheted forward with each message.
\capbox{\underline{\textbf{C2}}: Compromise of a single fan-out layer session public signing key. \\
\textbf{Properties violated}: P4}
Message deniability (P4) is violated until the compromised peer completes session regeneration and re-transmission. An observer may check the signature on all messages, past and present, within the session and prove attribution to the sender. Note, (P7) is also violated, as both post-compromise deniability and perfect forward deniability within sessions does not hold.
\capbox{\underline{\textbf{C3}}: Compromise of a single fan-out layer session private signing key. \\
\textbf{Properties violated}: P3, P4 for nested ratchet and Sender Keys; P4 for Megolm}
Message deniability (P4) is violated for the nested ratchet protocol and its derivatives. Message authentication (P3) is violated for the nested ratchet protocol and Sender Keys; Megolm chooses to MAC messages, ensuring authentication remains in the case the private signing key is compromised. Both properties remain violated until session regeneration and re-transmission.
\capbox{\underline{\textbf{C4}}: Compromise of a single complete fan-out layer session, including the public and public signing keys, and ratchet key. \\
\textbf{Properties violated}: P2, P3, P4}
This case can be seen as a composition of the previous cases. In the case a singular fan-out layer session compromise, message secrecy (P2), authentication (P3), and deniability (P4) are compromised until session re-generation and re-transmission.
\capbox{\underline{\textbf{C5}}: Mutual Compromise of long-term static identity keys of a single P2P channel. \\
\textbf{Properties violated}: None}
No compromises: unless pre-keys remain unsigned, pre-key messages are secure. Equivalently, this result also expresses resistance against unknown keyshare attacks.
\capbox{\underline{\textbf{C6}}: Compromise of a single chain key within a double ratchet P2P channel. \\
\textbf{Properties violated}: P2, P3, P4}
We observe any compromise in P2P double ratchet chain keys results in the compromise of any fan-out layer sessions enclosed within any message keys derived from the chain key. Therefore, (P2), (P3), and (P4) will be violated. Note, (P6) holds due to the self-healing property of the double ratchet.
\capbox{\underline{\textbf{C7}}: Compromise private pre-key and static identity material of a single peer. \\
\textbf{Properties violated}: P2, P3, P4} The same as the previous scenario: (P2), (P3), and (P4) are violated until the next fan-out layer session is re-generated and re-transmitted.
\subsubsection{Observed Patterns \& Insights}
The previous failure taxonomy highlights a few patterns present across both Megolm and Sender Keys:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
\item\noindent\textbf{Universal deniability failures}. Across most compromise scenarios, both Megolm and Sender Keys fail deniability (P4) and strong deniability (P7).
\item \noindent\textbf{Megolm's MAC advantage}. Only Megolm maintains authentication (P3) when the private signing key is compromised (C3) due to its additional MAC verification on fan-out messages.
\item \noindent\textbf{P2P compromises cascade}. Any P2P layer compromise (C6, C7) breaks nearly everything at the fan-out layer, showing the compositional vulnerability at play.
\item\noindent\textbf{PFS/PCS resilience}. Both Sender Keys and Megolm maintain perfect forward secrecy (P5) and post-compromise security (P6) reasonably well, validating the ratcheting design. This conclusion, however, acknowledges each peer re-establishing sessions with each other peer is a polynomial-complexity operation; this is precisely the scenario where MLS improves upon Nested Ratchet-based protocol designs.
\end{itemize}
%- leaking peer-layer ephemeral keys compromises a single session
%- leaking peer-layer long-term keys only compromises pre-key messages, and thus a single session, if pre-keys are not signed
%- leaking session ratchet keys compromises all messages until retransmission
%- leaking session signatures compromises the deniability of all messages of the session
\subsection{P2P layer pre-key post-compromise message secrecy vs mutual deniability}
\label{sec:section label}
Designers of secure point-to-point messengers in the client-server model are faced with the decision of whether users should upload signed or unsigned pre-key material to the central server.
If pre-key messages are \textit{signed} by the long-term identity key, users no longer to place trust the central server; however, offline deniability for the uploader of the pre-key material (also generally the responder to the authenticated key exchange handshake) trivially does not hold, as a judge may simply be presented with the signed pre-keys as proof of protocol participation. If pre-key messages remain \textit{unsigned} more trust must be placed in the central server, as handshake becomes vulnerable to an unknown keyshare attack and thus results in pre-key message leakage if static identity keys are leaked. However, mutual deniability (i.e. property P8) is completely ensured as pre-keys are no longer signed.
The Olm documentation briefly alludes to this tradeoff. We go ahead and mechanize this tradeoff in \textsc{ProVerif}, and most importantly show the entire nested ratchet protocol is deniable to parties external to the group, modulo compromises, if all P2P-layer pre-keys remain unsigned. One observation we make is mutual deniability in the P2P layer is not sufficient to post-compromise and perfect forward deniability (a la property P7) for the entire protocol, which we now elaborate upon.
\subsection{Fan-out layer non-repudiation vs deniability}
\label{sec:section label}
We identify a similar trade-off in the fan-out layer of the nested ratchet protocol and its derivatives. Signing all fan-out layer messages with a private key serves to ensure messages cannot be forged within a group; however, in the case any one of the peers are compromised, thereby compromising the public key, deniability for all past and future messages within the session is compromised (i.e. property P7 is violated).
Thus, the nested ratchet protocol may choose to leave fan-out layer messages \textit{unsigned} and alternatively ensure message authentication via MACs. We mechanize this tradeoff in \textsc{ProVerif}. We observe signatures versus MACs trades non-reputation for mutually deniability both \textit{within} the group (i.e. property P8), as well post-compromise and forward secret deniability (i.e. property P7).
\input{diagrams/fail-cases}

215
sections/background.tex Normal file
View File

@@ -0,0 +1,215 @@
We build up the necessary machinery for our work, including descriptions of the underlying cryptographic primitives, handshakes, component sub-protocols, and a formal description of the nested ratchet protocol.
\subsection{Cryptographic Building Blocks}
\label{sec:section label}
We overview the relevant cryptographic primitives and specify the notation we employ.
\noindent\textbf{Cryptographic Signatures}. A signature scheme consists
of three algorithms: \textsf{SignGen()}, which outputs a public/private
keypair \textsf{(pk, sk)}; \textsf{Sign(\textsf{sk}, \textit{m})}, which
$\sigma$; \textsf{CheckSign(\textsf{pk}, \textit{m}, $\sigma$)},
which takes a public key \textsf{pk}, a message \textit{m}, and a signature
$\sigma$, and outputs a boolean indicating the validity of the signature.
For a given keypair \textsf{(pk, sk)}, the algebraic relation \textsf{CheckSign(pk, \textit{m}, Sign(sk, \textit{m}))} $=$ \textsf{True}
will hold.
\noindent\textbf{Diffie-Hellman (DH) key exchange}. The DH
key exchange is a cryptographic method that allows
two parties to establish a shard secret over an insecure channel.
Diffie-Hellman consists of two functions: \textsf{DH\_Gen()}, which outputs
a public/private keypair \textsf{(pk, sk)}, and \textsf{DH(sk, pk)},
which outputs a string. For two given keypairs generated by \textsf{DH\_Gen()}
\textsf{(pk\textsubscript{A}, sk\textsubscript{A})}, \textsf{(pk\textsubscript{B}, sk\textsubscript{B})}, the algebraic relation
\textsf{DH(sk\textsubscript{A}, pk\textsubscript{B})} $=$ \textsf{DH(sk\textsubscript{B}, pk\textsubscript{A})} holds.
\noindent\textbf{Message Authentication Codes (MAC)}.
A MAC provides message authenticity and integrity assurance.
It consists of two algorithms: \textsf{MAC(k, \textit{m})} which takes
a secret key \textsf{k} and message \textit{m}, outputting a
tag \textsf{t}; and \textsf{CheckMac(k, \textit{m}, t)} which outputs
a boolean indicating validity. Formally,
\textsf{CheckMac(k, \textit{m}, MAC(k, \textit{m}))} = \textsf{True} holds for all correctly computed tags. MACs instantiated with a hash function are denoted as \textsf{HMAC}.
\noindent\textbf{Symmetric Encryption}. Symmetric encryption provides confidentiality using a single shared secret key for both encryption and decryption. A symmetric encryption scheme consists of two algorithms: \textsf{Encrypt(k, \textit{m})} outputs ciphertext \textsf{c} given a key \textsf{k} and message \textit{m}; \textsf{Decrypt(k, c)} outputs the original message \textit{m}. The algebraic relation \textsf{Decrypt(k, Encrypt(k, m))} = \textsf{m} holds for all correctly encrypted messages.
\noindent\textbf{Authenticated Encryption with Associated Data (AEAD)}. AEAD schemes simultaneously ensure message confidentiality, authenticity, and integrity, optionally incorporating associated data. AEAD is defined by two algorithms: \textsf{AEAD\_ENC(k, \textit{m}, \textit{ad})}, outputting a ciphertext \textsf{c} given a key \textsf{k}, message \textit{m}, and associated data \textsf{ad}; \textsf{AEAD\_DEC(k, c, \textit{ad})} returns message \textit{m} or a symbol indicating failure otherwise. The algebraic relation \textsf{AEAD\_DEC(k, AEAD\_ENC(k, \textit{m}, \textit{ad}), \textit{ad})} = \textsf{m} holds.
\noindent\textbf{Hash Functions}. Cryptographic hash functions deterministically map arbitrary-length inputs to fixed-length outputs, satisfying collision resistance and pre-image resistance. Formally, a hash function \textsf{Hash(m)} outputs a string \textsf{h} of fixed length. Computational hardness ensures it is infeasible to find distinct \textsf{m} and \textsf{m'} where \textsf{Hash(m)} = \textsf{Hash(m')} or to derive \textsf{m} from \textsf{h}.
\noindent\textbf{Key Derivation Functions}. A Key Derivation Function (KDF) is a cryptographic algorithm that derives one or more secrets from a master secret. We universally employ HMAC-based Extract-and-Expand Key Derivation Function (HKDF) \cite{rfc5869} instantiated with a collision-resistant hash function. We denote HKDFs as \textsf{HKDF(master)} $\rightarrow$ \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})}, where \textsf{master} denotes master key, and \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} denotes the derived keys.
\newcolumntype{L}{>{\raggedright\arraybackslash}p{4cm}}
\newcolumntype{R}{>{\raggedright\arraybackslash}p{7cm}}
\begin{table}[h]
\footnotesize
% \small
\centering
% \begin{tabular}{L!{\color{gray}\vrule width 0.4pt}R}
% \begin{tabular}{L!{\color{black!70}\vrule width 0.4pt}R}
\begin{tabularx}{\columnwidth}{l!{\color{black!70}\vrule width 0.4pt\hspace{0.5em}}X}
\toprule
\textbf{Symbol} & \textbf{Description} \\
\midrule
\textsf{SignGen()} $\rightarrow$ \textsf{(sk, pk)} & Generates a signing keypair \\
\textsf{Sign(\textsf{sk}, \textit{m})} $\rightarrow$ $\sigma$ & Produces a signature $\sigma$ on \textit{m} using \textsf{sk} \\
\textsf{CheckSign(\textsf{pk}, \textit{m}, $\sigma$)} & Takes a public key \textsf{pk}, a string \textit{m}, and a signature $\sigma$; outputs a boolean indicating validity \\
\midrule
\textsf{DH\_Gen()} $\rightarrow$ \textsf{(sk, pk)} & Generates a DH keypair \\
\textsf{DH(sk, pk)} $\rightarrow$ str & Computes DH exponentiation \\
\midrule
\textsf{X25519\_Gen()} $\rightarrow$ \textsf{sk, pk} & Generates a keypair capable of signing and encryption \\
\midrule
\textsf{MAC(k, \textit{m})} $\rightarrow$ \textsf{t} & Takes
a key \textsf{k} and message \textit{m}; outputs a
tag \textsf{t} \\
\textsf{CheckMac(k, \textit{m}, t)} & Takes
a key \textsf{k}, a message \textit{m}, and a tag \textsf{t}; outputs a boolean indicating validity \\
\midrule
\textsf{Encrypt(k, \textit{m})} $\rightarrow$ \textsf{c} & Takes a key \textsf{k}, a plaintext \textit{m}; ouptuts a ciphertext \textsf{c} \ \\
\textsf{Decrypt(k, c)} $\rightarrow$ \textit{m} & Takes a key \textsf{k}, a ciphertext \textsf{c}; outputs the plaintext \textit{m} \\
\midrule
\textsf{AEAD\_ENC(k, \textit{m}, \textit{ad})} $\rightarrow$ \textsf{c} & Takes a key \textsf{k}, a plaintext \textit{m}, and associated data \textit{ad}; outputs a ciphertext \textsf{c} \\
\textsf{AEAD\_DEC(k, c, \textit{ad})} $\rightarrow$ \textit{m} & Takes a key \textsf{c}, a ciphertext \textsf{c}, and associated data \textit{ad}; outputs plaintext \textit{m} \\
\midrule
\textsf{Hash(m)} $\rightarrow$ \textsf{h} & Takes a string \textsf{m}; outputs a hash \textsf{h}. Assumed to be collision-resistant. \\
\midrule
\textsf{HKDF(mk)} $\rightarrow$ \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} & Takes a ``master'' key \textsf{mk}, and outputs a set of keys \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} \\
\bottomrule
\end{tabularx}
\caption{Notation for employed Cryptographic primitives.}
\label{tab:symbols}
\end{table}
\subsection{Handshakes}
\label{sec:section label}
Fundamental to nested ratchet protocols is authenticated key exchange (AKE), which builds on Diffie-Hellman with cryptographic signatures to establish authenticated, shared secrets between two parties over an insecure channel \cite{auth}. AKE generally relies upon verification of public/private key pairs between parties \textit{out-of-band} to prevent unknown key-share attacks.
To achieve \textit{asynchronous} messaging, as well as cryptographic deniability for the conversation intiator, AKE is extended with an always-online intermediary server that (1) retains initial key exchange ``pre-key'' material of users, and (2) retains and transmits encrypted messages. The intermediary server need not be trusted to ensure the secrecy, authenticity, and integrity of transmitted messages.
\input{diagrams/ake-server.tex}
In practice, pre-key bundles do not just consist of the (signed) public material of a single key pair. Multiple key pairs are generally employed, with Diffie-Hellman exchanges crossed between the different key pairs, and the result concatonated together and passed through a key derivation function.
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
\item \textit{Triple Diffie-Hellman} (3DH) combines DH operations between static long-term keys and signed ephemeral one-time use keys -- this way, the derived key remains secure in the case long-term are later compromised \cite{matrixorg_olm_repo}.
\item \textit{Extended Triple Diffie-Hellman} (X3DH), which was notably used by Signal for scaling asynchronous messaging, combines long-term static keys, signed medium-term pre-keys, and one-time use pre-keys -- one-time pre-keys are selected from a set of one-time pre-keys provided to the server. This way, each asynchronous handshake uses a unique one-time pre-key \cite{Marlinspike_Perrin_X3DH}.
\item \textit{Post-Quantum Extended Diffie-Hellman} (PQXDH), combines Diffie-Hellman key exchanges over classical primitives such as RSA or elliptic curve cryptography with post-quantum primtivies via key encaptulation \cite{Kret_Schmidt_PQXDH}.
\end{itemize}
\subsection{Signal, Olm, and the Double Ratchet}
\label{sec:section label}
After deriving a symmetric key through an authenticated handshake protocol, secure peer-to-peer messaging in modern protocols, including Signal and Olm, almost universally employ the \textit{Double Ratchet} algorithm. We only describe the Double Ratchet in brief -- we refer the reader to \cite{Moxie_DoubleRatchet} for the complete description. Messages transmitted with the Double Ratchet algorithm, notably, carry post-compromise security and perfect forward secrecy guarantees in addition to the secrecy and authentication guarantees one would achieve with an authenticated encryption scheme. In short, the Double Rachet achieves this by (1) continually hasing ``racheting'' forward the shared symmetric key, (2) continually re-negotiating new Diffie-Hellman keys, and encrypting messages with a key derived via passing the results from the aforemetioned steps into a key derivation function.
\subsection{Session Sharing \& Server-Side Fan-Out}
\label{sec:section label}
One could reasonably argue secure peer-to-peer messaging is ``solved'' by composing X3DH or PQXDH with the Double Ratchet, as we achieve our desired properties: secrecy, authentication, deniability, post-compromise security, perfect forward secrecy, and asynchronicity. However, the same certainly cannot be said for secure group communication protocols. Group messaging protocols must cope with two orthogonal scaling dimensions:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
\item \textit{Group fan-out}. A single ciphertext created by the sender should efficiently reach $n-1$ recipients without $n-1$ separate public-key operations at the sender.
\item \textit{Dynamic Group Membership}. Group membership does not remain fixed: peers may be added or removed, necessitating another round of key agreement to add the new user or exclude the old user.
\end{itemize}
\noindent Both problems are addressed with \textit{session-sharing messages} and a \textit{server-side fan-out} layer that sits \textit{above} the pairwise Double Ratchet channels and \textit{below} the application payload. In general, message communication within a group relies upon the key material within sessions as opposed to the pairwise Double Ratchet channels.
%To communicate within a group, peers rely on \textit{sessions} to secure transmitted messages as opposed to the pairwise Double Ratchet channels.
\noindent\textbf{Sessions}. Broadly, \textit{sessions} include a symmetric ratchet key for message encryption and encryption, as well as a signing key pair for message identification. Each peer creates their own session. Thus, a \textit{session-sharing message}, which each peer distributes to every other peer, encaptulates the symmetric ratchet key and the public key of the signing keypair. To transmit a message, the sending peer will ratchet their symmetric key forward with a cryptographic hash function, encrypt their message, and sign; the receiving peer will check the signature, ratchet their symmetric key forward to match the sender, and decrypt.
\input{diagrams/session-trans.tex}
\noindent\textbf{Server-side Fan-out}. Once a peer's sessions are transmitted to each other peer, group communication is conducted through \textit{server-side fan-out}. That is, the sending peer will send a message to the central server, and the server will ``fan'' the message out to each receiving peer. Combined with the already-established sessions, public-key operations between pairwise peers is avoided.
\noindent\textbf{Group Updates}. Whenever the group is updated, or a peer is compromised, each peer simply generates a new session and transmits it to each other peer through the already established peer-to-peer Double Ratchet channels.
One can see post-compromise security stems from session re-establishment through group updates, and perfect forward secrecy stems from ratcheting forward the session's symmetric key.
\subsection{Nested Ratchet Protocol Definition}
\label{sec:section label}
We are now ready to define the \textit{nested ratchet protocol}.
Informalaly, nested ratchet protocols \textit{stack} two layers of
distinct ratcheting protocols as such:
\begin{itemize}[leftmargin=*]
\item \textbf{P2P layer}. Ordinary Double Ratchets-based
communication channels that
carry only the occasional \textsc{SessionShare} control
messages; they provide secrecy and authentication, but may also
provide deniability, perfect forward secrecy, and post-compromise security.
\item \textbf{Fan-out layer}. A lightweight symmetric ratchet per
sender; one ciphertext is uploaded and blindly fanned out,
giving $O(1)$ work for the sender regardless of group size.
\end{itemize}
\noindent With this intution, we now provide the complete definition.
\begin{mydef}[Nested Ratchet Protocol]\hfill\\
Let $U=\{u_1,\dots,u_n\}$ be a dynamic set of participants.
A \emph{Nested Ratchet Protocol} consists of
\begin{enumerate}[label=(\roman*),leftmargin=*]
\item A set of pairwise \textbf{Double Ratchet channels}
$\mathcal D=\bigl\{D_{i,j}\mid u_i,u_j\in U,\;i\neq j\bigr\}$ established through authenticated key exchange between each pair of peers
\item For each sender $u_s\in U$, a \textbf{fan-out ratchet}
$R_s=(\mathsf{rk}_s,\mathsf{sk}_s, \mathsf{pk}_s, i)$,
where $\mathsf{rk}_s$ is a symmetric \emph{chain key},
$\mathsf{sk}_s/\mathsf{pk}_s$ is a signing key-pair, and $i$ is
the \textit{ratchet index}
\end{enumerate}
\noindent The protocol exports four operations:
\begin{itemize}[leftmargin=*]
\item \textsc{SessionShare}$(u_s)$ --- sends
$(\mathsf{rk}_s,\mathsf{pk}_s, i)$ through each $D_{s,j}$;
\item \textsc{SessionGen}$(u_s)$ --- generates a new ratchet key $\mathsf{rk}_s$,
computes a new signing key pair $(\textsf{pk}_s, \textsf{sk}_s)$ with $\textsf{SignGen()}$,
sets $i=0$, and stores $(\mathsf{rk}_s,\mathsf{sk}_s, \mathsf{pk}_s, i)$.
The previous session locally purged; however other peers may choose to hold onto the old session.
\item \textsc{Send}$(u_s,m)$ --- updates
$\mathsf{rk}_s\leftarrow \textsf{Hash}(\mathsf{rk}_s)$,
computes \texttt{++i},
$c = $ \textsf{Encrypt($k$, $m$)}, and $\sigma_c$ = \textsf{Sign(\textsf{sk}$_s$, $c$)},
then uploads ($c$, $\sigma_c$, $i$) to the fan-out server. The fan-out server forwards the message to each receiving peer.
\item \textsc{Recv}$(u_r,c, \sigma_c, i_c)$ --- retrives the
stored session $(\textsf{rk}_r, \textsf{pk}_r, i_r)$,
computes $\textsf{CheckSign}(\textsf{pk}_r, c, \sigma_c)$, iterates
$\mathsf{rk}_r\leftarrow \textsf{Hash}(\mathsf{rk}_r)$ $i_c - i_r$ times,
then finally computes $m = \textsf{Decrypt(rk$_r$, $c$)}$
\end{itemize}
\end{mydef}
\subsection{Real World Instantiations of the Nested Ratchet Protocol}
\label{sec:section label}
Instantions of the nested ratchet protocol as defined in the previous section are widely deployed in practice.
Megolm as deployed by the Matrix protocol uses 3DH for the P2P layer, and a custom ratchet scheme for the fan-out layer. ``Sender Keys'' as deployed
by Whatsapp and Facebook Messenger, choose to use X3DH for the P2P layer, while Signal uses PQXDH in composition with a secure membership maintence protocol \cite{Chase_Perrin_Zaverucha_2020} for the P2P layer. We provide an example of the Nested Ratchet Protocol instantiated with 3DH as the P2P layer (similar to Matrix's Megolm) in Figure \ref{fig:megolm}. Furthermore, a comparison of the prominent nested ratchet protocol implementations deployed in practice is shown in Table \ref{tab:crypto-summary}.
\input{diagrams/megolm.tex}
\begin{table*}[ht]
\caption{Comparison of Nested Ratchet Protocol Implementations}
\label{tab:crypto-summary}
\centering
\scriptsize
\setlength{\tabcolsep}{6pt} % More padding between columns
\renewcommand{\arraystretch}{1.2} % More vertical space
\begin{tabularx}{0.8\textwidth}{l Y Y Y Y Y}
\toprule
\textbf{Protocol} & \textbf{P2P AKE} & \textbf{P2P Msging} &
\textbf{Fan-out Ratchet} & \textbf{Fan-out Enc.} & \textbf{Fan-out Sig.} \\
\midrule
Signal\textsuperscript{\dag} \cite{SignalSenderKeysRust} & PQXDH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
WhatsApp \cite{WhatsAppSecurity2024} & X3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Matrix\textsuperscript{\P} \cite{matrixorg_megolm_doc} & 3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Facebook Messenger \cite{MetaMessengerE2EE2023} & X3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Session \cite{Jefferys2020SessionProtocol} & Auth.DH & XChaCha20-Poly1305 & None & XChaCha20-Poly1305 & Curve25519 \\
\bottomrule
\end{tabularx}
\vspace{1pt}
\textsuperscript{\dag}For Signal, Sender Keys was obscoleted by Signal Groups v2 \cite{Chase_Perrin_Zaverucha_2020}, yet still \\ employed in-practice \cite{Balbas_SK}. Key Transparency \cite{mcmillion2025keytransparencyarchitecture} is used for identity verification. \\
\textsuperscript{\P}For Matrix, a MAC is included in fan-out messages \cite{matrixorg_megolm_doc}.
\end{table*}

11
sections/conclusion.tex Normal file
View File

@@ -0,0 +1,11 @@
We have presented a formal description of the nested ratchet protocol, instantiated with representative designs used in practice
such as Sender Keys and Megolm, as well as the first
unified, mechanized cryptographic analysis.
Our results demonstrate that core guarantees, including
secrecy, mutual authentication, perfect forward secrecy,
and post-compromise security, hold under standard assumptions.
We also present the first mechanized treatment of offline deniability
for this class of protocols.
We provide a detailed analysis of the compromise scenarios across pairwise and fan-out layers, clarifying how failures propagate and when recovery occurs (e.g. upon session regeneration or double ratchet-based self-healing).
Our analysis and formal models underscore two key design choices when constructing and implementing a nested ratchet protocol: (1) the deniability and trust implications of signed vs. unsigned pre-keys, and (2) the trade-off between non-reputation and deniability when choosing signatures vs. MACs for fan-out. Based on the tradeoffs we identity, as well as our formal analysis, we provide additional concrete recommendations for protocol implementers. Ultimately, in addition to the concrete results our analysis provides, our work demonstrates the utility of formal methods and computer-aided reasoning for constructing and verifying secure protocols.

14
sections/discussion.tex Normal file
View File

@@ -0,0 +1,14 @@
We elaborate on our findings and provide concrete recommendations for implementers of the nested ratchet protocol and its derivatives.
\subsection{Recommendations for protocol implementers}
\label{sec:recs}
\noindent\textbf{Let users choose to leave P2P-layer pre-keys unsigned}. We recommend implementers of the nested ratchet protocol and its derivatives to provide users with the option to leave P2P-layer authenticated key exchange pre-keys (i.e. those used in 3DH/X3DH/PQXDH) unsigned. This way, mutual deniability of the P2P layer and thus mutual deniability of the cumulative nested ratchet protocol is maintained. In this case, the user should be notified of the tradeoff between trusting the central server and mutual deniability.
\noindent\textbf{Use forward-secure signatures for fan-out layer messaging}.
We find nested ratchet protocol derivatives, including Sender Keys and Megolm, needlessly compromise on post-compromise deniability guarantees by signing all messages in a single fan-out layer session with one private signing key. If a session's public signing key is compromised (i.e. any of the users in the group are compromised), every message in that session, past and future, may be provably attributed to the sender. To avoid this scenario, we advise implementors of the nested ratchet protocol or its derivatives to employ \textit{forward-secure signatures} in fan-out layer sessions as opposed to static signatures \cite{Itkis_Reyzin_2001}. Using such a primitive, signing keys may be ratcheted concurrently with the symmetric encryption key to provide perfect forward deniability.
% https://www.cs.bu.edu/~reyzin/papers/forwardsig-optimal.pdf
\noindent\textbf{Do not use fan-out layer encryption for two-party channels}.
Federated secure communication protocols like Matrix, which employ Megolm as a critical sub-protocol, choose to ensure even peer-to-peer messaging uses the fan-out layer protocol. Matrix ``rooms'' always use Megolm over Olm, even for 1:1 rooms.\footnote{See \href{https://matrix.org/docs/matrix-concepts/end-to-end-encryption/}{matrix.org/docs/matrix-concepts/end-to-end-encryption}. Note, direct messages (i.e. ``m.direct'' in element) are still constructed as rooms.} Our analysis clearly shows this should not be done: Olm should be always used for the 1:1 setting. Fan-out layer protocols such as Megolm intentionally compromise on post-compromise security and initiator deniability to accomidate the group chat setting specifically, and thus should not be used over Olm or Signal in the two-party setting.

80
sections/intro.tex Normal file
View File

@@ -0,0 +1,80 @@
End-to-end encryption represents the foundation for
security, privacy, trust, and compliance for all services
on the internet. Several protocols supporting end-to-end encryption
exist, all serving different purposes.
Transport Layer Security (TLS) \cite{rfc8446} and
Quick UDP Internet Connections (QUIC) \cite{rfc9369}
secure web traffic; Wireguard \cite{Donenfeld_2017},
OpenVPN \cite{openvpn} secure point-to-point tunneling, and Tor \cite{Dingledine_Mathewson_Syverson_2004} secures multi-hop routing.
One such use case --- instant messaging --- has become particularly
ubiquitous in today's society.
While secure instant messaging protocols strive for the same
high-level cryptographic properties as other standard protocols
such as TLS --- secrecy, authentication, and integrity --- instant
messaging carries several subtleties that lead to differences in
overall protocol design. First, messaging is \textit{asynchronous}:
an \textit{online} peer must be able to send messages to an
\textit{offline} peer, and the offline peer must then receive
the messages upon coming online. Therefore, parties must rely
on a potentially untrusted central server to initiate
authentication, key exchange, and message coordination. Second,
conversations are \textit{long-lived}; unlike TLS connections,
which typically last a few seconds, instant messaging conversations
may go on for years and carry both sensitive and non-sensitive
messages. Thus, it is likely an endpoint will be compromised
during a conversation lifetime, necessitating measures protecting
conversation contents in long-term key compromise scenarios.
Third, conversations are ideally \textit{deniable}, allowing
participants to protect their own privacy by plausibly denying
the authorship of a given message or transcript. Fourth, message
transcripts are ideally \textit{restorable}: users expect to be
able to restore their message conversation history from a server
given the right credentials, thereby introducing additional
challenges for the integrity, authentication, and confidentiality
of stored messages on untrusted servers.
For our properties of interest, it is unquestionable that the
Signal protocol has fully emerged as the best-in-class solution
for peer-to-peer messaging. The Signal protocol offers secrecy,
authentication, integrity, deniability, and asynchronicity
through the Extended Triple Diffie-Hellman handshake \cite{Marlinspike_Perrin_X3DH}, long-lived
conversation security through the post-compromise and perfect
forward secrecy guarantees of the double ratchet \cite{Moxie_DoubleRatchet}, restorability
through Signal's sub-protocol, Sesame \cite{Moxie_Sesame}, and post-quantum security
through the advent of the PQXDH key agreement protocol \cite{Kret_Schmidt_PQXDH}. Signal
has been the subject of ample formal analysis, rigorously proving
both on-paper and with proof assistants that each of the
aforementioned
properties hold \cite{Bhargavan_PQXDH, cremers_signal, alwen_doubleratchet, VatandasDeny, bhargavan_dy}.
Signal, today, is extremely dominant in secure messaging. It is a reasonable statement to say that every modern end-to-end encrypted messaging protocol is, in fact, a derivative of the Signal protocol. However, the Signal protocol has a single critical caveat: its excellent guarantees fall apart for group messaging. The group messaging scenario carries similar subtleties and desired high-level properties, with the obvious caveat of a shared transcript history. However, at the time of writing, no universal solution akin to what Signal is for peer-to-peer messaging exists for secure group messaging. The current state of secure group messaging is fragmented between tree-based group key agreement schemes such as Messaging Layer Security \cite{rfc9420}, and protocols that \textit{compose} a group fan-out layer with secure peer-to-peer channels such as Sender Keys and Megolm. While there has been excellent recent work done on Messaging Layer Security \cite{Wallez_TreeSync, Wallez_TreeKEM}, the composite-type protocols remain dominant for group messaging in-practice. Sender Keys is directly deployed by WhatsApp \cite{WhatsAppSecurity2024}, Facebook Messenger \cite{MetaMessengerE2EE2023}, Signal \cite{SignalSenderKeysRust}, and Session \cite{Jefferys2020SessionProtocol}; Megolm is deployed by Matrix,\footnote{See \href{https://matrix.org/docs/matrix-concepts/end-to-end-encryption/}{matrix.org/docs/matrix-concepts/end-to-end-encryption} for more details.} which is in turn widely deployed by both governemnts and the private sector \cite{Albrecht_Dowling_Jones}.
Thus, to analyze this flavor of group communication protocol, namely Sender Keys and Megolm, within a single security model, we define the generalized \textit{nested ratchet protocol} primitive. We define the nested ratchet protocol as a protocol that constitutes two main features: (1) a double ratchet-based peer-to-peer channel such as Signal, which securely communicates (2) a \textit{session} containing a signing keypair and symmetric key, which is then ratcheted forward to encrypt and decrypt fanned out group communications. Informally, we name these two composite features the \textit{peer-to-peer (P2P)} layer and the \textit{fan-out} layer respectively.
While there has been previous work studying nested ratchet protocols, namely Sender Keys \cite{Balbas_SK} and Megolm \cite{Albrecht_Dowling_Jones} in isolation, several questions are left unanswered by the literature:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
\item \noindent\textbf{Nested ratchet protocols generally assume the underlying peer-to-peer channel is secure. What happens if that assumption is broken?}
Broadly, nested ratchet protocols treat the underlying peer-to-peer channels as a black box and assume their security \cite{matrixorg_megolm_doc}. If this assumption fails, how is the broader fan-out layer protocol affected? If peer-to-peer channels have post-compromise security guarantees (i.e. double ratchet is used), what do recovery scenarios look like? The failure and recovery scenarios of nested ratchet protocols have yet to be thoroughly studied.
\item \noindent\textbf{What are the deniability guarantees of nested ratchet protocols?}
The deniability of Triple Diffie Hellman, Extended Triple Diffie Hellman, PQXDH, and Authenticated Key Exchange --- the peer-to-peer channel protocols --- are well-studied \cite{VatandasDeny, FiedlerPQXDHdeny}. However, it remains an open question whether or not nested ratchet protocols such as Sender Keys and Megolm preserve the deniability guarantees of their peer-to-peer channels.
\item \noindent\textbf{Do nested ratchet protocols hold up under the scrutiny of formal verification?}
In recent years it has become the standard to incorporate formal verification techniques into the design and evaluation of both novel and historical protocols, both inside and outside the cryptographic sphere \cite{SoK_CAC}. However, nested ratchet protocols, including Sender Keys and Megolm, have not received such a treatment. Previous work is hand-written and has not been mechanized \cite{Balbas_SK, Albrecht_Dowling_Jones}. Until this work, it has remained an open question whether nested ratchet protocols hold up under the scrutiny of formal verification.
\end{itemize}
A rigorous investigation into these unresolved issues is necessary to either validate or reconsider the widespread trust currently placed in nested ratchet protocols.
\textbf{Our contribution.} Our work seeks to round off and tie together the previous literature on nested ratchet protocols. To do so, we take an approach rooted in \textit{formal methods} to study the security of nested ratchet protocols, allowing us to construct computer-verified proofs or provable and explicit counterexamples. We make the following contributions.
\textbf{Models}. We present the first mechanization of nested ratchet protocols, primarily using the state-of-the-art symbolic cryptanalysis tool \pv \cite{Blanchet_2016}. Specifically, we present canonical models of the nested ratchet protocol; we also model Megolm, and Sender Keys according to their specifications. To support our larger nested ratchet protocol models, we also model 3DH, X3DH, Signal, and Olm (the secure point-to-point protocol employed by Megolm).
\textbf{Formal Analysis}. Using \pv's automated analysis suite, we prove secrecy, authentication, integrity, post-compromise security, and perfect forward secrecy for the nested ratchet protocol, including Megolm and Sender Keys. Using our models, we reason about offline deniability and the various failure cases of the nested ratchet protocol. To support our analysis, we present the first mechanizations of offline deniability, in addition to standard secrecy, authentication, integrity, post-compromise security, and perfect forward secrecy results for 3DH, X3DH, Signal, and Olm.
\textbf{Comparison}. Using our \pv models, we precisely compare the cryptographic properties and failure cases of the nested ratchet protocol. We quantify how differences in design between both the peer-to-peer and fan-out channels propagate to the overall high-level guarantees of hthe protocols.
\textbf{Code}. Our models are entirely open-source, and our results, environment, and dependencies are readily reproducible via a nix flake: \href{https://zenodo.org/records/16959099}{zenodo.org/records/16959099}

14
sections/limitations.tex Normal file
View File

@@ -0,0 +1,14 @@
We identify the limitations of our work, and discuss the respective potential lines of future work to address the limitations.
\noindent\textbf{Abstracted model}. Models in \pv generally abstract away implementation details of cryptographic protocols. Therefore, our models are not executable, and are not drop-in replacements for an already in-place implementation. Our models abstract away details required to actually implement the protocol, including low-level message formatting, key lifetimes, resource constraints, error handling, interfacing with other networking protocols, and caching. Thus, in order to further close the gap between the nested ratchet protocol specification and the real-world implementation, constructing a more detailed model in a more expressive cryptographic modeling tool is necessary. Frameworks such as DY* \cite{DY} and OwlC \cite{Gancher_2023} have shown great promise here, allowing for executable, compilable specifications whilst maintaining semi-automatic cryptanalysis capabilities.
\noindent\textbf{No post-quantum analysis}. Protocols supporting post-quantum encryption have increasingly seen real-world deployment in recent years, including PQXDH \cite{Kret_Schmidt_PQXDH}, PQ-Wireguard \cite{pqwg}, HPKE \cite{rfc9180}, and KEMTLS \cite{Schwabe_Stebila_Wiggers_2020}. The rollout of the aforementioned protocols was notably supported by formal methods tooling, most prominently \pv \cite{Lafourcade_Mahmoud_Ruhault_Taleb, Bhargavan_PQXDH} and \textsc{Tamarin} \cite{Celi_Hoyland_Stebila_Wiggers_2022}. In this work we do not support post-quantum primitives. While at the moment only Signal is the only implementor of a nested ratchet protocol that employs a post-quantum-secure P2P layer (using PQXDH), we expect the other implementors of the nested ratchet protocol to follow suit soon.
\noindent\textbf{No Key Transparency model}. Key Transparency provides a mechanism to allow communicating parties to verify each other's public identity keys via each party uploading key material to an append-only log on a central server \cite{mcMillion2025keytrans}.
Key Transparency has seen increasing popularity and deployment, being recently adopted by both Signal\footnote{Although Signal has yet to publicly document their usage of Key Transparency, the implementation is publicly available: \href{https://github.com/signalapp/key-transparency-server}{github.com/signalapp/key-transparency-server}} and Whatsapp.\footnote{WhatsApp is not open source; however, the implementation of the Key Transparency server is open source and available here: \href{https://github.com/facebook/akd}{github.com/facebook/akd}}
However, it is currently unclear how the incorporation of Key Transparency affects WhatsApp and Signal's existing deniability guarantees. Future work may seek to formalize Key Transparency and its guarantees, then fold the resulting model into existing models of Signal and WhatsApp.
\noindent\textbf{No mechanized computational analysis}. Computational reasoning about cryptographic protocols is fundamentally different from the symbolic model. As opposed to modeling cryptographic protocols with equational reasoning and specifying properties in terms of traces, the computational model is closer to hand-written cryptographic proofs in the random oracle model, proving protocols reduce to fundamental computational hardness assumptions through a sequence of computational games. The adversary in the computational model is slightly stronger than in the symbolic model \cite{Blanchet_2012}. Formal methods tooling exists for both the computational model and the symbolic model \cite{Blanchet_Jacomme}; in recent years, analysis of important protocols such as Signal have included mechanized proofs within both models \cite{Kobeissi_Bhargavan_Blanchet_2017} \cite{VatandasDeny}. Thus, future work could involve mechanizing the nested ratchet protocol, Sender Keys, or Megolm inside a computational verifier such as \textsc{CryptoVerif} \cite{Blanchet_Jacomme}

70
sections/models.tex Normal file
View File

@@ -0,0 +1,70 @@
We describe our approach for the analysis nested ratchet protocol, Megolm, and Sender Keys
%l, Megolm (as described by Matrix), Sender Keys (as described by Signal, Whatsapp, and Facebook Messenger), as well as the reliant sub-protocols 3DH, X3DH, Signal, and Olm,
using \textsc{ProVerif}. We also describe the claimed
security properties for these protocols.
\subsection{Modeling Strategy}
\label{sec:section label}
We do not go into the full details of the symbolic approach. One may refer to \cite{Blanchet_2012, Blanchet_2016} for a detailed presentation and comparison with alternative approaches.
\textbf{Cryptographic Primitives}. In the symbolic model, messages are represented as terms which can be either atomic (to represent fresh values or constants), or constructed through applying functional rules. For instance, \textsf{pk}(\textit{sk}), \textsf{Mac($key$, $msg$)}, \textsf{Encrypt(k, m)}, and \textsf{Decrypt(k, m)}. To model the behavior of cryptographic primitives, function symbols may be ruled by a set of equations, i.e. an \textit{equational theory}. For example, \textsf{Decrypt(k, Encrypt(k, m)) = m} will hold. Therefore, cryptography is assumed to be perfect in the symbolic model; if no equational rules are defined for a given primitive, it behaves as a perfect one-way function.
\textbf{Threat Model}. The Dolev-Yao attacker model is the standard attacker model assumed in symbolic cryptography \cite{Dolev_1983}. The Dolev-Yao attacker has full control over the network, and can observe, remove, duplicate, replay, synthesize, and subtitute messages on public channels. In \pv, the attacker may be configured to never gain access to certain terms (i.e. a private key), or have access to pre-computed values (i.e. precomputing \textsf{DH} to find additional attacker scenarios). \pv is capable of reasoning soundly, but not completely, about arbitrary protocols with respect to a Dolev-Yao attacker. That is, any attack or proof \pv is guaranteed within the model,\footnote{Though, one must look out for false attacks -- that is, where a property is violated by a trivial or unintended attack trace. Alternatively, cases where properties vacuously hold (i.e. because parts of a protocol are not reachable) should similarly be avoided.}
but \pv is not guaranteed to terminate.\footnote{And indeed, we employ a plethora of tricks and tweaks to ensure our models terminate reasonably quickly. See section \S 6.6.2 ``Settings'' and \S 6.7.5 ``Sources of incompleteness'' in the \pv manual \cite{ProverifManual}}
\textbf{Trace and Equivalence}. In the symbolic approach there exists two primary classes of security properties: \textit{trace properties} and \textit{equivalence} properties.
Trace properties are considered to be satisfied when, in all possible
traces or executions of the protocol, the property holds. Several security properties are expressable as trace properties. \textit{secrecy} is specified as a reachability property, i.e. whether an attacker can \textit{reach} a given secret term, such as a message. \textit{authentication} is specified as correspondance between two events; that is, if an event is reachable, another event is previously reachable. \textit{perfect forward securiy} and \textit{post-compromise security} relate secrecy queries with event timings, e.g. if a message can be compromised by the time an event occurs. On the other hand, equivalence properties decide if an attacker can \textit{tell apart} two processes; e.g. privacy properties such as deniability and unlinkability can be expressed as equivalence properties. In this work, we employ both trace and equivalence properties.
% Things to mention:
% - threat model
% - cryptographic primitives, how they're modeled
% - security and authentication
% - equivalence properties \& deniability
% - evaluation strategy
% claimed security properties
% - authentication of sessions, authentication of msgs sent with megolm
% - secrecy of sessions, secrecy of messages; resistant against unknown keyshare attacks
% - deniability
% - forward secrecy and post compromise security
\subsection{Claimed Security Properties}
\label{sec:section label}
We provide informal definitions for each property we claim the nested ratchet protocol to hold.
\attackerbox{\underline{\textbf{P1}}: Protocol participants are \textbf{Deadlock-Free}.} We claim an arbitrary number of peers who participate in the nested ratchet protocol \textit{do not deadlock} and complete the protocol, even in the presence of an attacker.
\attackerbox{\underline{\textbf{P2}}: Honest peers enjoy \textbf{Message Secrecy}.}
We claim only group members who hold the correct fan-out ratchet state can decrypt ciphertexts encrypted by honest peers, ruling out outsider eavesdropping and cross-group leakage.
\attackerbox{\underline{\textbf{P3}}: Honest protocol participants enjoy \textbf{Mutual Authentication} \& \textbf{Message Agreement.}.}
We claim that every plaintext accepted by an honest recipient is \textit{jointly authenticated}: if Alice outputs a message labelled as originating from Bob, then Bob must have previously emitted that exact message under the same epoch key. This guarantees resistance to replay, impersonation, and \textit{unknown-key-share} attacks.
\attackerbox{\underline{\textbf{P4}}: Honest peers enjoy \textbf{Offline Deniability}.}
We follow the form of \cite{Celi_Hoyland_Stebila_Wiggers_2022, Lafourcade_Mahmoud_Ruhault_Taleb} and claim it is impossible for an offline judge (that is, a judge that does not interfere with the protocol execution) to distinguish between a transcript generated between an honest initiator, and a transcript generated by a simulator.\footnote{No single approach to defining and demonstrating deniability guarantees has distinguished itself in the literature; see a classification here} On the other hand, it is well-known that responder deniability in authenticated key exchange, namely signal, is not ensured in the offline judge model. Thus, we also consider responder \textit{undeniability}.
%We claim that for the initiator of each Double Ratchet channel, message authorship cannot be proven to a coercer and deniability is retained. On the other hand, it is well-known that responder deniability in authenticated key exchange, namely Signal, is not ensured. Thus, we also claim all keys in the initial key exchange must remain unsigned by long-term static keys to ensure mutual deniability.
% https://eprint.iacr.org/2022/1111.pdf
\attackerbox{\underline{\textbf{P5}}: Messages sent by honest peers enjoy \textbf{Perfect Forward Secrecy}.}
We claim that the compromise of any static credential, including the identity keys and the medium-term pre-keys of the double ratchet channel, and the present fan-out ratchet key, does not reveal any earlier traffic from either the current session or any previous sessions.
\attackerbox{\underline{\textbf{P6}}: Between sessions, messages sent by honest peers enjoy \textbf{Post-Compromise Security}.}
We claim that once a compromised party executes a fresh \textsc{SessionGen} followed by \textsc{SessionShare}, full secrecy and authentication guaratnees are re-established for the subsequent traffic.
%The attacker’s influence is thus confined to the interval between the breach and the next honest ratchet refresh, matching the recovery guarantees of Signal.
\subsection{Additional Security Properties}
We additionally define exceptionally strong properties we do not expect to hold in composition with the aforementioned properties, but are desirable and relevant nonetheless.
\badpropbox{\underline{\textbf{P7}}: In the post-compromise and forward secrecy scenarios, deniability is maintained (i.e. \textbf{Strong Deniability} \cite{Unger_Goldberg_2018}). }
% https://cypherpunks.ca/~iang/pubs/dakez-popets18.pdf
Given the compromise of static or ephemeral credentials, ideally deniability of message authorship, both previous and future, is maintained.
\badpropbox{\underline{\textbf{P8}}: Deniability is maintained for all protocol participants (i.e. \textbf{Mutual Offline Deniability})}
%https://eprint.iacr.org/2021/642.pdf (or the thesis, https://academicworks.cuny.edu/cgi/viewcontent.cgi?article=6191&context=gc_etds)
%https://petsymposium.org/popets/2025/popets-2025-0018.pdf
It is well-known that responder deniability in mutually authenticated key exchange, namely Signal, is not ensured within the offline judge model \cite{VatandasDeny, Collins_Colombo_Huguenin-Dumittan_2025}.

74
sections/related.tex Normal file
View File

@@ -0,0 +1,74 @@
\input{diagrams/related}
We briefly review the relevant related works,
including the previous computational analysis of Megolm and Sender Keys,
the line of work on Messaging Layer Security, and relevant work on
cryptographic deniability.
\subsection{Computational Analysis}
\label{sec:section label}
The only existing work on Megolm and Sender Keys is computational.
Balbás et al. provided the first formal description and
analysis of the Sender Keys protocol, as employed by WhatsApp,
within the computational model \cite{Balbas_SK}.
Albrecht et al. introduced the
\textit{Device-Oriented Group Messaging}
model to capture the security guarantees of Matrix's Megolm protocol \cite{Albrecht_Dowling_Jones},
and later WhatsApp's Sender Keys protocol in great detail, also within
the computational model \cite{Albrecht_2025}.
We differ from the existing body of work in that our analysis is
in the \textit{symbolic} model of cryptography as opposed to the random oracle
``computational'' model. The symbolic and computational modeling approaches are
generally regarding to have complimentary benefits
\cite{Blanchet_2012}, and cryptographic analysis of
modern protocols generally involves both approaches
\cite{Kobeissi_Bhargavan_Blanchet_2017,Bhargavan_PQXDH}. In general,
symbolic analysis allows for simpler mechanization, explicit failure traces,
and greater ease in evaluating attack scenarios.
We take advantage of these benefits and provide the first
mechanization of Sender Keys \& Megolm, as well as a
cryptographic failure case analysis much more detailed than in previous works.
We also cover the same high-level properties as previous works; We provide
a complete comparison of the analyzed cryptographic properties
between the related works and our work in Table \ref{tab:symbolic-tools}.
% We differ from the existing body of work in three ways: (1)
% we \textit{mechanize} our results in a symbolic protocol analysis tool
% as opposed to hand-written analysis in the random
% oracle ``computational'' model,
% (2) we provide the first cryptographic \textit{deniability} analysis of
% Megolm and Sender Keys, and (3)
% our analysis is unified and general, covering both Megolm and Sender Keys,s
% within our \textit{nested ratchet protocol} primitive.
\subsection{Messaging Layer Security}
\label{sec:section label}
There exists a long and fruitful line of work studying the Messaging
Layer Security (MLS) protocol standardized by the IETF \cite{rfc9420},
and the underlying Continuous Group Key Agreement (CGKA) primitive \cite{Alwen_Coretti_Jost_Mularczyk_2020}. In
comparison with Sender Keys and Megolm, MLS boasts key updates that are
logarithmic (as opposed to linear) in the number of members \cite{rfc9420}; however,
MLS is relatively new and thus much less widely deployed. TreeKEM, the CGKA primitive used by MLS, has enjoyed a string of
mechanized cryptanalysis work using the symbolic protocol analysis
framework DY* \cite{Wallez_TreeKEM}. The group management scheme employed by MLS has
also seen verification using DY* \cite{Wallez_TreeSync}.
\subsection{Mechanized Deniability}
\label{sec:section label}
Cryptographic deniability, particularly of authenticated key exchanges,
has a long history in the literature. See the related works section of \cite{VatandasDeny}
for additional background. However, relatively few works have considered mechanizing
\textit{offline deniability} guarantees -- that is,
the ability to deny having participated in a communication session
\textit{after} it has taken place. One technique for proving offline
deniability with respect to a peer involves proving any execution of a session may be
\textit{simulated} with only access to the aforementioned peer's public key material \cite{VatandasDeny}.
In recent years there has
surfaced a technique for mechanizing offline deniability through the
\textit{observational equivalence} feature of symbolic cryptographic provers.
This technique has been employed to reason about the deniability of
KEMTLS \cite{Celi_Hoyland_Stebila_Wiggers_2022} and Wireguard \cite{Lafourcade_Mahmoud_Ruhault_Taleb} using \textsc{Tamarin} \cite{Basin_Cremers_Dreier_Sasse_2022} and \textsc{ProVerif} \cite{ProverifManual} respectively. We employ this technique to provide the first comprehensive, mechanized deniability analysis of Sender Key and Megolm. Furthermore, we evaluate deniability across different compromise scenarios, as is done in \cite{Lafourcade_Mahmoud_Ruhault_Taleb}.

13
sections/usenix.tex Normal file
View File

@@ -0,0 +1,13 @@
\section{Ethical Considerations}
\label{sec:section label}
We strictly adhere to ethical norms as well as the described ethical guidelines. We confirm our research considers potential negative impact and obeys legislation. Although we point out practical weaknesses in the design of deployed group communication protocols, namely Matrix, in Section \ref{sec:recs}, we emphasize these weaknesses concern only protocol design. Our research does not target a particular implementation nor product that embeds Matrix or other nested ratchet protocols. We plan to alert the stakeholders of our findings immediately after submission, and we do not plan to publicize our work onto a pre-print server until our findings are acknowledged.
\section{Open Science}
\label{sec:section label}
We strictly adhere to and enthusiastically embrace the Open Science guidelines of USENIX Security 2026. All of our methodology, models, symbolic analysis in \textsc{ProVerif}, and nix flakes for environment reproducibility are thoroughly documented and openly shared in our companion artifact publicly available on Zenodo: \href{https://zenodo.org/records/16959099}{zenodo.org/records/16959099}
Our companion artifact contains a nix flake, a nix lock file, a \texttt{README.md}, and a folder, \textbf{proverif}, that contains our \textsc{ProVerif} models of 3DH, X3DH, Signal, Olm, Sender Keys, and Megolm. Secrecy, authentication, post-compromise security, and forward secrecy properties are contained are grouped in the same model file, and offline deniability properties for both the initiator and responder are contained independently in a sub-folder \textbf{deniability}. Our README documents the many of the implementation-specific techniques and tricks that went into modeling Sender Keys, Megolm, and their sub-protocols.