Files
usenix-2026-nested/sections/analysis.tex
2025-10-25 03:54:21 -04:00

94 lines
9.0 KiB
TeX

We describe our analysis of the base nested ratchet protocol Megolm (as described by Matrix), and Sender Keys (as described by Signal, Whatsapp, and Facebook Messenger).
\subsection{Sub-Protocol Properties}
\label{sec:section label}
Building up to our models of the nested ratchet protocol, Megolm, and Sender Keys, we construct simpler models of 3DH, X3DH, Olm, and Signal in isolation. For 3DH and X3DH, we re-prove the classic results on message secrecy and mutual authentication. For Olm and Signal, we re-prove message secrecy, mutual authentication, perfect forward secrecy, and post-compromise security. Notably, we also offer the first mechanization of offline initiator deniability and offline responder undeniability results for 3DH, X3DH, Olm, and Signal.
\subsection{Symbolic Analysis Results}
\label{sec:section label}
We completely model the nested ratchet protocol, Megolm, and Sender Keys (encompassing Signal, Whatsapp, and Facebook Messenger) under X3DH within \textsc{ProVerif}. To inform and validate our models, we closely referenced the computational constructions of Sender Keys, Megolm, and Signal, referenced the relevant open-source implementations and documentation, and interfaced with the specification authors when necessary. We formalized and automatically proved properties P1-P6 with respect to the models of the nested ratchet protocol, Megolm, and Sender Keys. We also formalized P7 and P8 for use in our case studies.
%and Signal, Whatsapp, and Facebook Messenger (
%We prove properties P1-P6 hold
%- secrecy
%- mutual authentication
%- initiator deniability for megolm sessions
%- responder undeniability for megolm sessions
%- perfect forward secrecy over megolm msgs
\subsection{Failure Case Analysis}
\label{sec:section label}
Using our models, we analyze the various failure cases of the nested ratchet protocol, Megolm, and Sender Keys. For conciseness, we jointly refer to the aforementioned protocols cumulatively, as the results are very similar, unless otherwise stated. We also refer to offline deniability simply as ``deniability'' unless otherwise stated. A stratification of our failure case analysis is shown in Table \ref{tab:failure-comparison}.
\subsubsection{Failure Taxonomy}
\capbox{\underline{\textbf{C1}}: Compromise of a single fan-out layer session ratchet key. \\
\textbf{Properties violated}: P2}
Message secrecy (P2) is violated until the compromised peer completes session regeneration and re-transmission. Note, perfect forward secrecy within a single fan-out session (P5) is maintained because session symmetric keys are ratcheted forward with each message.
\capbox{\underline{\textbf{C2}}: Compromise of a single fan-out layer session public signing key. \\
\textbf{Properties violated}: P4}
Message deniability (P4) is violated until the compromised peer completes session regeneration and re-transmission. An observer may check the signature on all messages, past and present, within the session and prove attribution to the sender. Note, (P7) is also violated, as both post-compromise deniability and perfect forward deniability within sessions does not hold.
\capbox{\underline{\textbf{C3}}: Compromise of a single fan-out layer session private signing key. \\
\textbf{Properties violated}: P3, P4 for nested ratchet and Sender Keys; P4 for Megolm}
Message deniability (P4) is violated for the nested ratchet protocol and its derivatives. Message authentication (P3) is violated for the nested ratchet protocol and Sender Keys; Megolm chooses to MAC messages, ensuring authentication remains in the case the private signing key is compromised. Both properties remain violated until session regeneration and re-transmission.
\capbox{\underline{\textbf{C4}}: Compromise of a single complete fan-out layer session, including the public and public signing keys, and ratchet key. \\
\textbf{Properties violated}: P2, P3, P4}
This case can be seen as a composition of the previous cases. In the case a singular fan-out layer session compromise, message secrecy (P2), authentication (P3), and deniability (P4) are compromised until session re-generation and re-transmission.
\capbox{\underline{\textbf{C5}}: Mutual Compromise of long-term static identity keys of a single P2P channel. \\
\textbf{Properties violated}: None}
No compromises: unless pre-keys remain unsigned, pre-key messages are secure. Equivalently, this result also expresses resistance against unknown keyshare attacks.
\capbox{\underline{\textbf{C6}}: Compromise of a single chain key within a double ratchet P2P channel. \\
\textbf{Properties violated}: P2, P3, P4}
We observe any compromise in P2P double ratchet chain keys results in the compromise of any fan-out layer sessions enclosed within any message keys derived from the chain key. Therefore, (P2), (P3), and (P4) will be violated. Note, (P6) holds due to the self-healing property of the double ratchet.
\capbox{\underline{\textbf{C7}}: Compromise private pre-key and static identity material of a single peer. \\
\textbf{Properties violated}: P2, P3, P4} The same as the previous scenario: (P2), (P3), and (P4) are violated until the next fan-out layer session is re-generated and re-transmitted.
\subsubsection{Observed Patterns \& Insights}
The previous failure taxonomy highlights a few patterns present across both Megolm and Sender Keys:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
\item\noindent\textbf{Universal deniability failures}. Across most compromise scenarios, both Megolm and Sender Keys fail deniability (P4) and strong deniability (P7).
\item \noindent\textbf{Megolm's MAC advantage}. Only Megolm maintains authentication (P3) when the private signing key is compromised (C3) due to its additional MAC verification on fan-out messages.
\item \noindent\textbf{P2P compromises cascade}. Any P2P layer compromise (C6, C7) breaks nearly everything at the fan-out layer, showing the compositional vulnerability at play.
\item\noindent\textbf{PFS/PCS resilience}. Both Sender Keys and Megolm maintain perfect forward secrecy (P5) and post-compromise security (P6) reasonably well, validating the ratcheting design. This conclusion, however, acknowledges each peer re-establishing sessions with each other peer is a polynomial-complexity operation; this is precisely the scenario where MLS improves upon Nested Ratchet-based protocol designs.
\end{itemize}
%- leaking peer-layer ephemeral keys compromises a single session
%- leaking peer-layer long-term keys only compromises pre-key messages, and thus a single session, if pre-keys are not signed
%- leaking session ratchet keys compromises all messages until retransmission
%- leaking session signatures compromises the deniability of all messages of the session
\subsection{P2P layer pre-key post-compromise message secrecy vs mutual deniability}
\label{sec:section label}
Designers of secure point-to-point messengers in the client-server model are faced with the decision of whether users should upload signed or unsigned pre-key material to the central server.
If pre-key messages are \textit{signed} by the long-term identity key, users no longer to place trust the central server; however, offline deniability for the uploader of the pre-key material (also generally the responder to the authenticated key exchange handshake) trivially does not hold, as a judge may simply be presented with the signed pre-keys as proof of protocol participation. If pre-key messages remain \textit{unsigned} more trust must be placed in the central server, as handshake becomes vulnerable to an unknown keyshare attack and thus results in pre-key message leakage if static identity keys are leaked. However, mutual deniability (i.e. property P8) is completely ensured as pre-keys are no longer signed.
The Olm documentation briefly alludes to this tradeoff. We go ahead and mechanize this tradeoff in \textsc{ProVerif}, and most importantly show the entire nested ratchet protocol is deniable to parties external to the group, modulo compromises, if all P2P-layer pre-keys remain unsigned. One observation we make is mutual deniability in the P2P layer is not sufficient to post-compromise and perfect forward deniability (a la property P7) for the entire protocol, which we now elaborate upon.
\subsection{Fan-out layer non-repudiation vs deniability}
\label{sec:section label}
We identify a similar trade-off in the fan-out layer of the nested ratchet protocol and its derivatives. Signing all fan-out layer messages with a private key serves to ensure messages cannot be forged within a group; however, in the case any one of the peers are compromised, thereby compromising the public key, deniability for all past and future messages within the session is compromised (i.e. property P7 is violated).
Thus, the nested ratchet protocol may choose to leave fan-out layer messages \textit{unsigned} and alternatively ensure message authentication via MACs. We mechanize this tradeoff in \textsc{ProVerif}. We observe signatures versus MACs trades non-reputation for mutually deniability both \textit{within} the group (i.e. property P8), as well post-compromise and forward secret deniability (i.e. property P7).
\input{diagrams/fail-cases}