\begin{abstract}
Nested ratchet protocols—such as Sender Keys and 
Megolm—combine pairwise peer-to-peer
double-ratchet channels with a server-assisted fan-out layer to scale 
end-to-end encrypted group messaging. Despite the widespread deployment
of nested ratchet protocols, including in WhatsApp, Signal, Matrix, and Facebook Messenger, their security properties are typically analyzed
piecemeal rather than in a single unified model. Thus, we define the unified nested ratchet protocol primitive to capture the security guarantees of Sender Keys and Megolm. We present a symbolic, mechanized model of the nested ratchet protocol in ProVerif, and instantiate it with canonical designs and faithful encodings of Sender Keys and Megolm. We formalize and evaluate core properties, including message secrecy, mutual authentication, perfect forward secrecy, post-compromise security, and offline deniability. Using our models, we systematize compromise scenarios across the pairwise and fan-out layers, quantify how attacks propagate, and identify recovery conditions induced by
session (re)generation. We analyze two design tradeoffs, 
including signed vs. unsigned pre-keys at the peer-to-peer layer (server trust vs. mutual deniability), and signatures vs. MACs at the fan-out layer (non-reputation vs. deniability).
Our analysis also surfaces actionable recommendations for protocol implementors, including the use of forward-secure signatures, peer-to-peer layer pre-key signing, and fan-out layer encryption usage for two-party channels. We release our ProVerif models and scripts as reproducible artifacts to facilitate verification and comparison of future designs.
\end{abstract}
