We build up the necessary machinery for our work, including descriptions of the underlying cryptographic primitives, handshakes, component sub-protocols, and a formal description of the nested ratchet protocol.

\subsection{Cryptographic Building Blocks}
\label{sec:section label}

We overview the relevant cryptographic primitives and specify the notation we employ.

\noindent\textbf{Cryptographic Signatures}. A signature scheme consists
of three algorithms: \textsf{SignGen()}, which outputs a public/private 
keypair \textsf{(pk, sk)}; \textsf{Sign(\textsf{sk}, \textit{m})}, which 
$\sigma$; \textsf{CheckSign(\textsf{pk}, \textit{m}, $\sigma$)}, 
which takes a public key \textsf{pk}, a message \textit{m}, and a signature
$\sigma$, and outputs a boolean indicating the validity of the signature.
For a given keypair \textsf{(pk, sk)}, the algebraic relation \textsf{CheckSign(pk, \textit{m}, Sign(sk, \textit{m}))} $=$ \textsf{True}
will hold.

\noindent\textbf{Diffie-Hellman (DH) key exchange}. The DH
key exchange is a cryptographic method that allows 
two parties to establish a shard secret over an insecure channel. 
Diffie-Hellman consists of two functions: \textsf{DH\_Gen()}, which outputs
a public/private keypair \textsf{(pk, sk)}, and \textsf{DH(sk, pk)}, 
which outputs a string. For two given keypairs generated by \textsf{DH\_Gen()}
\textsf{(pk\textsubscript{A}, sk\textsubscript{A})}, \textsf{(pk\textsubscript{B}, sk\textsubscript{B})}, the algebraic relation
\textsf{DH(sk\textsubscript{A}, pk\textsubscript{B})} $=$ \textsf{DH(sk\textsubscript{B}, pk\textsubscript{A})} holds.

\noindent\textbf{Message Authentication Codes (MAC)}. 
A MAC provides message authenticity and integrity assurance. 
It consists of two algorithms: \textsf{MAC(k, \textit{m})} which takes
a secret key \textsf{k} and message \textit{m}, outputting a 
tag \textsf{t}; and \textsf{CheckMac(k, \textit{m}, t)} which outputs 
a boolean indicating validity. Formally, 
\textsf{CheckMac(k, \textit{m}, MAC(k, \textit{m}))} = \textsf{True} holds for all correctly computed tags. MACs instantiated with a hash function are denoted as \textsf{HMAC}.

\noindent\textbf{Symmetric Encryption}. Symmetric encryption provides confidentiality using a single shared secret key for both encryption and decryption. A symmetric encryption scheme consists of two algorithms: \textsf{Encrypt(k, \textit{m})} outputs ciphertext \textsf{c} given a key \textsf{k} and message \textit{m}; \textsf{Decrypt(k, c)} outputs the original message \textit{m}. The algebraic relation \textsf{Decrypt(k, Encrypt(k, m))} = \textsf{m} holds for all correctly encrypted messages.

\noindent\textbf{Authenticated Encryption with Associated Data (AEAD)}. AEAD schemes simultaneously ensure message confidentiality, authenticity, and integrity, optionally incorporating associated data. AEAD is defined by two algorithms: \textsf{AEAD\_ENC(k, \textit{m}, \textit{ad})}, outputting a ciphertext \textsf{c} given a key \textsf{k}, message \textit{m}, and associated data \textsf{ad}; \textsf{AEAD\_DEC(k, c, \textit{ad})} returns message \textit{m} or a symbol indicating failure otherwise. The algebraic relation \textsf{AEAD\_DEC(k, AEAD\_ENC(k, \textit{m}, \textit{ad}), \textit{ad})} = \textsf{m} holds.

\noindent\textbf{Hash Functions}. Cryptographic hash functions deterministically map arbitrary-length inputs to fixed-length outputs, satisfying collision resistance and pre-image resistance. Formally, a hash function \textsf{Hash(m)} outputs a string \textsf{h} of fixed length. Computational hardness ensures it is infeasible to find distinct \textsf{m} and \textsf{m'} where \textsf{Hash(m)} = \textsf{Hash(m')} or to derive \textsf{m} from \textsf{h}.

\noindent\textbf{Key Derivation Functions}. A Key Derivation Function (KDF) is a cryptographic algorithm that derives one or more secrets from a master secret. We universally employ HMAC-based Extract-and-Expand Key Derivation Function (HKDF) \cite{rfc5869} instantiated with a collision-resistant hash function. We denote HKDFs as \textsf{HKDF(master)} $\rightarrow$ \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})}, where \textsf{master} denotes master key, and \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} denotes the derived keys.

\newcolumntype{L}{>{\raggedright\arraybackslash}p{4cm}}
\newcolumntype{R}{>{\raggedright\arraybackslash}p{7cm}}
\begin{table}[h]
    \footnotesize
    % \small
    \centering
    % \begin{tabular}{L!{\color{gray}\vrule width 0.4pt}R}
    % \begin{tabular}{L!{\color{black!70}\vrule width 0.4pt}R}
    \begin{tabularx}{\columnwidth}{l!{\color{black!70}\vrule width 0.4pt\hspace{0.5em}}X}
    \toprule
    \textbf{Symbol} & \textbf{Description} \\
    \midrule
    \textsf{SignGen()} $\rightarrow$ \textsf{(sk, pk)} & Generates a signing keypair \\
    \textsf{Sign(\textsf{sk}, \textit{m})} $\rightarrow$ $\sigma$ & Produces a signature $\sigma$ on \textit{m} using \textsf{sk} \\
    \textsf{CheckSign(\textsf{pk}, \textit{m}, $\sigma$)} & Takes a public key \textsf{pk}, a string \textit{m}, and a signature $\sigma$; outputs a boolean indicating validity \\
    \midrule
    \textsf{DH\_Gen()} $\rightarrow$ \textsf{(sk, pk)} & Generates a DH keypair \\
    \textsf{DH(sk, pk)} $\rightarrow$ str & Computes DH exponentiation \\
    \midrule
    \textsf{X25519\_Gen()} $\rightarrow$ \textsf{sk, pk} & Generates a keypair capable of signing and encryption \\
    \midrule
    \textsf{MAC(k, \textit{m})} $\rightarrow$ \textsf{t} & Takes
a key \textsf{k} and message \textit{m}; outputs a 
tag \textsf{t} \\
    \textsf{CheckMac(k, \textit{m}, t)} & Takes
    a key \textsf{k}, a message \textit{m}, and a tag \textsf{t}; outputs a boolean indicating validity \\
    \midrule
    \textsf{Encrypt(k, \textit{m})} $\rightarrow$ \textsf{c} & Takes a key \textsf{k}, a plaintext \textit{m}; ouptuts a ciphertext \textsf{c} \ \\
    \textsf{Decrypt(k, c)} $\rightarrow$ \textit{m} & Takes a key \textsf{k}, a ciphertext \textsf{c}; outputs the plaintext \textit{m} \\
    \midrule
    \textsf{AEAD\_ENC(k, \textit{m}, \textit{ad})} $\rightarrow$ \textsf{c} & Takes a key \textsf{k}, a plaintext \textit{m}, and associated data \textit{ad}; outputs a ciphertext \textsf{c} \\
    \textsf{AEAD\_DEC(k, c, \textit{ad})} $\rightarrow$ \textit{m} & Takes a key \textsf{c}, a ciphertext \textsf{c}, and associated data \textit{ad}; outputs plaintext \textit{m} \\
    \midrule
    \textsf{Hash(m)} $\rightarrow$ \textsf{h} & Takes a string \textsf{m}; outputs a hash \textsf{h}. Assumed to be collision-resistant. \\
    \midrule
    \textsf{HKDF(mk)} $\rightarrow$ \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} & Takes a ``master'' key \textsf{mk}, and outputs a set of keys \textsf{(k\textsubscript{1}, \ldots, k\textsubscript{n})} \\
    \bottomrule
    \end{tabularx}
    \caption{Notation for employed Cryptographic primitives.}
    \label{tab:symbols}
\end{table}

\subsection{Handshakes}
\label{sec:section label}

Fundamental to nested ratchet protocols is authenticated key exchange (AKE), which builds on Diffie-Hellman with cryptographic signatures to establish authenticated, shared secrets between two parties over an insecure channel \cite{auth}. AKE generally relies upon verification of public/private key pairs between parties \textit{out-of-band} to prevent unknown key-share attacks.

To achieve \textit{asynchronous} messaging, as well as cryptographic deniability for the conversation intiator, AKE is extended with an always-online intermediary server that (1) retains initial key exchange ``pre-key'' material of users, and (2) retains and transmits encrypted messages. The intermediary server need not be trusted to ensure the secrecy, authenticity, and integrity of transmitted messages.

\input{diagrams/ake-server.tex}

In practice, pre-key bundles do not just consist of the (signed) public material of a single key pair. Multiple key pairs are generally employed, with Diffie-Hellman exchanges crossed between the different key pairs, and the result concatonated together and passed through a key derivation function. 

\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
  \item \textit{Triple Diffie-Hellman} (3DH) combines DH operations between static long-term keys and signed ephemeral one-time use keys -- this way, the derived key remains secure in the case long-term are later compromised \cite{matrixorg_olm_repo}.
  \item  \textit{Extended Triple Diffie-Hellman} (X3DH), which was notably used by Signal for scaling asynchronous messaging, combines long-term static keys, signed medium-term pre-keys, and one-time use pre-keys -- one-time pre-keys are selected from a set of one-time pre-keys provided to the server. This way, each asynchronous handshake uses a unique one-time pre-key \cite{Marlinspike_Perrin_X3DH}.
  \item  \textit{Post-Quantum Extended Diffie-Hellman} (PQXDH), combines Diffie-Hellman key exchanges over classical primitives such as RSA or elliptic curve cryptography with post-quantum primtivies via key encaptulation \cite{Kret_Schmidt_PQXDH}.
\end{itemize}

\subsection{Signal, Olm, and the Double Ratchet}
\label{sec:section label}

After deriving a symmetric key through an authenticated handshake protocol, secure peer-to-peer messaging in modern protocols, including Signal and Olm, almost universally employ the \textit{Double Ratchet} algorithm. We only describe the Double Ratchet in brief -- we refer the reader to \cite{Moxie_DoubleRatchet} for the complete description. Messages transmitted with the Double Ratchet algorithm, notably, carry post-compromise security and perfect forward secrecy guarantees in addition to the secrecy and authentication guarantees one would achieve with an authenticated encryption scheme. In short, the Double Rachet achieves this by (1) continually hasing ``racheting'' forward the shared symmetric key, (2) continually re-negotiating new Diffie-Hellman keys, and encrypting messages with a key derived via passing the results from the aforemetioned steps into a key derivation function.

\subsection{Session Sharing \& Server-Side Fan-Out}
\label{sec:section label}

One could reasonably argue secure peer-to-peer messaging is ``solved'' by composing X3DH or PQXDH with the Double Ratchet, as we achieve our desired properties: secrecy, authentication, deniability, post-compromise security, perfect forward secrecy, and asynchronicity. However, the same certainly cannot be said for secure group communication protocols. Group messaging protocols must cope with two orthogonal scaling dimensions:
\begin{itemize}[leftmargin=*, align=parleft, label=~$\bullet$~]
  \item \textit{Group fan-out}. A single ciphertext created by the sender should efficiently reach $n-1$ recipients without $n-1$ separate public-key operations at the sender.
  \item \textit{Dynamic Group Membership}. Group membership does not remain fixed: peers may be added or removed, necessitating another round of key agreement to add the new user or exclude the old user.
\end{itemize}

\noindent Both problems are addressed with \textit{session-sharing messages} and a \textit{server-side fan-out} layer that sits \textit{above} the pairwise Double Ratchet channels and \textit{below} the application payload. In general, message communication within a group relies upon the key material within sessions as opposed to the pairwise Double Ratchet channels.

%To communicate within a group, peers rely on \textit{sessions} to secure transmitted messages as opposed to the pairwise Double Ratchet channels.
\noindent\textbf{Sessions}. Broadly, \textit{sessions} include a symmetric ratchet key for message encryption and encryption, as well as a signing key pair for message identification. Each peer creates their own session. Thus, a \textit{session-sharing message}, which each peer distributes to every other peer, encaptulates the symmetric ratchet key and the public key of the signing keypair. To transmit a message, the sending peer will ratchet their symmetric key forward with a cryptographic hash function, encrypt their message, and sign; the receiving peer will check the signature, ratchet their symmetric key forward to match the sender, and decrypt.

\input{diagrams/session-trans.tex}

\noindent\textbf{Server-side Fan-out}. Once a peer's sessions are transmitted to each other peer, group communication is conducted through \textit{server-side fan-out}. That is, the sending peer will send a message to the central server, and the server will ``fan'' the message out to each receiving peer. Combined with the already-established sessions, public-key operations between pairwise peers is avoided.

\noindent\textbf{Group Updates}. Whenever the group is updated, or a peer is compromised, each peer simply generates a new session and transmits it to each other peer through the already established peer-to-peer Double Ratchet channels.

One can see post-compromise security stems from session re-establishment through group updates, and perfect forward secrecy stems from ratcheting forward the session's symmetric key. 

\subsection{Nested Ratchet Protocol Definition}
\label{sec:section label}

We are now ready to define the \textit{nested ratchet protocol}. 
Informalaly, nested ratchet protocols \textit{stack} two layers of 
distinct ratcheting protocols as such:

\begin{itemize}[leftmargin=*]
  \item \textbf{P2P layer}. Ordinary Double Ratchets-based 
        communication channels that
        carry only the occasional \textsc{SessionShare} control
        messages; they provide secrecy and authentication, but may also 
        provide deniability, perfect forward secrecy, and post-compromise security.
  \item \textbf{Fan-out layer}. A lightweight symmetric ratchet per
        sender; one ciphertext is uploaded and blindly fanned out,
        giving $O(1)$ work for the sender regardless of group size.
\end{itemize}

\noindent With this intution, we now provide the complete definition. 

\begin{mydef}[Nested Ratchet Protocol]\hfill\\
Let $U=\{u_1,\dots,u_n\}$ be a dynamic set of participants.  
A \emph{Nested Ratchet Protocol} consists of  
\begin{enumerate}[label=(\roman*),leftmargin=*]
  \item A set of pairwise \textbf{Double Ratchet channels}
        $\mathcal D=\bigl\{D_{i,j}\mid u_i,u_j\in U,\;i\neq j\bigr\}$ established through authenticated key exchange between each pair of peers
  \item For each sender $u_s\in U$, a \textbf{fan-out ratchet}  
    $R_s=(\mathsf{rk}_s,\mathsf{sk}_s, \mathsf{pk}_s, i)$,  
        where $\mathsf{rk}_s$ is a symmetric \emph{chain key},
        $\mathsf{sk}_s/\mathsf{pk}_s$ is a signing key-pair, and $i$ is 
        the \textit{ratchet index}
\end{enumerate}

\noindent The protocol exports four operations:
\begin{itemize}[leftmargin=*]
  \item \textsc{SessionShare}$(u_s)$ --- sends  
        $(\mathsf{rk}_s,\mathsf{pk}_s, i)$ through each $D_{s,j}$;
  \item \textsc{SessionGen}$(u_s)$ --- generates a new ratchet key $\mathsf{rk}_s$, 
        computes a new signing key pair $(\textsf{pk}_s, \textsf{sk}_s)$ with $\textsf{SignGen()}$,
        sets $i=0$, and stores $(\mathsf{rk}_s,\mathsf{sk}_s, \mathsf{pk}_s, i)$. 
        The previous session locally purged; however other peers may choose to hold onto the old session.
  \item \textsc{Send}$(u_s,m)$ --- updates  
        $\mathsf{rk}_s\leftarrow \textsf{Hash}(\mathsf{rk}_s)$, 
        computes \texttt{++i},
        $c = $ \textsf{Encrypt($k$, $m$)}, and $\sigma_c$ = \textsf{Sign(\textsf{sk}$_s$, $c$)},
        then uploads ($c$, $\sigma_c$, $i$) to the fan-out server. The fan-out server forwards the message to each receiving peer.
  \item \textsc{Recv}$(u_r,c, \sigma_c, i_c)$ --- retrives the 
        stored session $(\textsf{rk}_r, \textsf{pk}_r, i_r)$, 
        computes $\textsf{CheckSign}(\textsf{pk}_r, c, \sigma_c)$, iterates
        $\mathsf{rk}_r\leftarrow \textsf{Hash}(\mathsf{rk}_r)$ $i_c - i_r$ times, 
        then finally computes $m = \textsf{Decrypt(rk$_r$, $c$)}$
\end{itemize}
\end{mydef}

\subsection{Real World Instantiations of the Nested Ratchet Protocol}
\label{sec:section label}
Instantions of the nested ratchet protocol as defined in the previous section are widely deployed in practice. 
Megolm as deployed by the Matrix protocol uses 3DH for the P2P layer, and a custom ratchet scheme for the fan-out layer. ``Sender Keys'' as deployed 
by Whatsapp and Facebook Messenger, choose to use X3DH for the P2P layer, while Signal uses PQXDH in composition with a secure membership maintence protocol \cite{Chase_Perrin_Zaverucha_2020} for the P2P layer. We provide an example of the Nested Ratchet Protocol instantiated with 3DH as the P2P layer (similar to Matrix's Megolm) in Figure \ref{fig:megolm}. Furthermore, a comparison of the prominent nested ratchet protocol implementations deployed in practice is shown in Table \ref{tab:crypto-summary}. 

\input{diagrams/megolm.tex}

\begin{table*}[ht]
\caption{Comparison of Nested Ratchet Protocol Implementations}
\label{tab:crypto-summary}
\centering
\scriptsize
\setlength{\tabcolsep}{6pt} % More padding between columns
\renewcommand{\arraystretch}{1.2} % More vertical space
\begin{tabularx}{0.8\textwidth}{l Y Y Y Y Y}
  \toprule
  \textbf{Protocol} & \textbf{P2P AKE} & \textbf{P2P Msging} &
\textbf{Fan-out Ratchet} & \textbf{Fan-out Enc.} & \textbf{Fan-out Sig.} \\
\midrule
Signal\textsuperscript{\dag} \cite{SignalSenderKeysRust} & PQXDH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\ 
WhatsApp \cite{WhatsAppSecurity2024} & X3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Matrix\textsuperscript{\P} \cite{matrixorg_megolm_doc} & 3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Facebook Messenger \cite{MetaMessengerE2EE2023} & X3DH & Double Ratchet & HMAC-SHA256 & AES-CBC & Curve25519 \\
Session \cite{Jefferys2020SessionProtocol} & Auth.DH & XChaCha20-Poly1305 & None & XChaCha20-Poly1305 & Curve25519 \\
\bottomrule
\end{tabularx}

\vspace{1pt}
\textsuperscript{\dag}For Signal, Sender Keys was obscoleted by Signal Groups v2 \cite{Chase_Perrin_Zaverucha_2020}, yet still \\ employed in-practice \cite{Balbas_SK}. Key Transparency \cite{mcmillion2025keytransparencyarchitecture} is used for identity verification. \\
\textsuperscript{\P}For Matrix, a MAC is included in fan-out messages \cite{matrixorg_megolm_doc}.
\end{table*}


