Implementation and Analysis of QUIC for MQTT

Implementation and Analysis of QUIC for MQTT

Puneet Kumar and Behnam Dezfouli
Internet of Things Research Lab, Department of Computer Engineering, Santa Clara University, USA

Transport and security protocols are essential to ensure reliable and secure communication between two parties. For IoT applications, these protocols must be lightweight, since IoT devices are usually resource constrained. Unfortunately, the existing transport and security protocols – namely TCP/TLS and UDP/DTLS – fall short in terms of connection overhead, latency, and connection migration when used in IoT applications. In this paper, after studying the root causes of these shortcomings, we show how utilizing QUIC in IoT scenarios results in a higher performance. Based on these observations, and given the popularity of MQTT as an IoT application layer protocol, we integrate MQTT with QUIC. By presenting the main APIs and functions developed, we explain how connection establishment and message exchange functionalities work. We evaluate the performance of MQTTw/QUIC versus MQTTw/TCP using wired, wireless, and long-distance testbeds. Our results show that MQTTw/QUIC reduces connection overhead in terms of the number of packets exchanged with the broker by up to 56%. In addition, by eliminating half-open connections, MQTTw/QUIC reduces processor and memory usage by up to 83% and 50%, respectively. Furthermore, by removing the head-of-line blocking problem, delivery latency is reduced by up to 55%. We also show that the throughput drops experienced by MQTTw/QUIC when a connection migration happens is considerably lower than that of MQTTw/TCP.

Internet of Things (IoT); Transport layer; Application layer; Latency; Security

I Introduction

The Internet of Things is the enabler of many applications, such as the smart home, smart cities, remote medical monitoring, and industrial control, by connecting a large number of sensors and actuators to the Internet. Existing studies predict that the number of connected devices will surpass 50 billion by 2020 [1]. To facilitate interconnection and software development, the communication between IoT devices usually employs a protocol stack similar to that of regular Internet-connected devices such as smartphones and laptops. Specifically, IP (or 6LowPAN [2]) and transport layer protocols are provided by various protocol stacks (e.g., IP [3], LwIP [4]) to enable interconnectivity.

I-a TCP and UDP

The primary responsibility of the transport layer is to support exchanging segments between the two end-to-end communicating applications. Among the transport layer protocols, TCP (Transport Layer Protocol) and UDP (User Datagram Protocol) are the most widely used, depending on the application at hand. TCP provides a reliable end-to-end connection and implements congestion control mechanisms to avoid buffer overflow at the receiver. During the past couple of decades, several improved versions of TCP have been proposed to address the increasing demand for throughput [5, 6]. However, these features impose high overhead in terms of connection establishment and resource (i.e., processor, memory, energy) utilization. UDP, on the other hand, does not provide any of the above-mentioned features and therefore, its overhead is significantly lower than that of TCP.

While throughput is the main performance metric for user traffic such as voice and video, the prevalent communication type of IoT, which is machine-to-machine (M2M), is characterized by short-lived bursts of exchanging small data chunks [7, 8]. In addition, compared to user devices such as smartphones and laptops, IoT devices are usually resource constrained in terms of processing, memory, and energy [9, 10]. Using TCP in IoT domains to satisfy reliability and security requirements, therefore, imposes high overhead. Specifically, the shortcomings of TCP when used in IoT applications are as follows:

  • [itemsep=0pt, topsep=0pt, leftmargin=*]

  • Connection startup latency is highly affected by the TCP handshake. This handshake requires 1 Round-Trip Time (RTT) for TCP and 2 or 3 RTTs when TLS (Transport Layer Security) is added to this protocol [11]. The overhead impact is even higher in IoT scenarios where unreliable wireless links cause frequent connection drops [12]. In these scenarios, imposing a high connection establishment overhead for the exchange of a small amount of data results in wasting the resources of devices. TCP Fast Open [13] seeks to address this problem by piggybacking data in SYN segments in repeated connections to the same server. This solution is not scalable since the TCP SYN segment can only fit a limited amount of data [14].

  • IoT devices are often mobile, and as such, supporting connection migration is an essential requirement [15, 16, 17, 18]. However, any change in network parameters (such as IP address or port) breaks the connection. In this case, either the connection must be re-established, or a gateway is required to reroute the data flow. Unfortunately, these solutions increase communication delay and overhead, which might not be acceptable in mission-critical applications such as medical monitoring [8].

  • To preserve energy resources, IoT devices usually transition between sleep and awake states [19, 20]. In this case, a TCP connection cannot be kept open without employing keep-alive packets. These keep-alive mechanisms, however, increase resource utilization and bandwidth consumption. Without an external keep-alive mechanism, IoT devices are obliged to re-establish connections every time they wake from the sleep mode.

  • In disastrous events such as unexpected reboots or a device crash, TCP connections between client and server might end up out of state. This undefined state is referred to as TCP half-open connections [21]. A half-open connection consumes resources such as memory and processor time. In addition, it can impose serious threats such as SYN flooding [22, 23].

  • If packets are dropped infrequently during a data flow, the receiver has to wait for dropped packets to be re-transmitted in order to complete the packet re-ordering. This phenomena, which impedes packet delivery performance, is called the head-of-line blocking [24, 25, 26].

Despite the aforementioned shortcomings, several IoT application layer protocols rely on TCP, and some of them offer mechanisms to remedy these shortcomings. For example, MQTT [27] employs application layer keep-alive messages to keep the connection alive. This mechanism also enables MQTT to detect connection breakdown and release the resources.

Another transport layer protocol used in IoT networks is UDP. Generally, UDP is suitable for applications where connection reliability is not essential. Although this is not acceptable in many IoT scenarios, several IoT application layer protocols rely on UDP due to its lower overhead compared to TCP. These protocols usually include mechanisms to support reliability and block transmission (e.g., CoAP [28]).


In addition to reliability, it is essential for IoT applications to employ cryptographic protocols to secure end-to-end data exchange over transport layer. TLS [29] is the most common connection-oriented and stateful client-server cryptographic protocol. Symmetric encryption in TLS enables authenticated, confidential, and integrity-preserved communication between two devices. The key for this symmetric encryption is generated during the TLS handshake and is unique per connection. In order to establish a connection, the TLS handshake can require up to two round-trips between the server and client. However, since connections might be dropped due to phenomenons such as sleep phases, connection migration, and packet loss, the overhead of establishing secure connections imposes high overhead. In order to address this concern, a lighter version of TLS for datagrams, named DTLS (Datagram Transport Layer Security) [30], has been introduced. Unlike TLS, DTLS does not require a reliable transport protocol as it can encrypt or decrypt out-of-order packets. Therefore, it can be used with UDP. Although the security level offered by DTLS is equivalent to TLS, some of the technical differences include: the adoption of stream ciphers is prohibited, and an explicit sequence number is included in every DTLS message. Compared to TLS, DTLS is more suitable for resource-constrained devices communicating through an unreliable channel. However, similar to TLS, the underlying mechanism in DTLS has been primarily designed for point-to-point communication. This creates a challenge to secure one-to-many connections such as broadcasting and multicasting. In addition, since DTLS identifies connections based on source IP and port number, it does not support connection migration [31]. Furthermore, DTLS handshake packets are large and may fragment each datagram into several DTLS records where each record is fit into an IP datagram. This can potentially cause record overhead [32].

I-C Contributions

Given the shortcomings of UDP and TCP, we argue that the enhancement of transport layer protocols is a necessary step in the performance improvement of IoT applications. In order to address this concern, this paper presents the implementation and studies the integration of QUIC [33] with application layer to address these concerns. QUIC is a user space, UDP-based, stream-based, and multiplexed transport protocol developed by Google. According to [14], around 7% of the world-wide Internet traffic employs QUIC. This protocol offers all the functionalities required to be considered a connection-oriented transport protocol. In addition, QUIC solves the numerous problems faced by other connection-oriented protocols such as TCP and SCTP [34]. Specifically, the addressed problems are: reducing the connection setup overhead, supporting multiplexing, removing the head-of-line blocking, supporting connection migration, and eliminating TCP half-open connections. QUIC executes a cryptographic handshake that reduces the overhead of connection establishment by employing known server credentials learned from past connections. In addition, QUIC reduces transport layer overhead by multiplexing several connections into a single connection pipeline. Furthermore, since QUIC uses UDP, it does not maintain connection status information in the transport layer. This protocol also eradicates the head-of-line blocking delays by applying a lightweight data-structure abstraction called streams.

At present, there is no open source or licensed version of MQTT using QUIC. Current MQTT implementations (such as Paho [35]) rely on TCP/TLS to offer reliable and secure delivery of packets. Given the potentials of QUIC and its suitability in IoT scenarios, in this paper we implement and study the integration of MQTT with QUIC. First, since the data structures and message mechanisms of MQTT are intertwined with the built-in TCP and TLS APIs, it was necessary to redesign these data structures. The second challenge was to establish IPC (Inter-Process Communication) between QUIC and MQTT. MQTTw/TCP utilizes the available APIs for user space and kernel communication. However, there is no available API for QUIC and MQTT to communicate, as they are both user space processes. To address these challenges, we have developed new APIs, which are referred to as agents. Specifically, we implemented two types of agents: server-agent and client-agent, where the former handles the broker-specific operations and the latter handles the functionalities of publisher and subscriber. This paper presents all the functions developed and explains the connection establishment and message exchange functionalities by presenting their respective algorithms. The third challenge was to strip QUIC of mechanisms not necessary for IoT scenarios. QUIC is a web traffic protocol composed of a heavy code footprint (1.5GB [36]). We have significantly reduced the code footprint (to around 22MB) by eliminating non-IoT related code segments such as loop network and proxy backend support.

Three types of testbeds were used to evaluate the performance of MQTTw/QUIC versus MQTTw/TCP: wired, wireless, and long-distance. Our results show that, in terms of the number of packets exchanged during the connection establishment phase, MQTTw/QUIC offers a 56.2% improvement over MQTTw/TCP. By eliminating half-open connections, MQTTw/QUIC reduces processor and memory usage by up to 83.2% and 50.3%, respectively, compared to MQTTw/TCP. Furthermore, by addressing the head-of-line blocking problem, MQTTw/QUIC reduces message delivery latency by 55.6%, compared to MQTTw/TCP. In terms of connection migration, the throughput drop experienced by MQTTw/QUIC is significantly lower than that of MQTTw/TCP.

The rest of this paper is organized as follows. Section II explains the QUIC protocol along with its potential benefits in IoT applications. The implementation of QUIC for MQTT is explained in Section III. Performance evaluation and experimentation results are given in Section IV. Section V overviews the existing studies on QUIC and IoT application layer protocols. The paper is concluded in Section VI.

Ii Quic

QUIC employs some of the basic mechanisms of TCP and TLS, while keeping UDP as its underlying transport layer protocol. QUIC is in fact a combination of transport and security protocols by performing tasks including encryption, packet re-ordering, and retransmission. This section overviews the main functionalities of this protocol and justifies the importance of its adoption in the context of IoT.

Ii-a Connection Establishment

QUIC combines transport and secure layer handshakes to minimize the overhead and latency of connection establishment. To this end, a dedicated reliable stream is provided for the cryptographic handshake. Figure 1(a) and (b) show the packets exchanged during the 1-RTT and 0-RTT connection establishment phases, respectively. Connection establishment works as follows:

Fig. 1: The messages exchanged by QUIC during (a) 1-RTT and (b) 0-RTT connections.
  • [leftmargin=*]

  • First handshake. In order to retrieve the server’s configuration, the client sends an inchoate client hello (CHLO) message. Since the server is an alien to the client, the server must send a REJ packet. This packet carries the server configuration including the long-term Diffie-Hellman value, connection ID (cid), port numbers, key agreement, and initial data exchange. After receiving the server’s configuration, the client authenticates the server by verifying the certificate chain and the signature received in the REJ message. At this point, the client sends a complete CHLO packet to the server. This message contains the client’s ephemeral Diffie-Helman public value. This concludes the first handshake.

  • Final and repeat handshake. After receiving the complete CHLO packet, the client has the initial keys for the connection and starts sending application data to the server. For 0-RTT, the client must initiate sending encrypted data with its initial keys before waiting for a reply from the server. If the handshake was successful, the server sends a server hello (SHLO) message. This concludes the final and repeat handshake.

Except some handshake and reset packets, QUIC packets are fully authenticated and partially encrypted. Figure 2 shows the non-encrypted and encrypted parts of the packet using solid and dotted lines, respectively. The non-encrypted packet header is used for routing and decrypting the packet content. The flags encode the presence of cid and the length of the Packet Number (PN) field, which are visible to read the subsequent fields.

Fig. 2: The solid and dashed lines show the clear-text and encrypted parts of a QUIC packet, respectively. The non-encrypted part is used for routing and decrypting the encrypted part of the packet.

Ii-B Connection Migration

The QUIC connections are identified by a randomly generated 64-bit Connection Identifier (cid). A cid is allocated per connection and allows the clients to roam between networks without being affected by the changes in the network or transport layer parameters. As shown in Figure 2, cid resides in the header (non-encrypted part) and makes the clients independent of network address translation (NAT) and restoration of connections. The cid plays an important role in routing, specifically for connection identification purposes. Furthermore, using cids enables multipath by probing a new path for connection. This process is called path validation [37]. During a connection migration, the end point assumes that the peer is willing to accept packets at its current address. Therefore, an end point can migrate to a new IP address without first validating the peer’s IP address. It is possible that the new path does not support the current sending rate of the endpoint. In this case, the end point needs to reconstitute its congestion controller [38]. On the other hand, receiving non-probe packets [39] from a new peer address confirms that the peer has migrated to the new IP address.

Ii-C Security

For transport layer encryption, MQTTw/TCP usually relies on TLS/SSL. The primary reasons why TLS/SSL cannot be used in QUIC were described in [40]. In short, the TLS security model uses one session key, while QUIC uses two session keys. This difference, in particular, enables QUIC to offer 0-RTT because data can be encrypted before the final key is set. Thus, the model has to deal with data exchange under multiple session keys [41]. Therefore, MQTTw/QUIC uses its own encryption algorithm named QUIC Crypto [42]. This algorithm decrypts packets independently to avoid serialized decoding dependencies. The signature algorithms supported by Crypto are ECDSA-SHA256 and RSA-PSS-SHA256.

Ii-D Multiplexing

Unlike TCP, QUIC is adept in transport layer header compression by using multiplexing. Instead of opening multiple connections from the same client, QUIC opens several streams multiplexed over a single connection. Each connection is identified by a unique cid. The odd cids are for client-initiated streams and even cids are for server-initiated streams. A stream is a lightweight abstraction that provides a reliable bidirectional byte-stream. A QUIC stream can form an application message up to bytes. Furthermore, in the case of packet loss, the application is not prevented from processing subsequent packets. Multiplexing is useful in IoT applications where a large amount of data transfer is required per transaction. For example, this feature enhances performance for remote updates and industrial automation [43].

Ii-E Flow and Congestion Control

Similar to TCP, QUIC implements a flow control mechanism to prevent the receiver’s buffer from being inundated with data [14]. A slow TCP draining stream can consume the entire receiver buffer. This can eventually block the sender from sending any data through the other streams. QUIC eliminates this problem by applying two levels of flow control: (i) Connection level flow control: limits the aggregate buffer that a sender can consume across all the streams on a receiver. (ii) Stream level flow control: limits the buffer per stream level. A QUIC receiver communicates the capability of receiving data by periodically advertising the absolute byte offset per stream in window update frames for sent, received, and delivered packets.

QUIC incorporates a pluggable congestion control algorithm and provides a richer set of information than TCP [44]. For example, each packet (original or re-transmitted) carries a new Packet Number (PN). This enables the sender to distinguish between the re-transmitted and original ACKs, hence removing TCP’s re-transmission ambiguity problem. QUIC utilizes a NACK based mechanism, where two types of packets are reported: the largest observed packet number, and the unseen packets with a packet number lesser than that of the largest observed packet. A receive timestamp is also included in every newly-acked ACK frame. QUIC’s ACKs can also provide the delay between the receipt of a packet and its acknowledgement, which helps in calculating RTT. QUIC’s ACK frames support up to 256 NACK ranges in opposed to the TCP’s 3 NACK range [45]. This makes QUIC more resilient to packet reordering than TCP (with SACK). The congestion control algorithm of QUIC is based on TCP Reno to determine the pacing rate and congestion window size [45]. In addition, QUIC supports two congestion control algorithms: (i) Pacing Based Congestion Control Algorithm (PBCCA) [46], and (ii) TCP CUBIC [47]. The superior performance of QUIC’s flow control over TCP for HTTP traffic has been demonstrated in the literature [48].

Iii Integration and Implementation
of MQTT with QUIC

This section presents the integration of MQTT with QUIC and is divided into six sub-sections. The first sub-section overviews the system architecture. The second sub-section describes the definitions, methods, and assumptions. The third and fourth sub-sections explain the operations of the APIs and functions developed for the broker and clients, respectively. We present the common APIs and functions, which are used by the broker and client in the fifth sub-section. A short discussion about code reduction is presented in the sixth sub-section. In order to simplify the discussions, we refer to the publisher and the subscriber as client. The presented implementation for the client and broker are based on the open-source Eclipse Paho and Mosquitto [35], respectively.

Iii-a Architecture

Both MQTT and QUIC belong to the application layer. To streamline their integration, either the QUIC library (ngtcp2 [49]) must be imported into MQTT, or new interfaces must be created. However, the former approach is not suitable for resource-constrained IoT devices as QUIC libraries are developed mainly for HTTP/HTTPS traffic, and therefore, impose a heavy code footprint. In our implementation, we chose the latter approach and built a customized broker and client interfaces for MQTT and QUIC. We refer to these interfaces as agents, which enable IPC between the MQTT and QUIC. Figure 3 shows a high level view of the protocol stack architecture.

Fig. 3: The high-level architecture of the proposed implementation.

Since QUIC uses UDP, connection-oriented features such as reliability, congestion control, and forward error correction are implemented in QUIC. In addition, QUIC incorporates cyptographic shields, such as IP spoofing protection and packet reordering [50]. To keep the QUIC implementation lightweight and abstracted, we segregated the implementation into two parts: (i) the QUIC client and server only deal with UDP sockets and streams; (ii) the agents deal with reliability and security.

Figure 4 shows the high-level view of the server and client agent implementations. The implemented entities are as follows: (i) Server-Agent APIs and functions: They handle server specific roles such as accepting incoming UDP connections, setting clients’ state, and storing and forwarding packets to the subscribers based on topics. (ii) Client-Agent APIs and functions: They perform client-specific tasks such as opening a UDP connection, and constructing QUIC header and streams. (iii) Common APIs and functions: They are utilized by both the server and client. However, their tasks differ based on their roles.

After initializing the transmit and receive message queues by initialize_rx_tx_msg_queue(), the server and client agents process the messages differently. In the server-agent, the received message queue is fed to quic_input_message(). This API begins the handshake process to negotiate the session keys. However, if the client has contacted the server in the past, then quic_input_message() is directly available for the crypt_quic_message() to parse the incoming MQTT message. On the other hand, the client-agent first processes the MQTT message and then sends that message to the server by using start_connect(). Based on the communication history between the server and client, the start_connect() API either enters or skips the handshake process with the server. The common APIs handle the handshake control messages.

Fig. 4: Server-agent and client-agent are two entities between QUIC and MQTT. This figure represents the packet processing flow by the server and client agents.
Variable Description

Represents a rejected message

Protocol-associated security parameter used to derive the session keys

Input key
Deterministic algorithm used in AEAD
SHA-256 function
Signature used in digital signature scheme

Key pair representing public key and secret key

Derived by

Flag used to determine if packet is subject to transmission or processing

Derived by

Client (i.e., publisher or subscriber)

Client hello message in QUIC, a.k.a., inchoate hello (c_i_hello)
cid Connection identifier
Struct used in storing client info such as cid, socket, etc.

Client’s initial state, which is the state immediately after receiving REJ message from server

Deterministic algorithm used in AEAD

Diffie-Helman public values

Flag in MQTT (value 1 indicates the packet is a retransmission)

Key hash message authentication used in expansion of keys [51]
Initial key variable set in the initial phase of QUIC’s connection establishment

bit initializer, init

Initialization vector
Key to derive data in final_data() phase
Kg Key generation algorithm takes as security parameter and generates (,) as key pair in QUIC
Derived same as session key (for simplicity, in our implementation we treat as random string to replicate unpredictable input)
Message to be sent or received by client or server in QUIC
The message part of Msg

Message ID for MQTT queuing
Message space used in QUIC
(Msg consists of bitstrings starting with a 1, while key exchange messages to be encrypted start with a 0)

pub Public DH Values
pk Public key

Reject message (message from server after CHLO in QUIC)
retained Flag used in MQTT to prevent loosing subscribed topics when a connection loss happens
RTT Round-trip time


Variable (output by scfg_gen()) to represent the global state of server in QUIC

Server’s cid

Server Hello packet (s_hello) in QUIC

Secret key

Sequence number (used in signalling for every segment in QUIC)
Source address token to guard IP-spoofing in QUIC
Strike variable used by QUIC
TABLE I: Key Notations and Abbreviations

Iii-B Definitions, Methods, and Assumptions

This section presents the primitives used to present and explain the implementations. Table I shows the main acronyms and notations used in the rest of this paper. We chose most of these notations based on the RFCs published relevant to this work.

Notation represents the set of all finite-length binary strings. If and are two uniquely decodable strings, then () simply represents the concatenation of both. If , then represents the consecutive strings of 1 bits. The notation S represents the uniform random selection of from a finite set S. The set of integers [] is represented by [], where . We also assume that a public key infrastructure (PKI) is available. This means that public keys are user-identity bounded, valid, and publicly known. Therefore, certificates and their verification are excluded from the implementation.

A Digital Signature Scheme (SS) with message space is used by the broker during a connection establishment to authenticate certain data. The scheme is defined as follows,


where Kg, Sign and Ver are the randomization key generation algorithm, signing algorithm, and verification algorithm, respectively. The input of Kg is the security parameter and its output is a public and secret key pair, as follows,


The signing algorithm returns a signature,


where is the secret key and . The verification algorithm is denoted as follows,


where is the public key and . The output value , which is a bit, shows whether the signature is valid or invalid.

The requirement for correctness of SS is Ver(,, Sign(,)) = 1 for every Msg and Kg(). Correctness is defined by the requirement that the input of one party’s be equal to the output of the other party’s .

Secure channel implementation is based on an authenticated encryption with associative data schemes (AEAD) [52]. AEAD consists of two algorithms: and D. First, is a deterministic encryption algorithm defined as follows,


where is ciphertext, , message , is an additional authenticated data, and key is defined as


Second, D is a deterministic decryption algorithm defined as,


where is plaintext.

The correctness requirement of AEAD is ,nonc,,(,nonc,,     for all , nonc , and .

1 function main()
2         /* Initialize tx and rx queues */
3         initialize_rx_tx_msg_queue();
4         ;
5         while true do
6                 if event = process_packet then
7                         if packet must be transmitted then
10                         else
11                                 recvfrom();
13                                 process_rx_packets();
16                 else if event = client_disconnect socket_timeout then
17                         disconnect ;
18                         break;
22 function process_rx_packets()
23         while rx_message_queue do
24                 quic_input_message();
26        return ;
28 function quic_input_message()
29         if !(handshake_completed) then
30                 if !(do_handshake()) then
31                         = ;
32                         return ;
34                 else
35                         =
38         else
39                 = ;
40                 do_handshake_once();
42        return ;
Algorithm 1 Server-agent APIs and functions
1 function main()
2         initialize_tx_rx_msg_queue();
3         ;
4         while true do
5                 if event = process_packet then
6                         if packet must be transmitted then
7                                 create_udp_socket();
8                                 ;
9                                 mqtt_message_initializer();
11                         else
12                                 insert
13                                 process rx_msg_queue for received packets;
16                 else if event = client_disconnect socket_timeout then
17                         disconnect ;
22 function mqtt_message_initializer()
23         if (!MQTTClient_create()) then
24                 return ;
26         if (!MQTTClient_connectOptions_initializer()) then
27                 return ;
29         if (!mqtt_client_connect()) then
30                 return ;
32         return ;
34 function mqtt_client_connect()
35         if  then
36                 return error;
38         initialize_quic();
39         return ;
41 function initialize_quic()
42         /* filling quic header */
43         if ! then
44                 = get_cid(); /*Assign cid to client */
46         populate_quic_header();
47         start_connect();
48         return ;
50 function start_connect()
51         quic_stream_gen();
52         if ! then
53                 do_handshake();
55         else
56                 do_handshake_once();
58        return ;
60 function quic_stream_gen()
61         /* Check stream presence */
62         if  then
63                 if  then
64                         clear_entry();
65                         return ;
67                 else
68                         /* Find stream */
69                         find_stream();
72         else
73                 /* Create a stream and associate it with the cid*/
75                 create_stream_buf();
77        return;
Algorithm 2 Client-agent APIs and functions
1 function do_handshake()
2         if !(protocol_connect) then
3                 return ;
6         if !(do_handshake_once()) then
7                 return ;
9         return ;
11 function do_handshake_once()
12         if  =  then
13                 = ;
15         else
16                 = ;
18        if !(crypt_quic_message()) then
19                 return ;
21         return ;
23 function crypt_quic_message()
24         if flag = decrypt then
25                 = decrypt_message
26                 quic_dispatcher();
28         else if flag = encrypt then
29                 = encrypt_message
30                 insert
32         return ;
34 function protocol_connect()
35         if session files not found then
36                 connect(); /* 1-RTT Implementation */
38         else
39                 resume(); /* 0-RTT Implementation */
41        return ;
43 function quic_dispatcher()
44         if  then
45                 /* If the client is calling the API */
46                 process rx_msg_queue for mqtt application;
47                 return ;
49         else
50                 /* MQTT parsing for received messages */
51                 if (valid_mqtt_header()) then
52                         mqtt_parse_args
53                         return ;
55                 else
56                         return
59        return ;
Algorithm 3 Common APIs and functions
1 function encrypt_message()
2         if !() then
3                 /* Initial session key is used */
4                 get_iv(H, )
5                 if  was used then
6                         return ;
8                 else
9                         /* For client */
10                         if is_client then
11                                 return (H, H, );
13                         /* For server */
14                         else
15                                 return (H, H, );
19         else
20                 /* The stored established key is used */
21                 get(H,
22                 if  is used then
23                         return
25                 /* For client */
26                 if is_client then
27                         return (H, H, );
29                 /* For server */
30                 else
31                         return (H, H, );
35 function decrypt_message()
36         /* Extracting ciphertext*/
37         ;
38         if !() then
39                 /* initial data phase key is used */
40                 get_iv(H, )
41                 /* For client */
42                 if is_client then
43                         if D(, , H, ci)  then
44                                 return ;
46                         else
47                                 return
50                 else
51                         /* For Server */
52                         if D(, , H, ci)  then
53                                 return ;
57         else
58                 /* Final data phase key is used */
59                 get_iv(H, )
60                 /* For client */
61                 if is_client then
62                         if D(, , H, ci)  then
63                                 return ;
65                         else
66                                 return
69                 else
70                         /* For server */
71                         if D(, , H, ci)  then
72                                 return ;
Algorithm 3 Common APIs and functions (continued)

Iii-C Server-agent

Algorithm 1 shows the implementation of these APIs and functions. Our server-agent implementation is event-based. When an event occurs, the server-agent determines whether to transmit the packet or insert it into the receiving queue. Server-agent APIs and functions are described as follows:

Iii-C1 main()

If the packet is subject to transmission, then action_flag is set, otherwise the packet will be processed by process_rx_packets(). This function also handles the disconnect events. If the server receives a disconnect event, then particular clients will be disconnected. However, the server might receive a disconnect event for all the clients. This happens, for example, when a reboot event occurs.

Iii-C2 process_rx_packets()

It processes the packets in the receive queue by quic_input_message().

Iii-C3 quic_input_message()

This API first checks whether the handshake process with the client has been completed or not. In order to determine this, it checks the flag. If it has been set, then it assumes that the connection is alive and handshake has been completed. In this case, the API skips the handshake process and enters the do_handshake_once() API to set the flag for encryption or decryption. If the flag is not set, then the API runs the handshake process by executing do_handshake(), where the client and server either follow the 1-RTT or 0-RTT implementation. This API also sets the client’s state according to its operation. For instance, if the client is contacting the server for the first time, then the client’s state is set to client_initial. If the handshake fails, then the client’s state is set to client_handshake_failed. If the handshake has to be skipped, then it sets the client’s state to client_post_handshake.

Iii-D Client-agent

This section deals with the APIs and functions specifically used by the client-agent. Algorithm 2 shows the client agent implementation.

Iii-D1 main()

Similar to the server-agent, the main() function initializes both the transmit and receive queues by calling initialize_rx_tx_queue(). The client-agent is also event-driven and processes both incoming and outgoing packets based on the packet_process event. The main() function checks to determine if the packet is meant to be sent or processed by process_rx_packets().

Iii-D2 mqtt_message_initializer()

This API creates an MQTT client instance via MQTTClient_create() and sets the client ID, the persistence parameter (retained), and the server IP address. When a subscriber connects to a broker, it creates subscriptions for all the topics it is interested in. When a reboot or reconnect event occurs, the client needs to subscribe again. This is perfectly normal if the client has not created any persistent sessions. The persistence parameter creates a persistent session if the client intends to regain all the subscribed topics after a reconnection or reboot event.

Immediately after the creation of a client instance, a message is initialized by MQTTClient_connectOptions_initializer(). This API sets the msgid, dup, retained, payload and version fields of this packet. MQTTClient_connectOptions_initializer() creates a directory to store all the topics and includes this in to be easily retrieved later.

Iii-D3 mqtt_client_connect()

At this point, all the message construction functionalities related to MQTT are completed, and finally the client instance is ready for QUIC related operations. Sanity checking (e.g., header size, data length, etc.) is performed before entering the main QUIC API (i.e., initialize_quic()).

1 function connect()
2         if initial_key_exchange() then
3                 if initial_data() then
4                         if key_settlement() then
5                                 if final_data() then
6                                         return ;
10         return ;
11 function initial_key_exchange()
12         ;
13         /* For client */
14         if is_client then
15                 switch phase do
16                         case initial do
17                                 /* is public key */
18                                 c_i_hello();
19                                 break;
21                        case received_reject do
22                                 /* received packet from server (REJ) */
23                                 c_hello()
24                                 break;
26                        case complete_connection_establishment do
27                                 /* is initial key variable set during initial phase */
28                                 get_i_key_c()
29                                 break;
33         /* For server */
34         else
35                 switch phase do
36                         case received_chlo do
37                                 s_reject()
38                                 break;
40                        case received_complete_chlo do
41                                 get_i_key_s()
42                                 break;
46        return ;
48 function initial_data()
49         /* For client */
50         if is_client then
51                 for each ;
52                 + 2 sqn;
53                 /* sqn = Client sequence number */
54                 /* = Client constructed message */
55                 pak(, sqn, )
56                 process_packets();
58         /* For server */
59         else
60                 for each ;
61                 + 1 sqn
62                 /* sqn = Server sequence number*/
63                 /* = Server constructed message */
64                 pak(, sqn)
65                 process_packets();
67        return ;
Algorithm 4 1-RTT Implementation
1 function key_settlement()
2         if is_client then
3                 get_key_c(, sqn)
5         else
6                 2 + sqn;
7                 s_hello(, sqn)
8                 get_key_s(
10        return ;
12 function final_data()
13         if is_client then
14                 for each { + 1,…,}
15                 + 2 sqn;
16                 pak(, sqn, )
18                 process_packets;
20         else
21                 for each { + 1,…,}
22                 + 2 sqn
23                 pak(, sqn, )
25                 process_packets(, );
27        return ;
Algorithm 4 1-RTT Implementation (continued)

Iii-D4 initialize_quic()

This API first determines if the client has an assigned cid, and if not, then a new cid is generated by . The QUIC header is initialized after cid assignment. The non-encrypted part of the QUIC header consists of cid, diversification nonce, packet number or sequence number, pointer to payload, size of the payload, size of the total packet, QUIC version, flags and encryption level. The encrypted part of the QUIC header is made of frames, where each frame has a stream ID and a final offset of a stream. The final offset is calculated as a number of octets in a transmitted stream. Generally, it is the offset to the end of the data, marked as the FIN flag and carried in a stream frame. However, for the reset stream, it is carried in a RST_STREAM frame [37]. Function populate_quic_header() fills both the encrypted and non-encrypted parts of the QUIC header and enters the API start_connect() to connect the client.

Iii-D5 start_connect()

This API is the entering point to the common APIs (described in Section III-E). Its first task is to find an appropriate stream for the connection. Based on the handshake_completed flag, start_connect() API determines if the client and server have completed the handshake. If the handshake has been completed, then it means that QUIC connection is alive and data can be encrypted or decrypted. If not, then it enters the handshake process by executing the do_handshake() API.

Iii-D6 quic_stream_gen()

The only purpose of this API is to detect and create streams. As mentioned earlier, QUIC is capable of multiplexing several streams into one socket. This API first detects whether there is already an open stream for a particular client, and if not, then it creates a new stream by running create_stream_buf().

Iii-E Common APIs and functions

Several APIs and functions such as encryption, decryption and processing transmission packets are mandated on both the server and client sides. Although the packets handled by these functions are different in the client-agent and server-agent, the underlying mechanisms are almost similar. Algorithm 3 shows the implementation.

Iii-E1 do_handshake()

This API is the starting point of the QUIC connection establishment and key exchange process. It behaves as an abstraction of the handshake process. First, it calls protocol_connect() for protocol communication. If protocol_connect() is executed successfully, then the handshake_completed flag is set. Based on this flag, the client determines whether to execute the handshake process or skip it. Lastly, this API calls the do_handshake_once() API to set the encrypt or decrypt flag for further processing.

Iii-E2 do_handshake_once()

The primary responsibility of this API is to set the encryption or decryption flags. This decision is made based on the type of the next operation, which is transmission or processing. If action_flag is set, then the packet must be encrypted and sent. Finally, it enters the crypt_quic_message() API.

Iii-E3 crypt_quic_message()

This API is the starting point in establishing a secure connection. It checks the to determine if the packet should undergo decryption (decrypt_message()) or encryption (encrypt_message()).

Iii-E4 protocol_connect()

In this API, the client () checks the existence of a session file to determine whether it has communicated with the server () during the last seconds. If and are interacting for the first time, then the connect() API completes the 1-RTT scenario in four phases, as shown in Figure 5 and Algorithm 4. If and have communicated before, then the resume() API follows the 0-RTT scenario shown in Figure 6 and Algorithm 7.

1-RTT Connection. The 1-RTT implementation is divided into four phases. The first phase exchanges the initial keys to encrypt the handshake packets until the final key is set. The second phase starts exchanging encrypted initial data. The third phase sets the final key. Last, the fourth phase starts exchanging the final data. We explain the details of these phases as follows.

Fig. 5: The four phases of the 1-RTT connection.
1 function get_scfg()
4         pub
5         sec
6         expy;
7         (pub, expy) scid; /* is a SHA-256 function */
8         "QUIC Server Config Signature" str;
9         Sign(, (str, 0x00, scid, pub, expy))prof;
10         (scid, pub, expy) scfg
11         return scfg;
Algorithm 5 Server Configuration State

[1-RTT]: Phase 1. This phase is handled by initial_key_exchange() and consists of three messages, and . The client runs c_i_hello(), which returns a packet with sequence number 1. is an initial connection packet sent to , containing a randomly generated cid. In response to , sends a REJ packet , generated using s_reject(). contains a source-address token (similar to TLS session tickets [53]), which is later used by to prove its identity to for the ongoing session and future sessions. This is performed by checking if the source IP address equals the IP address in . Fundamentally, the consists of an encryption block of ’s IP address and a timestamp. In order to generate , uses the same deterministic algorithm (i.e., AEAD) with (derived by ). The initialization vector for (i.e., iv) is selected randomly and is used in s_reject. For simplicity, we implemented a validity range for , which is bounded by the time period during which it was either generated or set up. Another important parameter in is the s current state scfg_pub (refer to Algorithm 5). It contains ’s DH values with an expiration date and a signature prof. This signature is signed by SS over all the public values under the s secret key .

Upon receiving , the client checks scfg for its authenticity and expiration. The algorithms for this purpose can be found in [52]. As we mentioned in Section III-B, our implementation assumes that a PKI is in place. After possessing the public key of , the client generates a nonc and DH values by running c_hello(), and sends them to the server in message . At this point, both and derive the initial key material , using get_i_key_c() and get_i_key_s(), respectively. The server keeps track of the used nonc values in order to make sure that it does not process the same connection twice. This mechanism is referred as strike-register or strike. The timestamp is included in the nonc by the client. The server only maintains the state of a connection for a limited duration of time. Any connection request from a client is rejected by the server if its nonc is already included in its strike or contains a timestamp that is outside the permitted range strike. The initial key = (, , iv) is made of two parts: two 128-bit application keys () and two 4-byte initialization vector prefixes iv = (iv, iv). The client uses and iv to encrypt the data and send it to . On the other hand and iv assist in decryption and encryption. This phase happens only once during the time period until scfg and are not expired.

1 function c_i_hello(pk)
2         cid;
3         return {IP, IP, port, port, cid, 1};
4 function s_reject(m)
5         iv;
6         (iv, , iv, , 0) (IP, current_time)) stk;
7         return {IP,IP,port,port,cid,1,scfg, prof, stk};
8         /* prof is generated by get_scfg */
9 function c_hello(m)
10         if expy   then
11                 return ;
13         ;
14         if Verstr,scid, pub, expy), prof))  then
15                 return ;
17         ;
20         pkt_info;
21         return (pkt_info, cid, 2, stk, scid, nonc, pub);
22 function get_i_key_c(m)
23         ipm;
24         return extract_expand(ipm, nonc, cid, , 40, 1)
26 function get_i_key_s(m)
27         stk(iv, tk);
28         d;
29         if (d =  then
30                 if ) then
31                         return;
32                 return;
33         if last 4 bytes corresponds to outside strike then
34                 return ;
36         if   then
37                 return ;
39         if scid is unknown then
40                 return ;
42         if scid corresponds to expired scfg then
43                 /* where  <  */
44                 return ;
46         if  then
47                 return ;
49         ipm;
50         return extract_expand(ipm,nonc,cid,m,40,1);
52 function extract_expand(ipm, nonc, cid, m, l, init)
53         HMAC(nonc,ipm) ms;
54         if init = 1 then
57         else
60         cid, , scfg info;
61         return first bytes (octets) of T = (T(1), T(2),...),
62         if all , T() = HMAC(ms, (T(-1), info, 0x0)) and T(0) =
Algorithm 6 QUIC messages exchange APIs and functions

[1-RTT]: Phase 2. This phase is handled by initial_data() and consists of two messages, and . The client and server exchange the initial data message and , which are encrypted and authenticated with in function pak(,sqn, ) for every and pak(, sqn, ) for every , respectively. Here, sqn and sqn represent the sequence number of packets sent by and , respectively. and represent the maximum number of message blocks that and can exchange prior to the first phase. The initialization vector iv is generated based on the server or client role. When sends a packet, get_iv() outputs iv by concatenating iv and sqn. When sends a packet, get_iv() generates iv by concatenating iv and sqn. The total length of each iv is 12 bytes since both the server and client initialization vector prefixes (i.e., iv and iv) are 4 bytes in length and sequence numbers (i.e., sqn and sqn) are 8 bytes in length. When receives packets from , it uses the process_packets() function to decrypt those packets to extract their payloads and concatenates them based on their sequence number. The server performs a similar mechanism for the packets received from .

1 function get_iv()
2         /* Client key,
3         * Server key
4         * Client initialization vector,
5         * Server initialization vector */
7         if is_client then
10         else
11                 ,
13        /* sqn is packet sequence number */
14         (cid, sqn);
15         return (, sqn);
16 function pak(, sqn, )
17         ,iv, iv);
18         if is_client then
19                 and ;
21         else
22                 and ;
24        IP, IP, port, port pkt_info;
25         (cid, sqn)
26         iv;
27         return (pkt_info, (,iv,, (1 )));
28 function process_packets()
30         if is_client then
31                 and
33         else
34                 and
36        for each
40             if  Msg then
41                 return  
42         return (, , …. );
43 function s_hello(, , sqn)
46         (cid, sqn)
47         ;
48         return (ip,ip,port,port, , );
50 function get_key_s()
51         pms;
52         return extract_expand(pms, nonc, cid, , 40, 0);
53 function get_key_c()
54         (IP,IP,port, port, cid, sqn, );
55         if D(,(iv, sqn),(cid,sqn),) =  then
56                 return 
57         if first  bit  of  the  message 0 then
58                 return 
59         pms;
60         return extract_expand(pms, nonc, cid, , 40, 0);
Algorithm 6 QUIC messages exchange APIs and functions (continued)
1 function resume()
2         if (pub) then
3                 if stk then
4                         if scid then
5                                 return c_hello(stk, scfg);
8         /* Jumping back on 1-RTT */
9         return connect();
10 function c_hello(stk, scfg)
11         cid;
12         , (current_time, ) nonc;
13         , , pub;
14         (IP, IP,port,port)pkt_info;
15         return (pkt_info, cid, 1, stk, scid, nonc, pub)
Algorithm 7 0-RTT Implementation

[1-RTT]: Phase 3. This phase is handled by key_settlement() and involves message . The server produces new DH values (authenticated and encrypted via AEAD with ) and transmits them to the client using s_hello(,, sqn). The client verifies the server’s new DH public values with the help of . At this point, the server and client both derive the session key by get_key_s() and get_key_c() and use extract_expand() for key expansion, as defined in Algorithm 6.

Fig. 6: In 0-RTT scenario, client sends the first packet c_hello with previous server global state scfg and strike. This step itself sends initial data to server.

[1-RTT]: Phase 4. This phase is handled by final_data() and consists of two messages, and . Instead of initial key , the established key is used to encrypt and authenticate the remaining data (Msg) by both and . Similar to , is derived from , , and iv, and consists of two parts: the two 128-bit application keys (, ) and the two 4-bytes initialization vector prefixes iv = (iv,iv