A Appendix

# Enhancing the Security of Protocols against Actor Key Compromise Problems

## Abstract

Security of complex systems is an important issue in software engineering. For complex computer systems involving many actors, security protocols are often used for the communication of sensitive data. Actor key compromise (AKC) denotes a situation where the long-term secret key of an actor may be known to an adversary for some reasons. Many protocols are not secure enough for ensuring security in such a situation. In this paper, we further study this problem by looking at potential types of attacks, defining their formal properties and providing solutions to enhance the level of security. As case studies, we analyze the vulnerabilities (with respect to potential AKC attacks) of practical protocols, including PKMv2RSA and Kerberos, and provide solutions to enhance the level of security of such protocols.

## 1 Introduction

Security of complex systems is an important issue in software engineering. For complex computer systems involving many actors, security protocols are often used for the communication of sensitive data. However, security protocols are not always secure enough, because of reasons including that there may be weakness in the methods for generation of secrete keys, storage of keys and so on. If an actor’s key is revealed and used by an adversary to impersonate another party communicating with the actor, then there is a key compromise impersonation (KCI) attack [1], and then the attacker may obtain sensitive data through such an impersonation. Actor key compromise (AKC) attack is a generalization of this kind of attacks. This has been studied in [2], where this property is formalized and conditions under which it can and cannot be achieved are identified.

Previous works focus on KCI attacks in the domain of key establishment protocols. In [3] and [4] some concrete two-party protocols have been studied and countermeasures to prevent such attacks provided. The type of KCI attacks is classified in [5] and [6] based on whether the responder authenticates the initiator, and use digital signatures and time-stamps as a help. [7] is the first to study security attribute of group key exchange protocols under KCI attack. The first computation model of KCI is provided in [9]. Then [2] provides a systematic analysis of the consequences of compromising the actor’s secret key and countermeasures, and shows both constructive and impossibility results.

There are additional issues that need to be investigated. Firstly, the classification of KCI attacks based on adversary’s capability of eavesdropping and sending messages is generic and may not reveal the particular feature of such attacks. Furthermore, providing definitions of attack types may make it easier to analyze the vulnerabilities and then modify the protocol for enhancing security. Second, the work in [2] focuses on the problem where a given actor may have the secret key being compromised, and we focus on solutions for enhancing the security in case one of the actors (however, which one is unknown) has the secret key being compromised, and we also consider multi-party protocols and a different type of security claims. Third, no practical algorithms have been provided in transforming a protocol into an AKC resilience one, which is also important for the practical use of the methods.

The purpose of this work is to provide practical solutions for transforming protocols to achieve higher security levels against AKC attacks. The work includes classifying types of AKC attacks and providing their formal definitions, furnishing solutions, and providing practical algorithms.

The rest of this paper is organized as follows. Section 2 introduces the modeling framework and gives formalization of security properties. In Section 3, we classify four types of AKC attacks and give formal definition of the attacks. In Section 4, we propose solutions to prevent such attacks. We present case studies in section 5 and concluding remarks in Section 6.

The proofs of the propositions and corollaries, and the algorithms for the transformation of protocols are to be found in the appendix.

## 2 Preliminaries

We follow the formal framework for protocol specification and the execution model defined in [10][11].

### 2.1 Protocol Specification

A partial function from to is denoted . The domain and range of are denoted and , respectively. denotes a function such that , and otherwise it coincides with . We write to denote the sequence of elements from to .

Let A, R, Fresh, Var, Func, and TID denote sets of agents, roles, Fresh and so on. TID contains two distinguished thread identifiers, Test and which stands for a thread of an arbitrary agent and that of an adversary thread.

binds the local term to the protocol thread identified by . By we denote X’s asymmetric long-term public key, and denotes the corresponding secret key. The superscript in denotes the arity of parameter. is a special case of with arity . The use of symmetric cryptography and hashing is not sufficient to ensure AKC resilience [2]. For brevity, we do not consider symmetric cryptography in this paper and therefore omit symmetric cryptographic terms in the definition of the basic elements of protocols.

Definition 1 (Terms):

 Term::= A|R|Fresh|Var |Fresh♯TID|Var♯TID|Func(Termn) |(Term,Term)|{Term}Term |sk(A)|pk(A)|sk(R)|pk(R)

We define as the set of terms that have no subterms in , and as the set of terms that have no subterms in . A role term is transformed into a run term by applying an instantiation from the set :

 TID↪((R↪A)∪((Fresh∪Var)↪RunTerm)).

We define a binary relation on terms, where denotes that the term can be inferred from the set of terms M. Let denote the inverse function on terms such that for all agents , and , and for all other terms, . Let and let . The relation is the smallest relation satisfying:

 t∈M⇒M⊢t M⊢t1∧M⊢t2⇔M∈(t1,t2) M⊢{t1}t2∧M⊢(t2)−1⇒M⊢t1 M⊢t1∧M⊢t2⇒M⊢{t1}t2 ⋀0≤i≤nM⊢ti⇒M⊢f(t0,...,tn)

The subterm relation is defined as the reflexive, transitive closure of the smallest relation satisfying the following, for all terms and function names :

 t1⊑(t1,t2),t2⊑(t1,t2) t1⊑{t1}t2,t2⊑{t1}t2 t1⊑pk(t1),t1⊑sk(t1) ti⊑f(t1,...,tn)for1≤i≤n

The accessible subterm relation identifies potentially retrievable subterms, is defined as a subset of subterm relation such that and . In order to identifies position of and , we define another subterm relation such that .

Definition 2 (Event): Let be a given set of claims including the following claims , , , . Let be a set of labels. The set of events is defined as follows.

 RoleEvent:: =sendLabel(R,R,RoleTerm) |revcLabel(R,R,RoleTerm) |claimLabel(R,Claim[,R][,RoleTerm]) RunEvent:: =create(R,A) |sendLabel(A,A,RunTerm) |revcLabel(A,A,RunTerm) |claimLabel(A,Claim[,A][,RunTerm]) AdvEvent:: =LKR(A) Event:: =RoleEvent|RunEvent|AdvEvent

describes how agents start threads, send and receive messages. is an event where the adversary compromises ’s long term secret key. The is executed in the single adversary thread .

As an example, the event

 sendl(Alice,Bob,{n#tid}sk(Alice))

denotes that sends a nonce in the run and encrypted with its secret key.

An event has an event-type and a label which are denoted and , and the contents of a send-event is denoted .

In order to simplify the typing constraint, in the following, stand for events, stand for sequence of events, stand for roles, stand for agents, stand for labels, stand for role terms and run terms (should be clear from the context), stand for run terms that are used in a message, for TID. Let be a set. A sequence of elements of is denoted . An element in a sequence is denoted . The operation denotes the concatenation of two sequences. The powerset of is denoted .

A sequence of RoleEvent is well-formed, if all variables initialized in an accessible position in a event are not used before that event. Let denote the set of variables appearing in .

 wellformed(ρ) ⇔ ∀ρ′,l,a,b,t,ρ′′,v: ρ=ρ′⋅⟨recvl(a,b,t)⟩⋅ρ′′ ⇒(v⊑acct⇒v∉vars(ρ′).

A protocol is a partial function from to together with a function that formalizes which terms may be stored in a given variable. For each role, the sequence of events must be wellformed.

Definition 3 (Protocol): Let and . If for all , is wellformed, then is a protocol.

For convenience, we extend the domain of to RunTerm such that for a run term is the set of run terms such that variables in is substituted according to the initial .

In a protocol, a label is associated with a send-role and a receive-role, denoted respectively and , defined by , if and, or for some ; , if and, or .

### 2.2 Execution Model

Protocol execution is modeled as a labeled transition system . A state consists of a trace , the adversary’s knowledge , a partial function and a role and variable instantiation . We denote as , and as which is the -th event of the trace. The initial state is where is the initial adversary knowledge.

The operational semantics of a protocol is defined by a transition system which are composed of execution rules from Fig 1 with a selected subset of adversary rules in Fig 2. The rule starts a new thread of a protocol role . The rule sends a message to the network and add it to adversary knowledge. The rule accepts message if it match the pattern . The rule states a security property that is expected to hold. The rule allows the adversary to learn the long-term keys of the agent executing the test run.

Let the protocol with an initial role , and a set of adversary rules be given. If there is a rule such that , then we write . The set of reachable states denoted is . The set of all possible traces of the protocol is denoted .

In a state , we have a trace and each thread in the trace is created by a role. The special thread is created by . Let be a function that identifies a with a in . Then and , if .

### 2.3 Security Property

Security properties are modeled as reachability properties. A claim on a role term is of the form for some label and role .

Definition 4 (secrecy claim): Let be a state. If is a secrecy claim on , and , then

 s⊨γ⇔AKs⊬σs,Test(t#Test)

The following two properties are related to data agreement.

The property means that the initiator agree on some data with the responder. The property means whenever initiator completes a run of the protocol with responder , then has previously been running the protocol with , and the two agents agreed on all the variables. A commit claim on a role term is of the form for some label and roles and . A corresponding claim for such a claim is of the form .

Definition 5 (commit claim): Let be a state. If is a commit claim, and , then , iff

• there is a such that , and

• there is a running claim such that , and there exists a send-event , such that , .

Let denote the total order of events in a sequence (for the sequence of events , we have , , and ). The order on events which is induced by the communications is defined as . The transitive closure of the union of the role event order and the communication relation is called the protocol order . is the set of causally preceding communications of a claim event labeled with : .

Let denote the function that maps roles to runs according to of the state . Let denote iff for some .

A claim is of the form for some and for stating the correspondence between send-messages and recv-messages.

Definition 6 (nisynch claim): Let be a state. If is a nisynch claim, and for some , then

 s⊨γ⇔ ∀l′∈prec(l),a,b,m, ∀j

A protocol is AKC secure if its security claim holds under AKC attacks. This property has been formalised in [2]. Here we use to denote that for all , .

Definition 7 (Actor key compromise security, AKCS): Let be a protocol, , an adversary (represented by a set of adversary rules) such that , and a security claim. is (AKCS) if .

For the correctness of security properties, we assume that no asymmetric long-term secret keys appear in accessible positions in any messages of a protocol, in the subsequent sections.

## 3 Attack Types

Understanding adversary’s techniques to launch attacks and their attack objectives is helpful in identifying weakness of protocols. Some work has been done on categorizing attacks with traditional Dolev-Yao adversary model. In [12], there is a classification of known-key attacks, where they study AK protocols and categorize attacks based on adversary’s capability of modifying messages. In [13] one-pass two-party key establishment protocols under KCI attacks are studied, two classes of KCI attacks are described. Here we study types of attacks under stronger adversary models. Furthermore, we provide the formal definition of such attacks based on the trace model and techniques for fixing such protocols are provided in the next section.

Secrecy Attack One purpose of a protocol is to transmit a secret nonce from an initiator to a responder. In order to keep the nonce secret, The initiator will encrypt the nonce with the responder’s public key, which is not safe if intruders knows the responder’s secret keys.

Definition 8 (Secrecy attack):

Let be a protocol, , . If , , then there is secrecy attack on t, which we denote .

Example Suppose that the initiator wants to transmit a secret nonce to the responder before setting up a session key. In order to keep the nonce secret, the initiator will encrypt the nonce with the responder’s public key, which is not safe if intruders knows the responder’s secret keys. Consider the CCIT-ban1[19] protocol as follows.

 I→R: I,{Ta,Na,R,Xa,{Ya,{hash(Ya)}sk(I)}pk(R)}sk(I)

Clearly, there is secrecy attack on Ya, if the secret key of the responder is known to the intruder.

Substitution Attack An attack of this type occurs in a situation when an initiator and a responder try to use fresh values or secret keys to authenticate each other. The main characteristics of this type of attacks is that the adversary replaces terms in a message with another terms without being discovered.

Let denote that the thread instantiated by the agent is the corresponding thread communicating with instantiated by according to of the state . In other words, iff there is such that and for .

Let denote derived from by replacing in with . Let be a subset of labels, and be sets of terms, and be an access relation. The predicate is defined as follows.

 Replace(s,L,S,S′,≺,tid)⇔ ∃l∈L,m,m′,a,b′,tid′, x∈S,y∈S′,y≠x,x≺m′: tid′∈tidinsts(sr(l))∧tid∈tidinsts(rl(l))∧ Matchs(a,tid′,b,tid)∧ ∃k.(ev(trs,k)=recvl(a,b,m)#tid)∧ ∀j

In a substitution attack, the adversary eavesdrop the message and modify some of its fresh values by its own fresh values and transmit it to the receiver of the message.

Let denote the thread has been completed in , i.e., every event in the sequence has a corresponding event in .

Let and , where denote the subset of used by the adversary.

Definition 9 (Substitution Attack): For a security protocol , there is a substitution attack, if and a such that and hold, which we denote .

Example Consider the Bilateral Key Exchange (BKE) protocol as an example, which is supposed to guarantee the secrecy of and agreement on and .

 1.I→R:{ni,I}pk(R) 2.R→I:{hash(ni),nr,R,kir}pk(I) 3.I→R:{hash(nr)}kir

The protocol is vulnerable to substitution attacks. If the intruder (denoted ) knows the secret key of Bob (an agent of the role ), he can decrypt message 2 using the secret key, and constructing another message 2’ using its own nonces. In this way, the adversary impersonate Bob to Alice (an agent of ) and break agreement of and between them:

 1.Alice→Bob:{ni,Bob}pk(Bob) 2.Bob→DAlice:{hash(ni),nr,Alice,kir}pk(Alice) 3.DAlice decrypts\; message\; using sk(Alice) and \;learnshash(ni) 4.DAlice choosesnr′,kir′ and\; constructs{hash(ni),nr′,Alice,kir′}pk(Alice) 5.DAlice→Alice:{hash(ni),nr′,Alice,kir′}pk(Alice) 6.Alice→DAlice:{hash(nr′)}kir′

Role-mixup Attack An attack of this type has the result that the participating entities do not agree on who is playing what role in the protocol. We use to denote that there exists some label which contains role name in accessible position and there is no matching send-events for a recv-event in the trace.

 Termin(s,L,tid)⇔ ∃l∈L,a,b,m,n,tid′: tid′∈tidinsts(sr(l))∧tid∈tidinsts(rl(l))∧ Matchs(a,tid′,b,tid)∧ ∃k.(ev(trs,k)=recvl(a,b,m)#tid′)∧ ∀j

The role-mixup attack states that the messages which has agent names in accessible position have been replaced by the adversary, or the public(secret) key of some agent may be replace by other agent’s public(secret) key, or the adversary forged a message with agent names in accessible position to impersonate another party.

Definition 10 (Role-mixup attack): Let be a protocol, be the subset of such that agent names are accessible in the corresponding events, i.e. , . The role-mixup attack of , denoted , is defined as follows.

 RoleMixupAttack(Π,typeΠ)⇔ ∃s∈RS(Π,typeΠ,R,A),tid: Finish(s,tid)∧ (Replace(s,L,A,A,⊑acc,tid)∨Termin(s,L,tid)∨ Replace(s,Label,S,S,⊑ace,tid))

Example Consider the isoiec-9798-3-5 [20] protocol as an example:

 1.A→B:Cert(A),RA,Text1 2.B→A:Cert(B),RB,Text2 3.B→A:RB,RA,A,Text6,{RB,TA,A,Text5}sk(B) 4.A→B:RA,RB,B,Text4,{RA,RB,B,Text3}sk(A)

The protocol is vulnerable to role-mixup attacks. In this protocol Bob and Alice want to agree on fresh values , , and . The attack is shown in Fig 3, in which the adversary listens to the message between them and impersonate Alice and Bob, such that Alice assumes Bob as B and Bob assumes Alice as B, however both Alice and Bob are acting as A.

Parallel Attack In the environment that the same protocol has run as several threads, the authentication may not be preserved because A may communicate with B in the first thread, and with C which has run the same protocol later, but A still assumes he is communicating with B.

Definition 11 (Parallel Attack): Let be a protocol. The parallel attack of , denoted , is defined as follows.

 ParallelAttack(Π,typeΠ)⇔ ∃s∈RS(Π,typeΠ,R,A),l,a,b,m, ∃k,tid∈TID.(ev(trs,k)=recvl(a,b,m)#tid)∧ ∀j

Example Consider the following protocol, in which the two agents authenticate each other using three nonces.

 1.A→B:{na}sk(A) 2.B→A:{h(na,nb),nb}sk(B) 3.A→B:{h(nb,nc),nc}sk(A) 4.B→A:{h(nc)}sk(B)

The protocol is vulnerable to parallel attack when Alice has two runs of the protocol. The adversary can forge the message in the second run, which makes Bob initiate a session with Alice in run 1 but receive the last authentication message in run 2. We show the attack in Fig 4.

## 4 Preventing Attacks

In this section, we give constructive methods for avoiding potential AKC attacks. In [2], transformations to achieve unilateral security is provided. Our work tries to provide transformations that achieve bilateral secrecy and agreement, and instead of using secret keys to achieve agreement, we use hash function and public keys to achieve agreement. The argument here is that the content encrypted by public keys will not be compromised easily, and we can use hash function to commit values to be used as short term keys. Another particular point of our work is to use a special tag including role names to prevent role-mixup attack. Furthermore, we modify the -party NSL protocol in order to achieve the higher agreement property , which illustrates the practicability of the approach.

### 4.1 Resilience of Secrecy Attack

In [2], a tagging function for the transformation is provided. We recall that the function and the restricted one defined as follows .

Definition 12 (Tagging function) Let , , then for all :

 τc(t)=⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩t,if% tatomic\,or\,long-term\,key,(τc(t1),τc(t2)),ift=(t1,t2),{τc(t1),c}τc(t2),ift={t1}t2,f(τc(t1),...,τc(tn)),c),ift=f(t1,...,tn).

denotes the modification of which restricts the domain of to some set S of terms to avoid tagging unnecessary terms.

The transformation in Fig 5 shows how to ensure AKCS of secrecy. Three messages are added: the first one is a constant asking for a nonce, the second one contains an encrypted nonce, and the third one contains the secrecy encrypted by the nonce and the public key together. The last two works like encrypting secrecy with two pair of keys, which the adversary at most compromise either pair of them, thus achieving AKCS of secrecy for both sides. Here we add different constant tags on message to ensure the secrecy.

Let , , , and

 S1=⟨sendl1(R,R′,Req