Duty to Warn in Strategic Games
Abstract
The paper investigates the secondorder blameworthiness or duty to warn modality “one coalition knew how another coalition could have prevented an outcome”. The main technical result is a sound and complete logical system that describes the interplay between the distributed knowledge and the duty to warn modalities.
Key Words.:
epistemic logic; blameworthiness; responsibility completenessprintacmref=false \setcopyrightifaamas \copyrightyear2020 \acmYear2020 \acmDOI \acmPrice \acmISBN \acmConference[AAMAS’20]Proc. of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2020)May 9–13, 2020Auckland, New ZealandB. An, N. YorkeSmith, A. El Fallah Seghrouchni, G. Sukthankar (eds.) \orcid000000031687045X \affiliation\institutionTulane University \cityNew Orleans \stateLouisiana \affiliation\institutionLafayette College \cityEaston \statePennsylvania
1 Introduction
On October 27, 1969, Prosenjit Poddar, an Indian graduate student from the University of California, Berkeley, came to the parents’ house of Tatiana Tarasoff, an undergraduate student who recently immigrated from Russia. After a brief conversation, he pulled out a gun and unloaded it into her torso, then stabbed her eight times with a 13inch butcher knife, walked into the house and called the police. Tarasoff was pronounced dead on arrival at the hospital Aron (2017).
In this paper we study the notion of blameworthiness. This notion is usually defined through the principle of alternative possibilities: an agent (or a coalition of agents) is blamable for if is true and the agent had a strategy to prevent it Frankfurt (1969); Widerker (2017). This definition is also referred to as the counterfactual definition of blameworthiness Cushman (2015). In our case, Poddar is blamable for the death of Tatiana because he could have taken actions (to refrain from shooting and stabbing her) that would have prevented her death. He was found guilty of seconddegree murder and sentenced to five years Aron (2017). The principle of alternative possibilities, sometimes referred to as “counterfactual possibility” Cushman (2015), is also used to define causality Lewis (2013); Halpern (2016); Batusov and Soutchanski (2018). A sound and complete axiomatization of modality “statement is true and coalition had a strategy to prevent ” is proposed in Naumov and Tao (2019a). In related works, Xu Xu (1998) and Broersen, Herzig, and Troquard Broersen et al. (2009) axiomatized modality “took actions that unavoidably resulted in ” in the cases of single agents and coalitions respectively.
According to the principle of alternative possibilities, Poddar is not the only one who is blamable for Tatiana’s death. Indeed, Tatiana’s parents could have asked for a temporary police protection, hired a private bodyguard, or taken Tatiana on a long vacation outside of California. Each of these actions is likely to prevent Tatiana’s death. Thus, by applying the principle of alternative possibilities directly, we have to conclude that her parents should be blamed for Tatiana’s death. However, the police is unlikely to provide lifetime protection; the parents’ resources can only be used to hire a bodyguard for a limited period time; and any vacation will have to end. These measures would only work if they knew an approximate time of a likely attack on their daughter. Without this crucial information, they had a strategy to prevent her death, but they did not know what this strategy was. If an agent has a strategy to achieve a certain outcome, knows that it has a strategy, and knows what this strategy is, then we say that the agent has a knowhow strategy. Axiomatic systems for knowhow strategies have been studied before Ågotnes and Alechina (2019); Fervari et al. (2017); Naumov and Tao (2017, 2018c, 2018b, 2019b). In a setting with imperfect information, it is natural to modify the principle of alternative possibilities to require an agent or a coalition to have a knowhow strategy to prevent. In our case, parents had many different strategies that included taking vacations in different months. They did not know that a vacation in October would have prevented Tatiana’s death. Thus, they cannot be blamed for her death according to the modified version of the principle of alternative possibilities. We write this as
Although Tatiana’s parents did not know how to prevent her death, Dr. Lawrence Moore did. He was a psychiatrist who treated Poddar at the University of California mental clinic. Poddar told Moore how he met Tatiana at the University international student house, how they started to date and how depressed Poddar became when Tatiana lost romantic interest in him. Less than two months before the tragedy, Poddar shared with the doctor his intention to buy a gun and to murder Tatiana. Dr. Moore reported this information to the University campus police. Since the University knew that Poddar was at the peak of his depression, they could estimate the possible timing of the attack. Thus, the University knew what actions the parents could take to prevent the tragedy. In general, if a coalition knows how a coalition can achieve a certain outcome, then coalition has a secondorder knowhow strategy to achieve the outcome. This class of strategies and a complete logical system that describes its properties were proposed in Naumov and Tao (2018a). We write if is true and coalition knew how coalition could have prevented . In our case,
After Tatiana’s death, her parents sued the University. In 1976 the California Supreme Court ruled that “When a therapist determines, or pursuant to the standards of his profession should determine, that his patient presents a serious danger of violence to another, he incurs an obligation to use reasonable care to protect the intended victim against such danger. The discharge of this duty may require the therapist to take one or more of various steps, depending upon the nature of the case. Thus it may call for him to warn the intended victim or others likely to apprise the victim of the danger, to notify the police, or to take whatever other steps are reasonably necessary under the circumstances.” Tobriner (1976). In other words, the California Supreme Court ruled that in this case the duty to warn is not only a moral obligation but a legal one as well. In this paper we propose a sound and complete logical system that describes the interplay between the distributed knowledge modality and the secondorder blameworthiness or duty to warn modality . The (firstorder) blameworthiness modality mentioned earlier could be viewed as an abbreviation for . For example, because Poddar knew how he himself could prevent Tatiana’s death.
The paper is organized as follows. In the next section we introduce and discuss the formal syntax and semantics of our logical system. In Section 3 we list axioms and compare them to those in the related logical systems. Section 4 gives examples of formal proofs in our system. Section 5 and Section 6 contain the proofs of the soundness and the completeness, respectively. Section 7 concludes.
2 Syntax and Semantics
In this section we introduce the formal syntax and semantics of our logical system. We assume a fixed set of propositional variables and a fixed set of agents . By a coalition we mean any subset of . The language of our logical system is defined by grammar:
where and are arbitrary coalitions. Boolean connectives , , and are defined through and in the usual way. By we denote the formula and by the set of all functions from set to set .
Definition \thetheorem
A game is a tuple , where

is a set of “initial states”,

is an “indistinguishability” equivalence relation on the set of initial states , for each agent ,

is a set of “actions”,

is a set of “outcomes”,

a set of “plays” is an arbitrary set of tuples such that , , and . Furthermore, we assume that for each initial state and each function , there is at least one outcome such that ,

for each propositional variable .
By a complete (action) profile we mean any function that maps agents in into actions in . By an (action) profile of a coalition we mean any function from set .
Figure 1 depicts a diagram of the game for the Tarasoff case. It shows two possible initial states: October and November that represent two possible months with the peak of Poddar’s depression. The actual initial state was October, which was known to the University, but not to Tatiana’s parents. In other words, the University could distinguish these two states, but the parents could not. We show the indistinguishability relation by dashed lines. At the peak of his depression, agent Poddar might decide not to attack Tatiana (action 0) or to attack her (action 1). Parents, whom we represent by a single agent for the sake of simplicity, might decide to take vacation in October (action 0) or November (action 1). Thus, in our example, . Set consists of outcomes and . Recall that a complete action profile is a function from agents into actions. Since in our case there are only two agents (Poddar and parents), we write action profiles as where is an action of Poddar and is an action of the parents. The plays of the game are all possible valid combinations of an initial state, a complete action profile, and an outcome. The plays are represented in the diagram by directed edges. For example, the directed edge from initial state October to outcome is labeled with action profile . This means that . In other words, if the peak of depression is in October, Poddar decides to attack (1), and the parents take vacation in November (1), then Tatiana is dead. Multiple labels on the same edge of the diagram represent multiple plays with the same initial state and the same outcome.
Function specifies the meaning of propositional variables. Namely, is the set of all plays for which proposition is true.
Next is the core definition of this paper. Its item 5 formally defines the semantics of modality . Traditionally, in modal logic the satisfiability is defined as a relation between a state and a formula. This approach is problematic in the case of the blameworthiness modality because this modality refers to two different states: if statement is true in the current state and coalition knew how coalition could have prevented in the previous state. In other words, the meaning of formula depends not only on the current state, but on the previous one as well. We resolve this issue by defining the satisfiability as a relation between a play and a formula, where a play is a triple consisting of the previous state , the complete action profile , and an outcome (state) . We distinguish initial states from outcomes to make the presentation more elegant. Otherwise, this distinction is not significant.
We write if for each agent . We also write if for each element .
Definition \thetheorem
For any game , any formula , and any play , the satisfiability relation is defined recursively as follows:

if ,

if ,

if or ,

if for each such that ,

if and there is a profile such that for each play , if and , then .
Going back to our running example,
because and
for each play such that and .
Because the satisfiability is defined as a relation between plays and formulae, one can potentially talk about two forms of knowledge about a play in our system: a priori knowledge in the initial state and a posteriori knowledge in the outcome. The knowledge captured by the modality as well as the knowledge implicitly referred to by the modality , see item (5) of Definition 2, is a priori knowledge about a play. In order to define a posteriori knowledge in our setting, one would need to add an indistinguishability relation on outcomes to Definition 2. We do not consider a posteriori knowledge because one should not be blamed for something that the person only knows how to prevent postfactum.
Since we define the secondorder blameworthiness using distributed knowledge, if a coalition is blamable for not warning coalition , then any superset could be blamed for not warning . One might argue that the definition of blameworthiness modality should include a minimality condition on the coalition . We do not include this condition in item (5) of Definition 2, because there are several different ways to phrase the minimality, all of which could be expressed through our basic modality .
First of all, we can say that is the minimal coalition among those coalitions that knew how could have prevented . Let us denote this modality by . It can be expressed through as:
Second, we can say that is the minimal coalition that knew how somebody could have prevented :
Third, we can say that is the minimal coalition that knew how the smallest coalition could have prevented :
Finally, we can say that is the minimal coalition that knew how some smallest coalition could have prevented :
The choice of the minimality condition depends on the specific situation. Instead of making a choice between several possible alternatives, in this paper we study the basic blameworthiness modality without a minimality condition through which modalities , , , , and possibly others could be defined.
3 Axioms
In addition to the propositional tautologies in language , our logical system contains the following axioms:

Truth: and ,

Distributivity: ,

Negative Introspection: ,

Monotonicity: and ,
where and , 
None to Act: ,

Joint Responsibility: if , then
, 
Strict Conditional: ,

Introspection of Blameworthiness: .
The Truth, the Distributivity, the Negative Introspection, and the Monotonicity axioms for modality are the standard axioms from the epistemic logic S5 for distributed knowledge Fagin et al. (1995). The Truth axiom for modality states that a coalition can only be blamed for something that has actually happened. The Monotonicity axiom for modality captures the fact that both distributed knowledge and coalition power are monotonic.
The None to Act axiom is true because the empty coalition has only one action profile. Thus, if the empty coalition can prevent , then would have to be false on the current play. This axiom is similar to the None to Blame axiom in Naumov and Tao (2019a).
The Joint Responsibility axiom shows how the blame of two separate coalitions can be combined into the blame of their union. This axiom is closely related to Marc Pauly Pauly (2002) Cooperation axiom, which is also used in coalitional modal logics of knowhow Ågotnes and Alechina (2019); Naumov and Tao (2017, 2018c, 2018b) and secondorder knowhow Naumov and Tao (2018a). We formally prove the soundness of this axiom in Lemma 5.
Strict conditional states that formula is known to to imply . By contraposition, coalition knows that if is prevented, then is also prevented. The Strict Conditional axiom states that if could be secondorder blamed for , then it should also be secondorder blamed for as long as is true. A similar axiom is present in Naumov and Tao (2019a).
Finally, the Introspection of Blameworthiness axiom says that if coalition is secondorder blamed for , then knows that it is secondorder blamed for as long as is true. A similar Strategic Introspection axiom for secondorder knowhow modality is present in Naumov and Tao (2018a).
We write if formula is provable from the axioms of our system using the Modus Ponens and the Necessitation inference rules:
We write if formula is provable from the theorems of our logical system and an additional set of axioms using only the Modus Ponens inference rule.
Lemma \thetheorem
If , then .
Proof
By the deduction lemma applied times, assumption implies that Thus, by the Necessitation inference rule,
Hence, by the Distributivity axiom and the Modus Ponens rule,
Then, again by the Modus Ponens rule,
Therefore, by applying the previous steps more times.
The next lemma capture a wellknown property of S5 modality. Its proof could be found, for example, in Naumov and Tao (2018a).
Lemma \thetheorem (Positive Introspection)
.
4 Examples of Derivations
The soundness of our logical system is established in the next section. Here we prove several lemmas about our formal system that will be used later in the proof of the completeness.
Lemma \thetheorem
.
Proof
Note that by the Introspection of Blameworthiness axiom. Thus, , by the law of contrapositive. Then, by the Necessitation inference rule. Hence, by the Distributivity axiom and the Modus Ponens inference rule,
At the same time, by the Negative Introspection axiom:
Then, by the laws of propositional reasoning,
Thus, by the law of contrapositive,
Since is an instance of the Truth axiom, by propositional reasoning,
Therefore, by the definition of .
Lemma \thetheorem
If , then .
Proof
By the Strict Conditional axiom,
Assumption implies by the laws of propositional reasoning. Hence, by the Necessitation inference rule. Thus, by the Modus Ponens rule, Then, by the laws of propositional reasoning,
(1) 
Observe that by the Truth axiom. Also, by the assumption of the lemma. Then, by the laws of propositional reasoning, . Therefore, by the Modus Ponens inference rule from statement (1).
Lemma \thetheorem
.
Proof
By the Truth axioms, . Hence, by the law of contrapositive, . Thus, by the definition of the modality . Therefore, by the Modus Ponens inference rule.
The next lemma generalizes the Joint Responsibility axiom from two coalitions to multiple coalitions.
Lemma \thetheorem
For any integer ,
where sets are pairwise disjoint.
Proof
We prove the lemma by induction on . If , then disjunction is Boolean constant false . Hence, the statement of the lemma, , is provable in the propositional logic.
Next, assume that . Then, from Lemma 4 using Modus Ponens rule twice, we get .
Assume now that . By the Joint Responsibility axiom and the Modus Ponens inference rule,
Hence, by Lemma 4,
At the same time, by the induction hypothesis,
Thus,
Note that is provable in the propositional logic. Thus,
(2) 
Similarly, by the Joint Responsibility axiom and the Modus Ponens inference rule,
Because formula is provable in the propositional logic, by Lemma 4,
Hence, by Lemma 4,
At the same time, by the induction hypothesis,
Thus,
Note that is provable in the propositional logic. Thus,
(3) 
Finally, note that the following statement is provable in the propositional logic for ,
Therefore, from statement (2) and statement (3),
by the laws of propositional reasoning.
Our last example rephrases Lemma 4 into the form which is used in the proof of the completeness.
Lemma \thetheorem
For any , any sets , and any pairwise disjoint sets ,
Proof
Let . Then, by Lemma 4,
Hence, by the Monotonicity axiom,
by the Modus Ponens inference rule. Hence, by the Truth axiom,
The following formula is an instance of the Strict Conditional axiom . Thus, by the Modus Ponens applied twice,
Then,
by the Modus Ponens.
Thus,
by the deduction lemma.
Hence,
by Lemma 3 and the definition of set . Then,
by the Monotonicity axiom, the Modus Ponens inference rule, and the assumption . Thus,
by the definition of modality , the Negative Introspection axiom, and the Modus Ponens rule. Therefore, by Lemma 3 and the Modus Ponens inference rule, the statement of the lemma is true.
5 Soundness
The soundness of the Truth, the Distributivity, the Negative Introspection, the Monotonicity, and the None to Blame axioms is straightforward. Below we prove the soundness of the Joint Responsibility, the Strict Conditional, and the Introspection of Blameworthiness axioms as separate lemmas.
Lemma \thetheorem
If , , , and , then .
Proof
By Definition 2 and the definition of modality , assumption implies that there is a play such that and . Thus, again by Definition 2, there is an action profile such that for each play , if and , then . Recall that . Thus, for each play ,
(4) 
Similarly, assumption implies that there is a profile such that for each play ,
(5) 
Let be the action profile:
(6) 
Action profile is welldefined because . Statements (4), (5), and (6) by Definition 2 imply that for each play if and , then . Recall that . Therefore, by Definition 2.
Lemma \thetheorem
If , , and , then .
Proof
By Definition 2, assumption implies that for each play of the game if , then .
By Definition 2, assumption implies that there is an action profile such that for each play , if and , then .
Hence, for each play , if and , then . Therefore, by Definition 2 and the assumption of the lemma.
Lemma \thetheorem
If , then .
Proof
By Definition 2, assumption implies that there is an action profile such that for each play , if and , then .
Let be a play where and . By Definition 2, it suffices to show that .
Consider any play such that and . Then, since is an equivalence relation, assumptions and imply . Thus, by the choice of action profile . Therefore, by Definition 2 and the assumption .
6 Completeness
The standard proof of the completeness for individual knowledge modality defines states as maximal consistent sets Fagin et al. (1995). Two such sets are indistinguishable to an agent if these sets have the same formulae. This construction does not work for distributed knowledge because if two sets share formulae and formulae, they do not necessarily have to share formulae. To overcome this issue, we use the Tree of Knowledge construction, similar to the one in Naumov and Tao (2018c). An important change to this construction proposed in the current paper is placing elements of a set on the edges of the tree. This change is significant for the proof of Lemma 6.
Let be an arbitrary set of cardinality larger than that of the set . Next, for each maximal consistent set of formulae , we define the canonical game