Metrics for Signal Temporal Logic Formulae
Abstract
Signal Temporal Logic (STL) is a formal language for describing a broad range of realvalued, temporal properties in cyberphysical systems. While there has been extensive research on verification and control synthesis from STL requirements, there is no formal framework for comparing two STL formulae. In this paper, we show that under mild assumptions, STL formulae admit a metric space. We propose two metrics over this space based on i) the PompeiuHausdorff distance and ii) the symmetric difference measure, and present algorithms to compute them. Alongside illustrative examples, we present applications of these metrics for two fundamental problems: a) design quality measures: to compare all the temporal behaviors of a designed system, such as a synthetic genetic circuit, with the “desired” specification, and b) loss functions: to quantify errors in Temporal Logic Inference (TLI) as a first step to establish formal performance guarantees of TLI algorithms.
I Introduction
Temporal logics [1] are increasingly used for describing specifications in cyberphysical systems such as robotics [2], synthetic biology [3], and transportation [4]. Variants of temporal logics, such as Computation Tree Logic (CTL) [5], Linear Temporal Logic (LTL) [6], or Signal Temporal Logic (STL) [7], can naturally describe a wide range of temporal system properties such as safety (never visit a “bad” state), liveness (eventually visit a “good” state), sequentiality, and their arbitrarily elaborate combinations.
Using model checking [1] techniques, signals or traces can be checked to determine whether or not they satisfy a specification. For STL in particular, the degree of satisfaction or robustness is a quantitative measure to characterize how far a signal is from satisfaction [8, 9, 10] of an STL formula. There is currently, however, no formal way to directly compare specifications against each other. Previous related approaches in planning and control have looked into specification relaxation, where the goal is to minimally enlarge the specification language to include a satisfying control policy for the system model. Various specification relaxations have been defined including minimum violation [11, 12, 13] for selfdriving cars, temporal relaxation of deadlines [14], minimum revision of Büchi automata [15], and diagnosis and repair in reactive synthesis [16]. While language inclusion and equivalence problems are of paramount importance in computer science and control theory, they are only qualitative measures while we are interested in quantitative metrics.
This paper presents two metrics that can be used to compute the distance between two STL specifications. Under mild assumptions, we propose the metrics based on the languages of STL formulae. We propose two distance functions. The first is based on the PompeiuHausdorff (PH) distance [17], which captures how much the language of one formula must be enlarged to include the other, and the second is based on the symmetric difference (SD) [18], which characterizes how much overlap there is between the two formulae. The theoretical contributions of this paper are:

formalization of STL formulae metrics based on the PH and the SD distances, and

methods for computing the PH using mixedinteger linear programming (MILP), and the SD using a recursive algorithm based on the area of satisfaction.
We discuss the comparison of the two metrics in detail and provide examples that highlight their differences.
This paper additionally presents applications of the proposed metrics to a behavioral synthesis problem and to the evaluation of temporal logic inference (TLI) [19, 20, 21, 22, 23] methods. In the first case, we are interested in generating designs that exhibit desired behaviors specified in STL. For example, we study synthetic genetic circuits. Possible circuit designs are constructed and measured in laboratory experiments, and the resulting traces are abstracted into STL specifications using TLI. These formulae are compared quantitatively against the desired design specification using the proposed metrics. The second setup considers the fundamental problem of evaluating TLI methods themselves. Under the assumption that data used for inference can be characterized by ground truth STL formulae, we ask the question of how well the TLI algorithms perform. As opposed to empirical evaluation used in previous work, we propose to use our metrics as loss functions as the first step in establishing theoretical foundations for TLI. The related contributions are:

a design quality measure for evaluating proposed implementations against an STL specification, and

a loss function to quantify errors in TLI as a first step in establishing formal performance guarantees of TLI algorithms.
Ii Preliminaries
Notation
Let , denote the set of real, nonnegative real, and natural numbers, respectively. We use to denote the absolute value of . Given ,  where is the ’th component of  is its infinitynorm. A scalarvalued function is rectangular if for some , . A metric space is an ordered pair , where is a set and is a distance function such that i) ; ii) iii) . If and are metric spaces, then , is also a metric space for any .
We use discrete notion of time throughout this paper. Time intervals in the form , , are interpreted as . is denoted by , . The continuous interval is denoted by . An dimensional, real, infinitetime, discretetime signal is defined as a string of real values , where , . The suffix of at , denoted by , is a signal such that for all . We use to refer to a particular portion of a signal. The set of all signals with values taken in is denoted by . The set of all signal prefixes with time bound is defined as For the convenience of notation, we use to say . The distance between two signals is defined as
Signal Temporal Logic
The syntax of STL is defined as follows [7]:
where is the Boolean true constant; is a predicate over in the form of , , , and ; and are the Boolean operators for negation and conjunction, respectively; and is the temporal operator until over bounded interval . A predicate is rectangular if is rectangular. Other Boolean operations are defined in the usual way. Additional temporal operators eventually and globally are defined as and , respectively, where is an interval. The set of all STL formulae over signals in is denoted by . The STL score, also known as robustness degree is a function , which is recursively defined as [7]:
(1) 
As one can inspect from (1), a signal satisfies an STL specification at a certain time if and only if its corresponding STL score is positive: where is read as “satisfies”. The case of is usually left ambiguous  this is never a concern in practice due to issues with numerical precision. In this paper, we consider as satisfaction, but by doing so, we sacrifice the principle of contradiction: and if .
The horizon of an STL formula is defined as the minimum length of the time window required to compute its score, and it is recursively computed as [24]:
(2) 
The set of all STL formulae over signals in such that their horizons are less than is denoted by . Note that computing requires , and the rest of the values are irrelevant.
Definition 1 (BoundedTime Language)
Given , we define the boundedtime language as:
(3) 
Note that . When the predicates are rectangular, the boundedtime language becomes a finite union of hyperrectangles in .
Example 1
Let . Consider the following six STL formulae in :
(4) 
where and . We have . Two examples of boundedtime languages are: Consider two constant signals and , where . The STL scores are computed from (1). For instance, , (minimizer at ), and (maximizer at ).
Iii Metrics
In this section, we introduce two functions that quantify the dissimilarity between the properties captured by the two STL formulae. However, it is possible that different formulae may describe the same properties. For example, and in (4) are describing the same behavior, since any signal that satisfies already satisfies and vice versa. The key idea is to define the distance between two STL formulae as the distance between their timebounded languages.
Assumption 1
The set is compact.
Assumption 2
All of the predicates are rectangular.
Note that boundedtime languages are constructed in finitedimensional Euclidean spaces. Also, since all inequalities in the predicates are nonstrict, boundedtime languages are compact sets. Assumption 2 is theoretically restrictive, but not in most applications  usually it is the case that all predicates are rectangular as they describe thresholds for state components of a system.
Definition 2
We say that the two STL formulae and are semantically equivalent, denoted by , if both induce the same language: .
The set of equivalence classes of induced by is denoted by . Distance functions are effectively pseudometrics on , but proper metrics on , where is the induced metric, is the equivalence class associated with , and and are formulae in the two equivalence classes. Note that, by definition, there is a onetoone map between the equivalence classes of STL formulae and their formulae. Moreover, for any , we have .
We adapt two common metrics between sets: (a) the PompeiuHausdorff (PH) distance based on the underlying metric between signals, and (b) a measure of Symmetric Difference (SD) between sets. As it will be clarified in the paper, the choice of , as long as it is larger than the horizons of the formulae that are considered, does not affect the fundamental properties of the defined metrics. In the case of the PH distance, it does not have any effect at all. For the SD metric, the computed distances are scaled with respect to the inverse of . These details are explained in Section IIIB.
Iiia PompeiuHausdorff Distance
Definition 3
The (undirected) PH distance is defined as:
(5) 
where denotes the directed PH distance:
(6) 
Note that the directed PH distance is obviously not a metric as it is possible to have . We have if and only if . Another way to interpret the PH distance is as follows [17]:
(7) 
where is the unit ball in and addition of sets is interpreted in the Minkowski sense. In words, is the radius of the minimum ball that should be added to such that it contains .
Proposition 1
The is a metric space.
Note that is effectively defined as  remember that languages are compact subsets of finite dimensional Euclidean space, for which it is known that the PH distance is a metric [17]. Moreover, there is a one to one map between an equivalency class formula of a formula in and its language.
It is possible to interpret (6) as the distance between an STL formula and a signal: . It is easy to see that we have if and only if . The following result is a reformulation of Definition 23 in [8], which establishes a connection between the STL score and the notion of signed distance.
Proposition 2
Given any and , the STL score is a signed distance in the sense that:
(8) 
The following results are extensions of classical results for signed distances [25].
Corollary 1
For any given two formulae and a signal , we have the following inequalities:
Corollary 2
Given , define neighborhood of an STL formula as . Then, implies that .
IiiB Symmetric Difference
The SD is denoted by , and defined as , where and are two sets. It induces a distance between compact sets as the measure of the SD [18].
Definition 4
The SD metric is defined as:
where is the Lebesgue measure.
Proposition 3
The is a metric space.
Follows immediately from the definition. The metric is welldefined since the formulae have time horizons bounded by over discretetime signals. Their languages are compact subsets of the Euclidean space . Thus, the Lebesgue measure is defined.
We define the coverage of signal sets in the spacetime value set. Formally, we have the map such that , where . For an STL formula , .
Let , and be a rectangular predicate with and . The coverage of is .
Theorem 1
If and are two STL formulae with the same language, then they cover the same space. Formally, we have implies .
Immediately follows from the fact that is a projection of onto spacetime .
Note that the converse is not true in general. In particular, it can fail for formulae containing disjunctions.
IiiC Comparison
While the PH distance and the SD difference are both metrics, they have quite different behaviors. Here, we elaborate on these differences and show an illustrative example.
Informally, the PH distance has a stronger spatial notion, and it is closely connected to STL score, as stated in Corollary 1 and Corollary 2. The PH distance, captures the worstcase spatial difference between formulae. On the other hand, the SD is a more temporal notion, as the areas also capture the length of temporal operators. It is possible that two STL formulae have a large PH distance, but a small SD distance, and vice versa. In applications, the choice is dependent on the user. The most useful may be a convex combination  which is a metric by itself  with a usergiven convex coefficient.
Example 2
Consider the six STL formulae in (4). We compute the PH and the SD distances between all pairs of formulae using the methods proposed in Section IV. The directed PH distances are also reported.
1) Directed PH distance: The results are shown in Table I, where the value in the ’th row and ’th column is . Each maximizer (the PH distance) is bolded.
We have also included the truth constant in the distance table. It is observed that , which implies the fact that the language of each formula is contained within the language of , which is the set of all signals. The opposite direction, , is, informally, the quantification of how restrictive is.
Note that . It is observed that most values are either or , which correspond to the extreme signal that one language contains but the other does not, or it is , indicating that one language is a subset of another. For instance, is a “weak” specification in the sense that its language is broad  any signal with some value in at some time satisfies it  so the directed distances from other formulae to are zero. Another notable example is the relation between and . The directed PH distance is zero in both directions  the two formulae are equivalent. This is due to the fact that any signal that satisfies , already satisfies . The other direction also trivially holds. Note that some pairs, like and , have nonzero PH distances in both directions.
2) SD distance: The results are shown in Table II along with the PH distances for comparison. Here, it can be seen that in most cases, the SD distance is either a lot larger or a lot smaller than the PH distance. This is largely due to the fact that this metric is based on area which is particularly highlighted when comparing any of the formulae to . Since each formula’s satisfaction space is very small in comparison to the entire bounded signal space, each of these values is quite large. In contrast, the SD distance between and is fairly small since the satisfaction regions for each of these formulae cover a similar area. The SD distance between and is on the larger side as their areas of satisfaction are quite different; however, these areas are still much closer to each other than they are to the entire bounded satisfaction area represented by . Similar to the PH distance, the SD distance between and is zero as they have completely overlapping areas of satisfaction.
The results in Table II illustrate that there are different situations when the PH distance might be favored over the SD distance and vice versa. In cases where one cares about the area covered by the satisfaction region of a formula, the SD distance should be used. For instance, the SD could be used to find a formula close to one that requires a signal to be held at a particular value for a long time interval. However, if one only cares about how close the signal bounds of the formulae are to each other, the PH distance should be used.
Iv Computation
This section presents algorithms for computing the PH and the SD distances between STL specifications.
Iva PompeiuHausdorff Distance
In this section, we propose an optimizationbased method to compute the PH distance between two STL formulae.
Definition 5
Given an STL formula that contains no negation, we define with the same logical structure as with predicates replaced as follows:

replaced with ;

replaced with .
Intuitively, is a relaxed version of . It is easy to verify from (1) that if .
Lemma 1
The following relation holds:
(9) 
(sketch) The result is a direct consequence of (7) as we have . The following statement provides the main result, and the base for the computational method of this section.
Theorem 2
Given , , define as the following optimum:
(10) 
Then the following holds:
(11) 
First, consider the case that (10) is infeasible or its value is . Then, it means that the constraints are infeasible for all , which implies that for any , we have . Thus, and consequently .
Now consider the case (10) is feasible and . Then, is an active constraint, which implies . Note that is also optimized in (10). Thus , or . We can rewrite (9) as:
(12) 
Note that we have used instead of as is a strict relation. Also note that such the supremum exists as i) the condition is satisfied for and ii) the language sets are bounded. We show that (10) captures (12). If , then it means that but . This is what is captured by the constant in , with the difference that is replaced by a nonstrict inequality and is replaced by . We convert (10) into a MILP problem. The procedure for converting STL into MILP constraints is straightforward, see, e.g., [26]. The encoding details are omitted here. By solving two MILPs, we are able to obtain the PH distance. Two MILPs can be aggregated into a single MILP, but that usually more than doubles the computation time due to larger branch and bound trees. Moreover, it is often useful to have the knowledge of the directed PH distances.
Theorem 2 requires that formulae do not contain negation. Negation elimination is straightforward: first, the formula is brought into its Negation Normal Form (NNF), where all negations appear before the predicates. Next, the predicates are negated. For example, we replace by . We remind the reader that we do not consider strict inequalities, hence and are both true if . Finally, observe that the choice of does not effect the values of PH distance, as long as it is larger than the horizons of two formulae that are compared. Given , the values of for do not have any associated constraints in (10).
Complexity
The complexity of (10) is exponential in the number of integers, which grows with the number of predicates and horizons of the formulae. However, since signal values do not have any dynamical constraints, we found solving (10) to be orders of magnitudes faster than comparable STL control problems, such as those studied in [26]. All the values obtained in Table I were evaluated almost instantaneously using Gurobi MILP solver on a personal computer.
IvB Symmetric Difference
This section presents an algorithm for computing boxes representing the area of satisfaction of a formula as well as a method for determining the SD between two sets of boxes. Each set of boxes approximates the projection () of the formula and represents the valid valuespace that a timevarying signal can take such that traces that are contained entirely within the boxes satisfy the formula.
Computing the set of boxes representing the area of satisfaction is a recursive process that takes as input an STL formula, , a set of max values, , for each signal, (used to normalize the signal values to a unit space), and a discretization threshold, . This algorithm, , is presented in Algorithm 1 Here, creates a new box with minimum and maximum times and , minimum and maximum values and , and spatial dimension , respectively; determines the time window of the overlap between two boxes; takes two boxes and produces a set of boxes representing the intersection of the overlapping time window region; , , , and return the lower time window, upper time window, lower variable, and upper variable values for box , respectively; in the definition of a box denotes that it may restrict multiple spatial dimensions; and the operator is used to create a “choice” set representing that either of the two sets separated by it can be selected as the set of boxes representing the area of satisfaction.
To address the problem of projection of formulae containing disjunction (the converse to Theorem 1), utilizes the operator. If this algorithm instead generated boxes representing the projection of all formulae, it would be possible for the satisfaction space represented by the boxes to capture signals that the original formula does not allow. The application in Section V highlights this problem and presents a way of dealing with it for that particular example.
For operators such as globally (), is exact and produces boxes bound by the time bounds of the operator that represent the projection of the primitive. However, operators such as eventually () do not immediately lend themselves to conversion into a set of boxes. In order to deal with this operator, we approximate it by converting it into a disjunction of globally predicates. Each globally predicate is generated using a small threshold value () for its time window width. The new formula requires that the expression be true in at least one of the smaller time windows essentially introducing a mandatory “hold” time for eventually operators. The tunability of allows for a user to give up some accuracy for gains in performance of box computation and ultimately distance comparison. Examples of computing the area of satisfaction boxes for some of the formulae in Example 2 are shown in Figure 4.
The SD between two sets of boxes is computed by calculating the area of the sum of the nonintersected area for each box set. This value is normalized by the maximum time horizon, , and results in the SD computation:
Figure (c)c visually illustrates how the SD between and is computed.
Note that the SD distance is scaled by if the maximum horizon is increased by . This again shows the temporal nature of the SD as opposed to the PH distance which does not change.
Complexity
The complexity of Algorithm 1 depends on the complexity of the operation which may be exponential depending on how it is implemented. Otherwise, the algorithm is polynomial due to the box combination operations carried out whenever a conjunction predicate is encountered. In practice, computing the SD distance using this method for formulae with a few dozen predicates typically takes only a few seconds.
V Quantification of design quality
In our first application, we show an example of how the proposed metrics can be used in behavioral synthesis. Behavioral synthesis is an important process in design automation where the description of a desired behavior is interpreted and a system is created that implements the desired behavior. Our goal is to check if the characterized implementations satisfy the specifications of a system. Implementations include simulations and execution traces of a system. These implementations are characterized into formal specifications using TLI. We show that the proposed metrics can be used in the synthesis step to choose a design from the solution space that can best implement the desired specification. The specific example we have chosen to highlight this application is the synthesis of genetic circuits in synthetic biology.
Synthetic Genetic Circuit Synthesis
In this example, we have a set of desired behaviors (each formally represented by STL) which describe the various behaviors expected of a genetic circuit. This set of behaviors is referred to as a performance specification: . consists of 2 STL formulae: and which describe the desired amount of output produced by the genetic circuit over time:
In this case, the output of the circuit corresponds to the expression of a fluorescent protein. specifies that the output must consistently be below 40 units from time 0 to 300, and specifies that that output must gradually increase over time and must end up between 150 and 320 units between time 200 and 300. Our solution space consists of two genetic circuits. The first circuit has a constitutive promoter as shown in Figure (a)a. Constitutive expression removes flexibility for consistency allowing constant protein production independent of the state or inputs of the system, which is highlighted in Figure (c)c. The second circuit has an inducible promoter: a sugar detecting transcription factor AraC*, which will turn on the protein production if and only if it is in the presence of a specific input molecule (arabinose) as shown in Figure (b)b. Figure (d)d shows the output of the circuit for various concentrations of arabinose. Both of these synthetic genetic circuits were built in Escherichia coli. The traces were obtained from biological experiments by measuring fluorescence.
Our goal is to choose the circuit that can “satisfy as many behaviors as possible” in . It is important to note here that it is difficult to express the term “satisfy as many behaviors as possible” using the syntax and semantics of STL. For instance, expressing the desired specification as a disjunction of all the formulae in would imply that satisfying any one specification is sufficient for the genetic circuit to satisfy the performance specification. Similarly, expressing the desired specification as a conjunction of all the formulae in would imply that at any point in time, the output of a genetic circuit must have multiple distinct values, which is physically impossible.
This conundrum is highlighted in the current example. The output of constitutive expression satisfies but cannot satisfy . The induction circuit produces traces that can satisfy both and . However, traditional model checking techniques may not help a designer choose the desired circuit. Using statistical model checking, for example, the circuit with constitutive expression yields a satisfaction likelihood of and the induction circuit yields a satisfaction likelihood of when checked against . With these results, one might think that the circuit with constitutive expression best satisfies the performance specification.
To address the issue of satisfying as many behaviors as possible, we treat the performance specification’s region of satisfaction as the union of the regions of satisfaction of all the formulae in as shown in Figure (a)a. We compute this region by taking the union of the generated boxes for each formula that are computed using Algorithm 1. The union of the box sets of all STL formulae in is represented as and is shown in Figure (b)b.
Using Grid TLI [22], we produce STL formulae, and , for each circuit using the traces shown in Figures (c)c and (d)d, respectively. We then use the SD metric and get the following values: = and = . Using the PH metric, we get: = and = . These results imply that the behavior of the induction circuit is closer to the desired specification than the circuit with constitutive expression, and thus, it should be selected as the desired circuit.
Vi Loss functions for TLI
Loss functions play a fundamental role in statistical inference and learning theory. In this framework, we usually have a pair of real vector spaces corresponding to the state and observation spaces, respectively. Three ingredients are used in the formalization: (a) a model of the states – the prior distribution, (b) a model of the observations given the state , and (c) a realvalued loss function . Let be a decision rule, which can also be interpreted as a partition of the state space based on observations, and be the set of all decision rules or the hypothesis space. The frequentist and Bayesian risks are defined based on the loss functions, and induce optimal decision rules. This general framework is the basis for the study and design of decision algorithms in statistical inference and learning. For more details, see [27].
In the following, we show how we adapt this framework for TLI, where we use the proposed metrics, PH and SD, as loss functions. In this paper, we only focus on the loss functions, while characterization of optimal decision rules, their computation, and regularization are left for future work.
For TLI, the state space is the set of all timebounded STL formulae, while the observation space is the set of all languages. The hypothesis space is composed of decision rules that map languages to STL formulae. Lastly, the loss functions are defined as such that represents the dissimilarity between the ground truth formula and the STL formula obtained by the decision rule using the signal set . We propose to use the PH and the SD metrics as loss functions .
We assess the performance of the two decision rules from TLI: TreeTLI [21] based on decision trees, and GridTLI [22] based on minimum covers of signals in spacetime , with respect to the two metrics as shown in Figure (b)b.
The ground truth STL formula that was used to generate the signals in Figure (a)a is
where and . Note that TreeTLI requires both positive and negative examples, while GridTLI only needs positive ones. For brevity, we omit here the formalization for rules that require both types of examples.
The results in Figure (b)b show the distances between the ground truth formula and the iterations of TreeTLI (lower plot) as the decision tree grows [21]. For GridTLI (upper plot), we varied the discretization thresholds [22] from rougher to finer grids, and for space and time, respectively. The upper plot for GridTLI highlights the overfitting phenomenon in the PH metric (red), where reducing the discretization thresholds helps reducing the error, but further reduction leads to overfitting. For the SD (blue), the loss has a decreasing trend which we hypothesize is due to a better temporal fitting that the PH distance does not capture. In the case of TreeTLI (lower plot), the PH distance (red) is constant. This masking behavior might be due to the compounding effect of i) the primitives used do not match the structure of , and ii) the incremental and local nature of TreeTLI. Thus, the first step of the decision tree is heavily penalized by the PH metric. The SD metric (blue), which shows an increasing trend, is consistent with this conclusion.
Thus, the statistical learning approach to TLI gives insight into the ability of algorithms to recover temporal logic rules assumed to underlie data. It also provides a formal framework to study TLI methods. A detailed account of problems GridTLI and TreeTLI are appropriate for based on the insights provided by the proposed metrics is left for future work.
Vii Discussion and Future Work
We presented two metrics for computing the distance of one STL formula to another. These methods are very useful in applications where temporal logic specifications are mined from simulation or experimental data, and need to be compared against a desired specification. Fields such as synthetic biology and robotics, where systems are characterized with performance specifications, can greatly benefit from our methods. We also showed how these metrics are useful as a first step in evaluating the performance of TLI methods.
An immediate theoretical extension is studying continuoustime signals. By assuming Lipschitz continuity of signals, it is possible to provide bounds between the metrics computed in discretetime and the ones in continuoustime. A similar idea was used in [8] to compute sampledtime STL scores. The second extension is relaxing the assumption on rectangular predicates.
References
 [1] C. Baier and J. Katoen, Principles of model checking. MIT Press, 2008.
 [2] H. KressGazit, G. E. Fainekos, and G. J. Pappas, “Temporallogicbased reactive mission and motion planning,” IEEE transactions on robotics, vol. 25, no. 6, pp. 1370–1381, 2009.
 [3] G. Batt, B. Yordanov, R. Weiss, and C. Belta, “Robustness analysis and tuning of synthetic gene networks,” Bioinformatics, vol. 23, no. 18, pp. 2415–2422, 2007.
 [4] S. Coogan, M. Arcak, and C. Belta, “Formal methods for control of traffic flow: Automated control synthesis from finitestate transition models,” IEEE Control Systems, vol. 37, no. 2, pp. 109–128, 2017.
 [5] E. A. Emerson and E. M. Clarke, “Using branching time temporal logic to synthesize synchronization skeletons,” Science of Computer Programming, vol. 2, no. 3, pp. 241 – 266, 1982.
 [6] A. Pnueli, “The temporal logic of programs,” in Foundations of Computer Science, 1977., 18th Annual Symposium on. IEEE, 1977, pp. 46–57.
 [7] O. Maler and D. Nickovic, “Monitoring temporal properties of continuous signals,” in Formal Techniques, Modelling and Analysis of Timed and FaultTolerant Systems. Springer, 2004, pp. 152–166.
 [8] G. E. Fainekos and G. J. Pappas, “Robustness of temporal logic specifications for continuoustime signals,” Theoretical Computer Science, vol. 410, no. 42, pp. 4262–4291, 2009.
 [9] A. Donzé and O. Maler, Robust satisfaction of temporal logic over realvalued signals. Springer, 2010.
 [10] A. Donzé, T. Ferrere, and O. Maler, “Efficient robust monitoring for STL,” in Computer Aided Verification. Springer, 2013, pp. 264–279.
 [11] J. Tumova, L. ReyesCastro, S. Karaman, E. Frazzoli, and D. Rus, “Minimumviolating planning with conflicting specifications,” in American Control Conference (ACC), 2013.
 [12] J. Tumova, G. C. Hall, S. Karaman, E. Frazzoli, and D. Rus, “Leastviolating control strategy synthesis with safety rules,” in International Conference on Hybrid Systems: Computation and Control, Philadelphia, PA, USA, 2013, pp. 1–10.
 [13] C.I. Vasile, J. Tumova, S. Karaman, C. Belta, and D. Rus, “Minimumviolation scLTL motion planning for mobilityondemand,” in IEEE International Conference on Robotics and Automation, Singapore, Singapore, May 2017, pp. 1481–1488.
 [14] C. I. Vasile, D. Aksaray, and C. Belta, “Time Window Temporal Logic,” Theoretical Computer Science, vol. 691, no. Supplement C, pp. 27–54, August 2017.
 [15] K. Kim, G. Fainekos, and S. Sankaranarayanan, “On the minimal revision problem of specification automata,” The International Journal of Robotics Research, 2015.
 [16] S. Ghosh, D. Sadigh, P. Nuzzo, V. Raman, A. Donzé, A. L. SangiovanniVincentelli, S. S. Sastry, and S. A. Seshia, “Diagnosis and repair for synthesis from signal temporal logic specifications,” in Proc. International Conference on Hybrid Systems: Computation and Control. ACM, 2016, pp. 31–40.
 [17] J. R. Munkres, Topology. Prentice Hall, 2000.
 [18] J. B. Conway, A course in abstract analysis. American Mathematical Soc., 2012, vol. 141.
 [19] E. Bartocci, L. Bortolussi, and G. Sanguinetti, “Datadriven statistical learning of temporal logic properties,” in Formal Modeling and Analysis of Timed Systems, A. Legay and M. Bozga, Eds. Cham: Springer International Publishing, 2014, pp. 23–37.
 [20] X. Jin, A. DonzÃ©, J. V. Deshmukh, and S. A. Seshia, “Mining requirements from closedloop control models,” IEEE Transactions on ComputerAided Design of Integrated Circuits and Systems, vol. 34, no. 11, pp. 1704–1717, 2015.
 [21] G. Bombara, C.I. Vasile, F. Penedo, H. Yasuoka, and C. Belta, “A Decision Tree Approach to Data Classification Using Signal Temporal Logic,” in Proc. International Conference on Hybrid Systems: Computation and Control. New York, NY, USA: ACM, 2016, pp. 1–10.
 [22] P. Vaidyanathan, R. Ivison, G. Bombara, N. A. DeLateur, R. Weiss, D. Densmore, and C. Belta, “Gridbased temporal logic inference,” in Annual Conference on Decision and Control (CDC). IEEE, 2017, pp. 5354–5359.
 [23] B. Hoxha, A. Dokhanchi, and G. Fainekos, “Mining parametric temporal logic properties in modelbased design for cyberphysical systems,” International Journal on Software Tools for Technology Transfer, vol. 20, no. 1, pp. 79–93, Feb 2018.
 [24] A. Dokhanchi, B. Hoxha, and G. Fainekos, 5th International Conference on Runtime Verification, Toronto, ON, Canada. Springer, 2014, ch. OnLine Monitoring for Temporal Logic Robustness, pp. 231–246.
 [25] D. Kraft, “Computing the Hausdorff distance of two sets from their signed distance functions,” Computational Geometry, Submitted March 2015.
 [26] V. Raman, A. Donzé, M. Maasoumy, R. M. Murray, A. SangiovanniVincentelli, and S. A. Seshia, “Model predictive control with signal temporal logic specifications,” in Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on. IEEE, 2014, pp. 81–87.
 [27] V. Vapnik, The nature of statistical learning theory. Springer, 2013.