Equivalence Problems for Tree Transducers:A Brief Survey

Equivalence Problems for Tree Transducers:
A Brief Survey

Sebastian Maneth \IfArrayPackageLoaded
School of Informatics University of Edinburgh
School of Informatics University of Edinburgh

The decidability of equivalence for three important classes of tree transducers is discussed. Each class can be obtained as a natural restriction of deterministic macro tree transducers (mtts): (1) no context parameters, i.e., top-down tree transducers, (2) linear size increase, i.e., mso definable tree transducers, and (3) monadic input and output ranked alphabets. For the full class of mtts, decidability of equivalence remains a long-standing open problem.

Z. Ésik and Z. Fülöp (Eds.): Automata and Formal Languages 2014 (AFL 2014) EPTCS 151, 2014, pp. Equivalence Problems for Tree Transducers: A Brief SurveyLABEL:LastPage, doi:10.4204/EPTCS.151.5 © S. Maneth This work is licensed under the Creative Commons Attribution License.

Equivalence Problems for Tree Transducers:

A Brief Survey

Sebastian Maneth \IfArrayPackageLoaded
School of Informatics
University of Edinburgh
School of Informatics
University of Edinburgh

1 Introduction

The macro tree transducer (mtt) was invented independently by Engelfriet [21, 30] and Courcelle [14, 15] (see also [38]). As a model of syntax-directed translations, mtts generalize the attribute grammars of Knuth [47]. Note that one (annoying) issue of attribute grammars is that they can be circular; mtts always terminate. Macro tree transducers are a combination of context-free tree grammars, invented by Rounds [58] and also known as “macro tree grammars” [35], and the top-down tree transducer of Rounds and Thatcher [59, 68]: the derivation of the grammar is (top-down) controlled by a given input tree. In terms of a top-down transducer, the combination is obtained by allowing nesting of state calls in the rules (similar to the nesting of nonterminals in the productions of context-free tree grammars). Top-down tree transducers generalize to trees the finite state (string) transducers (also known as “generalized sequential machines”, or gsms, see [39, 9]). In terms of formal languages, compositions of mtts give rise to a large hierarchy of string languages containing, e.g., the io and oi hierarchies (at level one they include the indexed languages of Aho [2]), see [23]. mtts can be applied in many scenarios, e.g., to type check xml transformations (they can simulate the -pebble transducers of Milo, Suciu, and Vianu [52]), see [24, 50, 51], or to efficiently implement streaming XQuery transformations [43, 53]. In terms of functional programs, mtts are particularly simple programs that do primitive recursion over an input tree and only produce trees as output. Applications in programming languages exist include [69, 70, 53].

Equivalence of nondeterministic transducers is undecidable, already for restricted string transducers [41]. We therefore only consider deterministic transducers. What is known about the equivalence problem for deterministic macro tree transducers? Unfortunately not much in the general case. Only a few subcases are known to be decidable. Here we describe three of them:

  1. top-down tree transducers

  2. linear size increase transducers

  3. monadic tree transducers.

The first one was solved long ago by Esik [31], but was revived through the “earliest canonical normal form” by Engelfriet, Maneth, and Seidl [27]. The latter implies Ptime equivalence check for total top-down tree transducers. The second one is solved by the decidability of equivalence for deterministic mso tree transducers of Engelfriet and Maneth [26]. The same authors have shown that every mtt of linear size increase is effectively equal to an mso transducer [25]. Hence, decidability of equivalence follows for mtts of linear size increase. Here we give a direct proof using mtts. The third result is about mtts over monadic trees. These are string transducers with copying. The decidability of their equivalence problem follows through a relationship with L-systems [29]; in particular, with the sequence equivalence problem of hdt0l systems. The latter was first proved decidable by Culik II and Karumhäki [19].

2 Preliminaries

We deal with finite, ordered, ranked trees. In such a tree, each node is labeled by a symbol from a ranked alphabet such that the rank of the symbol is equal to the number of children of the node. Formally, a ranked alphabet consists of a finite set together with a mapping associating to each symbol its rank. We write to denote that the rank of is equal to . By we denote the subset of symbols of that have rank . Let be a ranked alphabet. The set of all trees over , denoted , is the smallest set of strings such that if , , and , then also . For a tree of the form we simply write . For a set , we denote by the set of all trees over such that the rank of each is zero. Let be distinct symbols in and let such that none of the leaves in are labeled by for . Then by we denote the tree substitution that replaces each leaf labeled by the tree . Thus denotes the tree .

Let for some ranked alphabet . We denote the nodes of by their Dewey dotted decimal path and define the set of nodes of as if with , , and . Thus, denotes the root node, and denotes the th child of the node . For a node we denote by its label, and, for a tree we denote by the tree obtained from by replacing its subtree rooted at by the tree . The size of , denoted , is its number of nodes. The height of , denoted , is defined as for , , and .

We fix the set of input variables and the set of formal context parameters . For we define and .

A deterministic finite-state bottom-up tree automaton is a tuple where is a finite set of states, is the set of final states, and for every and , is a function from to . The transition function is extended to a mapping from to in the obvious way, and the set of trees accepted by is .

3 Macro Tree Transducers

Definition 1

A (deterministic) macro tree transducer is a tuple such that is a ranked alphabet of states with , and are ranked alphabets of input and output symbols, respectively, is the initial state, and for every , , and , , the set of rules contains at most one rule of the form

where such that if a node in has its label in , then its first child has its label in . A rule as above is called a -rule and its right-hand side is denoted by .

The translation (often just denoted ) realized by an mtt is a partial function recursively defined as follows. For each state of rank , is the translation of starting in state , i.e., “starting” with a tree where . For instance, for , simply equals . In general, for an input tree , is obtained from by repeatedly replacing a subtree of the form with by the tree . The latter tree will also be written . We define .

By the definition above, all our mtts are deterministic and we refer to them as “macro tree transducers” and denote their class of translations by mtt. An mtt is total if there is exactly one rule of the above form. If each state of an mtt is of rank one, then it is a top-down tree transducer. The class of translations realized by (deterministic) top-down tree transducers is denoted by t. In Lemmas 3 and 5 we make use of nondeterministic top-down tree transducers and mark the corresponding class there by the letter “N”. A transducer is nondeterministic if there are several -rules in . An mtt is monadic if its input and output ranked alphabets are monadic, i.e., only contain symbols of rank and . An mtt is of linear size increase if there is a constant such that for every .

As an example, consider the transducer consisting of the rules in Figure 1.

Figure 1: The macro tree transducer adds reverse Dewey paths to leaves

The alphabets of this transducer are , , and . The mtt translates a binary tree into the same tree, but additionally adds under each leaf the reverse (Dewey) path of the node. For instance, is translated into as can be verified in this computation of

Note that is not of linear size increase. Hence, it is not MSO definable. Since the translation is neither top-down nor monadic, it falls into a class of mtts for which we do not know a procedure to decide equivalence. As an exercise, the reader may wonder whether there is an mtt that is similar to , but outputs Dewey paths below leaves (instead of their reverses). In contrast, consider input trees with only exactly one -leaf (and all other leaves labeled differently). Then an mtt of linear size increase can output under the unique -leaf its reverse Dewey path, i.e., this translation is mso definable. Is it now possible with an mtt to output the non-reversed Dewey path?

One of the most useful properties of mtts is the effective preservation of regular tree languages by their inverses. This is used inside the proofs of several results presented here. For instance, to prove that for every mtt there is an equivalent one which is nondeleting in the parameters (= strict), one can use the above property as follows: given a state of rank and a subset , the language is regular, and hence is the regular set of inputs for which outputs only parameters in the set . Thus, by using regular look-ahead we can determine which parameters are used, and can change the rules to call an appropriate state which is only provided the parameters in which it uses. Regular look-ahead is explained in Section 4.1.

The following result is stated in Theorem 7.4 of [30]. A similar proof as below is given at the end of [24]. A slightly simpler proof, for a slightly larger class, is presented by Perst and Seidl for macro forest transducers [55].

Lemma 1

Let be an mtt with output alphabet and let be a regular tree language (given by a bottom-up tree automaton ). Then is effectively regular. In particular, the domain is effectively regular.

Proof.  Let and . We construct the new automaton . The states of are mappings that associate with each state a mapping . The set consists of all such that . For we define such that for every and , . Here, is the extension of to trees in by the rule for all . Now let with . For we define where for every and , . Now, is the extension of of above to symbols by for every .

3.1 Bounded Balance

Many algorithms for deciding equivalence of transducers are based on the notion of bounded balance. Intuitively, two transducers have bounded balance if the difference of their outputs on any “partial” input is bounded by a constant. For instance, for two d0l systems with , Culik II defines [17] the balance of a string as the difference of the lengths of and . He shows that equivalence is decidable for d0l systems that have bounded difference. In a subsequent article [18], Culik II and Fris show that any two equivalent d0l systems in normal form have bounded difference, thus giving the first solution to the famous d0l equivalence problem.

For a tree transducer a partial input is an input tree which contains exactly on distinguished leaf labeled , and is a fresh symbol not in . More precisely, a partial input is a tree where is in the domain of and is a node of . Since the transducer has no rules for , the computation on the input “blocks” at the -labeled node. Thus, the tree may contain subtrees of the form where is a state of rank . For instance, consider the transducer shown in the left of Figure 2 and consider the partial input tree . Then

It should be clear that if we replace each by so that is a tree with , then we obtain the output of on input tree . For instance, we may pick above; then we obtain which indeed equals .

Consider the trees and for two mtts and . What is the balance of these two trees? There are two natural notions of balance: either we compare the sizes of , or we compare their heights. Given two trees we define their size-balance (for short, s-balance) as and their height-balance (for short, h-balance) as . Two transducers have bounded s-balance (resp. h-balance) if there exists a such that for any partial input the s-balance (resp. h-balance) of and is at most . Obviously, bounded h-balance implies bounded s-balance, but not vice versa.

Let and be equivalent mtts. Do and have bounded size-balance? To see that this is in general not the case, it suffices to consider two simple top-down tree transducers: translates a monadic partial input of the form into the full binary tree of height containing occurrences of the subtree . The transducer translates into the full binary tree of height with occurrences of the subtree .

Figure 2: Equivalent top-down tree transducers and with unbounded size-balance

The rules of and are shown in Figure 2. Clearly, and are of bounded height-balance (with constant ). But, their size-balance is not bounded.

In a similar way it can be seen that equivalent mtts need not have bounded height-balance. This even holds for monadic mtts: Consider the transducers and with rules shown

Figure 3: Equivalent monadic macro tree transducers and with unbounded height-balance

in Figure 3. Their height-balance on input is equal to ; for instance, for the tree we obtain

which are trees of height and , respectively. We may verify that replacing by results in equal trees:

And for we obtain that

Let us now show that equivalent top-down tree transducers have bounded height-balance.

Lemma 2

Equivalent top-down tree transducers effectively have bounded height-balance.

Proof.  Let be equivalent top-down tree transducers with sets of states , respectively. Note that they have the same domain . Let and . Consider the two trees for . Let be a smallest input tree such that . It should be clear that the height of is bounded by some constant . In fact, let be the height of a smallest tree in the set , for any subsets and . Since is in such a set, its height is at most . This bound can be computed because by Lemma 1 the sets and are effectively regular, and regular tree languages are effectively closed under intersection [12]. In fact, it is not difficult to see that we can choose . Hence, there is a constant such that for any appearing in . Clearly we can take , where is the maximal height of the right-hand side of any rule of and . This means that because and the substitutions increase the height of by at most .

If the transducers of Lemma 2 are total, then and is the maximal size of the right-hand side of any rule for an input leaf symbol, i.e., .

Let us consider an example of two equivalent top-down tree transducers and with

Figure 4: Equivalent top-down tree transducers and

output paths of different height. The rules of and are given in Figure 4. Let us consider the input tree . We omit some parentheses in monadic input trees. We obtain

Similarly, for the transducer we obtain

As the reader may verify, if is replaced by the leaf , then indeed the output trees and are the same, i.e., the transducers are equivalent. Let us compare the trees and . On the one hand, the transducer is “ahead” of the transducer in the output branch . It has already produced a -node at that position, while has not (and is in state at that position). On the other hand, is ahead of at two other positions in the output: at the node the transducer has produced an -node already, while at that node is in state , and, at node the transducer has output an -node, while also here is in state .

4 Decidable Equivalence Problems

4.1 Top-Down Tree Transducers

It was shown by Ésik [31] that the bounded height-difference of top-down tree transducers can be used to decide equivalence.

Theorem 1

([31]) Equivalence of top-down tree transducers is decidable.

Proof.  We follow the version of the proof given by Engelfriet [21].

Figure 5: Two equivalent top-down tree transducers

Consider two equivalent top-down tree transducers and . By Lemma 2 they have bounded height-balance by some constant . Consider the trees and . An “overlay” of these two trees is shown in Figure 5 (this is a copy of Figure 10 of [21]). At the node where is in state , the transducer has already produced the tree , i.e., at this node is “ahead” of by the amount . Similarly, at the -labeled node, is ahead of by the amount . Clearly, the height of and is bounded by . Hence, there are only finitely many such trees and . We can construct a top-down tree automaton which in its states keeps track of all such “difference trees” and , while simulating the runs of and . It checks if the outputs are consistent, and rejects if either the outputs are different or if the height of a difference tree is too large. Finally, we check if accepts the language ; this is decidable because is regular by Lemma 1, and equivalence of regular tree languages is decidable (see [12]).

Note that Ésik [31] shows that even for single-valued (i.e., functional) nondeterministic top-down tree transducers, equivalence is decidable. It is open whether or not equivalence is decidable for -valued nondeterministic top-down tree transducers (but believed to be decidable along the same lines as for bottom-up tree transducers [64], cf. the text below Theorem 5). A top-down tree transducer is letter-to-letter if the right-hand side of each rule contains exactly one output symbol in . It was shown by Andre and Bossut [7] that equivalence is decidable for nondeleting nondeterministic letter-to-letter top-down tree transducers. An interesting generalization of Theorem 1 is given by Courcelle and Franchi-Zannettacci [16]. They show that equivalence is decidable for “separated” attribute grammars which are evaluated in two independent phases: first a phase that computes all inherited attributes, followed by a phase that computes all synthesized attributes (top-down tree transducers are the special case of synthesized attributes only).

Top-down Tree Transducers with Regular Look-Ahead. Regular look-ahead means that the transducer (mtt or t) comes with a (complete) deterministic bottom-up automaton (without final states), called the “look-ahead automaton of ”. A rule of the look-ahead transducer is of the form

and is applicable to an input tree only if the look-ahead automaton recognizes in state for all . Given two top-down tree transducers with regular look-ahead , we can transform them into ordinary transducers (without look-ahead) such that the resulting transducers are equivalent if and only if the original ones are. This is done by changing the input alphabet so that for every original input symbol of rank , it now contains the symbols for all possible look-ahead states of and of . Thus, for every , the new input alphabet has -many symbols. It is easy to see that Lemma 2 also holds for transducers with look-ahead, by additionally requiring that the domain of the is intersected with all input trees that represent correct runs of the look-ahead automata. Thus, Theorem 1 also holds for top-down tree transducers with look-ahead.

Theorem 2

Equivalence of top-down tree transducers with regular look-ahead is decidable.

Canonical Normal Form. Consider two equivalent top-down tree transducers and let be their domain. As the example transducers and with rules in Figure 4 show, for a partial input tree , there may be positions in the output trees where is ahead of , and other positions where is ahead of . Such a scenario is also depicted in Figure 5.

We say that is earlier than , if for every and , the tree is a prefix of the tree . A tree is a prefix of a tree if for every with it holds that . The question arises whether for every top-down translation there is an equivalent unique earliest transducer such that for ; we call such a transducer a canonical transducer. The question was answered affirmative by Engelfriet, Maneth, and Seidl [27]. We only state this result for total transducers.

Theorem 3

Let be a total top-down tree transducer. An equivalent canonical transducer can be constructed in polynomial time.

Proof.  The canonical transducers are top-down tree transducers without an initial state, but with an axiom tree . This means that the translation on input tree starts with the tree (instead of for ordinary transducers).

Starting with , we define its axiom . In a first step, an earliest transducer is constructed: if there is a state and an output symbol (of rank ) such that for every input symbol , then is not earliest. Intuitively, the symbol should be produced earlier, at each call of the state . Thus, the construction replaces in all right-hand sides (and in the axiom ) by where the are new states. For every , is defined as the -th subtree of the root of – beware, this right-hand side may have changed due to the replacement above. Finally we remove and its rules. This step is repeated until it cannot be applied anymore. In this case has become earliest and no and exists such that for all input trees of . It should be clear that the earliest step can be carried out in polynomial time. In the second step, equivalent states are merged to obtain the canonical transducer; the corresponding equivalence relation on states is computed using fixed point iteration in cubic time (with respect to the size of ). It is computed in such a way that if , then .

Note that the availability of a canonical (“minimal”) transducer has many advantages. For instance, it makes possible to formulate a Myhill-Nerode like theorem which, in turn, makes possible Gold-style learning of top-down tree transducers (in polynomial time), as shown by Lemay, Maneth, and Niehren [48].

Consider the two transducers and with the rules given in Figure 4. To construct a canonical equivalent transducer for according to Theorem 3, we observe that state of is not earliest: the root equals for the right-hand sides of all -rules. We replace by in the -rule, and introduce the two rules and . We remove and have obtained an earliest transducer. The canonical transducer is constructed by realizing that the states and are equivalent and hence can be merged. The rules of the canonical transducer can thus be given as

To construct the canonical transducer for , we observe that state is not earliest: the root equals in both rules. We thus replace everywhere by where are new states. After this replacement, the current -rule is:

Thus, new -rules are and . The -rules are and . The resulting transducer is earliest. We now compute that and that . We merge these pairs of states and obtain the same transducer (up to renaming of states) as the canonical one of above. Hence, and are equivalent.

As a consequence of Theorem 3 we obtain that equivalence of total top-down tree transducers can be decided in polynomial time.

Theorem 4

Equivalence of total top-down tree transducers can be decided in polynomial time.

The earliest normal form has also certain “disadvantages”. For instance, it does not preserve linearity (or nondeletingness) of the transducer. Consider for the rules and . When making earliest, these rules are removed and a rule such as is replaced by which is non-linear. It is also deleting: .

4.2 Bottom-Up Tree Transducers

As a corollary of Theorem 2 we obtain that also for deterministic bottom-up tree transducers, equivalence is decidable. This follows from the fact that every deterministic bottom-up tree transducer can be transformed into an equivalent deterministic top-down tree transducer with regular look-ahead [20].

Theorem 5

Equivalence of deterministic bottom-up tree transducers is decidable.

Proof.  A deterministic bottom-up tree transducer is a tuple where is the set of final states and contains for every , , and at most one rule of the form where is a tree in . We construct in linear time a deterministic bottom-up tree automaton which for every rule as above has the transition . This automaton serves as the look-ahead automaton of a top-down tree transducer with the unique state . For a rule as above, the transducer has the rule

It should be clear that the resulting top-down tree transducer with look-ahead (which has only the single state ) is equivalent to the given bottom-up tree transducer .

The equivalence problem for bottom-up tree transducers was first solved by Zachar [71]. It was shown by Seidl [61] that equivalence can be decided in polynomial time for single-valued (i.e., functional) nondeterministic bottom-up tree transducers. Note that this also follows from Theorem 8 and the (polynomial time) construction in the proof of Theorem 5. This result was extended to finite-valued nondeterministic bottom-up tree transducers by Seidl [62]. For nondeterministic letter-to-letter bottom-up tree transducers, equivalence was shown decidable by Andre and Bossut [6]; such transducers contain exactly one output symbol in the right-hand side of each rule. They reduce the problem to the equivalence of bottom-up relabelings which was solved by Bozapalidis [10]. For deterministic bottom-up tree transducers the effective existence of a canonical normal form, similar in spirit to the earliest normal form of top-down tree transducers, was shown by Friese, Seidl, and Maneth [37]. They show that this normal form can be constructed in polynomial time, if each state of the given transducer produces either none or infinitely many outputs; hence, equivalence is decidable in polynomial time for such transducers. Friese presents in her PhD thesis [36] a Myhill-Nerode theorem for bottom-up tree transducers.

4.3 Linear Size Increase mtts

It was shown by Engelfriet and Maneth [25] that total deterministic mtts of linear size increase characterize the total deterministic mso definable tree translations. In fact, even any composition of total deterministic mtts, when restricted to linear size increase, is equal to an mso definable translation, as shown by Maneth [49]. The mso definable tree translations are a special instance of the mso definable graph translations, introduced by Courcelle and Engelfriet, see [13]. Decidability of equivalence for deterministic mso graph-to-string translations on a context-free graph language was proved by Engelfriet and Maneth [26]. It implies decidable equivalence also for mso tree translations. We present a proof of the latter here that only uses mtts and avoids going through mso.

The idea of the proof stems from Gurari’s proof [42] of the decidability of equivalence for 2dgsm. In a nutshell: the ranges of all the above translations are Parikh. A language is Parikh if its set of Parikh vectors is equal to the set of Parikh vectors of a regular language. Let be an alphabet. The Parikh vector of a string is the -tuple of natural numbers such that for , equals the number of occurrences of in . For a language that is Parikh, it is decidable whether or not it contains a string with Parikh vector for some natural number . This property is used to prove equivalence as follows. Given two tree-to-string transducers we first change to produce a new end marker at the end of each output string. Then, given the regular domain language of and , and two distinct output letters we construct the Parikh language

Here denotes the -th letter in the string . We now decide if there is an such that , using the fact that is Parikh. If such an exists, then the transducers are not equivalent. If, for all possible , , no such exists, then we know that the transducers are equivalent.

It was shown by Engelfriet, Rozenberg, and Slutzki in Corollary 3.2.7 of [29] that ranges of nondeterministic finite-copying top-down tree transducers with regular look-ahead (for short, s) possess the Parikh property. The nondeterminism of this result is useful for defining the language , because we need to nondeterministically choose and positions and of the output strings. A nondeterministic top-down tree transducer is finite-copying if there is number such that for every and , the number of occurrences of states (more precisely, subtrees such that is a state of ) in the tree is . We denote the class of translations of nondeterministic finite-copying top-down tree transducers by .

For a tree we denote by its yield, i.e., the string of its leaf labels from left to right. For a class of tree translations we denote by the corresponding class of tree-to-yield translations. The tree-to-yield translations of top-down tree transducers can be obtained by top-down tree-to-string transducers which have strings over output symbols and state calls in the right-hand sides of their rules. We repeat the argument given in [29]. By regt we denote the class of regular tree languages, i.e., those languages recognized by (deterministic) finite-state bottom-up tree automata.

Lemma 3

Languages in are Parikh.

Proof.  Let be a transducer and let . A top-down transducer is linear if no appears more than once in any of the right-hand sides of its rules. We construct a linear transducer such that and the string is a permutation of the string , for every . The new transducer computes in its states the state sequences of , i.e., the sequence of states that are translating the current input node. Since is finite-copying, there effectively exists a bound on the length of the state sequences. For a new state with the right-hand side of a rule is obtained by simply concatenating the right-hand sides of the corresponding rules for . It is well known that linear top-down tree transducers preserve regularity and hence the language is in , i.e., it is the yield language of a regular tree language. The latter is obviously a context-free language (cf. Theorem 3.8 of [68]) which is Parikh by Parikh’s theorem [54].

A macro tree transducer is finite-copying if there exist constants and such that

  1. for every input tree with and , the number of occurrences of states in is and

  2. for every state of rank , , and , the number of occurrences of in is .

Recall that an mtt is of linear size increase if there is a constant such that for every . We denote the class of translations realized by mtts of linear size increase by