Loss Tomography from Tree Topologies to General Topologies

# Loss Tomography from Tree Topologies to General Topologies

Weiping Zhu, Ke Deng Weiping Zhu is with University of New South Wales, Australia
###### Abstract

Loss tomography has received considerable attention in recent years and a number of estimators based on maximum likelihood (ML) or Bayesian principles have been proposed. Almost all of the estimators are devoted to the tree topology despite the general topology is more common in practice. There has been few likelihood function devoted to the general topology, not to mention the estimator. To overcome this, two sets of sufficient statistics for the tree and general topologies, respectively, are proposed in this paper. Using the statistics, two likelihood functions, one for a topology, are proposed here and subsequently two likelihood equations for the general topology, one is link-based and the other is path-based, are obtained. In addition, a dependence between subtrees in terms of their estimates is identified for the general topology and a divide-and-conquer strategy is proposed to deal with the dependence, which divides a general network into two types of independent trees. Further, two algorithms, one for a type of the independent trees, are proposed to estimate the loss rates of each type.

Decomposition, General topology, Likelihood Equations, Loss tomography, Tree topology.

## I Introduction

Network characteristics, such as loss rate, delay, available bandwidth, and their distributions, are critical to various network operations and also important to network research, e.g modeling. Because of these, considerable attention has been given to network measurement, in particular to large networks. However, due to various reasons, e.g. security, commercial interests and administrative boundary, some of the characteristics cannot be obtained directly from a large network. To overcome the limitations, network tomography is proposed in [1], where the author suggests to use end-to-end measurement and statistical inference to estimate the characteristics of a large network. The characteristics that have been estimated in this manner include link-level loss rates [2], delay distribution [3], [4], [5], [6], [7], and loss pattern [8]. In this paper, our attention is focused on the loss rate inference, which can be easily extended to estimate loss pattern.

In an active approach, a number of sources attached to a network send probes to the receivers attached to the other side of the network, where the paths from the sources to the receivers cover the network of interest. To create the correlation required for estimation, probes need to be sent in a coordinated manner to the receivers, where a multicast scheme is often used to achieve this. Then, the arrivals, arrival orders and arrival time of the probes are used by an estimator to infer the characteristics of interest. If the probes sent in an experiment are far apart and the traffic is statistically stable, the observations obtained at receivers are considered independent identical distributed () and the likelihood function of the observations is the product of the individual ones. To determine the parameter(s) embedded into a likelihood function, ML or Bayesian principles are often used in the determination. No matter which principle is used in estimation, the sufficient statistics from observations are crucial for an estimator. Unfortunately, this issue has been overlooked in the past that leads some of the estimators proposed so far to use a part of the available information.

Apart from statistics, most of works in loss tomography have been focused on the tree topology although a general topology is more common in practice. The tree topology as named has a single source attached to the root of a multicast tree to send probes to the receivers attached to the leaf nodes of the multicast tree, where all of the receivers have a common view of the probing process. A network of the general topology is different from the tree one that is composed of a number of trees and some parts of the trees are intersected with each other. Thus, the receivers, nodes, and links in an intersected subtree can receive probes from multiple sources and observe a number of probing processes. Because of the difference, an estimator for the general topology, in particular for the intersected areas, must consider the probes sent by different sources. In addition, the estimator must consider the possible dependence between intersected areas and other areas. The differences make the estimators or algorithms developed for the tree topology incapable of the general topology. This paper aims at overcoming the shortage that will address the statistics, the correlation and the dependence raised above. In addition, it provides a number of ML estimators for the general topology.

### I-a Contribution and Paper Organization

As stated, there has been a lack of ML estimators in loss tomography for the tree topology and there has been a lack of both analytical results and estimators for the general topology. To fill the gaps, we in this paper present the results obtained recently that partially solve the problems, which, comparing to [9]-[10], have two-fold contributions to loss tomography, one for a topology. For the tree topology, there are 2 contributions:

1. A set of the minimal sufficient statistics is introduced and used to rewrite the widely used likelihood function into a different format. From the likelihood function, a set of likelihood equations is derived that shows the loss rate of a link depends on the loss rates of its ancestors and descendants, where the polynomial proposed in [9] is a special case of the likelihood equations.

2. The solution space is proved to be concave if a Bernoulli model is assumed to describe the loss behavior of a link. The finding ensures that the estimates obtained by an iterative procedure, e.g. EM, are MLEs.

For the general topology, there are four contributions:

1. The analytical results obtained from the tree topology are extended to the general topology, and a direct expression of the MLE is derived for the loss rate of a link. The direct expression has a similar structure as that of the tree topology.

2. By proving the direct expression leading to an MLE, we also prove the solution space is concave and the estimate obtained by an iterative algorithm, such as the EM, the fixed point and MCMC, is an MLE.

3. Apart from the link-based estimator, a path-based estimator is proposed that not only generalizes the results presented in [9], but also points out the specialty of the general topology. The specialty rests on the demand of consistency in estimation that leads to an order in estimation.

4. A divide-and-conquer strategy is proposed to impose the order raised above that decomposes a general network into two type of independent trees; one is for those without any intersection with others, the other is for those with intersection(s). Further, two estimators are proposed, one for each of the above.

Apart from the above, a practical issue that has been overlooked by previous works is raised in this paper, which is about the number of probes needed to capture the loss rate of a link. In contrast to previous works, the focus here is not only on the number of probes that need to be sent but also on the consistency between the background traffic and the model used by an estimator. Only if the model used to develop the estimator is consistent with that of the background traffic, is the accuracy of the estimator guaranteed. If not, however, there is no guarantee that an estimate obtained from more samples is better than that obtained from less ones. Our simulation study shows that for some background traffic, 5% variance can be easily achieved with a few hundreds of samples. Nevertheless, for some other traffic, 10% variance is hardly achieved even with a few thousands of samples. In this case, increasing the percentage of the probes in traffic can improve estimation accuracy, i.e. sacrificing efficiency for accuracy. Despite this, the estimate obtained may not be the true one since the probes sent can intervene the background traffic.

The rest of the paper is organized as follows. In Section II, we present the related works and the notations used in this paper. In addition, a set of sufficient statistics are introduced in this section. There are four parts in Section III: a) a likelihood function of the tree topology is created from the set of sufficient statistics; b) the likelihood equation from the likelihood function is derived; c) the likelihood equation derived in b) and that presented previously are compared and the difference between them are identified; and d) the statistics introduced in Section II are proved to be the complete minimal sufficient ones. We then extend the results from the tree topology to the general topology in the rest of the paper. Section IV contains the extended notation and the set of sufficient statistics. Then, the link-based and the path-based likelihood equations are presented in Section V. Section VI is devoted to the solution of the likelihood equations proposed in Section V. Statistical properties of the estimators are discussed in Section VII. Section VIII presents simulation results. The last section is devoted to concluding remark.

## Ii Related Works and Problem Formulation

### Ii-a Related Works

Multicast Inference of Network Characters (MINC) is the pioneer of using multicast probes to create correlated observations, where a Bernoulli model is used to model the loss behaviors of a link. Using this model, the authors of [9] derive an MLE for the pass rate of a path connecting the source to a node. The MLE is expressed in a polynomial that is one degree less than the number of descendants of the node [9], [11], [12]. To ease the concern of using numeric method to solve a higher degree polynomial , the authors of [13] propose an explicit estimator. Although the estimator has the same asymptotic variance as that of the MLE to first order, it is not an MLE and there is a noticeable difference between the estimate obtained by the estimator and an MLE if . The two estimators are dedicated to the tree topology. Later, Bu et. al. attempted to extend the strategy to the general topology without success in [2]. The authors then resorted on iterative procedures, i.e. the EM, to approximate the maximum of the likelihood function. In addition, a minimum variance weighted average (MVWA) method is used for comparison with the EM one. The problem of MVWA is the bias of its estimate when the sample size is small [14]. The experimental results presented in [2] confirm this, where the EM outperforms the MVWA when the sample size is small. Nevertheless, the estimates obtained by the EM algorithm could not be verified to be the MLE since there was no proof that the search space is concave. In addition, iterative procedures, e.g. EM algorithm, have their interionic weakness as previously stated in [15]. Considering the unavailability of multicast in some networks, Harfoush et al. and Coates et al. independently proposed the use of the unicast-based multicast to send probes to receivers [16], [17], where Coates et al. also suggested the use of the EM algorithm to estimate link-level loss rates. Apart from those, Rabbat et al. in [18] consider network tomography on general networks and found a general network comprised of multiple sources and multiple receivers can be decomposed into a number of 2 by 2 components. The authors further proposed the use of the generalized likelihood ratio test to identify network topology. To improve the scalability of an estimator, Zhu and Geng propose a bottom up estimator for the tree topology in [19], which later is found to be topology independent [10]. The estimator adopts a step by step approach to estimate the loss rate of a link, one at a time and from bottom up. At each step the estimator uses a formula to compute the loss rate of a link. Despite the effectiveness, scalability, and extensibility to the general topology, the estimate obtained by the estimator is not the MLE as the one proposed in [13] because the statistics used by the two estimators are not sufficient ones.

### Ii-B Notation

Let donate the multicast tree, where is the set of nodes representing routers and switches of a network; is the set of directed links connecting the nodes of , the two nodes connected by a link are called the parent node and the child node of the link, where the parent forwards probes received from its parent to the child; and is the set of parameters to be estimated, one for a link. The multicast tree used to deliver probes to receivers is slightly different from an ordinary one at its root that has only a single descendant. Figure 1 presents an example of the multicast tree.

### Ii-C Internal State and Internal View

Given the loss model of a link, statistical inference is used to determine the unknown parameters of the likelihood function. If the ML principle is applied, the task is frequently described as the maximization of the following log-likelihood function [9]:

 argmaxθ∈ΘL(θ)=argmaxθ∈Θ∑x∈Ωn(x)logP(x;θ) (1)

where is the value space of parameter , is the number of observation , and is the joint probability of observation with parameter . However, it is hard, if not impossible, to derive the likelihood equation from a log-likelihood function in the form of (1).

Instead of using the log-likelihood function as (1), we consider to rewrite it in a different form. Let denote the likelihood function. Under the i.i.d. assumption of probes, we have

 PΩ(θ)=n∏i=1P(X(i),θ)

and under the Bernoulli assumption of losses at a link, we have

 P(Xi,θ)=∏j∈E1(1−θj)∏k∈E2(θk+(1−θk)(1−βk))

for , where denotes the set of links that have ; denotes the set of links whose observation of the probe cannot be confirmed but their parents’ are confirmed, i.e. and ; and is the pass rate of subtree . If we do not distinguish the difference between and and let denote the probability of link for , we have

 PΩ(θ)=n∏i=1∏j∈EPij(θj)

With a finite and , the order of the products is commutable. To swap the order of the two products, we have

 PΩ(θ)=∏j∈En∏i=1Pij(θj). (2)

Having a likelihood function as above, we need to have the state of each link for each probe, which results in two concepts called internal state and internal view. The formal definitions of them are as follows.

#### Ii-C1 Internal State

Given , one is able to obtain and then we can have

 Yji=maxk∈R(i)xjk, j∈{1,..,n}

called the internal state of link for probe . If , probe for sure passes link . Further, considering the values of and , together, we have three possible combinations:

1. , this also means that probe passes link ; or

2. and , this means that probe reaches node and then become uncertain in , i.e. it is not sure whether the probe is lost on link or lost in the subtrees rooted at node ; or

3. , this means the probe becomes uncertain at one of and the uncertainty is transferred from the ancestor to node .

For link , if the first occurs, we need to have in the likelihood function for this probe; if the second one occurs, we need to have in the likelihood function; and if the last one occurs, we need to have 1 in the likelihood function for this probe since . Because the likelihood function is in a product form, we only need to consider the first two in the likelihood function.

#### Ii-C2 Internal View

Accumulating the states of each link in an experiment, and let

 ni(1)=n∑j=1Yji,

we have the number of probes passing link confirmed from , i.e. at least probes pass link in the experiment. In addition, let

 ni(0)=nf(i)(1)−ni(1)

denote the number of probes that become uncertain in . and are called the internal view of link . We will prove that is a set of sufficient statistics later in the paper. Using , we are able to have a likelihood function of in the format of (2).

## Iii New Likelihood Function and Solution

### Iii-a New Likelihood Function

Given and , we then have the following theorem to write the log-likelihood function.

###### Theorem 1.

Given , the log-likelihood function of an experiment on the tree topology can be written as follows:

 L(PΩ(θ)) = ∑i∈E[ni(1)log(1−θi)+ (3) ni(0)log(θi+(1−θi)(1−βi))]

where and are the loss rate of link and the loss rate of subtree , respectively.

###### Proof:

Based on the definition of and , we have

 P(Xj;θ)=∏i∈E[(1−θi)Yji(θi+(1−θi)(1−βi))(Yjf(i)−Yji)].

Thus,

 L(PΩ(θ)) = n∑k=1logP(Xk;θ) = n∑k=1∑i∈E[Yjilog(1−θi)+ (Yjf(i)−Yji)log(θi+(1−θi)(1−βi))] = ∑i∈E[ni(1)log(1−θi)+ ni(0)log(θi+(1−θi)(1−βi))].

The theorem follows. ∎

If the derivative of (3) can be obtained and expressed explicitly, the likelihood equation of the tree topology becomes available.

### Iii-B Likelihood Equations and Solution

Differentiating (3) with respect to (wrt.) each parameter and letting the derivatives be 0, we have a set of likelihood equations as:

 ∂L(PΩ(Θ))∂θi=−ni(1)1−θi+ni(0)βiθi+(1−θi)(1−βi)+ βi∑k∈a(i)nk(0)∏l∈a(i)l≥k[(1−θl)1−βf(l)θl+(1−θl)(1−βl)]θk+(1−θk)(1−βk) =0, i=1,⋅⋅,m.

Reorganizing it, we have

 θi=⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1−ni(1)nβi,i=11−ni(1)nf(i)(1)+imp(f(i))βi,i∈E∖(L∪1)ni(0)+imp(f(i))nf(i)(1)+imp(f(i)),i∈L (4)

where denotes the set of links connecting . It is clear that (4) can be solved by approximation if is a function of , where

 imp(f(i))=∑k∈a(i)nk(0)∗pai(k)∗ξi∗∏l∈a(i)l≥k1−βf(l)ξlξk

is the estimated number of that reaches node before being lost in , where

 ξi = θi+(1−θi)(1−βi) βi = 1−∏j∈diξj pai(k) = ∏l∈a(i)l≥k(1−θl).

Each term in the summation of (LABEL:ancterm) is for an ancestor and represents the portion of that reaches link , where is the pass rate of the path from node to node ; and is the estimate of the number of probes reaching node .

### Iii-C Similarity and Difference from Previous

Let , , the three formulae of (4) become one as:

 θi=1−γ(a(i))βi, (7)

where is the pass rate of , its empirical pass rate is equal to . As stated, if is a function of , can be obtained from 4. In fact, the expression of depends on the observation of . If the observation satisfies

 {(n∑l=1⋀k∈jYlk)>0|∀j,j∈2di∖{∅,{e}}}, (8)

where stands for the set of all subsets of and denotes all single element set, we have,

 βi=1−∏j∈di(1−^γ(a(j))1−θi). (9)

If knowing , we have , and then (9) turns to a polynomial as that derived in [9], i.e.

 Hk(Ak,γ)=1−γkAk−∏j∈dk(1−γjAk)=0 (10)

Note that (10) is not equivalent to (4) since (4) is not restricted to (8). If (8) does not hold, can not be expressed by (9). For instance, if has no intersection with others, we need to have a different rather than (9) in (4). There are various s, depending on observations. For those who are interested in the detail, please refer to [20].

### Iii-D Proof of Sufficient Statistics

There are a number of approaches to prove (4) is the MLE. One of them is to prove the equivalence between (4) and (10) that has been achieved in the previous section. An alternative is to prove the internal view proposed in this paper is the complete minimal sufficient statistics, To prove this, we need to prove that (3) belongs to the exponential families. That is obvious since (3) follows a Bernoulli distribution. Alternatively, according to the following definition presented in [22],

###### Definition 1.

Let be a random sample, governed by the probability mass function . The statistic is sufficient for if the conditional distribution of , given , is independent of .

we have theorem 2 that confirms the internal views are sufficient statistics.

###### Theorem 2.

is a set of sufficient statistics.

###### Proof:

Given the loss process yields the Bernoulli model and , (3) shows the likelihood function is as follows:

 fAk(x)=Ank(1)k(1−Akβk)n−nk(1).

The distribution of in independent Bernoulli trials is a binomial as

 fAk(x)=(nnk(1))Ank(1)k(1−Akβk)n−nk(1).

Then, the conditional distribution can be written as

 fAk|nk(1)(x) = Ank(1)k(1−Akβk)n−nk(1)(n)nk(1)Ank(1)k(1−Akβk)n−nk(1). =1(nnk(1)). (11)

According to definition 1, are sufficient statistics. Since is of the standard exponential family, the statistics are minimal complete sufficient ones. ∎

## Iv Loss Rate Analysis for General Networks

### Iv-a Goals and Background

As stated, using multiple trees to cover a general network makes estimation harder than that in the tree topology. Because of this, no likelihood equation has yet been proposed for the general topology. Without a likelihood equation, it is impossible to know the shape of the search space of an iterative algorithm, e.g. the EM algorithm. Subsequently, there is no guarantee that the estimate obtained by an iterative method is the MLE. To overcome this, two likelihood equations for the general topology are presented in the next section: one is a link-based estimator and the other is a path-based one. The former has a similar structure as (4) that clearly shows the independency of the probes sent by multiple sources on the estimate of a shared link; the latter defies previous approaches by considering the number of probes reaching a node from multiple sources. Subsequently, the path-based likelihood equation is obtained and the solution space of the equation is proved to be strictly concave. Thus, there is a unique solution to the likelihood equation that is the MLE.

### Iv-B Extended Notation

Let denote a general network consisting of trees, where denote the sources attached to the trees. In addition, and denote the nodes and links of . Each node is assigned a unique number, so is a link. and denote the number of nodes and the number of link of the network, respectively. Each of the multicast trees is named after the number assigned to its root link, where denotes the subtree with link as its root link. Further, let denote the receivers attached to the multicast tree rooted at and let denotes the receivers attached to the multicast subtree rooted at link . The largest intersection between two or more trees is called the intersection of the trees and the root of an intersection is called the joint node of the intersection. An intersection is an ordinary tree and named after its root node. Let denote all joint nodes and let denote the sources that send probes to node . Since each node can have more than one parents in a general network, , simply later, is used to denote the parent of node on the way to source and to denote the ancestor that is hops away from node in the path to source . Recursively, we have . Further, let , where , denote the ancestors of node in the path to . Let denote all the ancestors of node .

If denotes the number of probes sent by source , each probe gives rise of an independent realization of the probe process . As the tree topology, if probe sent by source passing link ; otherwise . Further, , , comprises the data set of estimation. As the tree, let denote the state of link obtained from for probe . if the probe passes link , otherwise . Further, let denote the observation obtained by from the probes sent by source .

### Iv-C Sufficient Statistic

It is assumed that each source sends probes independently, and the arrivals of probes at a node or a receiver are also assumed independent. Then, the loss process of a link is considered an i.i.d. process. This also makes the collective impacts of probes sent by the sources to a link i.i.d.. Thus, the likelihood function of an experiment takes either a product form of the individual likelihood functions or a summation form of the individual log-likelihood functions.

Using the internal view proposed in the tree topology, the confirmed passes of link for the probes sent by source can be obtained from that is equal to

 ni(s,1)=ns∑j=1Ysi(j).

 ni(s,0)=np(s,i)(s,1)−ni(s,1)

be the number of probes sent by source that become uncertain in . Further, considering all sources, we have

 ni(1)=∑s∈S(i)ni(s,1)ni(0)=∑s∈S(i)ni(s,0)

be the total number of probes confirmed from that pass link and the total number of probes that turn to uncertain in , respectively. Using the same method as that presented in section III-D, we can prove is a set of sufficient statistics.

## V Maximum Likelihood Estimator

Given the similarity between the statistics presented in the last section and that presented in section II-C, a link-based estimator can be obtained by a similar procedure as that presented in section III-B.

Using the set of sufficient statistics, we can write the log-likelihood function of , and using the same strategy as [23], we have the log-likelihood function of a general network as follows:

 L(θ) = ∑i∈E[ni(1)⋅log(1−θi)+ni(0)⋅logξi] (12)

where . Differentiating wrt. and setting the derivatives to 0, we have a set of equations as follows:

 θi=⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1−ni(S(i),1)niβi, i∈RL,1−ni(S(i),1)nf(i)(S(i),1)+imp(S(i),f(i))βi,i∈SBRL,1−∑j∈S(i)ni(j,1)∑j∈S(i)[nf(i)(j,1)+imp(j,f(i))]βi, i∈SSNL,∑j∈S(i)[ni(j,0)+imp(j,f(i))]∑j∈S(i)[nf(i)(j,1)+imp(j,f(i))], i∈AOL, (13)

where RL denotes root links, SBRL denotes the links that are not root link but only receive probes from a single source, SSNL denotes the links in a intersection but leaf ones, and AOL denotes all others, i.e. leaf links. As (4), denotes the impact of , , on the loss rate of link .

 imp(j,i)= ∑k∈a(j,i)nk(j,0)⋅pai(k)⋅ξi∗∏l∈a(i)l≥k∏q∈Cl∖lξqθk+(1−θk)∏q∈Ckξq

where

 pai(k)=∏l∈a(j,i)l≥k(1−θl).

In fact, the second equation of (13) can be merged into the third one since each link in SBRL has its . To distinguish SBRL from SSNL here is for later to divide a general network into a number of independent trees at .

The estimate obtained from (13) can be proved to be the MLE by using the same procedure as that presented in section III-D. In addition, almost all of the results obtained in the tree topology can be extended to the general topology. For instance, let be the pass rate of , the empirical pass rate of is

 ^γi(a(i))=∑j∈S(i)ni(j,1)∑j∈S(i)[nf(i)(j,1)+imp(j,f(i))]. (14)

In addition, if can be expressed as a function of , the likelihood equation is expressed by a polynomial as (9). However, there is a lack of methods to express for the general topology because there can have a number of sources sending probes to subtree .

### V-B Insight and Remark

Despite it is unclear how to express , the similarity between the four equations presented in (13) and the three in (4) provides such a hope that a path-based estimator as (10) is available to the general topology. To make this happen, equations (4), (13) and (10) are examined and an insight is emerged that is presented in the following remark:

Remark: regardless of the topology and the number of sources, the MLE of the loss rate of a link can be obtained if we know:

1. the total number of probes reaching the parent node of the link, e.g. , , or of (13);

2. the total number of probes reaching the receivers via the link, e.g. , or of (13); and,

3. the pass rate of the subtree rooted at the child node of the link, e.g. of (13).

Note that the link stated in the remark can be a path consisting of a number of links serially connected. For the tree topology, there is only one source, 1) and 2) can be suppressed to the pass rate of the path connecting the source node to an internal node. (10) as an example takes advantage of this to express , the loss rate of subtree , in two different functions of : a) the product of the loss rates of the multicast subtrees rooted at node ; and b) since .

### V-C Path-based Estimator

As stated, there has been a lack of likelihood function proposed for the general topology regardless of whether it is a link-based or a path-based. This is partially due to the lack of sufficient statistics of a link or a path from observations. With the help of the minimal sufficient statistics , a link-based estimator has been presented in the paper. Further, using the remark presented above and the statistics, we are able to have a path-based log-likelihood function for the general topology. Among the three factors listed in the remark, two of them are available, i.e.:

• the total number of probes sent by a source; and

• the total number of probes reaching and know the source of each arrived probe.

Given two of the three, a likelihood equation with the third factor as the variable can be derived. Let

 A(s,i)=∏j∈a(s,i)(1−θj), i∈V, and s∈S(i) (15)

be the pass rate of the path from source to node . It is easy to prove that (15) is the bijection, , from to , where is the support space of and is the support space of . To have a polynomial likelihood equation, we need to express by .

In contrast to the tree topology, there are intersections in the general topology. For intersection , must be consistent for all paths that connect a sources to node . To prove that is a necessary condition for the MLE, we have the following theorem.

###### Theorem 3.

The likelihood equation describing the pass rate of a path connecting source to node , can be expressed as

 A(s,i)=γi(s)βi, s∈S(i). (16)

As the tree topology, the empirical probability of can be obtained by .

###### Proof:

Using the sufficient statistics, we can write the path-based likelihood function as

 P(A(s,i)) = ∏i∈V∖S∏s∈S(i)[A(s,i)ni(s,1)(1−A(s,i)+ A(s,i)(1−βi))(ns(1)−ni(s,1))] = ∏i∈V∖S∏s∈S(i)[A(s,i)ni(s,1)× (1−A(s,i)βi)(ns(1)−ni(s,1))].

The log-likelihood function takes the following form:

 L(P(A(s,i))=∑i∈V∖S∑s∈S(i)[ni(s,1)logA(s,i) +(ns(1)−ni(s,1))log(1−A(s,i)βi)].

Differentiating the log-likelihood function wrt and letting the derivative be 0, we have

 A(s,i)=γi(s)βi, s∈S(i).

(16) is almost identical to that obtained for the tree topology except that the is consistent to all . We call this the consistent condition since it ensures the estimates obtained for consistent with each other. The condition is also implicitly presented in (13) that indicates there is an order in the estimation of a general network, where should be estimated before , .

To have the MLE satisfying the consistent condition, we have the following two theorems.

###### Theorem 4.

Let be the pass rate of the path connecting to in a network of the general topology. There is a polynomial, , as follows to express the estimate of .

 H(A(k,i),S(i))=1−^γi(k)A(k,i)− ∏j∈di(1−^γi(k)∑s∈S(i)nj(s,1)A(k,i)⋅∑s∈S(i)ni(s,1))=0 (17)

where

 ^γi(k)=ni(k,1)nk
###### Proof:

Assume source sending probes to node . Based on the first equation of (13), we have

 A(k,i)βi=ni(k,1)nk=^γi(k), k∈S(i)

Since , there are more than one sources sending probes to node , the pass rates of two sources, and , to node are correlated that can be expressed as

 A(s,i)=A(k,i)^γi(s)^γi(k), s,k∈S(i) (18)

Let be the total number of probes reaching node , we have

 n∗i(1)=A(k,i)^γi(k)∑s∈S(i)ns^γi(s)

Then, we have two ways to express . If the observations of the subtrees rooted at node satisfy (8), we have

 1−βi=∏j∈di[1−∑s∈S(i)nj(s,1)n∗i(1)] (19)

and using (16), we have

 1−βi = 1−^γi(k)A(k,i)

Connecting the two, we have

 1−^γi(k)A(k,i) = ∏j∈di(1−^γi(k)⋅∑s∈S(i)nj(s,1)A(k,i)⋅∑s∈S(i)ns^γi(s))

Except for , all others, e.g. , are either known or obtainable from observations. Thus, the above equation is a polynomial of . Alternatively, using (19) to replace from (13), we have the same result. ∎

(17) generalizes (10) that considers various paths ending at a common node, including those having . For node having , (17) degrades to (10). For , if ,

 ∑s∈S(i)nj(s,1)∑s∈S(i)ni(s,1) (20)

is the estimate of the pass rate of the link connecting node to node , where the numerator is the sum of the probes sent by that reach and the denominator is the sum of the probes sent by that reach . Both are obtainable from . The estimate is built on the arithmetic mean that considers the contribution of all sources sending probes to the link.

Solving (17), we have . Then, we can have from (18) and

 ^βi=nsˆA(s,i),s∈S.

Using Lemma 1 in [9], we are able to prove there is only one solution to (17) in , To prove the estimate obtained from (17) is the MLE, we resort on a well known theorem for the MLE of a likelihood function yielding the exponential family.

###### Theorem 5.

If a likelihood function belongs to a standard exponential family with as the natural parameters, we have the following results:

1. the likelihood equation has at most one solution ;

2. if exists, is the MLE.

###### Proof:

This theorem can be found from a classic book focusing on exponential families, such as [24]. The likelihood function presented in (V-C) belongs to the exponential family, where is the natural parameters. Then, the estimate obtained from (17) is unique in its support space and it is the MLE of . ∎

## Vi Solutions

Theorem 4 is not only valid to estimate the pass rates of a path connecting a source to a joint node, but also valid to estimate the pass rate of a path connecting a source to a node in an intersection. Then, the loss rate of a link in an intersection can be obtained easily since (15) is the bijection function from to , and then , from to , is as follows

 1−θi=∑s∈S(i)nsA(s,i)∑s∈S(i)nsA(s,f(i)). (21)

Unfortunately, cannot be applied to the links falling into SBRL and having at least a descendant connecting an intersection, i.e. the ancestors of an intersection, because of the consistent condition.

### Vi-a Consistency and Decomposition

To estimate the loss rate of link , called ancestor links, a divide-and-conquer strategy is proposed here that is not only an algorithm to estimate the loss rates of a general network, but also a necessary measure to ensure the consistency established in theorem 3. Instead of directly finding a likelihood equation for the ancestor links from the statistics proposed in Section IV-C, a general network covered by a number of trees is divided into a number of independent ones before being estimated. The strategy is built on equation (17) that shows the parameter estimation in a hierarchical structure with probes flowing in one direction can be divided into a number of sections as the d-separation stated in [25].

Given the number of probes reaching a node, the descendants of the node not only become independent from each other, but also become independent from their ancestors. Since each of the independent parts is a tree, the estimators proposed for the tree topology can be used to estimate the loss rates of the independent parts. Then, the immediate question is which nodes should be selected as the decomposing points that can minimize the cost of estimation. Two criteria are proposed here for the optimal decomposition, which should:

1. ensure that all the nodes and links in an independent tree receive probes from the same set of sources. Then the sources can be considered a virtual source sending probes to the receivers of the independent tree; and

2. minimize the number of independent trees created after decomposition since the computation cost is proportional to the number of the independent trees.

The following theorem is presented for the optimal strategy to select decomposing points.

###### Theorem 6.

Given a general network covered by multiple trees, the optimal strategy to decompose the overlapped multiple trees into a number of independent trees is to use the nodes of as decomposing points.

###### Proof:

Obviously the nodes or links of an intersection can only receive probes from the same set of sources. The minimum number of independent trees is proved by contradiction. Firstly, we assume there are two strategies, say A and B. Strategy A decomposes a general network into a number of independent subtrees and there is at least one of the decomposing points that is not a joint point and the total number of the independent subtrees obtained is . On the other hand, strategy B decomposes the network at the joint points only and the total number of independent subtrees is . If strategy A is better than strategy B, we should have . However, this is impossible: given the fact that all of the subtrees rooted at a joint point can only become independent from each other if we know the state of the joint point according to d-separation. Then, we have , which contradicts to the assumption. Then, the theorem follows. ∎

To start the the divide-and-conquer strategy, we need to have a method to estimate . If has more than 6 descendants, there is not a closed form solution according to Galois theory. Then, an approximation method, such as (17) or the EM algorithm, is needed to find the solution. Apart from them, an iterative procedure based on fixed point theory can also accomplish this. The iterative procedure uses:

 (1−βi)(q+1)= ∏j∈di[∑k∈S(j)nkj(0)∑k∈S(i)nki(1)+∑k∈S(j)nkj(1)∑k∈S(i)nki(1)⋅(1−βi)(q)]

to compute [26]. According to Lemma 1 in [9], there is only one in satisfying (17). Then, the iterative procedure is assured to coverage at and the number of iterations depends on the initial value of , . If the initial value is selected properly, i.e. close to the fixed point, only a few of iterations are needed. A number of methods can be used to obtain the initial value, one of them is to use a weighted average obtained from the estimates of the trees involving in subtree ; another is to use the top down algorithm proposed in [10]. In our experience for a network as Figure 2, the procedure needs 5.25 iterations on average to converge to the fixed point when is set to . Alternatively, Newton Raphson algorithm can be applied to (LABEL:munique) to find the that can converge to quickly as well. Given , we have and .

As stated, the independent trees obtained from decomposition can be divided into two groups on the basis of ingress or egress of the links connecting or connected to the decomposing points. The group consisting of the trees that have at least an ingress link to a joint node is called the ancestor group, all others that have an egress link from a joint node is called the descendant group. For the descendant group, given , the maximum likelihood estimators developed for the tree topology can be applied to estimate the loss rates of the independent trees.

Although the trees in the ancestor group as those in the descendant group are independent from each other given , they cannot be estimated directly by an estimator developed for the tree topology. As previously stated, the pass rate of a path ending at a node that is an ancestor of a joint node depends on the estimate obtained for the intersection. Then, the pass rate of the path connecting source to node should be . If is the last link of the path, i.e the link ending at node , the statistic of link , , is equal to

 ^nrj(1)=nrj(1)β∗i,j∈p(i). (23)

where denotes the parents of node . To ensure the consistency of estimation with the pass rates from to