Loss Rate Inference in Multi-Sources and Multicast-Based General Topologies
Loss tomography has received considerable attention in recent years and a number of estimators have been proposed. Unfortunately, almost all of them are devoted to the tree topology despite the general topology is more common in practice. In addition, most of the works presented in the literature rely on iterative approximation to search for the maximum of a likelihood function formed from observations, which have been known neither scalable nor efficient. In contrast to the tree topology, there is few paper dedicated to the general topology because of the lack of understanding the impacts created by the probes sent by different sources. We in this paper present the analytical results obtained recently for the general topology that show the correlation created by the probes sent by multiple sources to a node located in an intersection of multiple trees. The correlation is expressed by a set of polynomials of the pass rates of the paths connecting the sources to the node. In addition to the expression, a closed form solution is proposed to obtain the MLE of the pass rates of the paths connecting the sources to the node. Then, two strategies are proposed to estimate the loss rate of a link for the general topology: one is path-based and the other is link-based, depending on whether we need to obtain the pass rate of a path first. The two strategies are compared in the context of the general topology that shows each has its advantages and the link-based one is more general. Apart from proving the estimates obtained are the MLEs, we prove the estimator presented here has the optimal asymptotic property.
Network tomography was proposed in  that suggests the use of end-to-end measurement and statistical inference to estimate network characteristics of a large network. Since then, a considerable amount of works has been done in this area and the works reported so far have covered almost all of the characteristics that were previously obtained by direct measurement for a small network. The works can be classified into loss tomography , delay tomography , , , , , loss pattern tomography , source-destination traffic matrix  and shared congestion flows . Despite such a overwhelming enthusiasm and a wealth of publications, most of the works in this area are at the preliminary stage, and most of the estimators or algorithms proposed for estimations aim at proof of concept while the efficiency and scalability of them are very much overlooked. For instance, loss tomography has been studied for a decade and a number of algorithms have been published to estimate link-level loss rates from end-to-end measurement. Unfortunately, almost all of the algorithms are limited into the tree topology and most of them resort on iterative approximation. To overcome this problem, we in this paper propose a number of maximum likelihood estimators to estimate link-level loss rate of a general network, all of them are in closed form.
The topologies used to connect sources to receivers can be divided into two classes: tree and general, which leads to two types of estimators, one for a topology. As stated, the tree one has received much more attention than its counterpart. Almost all of the estimators proposed previously target the tree topology because of the simplicity of the correlation embedded into the observations of the receivers attached to a tree topology. Despite the large amount of works, there are only a few of them providing analytical solutions as   . That leads to the use of iterative procedures to approximate the root of a likelihood function although knowing such a procedure is neither efficient nor scalable . In contrast to the tree topology, there has been little systematically study in both theory and experiments for the general topology despite the general topology is more common than the tree one in practice. As a result, there are only a handful of estimators that have been proposed for the general topology. To improve this, we in this paper provide our findings about the impact of multiple sources on loss estimation that lead to a number of maximum likelihood estimators for the general topology, including an analytical solution, and two divide-and-conquer ones. All of them completely eliminate iterative procedure involved in the estimation. Compared with the estimators proposed previously for the general topology ,, the estimators proposed here not only perform better, but also extend our understanding of loss tomography from the tree topology to the general one. In fact, the results presented in this paper generalize those obtained from the tree topology since the tree is a special case of the general topology. Further, the results may be extended into other tomographies, such as delay, in the general topology.
Multicast Inference of Network Characters (MINC) is the pioneer of using multicast probes to create correlated observations at the receivers of the tree topology, where a Bernoulli model is used to model the losses occurred on a link. Using this model, the authors of  derive a direct expression of the pass rate of a path connecting the source to an internal node. The expression is in the form of a polynomial , , . To ease the concern of unsalability of using numeric method to solve higher degree polynomials , Duffield et. al. propose an explicit estimator in  for the tree topology. Although the estimator has the same asymptotic variance as the MLE to first order, it is not an MLE. When , the estimate obtained by the estimator can be very different from an MLE. In contrast,  proposed an iterative approach to search for the maximum of the general topology. Later, Bu et. al. attempted to use the same model as the one used in  for the general topology, but could not derive a direct expression as its predecessor . Without analytic results, the authors of  then resort on iterative procedures, e.g. the EM, to approximate the maximum of the likelihood function. In addition, a minimum variance weighted average (MVWA) estimator is proposed as an alternative in comparison with the EM one. The MVWA in fact is a variant of the minimum chi-square estimator . The experimental results confirm this problem , where the EM outperforms the MVWA when the sample size is small. Nevertheless, there is no proof that the solution obtained by the EM algorithm is the MLE since the iterative procedures, e.g. EM algorithm, can converge to a local maximum unless the solution space is proved to be strictly concave.
Considering the unavailability of multicast in some networks, Harfoush et al. and Coates et al. independently proposed the use of the unicast-based multicast to discover link-level characteristics , , where Coates et al. also suggested the use of the EM algorithm to estimate link-level loss rates that may not be scalable as previously stated.
To improve the scalability of estimation, Zhu and Geng propose a bottom up algorithm in . The algorithm is independent of topology, i.e. it can be applied to the general topology as well . The estimator, rather than estimating the loss rates of all links together as an iterative procedure, adopts a step by step approach to estimate the loss rate of a link one at a time and from bottom up. At each step the algorithm uses a closed form formula to estimate the loss rate of a link. Despite the effectiveness, scalability and extensibility to the general topology, the estimator is not the MLE as the one presented in  because the statistics used in estimation are not sufficient. Recently, the authors of  use a divide-and-conquer approach to tackle the general topology, which divides a general network into a number of independent trees at the roots of intersections. It uses a fixed point algorithm to estimate the number of probes reaching the roots of intersections. The fixed point algorithm is based on an iterative procedure to achieve its goal, although experimental studies show it has a fast convergence rate since the solution space is strictly concave.
1.2Contribution and Paper Organization
As stated, there has been a lack of understanding the impact of probes sent from multiple sources to a link or a node in the general topology. In addition, iterative procedures are still used in all estimators proposed for the general topology . We are aiming to improve the unsatisfied situation and present the latest findings in this paper that contribute to loss tomography in the following respects:
we derive a set of polynomials to express the pass rate of the paths connecting a source to a node in the general topology. The derivation shows the pass rate of a path in the general topology is no longer independent from each other in contrast to that in a tree topology. This finding generalizes the results of  and is applicable to various topologies.
we derive a closed form MLE solution to the set of the polynomials of related paths. The solution shows that the MLE considers the probes sent from all related sources that generalizes the result obtained from the tree topology.
we present a divide-and-conquer strategy to break up a general network into a number of independent trees to avoid the necessity of solving a large number of polynomials. Two algorithms are proposed subsequently to the independent trees, one for the descendants of a decomposing point, the other for the ancestors of the decomposing point. The algorithms previously developed for the tree topology can be applied to the former with little help while the latter requires its own algorithm.
we propose two estimators: link-based and path-based, for the ancestors of a decomposing point. The result shows each of the estimators has its own advantages and disadvantages. The link-based one is more general than the path-based one.
we clarify some misconceptions about using multiple sources, in particular to those focusing on the complexity created by multiple sources in estimation but ignoring the benefits brought by the multiple sources on data consistency, variance reduction, and convergence rate.
The rest of the paper is organized as follows. In Section 2 we present the essential background, including the notations and statistics used in this paper. In Section 3, we present our latest findings and contributions to the loss inference of the general topology. Section 4 provides the detail description of the three estimators, plus some asymptotical properties of the estimator proposed in this paper. Section 5 is devoted to the statistical properties of the estimators presented in this paper. Simulations are presented in Section 6 that shows the estimates of overlapped links are converged quicker than other links. The last section is devoted to concluding remark.
2Notations and Statistics
A general topology differs from a tree one in a number of aspects. The most important one is the use of multiple trees to cover a general network, which leads some parts of the network covered more than once by the trees. Those parts are called intersection areas, or simply intersections. If a source is assigned to the root of a tree to send probes to the receivers attached to the tree, the receivers attached to an intersection can observe the probes sent from multiple sources. Then, the question is whether we need to consider the probes sent by different sources in estimation. If not, the estimate obtained from a source can be different from that obtained from another source and the difference can be noticeable unless the number of probes sent by each source goes to infinite that is not appropriate in practice. If yes, we must determine the contribution of each source on the estimate. In addition, the 1-to-1 mapping between nodes and links held in the tree topology no longer exists in the general topology since some nodes can have more than one parents.
A multicast tree or subtree used to connect the source to receivers is slightly different from an ordinary tree at the root that has only a single child. The link connecting the root node to the child is called the root link of the tree. Let denote a general network consisting of trees, where denote the sources attached to the trees. In addition, and denote the nodes and links of . Each node is assigned a unique number, so is a link. and denote the number of nodes and the number of link of the network, respectively. The two nodes connected by a link are called the parent node and the child node of the link, where the parent receives probes from its parent and forward them to its children. In addition, denotes the parent link of link toward source . The nodes that have no parent(s) are the roots of the multicast trees. Each of the multicast trees and the subtrees is named after the number assigned to its root link, where denotes the subtree with link as its root link. Further, let denote the receivers attached to the multicast tree rooted at and let denotes the receivers attached to the multicast subtree rooted at link . The largest intersection between two trees is called the intersection of the trees and the root of an intersection is called the joint node of the intersection. Note that an intersection is an ordinary tree but a multicast one. We use to denote all joint nodes and use to refer the sources that send probes to node . If , simply later, is used to denote the parent of node on the way to source and to denote the ancestor that is hops away from node on the path to source , we have . Further, let , where , denote the ancestors of node to . Let denote all the ancestors of node . In addition, except for leaf nodes each node has a number of children, where denotes the children of node and denotes the number of children of node .
If denotes the number of probes sent by source , each probe gives rise of an independent realization of the probe process , if probe passing link ; otherwise . The observation of comprise the data set for inference. In addition, let denote the state of link obtained by examining for probe to see whether it reaches at least one of . if the probe reaches, otherwise .
We assume that each source sends probes independently, and the observations of the arrivals of probes at nodes and receivers are also assumed independent. Then, the loss process of a link is considered an independent identical distributed (i.i.d.) process. This also makes the collective impacts of probes sent by the sources to a link i.i.d.. Further, the likelihood function of an experiment takes either a product form of the individual likelihood function or a summation form of the individual log-likelihood function. Therefore, we will consider a single source first in the following discussion, and then add a number of the single ones together to form a multiple-sources general log-likelihood function.
If a probe sent by source reaches , the probe must pass link . Then, by examining the observations of , we are able to confirm some of the probes sent by the source passing link . Based on the confirmed passes of each link, we have a set of sufficient statistics to write the likelihood function of . To obtain the confirmed passes of link , we must examine each observation received by . For each , let
If probe reaches at least one of , that also implies it reaching node . Further, considering and , we can confirm some of the probes sent by passing link . If
, it is confirmed that probe passes link ; or
and , it is confirmed that probe passes link but uncertain whether the probe passes link ; or
, it is obvious since uncertainty is transferable from a node to its descendants.
If the first scenario occurs, we need to have in the likelihood function. If the second one occurs, we need to have in the likelihood function, where is the loss rate of subtree that considers all possible combinations that lead to since uncertainty is transferable. If the last one occurs, we do not need adding any thing into the likelihood function because its contribution to the likelihood function is considered by one of its ancestors that has the second scenarios for this probe.
Considering all probes sent by source , and let
denote the number of the confirmed passes of the probes sent by source on link . In addition, let
be the number of probes sent by source that turn to uncertain in . Further, considering all sources, we have
be the total number of probes confirmed from observations that pass link and the total number of probes that turn to uncertain in . consist of a set of sufficient statistics .
3 Maximum Likelihood Estimator
The statistics presented in the previous section are extended from those presented in . In , we also attempted to tackle the general topology, but did not derive a closed from solution. Instead, we came up with a strategy to decompose a general topology into a number of independent subtrees. Nevertheless, that experience inspires us to further investigate the correlation in , where an insight is discovered from this investigation. For completeness, the link-based estimator is presented first. Then, we present the insight which finally leads to a path-based estimator.
Given the set of sufficient statistics, we can write the log-likelihood function of . Using the same strategy as , the log-likelihood function of a general network is presented as follows:
where . Differentiating with respect to (wrt.) and setting the derivatives to 0, we have a set of equations. Solving them, we have the followings:
where RL denotes root links, SBRL denotes the links that are not root link but only receive probes from a single source, SSNL denotes the links in intersections but not leave, and AOL denotes all others, i.e. leaf links. Differentiating wrt. and setting the derivatives to 0. Then, we have a set of equations as follows, one for a group of links. Let and denotes the impact of , , on the loss rate of link .
Despite the estimate obtained by (Equation 2) has been proved to be the MLE in , we have not found an analytical solution as the one presented in  for the general topology since (Equation 2) is a transcendental equation in the form of , where is the set of the parameters to be estimated, one for a link in . To overcome this, we proposed a fixed-point algorithm to approximate the number of probes reaching a joint node in , and then a general network can be decomposed into a number of independent trees in order to use the estimator developed for the tree topology.
3.2Insight and Remark
Despite there is no a closed form solution to (Equation 2), the simple and uniform appearance of the four equations provides such an insight of loss tomography that can be expressed in the following remark:
Remark: regardless of the topology and the number of sources, the MLE of the loss rate of a link can be obtained if we know:
the total number of probes reaching the parent node of the link, e.g. , , or of (Equation 2);
the total number of probes reaching the receivers via the link, e.g. , or of (Equation 2); and,
the pass rate of the subtree rooted at the child node of the link, e.g. of (Equation 2).
Note that the link stated in the remark can be a path consisting of a set of links serially connected. For the tree topology, there is only one source, 1) and 2) can be suppressed to the pass rate of the path connecting the source node to an internal node. Then, a polynomial formula as the one presented in , i.e.
is obtained that expresses MLE of the path from the source to node . In fact, (Equation 3) expresses , the loss rate of subtree , in two different forms of function of : the product of the loss rates of the multicast subtrees rooted at node and . Then, how to solve a high degree polynomial becomes the key obstacle blocking a closed form solution to the tree topology. The problem has been solved by  where an equivalent transformation is proposed that merges multiple descendants into two virtual ones.
Using the same strategy as  and the set of sufficient statistics, (Equation 2) shows the impact of multiple sources on a link, regardless the link to be estimated is in an intersection or not. The result, on one hand, shows that as the tree topology, the cumulative impact is built from to link ; on the other hand, it shows that the impact comes from multiple sources, i.e. , in a general topology that makes a closed form estimator almost impossible. This is because (Equation 2) is a multivariate polynomial relating to all links in a general topology. To find a solution, we need to to solve a set of multivariate polynomials, one for a link. That is intractable at least at this moment except for using approximation.
There has been no path-based likelihood function for the general topology as far as we know. This is partially due to the lack of sufficient statistics of a link or a path from observations. With the help of the minimal sufficient statistics proposed in Section 2, we are able to write a path-based likelihood function here, and then derive a set of likelihood equations, and finally have a path-based estimator that provides MLE for the general topology.
Because of the intractability of the link-based estimator, our attention is switched to a path-based estimator although the number of paths is the same as the number of links. The path-based estimator differs from the link-based one, based on the remark, in the following aspacts:
we know the total number of probes sent by a source; and
we know the total number of probes reaching and know the sources of the arrived probes.
As the tree topology, is unknown and can be explicitly expressed by the available information. The difference between the estimator of the tree topology and that of the general one is at the intersections of trees in the general topology, which makes the trees intersected dependent. Then, we need to determine the correlation between the trees and the number of paths involved in an intersection.
be the pass rate of the path from source to node . It is easy to prove that (Equation 4) is the bijection, , from to , where is the support space of and is the support space of . The statistics, , are still the sufficient statistics for the path-based likelihood function since the number of probes sent by and observed by is the confirmed number of probes passing the path connecting to . Then, we have the following theorem for the MLE of a path in the general topology.
( ?) is identical to that obtained for the tree topology except for having a condition for . We call it the consistent condition because it states the necessity of a consistent for those trees intersected at node . This condition states that an MLE estimator must consider all probes sent from to . Compared with the link-based estimator, the path-based estimator only considers the intersected paths during estimation that substantially reduces the correlation involved in estimation. To obtain the MLE from the set of equations defined by ( ?), we need:
to obtain a consistent for all and to express in two different ways as the functions of ,
to link the two expressions by an equation, and to prove there is a unique solution to the equation.
The following two theorems address them, respectively.
is the estimate of the pass rate of the link connecting node to node , where the numerator is the sum of the probes sent by that reach while the denominator is the sum of the probes sent by that reach . Both are obtainable from . The estimate is built on the arithmetic mean that considers the contribution of all sources sending probes to the link.
Solving ( ?), we have . Then, we can have and
To prove the uniqueness of ( ?), we have the following Lemma
Based on Lemma ?, we have
To prove the estimate obtained by ( ?) is the MLE, we resort on a well known theorem for the MLE of a likelihood function yielding the exponential family.
Given Theorem ?, there are a number of ways to complete the estimation. One of them is to repeatedly use Theorem ? to obtain the pass rates of all paths, and then use , from path rate to link rate, to have the loss rate of each link, i.e.
The advantages of this is its uniformity, while the disadvantage could be the requirement of solving a large number of polynomials, some of them may be high degree polynomials.
4.1Closed Form Solution
It is easy to see the degree of ( ?) is one less than the number of descendants connected to node and we also know that there is no closed form solution for a polynomial that has a degree of 5 or greater. Then, the key to have a closed form solution to ( ?) is to replace a multi-descendant node, i.e. a node with more than 5 descendants, with a tree. In , such a replacement is proposed for the tree topology and called a statistical equivalent replacement since
it is equal to use a number of serially connected links to replace an egress link of a multi-descendant node, and
the statistics of the links connecting or connected to the node remain the same as that before the replacement.
. Comparing (Equation 3) with ( ?), one can notice that (Equation 3) is a special case of ( ?). Then, the replacement proposed for the tree topology must be a special case of the general topology, where the key rests on how to determine the statistics of the nodes located in the tree used to replace a multi-descendant node. The following theorem extends the result presented in  and provides the answer to the above question.
Given Theorem ?, node can be replaced by a binary tree when we estimate the pass rate of the paths connecting to node . Then, equation ( ?) becomes a linear equation of and a closed form solution follows.
Apart from the closed form solution as above, another two closed form solutions based on divide-and-conquer strategies are proposed as alternatives. This is because equation ( ?) shows such a fact that the parameter estimation in a hierarchical structure with the probes flowing in one direction can be carried out by dividing the structure into a few segments as the d-separation presented in . For the general topology, given the number of probes reaching a node, the descendants of the node become independent from each other and can be estimated independently. Then, the immediate issue is which nodes should be selected as the decomposing points, where the criterion used to evaluate the strategies is based on the number of independent trees created after a decomposition, the smaller the better since the computation cost of estimation is proportional to the number of independent trees. The following theorem is presented for the optimal strategy to select decomposing points.
Given and obtained by ( ?), a network of the general topology can be divided into a number of independent trees. The independent trees can be divided into two groups on the basis of the position of a tree relative to the decomposing point that separates it from others. We call the two groups: the ancestor group and the descendant one. For the descendant group, given , the subtrees rooted at node become independent from their ancestors and from each other. Then, the loss rates of the links in the subtrees can be estimated by assuming a source attached to the root that sends probes to the receivers. Further, the maximum likelihood estimators developed for the tree topology, such as the top down algorithm , can be applied to each of the subtrees to estimate the loss rate of each link of the subtree.
For the ancestor group, the subtrees are also independent from each other given . However, given but the observations at node , we need to either use the first equation of (Equation 2) or the property of to estimate the loss rates. The former leads to a modified estimator of (Equation 3). For , we use to replace in the modified path-based estimator that has the following form:
Here, we also need to solve a polynomial to have . Once having , we are able to estimate , then we move a level up and use (Equation 6) to estimate . This process continues from bottom up until reaching . If the tree being estimated has more than one intersections, at the common ancestors of the intersections, the RHS of (Equation 6) will have a number of the left-most terms, one for an intersection plus the product term for those subtrees that do not link to any intersection. If the total number of overlapped trees plus the number of independent subtrees is over 5, there is no direct solution to (Equation 6). Compared to the approach of using ( ?), this approach does not have significant advantage since it also needs to solve a polynomial for a link. This shows the path-based estimators are similar to each other in terms of performance regardless of the topologies.
The question that we are facing is whether there is a more effective approach than the path-based estimators and whether the link-based estimators are better than the path-based one in this circumstance. The answer is affirmative and the solution is reported in [?].
Data consistency was raised in  for three extreme circumstances in observation for the tree topology, which are , , and . The three extreme circumstances make the estimation of an impossible task. To continue the estimation without , three corresponding procedures were proposed to skip the impossible task. For the general topology, the three circumstances still affect the estimation of the links, the paths, and the subtrees that only observe the probes sent by a single source, and the procedures proposed previously are still applicable to the occurrences of the problems. However, due the the use of multi-sources sending probes in a general topology, the situation is different from the tree topology. We here specify the data consistent issues for the paths and links that can observe probes sent by more than one sources:
if , based on the location of link , we need to consider:
if , we remove except for those that intersects with other trees. The loss rates of those links located in an intersection can still be estimated as long as the probes sent by other sources can reach the receivers attached to the intersection.
if , no link will be deleted from the subtree rooted at node as long as observes some probes sent by the sources in . The loss rates of the links in cannot be estimated if no receiver in observes a probe from the sources.
as  stated, is not a feasible value. An estimator must avoid this circumstance in estimation. The top down algorithm proposed in  and the algorithm based on ( ?) ensure this would not occur by having .
was raised in . We call it partition circumstance here because the equation shows that the observations of are the partitions of the observations of , i.e. the receivers attached to each subtree of has the exclusive observations for a portion of the probes sent by the source. Given the partition circumstance at node of a tree topology, equation (Equation 3) cannot be used to estimate the loss rate of the links connecting to its children. For the general topology, the situation is improved if is a parent of node . Even , we are still able to estimate the loss rate of the link connecting to by ( ?) as long as the partition circumstance does not occur simultaneously for all ’s parents.
It is clear that (Equation 3) is a special case of ( ?). Each of the three circumstances raised previously for data consistency has its counterpart in the general topology that can be expressed as:
if , and
For the last one, cannot be determined by ( ?). If one of the three scenarios occurs, the corresponding procedure proposed in  can be used.
As discussed, data consistency has been improved in some degree for the general topology. The improvement is due to the use of multiple sources to send probes that reduces the probability of those extreme circumstances in comparison with the tree topology.
5Statistical Property of the Estimators
Apart from proving the three estimators are MLEs, we also study the statistical properties of the estimators, such as whether the estimators are minimum-variance unbiased estimators (MVUE); and/or the estimate obtained is the best asymptotically normal estimates (BANE), etc. This section provides the results obtained from the study.
5.1 Minimum-Variance Unbiased Estimator
The estimators proposed in this paper are MVUE and the following theorem proves this.
Based on Fisher information we can prove the variance of the estimates obtained by ( ?) from is also smaller than that of an estimate obtained by (Equation 3) from . Since the receivers attached to intersections observe the probes sent from different sources, the sum of the observed probes is at least larger than or equal to the maximum number of probes observed from a single source. Therefore, there is more information about the loss rates of the links located in the intersections since information is addictive under i.i.d. assumption. With more information, the variance of an estimate of a parameter must be smaller than another obtained from the probes sent by a single source according to Fisher information. Given the less varied estimates from the intersections, the variances of the estimates of other links that are not in intersections are also reduced, at least not increased. Therefore, the estimates obtained by ( ?) is better than those obtained from a single source.
5.2General Topology vs. Tree Topology
The results presented in this paper can be viewed as a generalization of the results presented in  since the tree topology is only a special case of the general topology. The findings and discovery presented in this paper cover those presented in . The following corollary confirms this:
For instance, ( ?) obtained for the general topology holds for the tree one. When , we have
On the other hand, we are also interested in whether those properties discovered from the tree topology can be extended into the general one and the difference between the original properties and the extended ones, in particular for the rates of convergence.
5.3Large Sample Behavior of the Estimator
Since the estimate obtained by the proposed estimator is the MLE , we are able to apply some general results on the asymptotic properties of MLEs in order to show that is asymptotically normally distributed as . Using this, we can estimate the number of probes required to have an estimate with a given accuracy for many applications. The fundamental object controlling convergence rates of the MLE is the Fisher Information Matrix at . Since yields exponential family, it is straightforward to verify that is consistent and that satisfies conditions under which is equal to
Eliminate singular on the boundary of , we have
Theorem ? states such a fact that with the increase of the number of probes sent from sources, there are more probes reaching the links of interest. Then, there is more information for the paths to be estimated. Let is the Fisher Information for a single observation, we have , where is the number of observations related to the link/path being estimated.
As , the difference in terms of Fisher information between the two estimators approaches to zero. Therefore, as , estimation can be carried out on the basis of individual tree and the asymptotical properties obtained previously for the tree topology  hold for the general topology as well. On the other hand, if , the estimate obtained by ( ?) is more accurate than those obtained from an individual tree. The simulation results presented in the next section illustrate this that shows the fast convergence of the estimates for the links located in the intersection because there are more information about the links.
Although there are a number of large sample properties that can be related to loss inference in the general topology, including various asymptotic properties, we would not discuss them further because means infinite number of probes be sent from sources to receivers that requires a long period of stationarity of the network, including traffic and connectivity, which is impractical based on the measurement .
For the purpose of proof of concept, a series of simulations were conducted on a simulation environment built on ns2, the network simulator 2 , the network topology used in the simulation is shown in Figure ?, where two sources located at node 0 and node 16 multicast probes to the attached receivers in an interval of 0.01 second. The binary network used here is for the simplicity reasons since no matter how many branches a node has they are always converted to a two-layer binary tree in estimation. Apart from the traffic created by probing, a number of TCP sources with various window sizes and a number of UDP sources with different burst rates and periods are added at the roots and internal nodes to produce cross-traffic, where TCP traffic takes about of the total traffic. The loss rate of each link is controlled by a random process that has drop rate except link that has drop rate. Each simulation run for 200 simulation seconds, and executed under the same condition except for using different seeds. The samples collected in this period are divided into groups to study the effect of group size on the accuracy of estimation. Five groups for 200, 400, 600, 800, and 1000 samples are presented here to illustrate the impact of the group sizes on the accuracy of estimation. The accuracy is measured by the relative error that is defined as:
where we use ns2 to report the losses and passes incurred on individual links. The ratio between the losses and the sum of the losses and passes is used as actual loss rate. The estimated loss rate of a link is obtained by only considering the end-to-end observations. The resultant relative errors of the 24 links are presented in Figure ? to Figure ?, each for 8 links. The figures show that in general with the increase of the group size, the relative errors are decreased as expected. Although there are a few cases of slight increase in the figures, that is due to the randomness of relative small group sizes used in estimation. The problem can be overcome with the increase of the group sizes. For example, observing Figures ? and ?, one is able to notice the consistent decrease of the relative errors with increase of the group size for those links falling into the intersection, i.e., link 4, 5, and 8-11 because there are almost 2 times of probes traversing through this intersections in comparison with other links. The relative errors of the shared links drop below 15% when each source sends 1000 probes; when 2000 probes are sent by each source, the relative errors of those links drop below 10% (not presented here). This phenomenon suggests the use of large group size in estimation for high accuracy. However, whether we are able to use more samples in estimation is a question that is not only related to the complexity of an algorithm but also related to the time scale of the stationarity of the network of interest. Based on the setup of our simulation, a 20-second (0.01 2000) stability is needed to have the relative error drops to 10%. If the time scale of the stability of a network is much larger than 20 seconds, the estimates obtained by the proposed estimators have potential to be used in traffic engineering to control traffic flows.
From Figures ? and ?, there is a noticeable difference between link 8 and link 9 in their converging rate in terms of the relative error. The two links are siblings, that means the same number of probes observed by their common parent first before sending to receiver 8 and 9. However, the relative error of link 8 converges much quick than that of link 9, while link 8 has a much higher loss rate than link 9 does. This indicates that the accuracy of estimating the loss rate of a link is proportional to the number of probes observed by its siblings. This can be explained analytically. For instance, given a 2 layer binary tree with 4 nodes, node 0 is the root that connects to node 1, node 1 has two children 2 and 3 and uses link 2 and 3 to connect them; the MLE of the loss rate of link 3 is equal to
. In this simulation, the loss rate of link 8 is set to 10% that is 10 times of that of link 9, which makes much smaller than and vise verse for and . This implies that we get more information for link 8 than that for link 9 from observations. That is why the relative error of link 9 is higher than that of link 8. This leads to a finding that extends the finding of  on branching ratio where the authors stated that estimate variance reduces with the increase of branching ratio. The fact of increasing branching ratio is equal to the decrease of the loss rate of the siblings of a link being estimated, that leads to more information about the link from the observations of the siblings. Therefore, to have an accurate estimate of a link, apart from increasing the number of samples in a group we can increase the number of siblings of a link instead. In contrast to previous algorithms, the increase of siblings would not significantly affect the complexity of the algorithms proposed in this paper since they all have complexity.
In this paper, we presents our recent findings on loss tomography of the general topology, including both theoretical results and practical algorithms. In theory, we extend the set of complete minimal sufficient statistics proposed previously for the link-based estimator into the path-based estimator. Based on the statistics, we provide the path-based likelihood function and have a set of likelihood equations. Solving the likelihood equations, we derive a direct expression of the MLE for the pass rate of the paths in a general topology. The direct expression has a similar structure as its counterpart in the tree topology, a polynomial is for a path. The expression in the general topology differs from the one in the tree topology in the collective constraint, where a set of polynomials is bounded together if the paths end at the same node that has a descendant tree. In this aspect, the polynomial derived for the tree topology  can be regarded as a special case of the one derived in this paper.
With the direct expression, we are able to estimate the number of probes reaching the root of an intersection. Given the estimated number of probes reaching the roots of all intersections, the network can be decomposed into a number of independent trees. For the descendant trees, the algorithms proposed for the tree topology can be applied to them to estimate the loss rates of the links. For the ancestor trees that are not from the intersections, new algorithms are needed since there is no exact observations for the leaf nodes that are the joint nodes before decomposition. Two types of algorithms, path-based and link-based, are presented and compared in this paper. The result shows that the link-based one is significantly better than the path-based one here in terms of performance.
The method proposed in  to merge subtrees rooted at the same link has been extended into the general topology to merge the statistics obtained from multiple sources. Using this method, we are able to reduce the degree of the polynomials used to express the pass rate of a path, and subsequently avoid the use of a numeric method to solve a high degree of polynomial. The estimator proposed in this paper consists of 3 steps
using ( ?) to estimate the pass rates of the paths connecting sources to joint nodes,
using the top down algorithm to estimate the loss rates of the links in descendant trees, and
using ( ?) to estimate the loss rate of the links in ancestor trees.
If no complicated overlapping occurs between the trees used to cover a network, there is no need to have any iterative procedures involved in estimation. The complexity of the estimator is . This performance is significantly better than the iterative algorithms, including the EM.
Data consistency raised previously for the tree topology  has been reconsidered for the general topology. Our study shows this problem remains but has been eased when multiple sources are used to send probes to receivers. In addition, our study shows the convergence rate of the links that receive probes from more sources is faster than those that only receive from one.
- Network Tomography: Estimating Source-Destination Traffic Intensities from Link Data.
Y. Vardi. Journal of Amer. Stat. Assoc.
- Multicast topology inference from measured end-to-end loss.
F. LoPresti & D.Towsley N.G. Duffield, J. Horowitz. IEEE Trans. Inform. Theory, 48, Jan. 2002.
- Maximum pseudo likelihood estimation in network tomography.
G. Liang and B. Yu. IEEE trans. on Signal Processing, 51(8), 2003.
- Network delay tomography.
Y. Tsang, M. Coates, and R. Nowak. IEEE Trans on Signal Processing, (8), 2003.
- Multicast-based inference of network-internal delay distribution.
F.L. Presti, N.G. Duffield, J. Horwitz, and D. Towsley. IEEE/ACM Trans. on Networking, (6), 2002.
- Unicast-based inference of network link delay distributions with finite mixture models.
Meng-Fu Shih and Alfred O. Hero III. IEEE Trans on Signal Processing, (8), 2003.
- Network delay tomography using flexicast experiments.
E. Lawrence, G. Michailidis, and V. Nair. Journal of Royal Statist. Soc, (Part5), 2006.
- Multicast inference of temporal loss characteristics.
V. Arya, N.G. Duffield, and D. Veitch. Performance Evaluation, (9-12), 2007.
- A fast lightweight approach to origin-destination ip traffic estimation using partial measurements.
G. Liang, N. Taft, and B. Yu. IEEE/ACM trans. on Networking, 14(6), 2006.
- Detecting Shared Congestion of Flows Via End-to-End Measurement.
D. Rubenstein, J. Kurose, and D. Towsley.
- Multicast-based inference of network-internal loss characteristics.
R. Cáceres, N.G. Duffield, J. Horowitz, and D. Towsley. IEEE Trans. on Information Theory, 45, 1999.
- Explicit loss inference in multicast tomography.
N. Duffield, J. Horowitz, F. Presti, and D. Towsley. IEEE Trans. on Information Theory, 52(8), Aug., 2006.
- A top down approach to estimate loss rate.
W. Zhu and K. Deng. In Proc of IEEE ChinaCom 2006, October 25-27, Beijing, China, 2006.
- Maximum likelihood from imcomplete data via the EM algorithm.
A. Dempster, N. Laird, and D. Rubin. Journal of the Royal Statistical Society, B, 1977.
- Network Tomography on General Topologies.
T. BU, N. Duffield, F.L. Presti, and D. Towsley. In Proc. of ACM SIGMETRICS 02, 2002.
- Inference of Internal Loss Rates in the MBone .
R. Cáceres, N.G. Duffield, S.B. Moon, and D. Towsley. In IEEE/ISOC Global Internet’99, 1999.
- Inferring link-level performance from end-to-end multicast measurements.
R. Cáceres, N.G. Duffield, S.B. Moon, and D. Towsley. Technical report, University of Massachusetts, 1999.
- Discussion – minimum chi-square, not maximum likelihood.
C.R. Rao. The Annals of Statistics, (3), 1980.
- Robust identification of shared losses using end-to-end unicast probes.
K. Harfoush, A. Bestavros, and J. Byers. In Technical Report BUCS-2000-013, Boston University, 2000.
- Unicast network tomography using EM algorthms.
M. Coates and R. Nowak. Technical Report TR-0004, Rice University, September 2000.
- A fast method to estimate loss rates.
W. Zhu and Z. Geng. Lecture Note in Computer Science, Vol. 3090/2004, 2004.
- A bottom up inference of loss rate.
W. Zhu and Z. Geng. Computer Communications, 28, 2005.
- Loss tomography from tree topology to general topology.
W. Zhu and K. Deng. Submitted for Publication, 2009.
- Loss rate estimation in general topologies.
W. Zhu. In Proc. of IEEE Communications Society/CreateNet BROADNETS, San Jose, CA, USA, 2006.
- Fundamentals of Statistical Exponential Families.
L. Brown. Institute of Mathematical Statistics, Hayward CA, 1986.
J. Pearl. Cambridge University Press, 2000.
- End-to-End Internet Packet Dynamics.
V. Paxson. IEEE/ACM Trans. on Networking, 7(3), 1999.
- Technical report, www.isi.edu/nsnam/ns2.
The network simulator 2.
- Pseudo maximum likelihood loss estimates for general topologies.
W. Zhu. In Proc. of IEEE GLOBECOM 2006, November 27- December 1. San Francisco, USA, 2006.