On the Optimality of the Kautz-Singleton Construction in Probabilistic Group Testing
We consider the probabilistic group testing problem where random defective items in a large population of items are identified with high probability by applying binary tests. It is known that tests are necessary and sufficient to recover the defective set with vanishing probability of error. However, to the best of our knowledge, there is no explicit (deterministic) construction achieving tests in general. In this work, we show that a famous construction introduced by Kautz and Singleton for the combinatorial group testing problem (which is known to be suboptimal for combinatorial group testing for moderate values of ) achieves the order optimal tests in the probabilistic group testing problem. This provides the first strongly explicit construction achieving the order optimal result in the probabilistic group testing setting. To prove the order-optimality of Kautz and Singleton’s construction in the probabilistic setting, we provide a novel analysis of the probability of a non-defective item being covered by a random defective set directly, rather than arguing from combinatorial properties of the underlying code, which has been the main approach in the literature. Furthermore, we use a recursive technique to convert this construction into one that can also be efficiently decoded with only a log-log factor increase in the number of tests.
The objective of group testing is to efficiently identify a small set of defective items in a large population of size by performing binary tests on groups of items, as opposed to testing the items individually. A positive test outcome indicates that the group contains at least one defective item. A negative test outcome indicates that all the items in the group are non-defective. When is much smaller than , the defectives can be identified with far fewer than tests.
The original group testing framework was developed in 1943 by Robert Dorfman . Back then, group testing was devised to identify which WWII draftees were infected with syphilis, without having to test them individually. In Dorfman’s application, items represented draftees and tests represented actual blood tests. Over the years, group testing has found numerous applications in fields spanning biology , medicine , machine learning , data analysis , signal processing , and wireless multiple-access communications [7, 8, 9, 10].
I-a Non-adaptive probabilistic group testing
Group testing strategies can be adaptive, where the test is a function of the outcomes of the previous tests, or non-adaptive, where all tests are designed in one shot. A non-adaptive group testing strategy can be represented by a binary matrix , where indicates that item participates in test . Group testing schemes can also be combinatorial [11, 12] or probabilistic [13, 14, 15, 16]. The goal of combinatorial group testing schemes is to recover any set of up to defective items with zero-error and require at least tests [17, 18]. Existing constructions in this setting achieve  and  tests. In contrast, probabilistic group testing schemes assume a random defective set of size , allow for an arbitrarily small probability of reconstruction error, and require tests only . In this paper, we are interested in non-adaptive probabilistic group testing schemes.
I-B Our contributions
To best of our knowledge, all known probabilistic group testing strategies that achieve tests are randomized (i.e., is randomly constructed) [13, 14, 15, 16]. Recently, Mazumdar  presented explicit schemes (deterministic constructions of ) for probabilistic group testing. This was done by studying the average and minimum Hamming distances of constant-weight codes (such as Algebraic-Geometric codes) and relating them to the properties of group testing strategies. However, the explicit schemes in  achieve , which is not order-optimal when is poly-logarithmic in . It is therefore of interest to find explicit, deterministic schemes that achieve tests.
This paper presents the first strongly explicit111We will call a matrix strongly explicit if any column of the matrix can be constructed in time . A matrix will be called explicit if it can be constructed in time . scheme that achieves . We show, perhaps surprisingly, that Kautz and Singleton’s construction , a well known sub-optimal (for moderate values of ) explicit scheme for combinatorial group testing, is actually optimal for probabilistic group testing. We prove this result for both the noiseless and noisy (where test outcomes can be flipped at random) settings of probabilistic group testing framework. We prove the order-optimality of Kautz and Singleton’s construction by analyzing the probability of a non-defective item being “covered” (c.f. Section II) by a random defective set directly, rather than arguing from combinatorial properties of the underlying code, which has been the main approach in the literature [19, 20, 21].
We say a group testing scheme is efficiently decodable if there is a decoding strategy that runs in -time. While we can achieve the decoding complexity of with the “cover decoder” (c.f. Section II), our goal is to bring the decoding complexity to . To this end, we use a recursive technique to convert the Kautz-Singleton construction into a strongly explicit construction with tests and decoding complexity . This provides an efficiently decodable scheme with only a log-log factor increase in the number of tests. Searching for order-optimal explicit or randomized constructions that are efficiently decodable remains an open problem.
The remainder of this paper is organized as follows. In Section II, we present the system model and necessary prerequisites. The optimality of the Kautz-Singleton construction in the probabilistic group testing setting is formally presented in Section III. We propose an efficiently decodable group testing strategy in Section IV. We defer the proof of the results to their corresponding sections in the appendix. We provide, in Section V, a brief survey of important results on group testing and a detailed comparison with Mazumdar’s recent work in . Finally, we conclude our paper in Section VI with a few interesting open problems.
Ii System Model and Basic Definitions
For any matrix , we use to refer to its ’th column and to refer to its ’th entry. The support of a column is the set of coordinates where has nonzero entries. For an integer , we denote the set by . The Hamming weight of a column of will be simply referred to as the weight of the column.
We consider a model where there is a random defective set of size among the items . We define as the set of all possible defective sets, i.e., the set of subsets of of cardinality and we let be uniformly distributed over .222This assumption is not critical. Our results carry over to the setting where the defective items are sampled with replacement. The goal is to determine from the binary measurement vector of size taking the form
where measurement matrix indicates which items are included in the test, i.e., if the item is participated in test , is a noise term, and denotes modulo-2 addition. In words, the measurement vector is the Boolean OR combination of the columns of the measurement matrix corresponding to the defective items in a possible noisy fashion. We are interested in both noiseless and noisy variant of the model in (1). In the noiseless case, we simply consider , i.e., . Note that the randomness in the measurement vector is only due to the random defective set in this case. On the other hand, in the noisy case we consider for some fixed constant , i.e., each measurement is independently flipped with probability .
Given and , a decoding procedure forms an estimate of . The performance measure we consider in this paper is the exact recovery where the average probability of error is given by
and is taken over the realizations of and (in the noisy case). The goal is to minimize the total number of tests while achieving an arbitrary but fixed average probability of error, i.e., satisfying .
We say that a column is covered by a set of columns if the support of is contained in the union of the supports of columns . A binary matrix is called -disjunct if any column of is not covered by any other columns. The -disjunctiveness property ensures that we can recover any defective set of size with zero error from the measurement vector in the noiseless case. This can be naively done using the cover decoder which runs in -time. The cover decoder simply scans through the columns of , and returns the ones that are covered by the measurement vector . When is -disjunct, the cover decoder succeeds at identifying all the defective items without any error.
In this work, we are interested in the probabilistic group testing problem where we allow an arbitrary but fixed average probability of error. Therefore we can relax the -disjunctiveness property. In the noiseless case, it is sufficient to ensure that at least fraction of all possible defective sets do not cover any other column. A binary matrix satisfying this relaxed form is called an almost disjunct matrix [21, 22, 23, 24] and with this condition one can achieve the desired average probability of error by applying the cover decoder.
Ii-B Kautz-Singleton Construction
Let be a finite field and be distinct elements from . Let . The Reed-Solomon code of dimension over , with evaluation points is defined with the following encoding function. The encoding of a message is the evaluation of the corresponding degree polynomial at all the ’s:
The Kautz-Singleton construction starts with a RS code with and . Each -ary symbol is then replaced by unit weight binary vectors of length , via “identity mapping” which takes a symbol and maps it to the vector in that has a 1 in the ’th position and zero everywhere else. Note that the resulting binary matrix will have tests. This construction achieves a -disjunct binary matrix with by choosing the parameter appropriately.
While this is a strongly explicit construction, it is suboptimal for combinatorial group testing in the regime : an explicit construction with smaller (achieving ) is introduced by Porat and Rothschild in . Interestingly, we will show in the next section that this same strongly explicit construction that is suboptimal for combintorial group testing in fact achieves the order-optimal result in both the noiseless and noisy versions of probabilistic group testing.
Iii Optimality of the Kautz-Singleton construction
We begin with the noiseless case ( in (1)). The next theorem shows the optimality of the Kautz-Singleton construction with properly chosen parameters and .
Under the noiseless model introduced in Section II, the Kautz-Singleton construction with parameters and achieves an arbitrary but fixed average probability of error with tests in the regime .
The proof of the above theorem can be found in Appendix -A. It is further possible to extend this result to the noisy setting where we consider for some fixed constant , i.e., each measurement is independently flipped with probability . Our next theorem shows the optimality of the Kautz-Singleton construction in this case.
Under the noisy model introduced in Section II with some fixed constant , the Kautz-Singleton construction with parameters and achieves an arbitrary but fixed average probability of error with tests in the regime .
The proof of the above theorem can be found in Appendix -B. Similar to the noiseless setting, the Kautz-Singleton construction provides a strongly explicit construction achieving optimal number of tests in the noisy case.
While the cover decoder, which has a decoding complexity of , might be reasonable for certain applications, there is a recent research effort towards low-complexity decoding schemes due to the emerging applications involving massive datasets [26, 27, 28, 29]. The target is a decoding complexity of . This is an exponential improvement in the running time over the cover decoder for moderate values of . For the model we consider in this work (i.e., exact recovery of the defective set with ), there is no known efficiently decodable scheme with optimal tests to the best of our knowledge. However, there is a recent line of work towards practical decoding schemes while preserving the order of as much as possible. The work  presented a randomized scheme which identifies all the defective items with high probability with tests and time complexity . Another recent result, , introduced an algorithm which requires tests with decoding complexity. Note that the decoding complexity reduces to when which is order-optimal (and sub-linear in the number of tests), although the number of tests is not. In both  and , the number of tests is away from the optimal number of tests by a factor of .
We can convert the strongly explicit constructions in Theorem 1 and 2 into strongly explicit constructions that are also efficiently decodable by using a recursive technique introduced in  where the authors construct efficiently decodable error-tolerant list disjunct matrices. For the sake of completeness, we next discuss the main idea applied to our case.
For a defective set , the cover decoder goes through the columns of and makes a decision whether an item is inside the defective set or not. This gives us the decoding complexity . However, if we were to somehow obtain a superset where it is guaranteed that , then the naive decoder would run in time which could potentially be more efficient depending on the size of . It turns out that we can construct this small set recursively.
Suppose that we have access to an efficiently decodable matrix that achieves exact recovery of the defective set with . We will construct two matrices and using as follows. The th column of for is identical to all th columns of for if the first bits of is where and are considered as their respective binary representations. Similarly, the th column of for is identical to all th columns of for if the last bits of is .
After getting the measurement vectors and from and we can apply the decoding algorithm for to and to obtain the estimate sets and respectively. Note that the sets and consist of -bit vectors and by union bound the set contains all the indices with error probability at most . We further note that .
|Reference||Number of tests||Decoding complexity||Construction|
|This work||Strongly explicit|
|This work||Strongly explicit|
We can now vertically stack and with a matrix which is not necessarily efficiently decodable to obtain our final matrix . The decoding is as follows. We first decode the components of corresponding to and to obtain and respectively. We next construct the set . We finally apply the naive cover decoder to the component of corresponding to over the set to compute the final estimate which can be done with an additional time. We provide this decoding scheme in Algorithm 1 for the special case for some non-negative integer . Note that by union bound the probability of error is bounded by . The next theorem is the result of applying this idea recursively.
Under the noiseless/noisy model introduced in Section II, there exists a strongly explicit construction achieving an arbitrary but fixed average probability of error with number of tests that can be decoded in time in the regime .
The proof of the above theorem can be found in Appendix -C. We note that with only extra factor in the number of tests, the decoding complexity can be brought to the desired complexity. We further note that the number of tests becomes order-optimal in the regime for some . In Table I we provide the results presented in this work along with the related results in the literature.
V Related Work
The literature on the non-adaptive group testing framework includes both explicit and random test designs . In combinatorial group testing, a famous construction introduced by Kautz and Singleton  achieves tests matching the best known lower bound [17, 18] in the regime where for some . However, this strongly explicit construction is suboptimal in the regime where . An explicit construction achieving was introduced by Porat and Rothschild in . While is the best known achievability result in combinatorial group testing framework, there is no strongly explicit construction matching it to the best of our knowledge. Regarding efficient decoding, recently Indyk, Ngo and Rudra  introduced a randomized construction with tests that could be decoded in time . Furthermore, their construction can be derandomized in the regime . Later Ngo, Porat and Rudra  removed the constraint on and provided an explicit construction that can be decoded in time .
On the other hand, there are various schemes relaxing the 0-error criteria in the group testing problem. For instance, a model where the decoder always outputs a super-set of the defectives containing less than other non-defective items was studied in [31, 32, 33]. Another framework where the goal is to recover at least a -fraction (for any arbitrarily small ) of the defective set with high probability was studied in  where the authors provided a scheme with order-optimal tests and the computational complexity. There are also different versions of the group testing problem in which a test can have more than two outcomes [34, 35] or can be threshold based [36, 37, 38]. More recently, sparse group testing frameworks for both combinatorial and probabilistic settings were studied in [39, 40, 41].
When the defective set is assumed to be uniformly random, it is known that is order-optimal for achieving the exact recovery of the defective set with error probability guarantee (which is the model considered in this work) using random designs and information-theoretical tools [13, 30, 16]. These results also include noisy variants of the group testing problem. Efficient recovery algorithms with nearly optimal number of tests were introduced recently in  and . Regarding deterministic constructions in this model, recently Mazumdar  introduced an analysis connecting the group testing properties with the average Hamming distance between the columns of the measurement matrix and obtained (strongly) explicit constructions with tests. While this result is order-optimal in the regime where for some , it is suboptimal for the moderate values of . The performance of the Kautz-Singleton construction in the random model has been studied empirically , but we are not aware of any theoretical analysis of it beyond what follows immediately from the distance of Reed-Solomon codes. To the best of our knowledge there is no known explicit/strongly explicit construction achieving tests in general for the noiseless/noisy version of the probabilistic group testing problem.
In this work, we showed that the Kautz-Singleton construction is order-optimal in the noiseless and noisy variants of the probabilistic group testing problem. To the best of our knowledge, this is the first (strongly) explicit construction achieving order-optimal number of tests in the probabilistic group testing setting. We provided a novel analysis departing from the classical approaches in the literature that use combinatorial properties of the underlying code. We instead directly explored the probability of a non-defective item being covered by a random defective set using the properties of Reed-Solomon codes in our analysis. Furthermore, by using a recursive technique, we converted the Kautz-Singleton construction into a construction that is also efficiently decodable with only a log-log factor increase in number of tests which provides interesting tradeoffs compared to the existing results in the literature.
There are a number of nontrivial extensions to our work. Firstly, it would be interesting to extend these results to the regime . Another interesting line of work would be to find a deterministic/randomized construction achieving order-optimal tests and is also efficiently decodable.
The third author would like to thank Atri Rudra and Hung Ngo for helpful conversations.
-A The proof of Theorem 1
Let be the number of items and be the size of the random defective set. We will employ the Kautz-Singleton construction. We use a RS code such that and we pick and appropriately in the following. Note that the resulting binary matrix has tests.
We note that for any defective set the cover decoder provides the exact recovery given that none of the non-defective items are covered. For , we define as the event that there exists a non-defective column of that is covered by the defective set . Define as the event that the non-defective column is covered by the defective set . We can bound the probability of error as follows:
where in the last equation is uniformly distributed on the sets of size among the items in and denotes the indicator function of an event.
Fix any distinct elements from . We denote . We note that due to the structure of mapping to the binary vectors in the Kautz-Singleton construction, a column is covered by the random defective set if and only if the corresponding symbols of is contained in the union of symbols of in RS code for all rows in . Denoting as the polynomial corresponding to the column , we have
where in the last step the random set of polynomials is generated by picking nonzero polynomials of degree at most without replacement. We define the random polynomial . Note that
We next bound the number of roots of the polynomial . We will use the following result from .
Lemma 1 (, Lemma 3.9).
Let denote the set of nonzero polynomials over of degree at most that have exactly distinct roots in . For all powers and integers
Let denote the number of roots of a random nonzero polynomial of degree at most . It is easy to observe that and using Lemma 1, we get
Hence we can bound . We denote as the number of roots of the polynomial and as the number of roots of the polynomial . Note that . We next use the Bernstein concentration bound for sampling without replacement :
We have , hence, under the regime , the last quantity is bounded by for some constant . Hence the number of roots of the polynomial is bounded by with high probability.
Given the condition that the number of roots of the polynomial is bounded by and the random set of polynomials is picked from the nonzero polynomials of degree at most without replacement, due to the symmetry, we claim that the probability of satisfying for all is bounded by the probability of covering elements from a field of size by picking elements randomly without replacement. We next prove this claim. We define the set and we emphasize that this is not a multiset, i.e., the repeated roots appear as a single element. We begin with the following observation.
Let , and condition on the event that . Then is uniformly distributed among all sets of size .
For , we can write
where is the corresponding multiplicity of the root and does not have any linear factor. We note that this decomposition is unique. For of size , let
Let such that and . Then . Indeed, let be a bijection such that . Then given by
and is a bijection.
We further note that , so
where is due to and we pick uniformly with replacement. ∎
Based on this, we have
Let us fix . We then have
Therefore, is bounded by
Applying the summation over all in (2), we obtain . Therefore, under the regime , an arbitrary but fixed average probability of error can be achieved with . The resulting binary matrix has tests.
-B The proof of Theorem 2
We begin with describing the decoding rule. Since we are considering the noisy model, we will slightly modify the cover decoder employed in the noiseless case. For any defective item with codeword weight , in the noiseless outcome the tests in which this item participated will be all positive. On the other hand, when the noise is added, of these tests will flip in expectation. Based on this observation (see No-CoMa in  for a more detailed discussion), we consider the following decoding rule. For any item , we first denote as the weight of the corresponding column and as the number of rows where both and . If , then the th item is declared as defective, else it is declared to be non-defective.
Under the aforementioned decoding rule, an error event happens either when for a defective item or for a non-defective item . Using the union bound, we can bound the probability of error as follows:
where in the first quantity of the last equation is randomly distributed on the sets of size among the items in . We denote the first quantity as and the second one as in the last equation.
We will employ the Kautz-Singleton construction. We use a RS code such that and we pick and appropriately in the following. Fix any distinct elements from . We denote .
We begin with . Fix any defective set in with size and fix an arbitrary element of this set. We first note that due to the Kautz-Singleton construction. We further note that before the addition of noise the noiseless outcome will have positive entries corresponding to the ones where . Therefore only depends on the number of bit flips due to the noise. Using Hoeffding’s inequality, we have
Summing over the defective items , we get .
We continue with . We fix an item and note that . We similarly define the random polynomial . Let be the event of having at most number of roots. We then have
Following similar steps as in the proof of Theorem 1 we obtain for some constant in the regime .
We next bound the first term in (3). We choose and define the random set . We then have
Let us first bound the second term . We note that
where in the last equality the random set of polynomials is generated by picking nonzero polynomials of degree at most without replacement. Following similar steps of the proof of Theorem 1 we can bound by considering the probability of having at least symbols from when we pick symbols from uniformly at random without replacement. Hence we have
where we use in the second inequality.
We continue with . Note that . We further note that
Since we have . Using Hoeffding’s inequality, we have