Truncated Random Measures

Truncated Random Measures

Trevor Campbell Computer Science and Artificial Intelligence Laboratory (CSAIL)
Massachusetts Institute of Technology
http://www.trevorcampbell.me/ tdjc@mit.edu
Jonathan H. Huggins Computer Science and Artificial Intelligence Laboratory (CSAIL)
Massachusetts Institute of Technology
http://www.jhhuggins.org/ jhuggins@mit.edu
Jonathan How Laboratory for Information and Decision Systems (LIDS)
Massachusetts Institute of Technology
http://www.mit.edu/ jhow/ jhow@mit.edu
 and  Tamara Broderick Computer Science and Artificial Intelligence Laboratory (CSAIL)
Massachusetts Institute of Technology
http://www.tamarabroderick.com tbroderick@csail.mit.edu
July 7, 2019
Abstract.

Completely random measures (CRMs) and their normalizations are a rich source of Bayesian nonparametric priors. Examples include the beta, gamma, and Dirichlet processes. In this paper we detail two major classes of sequential CRM representations—series representations and superposition representations—within which we organize both novel and existing sequential representations that can be used for simulation and posterior inference. These two classes and their constituent representations subsume existing ones that have previously been developed in an ad hoc manner for specific processes. Since a complete infinite-dimensional CRM cannot be used explicitly for computation, sequential representations are often truncated for tractability. We provide truncation error analyses for each type of sequential representation, as well as their normalized versions, thereby generalizing and improving upon existing truncation error bounds in the literature. We analyze the computational complexity of the sequential representations, which in conjunction with our error bounds allows us to directly compare representations and discuss their relative efficiency. We include numerous applications of our theoretical results to commonly-used (normalized) CRMs, demonstrating that our results enable a straightforward representation and analysis of CRMs that has not previously been available in a Bayesian nonparametric context.

First authorship is shared jointly by T. Campbell and J. H. Huggins.


\subfile

intro \subfilebackground \subfilesequentialcrms \subfiletruncation \subfilenormalization \subfilesampling \subfilesummary \subfilediscussion

Acknowledgments

The authors thank the anonymous reviewers for their thoughtful comments, which led to substantial improvements in both the presentation and content of the paper. The authors also thank Tin Nguyen for pointing out a flaw in the proof of a result appearing in a preprint draft of the paper. All authors are supported by the Office of Naval Research under MURI grant N000141110688. J. Huggins is supported by the U.S. Government under FA9550-11-C-0028 and awarded by the DoD, Air Force Office of Scientific Research, National Defense Science and Engineering Graduate (NDSEG) Fellowship, 32 CFR 168a. T. Campbell and T. Broderick are supported by DARPA award FA8750-17-2-0019.

\subfile

examples \subfiletechnicalresults \subfilesequentialproofs \subfiletruncproofs \subfilenormproofs

References

  • Abramowitz and Stegun (1964) Abramowitz, M. and Stegun, I. (eds.) (1964). Handbook of Mathematical Functions. Dover Publications.
  • Airoldi et al. (2014) Airoldi, E. M., Blei, D., Erosheva, E. A., and Fienberg, S. E. (2014). Handbook of Mixed Membership Models and Their Applications. CRC Press.
  • Arbel and Prünster (2017) Arbel, J. and Prünster, I. (2017). “A moment-matching Ferguson & Klass algorithm.” Statistics and Computing, 27(1): 3–17.
  • Argiento et al. (2016) Argiento, R., Bianchini, I., and Guglielmi, A. (2016). “A blocked Gibbs sampler for NGG-mixture models via a priori truncation.” Statistics and Computing, 26(3): 641–661.
  • Banjevic et al. (2002) Banjevic, D., Ishwaran, H., and Zarepour, M. (2002). “A recursive method for functionals of Poisson processes.” Bernoulli, 8(3): 295–311.
  • Blei and Jordan (2006) Blei, D. M. and Jordan, M. I. (2006). “Variational inference for Dirichlet process mixtures.” Bayesian Analysis, 1(1): 121–144.
  • Bondesson (1982) Bondesson, L. (1982). “On simulation from infinitely divisible distributions.” Advances in Applied Probability, 14: 855–869.
  • Brix (1999) Brix, A. (1999). “Generalized gamma measures and shot-noise Cox processes.” Advances in Applied Probability, 31: 929–953.
  • Broderick et al. (2012) Broderick, T., Jordan, M. I., and Pitman, J. (2012). “Beta processes, stick-breaking and power laws.” Bayesian Analysis, 7(2): 439–476.
  • Broderick et al. (2015) Broderick, T., Mackey, L., Paisley, J., and Jordan, M. I. (2015). “Combinatorial clustering and the beta negative binomial process.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2): 290–306.
  • Broderick et al. (2017) Broderick, T., Wilson, A. C., and Jordan, M. I. (2017). “Posteriors, conjugacy, and exponential families for completely random measures.” Bernoulli.
  • Campbell et al. (2018) Campbell, T., Huggins, J. H., How, J. P., and Broderick, T. (2018). “Supplement to “Truncated Random Measures”.”
  • Doshi-Velez et al. (2009) Doshi-Velez, F., Miller, K. T., Van Gael, J., and Teh, Y. W. (2009). “Variational inference for the Indian buffet process.” In International Conference on Artificial Intelligence and Statistics.
  • Ferguson (1973) Ferguson, T. S. (1973). “A Bayesian analysis of some nonparametric problems.” The Annals of Statistics, 1(2): 209–230.
  • Ferguson and Klass (1972) Ferguson, T. S. and Klass, M. J. (1972). “A representation of independent increment processes without Gaussian components.” The Annals of Mathematical Statistics, 43(5): 1634–1643.
  • Gautschi (1959) Gautschi, W. (1959). “Some elementary inequalities relating to the gamma and incomplete gamma function.” Journal of Mathematics and Physics, 38(1): 77–81.
  • Gelfand and Kottas (2002) Gelfand, A. and Kottas, A. (2002). “A computational approach for nonparametric Bayesian inference under Dirichlet process mixture models.” Journal of Computational and Graphical Statistics, 11(2): 289–305.
  • Gumbel (1954) Gumbel, E. J. (1954). Statistical theory of extreme values and some practical applications: A series of lectures. National Bureau of Standards Applied Mathematics Series 33. Washington, D.C.: U.S. Government Printing Office.
  • Hjort (1990) Hjort, N. L. (1990). “Nonparametric Bayes estimators based on beta processes in models for life history data.” The Annals of Statistics, 18(3): 1259–1294.
  • Ishwaran and James (2001) Ishwaran, H. and James, L. F. (2001). “Gibbs sampling methods for stick-breaking priors.” Journal of the American Statistical Association, 96(453): 161–173.
  • Ishwaran and James (2002) — (2002). ‘‘Approximate Dirichlet Process Computing in Finite Normal Mixtures: Smoothing and Prior Information.” Journal of Computational and Graphical Statistics, 11(3): 508–532.
  • Ishwaran and Zarepour (2002) Ishwaran, H. and Zarepour, M. (2002). “Exact and approximate sum representations for the Dirichlet process.” Canadian Journal of Statistics, 30(2): 269–283.
  • James (2002) James, L. F. (2002). “Poisson Process Partition Calculus with applications to Exchangeable models and Bayesian Nonparametrics.” arXiv preprint arXiv:0205093.
  • James (2013) — (2013). “Stick-breaking PG(,)-Generalized Gamma Processes.” arXiv preprint arXiv:1308.6570.
  • James (2014) — (2014). “Poisson Latent Feature Calculus for Generalized Indian Buffet Processes.” arXiv preprint arXiv:1411.2936v3.
  • James et al. (2009) James, L. F., Lijoi, A., and Prünster, I. (2009). “Posterior Analysis for Normalized Random Measures with Independent Increments.” Scandinavian Journal of Statistics, 36(1): 76–97.
  • Kim (1999) Kim, Y. (1999). “Nonparametric Bayesian estimators for counting processes.” The Annals of Statistics, 27(2): 562–588.
  • Kingman (1967) Kingman, J. F. C. (1967). “Completely random measures.” Pacific Journal of Mathematics, 21(1): 59–78.
  • Kingman (1975) — (1975). “Random discrete distributions.” Journal of the Royal Statistical Society B, 37(1): 1–22.
  • Kingman (1993) Kingman, J. F. C. (1993). Poisson Processes. Oxford Studies in Probability. Oxford University Press.
  • Lijoi et al. (2005) Lijoi, A., Mena, R., and Prünster, I. (2005). “Bayesian nonparametric analysis for a generalized Dirichlet process prior.” Statistical Inference for Stochastic Processes, 8: 283–309.
  • Lijoi et al. (2007) — (2007). “Controlling the reinforcement in Bayesian nonparametric mixture models.” Journal of the Royal Statistical Society B, 69(4): 715–740.
  • Lijoi and Prünster (2003) Lijoi, A. and Prünster, I. (2003). “On a normalized random measure with independent increments relevant to Bayesian nonparametric inference.” In Proceedings of the 13th European Young Statisticians Meeting, 123–124. Bernoulli Society.
  • Lijoi and Prünster (2010) — (2010). “Models beyond the Dirichlet process.” In Hjort, N. L., Holmes, C., Müller, P., and Walker, S. (eds.), Bayesian Nonparametrics, 80–136. Cambridge University Press.
  • Maddison et al. (2014) Maddison, C., Tarlow, D., and Minka, T. P. (2014). “A* Sampling.” In Advances in Neural Information Processing Systems.
  • Muliere and Tardella (1998) Muliere, P. and Tardella, L. (1998). “Approximating distributions of random functionals of Ferguson–Dirichlet priors.” Canadian Journal of Statistics, 26(2): 283–297.
  • Orbanz (2010) Orbanz, P. (2010). “Conjugate projective limits.” arXiv preprint arXiv:1012.0363.
  • Paisley et al. (2012) Paisley, J. W., Blei, D. M., and Jordan, M. I. (2012). “Stick-breaking beta processes and the Poisson process.” International Conference on Artificial Intelligence and Statistics.
  • Paisley et al. (2010) Paisley, J. W., Zaas, A. K., Woods, C. W., Ginsburg, G. S., and Carin, L. (2010). “A stick-breaking construction of the beta process.” International Conference on Machine Learning.
  • Perman (1993) Perman, M. (1993). “Order statistics for jumps of normalised subordinators.” Stochastic Processes and their Applications, 46(2): 267–281.
  • Perman et al. (1992) Perman, M., Pitman, J., and Yor, M. (1992). “Size-biased sampling of Poisson point processes and excursions.” Probability Theory and Related Fields, 92(1): 21–39.
  • Pitman (2003) Pitman, J. (2003). “Poisson-Kingman partitions.” Lecture Notes-Monograph Series.
  • Regazzini et al. (2003) Regazzini, E., Lijoi, A., and Prünster, I. (2003). ‘‘Distributional results for means of normalized random measures with independent increments.” The Annals of Statistics, 31(2): 560–585.
  • Rosiński (1990) Rosiński, J. (1990). “On series representations of infinitely divisible random vectors.” Annals of Probability, 18: 405–430.
  • Rosiński (2001) — (2001). “Series representations of Lévy processes from the perspective of point processes.” In Barndorff-Nielson, O., Resnick, S., and Mikosch, T. (eds.), Lévy processes: theory and applications, chapter VI, 401–415. Bikhäuser Boston.
  • Roy (2014) Roy, D. (2014). “The continuum-of-urns scheme, generalized beta and Indian buffer processes, and hierarchies thereof.” arXiv preprint arXiv:1501.00208.
  • Roychowdhury and Kulis (2015) Roychowdhury, A. and Kulis, B. (2015). “Gamma Processes, Stick-Breaking, and Variational Inference.” In International Conference on Artificial Intelligence and Statistics.
  • Sethuraman (1994) Sethuraman, J. (1994). “A constructive definition of Dirichlet priors.” Statistica Sinica, 4: 639–650.
  • Teh and Görür (2009) Teh, Y. W. and Görür, D. (2009). “Indian buffet processes with power-law behavior.” In Advances in Neural Information Processing Systems.
  • Teh et al. (2007) Teh, Y. W., Görür, D., and Ghahramani, Z. (2007). ‘‘Stick-breaking construction for the Indian buffet process.” In International Conference on Artificial Intelligence and Statistics.
  • Thibaux and Jordan (2007) Thibaux, R. and Jordan, M. I. (2007). “Hierarchical beta processes and the Indian buffet process.” In International Conference on Artificial Intelligence and Statistics.
  • Titsias (2008) Titsias, M. (2008). “The infinite gamma-Poisson feature model.” In Advances in Neural Information Processing Systems.
  • Zhou et al. (2012) Zhou, M., Hannah, L., Dunson, D., and Carin, L. (2012). “Beta-negative binomial process and Poisson factor analysis.” In Artificial Intelligence and Statistics.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
368735
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description