A method for assessing the success and failure of community-level interventions in the presence of network diffusion, social reinforcement, and related social effects\DisableLigatures
[f]encoding = *, family = *
A method for assessing the success and failure of community-level interventions in the presence of network diffusion, social reinforcement, and related social effects
Hsuan-Wei Lee1*, G. Robin Gauthier1, Jerreed D. Ivanich1, Lisa Wexler2, Bilal Khan1, Kirk Dombrowski1
1 Department of Sociology, University of Nebraska-Lincoln, USA
2 School of Public Health and Health Sciences, UMass Amherst, USA
Prevention and intervention work done within community settings often face unique analytic challenges for rigorous evaluations. Since community prevention work (often geographically isolated) cannot be controlled in the same way other prevention programs and these communities have an increased level of interpersonal interactions, rigorous evaluations are needed. Even when the ‘gold standard’ randomized control trials are implemented within community intervention work, the threats to internal validity can be called into question given informal social spread of information in closed network settings. A new prevention evaluation method is presented here to disentangle the social influences assumed to influence prevention effects within communities. We formally introduce the method and it’s utility for a suicide prevention program implemented in several Alaska Native villages. The results show promise to explore eight sociological measures of intervention effects in the face of social diffusion, social reinforcement, and direct treatment. Policy and research implication are discussed.
Assessments of the success and failure of interventions in a community (rather than clinical) setting tend to focus on individual behavior change or in the case of protective factors, behavior maintenance, normally by comparing the outcomes of intervention participants with those of non-participants . Social forces are recognized to influence those outcomes in two ways. In the first, participant/non-participant social context are thought to potentially influence the efficacy of the intervention—raising or lowering the impact on the basis of factors beyond the control of those performing it. To account for this possibility, analyses of outcomes generally depend on the use of “control variables” in statistical sense, i.e. measures such as gender or income or age that are assumed to serve as adequate stand-ins or predictors for the actual social position of the individual participant or non-participant and which may confound intervention results. Of course, the extent to which things like gender and age and income are in fact adequate stand-ins for that individual’s social position is seldom tested, and it is not altogether clear how they might be considered when our purpose is to examine specific individual behaviors.
The second way that social factors may influence the outcome of an intervention is when close social ties between participants and non-participants create the possibility of information transfer or behavioral influence between them . One can easily imagine a situation, for example, where participants of smoking cessation intervention have close family members who are smokers. If the former seek to influence the behavior of the latter, then community wide outcome measures will be subject to “contamination”. For this reason, most clinical trials require randomized control trials or RCT’s. Here participants and non-participants are assumed to be isolated from one another, thereby preventing contamination from taking place. Such conditions are often unachievable in community-based intervention settings . It is important to measure contamination, contagion, diffusion and a host of other transmission effects in community intervention settings where participants and nonparticipants are likely to be socially connected. Typically, these effects go un-noticed or un-measured thereby skew or confound results though to be more rigorous. Unfortunately, methods for measuring contamination and related social effects are scarce.
In what follows, we propose a method for measuring the success and failure of behavioral interventions in small, relatively closed community settings. This method makes use of data on dyadic (network) ties to supplant more coarse control variables. Using dyadic data, we locate actors in their actual social position viz those around them, rather than using control variables as proxies of those positions. The method proposes four measures of the success or failure of the behavior in creating behavior change (from a state of absence to a state of presences) at a later time) and four measures of the success or failure of the intervention in promoting the maintenance of already existing protective behaviors (from a state of presence to a state of presences at a later time). It first tests both change and retention in an individual in the context of all social influences, and then uses similar strategies to assess the overall social impact of the intervention, the reinforcing effects of social relationships, and the overall diffusion of the intervention effect to non-participants.
To demonstrate the utility of the method, we employ a case study. The case study puts the full method in application on data from a recent suicide prevention intervention that includes mental health promotion, primary and secondary prevention components. Here we examine intervention efficacy across 39 protective behaviors, using post-intervention results from six communities in Alaska.
Prior attempts to deal with the possibility of contamination in a community setting can be found in the work of An, who proposes strategies such as novel forms of randomization  and the strategies that take advantage of existing clustering . Similar strategies have been employed by Ugander , where by the authors propose a method to assess the effects of an A/B on-line experiment while being able to adjust for the effects of social influence. Others have proposed straightforward means for measuring social influence, including Actor-Partner Independence models  and various forms of binary outcomes assessment using multilevel logistic regression modeling . Others have proposed strategies for latent influence effects in a structural equation environment . Here we propose a simple, flexible way to pose questions related to social factors that potentially arise in the context of intervention strategies in close-knit community contexts, and show how several of these social and relational factors can be analyzed for their effect on the intervention outcomes. The case study described below occurred in relation to a community intervention, PC CARES [10, 11].
To study the effect of a social intervention on a group of people with an underlying network structure, we compare ego-alter dyads at different time steps. Our study assumes that data collection proceeds at two time points, one before and one after intervention survey. On the underlying graph at two time points, we observe dyads of ego-alter pairs and examine the state transitions of dyads.
Given a graph , define the participate function , we have
Also define the behavior function
At two time points , each node is either in or not in the intervention in both times, but at time and separately, each node could either be with or without the behavior. For and , let be a function that counts number of dyads defined as
where is the participation function and is the behavior function. For example, measures number of dyads satisfying: the ego is in the intervention (because ); the alter is not in the intervention (because ); the ego has the behavior at time (because ); the alter has the behavior at time (because ); the ego does not have the behavior at time (because ); the alter has the behavior at time (because ).
With this counting notation, we use 6 bits to record the participation state and the behavior state at two time points of each end of the dyads. Each bit consists of a binary number, therefore we have types of dyads. The total 64 dyads are shown in Fig 1.
Here we introduce more notations of counts. For each choice of we define
which captures the number of dyads in which is of type and is of type at time . For each choice of we define
which captures the number of dyads in which is of type and is of type at time . For each choice of we define
which captures the number of dyads in which is of type and is of type at time and is of type and is of type at time . Lastly, for each choice of we define
which captures the number of dyads in which is of participation type and is of type and is of type at time .
To understand the notations better, we consider some examples here. The notation captures the number of dyads in which has the behavior at time (because ) and does not have the behavior at time (because ). The notation captures the number of dyads in which is in the intervention (because ), is not in the intervention (because ), has the behavior at time (because ), and does not have the behavior at time (because ).
Eight Sociological Measures and Their Informal Interpretations
Using this set of definitions, we formalize eight questions related to prevention (the maintaining of a protective behavior) and treatment (the adoption of a protective behavior previously absent) that are often difficult to answer when doing community intervention work—even when using “gold standard” randomized control trials. The result of each test tell us whether we find a “significant” change in behavior, or in the case of protective behaviors, a “significant” continuation of already present protective behaviors.
Direct Treatment Success in a Social Context
Did the intervention promote a positive change in the behavior of someone who participated in PC-CARES  regardless of the intervention status or behavior change of their network “alters”. We measure this by comparing the transitions from not having the behavior (or performing the activity) to having the behavior (or performing the activity) in those who were in the PC-CARES intervention vs. those who were not in the intervention, across all states of behavior/activity of their network alters.
Our goal here is a basic measure of the success of the intervention on the participants given the full range of their social influences.
Direct Prevention in a Social Context
Did the intervention promote sustained protective behaviors (or performance of a protective activity) in participants of the PC-CARES intervention regardless of the intervention status or behaviors/activities of their network alters. We measure this by comparing the maintenance of having the desired behavior in a PC-CARES participant versus those who were not in the intervention, taking into account all of the social influences of their network alters.
Our goal here is a measure of the success of the intervention in maintaining protective behaviors among participants given the broad mix of their social connections.
Social Effect of Treatment
Did the participation of ego in the intervention induce a positive change in their alters’ behavior, regardless of whether an alter participated in the intervention or not, AND whether the PC-CARES participant’s behavior changed or not. We measure this by comparing the behavior change in the network alters of PC-CARES participants to the behavior change in the network alters of those who did not participate in PC-CARES.
Our goal here is a measure of the general “social effect” of the intervention in promoting behavior change.
Social Effect of Prevention
Did the participation of ego in the intervention induce a statistically significant continuation in their alters’ protective behavior, regardless of whether that alter participated in the intervention or not, AND whether the PC-CARES participant’s maintained the behavior or not. We measure this by comparing the continuation of protective behaviors in the network alters of PC-CARES participants to the continuation of protective behaviors in the network alters of those who did not participate in PC-CARES.
Our goal here is a measure of the general “social effect” of the intervention in promoting the maintenance of protective behaviors in the community as a whole.
Reinforcement of Change
Does it make a difference to the success of the treatment on PC-CARES participants when their network alters also participate in the PC-CARES intervention? We measure this by comparing behavior change of PC-Care participants whose network alters also participated with the behavior change of PC-CARES participants whose network alters did not participate.
Our goal here is to measure whether there is a social reinforcement of the intervention effects created by having people participate in the intervention along with those with whom they have an already existing social relationship.
Reinforcement of Prevention
Does it make a difference to the success of the PC-CARES intervention in promoting the maintenance of protective behaviors among those who participate in the intervention when their network alters also participate in the PC-CARES intervention? We measure this by comparing the continuation of protective behaviors of PC-Care participants whose network alters also participated with the maintenance of protective behaviors of PC-CARES participants whose network alters did not participate.
Our goal here is to measure whether there is a social reinforcement of the protective effects created by having people participate in the intervention along with those with whom they have an already existing social relationship.
Diffusion of Change
How effective is the intervention for promoting behavior change among nonparticipants in PC-CARES when their network alters participated in PC-CARES but the subject did not. We measure this by comparing positive behavior changes in egos who did not participate but whose alters did participate to egos who did not participate and whose alters also did not participate.
Our goal here is measure the diffusion of the PC-CARES intervention treatment on non-participants with a direct link to a participant.
Diffusion of Prevention
How effective is the intervention on preserving protective behaviors among nonparticipants whose alters participated in PC-CARES? We measure this by comparing maintenance of protective behaviors in egos who did not participate but whose alters did participate with egos who did not participate and whose alters also did not participate.
Our goal here is to measure the diffusion of the protective effects of the PC-CARES intervention on non-participants with a direct link to the a participant.
Investigating the effect of the intervention
Our main goal is to quantify and compare the effect of social interventions of the ego and alter behavior by asking sociological meaningful measures. To consider all the possible measures of ego and alter related to the intervention and behavior, we separate the counting function into two parts: participation and behavior parts. The participation part ( -part) cuts the probabilities into combinations of in and not in the intervention. The behavior part ( -part) cuts the probabilities into combinations of having or not having the behavior at different times and .
To verify whether the dyads in measure display some feature with sociological meanings, we set up a null model that is used as a term of comparison. In the null model assumption, we assume that the intervention does not make any change of egos’ and alters’ behavior. Although there are possibly different numbers of people in each configuration, under the null model assumption, they share the same probability distribution of changing their behavior at time . That is, we have 4 types of ego-alter participation combinations, e.g., , and they should all have the same probability transition matrix of the 16 types. Therefore our first step is to find out the probability transition matrix .
For each note that
collapses the coordinates together under the null hypothesis. Moreover, define
to be a matrix which has columns and has rows . Note that is a probability transition matrix and has row sum to be 1. Still under the null hypothesis, we assume for each pair of , it has a mass
This is the sum of people with pair of fixed values, and we use as the mass to generate the corresponding under the null hypothesis based on the transition matrix . Fix , the null model yields
where and is the indicator function. Therefore, in the null model, we have following a multinomial distribution with independent trials and the corresponding probabilities , where .
With the null model and the multinomial distributions computes from different sets of the observed , we can use the bootstrapping technique to simulate the approximated distributions for each measure by sampling the numbers number of times. For a given data set with have a specific value of each measure as , …, , we then compare each of them with the corresponding bootstrapped values , …, and see its tail probability of each measure. If the tail probability is less than some assigned significance level , then we have an -level of confidence to reject the null model assumption of the measure.
In addition to using the bootstrapping method to compute the tail probability of each measure, there is also analytical way of calculation. In each measure , there are two summations of counts. Each count satisfies a multinomial distribution, hence the summations are sums of the multinomial distribution. The combined sum then satisfies a Poisson-Binomial distribution, which is a discrete probability distribution of a sum of independent Bernoulli trials that are not necessarily identically distributed. Suppose we have independent trials with probabilities f success and failure, for the th trial, equal to and , respectively. Let be the number of success in trials. The number of success , or the probability mass function of could be written as the sum
where is the set of all subsets of integers that can be selected from . Furthermore, the subtraction in each measure makes the final distribution satisfies the convolution of two Poisson-Binomial distributions by using the convolution formula
where and are two independent Poisson-Binomial distributions. Since we know the distribution of null model, the tail probabilities of the observed values could be computed accordingly.
PC CARES Program Evaluation
Suicide among Indigenous populations has remained a problem across American Indian populations living on the reservation , and off the reservation . The problem, however, has remained a even more problematic for Alaska Native Youth [14, 15]. In line with the the recommendations of Okamoto , scholars working with Indigenous communities such as Allen et al. , and Whitbeck  suggest that using prepackaged prevention programs are problematic when working with Indigenous populations, that stand in need of localized mobilization of culturally tailored programs adapted or created for targeted needs.
To address concerns of suicide among ten Alaska Native villages, Promoting Community Conversations About Research to End Suicide (PC CARES) was created. The program is a community health education intervention created at the grassroots level with both general community feedback and targeted community leader (police, social work, local government officials, etc.) input. All curriculum and materials are created by the communities, for the communities, with the intent of leaving a sustainable and adaptable prevention program for years to come. Data for this study was collected among six of the ten villages.
To illustrate the full method discussed in this paper, we use data from the Promoting Community Conversations About Research to End Suicide (PC CARES) project conducted in a remote and rural region of Alaska. PC CARES is a community-based suicide prevention program that mobilizes community members to use information from research to determine how they will reduce risk and increase protective factors for suicide among Alaska Native communities. Detailed discussion of the model, community engagement, and justifications for PC CARES is found here . Collectively, PC CARES project was held in ten Alaska Native villages that share a similar culture and language. Of these participating communities, eight with the largest number of PC CARES participants were selected to participate in social network surveys. We collected data in eight villages. Five of the ten villages used respondent driven sampling (RDS) sampling strategy while the remaining three villages were small enough to achieve sufficient sampling with traditional random recruitment. In the five villages that used a RDS sample design, there were 423 responses. In total, 447 individuals across the six selected communities participated in the data collection.
With approval from the regional tribal health organization and the University of Massachusetts IRB, data collection for this novel method was done in six intervention villages by one or two researchers and their community partners. Before arriving in each village, announcements about the research let village members know that they could participate in a survey to identify what people are doing to help promote wellness in their lives and village. Importantly, research teams emailed and called those people who had participated in the intervention in order to maximize the number of PC CARES participants who filled out a survey. PC CARES participants were the first group recruited.
Before filling out the survey on iPad computers, participants went through a written consent process. They were told about the intervention research study, expectations of participation in the research, and possible risks and benefits. Focused on helping behaviors and support giving, the survey was minimal risk, and took about 20 minutes to complete. All participants received $20 for their time, whether or not they competed the survey.
After each person completed the survey, s/he was given either two or three \unichar8220coupons\unichar8221 to share with people close to them. The number of coupons given was related to the size of the village, and the level of participation in the data collection. Each \unichar8220coupon\unichar8221 was worth 5 dollars if the person they gave it to came in to complete their own survey. Data collection continued for 2-3 days in each of the five communities where RDS was used.
Individuals interviewed were asked a series of demographic characteristics, a series of 39 PC CARES program specific questions (see table 1), and were given a list of PC CARES participants and asked to identify if they were close to those on the list. In combination with the RDS edge list and the reported relationships in the individual surveys, a dyadic data set was created. Each dyad contained information on each person\unichar8217s PC CARES completion status, and their before and after state on the program specific questions listed in table 1. The survey asked about respondent conversations, understandings and behaviors related to suicide prevention. All questions were binary yes or no if they exhibit the behavior in question at the before state and again in the after state.
Fig 11 displays the overall evaluation results across 39 items measured for pre and post intervention using the method proposed in this paper. Along the y-axis, eight sociological tests of program evaluation in a social context. Along the x-axis each of the 39 items measured are located. Circles suggest a positive change, where a triangle suggests that negative change was introduced as a result of the PC CARES program. The size of each symbol corresponds to how significant—or how far from the null model the real data was. The larger the symbol, the farther away the more significant the relationship. Intersections absent of circles or triangles suggests that no significant relations found for the item and sociological test.
Overall, PC CARES seems to have introduced an overwhelming positive impact to the communities as noted with the number of circles. Specifically, 66/91 (73%) significant relationship found were positive changes in the community. However, 25 items across the eight sociological measures were found to have a significant negative relationship. Of the 25 items that were found to be a negative relationship, nearly half (12/25) were found for one specific sociological evaluation test “diffusion of change”. Furthermore, the inclusion of item 11, “Have you reached out to someone who attempted suicide?”, accounts for 60% (15/25) of all negative results are accounted for.
In terms of sociological tests, PC CARES made the most impact in terms of social effect of prevention. Across all 39 items, 26 (66.7%) items showed significant positive change with zero negative impacts. Individuals that had an alter participate in the program was associated with a statistically significant continuation of protective factors regardless of their own participation in PC CARES. Simply put, it appears that the PC CARES program induced positive change without hands on participation.
PC CARES had a large impact on two items of note. First, item 22, “Have you had conversations about making it harder for an ’at risk’ person to get a loaded gun?” Four of the eight tests were statistically significant for creating positive change on this item. Secondly, item 38, “Have you done something (subsistence, basketball, other activities) with a youth in the summer?” Item 38 showed statistical significance for positive change on all eight sociological tests.
Lastly, item 32, “Have you taken action, like removing guns or alcohol, to make a home safer?” also appears to have also been positively impacted by the PC CARES program. Although, this item showed significant negative change for reinforcement of change, it showed positive change on six other sociological tests.
In sum, PC CARES, using this evaluation method, has made a overwhelming positive change to the communities it served. Items related to gun safety also seemed to benefit from the PC CARES program. Rigorous results of this form are often difficult to disentangle in isolated communities that rely on kinship and social interaction that are not often seen as being problematic in Eurocentric communities.
Stanley and colleges  note the historical and contemporary difficulties establishing scientifically rigorous assessments in community prevention work. Opposed to clinical research, doing prevention evaluation within communities (often geographically isolated and heavily dependent on kinship) sits outside a controlled environment. The lack of exact control, presences of social dynamic process, and inclusion of cultural sharing systems establishes and environment that is often prone to critics against empirical assessments of community prevention work.
This paper presents a method for assessing, not only the success, but the failures of community-level intervention in the presence of complex social relationships. As illustrated in this article, the proposed method, using of dyadic data, can be used to explore eight sociological measures of prevention outcomes in community-intervention work that are often difficult to assess even when “gold standard” randomized control trials are employed. The method presented here allows for exploration of prevention effects in the wake of social diffusion and social reinforcement of treatment.
Investigators well situated to benefit from such a method include investigators that do prevention work among specialized populations, geographically isolated communities, and target populations where social interactions are assumed to occur naturally (and cannot be controlled). For instance, as the data suggests in this paper, investigators working with Alaska Native, American Indian, or Native Hawaiian can greatly benefit from this method—although these are only several examples. The benefits of using this method can equally extend to non-indigenous prevention scholarship, such as the current efforts among Appalachian populations  or school wide intervention efforts .
Limitation and Future Research
In spite of the contributions this paper provides to investigators doing prevention work within community settings, there are few key limitations that should be noted. First, the use of the proposed method requires the collection of social relations. While collecting relational data is not an inherit limitation, an added burden is placed on the researchers and respondent to know the prevention program status and behavior state of the respondents nominations. Second, to implement this method investigators should be sensitive to the thresholds required to “fill” the cells shown in Fig 1. For the method to be used, it is essential that counts of dyads are present for each cell. Lack of sufficient cell size may eliminate or reduce the sociological measures that can be assessed.
It is also important to note that the data presented here may present some sampling limitations. The data used for this study come from geographically isolated communities located in rural Alaska. Potential confounds and social complexities may be present in studies that work with populations that can more-freely access other study sites. Additionally, when doing prevention work, one should also be cognizant of recruitment and participation self-selection bias. These bias may include treatment effects, but also the bias present in social nominations collected.
Lastly, our analyses indicate that question 38, “Have you done something (subsistence, basketball, other activities) with a youth in the summer?” was significantly improved across all sociological measures. Here, we caution the interpretation of this result as this finding may be a product of measurement error. Given the timing of data collection for these communities, it is likely that engaging in outdoor activities and/or culturally specific activities, in optimal weather may have established a socially desirable response. Thus, this method is not robust against measurement error.
Notwithstanding these limitations, the current study makes make several contributions to scholarship on program assessment and prevention science. Given the findings presented here, future research would benefit by building on the work presented here in several ways. First, the sociological measures presented here are not an exhaustive list of potential social and dynamic processes that investigators may care to assess and more work is needed to expand the list of social processes of interest among prevention science assessment. Second, this method should be used across multiple modes of study design to assess the robustness of the method when, unlike here, an RDS study design is not used. Finally, scholars should employ this method among a variety of populations, social context, and geographic locations to evaluate the effectiveness of such a method in cases that are not as kinship based—unlike the data presented here.
Cras egestas velit mauris, eu mollis turpis pellentesque sit amet. Interdum et malesuada fames ac ante ipsum primis in faucibus. Nam id pretium nisi. Sed ac quam id nisi malesuada congue. Sed interdum aliquet augue, at pellentesque quam rhoncus vitae.
Appendix 1 - PC CARES Intervention Questions
Appendix 2 - Validation of the Bootstrapping Method and Direct Computation of the Poisson-Binomial Distributions
The data presented in Fig 12 was created through a simulation designed to produce a plausible representation of a community based intervention designed to reduce the prevalence of a targeted behavior. We based the structure of social relations in our simulation on the friendship network of 444 middle school children. Using this network structure, each actor in the network was assigned an intervention participation status (, see Eq 1) and an initial 0/1 behavior state was assigned stochastically based on the behavior states of their contacts (see Friedkin and Johnsen, 2011). The assignment of post-intervention states took place in two stages. An initial post-intervention behavioral state was generated for each node based on intervention participation. Actors who were randomly assigned to participate in the intervention had a low probability of a transition into the suppressed behavior (0\unichar85941: ) and a moderately high probability of transitioning out of that behavior state (1\unichar85940: ). Actors assigned to the non-participation (control) group were assigned an equally low probability of transition into a present behavior state (0\unichar85941: ), but given a much lower probability of transitioning into an absent one (1\unichar85940: ). Next, we update the post-intervention behavior state. Low to moderate levels of social conformity were enforced by allowing actors a twenty percent chance () to reassign their initial post-intervention behavior state to the state that is most common among their alters, based on these alters\unichar8217 initial post-intervention value.
The simulation procedure produced 2397 dyads. Within each simulated dyad, egos and alters each possess a behavioral attribute characterized by two states; present or absent before and after the intervention, yielding four distinct dyadic states: (1) both ego and alter report a present behavior state; (2) only ego reports a present behavior state, while alter’s state is absent; (3) ego\unichar8217s state is absent and alter’s is present; or (4) ego and alter report an absent state. Dyads are further divided into four groups based on the intervention status of both the ego and the alter. Within each group, a state transition matrix records counts of ego and alter’s joint (dyadic) post-intervention state according to its pre-intervention state. The four combinations of pre-intervention dyadic behaviors are graphically depicted along the rows of each scenario, while post-intervention states are depicted along the columns. The probability of movement from pre to post-intervention states is conditioned on the pre-intervention status of the dyad, thus all rows sum to 1.
In order to utilize the 64 numbers as raw counts, first, we transform the probabilities into counts by multiplying each probability by 100. Note that the total number of people in the study now becomes 1,600. Applying the bootstrapping technique to answer the eight measures with times of simulations, we got the p-values of the measures equal to correspondingly (see Fig 21).
- Spector PE. Research designs. vol. 23. Sage; 1981.
- Snijders TAB, van de Bunt GG, Steglich CEG. Introduction to stochastic actor-based models for network dynamics. Social Networks. 2010;32(1):44–60. doi:10.1016/j.socnet.2009.02.004.
- Sanson-Fisher RW, Bonevski B, Green LW, D\unichar8217Este C. Limitations of the randomized controlled trial in evaluating population-based health interventions. American journal of preventive medicine. 2007;33(2):155–161.
- An W. Multilevel meta network analysis with application to studying network dynamics of network interventions. Social Networks. 2015;43:48–56.
- An W. Peer effects on adolescent smoking and social network-based interventions. Dissertation, Department of Sociology, Harvard University; 2011.
- Ugander J, Karrer B, Backstrom L, Kleinberg J. Graph cluster randomization: Network exposure to multiple universes. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM; 2013. p. 329–337.
- Kenny DA, Kashy DA, Cook WL. Dyadic Data Analysis. Methodology in the social sciences. New York: The Gulford Press; 2006.
- McMahon JM, Pouget ER, Tortu S. A guide for multilevel modeling of dyadic data with binary outcomes using SAS PROC NLMIXED. Computational statistics & data analysis. 2006;50(12):3663–3680.
- Gonzalez R, Griffin D. The correlational analysis of dyad-level data in the distinguishable case. Personal Relationships. 1999;6(4):449–469.
- Wexler L, McEachern D, DiFulvio G, Smith C, Graham LF, Dombrowski K. Creating a community of practice to prevent suicide through multiple channels: describing the theoretical foundations and structured learning of PC CARES. International quarterly of community health education. 2016;36(2):115–122.
- Wexler L, Trout L, Rataj S, Kirk T, Moto R, McEachern D. Promoting Community Conversations About Research to End Suicide: learning and behavioural outcomes of a training-of-trainers model to facilitate grassroots community health education to address Indigenous youth suicide prevention. International journal of circumpolar health. 2017;76(1):1345277.
- Walls ML, Chapple CL, Johnson KD. Strain, emotion, and suicide among American Indian youth. Deviant behavior. 2007;28(3):219–246.
- Ivanich J, Teasdale B. Suicide Ideation among Adolescent American Indians: An Application of General Strain Theory. Deviant Behavior. 2017;0(0):1–14. doi:10.1080/01639625.2017.1304799.
- Allen J, Mohatt G, Fok CCT, Henry D, Awakening Team P. Suicide prevention as a community development process: understanding circumpolar youth suicide prevention through community level outcomes. International Journal of Circumpolar Health. 2009;68(3):274–291.
- Wexler L, Silveira ML, Bertone-Johnson E. Factors associated with Alaska Native fatal and nonfatal suicidal behaviors 2001\unichar82112009: trends and implications for prevention. Archives of suicide research. 2012;16(4):273–286.
- Okamoto SK, Kulis S, Marsiglia FF, Steiker LKH, Dustman P. A continuum of approaches toward developing culturally focused prevention interventions: From adaptation to grounding. The journal of primary prevention. 2014;35(2):103–112.
- Allen J, Mohatt GV, Fok CCT, Henry D, Burkett R, Team PA. A protective factors model for alcohol abuse and suicide prevention among Alaska Native youth. American journal of community psychology. 2014;54(1-2):125–139.
- Whitbeck LB. Some guiding assumptions and a theoretical model for developing culturally specific preventions with Native American people. Journal of Community Psychology. 2006;34(2):183–192.
- Stanley LR, Swaim RC, Kaholokula JK, Kelly KJ, Belcourt A, Allen J. The Imperative for Research to Promote Health Equity in Indigenous Communities. Prevention Science. 2017; p. 1–9.
- Hamby S, Segura A, Taylor E, Grych J, Banyard V. Meaning making in rural Appalachia: Age and gender patterns in seven measures of meaning. Journal of Happiness and Well-Being. 2017; p. 168–186.
- Valente TW, Ritt-Olson A, Stacy A, Unger JB, Okamoto J, Sussman S. Peer acceleration: effects of a social network tailored substance abuse prevention program among high-risk adolescents. Addiction. 2007;102(11):1804–1815.
- Fernández M, Williams S. Closed-form expression for the poisson-binomial probability density function. IEEE Transactions on Aerospace and Electronic Systems. 2010;46(2):803–817.