- Research
- Open access
- Published:
Hierarchical Bayesian adaptive lasso methods on exponential random graph models
Applied Network Science volume 9, Article number: 9 (2024)
Abstract
The analysis of network data has become an increasingly prominent and demanding field across multiple research fields including data science, health, and social sciences, requiring the development of robust models and efficient computational methods. One well-established and widely employed modeling approach for network data is the Exponential Random Graph Model (ERGM). Despite its popularity, there is a recognized necessity for further advancements to enhance its flexibility and variable selection capabilities. To address this need, we propose a novel hierarchical Bayesian adaptive lasso model (BALERGM), which builds upon the foundations of the ERGM. The BALERGM leverages the strengths of the ERGM and incorporates the flexible adaptive lasso technique, thereby facilitating effective variable selection and tackling the inherent challenges posed by high-dimensional network data. The model improvements have been assessed through the analysis of simulated data, as well as two authentic datasets. These datasets encompassed friendship networks and a respondent-driven sampling dataset on active and healthy lifestyle awareness programs.
Introduction
Multiple disciplines such as sociology, political science, and biology have extensively employed network analysis and random graph studies to comprehend and represent relationships among entities, ranging from friendships and global trading partners to proteins and genes. Early models generating random graphs assumed equal probability among graphs of the same size or independence among edges, but these models had evident limitations (Erdös and Rényi 1959). Holland and Leinhardt presented the next advancement by introducing a model for directed graphs that solely employed independent dyads (Holland and Leinhardt 1981). Subsequent work overcame the limitations of independence assumptions and introduced Markov random graph models, establishing the foundation for ERGMs that have been endured for decades (Frank and Strauss 1986), however, traditional statistical methods have limitations in effectively capturing the complexities of relational data. The ERGM has emerged as a valuable tool for quantifying such data, elucidating how local interactions shape the overall structure of a network. ERGMs acknowledge and capture the inherent interdependence embedded within network structures. The probability of an edge’s existence is influenced not only by the presence of other edges but also by various network configurations, such as triangles, and the characteristics of nodes throughout the entire network. This assumption of dependence aligns closely with our intuitive understanding of how networks are formed and operate. It is noteworthy that the development of ERGMs by Frank and Strauss (1986) was primarily motivated by the recognition of tie-dependence in networks.
Fundamentally, ERGMs are analogous to logistic regression when the dyads are independent, offering regression-like analysis on random networks. ERGMs estimate the probability of tie existence between pairs of nodes in a network. Since ERGMs share commonalities with logistic regression, let us recall the traditional lasso method in classical linear regression and discuss its development and relation to Bayesian theory, providing hints about the potential problems developing lasso estimates on the exponential random network. The lasso of Tibshirani is a method for simultaneous shrinkage and model selection in regression problems. Tibshirani (1996) In the context of linear regression, the lasso is a regularization technique for simultaneous estimation and variable selection where if \({\varvec{y}}={\varvec{X}}\varvec{\beta }+\varvec{\epsilon }\) where \({\varvec{y}}=(y_1,y_2,\cdots ,y_n)^{\top }\) is the response vector, \({\varvec{X}}=({\varvec{x}}_1,{\varvec{x}}_2,\cdots ,{\varvec{x}}_p)\) is an \(n \times p\) predictor matrix, \(\varvec{\beta }=(\beta _1,\beta _2,\cdots ,\beta _p)\) is a corresponding vector of regression coefficients, \(\varvec{\epsilon }=(\epsilon _1,\cdots ,\epsilon _n)\) are independent normal distributed errors, then the lasso estimates are defined as
where the second term in (1) is the so-called “\(l^1\) penalty”. The tuning parameter \(\lambda\) controls the amount of penalty. Fan and Li (2001) studied a class of penalized models including the lasso. They proved that the lasso can perform automatic variable section because of the singularity of \(l^1\) penalty at the origin. If certain conditions are not satisfied, the lasso estimates could be inconsistent. To overcome the above issues, Zou in 2006 and Wang et al. proposed to use an adaptive lasso that enjoys the consistency and the oracle properties: namely, it performs as well as if the true underlying model were given in advance. Zou (2006), Wang and Leng (2008) Tibshirani suggested that lasso estimates can be interpreted as posterior mode estimates when the regression parameters have independent and identical Laplace (i.e., double-exponential) priors. Tibshirani (1996) Targeting at finding this mode, several other authors studied subsequently different Bayesian contexts. Yuan and Lin (2006), Park and Casella (2008), Leng et al. (2014), Alhamzawi and Ali (2018) However, all these studies are for linear regressions and they are not built on random networks.
In the context of ERGMs, estimation encounters computational challenges when there is dependence among dyads. These challenges are primarily attributed to the intractability of the normalizing constant and the issue of degeneracy. Chatterjee and Diaconis (2013) Intractability refers to the computational difficulties associated with calculating the normalizing constant, which ensures that the probability mass function sums to one. On the other hand, degeneracy refers to the phenomenon where the models assign a significant proportion of their probability mass to a small subset of graphs. This leads to a cascading effect throughout the graph, resulting in the model assigning most of its probability mass to very sparse or very dense graphs. Bayesian computational methods have proven instrumental in circumventing these challenges. Caimo and Friel were the first to develop complete Bayesian frameworks for network models, enabling the incorporation of Bayesian analysis into real-world networks, which often exhibit large-scale, high-dimensional, and complex structures with numerous attribute variables associated with nodes (Caimo and Friel 2011). Subsequently, Caimo et al. integrated a transdimensional reversible jump Markov Chain Monte Carlo (RJMCMC) approach, initially introduced by Green (1995), with the exchange algorithm (Caimo and Friel 2013, 2014). This algorithm incorporates an independence sampler, utilizing a distribution that fits a parametric density approximation to the within-model posterior. This method is appealing in model selection since it relies exclusively on probabilistic considerations but is challenging computationally since it needs to estimate the posterior probability for each competing model. In scenarios with a high number of variables, the presence of numerous potential models becomes more pronounced. The increased dimensionality leads to a larger set of competing models, making the task of model selection more challenging and critical. This motivates the development of the penalized exponential random graph model developed in this paper.
While penalized estimation methods have been discussed in the context of graphical models by various researchers, these studies either lack a specific focus on ERGMs or fail to fully account for the inherent dependencies present in network data, often transforming the problem into generalized penalized linear regression. Meinshausen and Bühlmann (2006), Shojaie et al. (2012), Shojaie and Michailidis (2010), Shojaie (2013), Fan et al. (2009) Motivated by the need to explore network model uncertainty and achieve parsimony in exponential random graphs, we propose a more flexible and adaptive lasso-type penalized model within the framework of the ERGM. This model aims to improve parameter estimations and prediction accuracy, enabling effective variable selection within high-dimensional network data. Through comprehensive evaluations and comparisons with existing methods, our model demonstrates its superiority in terms of efficiency and effectiveness in selecting significant variables. It promises substantial improvements in the field by addressing the critical challenge of model selection in the analysis of high-dimensional network data.
In summary, the utilization of Bayesian adaptive lasso model offers two prominent advantages: (1) Enhanced convergence speed and improved parameter mixing: adaptive lasso addresses a notable limitation of the conventional lasso regularization technique, which often exhibits sluggish convergence and difficulties in selecting significant variables within high-dimensional datasets. Consequently, it facilitates faster convergence and more effective mixing of parameters. This characteristic proves particularly advantageous in scenarios involving extensive datasets or a substantial number of predictors. (2) Effective variable selection: Bayesian adaptive lasso exponential random graph model demonstrates exceptional proficiency in this task by automatically identifying pertinent variables while concurrently shrinking or eliminating less relevant ones. The process is facilitated through the utilization of multiple chains generated by a parallel direction sampling algorithm, which enhances the efficiency and accuracy of variable selection. These benefits are the primary focus of the discussed article.
This article is structured as follows. Section 2 provides a basic introduction to exponential random graph models, offering a foundation for the subsequent discussions. In Sect. 3, we introduce a Bayesian Exponential Adaptive Lasso Model for the exponential random graph, which enhances the Monte Carlo maximum likelihood method proposed by Geyer and the Bayesian ERGM (BERGM) presented by Caimo and Friel (Geyer 1991; Caimo et al. 2022). Section 4 presents a derivation of the Gibbs sampling theory underlying the model, shedding light on the underlying theoretical framework. In Sect. 5, we introduce the adaptive parallel direction sampling algorithm, which is incorporated into the Gibbs sampling theory to improve the mixing of the Monte Carlo chains, thereby enhancing the overall performance of the model. Section 6 outlines the algorithm procedure and provides a comparative analysis with the BERGM method proposed by Caimo et al., highlighting the strengths and advantages of our proposed approach. Caimo and Friel (2013), Caimo and Friel (2014), Caimo et al. (2022) In Sect. 7, we describe the network dataset called Faux Dixon High, which is used to test the model and present simulation results. Additionally, this section includes the results of applying the proposed model to data collected in a study conducted with the Prevention Research Center at USC and Sumter County Active Lifestyles (SCAL). In Sect. 8, we discuss the goodness of fit for the proposed Bayesian adaptive lasso method, providing an evaluation of its performance and suitability. Finally, in Sect. 9, we summarize the key findings and contributions of the paper and identify open problems and avenues for future research
Exponential random graph models
Examples and context
Exponential Random Graph Models (ERGMs) are widely applicable to research questions in the social and health sciences. In psychology, researchers studied Romanian school children’s friendship networks to find that sex and mental health showed patterns of homophily, concluding that ERGM are a “promising avenue for further research.” Baggio et al. (2017) Also in the social and health sciences, Becker et al. considered the friendship network of members of a sorority and the influence of disordering eating habits on friendship finding that women tended to have disordered eating habits, unlike their friends (Becker et al. 2018). This unexpected result has implications for understanding the complex social dynamics that go into a serious health concern. Solo et al. note the utility and suitability of ERGM for modeling connections within the brain compared to more traditional methods, though they also note the computational difficulty of ERGM (Solo et al. 2018). On a much larger scale, ERGM have been used to understand the influences of information sharing on tourism. The model helped answer questions about the existence of patterns in the network including whether or not the network exhibited the characteristic of homophily and how organizations should understand their role in the network (Williams and Hristov 2018). In the biological world, Stivala et al. show that ERGM can address some of the limitations that previous research had found in modeling biological processes (Stivala and Lomi 2021). These examples show the incredible flexibility and significance of exponential random graph models.
Model structure
For any network, it can be expressed with an adjacency matrix. The connectivity of the network’s graph is described by an \(n\times n\) adjacency matrix \({\varvec{Y}}\). Its i-j entry \(Y_{i,j} = 1\) if node i will give referral to node j and \(Y_{i,j} = 0\) otherwise. Let \({\mathcal {Y}}\) be the set of all possible graphs on n nodes and let \({\varvec{y}}\) be a realization of \({\varvec{Y}}\). A given network \({\varvec{y}}\) consists of n nodes and m edges that define a relationship between pairs of nodes called dyads. The adjacency matrix of the network graph \({\varvec{Y}}\) allows for the analysis of the structural relationship in the observed network.
For general exponential random graph models, the network has the following exponential family type density: (Lusher et al. 2013)
where \({\varvec{y}}\) is the observed network, \(\varvec{\theta }\) is a vector of parameters, and \(s({\varvec{y}})\) is a vector of network statistics. Each i-th network statistic \(s_{i}(\cdot )\) has a corresponding parameter \(\theta _{i}\). A positive value of \(\theta _{i}\) indicates that the edges involved in the formation of network statistics \(s_{i}\) are more likely to be connected with each other. The normalizing constant \(z(\varvec{\theta })\) is the summation \(\sum \limits _{{\varvec{y}} \in {\mathcal {Y}}}e^{\varvec{\theta }^{T}s({\varvec{y}})}\) where \({\mathcal {Y}}\) is the set of all possible graphs with the same number of nodes as \({\varvec{y}}\). The number of possible graphs with \(n\) nodes is \(2^{n(n-1)/2}\) which becomes very large for all but the smallest graphs. Lusher et al. (2013) Hence, the calculation of \(z(\varvec{\theta })\) is feasible only for small networks in computer computation. It becomes challenging to find this normalization constant for large networks or even moderate-sized networks.
Let \(\varvec{\delta }=s({\varvec{y}}_{ij}^{+})-s({\varvec{y}}_{ij}^{-})\) be the vector of changes in the statistics in \({\varvec{s}}\) when the edge \(y_{ij}\) between node i and j in the graph \({\varvec{y}}\) changes from 1 to 0 along with the complement part \({\varvec{y}}_{ij}^c\) same. Conditioned on the state of the rest of the graph represented \({\varvec{Y}}_{-ij}\), the \(\log\) odds of the probability of a tie existing between node i and j is:
These network statistics can be overlapping subgraph configurations such as the number of edges, mutual edges, triangles, and uniform homophily etc. The representation above gives the intuitive explanation of the model parameter \(\varvec{\theta }\) about their effect on the probability of an edge between node i and j.
Classical inference for ERGMs
Estimation methods
The inferential statistical goal is to find an appropriate estimate of \(\varvec{\theta }\) such that the corresponding generated network has the probability distribution centered on the observed network on average. That is, we want to solve the moment equation:
where \({\varvec{y}}_{obs}\) is the observed network and \(s({\varvec{y}})\) is a vector of network statistics in the proposed graph and \(s({\varvec{y}}_{\text {obs}})\) is a vector of the network statistics in the observed graph. However, in most cases, the moment equation cannot be solved analytically. This challenge leads to two mainstream simulations: Maximum Pseudolikelihood estimation and Monte Carlo Maximum Likelihood estimation.
Maximum pseudolikelihood estimation
The direct Maximum likelihood estimation of ERGMs is complicated since the likelihood function is difficult to compute for models and networks of moderate or large size. Strauss et al. proposed a standard approximation with maximum pseudolikelihood estimation (MPLE). Strauss and Ikeda (1990) Instead of conditioning each tie on the state of the entire graph, the assumption is that the dependence of each dyad is weak. In particular, the MPLE estimates can be obtained by assuming the independence among values of \(Y_{ij}\):
This allows for the pseudolikelihood function that has the strength of quick estimation but has been shown to not provide reliable estimates. van Duijn et al. (2009), Friel et al. (2009)
This will only provide the true estimate for ERGM with dyadic independence or when the change statistics can be found only considering one tie without knowing the rest of the graph. Research by van Duijn et al. compares the maximum pseudo-likelihood and maximum likelihood estimates, and their study shows the pseudo-likelihood estimation is biased and MPLE can only approximate the transitivity pattern in the network well. van Duijn et al. (2009)
Monte Carlo maximum likelihood estimation
Similar to methods in linear regression, ERGMs are log-linear, and a typical method for finding the maximum likelihood requires finding the roots of the derivative of the \(\log\) of the function. This results in the \(s({\varvec{y}})^{T} - {\mathbb {E}}_{\varvec{\theta }}(s({\varvec{y}})) = 0\) found earlier. The Monte Carlo maximum likelihood estimation in ERGM case needs to find the following important ratio: (van Duijn et al. 2009)
The log-likelihood equation, however, is not directly solvable without computing the normalizing constant. As previously mentioned, this is computationally intensive for all but the smallest graphs. With this approximation, though, the normalizing constant can be estimated by generating \(m\) graphs from the density \(\pi (\varvec{\pi |\varvec{\theta }_0})\) and finding \(e^{(\varvec{\theta }-\varvec{\theta }_{0})^{T}s({\varvec{y}}_{i})}\) for each graph and use importance sampling technique. The estimates of \(\varvec{\theta }\) can be obtained by maximizing the log-likelihood ratio approximated as the following:
However, in this method, the choice of the initial \(\varvec{\theta }_0\) is tricky and should be near the maximum likelihood estimate of \(\varvec{\theta }_0\). Poor choice of \(\varvec{\theta }_0\) can lead to the failure of the maximization log-likelihood function and degeneracy problem. van Duijn et al. (2009), Handcock (2003)
Bayesian adaptive lasso exponential random graph model
This work is motivated by the need to explore model uncertainty and flexibility. With these objectives, we consider the following exponential random graph model, this model is a particular class of discrete exponential random exponential families that represent the probability distribution of the adjacency matrix \({\varvec{Y}}\in {\mathcal {Y}}\) where \({\mathcal {Y}}\) is the set of all possible graphs on n nodes. Let \({\varvec{y}}\) a realization of \({\varvec{Y}}\). The likelihood function of an ERGM stands for the probability density of a random network and can be expressed as:
where \(q({\varvec{y}}|\varvec{\theta })=e^{\varvec{\theta }^{T} s({\varvec{y}})}\) is the unnormalized likelihood.
We consider the following adaptive lasso estimator on the exponential random network:
where \(l(\varvec{\theta }|{\varvec{y}})=\ln (\pi ({\varvec{y}}|\varvec{\theta }))\) is the log-likelihood function of \(\varvec{\theta }\) and each \(\lambda _j\) is a different penalty parameter used for the coefficients. In dyadic independence ERGMs, maximizing the log-likelihood function (10) is equivalent to maximizing the following log pseudo-likelihood function:
where \(\pi _{ij}=P(Y_{ij}=1|{\varvec{y}}_{ij}^c)=P(Y_{ij}=1)\). In this case, the network estimation problems are transformed into the classical adaptive lasso logistic linear regression model. For example, the coordinate descent algorithm developed in glmnet package for R (Tay et al. 2023; Friedman et al. 2010) can get estimations of the parameters \(\theta _j\), \(j=1,2,3,\cdots ,p\) with penalties include the lasso, ridge and the elastic net. However, different from the generalized linear regression models, the challenge of estimation on the dyadic dependent ERGMs relies on the intractable normalizing constant appearing in the log-likelihood function. With the review of ERGMs likelihood-based methods in Sect. 2, the solution to the equation (10) has similar obstacles. To get around those obstacles, we will study this problem with an adaptively Bayesian estimate obtained from the lasso penalized method on the random networks.
Assume that a prior distribution \(\pi (\varvec{\theta })\) is placed on \(\varvec{\theta }\), and we are interested in the posterior distribution
We consider a conditional Laplace prior specification of the form similar to the classical Bayesian lasso linear regression developed in Park and Casella (2008) but with different penalty terms so that we have \(\lambda _j \text { for } j=1, 2, 3, \cdots , p\):
We can now formulate a hierarchical model on the exponential random graph, which we can use to implement this version of the Bayesian lasso with a Gibbs sampler, using the Laplace distribution as a scale mixture of Gaussians. When the mixing distribution is exponential, the resulting distribution is Laplace. Andrews and Mallows (1974)
Now we use a latent parameter \(\tau ^2\) to make the prior (14) as a scale mixture of normal distributions (15). We can consider \(\tau _j\)s as additional parameters that assign different variances to the prior of \(\varvec{\theta }\). When \(\tau _j\rightarrow 0\), the coefficient of \(s_j({\varvec{y}})\) is shrunk to zero.
Assume \(\varvec{\theta } = ( \theta _{1}, \theta _{2},..., \theta _{p})\) follows normal distributions centered at zero with variance defined below.
where \(\sigma ^2>0\) and \({\varvec{D}}_{\tau }=diag(\tau _1^2,\tau _2^2,\cdots ,\tau _p^2)\) is a matrix that allows each parameter to come from a normal distribution with a different variance.
Different than the basic Bayesian lasso model proposed by Park and Casella (2008) in which \(\varvec{\tau }\) follows
our Bergm adaptive lasso model sets up different shrinkage parameters for different coefficients. This motivates us to define a more adaptive penalty in the hierarchical structure:
and an independent non-informative scale-invariant marginal prior \(\pi (\sigma ^2)\propto \displaystyle \frac{1}{\sigma ^2}\) on \(\sigma ^2\) suggested by Park and Casella. Park and Casella (2008) The conditional distribution on \(\sigma ^2\) guarantees a unimodal full posterior distribution for the estimate \(\varvec{\theta }\) on the network. (See Appendix A ). The unimodal posterior distribution ensures the quick convergence of the Gibbs sampling algorithm and ensures the meaningful single point estimate of \(\varvec{\theta }\).
Finally, the simplest prior for the penalty term \(\lambda _j\), for \(j=1,2,3,\cdots ,p\) would be a uniform distribution, but this proved to be problematic with complex networks, particularly when a model has many parameters. Thus, following the notation of Park and Casella (2008) we propose a prior such that \(\lambda _{j}^{2}\) follows Gamma distribution with shape parameter r and rate parameter \(\delta _j\):
This prior mixes well with the other choices for the Gibbs sampling and as Park and Casella (2008) note, this prior can approach 0 as \(\varvec{\lambda } \rightarrow \infty\) and can concentrate probability near the MLE.
In summary, the hierarchical formulation of the Bayesian adaptive lasso Model on the exponential random graph is as follows:
for \(\sigma ^{2}, r, \delta _j,j=1,2,3,\cdots ,p \text { and } \tau _1^2,\tau _2^2,\cdots ,\tau _p^2>0\).
The major differences of this formulation compared with the Bayesian lasso in Park and Casella (2008) are first, the Bayesian lasso method in Park and Casella (2008) is applied to linear regression model \({\varvec{y}}=\mu {\varvec{1}}_n+\varvec{X\beta }+\varvec{\epsilon }\) without any network structure. In other words, \({\varvec{y}}\) in Park and Casella (2008) follows the normal distribution \({\mathcal {N}}(\mu {\varvec{1}}_n+\varvec{X\beta }, \sigma ^2{\varvec{I}}_n)\), where \({\varvec{y}}\) is a \(n\times 1\) vector of responses which doesn’t involve random graph. Second, our model allows different penalty variables \(\lambda _j\), one for each different parameter. In this case, each \(\tau _{j}^{2}\) can have its own distribution and thus the variance of each normal distribution can be different. With the flexibility of the penalties, the lasso estimate of the parameter for less important random variables on the exponential random graph will have a larger penalty. And smaller penalty will be applied to those important random variables. And compared with the existing Bayesian Adaptive Lasso model (Leng et al. 2014),Alhamzawi and Ali (2018), our model is built on the random network. And compared with the Bayesian Exponential Random Graph Model (BERGM) by Caimo and Friel (2011), our model Bayesian Adaptive Lasso Exponential Random Graph Model(BALERGM) has more accurate estimations, and the structure is more flexible and adaptive to the network statistics level by adopting distinct shrinkage and penalties for different network statistics. The estimates \(\hat{\theta _j}\) of \(\theta _j \text { for } j=1,2,3, \cdots , p\) will be small and close to 0 if it does not provide much improvement on predicting the random network \({\varvec{Y}}\). So it naturally leads to an estimator with an automatic variable selection property. The value of \(\lambda _j\) will affect the estimates \(\theta _j\). The larger \({\hat{\lambda }}_j\) exists in the model, the sparser \(\varvec{\theta }\) will be. (namely, more coefficients are small and near 0) whereas smaller \({\hat{\theta }}_j\) leads to a less sparse \(\varvec{\theta }\). Sparsity is a common belief in high-dimensional statistics because we anticipate only a few covariates are actually related to the response and most covariates are useless. BALERGM is very powerful in this scenario because it leads to a sparse estimator on the network (many coefficients are near 0). Note that high-dimensional problems in network science are very common. For example, in genetics, there are many genes per individual but often we have few patients in our study, or in neuroscience, the fMRI machine produces many voxels per person at a given time.
Gibbs sampler implementation
Now we will implement the model with a Gibbs sampler. The Gibbs sampling method is a Markov Chain Monte Carlo (MCMC) algorithm. In our case, the joint distribution is difficult to sample from directly, but the conditional distribution of each variable is known and is easier to sample from. The Gibbs sampling algorithm generates an instance from the distribution of each variable in turn, conditioned on the current values of the other variables. The construction of the hierarchical model (20) makes the derivation of the full conditional distributions for each component of the estimates feasible.
Thus we can write the joint density as the product of the conditional density of \({\varvec{y}} |\varvec{ \theta }\) and the density of \(\varvec{\theta }\). Using the pieces of the hierarchical formulation of the model from (20) we can substitute in each piece that we have already chosen to find the joint distribution.
To implement the Gibbs sampling, we require the distribution of each parameter \(\tau _{j}, \lambda _{j}, \sigma ^{2}\) to update in turn. From the joint distribution (26), we consider all parts of that joint distribution that depend on each variable. As summarized in Table 1, we consider the full conditional distributions for \(\tau _{j}, \lambda _{j},\text { and } \sigma ^{2}\) respectively.
Sample \(\tau _{j}\)
For each \(\tau _{j}\) we have the following distribution.
To find what distribution each \(\tau _{j}\) follows, we begin by considering the following transformation. Chhikara and Folks (1988) If a random variable \(x \sim \text {Inverse Gaussian}(\mu , \lambda ')\), that is
then with the change of variable, we can find the density \(f'\) of \(w = x^{-1}\) as
Hence
Then we can rewrite equation 27 into the reciprocal of the Inverse Gaussian distribution
thus \(\displaystyle \frac{1}{\tau _j^2}\) follows inverse Gaussian distribution with parameters \(\displaystyle \sqrt{\frac{\lambda _{j}^{2} \sigma ^{2}}{\theta _{j}^{2}} }\) and \(\lambda _{j}^{2}\):
Sample \(\sigma ^{2}\)
Similar to the other parameters, we now look at \(\sigma ^{2}\) with the following conditional distribution:
If \(x\sim \text {Inverse Gamma }(\alpha ,\beta )\) with the shape parameter \(\alpha\) and scale parameter \(\beta\), then it has the following density function:
We can compare the conditional density (33) with (34) to find:
Sample \(\lambda _{j}^{2}\)
To sample the penalty term \(\varvec{\lambda }\), we have developed three methods providing flexibility depending on the network model requirements (Table 2).
Method A: The simplest prior for the penalty term \(\lambda _j\), for \(j=1,2,3,\cdots ,p\) would be a uniform distribution, but this proved to be problematic with complex networks, particularly when a model has many parameters. Thus, following the notation of Park and Casella (2008), we propose an adaptive prior such that \(\lambda _{j}^{2} \sim \text { Gamma } (r, \delta _{j})\). That is,
For the full Bayes estimation of \(\lambda _{j}^{2}\), we have to find the following distribution.
This shows us that \(\lambda _j^{2}\) is proportional to a gamma distribution with \(\alpha = r + 1\) and \(\beta = \frac{\tau _{j}^{2}}{2} + \delta _{j}\), since a standard gamma probability density function is \(\displaystyle f(x) = \frac{\beta ^{\alpha }}{\Gamma (\alpha )} x^{\alpha - 1}e^{-\beta x}\).
Therefore we can conclude:
where \(r \text { and } \varvec{\delta }\) are chosen constants/vectors of constants.
Method B: In contrast to the previous Method A, where the parameters \(\delta _j\), for \(j=1,2,\cdots ,p\), were treated as fixed constants, the proposed method incorporates an empirical update of the hyperparameter vector \(\varvec{\delta }\) using the Monte Carlo Expectation-Maximization (E-M) algorithm (Levine and Casella 2001). The empirical update of the parameters \(\delta _j\) is performed using the following formula:
The full derivation of this method is presented in Appendix B.
The empirical update of the parameters \(\delta _j\) using the E-M algorithm brings several advantages to the estimation process. Firstly, it eliminates the need for manually specifying appropriate hyperparameter values, as the parameter values are estimated directly from the observed data. This data-driven approach enables the selection of hyperparameters based on the characteristics of the data, enhancing the flexibility and adaptability of the model. Additionally, the empirical update of the parameters \(\delta _j\) allows the model to capture intricate nuances and complexities that may not be adequately accounted for by Method 1, which relies on a fixed hyperparameter. By updating the parameters based on the observed data, the model can better capture the intricacies and variability present in the data, leading to improved estimation accuracy and model performance.
Method C: This method uses a full empirical Bayes that directly estimates \(\varvec{\lambda }\) from observed data without assuming any specific distribution or model. The full derivation is in the Appendix B, but we can update \(\lambda _{j}^{2}\)
While this Method C offers several advantages, such as adapting to the data and improving exploration of the parameter space, they also have certain disadvantages that should be considered.
One of the primary disadvantages of full empirical MCMC is its computational cost. Empirical MCMC methods typically require additional iterations and computations compared to traditional MCMC algorithms. The empirical updates of parameters or proposal distributions can be computationally intensive, particularly when dealing with large datasets or complex models. This can result in longer execution times, limiting the scalability of the method.
Another disadvantage is the potential for bias or inefficiency in the estimation process. Empirical updates rely on the observed network data to estimate the parameters and the proposal distribution of the network. If the nodal sufficient statistics are not fully representative or the observations of nodal random variables contain outliers, the empirical estimates may introduce biases or inefficiencies in the MCMC sampling. Additionally, the convergence of this method 3 needs careful tuning of the other hyperparameters to achieve optimal performance. The optimization of hyperparameters can be nontrivial and needs expert knowledge or extensive experimentation.
Adaptive parallel direction sampling algorithm
There have been considerable developments in the approaches dealing with the problem of sampling from a distribution with a doubly intractable normalizing constant. For example, the easy-to-implement and more direct single variable exchange algorithm proposed by Murray et al. (2012). However, if there is strong temporal dependence in the state process and a strong correlation between model parameters, the exchange algorithm performs slow mixing. Caimo and Friel (2011) and Caimo and Mira (2015) apply the ideas in Murray et al. (2012) to increase MCMC sampling efficiency by combining delayed rejection and adaptive Monte Carlo techniques. First, a collection of H parallel Markov chains are generated. Then the next element of a current chain \(h\) is found using estimates from chains \(h_{1}\) and \(h_{2}\) as below.
Algorithm: Parallel Adaptive Sampling Algorithm | |
---|---|
while \(i = 1, . . . , N\) do | |
Define a scalar ADS move factor \(\gamma\), for each chain \({\varvec{h}}\in \{1,2,3,\cdots , H\}\): | |
1. Sample two current states \(h_1,h_2\) and \(h_1\ne h_2\ne h\). | |
2. Sample the error term from a symmetric normal distribution. \(\varvec{\epsilon }\sim N({\varvec{0}},\varvec{\sigma }_{\epsilon }^2)\). | |
3. The sampling of \(\varvec{\theta }_h\) performs a simple random walk: \(\varvec{\theta }_h^{'}=\varvec{\theta }_h+\gamma (\varvec{\theta }_{h_1}-\varvec{\theta }_{h_2})+\varvec{\epsilon }\). | |
4. Sample \({\varvec{y}}'\) from \(\pi (\cdot |\varvec{\theta }_h^{'})\). | |
5. Accept \(\varvec{\theta }_h^{'}\) with probability \(\min (1,\frac{q({\varvec{y}}|\varvec{\theta }'_h )\pi (\varvec{\theta }'_h)q({\varvec{y}}'|\varvec{\theta }_h)}{q({\varvec{y}}|\varvec{\theta }_h)\pi (\varvec{\theta }_h)q({\varvec{y}}'|\varvec{\theta }'_h)})\) (42) | |
where \(q({\varvec{y}}|\varvec{\theta })=e^{\varvec{\theta }^{T} s({\varvec{y}})}\) is the unnormalized likelihood. | |
end while |
The move of \(\varvec{\theta }\) is illustrated in Figure 1. Here, two other chains \(h_{1}\) and \(h_{2}\) are chosen at random. The difference between the corresponding estimates in the other two chains \(\varvec{\theta }_{h_1}\) and \(\varvec{\theta }_{h_2}\) are used to find the distance to move away from \(\varvec{\theta }_{h}\). A normal distribution with a very small variance is used to slightly adjust the estimate for the new \(\varvec{\theta }\).
Bayesian adaptive lasso algorithm
In this section, we will list the algorithm of the Bayesian Exponential Random Graph Model (BERGM) by Caimo et al. (2017) and the algorithm of our Bayesian Adaptive Lasso Exponential Random Graph Model (BALERGM)) for easy comparison. Caimo et al. (2017) set up the exchange algorithm with a Gibbs update of \(\varvec{\theta }'\) and then \({\varvec{y}}'\) using Markov Chain Monte Carlo iteration without penalized terms. The algorithm can be written in the following concise way:
Algorithm: Bayesian Exponential Random Graph Model | |
---|---|
while \(i = 1, . . . , N\) do | |
while \(h = 1, . . ., H\) do | |
1. generate \(h_{1}\) and \(h_{2}\) such that \(h_{1} \ne h_{2} \ne h\) | |
2. generate \(\varvec{\theta }_{h}'\) from \(\gamma (\varvec{\theta }_{h_{1}} - \varvec{\theta }_{h_{2}}) + \epsilon ( \ddots |\varvec{\theta }_{h})\) | |
3. simulate \(y'\) from \(\pi ( \ddots |\varvec{\theta }_{h}')\) | |
4. update \(\varvec{\theta }_{h} \rightarrow \varvec{\theta }_{h}'\) with the log of the probability | |
\(\text {min} \left( 0, [\varvec{\theta }_{h} - \varvec{\theta }_{h}']^{T} [s({\varvec{y}}') - s({\varvec{y}})] + \log \left[ \frac{\pi (\varvec{\theta }_{h}')}{\pi (\varvec{\theta }_{h})} \right] \right)\) | |
end while | |
end while |
Where \(s({\varvec{y}})\) and \(s({\varvec{y}}')\) are functions of the observed and simulated vector of network statistics respectively.
For the new Bayesian Adaptive Lasso model, we use the parallel adaptive direction sampler method suggested by BERGM and combine with Gibbs sampling to generate samples to find estimates for \(\varvec{\theta }\).
Algorithm: Bayesian Adaptive Lasso Exponential Random Graph Model Algorithm | |
---|---|
Require: Set the initial value for \(\varvec{\lambda }, \sigma ^{2}, \gamma\), Use ERGM to find MPLE (Maximizer to the Psuedolikelihood Function) to find initial values for \(\varvec{\theta }\). Denote samples of \(\varvec{\theta }\) in the \(h\)th chain, as \(\varvec{\theta }_{h}\). | |
while \(i = 1, . . . , N\) do | |
while \(h = 1, . . ., H\) do | |
1. sample \(\varvec{\theta }_{h}\) with Parallel Adaptive Direction Sampler: | |
a. generate \(h_{1}\) and \(h_{2}\) such that \(h_{1} \ne h_{2} \ne h\) | |
b. update \({\varvec{D}}_{\varvec{\tau }}^{-1}\) | |
c. generate \(\varvec{\theta }_{h}'\) from \(\gamma (\varvec{\theta }_{h_{1}} - \varvec{\theta }_{h_{2}}) + \epsilon ( \ddots |\varvec{\theta }_{h})\) | |
d. simulate \({\varvec{y}}'\) from \(\pi ( \ddots |\varvec{\theta }_{h}')\) | |
e. update \(\varvec{\theta }_{h} \rightarrow \varvec{\theta }_h'\) with the log of the probability | |
\(\text {min} \left( 0, [\varvec{\theta }_h - \varvec{\theta }_h']^{T} [s({\varvec{y}}') - s({\varvec{y}})] + \log \left[ \frac{\pi (\varvec{\theta }_h')}{\pi (\varvec{\theta }_h)} \right] \right)\) | |
where \(\pi (\varvec{\theta }) \sim {\mathcal {N}}(0_{p},\,\sigma ^{2}{\varvec{D}}_{\varvec{\tau }} )\) | |
2. sample \(\sigma ^{2} \text {from Inverse Gaussian} ( \frac{p}{2}, -\frac{1}{2}\varvec{\theta }^{T} {\varvec{D}}_{\varvec{\tau }}^{-1} \varvec{\theta } )\) | |
3. sample \(\tau _{j}^{2} \text { for } j = 1, 2, 3, . . ., p \text { from Inverse Gaussian } \left( \sqrt{\frac{\lambda _{j}^{2} \sigma ^{2}}{\theta _{j}^{2}} }, \lambda _{j}^{2} \right)\) | |
4a. full Bayes update of \(\varvec{\lambda }\) | |
1. sample \({\lambda }_{j}^{2} \text { for } j = 1, 2, 3, . . ., p \text { from Gamma } ( r + 1, \frac{\tau _{j}^{2}}{2} + \delta _{j} )\) | |
OR | |
4b. empirical update of \(\varvec{\delta }\) and update of \(\varvec{\lambda }\) | |
1. update \(\delta _{j} \text { for } j = 1, 2, 3, . . ., p\) with the mean of the last five \(\varvec{\lambda }\) samples estimating the expected value. | |
\(\delta _{j} = \frac{r}{{\textbf{E}}_{\delta _{j}^{(k - 1)}} \left[ \lambda _{j}^{2} | \delta _{j}^{(k -1)}, y^{(k -1)} \right] }\) | |
2. sample \({\lambda }_{j}^{2} \text { for } j = 1, 2, 3, . . ., p \text { from Gamma } ( r + 1, \frac{\tau _{j}^{2}}{2} + \delta _{j} )\) | |
OR | |
4c. full empirical update of \(\varvec{\lambda }\) | |
1. update \(\varvec{\lambda }\) with the mean of the last five \(\varvec{\tau }\) estimating the expected value | |
\(\lambda _{j}^{2} = \frac{r}{ {\textbf{E}}_{\varvec{\lambda }_{j}^{(k - 1)}} \left[ \frac{\tau _{j}^{2}}{2} | \lambda _{j}^{(k -1)}, y^{(k -1)} \right] + \delta _{j} }\) | |
end while | |
end while |
This code was built with R version 4.1.1 (2021-08-10). R Core Team (2021) The following package versions were also used: coda 0.19-4, mcmc 0.9-7, Bergm 5.0.3, ergm.count 4.0.2, ergm 4.1.2, mvtnorm 1.1-3.
This BALERGM package is shared on Github:xxxx. (The link will be provided upon the acceptance of this paper).
Simulation and application
In this section, we will show the strength of BALERGM in three key ways. The first way uses the Faux Dixon High School data set to simulate 100 graphs to compare BERGM and BALERGM. The results of trials shows BALERGM is a stable model with accurate estimation, in addition to providing improvements to BERGM with a higher acceptance rate and effective sampling size, and lower MSE. The next two ways showcase the parameter selection abilities of BALERGM. The first on a simulated parameter and the second with network data collected in a study by Prevention Research Center at USC and Sumter County Active Lifestyles (SCAL).
Data
The network object Faux Dixon High represents a friendship network among junior high and high school students based on data gathered by a National Longitudinal Study of Adolescent Health, see details in Resnick et al. (1997). This study, first conducted in 1994–1995, considered more than 90,0000 American students. Students were asked to list friends, and a tie is formed between them in the network if both students claimed friendship (Goodreau et al. 2008).
The final network has 248 nodes with 1,197 directed edges. Each node has three characteristics: grade, sex, and race. The grades include 7th-12th and race is first delineated by Hispanic and non-Hispanic which was further split into Asian, Black, Native American, Other, and White. Figure 2 shows the network plotted with nodes colored for each grade showing the homophily.
Executing the BALERGM algorithm requires choosing network statistics with both nodal and edge attributes and structural features such as triangles and triads (Morris et al. 2008). The count of these network statistics is found with the adjacency matrix realization \({\varvec{y}}\) of \({\varvec{Y}}\) with i-j entry in the matrix defined as \(y_{ij}\). For a directed network, the following summations demonstrate the counting procedure.
A natural network statistic for this data is the instances of homophily between students in the same grade, since as seen in Fig. 2, nodes with the same attribute (in this case grade) appear visually to have more connections. As seen in Table 3, with the diagonal entries of the mixing matrix from Grade \(i\) to Grade \(i\) for \(i \in \{7, 8, 9, 10, 11, 12\}\), most of the connections are between students in the same grade. This feature can be included in network models with the R code nodematch.
Simulation
To demonstrate the overall effectiveness of BALERGM, we conducted a comparative analysis between BALERGM and BERGM (Caimo and Friel 2014). Our evaluation involved the generation of 100 independent exponential random graphs using the Faux Dixon High dataset, with known and fixed parameters \(\varvec{\theta }\). Specifically, we focused on two selected network statistics: the count of edges in the network ( \(\theta _{1}:\) edges) and the count of occurrences of homophily, where students of the same grade have a friendship connection ( \(\theta _{2}:\) nodematch.Grade). Without loss of generality, we fixed the parameter values \(\varvec{\theta }=(-4.8, 2.3)\) and generated 100 independent exponential random graphs based on the Faux Dixon High dataset, considering them as new instances with associated node attributes. This approach allowed us to create 100 distinct opportunities to estimate the parameter vector \(\varvec{\theta }\) using both the BALERGM and BERGM algorithms, enabling a comprehensive performance comparison against the true parameter values \(\varvec{\theta }=(-4.8, 2.3)\).
In each run of BERGM and BALERGM, the main chain for either model consists of 2000 iterations and the burn-in number is 50 iterations. In 100 simulations, each model generates a sequence of values estimating each \(\varvec{\theta }\) in each simulation. To confirm the stability of the model, the following representation of the MCMC results (Fig. 3) shows the strength and stability of the BALERGM algorithm after relatively few iterations. The unimodal distribution of estimates is on the left, and the center column shows the trace of the estimates indicating a stable estimating process. The final column shows the autocorrelation plot with the lag decreasing quickly; by 50 iterations, the process has stabilized to minimal lag.
Using both the mean and median of these estimates we can compare several metrics. Table 4 compares the acceptance rate of generated estimates for each run, the mean effective sample size, and both the mean and median square error (MSE) of the estimates compared to the chosen true values, where \(MSE=\frac{1}{n}{\varvec{e}}^{\top }{\varvec{e}}\), \({\varvec{e}}\) is the error vector, that is \({\varvec{e}}=\hat{\varvec{\theta }}-(-4.8, 2.3)\).
Table 4 shows that using either the mean or median of the generated estimates in MCMC for \(\varvec{\theta }\) (1) BALERGM has a better overall acceptance rate and effective sample size on average than BERGM. The acceptance rate or the percentage of generated samples that are accepted in the MCMC process is increased. This implies BALERGM adjusts to the true parameter for each single variable faster than BERGM. (2) BALERGM offers an improvement over BERGM with a lower mean squared error (MSE). The mean squared error is dramatically lower with the BALERGM process no matter whether the mean or median in MCMC is used as the estimate for \(\varvec{\theta }\). This can be seen in the quantiles for each estimate of \(\varvec{\theta }\) since the true values are \(\varvec{\theta } = (-4.8, 2.3)\), the BALERGM estimates are much closer to these true values.
In Table 5, the true known value of each \(\varvec{\theta }\) is estimated by either the mean or median of the generated samples. The quantiles for estimates of \(\varvec{\theta }\) show the spread of each estimate.
Once results are generated, the estimates produced can be used to calculate the probability of a tie, using the \(\varvec{\theta }\). Using the previous example, if no new tie is created, so the change statistic for \(\theta _{1}\) is zero, then the probability that a tie is between students of the same grade can be calculated as follows.
That means the conditional probability of observing an edge (not involved in the creation of other network statistics included in the model) is about \(94.56\%\).
Variable selection
BALERGM not only improves sampling efficiency compared to previous models but also demonstrates strong performance in variable selection through its adaptive lasso component. This indicates the ability of the model to identify and highlight parameters that are either more or less significant to the network structure. The example using the following simulated dataset showcases the effectiveness of BALERGM in terms of variable selection.
For this example, we still use Faux Dixon high school dataset. The chosen network statistics are the count of the edges in the network (edges), the counts of the occurrences of homophily where students of the same grade have a friendship connection (nodematch.Grade), and the third artificial created term: the counts of the occurrences of homophily from a generated nodal attribute for the wealth of a parent (nodemath.Wealth). This additional nodal attribute “Wealth” is generated from the uniform distribution on 20 and 75. Given that this nodal variable is generated uniformly at random, it is intentionally designed to have no impact on the network structure. Our objective is to test whether BALERGM can effectively identify and exclude this artificially created insignificant nodal variable. Running the BALERGM algorithm produces the following results, demonstrating that the model accurately estimates the value of \(\theta _3\) to be close to zero. This outcome aligns with our expectations, as the variable has no meaningful influence on the network structure (Table 6, Fig. 4)
.
Sumter county active lifestyles (SCAL) network analysis
With reports by The US Burden of Disease Collaborators (2018) of worsening metrics of American health, communities are working on addressing and understanding the factors that might improve health outcomes. To this end, the University of South Carolina Prevention Research Center and Sumter County Active Lifestyles (SCAL) based in Sumter County, South Carolina conducted a respondent-driven sampling study in 2014 to better understand the dynamics of social networks and health outcomes.
In this study, community ambassadors chosen for their history of community involvement were given a set compensation for their participation. Each ambassador was instructed to share the survey with those in their social network. Each of these respondents was also compensated for both completion of the survey and sharing the survey with others that completed the survey. Using referral codes, a network can be created with nodes representing survey respondents, edges formed by survey sharing, and nodal characteristics from the results of the survey. The final network has 80 nodes with the data for 30 questions for each respondent. Figure 5 is the network plot labeled with one of the 30 questions: “Have you heard of a group called Sumter County Active Lifestyles (SCAL)?”
The survey was intended to be a brief but broad look at self-reported health benchmarks. Questions cover demographic characteristics revealing that the respondents are primarily white (87%), female (78%), likely to be older than 50 (44%), and more educated with 46% being college graduates. Other questions focused on self-reported health outcomes and activities including exercise habits, eating habits, and social support dynamics. The question forms included qualitative questions about physical activities and opportunities for physical activities in the community. For the purposes of this network, network attributes were assigned using the answers to only multiple-choice questions.
The resulting network contains many nodal attributes where ERGM and BERGM cannot be applied effectively. This motivates a model like BALERGM which enables understanding which of these network statistics contribute less to the network structure.
Results
Using the SCAL data set from the previous section, we use the network statistics in Table 7 to analyze this model, where we use the ergm terms “nodematch”, “nodefactor” and “nodecov”. These three terms all provide measures of homophily. “nodematch” counts the instances of nodes with the same attribute for a given attribute. “nodecov” performs a similar function but for continuous variables. “nodefactor” creates network statistics for each discrete level of a nodal attribute and counts the occurrences of connected nodes with the same attribute level. For more details, see Morris et al. (2008).
Table 7 shows the BALERGM output on the SCAL social network. Here the sparsity of the network can be seen in the large negative values for the network statistics for edges and the out-degree of the nodes. While the standard deviations vary with each estimate, the MCMC outputs show stable estimating with symmetric distributions as the quantile values indicate. It 7 provides valuable insights into the relationships between different variables in the network analysis. One interesting observation is that individuals who maintain a healthy diet ( \(\theta _6\)-\(\theta _{13}\) are all positive) tend to have positive connections with each other. This suggests a clustering effect among individuals with similar dietary habits, indicating a potential influence of shared health-conscious behaviors on network connections.
Furthermore, the result highlights that participation in a walking program (variables \(\theta _{18}\) to \(\theta _{19}\)) is positively associated with network connections. This implies that individuals who engage in walking programs are more likely to know each other within the network. This finding suggests a potential social bonding effect among individuals who actively participate in health-promoting activities, leading to the formation of connections and social ties.
The adaptive lasso penalty in BALERGM is useful for shrinking \(\varvec{\theta }\) values for network statistics that are less significant to the network structures. Depending on the model and network conditions, the parameter estimate might not reach exactly zero. For example, the estimate for both \(\theta _{26} = -.001\) and \(\theta _{20} = -.076\) are small, but this mean of the generated samples as the single factor utilized doesn’t allow for a nuanced ranking of how significant each parameter is. Using the distribution of \(\varvec{\theta }\) found in the Gibbs sampling process, we can find the probability that half of the distribution is less than zero at some significance level \(\alpha\).
A larger value of \(\alpha\) indicates a higher importance of the variable in the context of the model. This creates the ability to rank variables. The following Table 8 shows the parameters less significant to the construct of the network at various significance levels. For example, the network statistic of the age of the participant (\(\theta _{26}\)) is less significant than for the network statistic of having heard of the SCAL program (\(\theta _{20}\)). While both are not the primary factors, BALERGM gives researchers insights into the social dynamics of Sumter County allowing for targeted programs to improve health outcomes.
This example highlights the powerful functionalities of BALERGM, particularly in the context of variable selection and importance ranking in network analysis. In network studies, the presence of numerous network variables is common. The identification of the most relevant variables is crucial as it enables researchers to concentrate their analysis and interpretation on the factors that significantly influence the network’s structure and behavior. By focusing on these key variables, we can gain a deeper understanding of the underlying mechanisms that drive network dynamics.
Goodness of fit
To assess the performance and goodness of fit of Exponential Random Graph Models (ERGMs), various diagnostics can be employed. These diagnostics involve comparing key statistical measures calculated from observed networks with those obtained from simulated networks generated based on the estimated network parameters. In the Bayesian framework, evaluating the goodness of fit of the model involves conducting posterior predictive assessments. This entails comparing the observed network to a collection of networks simulated from the posterior distribution of the model’s parameter estimates, as determined by Caimo and Friel (2011).
The set of statistics used for the comparison contains degree distributions, the minimum geodesic distance, and the number of edge-wise shared partners. Since the SCAL network graph is a directed graph, the degree distributions for both in and out degrees are included. Since the graph includes isolated nodes and clusters such that there is no path between some nodes, the minimum geodesic distance or the minimum number of edges needed to connect any two nodes is infinite leading to the spike in the plot for minimum geodesic distance in 6. Finally, the edge-wise shared partners are concentrated in the lower values since the number of nodes in common for any number of edges is small.
The Bayesian goodness of fit diagnostic in Fig. 6 evaluates the implemented model in section 8. The observed network is compared with 300 randomly simulated network samples drawn from the estimated posterior distribution using 50 auxiliary iterations for the network simulation step. Figure 6 illustrates the summary results of these 300 generated graphs in black and gray, alongside the original network represented in red. The comparison reveals a strong alignment in the high-level characteristics that are not explicitly modeled. This indicates that the posterior mean obtained through BALERGM accurately generates networks with corresponding structures.
Discussion
Bayesian adaptive lasso exponential random graph model (BALERGM) offers several notable advantages in the field of network analysis. One key advantage is its ability to perform automatic variable selection, facilitated by the integration of the Lasso regularization technique. By employing the Lasso penalty, BALERGM effectively identifies and emphasizes the most relevant network parameters while diminishing the influence of less significant ones towards zero. This feature streamlines the modeling process and extracts valuable insights from intricate network data, enhancing the interpretability of the results. Moreover, the Lasso penalty promotes sparsity in parameter estimates, resulting in a more parsimonious model that aids in discerning the influential factors governing network behavior.
Another compelling advantage of BALERGM is its superior adaptive estimation performance. Through the adaptive adjustment of penalties for each parameter, the model swiftly adapts to the data, allowing it to focus on the most relevant network parameters and capture underlying patterns and relationships more effectively. Researchers can readily select key network factors based on their significance levels, providing valuable insights and actionable knowledge.
We have shown the effectiveness of the the proposed algorithm in simulation and compared its performance against the currently popular BERGM method. One promising direction for future work involves the development of more generalized penalized forms within the context of network analysis. While the Lasso penalty has demonstrated its efficacy in variable selection and sparsity promotion, the incorporation of ridge penalty distributions could offer additional benefits. Combining the strengths of both Lasso and ridge penalties would strike a balance between model complexity and over/underfitting issues, leading to more robust parameter estimation.
Availability of data and materials
The Faux Dixon High dataset is avlaible in ERGM R package (Hunter et al. 2008). The SCAL dataset can be requested from the University of South Carolina Prevention Research Center: http://prevention.sph.sc.edu/Resources/SCOTM.html by filling out an interest form. The R codes used during the current study are available in the GitHub repository https://github.com/dan-han-mathematics/Bergm-lasso.git.
References
Alhamzawi R, Ali HTM (2018) The Bayesian adaptive lasso regression. Math Biosci 303:75–82. https://doi.org/10.1016/j.mbs.2018.06.004
Andrews DF, Mallows CL (1974) Scale mixtures of normal distributions. J Roy Stat Soc: Ser B (Methodol) 36(1):99–102
Baggio S, Luisier V, Vladescu C (2017) Relationships between social networks and mental health. Swiss J Psychol 76(1):5–11. https://doi.org/10.1024/1421-0185/a000186
Becker KR, Stojek MM, Clifton A, Miller JD (2018) Disordered eating in college sorority women: a social network analysis of a subset of members from a single sorority chapter. Appetite 128:180–187. https://doi.org/10.1016/j.appet.2018.06.013
Caimo A, Bouranis L, Krause R, Friel N (2022) Statistical network analysis with bergm. J Stat Softw 104(1):1–23. https://doi.org/10.18637/jss.v104.i01
Caimo A, Friel N (2011) Bayesian inference for exponential random graph models. Soc Netw 33(1):41–55. https://doi.org/10.1016/j.socnet.2010.09.004
Caimo A, Friel N (2013) Bayesian model selection for exponential random graph models. Soc Netw 35(1):11–24. https://doi.org/10.1016/j.socnet.2012.10.003
Caimo A, Friel N (2014) Bergm: Bayesian exponential random graphs in R. J Stat Softw 61:1–25. https://doi.org/10.18637/jss.v061.i02
Caimo A, Mira A (2015) Efficient computational strategies for doubly intractable problems with applications to Bayesian social networks. Stat Comput 25(1):113–125
Caimo A, Pallotti F, Lomi A (2017) Bayesian exponential random graph modelling of interhospital patient referral networks. Stat Med 36(18):2902–2920. https://doi.org/10.1002/sim.7301
Chatterjee S, Diaconis P (2013) Estimating and understanding exponential random graph models. Ann Stat 41(5):2428–2461. https://doi.org/10.1214/13-AOS1155
Chhikara RS, Folks L (1988) The Inverse Gaussian Distribution: Theory, Methodology, and Applications. CRC Press
Erdös P, Rényi A (1959) On random graphs I. Publicationes Mathematicae Debrecen 6:290
Fan J, Feng Y, Wu Y (2009) Network exploration via the adaptive LASSO and SCAD penalties. Ann Appl Stat 3(2):521–541. https://doi.org/10.1214/08-AOAS215
Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96(456):1348–1360
Frank O, Strauss D (1986) Markov Graphs. J Am Stat Assoc 81(395):832–842
Friedman J, Tibshirani R, Hastie T (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33(1):1–22. https://doi.org/10.18637/jss.v033.i01
Friel N, Pettitt A, Reeves R, Wit E (2009) 06. Bayesian Inference in Hidden Markov Random Fields for Binary Data Defined on Large Lattices. J Comput Graph Stat 18:243–261. https://doi.org/10.1198/jcgs.2009.06148
Geyer CJ (1991) Markov chain monte carlo maximum likelihood
Goodreau SM, Handcock MS, Hunter DR, Butts CT, Morris M (2008) A statnet Tutorial. J Stat Softw 24(9):1–26. https://doi.org/10.18637/jss.v024.i09
Handcock MS (2003) Assessing degeneracy in statistical models of social networks. Technical report, Working paper
Holland PW, Leinhardt S (1981) An exponential family of probability distributions for directed graphs. J Am Stat Assoc 76(373):33–50
Hunter D, Handcock M, Butts C, Goodreau S, Morris M (2008) Ergm: a package to fit, simulate and diagnose exponential-family models for networks. J Stat Softw 24(3):1–29. https://doi.org/10.18637/jss.v024.i03
Leng C, Tran MN, Nott D (2014) Bayesian adaptive lasso. Ann Inst Stat Math 66(2):221–244
Levine RA, Casella G (2001) Implementations of the Monte Carlo em algorithm. J Comput Graph Stat 10(3):422–439
Lusher D, Koskinen J, Robins G (2013) Exponential random graph models for social networks: theory, methods, and applications. Cambridge University Press
Meinshausen N, Bühlmann P (2006) High-dimensional graphs and variable selection with the lasso. Ann Stat 34(3):1436–1462
Morris M, Handcock MS, Hunter DR (2008) Specification of exponential-family random graph models: terms and computational aspects. J Stat Softw 24(4):1–24. https://doi.org/10.18637/jss.v024.i04
Munkres JR, Davis L (2018) Topology. Pearson Prentice Hall
Murray I, Ghahramani Z, MacKay D (2012) MCMC for doubly-intractable distributions
Park T, Casella G (2008) The Bayesian Lasso. J Am Stat Assoc 103(482):681–686. https://doi.org/10.1198/016214508000000337
R Core Team (2021) R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing
Resnick M, Bearman P, Blum R, Bauman K, Harris K, Jones J, Tabor J, Beuhring T, Sieving R, Shew M, Ireland M, Bearinger L, Udry J (1997) Protecting adolescents from harm Findings from the National Longitudinal Study on Adolescent Health. JAMA 278(10):823–32
Shojaie A (2013) Link prediction in biological networks using multi-mode exponential random graph models. In: 11th Workshop on Mining and Learning with Graphs, pp 987–991. Citeseer
Shojaie A, Basu S, Michailidis G (2012) Adaptive thresholding for reconstructing regulatory networks from time-course gene expression data. Stat Biosci 4(1):66–83
Shojaie A, Michailidis G (2010) Penalized likelihood methods for estimation of sparse high-dimensional directed acyclic graphs. Biometrika 97(3):519–538
Solo V, Poline JB, Lindquist MA, Simpson SL, Bowman FD, Chung MK, Cassidy B (2018) Connectivity in fMRI: Blind spots and Breakthroughs. IEEE Trans Med Imaging 37(7):1537–1550. https://doi.org/10.1109/tmi.2018.2831261
Stivala A, Lomi A (2021) Testing biological network motif significance with exponential random graph models. Appl Netw Sci. https://doi.org/10.1007/s41109-021-00434-y
Strauss D, Ikeda M (1990) Pseudolikelihood estimation for social networks. J Am Stat Assoc 85(409):204–212
Tay JK, Narasimhan B, Hastie T (2023) Elastic net regularization paths for all generalized linear models. J Stat Softw 106:1
The US Burden of Disease Collaborators (2018) The State of US Health, 1990–2016: burden of diseases, injuries, and risk factors among US states. JAMA 319(14):1444–1472. https://doi.org/10.1001/jama.2018.0158
Tibshirani R (1996) Regression shrinkage and selection via the Lasso. J Roy Stat Soc: Ser B (Methodol) 58(1):267–288
van Duijn MA, Gile KJ, Handcock MS (2009) A framework for the comparison of maximum pseudo-likelihood and maximum likelihood estimation of exponential family random graph models. Social Networks 31(1):52–62. https://doi.org/10.1016/j.socnet.2008.10.003
Wang H, Leng C (2008) A note on adaptive group lasso. Comput Stat Data Anal 52(12):5277–5286
Williams NL, Hristov D (2018) An examination of DMO network identity using Exponential Random Graph Models. Tour Manage 68:177–186. https://doi.org/10.1016/j.tourman.2018.03.014
Yuan M, Lin Y (2006) Model selection and estimation in regression with grouped variables. J R Stat Soc Series B (Statistical Methodology) 68(1):49–67
Zou H (2006) The Adaptive Lasso and its Oracle Properties. J Am Stat Assoc 101(476):1418–1429
Funding
This work was not supported by any funding.
Author information
Authors and Affiliations
Contributions
RP and DH conceived the research. MF collected the data. DH developed the mathematical model and designed the MCMC algorithm. DH and VM generated the code and analyzed the data. DH and VM drafted the first version of the manuscript with input from all authors. All authors contributed to the critical revision of the manuscript for important intellectual content. All authors have seen and approved the final version and agreed to its publication. DH and VM had full access to all the data in the study and take responsibility for the accuracy of the mathematical analysis.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not Applicable.
Competing interests
The authors declare no competing fnancial or non-fnancial interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: proof of unimodal posterior
The chosen prior needs to result in a unimodal posterior for faster Gibbs sample convergence and confidence that the estimates found are actually best.
Theorem:
The joint posterior distribution is unimodal for typical choices of \(\pi (\sigma ^{2})\) and any choice of \(\lambda \ge 0\).
Proof:
We begin by representing the joint distribution of \(\theta\) and \(\sigma ^{2} > 0\) using distributions already defined.
We have choose the prior such that \(\pi (\sigma ^{2}) \propto \frac{1}{\sigma ^{2}}\) according to the recommendation of the literature. Park and Casella (2008)
We wish to show that the posterior is unimodal in the sense that every upper-level set of \(\{(\varvec{\theta }, \sigma ^{2}) | \pi ( \varvec{\theta }, \sigma ^{2})> x, \sigma ^{2}>0 \}, \text { for } x > 0\) is connected. We will show this is true under a continuous transform with continuous inverse since the continuous image of a connected set is connected. Munkres and Davis (2018)
The posterior is shown here
We now take the natural log of the equation above.
The following transform allows for easier calculation.
This is continuous with a continuous inverse when \(0< \sigma ^{2} < \infty\), so the upper-level sets for the original parameters correspond under the transformation to upper-level sets for the original parameters.
Let \(\varvec{\phi } = (\phi _{1}, \phi _{2},... \phi _{p})^{T}\) be the column vector for ease of notation. This transform is one-to-one and continuous for \(0< \sigma ^{2} < \infty\), therefore the unimodality of the transformed equation is equivalent to the unimodality of the original equation.
Using the transform and algebra we get the following expression
We can show that (A8) is unimodal by showing it is a concave function in \((\varvec{\phi }, \rho )\). We will do that by considering each term of the equation in turn.
We will determine the concavity of the first two functions by checking the spectral property of the corresponding Hessian matrix.
For the first term and the second term \(h_{1} = \ln (\rho )\) and \(h_{2} = \frac{\varvec{\phi }^{T}s({\varvec{y}})}{\rho }\), the corresponding Hessian matrix \(H_{h_1}\) and \(H_{h_2}\) are both negative semi-definite and thus \(h_{1}\) and \(h_{2}\) are both concave in \((\varvec{\phi }, \rho )\).
For the third term \(h_{3} = - \sum _{j = 1}^{p} \lambda _{j}|\phi _{j}|\), we see this is a sum of the negative of a constant times an absolute value function. This is a concave function in \(\varvec{\phi }, \rho\), since the j the term in \(h_3\) is \(h_{3}(j) = - \lambda _{j}|\phi _{j}|\) which is a concave function of \(\phi _{j}\) and the sum of concave functions is a concave function.
Using the same reasoning that the sum of concave functions is concave gives that (A8) is concave, and hence the posterior distribution is concave.
Therefore, we can conclude that our posterior distribution is unimodal.
Appendix B: empirical bayes
The Monte Carlo Expectation-maximization algorithm for empirical Bayes estimation of hyperparameters proposed by Levine and Casella (2001) essentially treats the parameters as missing data and then uses the E-M algorithm to iteratively approximate the hyperparameters substituting Monte Carlo estimates for any expected values that cannot be computed explicitly. For BALERGM, the Gibbs sampler is used to estimate the expected values.
Method B: Estimating \(\delta _{j}\)
To begin this process, we consider the part of the joint distribution that depends on \(\varvec{\delta }\), since when taking the derivative all other terms will become zero.
We then take the natural log of the resulting equation.
1. Expectation step
2. Maximization step
Solve
we get
Method C: Estimating \(\lambda _{j}\)
The empirical process of estimating \(\lambda _{j}\) begins with the joint distribution terms that depend on \(\lambda _{j}\).
Next, we take the natural log
1. Expectation step
2. Maximization step
Solve
we get
Thus these conditional expectations are just the posterior expectations under the hyperparameter \(\lambda ^{(k-1)}\) thus they can be estimated using the sample averages from a run of the Gibbs sampler described in the section.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Han, D., Modisette, V., Forthofer, M. et al. Hierarchical Bayesian adaptive lasso methods on exponential random graph models. Appl Netw Sci 9, 9 (2024). https://doi.org/10.1007/s41109-024-00619-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s41109-024-00619-1