A Review of MixedEffect Modeling in the Longitudinal Studies Using Medical Images of Patients
Abstract
In this review paper, some applications of the mixed effect modeling in medial image processing and longitudinal analysis is studied. For this purpose, a general structure is extracted from some of the researches in the literature. This structure includes a number of essential elements, each of which having a few design choices, namely 1) tracked features, 2) model’s mathematical expression and random effects and finally 3) response prediction. Two research study examples in Alzheimer’s disease and prostate tomography are also briefly introduced to further discuss the above design choices.
Mixed effect model, Medical image processing, Longitudinal studies.
1 Introduction
\IEEEPARstartWith the increasing prevalence of the signal processing applications in the medical science, the need for sophisticated automated processing tools is more perceived. One group of these applications deals with different imaging systems (e.g. MRI, CT, Xray etc.), where one can derive useful information by studying big and diverse datasets of images taken from different patients in different stages of a disease. However, this diversity and plurality of data is helpful only if a proper and accurate automated tool is designed to process them.
Longitudinal study and disease tracking using medical imaging is one of the most important and effective processes during the medical treatment. Generally speaking, this procedure usually requires images regularly taken from patients’ organ(s) involving in disease evolution. The information in these images hopefully guides to make better estimations regarding the future state of the organs and observe the impact of different drugs in different situations. For instance, one might be interested to discover the impact of a specific dosage of an anticancer drug on the size of the tumor through the time. Therefore, the question is how to benefit from the latent information in the temporally diverse set of medical images taken from different patients.
Statistical modeling and prediction is a practical and effective solution to address the above problem. The main idea is to exploit the repetitive behavior of the organ/tumors in response to a new condition (e.g. drug injection, aging etc.) and derive an accurate pattern. Like other similar machine learning and statistical modeling problems, this task requires a diverse image dataset with a reasonable size taken in different conditions. However, it has always been costly and more importantly dangerous to take excessive images from individual patients to specifically model their disease. The most feasible solution to this problem is to combine image datasets from different patients with the same disease condition and produce a rather general model. However, this simplification would compromise accuracy of the model due to the patientspecific behaviors of the treatments.
In this paper, we review the use of a statistical technique called mixedeffect modeling, to take into account patientspecific behaviors. As will be discussed, this technique helps the modeling process improve the accuracy in the longitudinal organ/tumor tracking. The rest of this paper is organized as follows. In section 2, the general framework of a prediction system is described in detail. In section 3, two example research projects with different prediction systems for tracking two different medical phenomenons have been studied. Finally, section 4 concludes the paper with some discussions about the topic.
2 General framework of a prediction system using mixed effect model
In this section, we first explain principles of the mixedeffect modeling and its necessity in the modeling of a complex medical treatment process. Then, the main elements of a prediction system based on the mixedeffect modeling and their different choices are discussed.
2.1 Principles of the mixedeffect modeling
Basically, mixedeffect modeling or mixed modeling is a flexible statistical technique to exploit regularity in the pattern of a phenomenon which consists of both fixed and random effects. Herein, the term effect refers to a system parameter that somehow impact the prediction value. This impact can be either nonrandom (i.e. fixed effects) or random (i.e. random effect) [1] [cite mixed effect.pdf]. In matrix notation, a mixedeffect model can be represented as eq. 1:
(1) 
where and are fixed and random effects of the model and is the prediction error.
2.2 Key elements of a mixed effect modeling system
By taking a look at the longitudinal studies with medical image processing, one can spot a few common essential elements. Here we briefly describe these elements [2].
Tracked features
Depending on the disease under study, the tracked feature can be different. In the problems dealing with different types of cancers, the objective is to observe and model the growth in the size of tumor. “Sum of longest diameters” and “Mean tumor diameter” are two sizerelated features that can fairly represent the status of the tumor and also can be helpful in tracking it. According to the Response Evaluation Criteria in Solid Tumors (RECIST) [3], these features are allowed to be measured and tracked for on a limited set of organs associated with lesions.
However in other problems than the tomography, one might aim at monitoring more complicated features of the organ e.g. shape, displacement etc. For this purpose, some of the researches directly deal with pixels/voxels of the medical images taken from patients. In the simplest scenario, one might use each pixel/voxel of the image as one independent feature to track. More advanced approaches apply socalled feature reduction techniques such as Principle Component Analysis (PCA) to shrink the feature space [4].
There are also some other features that are used for tracking an organ’s status through the time. For instance, in a research for studying the tumor growth rate, the ProstateSpecific Antigen (PSA) was tracked in order to monitor the prostate cancer status [5].
Mixedeffect modelisation
In the design of a mixedeffect model, there are different choices to make. One important choice is the mathematical expression used in modelisation. There are mainly two types of mixedeffect models in the literature:

Algebraic equations with the general form of:
(2) 
Differential equation with the general form of:
(3)
In both equations, is the tracked feature, is time and is the model parameter. In this equation, the term exponential term of is the random effect and the the term is the fixed effect in the model.
Once the mathematical expression is decided, one should decide about the number and the nature of random effects of the model. For example, here is a list of some popular random effects used in the literature of anticancer drug treatment:

Drugindudec decay of tumor

Net growth of tumor size

Tumor size nadir (the transition between decay and growth)

Drug concentration
Response prediction
As soon as the mixed effect model is trained with the training data, it can be used as a tool for prediction. This prediction generally includes estimation of future state of the tracked feature. The Expectation Maximization (EM) is one of the popular methods for addressing the response prediction problem.
3 Example researches using the mixed effect model based prediction framework
After introducing the mixed effect modeling and its adoption in the longitudinal studies, here we briefly introduce two example researches in this domain.
3.1 A study on Alzheimer’s disease by Opsina et al. [6]
In this research, the authors have studied the temporal evolution the “gray matter”, which is a very important indicator of progress in Alzheimer’s disease. Gray matter is the darker tissue of the brain and spinal cord, consisting mainly of nerve cell bodies and branching dendrites [7]. The volume of this tissue is usually measured and tracked during treatment of Alzheimer’s disease.
The main purpose in this study is to model the impact of aging on the cortical gray matter quantification for two groups of subjects: healthy elderly (HE) and Alzheimer’s disease (AD) patients. For this purpose, the highlevel feature of gray matter volume is represented by lowlevel voxelbased morphology from MRI, PET and other biological markers. These voxels are then analyzed both in the temporal and spatial domains to build a mixed effect model.
A nonparametric framework is used to model both groups of HD and AD subjects. For each normalized voxel from each subject , where , this framework is expressed as:
(4) 
where , and are unknown functions and is a random error. Moreover, the flag is one if the th subject is an AD and is zero, otherwise. In the above equation, The term represents the expected density of the gray matter in an AD subject. Therefore, it can be understood that the role of flag is to consider the gray matter emission in the AD group of subjects, due to their disease. Furthermore, is deviation of the th subject from the voxel mean value corresponding to its group.
The analysis of this paper confirms that an agedependent gray matter volume decrease usually happens both in HE and AD subjects. It also confirms that this decrease is more significant in the AD subjects than the HE subjects. Among the AD subjects, it was concluded that the rate of reduction in the gray matter become steeper with aging.
3.2 A study on Prostate cancer by Rios et al. [4]
In this research, the sideeffects of radiotherapy in the prostate cancer treatment is analyzed using mixed effect modeling. The particular effect that has been studied in this paper is the bladder movement. The ultimate goal is to propose a mixed effect model, in order to enable an accurate and efficient bladder motion estimation under exposure of the cancer treatment drug. In contrast to the previous research example, here all the subjects are patients and the proposed approach tries to effectively model inter and intrapatient behaviors of the bladder. Hence, the interpatient features and the intrapatient features are considered as the fixed effects and random effects of the model, respectively.
From a highlevel point of view and considering the framework introduced in section 2, the proposed model of this paper can be summarized in the following 5 steps.
Imaging
The very first step of the proposed process is the computed tomography imaging. In this step, raw data is taken from a diverse set of patient s through the treatment period. As this data naturally contains enormous outlier and noise, the next steps are performed to refine it and extract the most useful information from it.
Spatial normalization of the input images
In this step, raw images taken in step one are processed to clip and extract the region of interest (i.e. the bladder). For this purpose, the threedimensional space of voxels are processed and shrank to a reduced space of voxels containing the bladder voxels.
Surface representation
Each bladder image extracted from step 2 is a solid volume. Therefore, all voxels in the threedimensional space which occur inside that volume can be ignored during the process. This is due to the fact that each 3D volume can be perfectly represented only by using its outer surface. To this end, a mehtod called Spherical Harmonic (SPHARM) was adopted. SPHARM takes each image from the previous step in the space and projects it to a further reduced space of 272 voxels.
Feature reduction using PCA
Considering the relatively small size of the dataset and also the fact that increasing it is extremely costly and harmful to the patients, one might further reduce the feature space. This step is necessary, since it is a proven rule in the machine learning that, when the size number of samples are limited, the size of feature space should be small enough to be trainable.
The Principle Component Analysis (PCA) is a wellknown method for efficient feature reduction. In this method, a statistical analysis is performed on the data and the target model, in order to detect uncorrelated part of the dataset. The expected output of this process is a reduced space which still contains a reasonably informative part of the data. In the research conducted in this paper, the PCA step reduces the feature space from 272 features to only 40 features. This set of 40 values, is considered as the final feature vector of each sample ans used in the rest of the mixed effect modeling process.
Mixed effect modeling
According to the four previous steps, each image taken from patients is represented by a set of 40 features and the goal is to produce 40 independent mixed effect models associated to each feature. Let assume that there are patients in the dataset and number of () images have been taken from each patient. Also, let be the measure of motion/deformation that is aimed to be modeled (i.e. represented by the feature vectors). Therefore, it can be said that where , and indicates the th feature of the th patient, taken at the th period of his/her treatment. The mixed effect modeling of this paper can be described as:
(5) 
The term in the above equation represents the average of the th features among all patients in all of their images. The term is a random variable representing deviation of the th patient from its mean and finally, is a random variable representing deviation of the th variable from its personal mean. The these random variables follow the Gaussian distribution functions:
(6) 
and:
(7) 
4 Conclusion
In this paper, the use of mixed effect modeling in some of the medical image processing application was reviewed. To do this, first general philosophy behind the mixed effect modeling and its necessity in such applications is explained. Then the general principles and common elements of the different models in the literature was described. For this purpose, three different design elements in the process of mixed effect modeling were detected. These elements and their possible choices are summarized in Table 1.
Element  Choices  


Algebraic equation  
Differential equation  





Other features line PSA.  

Druginduced decay in tumor.  
Net growth of tumor size.  
Nadir tumor size  
Drug concentration 
References
 Mary J Lindstrom and Douglas M Bates. Newtonâraphson and em algorithms for linear mixedeffects models for repeatedmeasures data. Journal of the American Statistical Association, 83(404):1014–1022, 1988.
 B Ribba, Nick H Holford, P Magni, I Trocóniz, I Gueorguieva, P Girard, C Sarr, M Elishmereni, C Kloft, and Lena E Friberg. A review of mixedeffects models of tumor growth and effects of anticancer drug treatment used in population analysis. CPT: pharmacometrics & systems pharmacology, 3(5):1–10, 2014.
 Yoshiaki Tsuchida and Patrick Therasse. Response evaluation criteria in solid tumors (recist): new guidelines. Pediatric Blood & Cancer, 37(1):1–3, 2001.
 Richard Rios, Renaud De Crevoisier, Juan D Ospina, Frederic Commandeur, Caroline Lafond, Antoine Simon, Pascal Haigron, Jairo Espinosa, and Oscar Acosta. Population model of bladder motion and deformation based on dominant eigenmodes and mixedeffects models in prostate cancer radiotherapy. Medical image analysis, 38:133–149, 2017.
 Wilfred D Stein, James L Gulley, Jeff Schlom, Ravi A Madan, William Dahut, William D Figg, Yangmin Ning, Phil M Arlen, Doug Price, Susan E Bates, et al. Tumor regression and growth rates determined in five intramural nci prostate cancer trials: the growth rate constant as an indicator of therapeutic efficacy. Clinical Cancer Research, 2010.
 J Ospina, Pascal Benquet, J Correa, and Oscar Acosta. A mixed effects spatio temporal model to assess the joint effect of ageing and alzheimerâs disease in gray matter volume, in â. In ImageGuidance and Multimodal Dose Planning in Radiation Therapy, pages 190–7, 2012.
 Elizabeth R Sowell, Paul M Thompson, Kevin D Tessner, and Arthur W Toga. Mapping continued brain growth and gray matter density reduction in dorsal frontal cortex: inverse relationships during postadolescent brain maturation. Journal of Neuroscience, 21(22):8819–8829, 2001.