Step 3: Model Optimization

One of ILLMO’s strenghts is its visual way of optimizing models. Optimizing a model to fit observed data can be seen as a game. We will show the basic rules, but feel free to play around for optimal results! See Step 2 for how to import data in ILLMO.

After reading in the data, ILLMO performs a so-called exploratory data analysis. As already explained in Step 2, the default choice is to approximate observed histograms by Gaussian distributions with averages that vary across the conditions and a standard deviation that is the same for all conditions (in the example, this implies that there are 3 parameters). The number expressing the lack-of-fit between the histograms and the distributions is the LLC (which stands for log-likelihood criterium). ILLMO determines the optimal values for the parameters in the statistical model by minimizing the LCC. Note that the value of the LLC only has an absolute interpretation in case of dicrete data, where the LLC is always positive, and LLC=0 corresponds to a perfect model where the histogram and the distribution coincide perfectly. In case of continuous data, the LLC can be either positive or negative, and lower values correspond to a better fit, but the concept of a perfect model doesn’t carry over to continuous data.

The (lack of) quality of a model is expressed by means of the Akaike Information criterium (AIC) and is determined by two aspects. The first is the lack-of-fit LLC; the second is the number of parameters used in the model (where AIC increases, and the quality hence decreases, with the number of distribution parameters that are used in the model, and that can be optimized to reduce the lack-of-fit). The ‘game’ of the optimization process is to try and make the value of the AIC as small as possible by exploring alternative models.

While the default choice in ILLMO is to assume Gaussian distributions, alternative distributions can be selected in the prob dialog window. In the example, we have selected Student’s T distribution as an alternative for the Gaussian distribution.

data40_studentt

Dialog window prob used to both specify properties of the distributions (or PDFs) used, and for inspecting distribution parameters

The effect of this change in statistical model can be visualized in several ways. First of all, in the bottom-right graph showing the cumulative histograms and distributions, we can visually inspect whether or not the models (in green and blue) fit more closely to the actual data (in red and black) in the case of Student-T distributions, as compared to the case of Gaussian distribution. An easier and more objective way is to inspect the changes in  LLC and AIC.

Changes in the LLC and AIC are visualized in the ‘Model history (LLC/CRC & AIC)’ graph in the bottom left. The black line shows the change in the LLC, which has indeed decreased by changing the distribution. As the number of parameters is the same in both models, the AIC and the LLC run parallel.

ILLMO interface after changing the distributions from Gaussian into Student T

ILLMO also allows to model transformed data. Transformed data can often be more closely approximated by standard distributions (such as a Gaussian) than original data. For instance, time measurements, that are by definition positively valued, are often transformed (most frequently, using a logarithmic function) before modeling them (which corresponds to using log-normal dsitributions instead of normal distributions for modeling the observed histograms).

ILLMO offers a choice between two types of (monotonically increasing) transformations, i.e., spline functions and power-like functions, which can be accessed through separate dialog windows. The option spline also offers the possibility to include a rank-order transformation, as is customary in non-parametric statistics.

The option BoxCox offers the possibility to perform power-like transformations, including a logarithmic function, as shown in the dialog window below. When the BoxCox power is specified as a ‘Private Variable’, this implies that ILLMO can vary it in order to minimize the LLC. The optimized tranformation is displayed in the graph on the left.

Dialog window for specifying and examining BoxCox transformations on the input data

The model will now have an extra parameter which makes it slightly more complex, but this might be worthwhile if it results in a significantly better correspondence between observed histograms and modeled distributions. In the case of our example data, we observe in the left graph that after adding a private variable for the Box-Cox transformation, the LLC (in black) decreases but the AIC (in red) actually increases slightly. This implies that it would be best in this case to undo the previous action by revisiting the Box-Cox dialog window and setting  the variable BoxCox power back to ‘fixed’ at a value of 1.

ILLMO interface after including a Box-Cox transformation on the input data

The AIC values can be converted into model likelihoods, using the method of multi-model comparison. The model likelihoods (which sum to 1 over all models considered) are shown as the red curve in the graph on the right. The first model (with Gaussian distributions) is much less likely than the other two models (with Student-T distributions), and the highest likelihood is assigned to the model without a data transformation.

Don’t be afraid to try other model options and explore their effect on the LLC and AIC!

Next: Drawing Conclusions in Illmo

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s