A-quick-tour-of-NMoE

Introduction

NMoE (Normal Mixtures-of-Experts) provides a flexible modelling framework for heterogenous data with Gaussian distributions. NMoE consists of a mixture of K Normal expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

  • The gating network parameters alpha’s of the softmax net.
  • The experts network parameters: The location parameters (regression coefficients) beta’s and variances sigma2’s.

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(1, 1) # Standard deviations of the experts
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivNMoE(alphak = alphak, betak = betak, sigmak = sigmak, x = x)
y <- sample$y

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: -809.267928022971
## EM NMoE: Iteration: 2 | log-likelihood: -809.236448999835
## EM NMoE: Iteration: 3 | log-likelihood: -809.216969908254
## EM NMoE: Iteration: 4 | log-likelihood: -809.194910716206
## EM NMoE: Iteration: 5 | log-likelihood: -809.153324110972
## EM NMoE: Iteration: 6 | log-likelihood: -809.054745541627
## EM NMoE: Iteration: 7 | log-likelihood: -808.804783554818
## EM NMoE: Iteration: 8 | log-likelihood: -808.163703827471
## EM NMoE: Iteration: 9 | log-likelihood: -806.541711469595
## EM NMoE: Iteration: 10 | log-likelihood: -802.60714888104
## EM NMoE: Iteration: 11 | log-likelihood: -793.875337533756
## EM NMoE: Iteration: 12 | log-likelihood: -777.248083050422
## EM NMoE: Iteration: 13 | log-likelihood: -752.026700905812
## EM NMoE: Iteration: 14 | log-likelihood: -724.241513194637
## EM NMoE: Iteration: 15 | log-likelihood: -703.573102893513
## EM NMoE: Iteration: 16 | log-likelihood: -692.478664410885
## EM NMoE: Iteration: 17 | log-likelihood: -687.403757628239
## EM NMoE: Iteration: 18 | log-likelihood: -685.210516914826
## EM NMoE: Iteration: 19 | log-likelihood: -684.231239884564
## EM NMoE: Iteration: 20 | log-likelihood: -683.75080478786
## EM NMoE: Iteration: 21 | log-likelihood: -683.49084681721
## EM NMoE: Iteration: 22 | log-likelihood: -683.338539074829
## EM NMoE: Iteration: 23 | log-likelihood: -683.243534693673
## EM NMoE: Iteration: 24 | log-likelihood: -683.181197582914
## EM NMoE: Iteration: 25 | log-likelihood: -683.138532734512
## EM NMoE: Iteration: 26 | log-likelihood: -683.108258853475
## EM NMoE: Iteration: 27 | log-likelihood: -683.086092286252
## EM NMoE: Iteration: 28 | log-likelihood: -683.069408626634
## EM NMoE: Iteration: 29 | log-likelihood: -683.056544120017
## EM NMoE: Iteration: 30 | log-likelihood: -683.046412212892
## EM NMoE: Iteration: 31 | log-likelihood: -683.038284200751
## EM NMoE: Iteration: 32 | log-likelihood: -683.031659330274

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -683.0317  8 -691.0317 -707.8901 -769.1578
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 258 242 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1   -0.06605732  -0.3284045
## X^1  2.49345739  -2.2278075
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##      0.7568672     0.8716871

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")

Application to a real dataset

Load data

data("tempanomalies")
x <- tempanomalies$Year
y <- tempanomalies$AnnualAnomaly

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: 48.5220094739613
## EM NMoE: Iteration: 2 | log-likelihood: 48.608071098625
## EM NMoE: Iteration: 3 | log-likelihood: 48.7645358211608
## EM NMoE: Iteration: 4 | log-likelihood: 49.1681564071126
## EM NMoE: Iteration: 5 | log-likelihood: 50.293922037973
## EM NMoE: Iteration: 6 | log-likelihood: 53.2173304028115
## EM NMoE: Iteration: 7 | log-likelihood: 59.1904068708332
## EM NMoE: Iteration: 8 | log-likelihood: 66.6754480486616
## EM NMoE: Iteration: 9 | log-likelihood: 71.7701632070882
## EM NMoE: Iteration: 10 | log-likelihood: 74.313700538944
## EM NMoE: Iteration: 11 | log-likelihood: 75.9574685881716
## EM NMoE: Iteration: 12 | log-likelihood: 77.5319403274288
## EM NMoE: Iteration: 13 | log-likelihood: 79.3075196157022
## EM NMoE: Iteration: 14 | log-likelihood: 81.5088144790198
## EM NMoE: Iteration: 15 | log-likelihood: 84.5031560383842
## EM NMoE: Iteration: 16 | log-likelihood: 88.6669360033668
## EM NMoE: Iteration: 17 | log-likelihood: 93.0577651899986
## EM NMoE: Iteration: 18 | log-likelihood: 95.3219365739457
## EM NMoE: Iteration: 19 | log-likelihood: 96.034953073684
## EM NMoE: Iteration: 20 | log-likelihood: 96.2871664572323
## EM NMoE: Iteration: 21 | log-likelihood: 96.41073085095
## EM NMoE: Iteration: 22 | log-likelihood: 96.4983766031633
## EM NMoE: Iteration: 23 | log-likelihood: 96.5788674613412
## EM NMoE: Iteration: 24 | log-likelihood: 96.6623844777583
## EM NMoE: Iteration: 25 | log-likelihood: 96.7530666857033
## EM NMoE: Iteration: 26 | log-likelihood: 96.852592528494
## EM NMoE: Iteration: 27 | log-likelihood: 96.9611443891773
## EM NMoE: Iteration: 28 | log-likelihood: 97.0776185782464
## EM NMoE: Iteration: 29 | log-likelihood: 97.1997377623277
## EM NMoE: Iteration: 30 | log-likelihood: 97.3243397606555
## EM NMoE: Iteration: 31 | log-likelihood: 97.447938507801
## EM NMoE: Iteration: 32 | log-likelihood: 97.5675042133918
## EM NMoE: Iteration: 33 | log-likelihood: 97.6812614030931
## EM NMoE: Iteration: 34 | log-likelihood: 97.78924982916
## EM NMoE: Iteration: 35 | log-likelihood: 97.8934262042415
## EM NMoE: Iteration: 36 | log-likelihood: 97.9972067576968
## EM NMoE: Iteration: 37 | log-likelihood: 98.104594187622
## EM NMoE: Iteration: 38 | log-likelihood: 98.2191702924571
## EM NMoE: Iteration: 39 | log-likelihood: 98.3433148281921
## EM NMoE: Iteration: 40 | log-likelihood: 98.477951140033
## EM NMoE: Iteration: 41 | log-likelihood: 98.622907743054
## EM NMoE: Iteration: 42 | log-likelihood: 98.7777047694852
## EM NMoE: Iteration: 43 | log-likelihood: 98.9423173313282
## EM NMoE: Iteration: 44 | log-likelihood: 99.1177161286786
## EM NMoE: Iteration: 45 | log-likelihood: 99.3061060048853
## EM NMoE: Iteration: 46 | log-likelihood: 99.5111146160274
## EM NMoE: Iteration: 47 | log-likelihood: 99.7382172862418
## EM NMoE: Iteration: 48 | log-likelihood: 99.9956237371547
## EM NMoE: Iteration: 49 | log-likelihood: 100.295732283336
## EM NMoE: Iteration: 50 | log-likelihood: 100.656840558688
## EM NMoE: Iteration: 51 | log-likelihood: 101.102216517172
## EM NMoE: Iteration: 52 | log-likelihood: 101.641388030805
## EM NMoE: Iteration: 53 | log-likelihood: 102.195211651685
## EM NMoE: Iteration: 54 | log-likelihood: 102.606502561399
## EM NMoE: Iteration: 55 | log-likelihood: 102.693445562009
## EM NMoE: Iteration: 56 | log-likelihood: 102.721838668926
## EM NMoE: Iteration: 57 | log-likelihood: 102.721845652917

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df      AIC      BIC      ICL
##        102.7218  8 94.72185 83.07123 83.18296
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2 
## 84 52 
## 
## Regression coefficients:
## 
##       Beta(k = 1)  Beta(k = 2)
## 1   -12.667262617 -42.36303110
## X^1   0.006474792   0.02149314
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##     0.01352322     0.0119306

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")