A-quick-tour-of-NMoE

Introduction

NMoE (Normal Mixtures-of-Experts) provides a flexible modelling framework for heterogenous data with Gaussian distributions. NMoE consists of a mixture of K Normal expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

  • The gating network parameters alpha’s of the softmax net.
  • The experts network parameters: The location parameters (regression coefficients) beta’s and variances sigma2’s.

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(1, 1) # Standard deviations of the experts
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivNMoE(alphak = alphak, betak = betak, sigmak = sigmak, x = x)
y <- sample$y

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: -823.304470745821
## EM NMoE: Iteration: 2 | log-likelihood: -822.587209164833
## EM NMoE: Iteration: 3 | log-likelihood: -821.541685223325
## EM NMoE: Iteration: 4 | log-likelihood: -819.281734947075
## EM NMoE: Iteration: 5 | log-likelihood: -814.211272751231
## EM NMoE: Iteration: 6 | log-likelihood: -803.908302674227
## EM NMoE: Iteration: 7 | log-likelihood: -786.675397958609
## EM NMoE: Iteration: 8 | log-likelihood: -764.685167141451
## EM NMoE: Iteration: 9 | log-likelihood: -744.448360346408
## EM NMoE: Iteration: 10 | log-likelihood: -730.969477320832
## EM NMoE: Iteration: 11 | log-likelihood: -723.638334771265
## EM NMoE: Iteration: 12 | log-likelihood: -719.884117124528
## EM NMoE: Iteration: 13 | log-likelihood: -717.926159199101
## EM NMoE: Iteration: 14 | log-likelihood: -716.837511658022
## EM NMoE: Iteration: 15 | log-likelihood: -716.184439268261
## EM NMoE: Iteration: 16 | log-likelihood: -715.765380036735
## EM NMoE: Iteration: 17 | log-likelihood: -715.480017410937
## EM NMoE: Iteration: 18 | log-likelihood: -715.274769414764
## EM NMoE: Iteration: 19 | log-likelihood: -715.119635774261
## EM NMoE: Iteration: 20 | log-likelihood: -714.997254795089
## EM NMoE: Iteration: 21 | log-likelihood: -714.897290225883
## EM NMoE: Iteration: 22 | log-likelihood: -714.813419713725
## EM NMoE: Iteration: 23 | log-likelihood: -714.741670056024
## EM NMoE: Iteration: 24 | log-likelihood: -714.679470950786
## EM NMoE: Iteration: 25 | log-likelihood: -714.625101834939
## EM NMoE: Iteration: 26 | log-likelihood: -714.577360474997
## EM NMoE: Iteration: 27 | log-likelihood: -714.535361770126
## EM NMoE: Iteration: 28 | log-likelihood: -714.498413483205
## EM NMoE: Iteration: 29 | log-likelihood: -714.465944175989
## EM NMoE: Iteration: 30 | log-likelihood: -714.437459754273
## EM NMoE: Iteration: 31 | log-likelihood: -714.41251978449
## EM NMoE: Iteration: 32 | log-likelihood: -714.390725214837
## EM NMoE: Iteration: 33 | log-likelihood: -714.371712460508
## EM NMoE: Iteration: 34 | log-likelihood: -714.355150678737
## EM NMoE: Iteration: 35 | log-likelihood: -714.340740352302
## EM NMoE: Iteration: 36 | log-likelihood: -714.328212165655
## EM NMoE: Iteration: 37 | log-likelihood: -714.317325705467
## EM NMoE: Iteration: 38 | log-likelihood: -714.307867838082
## EM NMoE: Iteration: 39 | log-likelihood: -714.299650784494
## EM NMoE: Iteration: 40 | log-likelihood: -714.292509985919

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -714.2925  8 -722.2925 -739.1509 -785.1356
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 299 201 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1    -0.2328162  -0.1252085
## X^1   2.1979930  -2.4675262
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##      0.9248235     0.9789857

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")

Application to a real dataset

Load data

data("tempanomalies")
x <- tempanomalies$Year
y <- tempanomalies$AnnualAnomaly

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

nmoe <- emNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM NMoE: Iteration: 1 | log-likelihood: 48.5826614147941
## EM NMoE: Iteration: 2 | log-likelihood: 48.6919224201266
## EM NMoE: Iteration: 3 | log-likelihood: 48.9248334085462
## EM NMoE: Iteration: 4 | log-likelihood: 49.5774940977518
## EM NMoE: Iteration: 5 | log-likelihood: 51.3824958237511
## EM NMoE: Iteration: 6 | log-likelihood: 55.6961759010577
## EM NMoE: Iteration: 7 | log-likelihood: 62.9471753697443
## EM NMoE: Iteration: 8 | log-likelihood: 69.7116011350829
## EM NMoE: Iteration: 9 | log-likelihood: 73.4211412359876
## EM NMoE: Iteration: 10 | log-likelihood: 75.3866703334103
## EM NMoE: Iteration: 11 | log-likelihood: 76.9770515275983
## EM NMoE: Iteration: 12 | log-likelihood: 78.6775973684867
## EM NMoE: Iteration: 13 | log-likelihood: 80.7169614820909
## EM NMoE: Iteration: 14 | log-likelihood: 83.4100476918588
## EM NMoE: Iteration: 15 | log-likelihood: 87.1924725189124
## EM NMoE: Iteration: 16 | log-likelihood: 91.7793242911418
## EM NMoE: Iteration: 17 | log-likelihood: 94.8410581834594
## EM NMoE: Iteration: 18 | log-likelihood: 95.8866553498857
## EM NMoE: Iteration: 19 | log-likelihood: 96.2313988898587
## EM NMoE: Iteration: 20 | log-likelihood: 96.381861500752
## EM NMoE: Iteration: 21 | log-likelihood: 96.4772660292863
## EM NMoE: Iteration: 22 | log-likelihood: 96.5593006053922
## EM NMoE: Iteration: 23 | log-likelihood: 96.6420739787399
## EM NMoE: Iteration: 24 | log-likelihood: 96.7310782450135
## EM NMoE: Iteration: 25 | log-likelihood: 96.8285838426276
## EM NMoE: Iteration: 26 | log-likelihood: 96.9351505473574
## EM NMoE: Iteration: 27 | log-likelihood: 97.0499924966052
## EM NMoE: Iteration: 28 | log-likelihood: 97.171101059581
## EM NMoE: Iteration: 29 | log-likelihood: 97.2954889379084
## EM NMoE: Iteration: 30 | log-likelihood: 97.4196867447229
## EM NMoE: Iteration: 31 | log-likelihood: 97.54048980238
## EM NMoE: Iteration: 32 | log-likelihood: 97.6557716890443
## EM NMoE: Iteration: 33 | log-likelihood: 97.7651227230171
## EM NMoE: Iteration: 34 | log-likelihood: 97.8700736953341
## EM NMoE: Iteration: 35 | log-likelihood: 97.9737612919138
## EM NMoE: Iteration: 36 | log-likelihood: 98.0801319843145
## EM NMoE: Iteration: 37 | log-likelihood: 98.1929362790738
## EM NMoE: Iteration: 38 | log-likelihood: 98.3148671734183
## EM NMoE: Iteration: 39 | log-likelihood: 98.4471698359823
## EM NMoE: Iteration: 40 | log-likelihood: 98.5898704235669
## EM NMoE: Iteration: 41 | log-likelihood: 98.7424967969814
## EM NMoE: Iteration: 42 | log-likelihood: 98.9048716036695
## EM NMoE: Iteration: 43 | log-likelihood: 99.0777200036317
## EM NMoE: Iteration: 44 | log-likelihood: 99.2629588382769
## EM NMoE: Iteration: 45 | log-likelihood: 99.4638740347342
## EM NMoE: Iteration: 46 | log-likelihood: 99.6854712565111
## EM NMoE: Iteration: 47 | log-likelihood: 99.9352462944205
## EM NMoE: Iteration: 48 | log-likelihood: 100.224513876809
## EM NMoE: Iteration: 49 | log-likelihood: 100.570150279726
## EM NMoE: Iteration: 50 | log-likelihood: 100.994858190352
## EM NMoE: Iteration: 51 | log-likelihood: 101.515078534317
## EM NMoE: Iteration: 52 | log-likelihood: 102.08189475923
## EM NMoE: Iteration: 53 | log-likelihood: 102.536867686652
## EM NMoE: Iteration: 54 | log-likelihood: 102.688621527879
## EM NMoE: Iteration: 55 | log-likelihood: 102.719129621943
## EM NMoE: Iteration: 56 | log-likelihood: 102.721232928614
## EM NMoE: Iteration: 57 | log-likelihood: 102.721862894942

Summary

nmoe$summary()
## ------------------------------------------
## Fitted Normal Mixture-of-Experts model
## ------------------------------------------
## 
## NMoE model with K = 2 experts:
## 
##  log-likelihood df      AIC      BIC      ICL
##        102.7219  8 94.72186 83.07124 83.17719
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2 
## 84 52 
## 
## Regression coefficients:
## 
##       Beta(k = 1)  Beta(k = 2)
## 1   -12.667293979 -42.36197305
## X^1   0.006474808   0.02149261
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##     0.01352347    0.01193111

Plots

Mean curve

nmoe$plot(what = "meancurve")

Confidence regions

nmoe$plot(what = "confregions")

Clusters

nmoe$plot(what = "clusters")

Log-likelihood

nmoe$plot(what = "loglikelihood")