A-quick-tour-of-SNMoE

Introduction

SNMoE (Skew-Normal Mixtures-of-Experts) provides a flexible modelling framework for heterogenous data with possibly skewed distributions to generalize the standard Normal mixture of expert model. SNMoE consists of a mixture of K skew-Normal expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

  • The gating network parameters alpha’s of the softmax net.
  • The experts network parameters: The location parameters (regression coefficients) beta’s, scale parameters sigma’s, and the skewness parameters lambda’s. SNMoE thus generalises mixtures of (normal, skew-normal) distributions and mixtures of regressions with these distributions. For example, when q = 0, we retrieve mixtures of (skew-normal, or normal) regressions, and when both p = 0 and q = 0, it is a mixture of (skew-normal, or normal) distributions. It also reduces to the standard (normal, skew-normal) distribution when we only use a single expert (K = 1).

Model estimation/learning is performed by a dedicated expectation conditional maximization (ECM) algorithm by maximizing the observed data log-likelihood. We provide simulated examples to illustrate the use of the model in model-based clustering of heterogeneous regression data and in fitting non-linear regression functions.

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
lambdak <- c(3, 5) # Skewness parameters of the experts
sigmak <- c(1, 1) # Standard deviations of the experts
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivSNMoE(alphak = alphak, betak = betak, sigmak = sigmak, 
                          lambdak = lambdak, x = x)
y <- sample$y

Set up SNMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-6
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

snmoe <- emSNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
                 threshold, verbose, verbose_IRLS)
## EM - SNMoE: Iteration: 1 | log-likelihood: -607.605595946828
## EM - SNMoE: Iteration: 2 | log-likelihood: -555.856733593601
## EM - SNMoE: Iteration: 3 | log-likelihood: -553.962566805216
## EM - SNMoE: Iteration: 4 | log-likelihood: -553.521073129669
## EM - SNMoE: Iteration: 5 | log-likelihood: -553.310412805982
## EM - SNMoE: Iteration: 6 | log-likelihood: -553.141039010359
## EM - SNMoE: Iteration: 7 | log-likelihood: -552.98731283049
## EM - SNMoE: Iteration: 8 | log-likelihood: -552.848343371803
## EM - SNMoE: Iteration: 9 | log-likelihood: -552.724760392298
## EM - SNMoE: Iteration: 10 | log-likelihood: -552.616109889107
## EM - SNMoE: Iteration: 11 | log-likelihood: -552.520976400206
## EM - SNMoE: Iteration: 12 | log-likelihood: -552.437820571431
## EM - SNMoE: Iteration: 13 | log-likelihood: -552.365115751785
## EM - SNMoE: Iteration: 14 | log-likelihood: -552.301440533455
## EM - SNMoE: Iteration: 15 | log-likelihood: -552.245571451252
## EM - SNMoE: Iteration: 16 | log-likelihood: -552.196455947324
## EM - SNMoE: Iteration: 17 | log-likelihood: -552.153180236604
## EM - SNMoE: Iteration: 18 | log-likelihood: -552.114894625712
## EM - SNMoE: Iteration: 19 | log-likelihood: -552.080959319886
## EM - SNMoE: Iteration: 20 | log-likelihood: -552.050837081958
## EM - SNMoE: Iteration: 21 | log-likelihood: -552.024071484899
## EM - SNMoE: Iteration: 22 | log-likelihood: -552.000194499001
## EM - SNMoE: Iteration: 23 | log-likelihood: -551.978833866763
## EM - SNMoE: Iteration: 24 | log-likelihood: -551.959704223198
## EM - SNMoE: Iteration: 25 | log-likelihood: -551.942511816999
## EM - SNMoE: Iteration: 26 | log-likelihood: -551.927038583672
## EM - SNMoE: Iteration: 27 | log-likelihood: -551.913104147396
## EM - SNMoE: Iteration: 28 | log-likelihood: -551.900493017746
## EM - SNMoE: Iteration: 29 | log-likelihood: -551.889073557798
## EM - SNMoE: Iteration: 30 | log-likelihood: -551.878732406668
## EM - SNMoE: Iteration: 31 | log-likelihood: -551.869335112066
## EM - SNMoE: Iteration: 32 | log-likelihood: -551.860799182716
## EM - SNMoE: Iteration: 33 | log-likelihood: -551.853008051041
## EM - SNMoE: Iteration: 34 | log-likelihood: -551.845872386412
## EM - SNMoE: Iteration: 35 | log-likelihood: -551.83934144868
## EM - SNMoE: Iteration: 36 | log-likelihood: -551.833348806943
## EM - SNMoE: Iteration: 37 | log-likelihood: -551.827866371895
## EM - SNMoE: Iteration: 38 | log-likelihood: -551.822829825501
## EM - SNMoE: Iteration: 39 | log-likelihood: -551.818184417606
## EM - SNMoE: Iteration: 40 | log-likelihood: -551.813902902261
## EM - SNMoE: Iteration: 41 | log-likelihood: -551.809963741912
## EM - SNMoE: Iteration: 42 | log-likelihood: -551.806315880244
## EM - SNMoE: Iteration: 43 | log-likelihood: -551.802936319084
## EM - SNMoE: Iteration: 44 | log-likelihood: -551.79980228184
## EM - SNMoE: Iteration: 45 | log-likelihood: -551.796909230691
## EM - SNMoE: Iteration: 46 | log-likelihood: -551.794211402885
## EM - SNMoE: Iteration: 47 | log-likelihood: -551.791704479983
## EM - SNMoE: Iteration: 48 | log-likelihood: -551.789370921817
## EM - SNMoE: Iteration: 49 | log-likelihood: -551.78719181166
## EM - SNMoE: Iteration: 50 | log-likelihood: -551.785157693808
## EM - SNMoE: Iteration: 51 | log-likelihood: -551.783265986979
## EM - SNMoE: Iteration: 52 | log-likelihood: -551.781493542897
## EM - SNMoE: Iteration: 53 | log-likelihood: -551.779831712305
## EM - SNMoE: Iteration: 54 | log-likelihood: -551.778274665635
## EM - SNMoE: Iteration: 55 | log-likelihood: -551.776816134204
## EM - SNMoE: Iteration: 56 | log-likelihood: -551.775449201255
## EM - SNMoE: Iteration: 57 | log-likelihood: -551.774159339489
## EM - SNMoE: Iteration: 58 | log-likelihood: -551.772946520857
## EM - SNMoE: Iteration: 59 | log-likelihood: -551.771809130833
## EM - SNMoE: Iteration: 60 | log-likelihood: -551.770736784775
## EM - SNMoE: Iteration: 61 | log-likelihood: -551.769725785702
## EM - SNMoE: Iteration: 62 | log-likelihood: -551.76877002089
## EM - SNMoE: Iteration: 63 | log-likelihood: -551.767865082421
## EM - SNMoE: Iteration: 64 | log-likelihood: -551.767007936293
## EM - SNMoE: Iteration: 65 | log-likelihood: -551.76620176252
## EM - SNMoE: Iteration: 66 | log-likelihood: -551.765439263737
## EM - SNMoE: Iteration: 67 | log-likelihood: -551.764718577861
## EM - SNMoE: Iteration: 68 | log-likelihood: -551.764035375983
## EM - SNMoE: Iteration: 69 | log-likelihood: -551.763386057941
## EM - SNMoE: Iteration: 70 | log-likelihood: -551.76276928926
## EM - SNMoE: Iteration: 71 | log-likelihood: -551.762185691555
## EM - SNMoE: Iteration: 72 | log-likelihood: -551.76163111877
## EM - SNMoE: Iteration: 73 | log-likelihood: -551.761098622927

Summary

snmoe$summary()
## -----------------------------------------------
## Fitted Skew-Normal Mixture-of-Experts model
## -----------------------------------------------
## 
## SNMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -551.7611 10 -561.7611 -582.8341 -582.8866
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 249 251 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1     0.8343338   0.8131996
## X^1   2.4693887  -2.4727475
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##      0.5067685     0.5629645

Plots

Mean curve

snmoe$plot(what = "meancurve")

Confidence regions

snmoe$plot(what = "confregions")

Clusters

snmoe$plot(what = "clusters")

Log-likelihood

snmoe$plot(what = "loglikelihood")

Application to a real dataset

Load data

data("tempanomalies")
x <- tempanomalies$Year
y <- tempanomalies$AnnualAnomaly

Set up SNMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-6
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

snmoe <- emSNMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
                 threshold, verbose, verbose_IRLS)
## EM - SNMoE: Iteration: 1 | log-likelihood: 66.7864945329767
## EM - SNMoE: Iteration: 2 | log-likelihood: 88.2296206612406
## EM - SNMoE: Iteration: 3 | log-likelihood: 89.1760592112306
## EM - SNMoE: Iteration: 4 | log-likelihood: 89.5475038262082
## EM - SNMoE: Iteration: 5 | log-likelihood: 89.815042827909
## EM - SNMoE: Iteration: 6 | log-likelihood: 89.9880098300878
## EM - SNMoE: Iteration: 7 | log-likelihood: 90.0865630579272
## EM - SNMoE: Iteration: 8 | log-likelihood: 90.1418235340948
## EM - SNMoE: Iteration: 9 | log-likelihood: 90.1750882514923
## EM - SNMoE: Iteration: 10 | log-likelihood: 90.1986220170872
## EM - SNMoE: Iteration: 11 | log-likelihood: 90.2181274647215
## EM - SNMoE: Iteration: 12 | log-likelihood: 90.2353637801516
## EM - SNMoE: Iteration: 13 | log-likelihood: 90.2510168366834
## EM - SNMoE: Iteration: 14 | log-likelihood: 90.265641057864
## EM - SNMoE: Iteration: 15 | log-likelihood: 90.2795429063615
## EM - SNMoE: Iteration: 16 | log-likelihood: 90.292867964742
## EM - SNMoE: Iteration: 17 | log-likelihood: 90.3056836578867
## EM - SNMoE: Iteration: 18 | log-likelihood: 90.3180428692306
## EM - SNMoE: Iteration: 19 | log-likelihood: 90.3299727615159
## EM - SNMoE: Iteration: 20 | log-likelihood: 90.3415012973919
## EM - SNMoE: Iteration: 21 | log-likelihood: 90.3526543723541
## EM - SNMoE: Iteration: 22 | log-likelihood: 90.3634613845533
## EM - SNMoE: Iteration: 23 | log-likelihood: 90.373933134571
## EM - SNMoE: Iteration: 24 | log-likelihood: 90.3840914162107
## EM - SNMoE: Iteration: 25 | log-likelihood: 90.3939604387388
## EM - SNMoE: Iteration: 26 | log-likelihood: 90.4035578359143
## EM - SNMoE: Iteration: 27 | log-likelihood: 90.412900510967
## EM - SNMoE: Iteration: 28 | log-likelihood: 90.4220353194485
## EM - SNMoE: Iteration: 29 | log-likelihood: 90.4309440116482
## EM - SNMoE: Iteration: 30 | log-likelihood: 90.4396347026313
## EM - SNMoE: Iteration: 31 | log-likelihood: 90.4481253806957
## EM - SNMoE: Iteration: 32 | log-likelihood: 90.456471803214
## EM - SNMoE: Iteration: 33 | log-likelihood: 90.4645355523349
## EM - SNMoE: Iteration: 34 | log-likelihood: 90.4724958329427
## EM - SNMoE: Iteration: 35 | log-likelihood: 90.4803416421911
## EM - SNMoE: Iteration: 36 | log-likelihood: 90.4880689816315
## EM - SNMoE: Iteration: 37 | log-likelihood: 90.4957386877037
## EM - SNMoE: Iteration: 38 | log-likelihood: 90.5032698117254
## EM - SNMoE: Iteration: 39 | log-likelihood: 90.5106919577003
## EM - SNMoE: Iteration: 40 | log-likelihood: 90.518012853572
## EM - SNMoE: Iteration: 41 | log-likelihood: 90.5252452781482
## EM - SNMoE: Iteration: 42 | log-likelihood: 90.5323832086457
## EM - SNMoE: Iteration: 43 | log-likelihood: 90.539427148604
## EM - SNMoE: Iteration: 44 | log-likelihood: 90.5463682209391
## EM - SNMoE: Iteration: 45 | log-likelihood: 90.5532211289775
## EM - SNMoE: Iteration: 46 | log-likelihood: 90.5599694504909
## EM - SNMoE: Iteration: 47 | log-likelihood: 90.5666212004554
## EM - SNMoE: Iteration: 48 | log-likelihood: 90.5731716724327
## EM - SNMoE: Iteration: 49 | log-likelihood: 90.5796316404401
## EM - SNMoE: Iteration: 50 | log-likelihood: 90.5859989899081
## EM - SNMoE: Iteration: 51 | log-likelihood: 90.592271315783
## EM - SNMoE: Iteration: 52 | log-likelihood: 90.5984572285162
## EM - SNMoE: Iteration: 53 | log-likelihood: 90.6045568957703
## EM - SNMoE: Iteration: 54 | log-likelihood: 90.6105709312385
## EM - SNMoE: Iteration: 55 | log-likelihood: 90.6165000155071
## EM - SNMoE: Iteration: 56 | log-likelihood: 90.6223447423574
## EM - SNMoE: Iteration: 57 | log-likelihood: 90.6281071869395
## EM - SNMoE: Iteration: 58 | log-likelihood: 90.6337869163912
## EM - SNMoE: Iteration: 59 | log-likelihood: 90.6393836666921
## EM - SNMoE: Iteration: 60 | log-likelihood: 90.6448977541574
## EM - SNMoE: Iteration: 61 | log-likelihood: 90.6503288992808
## EM - SNMoE: Iteration: 62 | log-likelihood: 90.6556761472245
## EM - SNMoE: Iteration: 63 | log-likelihood: 90.6609397548337
## EM - SNMoE: Iteration: 64 | log-likelihood: 90.6661229391288
## EM - SNMoE: Iteration: 65 | log-likelihood: 90.6712175513635
## EM - SNMoE: Iteration: 66 | log-likelihood: 90.6762276904094
## EM - SNMoE: Iteration: 67 | log-likelihood: 90.6811528133246
## EM - SNMoE: Iteration: 68 | log-likelihood: 90.6859917842208
## EM - SNMoE: Iteration: 69 | log-likelihood: 90.6907394247067
## EM - SNMoE: Iteration: 70 | log-likelihood: 90.6954017246093
## EM - SNMoE: Iteration: 71 | log-likelihood: 90.699984614447
## EM - SNMoE: Iteration: 72 | log-likelihood: 90.7044817471968
## EM - SNMoE: Iteration: 73 | log-likelihood: 90.7088848383768
## EM - SNMoE: Iteration: 74 | log-likelihood: 90.71319318317
## EM - SNMoE: Iteration: 75 | log-likelihood: 90.7174057126423
## EM - SNMoE: Iteration: 76 | log-likelihood: 90.721518236862
## EM - SNMoE: Iteration: 77 | log-likelihood: 90.7255331870858
## EM - SNMoE: Iteration: 78 | log-likelihood: 90.7295125560891
## EM - SNMoE: Iteration: 79 | log-likelihood: 90.7331771506779
## EM - SNMoE: Iteration: 80 | log-likelihood: 90.7369939445543
## EM - SNMoE: Iteration: 81 | log-likelihood: 90.7404821884587
## EM - SNMoE: Iteration: 82 | log-likelihood: 90.7439179511987
## EM - SNMoE: Iteration: 83 | log-likelihood: 90.7472556020719
## EM - SNMoE: Iteration: 84 | log-likelihood: 90.7505281858017
## EM - SNMoE: Iteration: 85 | log-likelihood: 90.753662816102
## EM - SNMoE: Iteration: 86 | log-likelihood: 90.7567035532062
## EM - SNMoE: Iteration: 87 | log-likelihood: 90.7596679537007
## EM - SNMoE: Iteration: 88 | log-likelihood: 90.7625267879965
## EM - SNMoE: Iteration: 89 | log-likelihood: 90.7652842688913
## EM - SNMoE: Iteration: 90 | log-likelihood: 90.7679453697985
## EM - SNMoE: Iteration: 91 | log-likelihood: 90.7705138037305
## EM - SNMoE: Iteration: 92 | log-likelihood: 90.7729937630342
## EM - SNMoE: Iteration: 93 | log-likelihood: 90.7753864823853
## EM - SNMoE: Iteration: 94 | log-likelihood: 90.7778353330687
## EM - SNMoE: Iteration: 95 | log-likelihood: 90.7800804426911
## EM - SNMoE: Iteration: 96 | log-likelihood: 90.7821994865698
## EM - SNMoE: Iteration: 97 | log-likelihood: 90.7842393850078
## EM - SNMoE: Iteration: 98 | log-likelihood: 90.7862227397057
## EM - SNMoE: Iteration: 99 | log-likelihood: 90.7881067708544
## EM - SNMoE: Iteration: 100 | log-likelihood: 90.7899281398768
## EM - SNMoE: Iteration: 101 | log-likelihood: 90.7916652036599
## EM - SNMoE: Iteration: 102 | log-likelihood: 90.7933438861422
## EM - SNMoE: Iteration: 103 | log-likelihood: 90.794942114426
## EM - SNMoE: Iteration: 104 | log-likelihood: 90.7964747094526
## EM - SNMoE: Iteration: 105 | log-likelihood: 90.7979436353581
## EM - SNMoE: Iteration: 106 | log-likelihood: 90.7993506531547
## EM - SNMoE: Iteration: 107 | log-likelihood: 90.8006978620188
## EM - SNMoE: Iteration: 108 | log-likelihood: 90.801987236878
## EM - SNMoE: Iteration: 109 | log-likelihood: 90.8033328016103
## EM - SNMoE: Iteration: 110 | log-likelihood: 90.8045059036114
## EM - SNMoE: Iteration: 111 | log-likelihood: 90.8056267231428
## EM - SNMoE: Iteration: 112 | log-likelihood: 90.8066977452866
## EM - SNMoE: Iteration: 113 | log-likelihood: 90.8077210574807
## EM - SNMoE: Iteration: 114 | log-likelihood: 90.8086977240443
## EM - SNMoE: Iteration: 115 | log-likelihood: 90.8096459434809
## EM - SNMoE: Iteration: 116 | log-likelihood: 90.8105338284222
## EM - SNMoE: Iteration: 117 | log-likelihood: 90.8113887843496
## EM - SNMoE: Iteration: 118 | log-likelihood: 90.8121996067431
## EM - SNMoE: Iteration: 119 | log-likelihood: 90.812971775168
## EM - SNMoE: Iteration: 120 | log-likelihood: 90.8137042194172
## EM - SNMoE: Iteration: 121 | log-likelihood: 90.814401206627
## EM - SNMoE: Iteration: 122 | log-likelihood: 90.8150650334176
## EM - SNMoE: Iteration: 123 | log-likelihood: 90.8156973970845
## EM - SNMoE: Iteration: 124 | log-likelihood: 90.8162989945232
## EM - SNMoE: Iteration: 125 | log-likelihood: 90.8168717749886
## EM - SNMoE: Iteration: 126 | log-likelihood: 90.8174170233841
## EM - SNMoE: Iteration: 127 | log-likelihood: 90.8179994874258
## EM - SNMoE: Iteration: 128 | log-likelihood: 90.8184915790245
## EM - SNMoE: Iteration: 129 | log-likelihood: 90.81895974813
## EM - SNMoE: Iteration: 130 | log-likelihood: 90.8194021116382
## EM - SNMoE: Iteration: 131 | log-likelihood: 90.8198219486653
## EM - SNMoE: Iteration: 132 | log-likelihood: 90.8202205339426
## EM - SNMoE: Iteration: 133 | log-likelihood: 90.8205989979743
## EM - SNMoE: Iteration: 134 | log-likelihood: 90.8209682636082
## EM - SNMoE: Iteration: 135 | log-likelihood: 90.8213091488078
## EM - SNMoE: Iteration: 136 | log-likelihood: 90.8216328059873
## EM - SNMoE: Iteration: 137 | log-likelihood: 90.821950072048
## EM - SNMoE: Iteration: 138 | log-likelihood: 90.8222413778547
## EM - SNMoE: Iteration: 139 | log-likelihood: 90.8225178498794
## EM - SNMoE: Iteration: 140 | log-likelihood: 90.8227801907392
## EM - SNMoE: Iteration: 141 | log-likelihood: 90.8230290866845
## EM - SNMoE: Iteration: 142 | log-likelihood: 90.8232651933176
## EM - SNMoE: Iteration: 143 | log-likelihood: 90.8234891364826
## EM - SNMoE: Iteration: 144 | log-likelihood: 90.8237015133225
## EM - SNMoE: Iteration: 145 | log-likelihood: 90.823902893288
## EM - SNMoE: Iteration: 146 | log-likelihood: 90.8240938191086
## EM - SNMoE: Iteration: 147 | log-likelihood: 90.8243194753583
## EM - SNMoE: Iteration: 148 | log-likelihood: 90.8244455732406
## EM - SNMoE: Iteration: 149 | log-likelihood: 90.8246531859002
## EM - SNMoE: Iteration: 150 | log-likelihood: 90.8247616751733
## EM - SNMoE: Iteration: 151 | log-likelihood: 90.8249529367426
## EM - SNMoE: Iteration: 152 | log-likelihood: 90.8250909014801
## EM - SNMoE: Iteration: 153 | log-likelihood: 90.8252219403842
## EM - SNMoE: Iteration: 154 | log-likelihood: 90.8253461184383
## EM - SNMoE: Iteration: 155 | log-likelihood: 90.8254182721249

Summary

snmoe$summary()
## -----------------------------------------------
## Fitted Skew-Normal Mixture-of-Experts model
## -----------------------------------------------
## 
## SNMoE model with K = 2 experts:
## 
##  log-likelihood df      AIC      BIC      ICL
##        90.82542 10 80.82542 66.26214 66.15802
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2 
## 69 67 
## 
## Regression coefficients:
## 
##       Beta(k = 1)  Beta(k = 2)
## 1   -14.234311502 -32.64896010
## X^1   0.007306007   0.01669469
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##     0.01440633    0.03715067

Plots

Mean curve

snmoe$plot(what = "meancurve")

Confidence regions

snmoe$plot(what = "confregions")

Clusters

snmoe$plot(what = "clusters")

Log-likelihood

snmoe$plot(what = "loglikelihood")