---
title: 'Partitioning Cluster Analysis Using Fuzzy C-Means'
author: 'Zeynel Cebeci'
date: '2017-11-20'
output:
  html_document:
    theme: yeti
    highlight: kate
    toc: true
    toc_depth: 2
    toc_float:
      collapsed: false
      smooth_scroll: false
    df_print: paged
    code_folding: show
    number_sections: true
vignette: >
  %\VignetteIndexEntry{Partitioning Cluster Analysis Using Fuzzy C-Means}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding[UTF-8]{inputenc}
---
<style>
body {text-align: justify}
</style>

# PREPARING FOR THE ANALYSIS
***
## Install and load the package ppclust
This vignette is designed to be used with the ppclust package. You can download the recent version of the package '`ppclust`' from CRAN with the following command:

```{r, eval=FALSE, message=FALSE, warning=FALSE}
install.packages("ppclust")
```

If you have already installed '`ppclust`', you can load it into R working environment by using the following command:

```{r, echo=TRUE, message=FALSE, warning=FALSE}
library(ppclust)
```

## Load the required packages
For visualization of the clustering results, some examples in this vignette use the functions from some cluster analysis packages such as '`cluster`', '`fclust`' and '`factoextra`'. Therefore, these packages should be loaded into R working environment with the following commands:

```{r, echo=TRUE, message=FALSE, warning=FALSE}
library(factoextra)
library(cluster)
library(fclust)
```

## Load the data set 
We demonstrate FCM on the Iris data set (Anderson, 1935). It is a real data set of the four features (Sepal.Length, Sepal.Width, Petal.Length and Petal.Width) of 150 iris flowers with three species (in the last column as the class variable). This four-dimensional data set contains 50 samples  each of three iris species. One of these three natural clusters (Class 1) is linearly well-separated from the other two clusters, while Classes 2 and 3 have some overlap as seen in the plot below.

```{r echo=TRUE, message=FALSE, warning=FALSE, cols.print=5, rows.print=10}
data(iris)
x=iris[,-5]
x
```

Plot the data by the classes of iris species

```{r fig.width=7, fig.height=6}
pairs(x, col=iris[,5])
```

# FUZZY C-MEANS CLUSTERING
***
Fuzzy C-Means (FCM) is a soft custering algorithm proposed by Bezdek (1974; 1981). Unlike *K-means* algorithm in which each data object is the member of only one cluster, a data object is the member of all clusters with varying degrees of fuzzy memberhip between `0` and `1` in FCM. Hence, the data objects closer to the centers of clusters have higher degrees of membership than objects scattered in the borders of clusters. 

## Run FCM with Single Start
### Initialization
In order to start FCM as well as the other alternating optimization algorithms, an initialization step is required to build the initial cluster prototypes matrix and fuzzy membership degrees matrix. Although this task is usually performed in the initialization step of the clustering algorithm, the initial prototypes and memberships can also be directly input by the user. 

FCM is usually started by using an integer specifying the number of clusters. In this case, the prototypes matrix is internally generated by using any of the prototype initalization algorithms which are included in the package '`inaparc`'. The default initialization technique is *K-means++* with the current version of `fcm` function in this package. In the following code block, FCM runs for three clusters with the default values of the remaining arguments.

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3)
```
Although it is a usual way to run FCM with a pre-determined number of clusters, sometimes the users may wish to use a certain cluster prototypes matrix to start the algorithm. In this case, FCM uses the user-specified prototype matrix instead of internally generated ones as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
v0 <- matrix(nrow=3, ncol=4,
     c(5.0, 3.4, 1.4, 0.3,
       6.7, 3.0, 5.6, 2.1,
	   5.8, 2.7, 4.3, 1.4),
	   byrow=TRUE)
print(v0)
res.fcm <- fcm(x, centers=v0)
```
In the following code block, there is another example for initialization of cluster prototypes for three clusters. `v`, the cluster prototype matrix is initialized by using the *K-means++* initalization algorithm (`kmpp`) in the R package '`inaparc`'. 

```{r echo=TRUE, message=FALSE, warning=FALSE}
v0 <- inaparc::kmpp(x, k=3)$v
print(v0)
res.fcm <- fcm(x, centers=v0)
```
> There are many other techniques for initialization of cluster prototype matrices in the R package '`inaparc`'. See the package documentation for other alternatives.

FCM can be alternatively started with a user-specified membership degrees matrix. In this case, the function `fcm` is called with the argument `memberships` as follows:
 
```{r echo=TRUE, message=FALSE, warning=FALSE}
u0 <- inaparc::imembrand(nrow(x), k=3)$u
res.fcm <- fcm(x, centers=3, memberships=u0)
```
In order to generate the matrices of cluster prototypes and membership degrees, users can choose any of existing initialization algorithms in the R package '`inaparc`'. For this purpose, `alginitv` and `alginitu` can be speficied for cluster prototypes and membership degrees, respectively.

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3, alginitv="hartiganwong", alginitu="imembrand")
```
In general, the squared Euclidean distance metric is used with FCM in order to compute the distances between the cluster centers and the data objects. Users can specify the other distance metrics with the argument `dmetric` as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3, dmetric="correlation")
```
Although the argument `m`, fuzzy exponent is usually choosen as `2` in most of the applications using FCM, users can increase `m` to get more fuzzified results as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3, m=4)
```

> There are other input arguments available with the function `fcm` to control the run of FCM algoritm. For the details, see `fcm` in  the package manual of `ppclust`.

### Clustering Results
#### Fuzzy Membership Matrix
The fuzzy membership degrees matrix is the main output of the function `fcm`. 

```{r echo=TRUE, message=FALSE, warning=FALSE, cols.print=3, rows.print=10}
res.fcm <- fcm(x, centers=3)
as.data.frame(res.fcm$u)
```

#### Initial and Final Cluster Prototypes
Initial and final cluster prototypes matrices can be achieved as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm$v0
res.fcm$v
```
#### Summary of Clustering Results
The function `fcm` returns the clustering results as an intance of '`ppclust`' class. This object is summarized with the `summary` method of the package '`ppclust`' as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
summary(res.fcm)
```

All available components of the '`ppclust`' object are listed at the end of summary. These elements can be accessed using as the attributes of the object. For example, the execution time of the run of FCM is accessed as follows:

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm$comp.time
```

## Run FCM with Multiple Starts
In order to find an optimal solution, the function `fcm` can be started for multiple times. As seen in the following code chunk, the argument `nstart` is used for this purpose. 

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3, nstart=5)
```

When the multiple start is performed, either initial cluster prototypes or initial membership degrees could be wanted to keep unchanged between the starts of algorithm. In this case, the arguments `fixcent` and `fixmemb` are used for fixing the initial cluster prototypes matrix and initial membership degrees matrix, respectively.

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm <- fcm(x, centers=3, nstart=5, fixmemb=TRUE)
```

> Both the prototypes and memberships cannot be fixed at the same time.

### Display the best solution
The clustering result contains some outputs providing information about some components such as the objective function values, number of iterations and computing time obtained with each start of the algorithm. The following code chunk demonstrates these outputs. 

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm$func.val
res.fcm$iter
res.fcm$comp.time
```
Among the outputs from succesive starts of the algorithm, the best solution is obtained from the start giving the minimum value of the objective function, and stored as the the final clustering result of the multiple starts of FCM.

```{r echo=TRUE, message=FALSE, warning=FALSE}
res.fcm$best.start
```

### Display the summary of clustering results
As described for the single start of FCM above, the result of multiple starts of FCM is displayed by using `summary` method of the '`ppclust`' package.

```{r echo=TRUE, message=FALSE, warning=FALSE}
summary(res.fcm)
```

# VISUALIZATION OF THE CLUSTERING RESULTS
***
## Pairwise Scatter Plots
There are many ways of visual representation of the clustering results. One common techique is to display the clustering results by using pairs of the features. The '`plotcluster`' can be used to plot the clustering results as follows:

```{r fig.width=7, fig.height=6}
plotcluster(res.fcm, cp=1, trans=TRUE)
```

In the plot above, clustering result is shown with pairwise scatter plots by using color palette 1 and transparent colors. The details can be viewed on the package manual for the function '`plotcluster`'.

## Cluster Plot with fviz_cluster
There are some nice versions of the cluster plots which are available in some of the R packages. One of them is the function `fviz_cluster` of the package '`factoextra`' (Kassambara & Mundt, 2017). In order to use this function for the fuzzy clustering result, first the fuzzy clustering object of `ppclust` should be converted to `kmeans` object by using the `ppclust2` of the package '`ppclust`' as shown in the first line of the code chunk as follows:

```{r fig.width=7, fig.height=6}
res.fcm2 <- ppclust2(res.fcm, "kmeans")
factoextra::fviz_cluster(res.fcm2, data = x, 
  ellipse.type = "convex",
  palette = "jco",
  repel = TRUE)
```

## Cluster Plot with clusplot
User can also use the function `clusplot` in the package '`cluster`' (Maechler et al, 2017) for plotting the clustering results. For this purpose, the fuzzy clustering object of `ppclust` should be converted to `fanny` object by using the `ppclust2` function of the package '`ppclust`' as seen in the following code chunk.

```{r fig.width=7, fig.height=6}
res.fcm3 <- ppclust2(res.fcm, "fanny")

cluster::clusplot(scale(x), res.fcm3$cluster,  
  main = "Cluster plot of Iris data set",
  color=TRUE, labels = 2, lines = 2, cex=1)
```	 

# VALIDATION OF THE CLUSTERING RESULTS
***
Cluster validation is an evaluation process for the goodness of the clustering result. For this purpose, various validity indexes have been proposed in the related literature. Since clustering is an unsupervised learning analysis which does not use any external information, the internal indexes are used to validate the clustering results. Although there are many internal indexes have originally been proposed for working with hard membership degrees produced by the K-means and its variants, most of these indexes cannot be used for fuzzy clustering results. In R environment, Partition Entropy (PE), Partition Coefficient (PC) and Modified Partition Coefficient (MPC) and Fuzzy Silhouette Index are available in '`fclust`' package (Ferraro & Giordani, 2015), and can be used as follows: 

```{r}
res.fcm4 <- ppclust2(res.fcm, "fclust")
idxsf <- SIL.F(res.fcm4$Xca, res.fcm4$U, alpha=1)
idxpe <- PE(res.fcm4$U)
idxpc <- PC(res.fcm4$U)
idxmpc <- MPC(res.fcm4$U)
```	 
```{r}
cat("Partition Entropy: ", idxpe)
cat("Partition Coefficient: ", idxpc)
cat("Modified Partition Coefficient: ", idxmpc)
cat("Fuzzy Silhouette Index: ", idxsf)
```	 

# References{-}
***
Anderson, E. (1935). The irises of the GASPE peninsula, in *Bull. Amer. Iris Soc.*, 59: 2-5.

Bezdek, J.C. (1974). Cluster validity with fuzzy sets. *J. Cybern.*, 3: 58-73.

Bezdek J.C. (1981). \emph{Pattern recognition with fuzzy objective function algorithms}. Plenum, NY. https://isbnsearch.org/isbn/0306406713

Ferraro, M.B. & Giordani, P. (2015). A toolbox for fuzzy clustering using the R programming language. *Fuzzy Sets and Systems*, 279, 1-16. http://dx.doi.org/10.1016/j.fss.2015.05.001

Kassambara, A. & Mundt, F. (2017). factoextra: Extract and Visualize the Results of Multivariate Data Analyses. R package version 1.0.5. https://CRAN.R-project.org/package=factoextra

Maechler, M., Rousseeuw, P., Struyf, A., Hubert, M. & Hornik, K.(2017).  cluster: Cluster Analysis Basics and Extensions. R package version 2.0.6. https://CRAN.R-project.org/package=cluster