Mercator for Continuous Data
Introduction
The Mercator package is intended to facilitate the exploratory analysis of data sets. It consists of two main parts, one devoted to tools for binary matrices, and the other focused on visualization. These visualization tools can be used with binary, continuous, categorical, or mixed data, since they only depend on a distance matrix. Each distance matrix can be visualized with multiple techniques, providing a consistent interface to thoroughly explore the data set. In thus vignette, we illustrate the visualization of a continuous data set.
The Mercator Class
First we load the package.
Now we load a “fake” set of synthertic continuous data that comes with the Mercator package. We will use this data set to illustrate the visualization methods.
## [1] "fakeclin" "fakedata"## [1] 776 300## [1] 300 4Visualization
The Mercator Package currently supports visualization of data with methods that include standard techniques (hierarchical clustering) and large-scale visualizations (multidimensional scaling (MDS),T-distributed Stochastic Neighbor Embedding (t-SNE), and iGraph.) In order to create a Mercator object, we must provide
- a distance matrix,
- a character string describing the metric that was used,
- a character string defining an initial visualization method,
- and a desired number of clusters.
We are going to start with hierarchical clustering, with an arbitrarily assigned number of 4 groups.
## An object of the 'Mercator' class, using the ' euclid ' metric, of size
## [1] 300 300
## Contains these visualizations: hclustHierarchical Clustering
Here is a “view” of the dendrogram produced by hierarchical clustering. Note that view is an argument to the plot function for Mercator objects. If omitted, the first view in the list is used.
Hierarchical clustering.
The dendrogram suggests that there might actually be more than 4 subtypes in the data, but we’re going to wait until we see some other views of the data before doing anything about that.
t-Distributed Stochastic Neighbor Embedding
Mercator can use t-distributed Stochastic Neighbor Embedding (t-SNE) plots for visualizing large-scale, high-dimensional data in a 2-dimensional space.
mercury <- addVisualization(mercury, "tsne")
plot(mercury, view = "tsne", main="t-SNE; Euclidean Distance")
A t-SNE Plot.
The t-SNE plot also suggests more than four subtypes; perhaps as many as seven or eight.
Optional t-SNE parameters, such as perplexity, can be used to fine-tune the plot when the visualization is created. Using addVisualization to create a new, tuned plot of an existing type overwrites the existing plot of that type.
## Warning in addVisualization(mercury, "tsne", perplexity = 15): Overwriting an
## existing visualization:tsne
A t-SNE plot with smaller perplexity.
Multi-Dimensional Scaling
Mercator allows visualization of multi-dimensional scaling (MDS) plots, as well.
mercury <- addVisualization(mercury, "mds")
plot(mercury, view = "mds", main="MDS; Euclidean Distance")
Multi-dimensional scaling.
Interestingly, the MDS plot (which is equivalent to principal components analysis, PCA, when used with Euclidean distances) doesn’t provide clear evidence of more than three or four subtypes. That’s not surprising, since groups separated in high dimensions can easily be flattened by linear projections.
iGraph
Mercator can visualize complex networks using iGraph. IN the next chunk of code, we add an iGraph visualization. We then look at the resulting graph, using three different “layouts”. The Q parameter is a cutoff (qutoff?) on the distance used to include edges; if omitted, it defaults to the 10th percentile. We arrived at the value Q=24 shown here by trial-and-error, though one could plot a histogram of the distances (via hist(mercury@distance)) to make a more informed choice.
## Warning: Non-positive edge weight found, ignoring all weights during graph
## layout.
iGraph views.
iGraph views.
iGraph views.
The last layout, in this case, is possibly not so nice.
Cluster Identities
We can use the getClusters function to determine the cluster assignments and use these for further manipulation. For example, we can easily determine cluster size.
## my.clust
## 1 2 3 4
## 82 68 74 76We might also compare the cluster labels to the “true” subtypes in our “fake” data set.
##
## my.clust 1 2 3 4 5 6 7 8
## 1 0 40 0 1 0 41 0 0
## 2 30 0 36 0 0 0 2 0
## 3 4 0 0 34 0 0 36 0
## 4 0 0 0 0 41 0 0 35Silhouette-Width Barplots
The barplot method produces a version of the “silhouette width” plot from Kaufman and Rouseeuw (and borrowed from the cluster package).
Silhouette widths.
For each observation in the data set, the silhouette width is a measure of how much we believe that it is placed in the correct cluster. Here we see that about 10% to 20% of the observations in each cluster may be incorrectly classified, since their silhouette widths are negative.
Reclustering
We can “recluster” by specifying a different number of clusters.
A t-SNE plot after reclustering.
The silhouette-width barplot changes with the number of clusters. In this case, it suggests that eight clusters may not describe the data as well as four. However, the previous t-SNE plot also shows that the algorithmically derived cluster labels don’t seem to match the visible clusters very well.
Silhouette widths with eight clusters.
Hierarchical Clusters
The clustering algorithm used within Mercator is partitioning around medoids (PAM). You can run any clustering algorithm of your choice and assign the resulting cluster labels to the Mercator object. As part of our visualizations, we have laready pefomred hierarchcai clustering. So, we can assign cluster labels by cutting the branches of the dendrogram. We can use the cutree function after extracting the dendrogram from the view. (Note that we use the remapColors function here to try to keep the same color assignments for the PAM-defined clusters and the hierarchical clusters.)
hclass <- cutree(mercury@view[["hclust"]], k = 8)
neptune <- setClusters(mercury, hclass)
neptune <- remapColors(mercury, neptune)A t-SNE plot colored by heierachical clustering.
The assignments by hierarchical clustering appear to more consistent thant eh PAM clusters with the t-SNE plot, though one suspect that the assignemnts among the pink, red, and orchid groups may be difficult. The silhouette width barplot (below) confirms that hierarchical clustering works better than PAM on this data set. Only the “red” group #4 contains a large number of apparently misclassified samples.
True Clusters
For our fake data set, since we simulated it, we know the “true” labels. So, we can “recluster” using the true assignments.
venus <- setClusters(neptune, fakeclin$Type)
venus <- remapColors(neptune, venus)
plot(venus, view = "tsne")
A t-SNE plot with true cluster labels.
Silhouette widths with true clusters.
We can also see how the hierarchical clustering compare to the true cluster assignments.
##
## 1 2 3 4 5 6 7 8
## 1 41 0 2 1 0 0 0 0
## 2 0 21 0 0 0 0 0 1
## 3 0 0 38 0 0 0 0 0
## 4 0 4 0 34 0 0 0 0
## 5 0 0 0 0 35 2 0 0
## 6 0 0 0 0 0 39 0 0
## 7 0 0 0 0 0 0 36 0
## 8 0 9 0 0 0 0 2 35Appendix
This analaysis was performed in the following environment:
## R version 4.4.1 (2024-06-14)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.1 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: Etc/UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] Mercator_1.1.5 Thresher_1.1.4 PCDimension_1.1.13
## [4] ClassDiscovery_3.4.5 oompaBase_3.2.9 cluster_2.1.6
## [7] rmarkdown_2.28
##
## loaded via a namespace (and not attached):
## [1] gtable_0.3.5 xfun_0.48 bslib_0.8.0
## [4] ggplot2_3.5.1 lattice_0.22-6 vctrs_0.6.5
## [7] tools_4.4.1 stats4_4.4.1 flexmix_2.3-19
## [10] Polychrome_1.5.1 tibble_3.2.1 fansi_1.0.6
## [13] highr_0.11 pkgconfig_2.0.3 Matrix_1.7-1
## [16] KernSmooth_2.23-24 scatterplot3d_0.3-44 lifecycle_1.0.4
## [19] kohonen_3.0.12 compiler_4.4.1 munsell_0.5.1
## [22] movMF_0.2-8 htmltools_0.5.8.1 sys_3.4.3
## [25] buildtools_1.0.0 sass_0.4.9 yaml_2.3.10
## [28] pillar_1.9.0 jquerylib_0.1.4 MASS_7.3-61
## [31] openssl_2.2.2 cachem_1.1.0 viridis_0.6.5
## [34] mclust_6.1.1 RSpectra_0.16-2 cpm_2.3
## [37] digest_0.6.37 Rtsne_0.17 slam_0.1-54
## [40] kernlab_0.9-33 maketools_1.3.1 changepoint_2.2.4
## [43] ade4_1.7-22 fastmap_1.2.0 grid_4.4.1
## [46] oompaData_3.1.4 colorspace_2.1-1 cli_3.6.3
## [49] magrittr_2.0.3 utf8_1.2.4 scales_1.3.0
## [52] umap_0.2.10.0 igraph_2.1.1 nnet_7.3-19
## [55] reticulate_1.39.0 gridExtra_2.3 png_0.1-8
## [58] askpass_1.2.1 zoo_1.8-12 modeltools_0.2-23
## [61] evaluate_1.0.1 knitr_1.48 viridisLite_0.4.2
## [64] rlang_1.1.4 Rcpp_1.0.13 dendextend_1.18.1
## [67] glue_1.8.0 jsonlite_1.8.9 R6_2.5.1