| Title: | Information-Theoretic Measures |
|---|---|
| Description: | Implements various measures of information theory based on several entropy estimators. |
| Authors: | Patrick E. Meyer |
| Maintainer: | Patrick E. Meyer <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 1.2.0.1 |
| Built: | 2024-11-18 06:42:04 UTC |
| Source: | CRAN |
condentropy takes two random vectors, X and Y, as input and returns the
conditional entropy, H(X|Y), in nats (base e), according to the entropy estimator method.
If Y is not supplied the function returns the entropy of X - see entropy.
condentropy(X, Y=NULL, method="emp")condentropy(X, Y=NULL, method="emp")
X |
data.frame denoting a random variable or random vector where columns contain variables/features and rows contain outcomes/samples. |
Y |
data.frame denoting a conditioning random variable or random vector where columns contain variables/features and rows contain outcomes/samples. |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
condentropy returns the conditional entropy, H(X|Y), of X given Y in nats.
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.
entropy, mutinformation, natstobits
data(USArrests) dat<-discretize(USArrests) H <- condentropy(dat[,1], dat[,2], method = "mm")data(USArrests) dat<-discretize(USArrests) H <- condentropy(dat[,1], dat[,2], method = "mm")
condinformation takes three random variables as input and computes the
conditional mutual information in nats according to the entropy estimator method.
If S is not supplied the function returns the mutual information between X and Y - see mutinformation
condinformation(X, Y, S=NULL, method="emp")condinformation(X, Y, S=NULL, method="emp")
X |
vector/factor denoting a random variable or a data.frame denoting a random vector where columns contain variables/features and rows contain outcomes/samples. |
Y |
another random variable or random vector (vector/factor or data.frame). |
S |
the conditioning random variable or random vector (vector/factor or data.frame). |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
condinformation returns the conditional mutual information, I(X;Y|S), in nats.
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.
mutinformation, multiinformation, interinformation, natstobits
data(USArrests) dat<-discretize(USArrests) I <- condinformation(dat[,1],dat[,2],dat[,3],method="emp")data(USArrests) dat<-discretize(USArrests) I <- condinformation(dat[,1],dat[,2],dat[,3],method="emp")
discretize discretizes data using the equal frequencies or equal width binning algorithm.
"equalwidth" and "equalfreq" discretizes each random variable (each column) of the data into nbins.
"globalequalwidth" discretizes the range of the random vector data into nbins.
discretize( X, disc="equalfreq", nbins=NROW(X)^(1/3) )discretize( X, disc="equalfreq", nbins=NROW(X)^(1/3) )
X |
A data.frame containing data to be discretized. The columns contains variables and the rows samples. |
disc |
The name of the discretization method to be used :"equalfreq", "equalwidth" or "globalequalwidth" (default : "equalfreq") - see references. |
nbins |
Integer specifying the number of bins to be used for the discretization. By default the number of bins is set to
|
discretize returns the discretized dataset.
Patrick E. Meyer, Frederic Lafitte, Gianluca Bontempi, Korbinian Strimmer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Dougherty, J., Kohavi, R., and Sahami, M. (1995). Supervised and unsupervised discretization of continuous features. In International Conference on Machine Learning.
Yang, Y. and Webb, G. I. (2003). Discretization for naive-bayes learning: managing discretization bias and variance. Technical Report 2003/131 School of Computer Science and Software Engineering, Monash University.
data(USArrests) nbins<- sqrt(NROW(USArrests)) ew.data <- discretize(USArrests,"equalwidth", nbins) ef.data <- discretize(USArrests,"equalfreq", nbins) gew.data <- discretize(USArrests,"globalequalwidth", nbins)data(USArrests) nbins<- sqrt(NROW(USArrests)) ew.data <- discretize(USArrests,"equalwidth", nbins) ef.data <- discretize(USArrests,"equalfreq", nbins) gew.data <- discretize(USArrests,"globalequalwidth", nbins)
entropy takes the dataset as input and computes the
entropy according
to the entropy estimator method.
entropy(X, method="emp")entropy(X, method="emp")
X |
data.frame denoting a random vector where columns contain variables/features and rows contain outcomes/samples. |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
entropy returns the entropy of the data in nats.
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
J. Beirlant, E. J. Dudewica, L. Gyofi, and E. van der Meulen (1997). Nonparametric entropy estimation : An overview. Journal of Statistics.
Hausser J. (2006). Improving entropy estimation and the inference of genetic regulatory networks. Master thesis of the National Institute of Applied Sciences of Lyon.
condentropy, mutinformation, natstobits
data(USArrests) H <- entropy(discretize(USArrests),method="shrink")data(USArrests) H <- entropy(discretize(USArrests),method="shrink")
The package infotheo provide various estimators for computing information-theoretic measures from data
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
entropy, condentropy, mutinformation, condinformation, multiinformation, interinformation, natstobits
interinformation takes a dataset as input and computes the
the interaction information among the random variables in the dataset using
the entropy estimator method. This measure is also called synergy or complementarity.
interinformation(X, method="emp")interinformation(X, method="emp")
X |
data.frame denoting a random vector where columns contain variables/features and rows contain outcomes/samples. |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
interinformation returns the interaction information (also called synergy or complementarity), in nats, among the random variables (columns of the data.frame).
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Jakulin, A. and Bratko, I. (2004). Testing the significance of attribute interactions. In Proc. of 21st International Conference on Machine Learning (ICML).
McGill, W. J. (1954). Multivariate information transmission. Psychometrika, 19.
condinformation, multiinformation, mutinformation, natstobits
data(USArrests) dat<-discretize(USArrests) ii <- interinformation(dat, method = "sg")data(USArrests) dat<-discretize(USArrests) ii <- interinformation(dat, method = "sg")
multiinformation takes a dataset as input and computes the
multiinformation (also called total correlation) among the random variables in the dataset.
The value is returned in nats using the entropy estimator estimator.
multiinformation(X, method ="emp")multiinformation(X, method ="emp")
X |
data.frame containing a set of random variables where columns contain variables/features and rows contain outcomes/samples. |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
multiinformation returns the multiinformation (also called total correlation) among the variables in the dataset (in nats).
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Studeny, M. and Vejnarova, J. (1998). The multiinformation function as a tool for measuring stochastic dependence. In Proceedings of the NATO Advanced Study Institute on Learning in graphical models,
condinformation, mutinformation, interinformation, natstobits
data(USArrests) dat<-discretize(USArrests) M <- multiinformation(dat)data(USArrests) dat<-discretize(USArrests) M <- multiinformation(dat)
mutinformation takes two random variables as input and computes the
mutual information in nats according to the entropy estimator method.
If Y is not supplied and X is a matrix-like argument, the function returns a matrix of mutual
information between all pairs of variables in the dataset X.
mutinformation(X, Y, method="emp")mutinformation(X, Y, method="emp")
X |
vector/factor denoting a random variable or a data.frame denoting a random vector where columns contain variables/features and rows contain outcomes/samples. |
Y |
another random variable or random vector (vector/factor or data.frame). |
method |
The name of the entropy estimator. The package implements four estimators :
"emp", "mm", "shrink", "sg" (default:"emp") - see details.
These estimators require discrete data values - see |
"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
mutinformation returns the mutual information I(X;Y) in nats.
Patrick E. Meyer
Meyer, P. E. (2008). Information-Theoretic Variable Selection and Network Inference from Microarray Data. PhD thesis of the Universite Libre de Bruxelles.
Cover, T. M. and Thomas, J. A. (1990). Elements of Information Theory. John Wiley, New York.
condinformation, multiinformation, interinformation, natstobits
data(USArrests) dat<-discretize(USArrests) #computes the MIM (mutual information matrix) I <- mutinformation(dat,method= "emp") I2<- mutinformation(dat[,1],dat[,2])data(USArrests) dat<-discretize(USArrests) #computes the MIM (mutual information matrix) I <- mutinformation(dat,method= "emp") I2<- mutinformation(dat[,1],dat[,2])
natstobits takes a value in nats (a double) as input and returns the value in bits (a double).
natstobits(H)natstobits(H)
H |
double denoting a value (in nats), as returned by one of the function of the infotheo package |
Information-theoretic quantities can have different units depending on the base of the logarithm used in their computation. All the function of tha package use a base e, hence the unit is the nat. The value in bit is given by using the base 2, hence the conversion is done by multiplying by log2(e) = 1.442695.
natstobits returns a double that is the conversion of the nats value into bits.
Patrick E. Meyer
data(USArrests) H <- entropy(discretize(USArrests)) natstobits(H)data(USArrests) H <- entropy(discretize(USArrests)) natstobits(H)