Home > voicebox > entropy.m

# entropy

## PURPOSE

ENTROPY calculates the entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP)

## SYNOPSIS

function h=entropy(p,dim,cond,arg,step)

## DESCRIPTION

```ENTROPY calculates the entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP)

Inputs:  P        is a vector or matrix of probabilities - one dimension per variable
DIM      lists dimensions along which to evaluate the entropy [default: 1st non singleton dimension]
COND     lists dimensions to use as conditional variables [default - none]
ARG      lists dimensions to use as parameters in the ouput [default - none]
STEP     for continuous distributions STEP gives the sample increment for each dimension of P
if STEP is a scalar, the increment is assumed to be the same for each dimension

Outputs:  H        is the entropy. It will have the same number of dimensions as the length of the ARG input.
If the STEP argument is specified then this will be the differential entropy.

Example: Suppose P(W,X,Y,Z) represents the joint probability of four correlated random variables

(a) H(W,X,Y,Z) = entropy(P,[1 2 3 4]).
(b) H(W) = entropy(P), or equivalently entropy(P,1)
(c) H(W,Z | X,Y) = entropy(P,[1 4],[2 3])
(d) H(W | X, Z=z) = entropy(P,1,2,4); this is a function of z and will be a column vector

As a special case, if the dimensions included in DIM are all singletons, the entries in P are treated
as Bernoulli variable probabilities.```

## CROSS-REFERENCE INFORMATION

This function calls:
• entropy ENTROPY calculates the entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP)
This function is called by:
• entropy ENTROPY calculates the entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP)

## SOURCE CODE

```0001 function h=entropy(p,dim,cond,arg,step)
0002 %ENTROPY calculates the entropy of discrete and sampled continuous distributions H=(P,DIM,COND,ARG,STEP)
0003 %
0004 %  Inputs:  P        is a vector or matrix of probabilities - one dimension per variable
0005 %           DIM      lists dimensions along which to evaluate the entropy [default: 1st non singleton dimension]
0006 %           COND     lists dimensions to use as conditional variables [default - none]
0007 %           ARG      lists dimensions to use as parameters in the ouput [default - none]
0008 %           STEP     for continuous distributions STEP gives the sample increment for each dimension of P
0009 %                    if STEP is a scalar, the increment is assumed to be the same for each dimension
0010 %
0011 % Outputs:  H        is the entropy. It will have the same number of dimensions as the length of the ARG input.
0012 %                    If the STEP argument is specified then this will be the differential entropy.
0013 %
0014 % Example: Suppose P(W,X,Y,Z) represents the joint probability of four correlated random variables
0015 %
0016 %               (a) H(W,X,Y,Z) = entropy(P,[1 2 3 4]).
0017 %               (b) H(W) = entropy(P), or equivalently entropy(P,1)
0018 %               (c) H(W,Z | X,Y) = entropy(P,[1 4],[2 3])
0019 %               (d) H(W | X, Z=z) = entropy(P,1,2,4); this is a function of z and will be a column vector
0020 %
0021 % As a special case, if the dimensions included in DIM are all singletons, the entries in P are treated
0022 % as Bernoulli variable probabilities.
0023
0024 %       Copyright (C) Mike Brookes 2006
0025 %      Version: \$Id: entropy.m 713 2011-10-16 14:45:43Z dmb \$
0026 %
0027 %   VOICEBOX is a MATLAB toolbox for speech processing.
0029 %
0030 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
0031 %   This program is free software; you can redistribute it and/or modify
0033 %   the Free Software Foundation; either version 2 of the License, or
0034 %   (at your option) any later version.
0035 %
0036 %   This program is distributed in the hope that it will be useful,
0037 %   but WITHOUT ANY WARRANTY; without even the implied warranty of
0038 %   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
0039 %   GNU General Public License for more details.
0040 %
0041 %   You can obtain a copy of the GNU General Public License from
0042 %   http://www.gnu.org/copyleft/gpl.html or by writing to
0043 %   Free Software Foundation, Inc.,675 Mass Ave, Cambridge, MA 02139, USA.
0044 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
0045
0046 if nargin<5
0047     stp=zeros(ndims(p),1);
0048 else
0049     stp=repmat(step(1),ndims(p),1);
0050     stp(1:length(step))=step(:);
0051     stp=log2(stp);
0052 end
0053 if nargin<4
0054     arg=[];
0055 else
0056     arg=arg(arg>0);
0057 end
0058 if nargin<3
0059     cond=[];
0060 else
0061     cond=cond(cond>0);
0062 end
0063 if ~length(cond)
0064     s=size(p);
0065     if nargin<2
0066         dim=find(s>=min(2,max(s)));
0067         dim=dim(1);
0068     else
0069         dim=dim(dim>0);
0070     end
0071     st=prod(s);
0072     sd=prod(s(dim));
0073     sa=prod(s(arg));
0074     marg=1:length(s);
0075     marg(arg)=0;
0076     marg(dim)=0;
0077     marg=marg(marg>0);
0078     sm=st/sd/sa;
0079     if sm>1
0080         ip=[arg dim(:)' marg(:)'];
0081         sp=[s([arg dim(:)']) prod(s(marg))];
0082         q=sum(reshape(permute(p,[arg(:)' dim(:)' marg]),sa,sd,sm),3);
0083     else
0084         q=reshape(permute(p,[arg(:)' dim(:)' marg]),sa,sd);
0085     end
0086     if sd==1
0087         h=-log2(q+(q==0)).*q-log2(1-q+(q==1)).*(1-q);   % special treatment for bernoulli variables
0088     else
0089         sq=sum(q,2);
0090         h=sum(-log2(q+(q==0)).*q,2)./sq+log2(sq);
0091     end
0092     if length(arg)>1
0093         h=reshape(h,s(arg));
0094     end
0095     h=h+sum(stp(dim));
0096 else
0097     % we could probably make this more efficient by avoiding the recursive call
0098     h=entropy(p,[dim(:); cond(:)],0,arg)-entropy(p,cond,0,arg);
0099 end```

Generated on Mon 06-Aug-2018 14:48:32 by m2html © 2003