Pattern Classification

for postgraduates

(9 weeks from 21 February to 17 April, 2007)

Akira Imada

Brest State Technical University, Belarus

(last modified on)

May 7, 2007

INDEX

• Introduction

– What is Classification?

∗ Let’s start with an example — Female sermon and Male sermon.

• Bayesian Classification – I – What if we know only prior density of each class?

– How conditional posterior distribution probability improves liklyhood of classes?

– Bayesian Decision Theory.

– Classiffication by likelyhood-ratio.

• Bayesian Classification – II

– When class-conditional distribution of obserbations is Gaussian pdf? ∗ One-dimensional case.

∗ Two-dimensional case.

∗ Higher-dimensional case.

• Bayesian Classification – III

– When we know class-conditional distribution p(x|ω) but don’t know some of

its parameters θ? ∗ Maximam likelihood parameter estimation.

∗ Bayesian parameter estimation.

• Bayesian Classification – IV

– When we don’t know know class-conditional distribution p(x|ω)?

∗ Parzen window approach.

∗ Probabilistic Neural Network implementaion of Parzen window approach.

∗ k_{m}-nearest neighbor approach.

• Bayesian Belief Network.

• Hidden Markov Model.

• Fuzzy Logic Classification.

1 An example to start with.

In general, we need what features can be used for pattern classification purpose. This is, of course, one of big issues in pattern classification, but here we assume that we already know features which efficiently can be used to classify our target patterns.

1.1 Which do you like better female or male?

I mean not human, but salmon. Well, we now assume, as an example, we want to classify salmons into female ones and male ones. The situation is we have thousands of salmons and our task is classify salmon not yet seen into female and male only by observing one feature x — length of the salmon.

1.2 If we only know prior probability.

If we don’t know any prior information except for how many so far we have were female and male. Now we denote the class of female salmon as ω_{1} and the class of male salmon as ω_{2}. Then above mentioned “how many so far we have were female and male” are denoted as P (ω_{1}) and P (ω_{2}), respectively, and called prior probability.

In this case, all we should act is with the following rule.

Rule 1 (Classification only with prior probability) If P (ω_{1}) > P (ω_{2}) then classify it to ω_{1} otherwise ω_{2}.

This might seem to be a trivial reaction.

1.3 If we also know posterior distribution of features.

If we assume that we know further distribution of length s in each of female and male salmon, that is, p(x|ω_{1}) and p(x|ω_{2}), then our task of classify salmons with an observation of x is a little more sophisticated.

Note here p(·) denotes a probability and P (·) denotes a density distribution.

Instinctively, we might put the threshold θ as in Fig. 1. If both classes are equally important then we put it at the center of those two distribution functions. But sometimes we might put it in different way like in Fig. 2, if we take it into acount that the risk of misclassify x into ω_{2} while x is ω_{2} is important^{1}.

In the example of Fig. 1.3, you might imagine a classification fo human female and male by one feature — frequency of her/his voice. In this case p(x|ω_{1}) and p(x|ω_{1}) would be well separated and easy to classify.

1In the case of salmon, we tend to be happier if the one we bought a male salmon with cheaper price than female due to eggs, was mistakenly female.

Figure 1: An example of two distribution of the observation x w.r.t. each class of ω_{1} and ω_{2}. Here reader might image that distribution of length of female salmon and male sarmon.

Figure 2: Yet another example where two distributions are closer than the first example. Two threshold are shown, considering a risk of miss-classifcation.

2 What is Bayesian Classification?

Assume we should guess the class that will come next with the knowledge of

(1) Prior probabilities of class-1 ω_{1} and class-2 ω_{2}, that is, P (ω_{1}) and P (ω_{2}).

(2) Class conditional probability of the observatin of x of each of the two classes, that is, p(x|ω_{1}) and p(x|ω_{2}).

Then we can calculate P (ω_{i})|x) — the probability that class is ω_{i} under the condition where x is observed as:

P (ω_{i}|x) = \frac{p(x|ω_{i})P (ω_{i})}{p(x)} (i = 1, 2),  (1)

where

p(x) = p(x|ω_{1})P (ω_{1}) + p(x|ω_{2})P (ω_{2})  (2)

which is sometimes called evidence. Then our rule is:

Rule 2 (Classification with posterior probability) If P (ω_{1}|x) > P (ω_{2}|x) then classify it to ω_{1} otherwise ω_{2}.

2.1 Bayesian Decision Theory

Assume now we have multiple features:

x_{1}, x_{2}, ⋯, x_{d}

mulitple classes:

ω_{1}, ω_{2}, ⋯, ω_{c}

also actions:

α_{1}, α_{2}, ⋯, α_{a}

instead of guessing which class. Then loss function λ(α_{i}|ω_{j}) is defined as the loss when we take an action alpha_{i} when the situation is ω_{j}).

If we take an action α_{i} when we observe x while the true situation is ω_{j}, the conditional risk is defiened as

R(α_{i}|x) = ∑_{j=1}^{c} λ(α_{i}|ω_{j})p(ω_{j}|x).  (3)

Note that we want to take the action so as to minimize this risk.

Under continuous feature x we have observed, assume c finite categories: ω_{1}, ω_{2}, ...ω_{c} and a finite possible actions α_{1}, α_{2}, ...α_{a}. Then a loss function λ(α_{i}|ω_{j}) is defiened as the loss caused by taking an action α_{i} when ω_{j} is the situation.

We now assume that we take action α_{i} when we observe x while the true situation is ω_{j} the loss function λ(α_{i}|ω_{j}) will be defiend. Then expected loss will be

R(α_{i}|x) = ∑_{j=1}^{c} λ(α_{i}|ω_{j})P (ω_{j}|x)  (4)

which called a risk. Recall here

P (ω_{j}|x) = \frac{p(x|ω_{j})P (ω_{j})}{p(x)}  (5)

where

p(x) = ∑_{j=1}^{c} p(x|ω_{j})P (ω_{j}).  (6)

2.2 Two-category classification

Let’s simplify as λ_{ij} = λ(α_{i}|ω_{j}), that is,

Λ = ( \begin{matrix}λ_{11} & λ_{12} \\ λ_{21} & λ_{22} \end{matrix} ) ,  (7)

Then

R(α_{1}|x) = λ_{11}P (ω_{1}|x) + λ_{12}P (ω_{2}|x)
R(α_{2}|x) = λ_{21}P (ω_{1}|x) + λ_{22}P (ω_{2}|x)  (8)

Here we have:

Rule 3 (Rule for classification) If R(α_{1}|x) < R(α_{1}|x) then classify xto ω_{1} otherwise to ω_{2}.

2.2.1 likelihood ratio

If we further assume that we have one feature x instead of multiple x, the if part of the rule above will be

(λ_{21} − λ_{11})P (ω_{1}|x) > (λ_{12} − λ_{22})P (ω_{2}|x).  (9)

And usually we can assume by definition that λ_{21} > λ_{21} > and λ_{21} > λ_{21} >, the rule will be:

Rule 4 (Rule for classification) If the following holds

\frac{p(x|ω_{1})}{p(x|ω_{2})} > \frac{λ_{12} − λ_{22}}{λ_{21} − λ_{11}} \frac{P (ω_{2})}{P (ω_{1})}  (10)

then classify x to ω_{1} otherwise to ω_{2}.

Note that the left-hand side is a function of x and is called a likelihood ratio and the right-hand side is a quantity and is a threshold to discriminate two classes.

2.3 Discriminant function

Let’s define a function, not probability or not density but just a function for the purpose of discriminate the space in to categories.

3 Examples of Decision Surface

3.1 1-D Gaussian case.

We now assume

p(x|ω_{1}) = \frac{1}{\sqrt{2π}σ_{1}} exp\{−\frac{1}{2} \frac{(x − μ_{1})^{2}}{σ_{1}^{2}} \}

and

p(x|ω_{2}) = \frac{1}{\sqrt{2π}σ_{2}} exp\{−\frac{1}{2} \frac{(x − μ_{2})^{2}}{σ_{2}^{2}} \}.

Hence, one-dimensional version of our Rule-2 is,

If P (ω_{1}|x) > P (ω_{2}|x) then classify x to ω_{1} otherwise ω_{2}.

So, recalling

P (ω_{i}|x) = \frac{p(x|ω_{i})P (ω_{i})}{P (x)}

and P (x) is common for both i = 1 and i = 2, the IF-part of the above rule is

\frac{1}{\sqrt{2π}σ_{1}} exp\{−\frac{1}{2} \frac{(x − μ_{1})^{2}}{σ_{1}^{2}} \}P (ω_{1}) > \frac{1}{\sqrt{2π}σ_{2}} exp\{−\frac{1}{2} \frac{(x − μ_{2})^{2}}{σ_{2}^{2}} \}P (ω_{2})

Now that we see all the terms in the equation are positive, the comparison holds true if we take logarithm based on e on both-hands of the equation.^{2} Therefore

ln \frac{1}{\sqrt{2π}} + ln \frac{1}{σ_{1}} − \frac{1}{2} \frac{(x − μ_{1})^{2}}{σ_{1}^{2}} + ln P (ω_{1}) > ln \frac{1}{\sqrt{2π}} + ln \frac{1}{σ_{2}} − \frac{1}{2} \frac{(x − μ_{2})^{2}}{σ_{2}^{2}} + ln P (ω_{2})

Excersize 1 Obtain the decision boundary x_{0} when the two classes follow the Gaussian distributions with N(1, \frac{1}{2}) and N(3, \frac{1}{2}) respectively.

3.2 2-D Gaussian case

Recalling Baysian formula Eq. (1)

P (ω_{i}|x) = \frac{p(x|ω_{i})P (ω_{i})}{p(x)} (i = 1, 2),

where p(x) = p(x|ω_{1})P (ω_{1}) + p(x|ω_{2})P (ω_{2}) is just a term to normalize P (ω_{i}|x) to 1. Further, we assume P (ω_{1}) = P (ω_{1}) = 1/2 just for a sake of simplicity here. So our decision rule to determine which class of ω_{1} or ω_{2} should be assigned to x:

“If P (ω_{1}|x) > P (ω_{2}|x) then x is ω_{1}, otherwise ω_{2}”

just depends on p(x|ω_{i}) which we assume to be Gaussian p.d.f. That is

p(x|ω_{i}) = \frac{1}{(2π)^{d/2}|Σ_{i}|^{1/2}} exp\{−\frac{1}{2}(x − μ_{i})^{t}Σ_{i}^{−1}(x − μ_{i})\}  (11)

2That is,

A > B ⇔ ln(A) > ln(B).

3.2.1 What will borders look like on what condition?

Now that we restrict our universe in two-dimensional space, we use a notation (x, y) instead of (x_{1}, x_{2}). So we now express x = (x, y) Furthermore, both of our two classes are assumed to follow the Gaussian p.d.f. whose μ are μ_{1} = (0, 0) and μ_{2} = (1, 0), and Σ are

Σ_{1} = ( \begin{matrix}a_{1} & 0 \\ 0 & b_{1} \end{matrix} ) , Σ_{2} = ( \begin{matrix}a_{2} & 0 \\ 0 & b_{2} \end{matrix} )

Under this simple condition, our inverse matrix is simply, |Σ_{1}| = a_{1}b_{1} and |Σ_{2}| = a_{2}b_{2}. So, we now know

Σ_{1}^{−1} = \frac{1}{a_{1}b_{1}} ( \begin{matrix}b_{1} & 0 \\ 0 & a_{1} \end{matrix} ) = ( \begin{matrix}1/a_{1} & 0 \\ 0 & 1/b_{1} \end{matrix} )

and in the same way

Σ_{2}^{−1} = \frac{1}{a_{2}b_{2}} ( \begin{matrix}b_{2} & 0 \\ 0 & a_{2} \end{matrix} ) = ( \begin{matrix}1/a_{2} & 0 \\ 0 & 1/b_{2} \end{matrix} )

Now Eq. (11) is more specifically

p(x|ω_{1}) = \frac{1}{2π\sqrt{a_{1}b_{1}}} exp\{−\frac{1}{2}(x y) ( \begin{matrix}1/a_{1} & 0 \\ 0 & 1/b_{1} \end{matrix} ) ( \begin{matrix}x \\ y \end{matrix} )\}

and

p(x|ω_{2}) = \frac{1}{2π\sqrt{a_{2}b_{2}}} exp\{−\frac{1}{2}(x − 1 y) ( \begin{matrix}1/a_{2} & 0 \\ 0 & 1/b_{2} \end{matrix} ) ( \begin{matrix}x − 1 \\ y \end{matrix} )\}

Then we can define our discriminat function g_{i}(x i = 1, 2 taking logarithm based natural number e as

g_{1}(x) = −\frac{1}{2}(x y) ( \begin{matrix}1/a_{1} & 0 \\ 0 & 1/b_{1} \end{matrix} ) ( \begin{matrix}x \\ y \end{matrix} ) + ln(2π) + \frac{1}{2} ln(a_{1}b_{1})

and

g_{2}(x) = −\frac{1}{2}(x − 1 y) ( \begin{matrix}1/a_{2} & 0 \\ 0 & 1/b_{2} \end{matrix} ) ( \begin{matrix}x − 1 \\ y \end{matrix} ) + ln(2π) + \frac{1}{2} ln(a_{2}b_{2})

Neglecting here the common term for both equation ln(2π), our new discriminant functions are

g_{1}(x) = −\frac{1}{2}\{\frac{x^{2}}{a_{1}} + \frac{y^{2}}{b_{1}} \} + \frac{1}{2} ln(a_{1}b_{1})

and

g_{2}(x) = −\frac{1}{2}\{\frac{(x − 1)^{2}}{a_{2}} + \frac{y^{2}}{b_{2}} \} + \frac{1}{2} ln(a_{2}b_{2})

Finally, we obtain the border equation from g_{1}(x) − g_{2}(x) = 0.

(\frac{1}{a_{1}} − \frac{1}{a_{2}} )x^{2} + \frac{2}{a_{2}} x + (\frac{1}{b_{1}} − \frac{1}{b_{2}} )y^{2} = \frac{1}{a_{2}} + ln \frac{a_{1}b_{1}}{a_{2}b_{2}}  (12)

We now know that the shape of the border will be either of the following five cases: (i) straight line (ii) circle; (iii) ellipse; (iV) parabola; (v) hyperbola; (Vi) two straight lines, depending on how the points distribute, that is, depending on a_{1}, b_{1}, b_{1} and b_{2} in our situation above.

3.2.2 Examples

Let’s try some calculations under the above assumptions, that is,

(1) Σ_{1} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.10 \end{matrix} ) , Σ_{2} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.10 \end{matrix} )

(2) Σ_{1} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.10 \end{matrix} ) , Σ_{2} = ( \begin{matrix}0.20 & 0 \\ 0 & 0.20 \end{matrix} )

(3) Σ_{1} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.15 \end{matrix} ) , Σ_{2} = ( \begin{matrix}0.20 & 0 \\ 0 & 0.25 \end{matrix} )

(4) Σ_{1} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.15 \end{matrix} ) , Σ_{2} = ( \begin{matrix}0.15 & 0 \\ 0 & 0.10 \end{matrix} )

(5) Σ_{1} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.20 \end{matrix} ) , Σ_{2} = ( \begin{matrix}0.10 & 0 \\ 0 & 0.10 \end{matrix} )

The example the below is somewhat tricky. I wanted an example in which the right-hand side of the equation (12 becames zero and the left-hand side is a product of one-order equations of x and y. As you might know, this is the case where border is made up of two straight lines.

(6) Σ_{1} = ( \begin{matrix}2e & 0 \\ 0 & 0.5 \end{matrix} ) , Σ_{2} = ( \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} )

My quick calculation tentatively results in as follows. See also the Figure below.

(1) 2x = 1

(2) 5(x + 1)^{2} + 5y^{2} = 10 − ln 4

(3) 5(x + 1)^{2} + (8/3)y^{2} = 10 − ln(10/3)

Figure 3: A cloud of 100 points each extracted from a set of two classes and border of the two classes calculated on six different conditions. (Results of (5) and (6) are still fishy and under another trial.)

(4) 5(x + 1)^{2} − (10/3)y^{2} = 10

(5) 20x − 5y^{2} = 10 − ln 2

(6) (1 − 1/2e)x^{2} − x − y^{2} = 0

The final example in this sub-section is more general 2-dimensional case, but (artificially) devised so that calculations won’t become very complicated. We now assume μ_{1} = (0, 0) and μ_{2} = (1, 1), and we both classes share the same Σ:

(7) Σ_{1} = ( \begin{matrix}2.0 & 0.5 \\ 0.5 & 2.0 \end{matrix} ) , Σ_{2} = ( \begin{matrix}5.0 & 4.0 \\ 4.0 & 5.0 \end{matrix} )

3.3 3-D Gaussian case

Here we study only one example. We assume two classes where P (ω_{1}) = P (ω_{1}) = 1/2. In each class, the patterns are distributed with Gaussian p.d.f both have the same covariance matrix

Σ = ( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0.1 & 0.3 & −0.1 \\ 0.1 & −0.1 & 0.3 \end{matrix} )

and means of the distribution are (0, 0, 0)^{T} and (1, 1, 1)^{T}. We now take a look at what our discriminat function

g_{i}(x) = −\frac{1}{2}(x − μ_{i})^{T} Σ_{i}^{−1}(x − μ_{i})  (13)

leads to?

Since we calculate (see APPENDIX III.)

Σ^{−1} = \frac{1}{30} ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} )

Now our discriminat equation g_{1}(x) = g_{2}(x) is

(x_{1}x_{2}x_{3}) ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} ) ( \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ) =

((x_{1} − 1)(x_{2} − 1)(x_{3} − 1)) ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} ) ( \begin{matrix}x_{1} − 1 \\ x_{2} − 1 \\ x_{3} − 1 \end{matrix} )

Further caluculation leads to

((5x_{1} − 2x_{2} − x_{3})(−3x_{1} + 6x_{2} + 3x_{3})) ( \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ) =

((5x_{1} − 2x_{2} − x_{3} − 2)(−3x_{1} + 6x_{2} + 3x_{3} − 6)(−3x_{1} + 3x_{2} + 6x_{3} − 6)) ( \begin{matrix}x_{1} − 1 \\ x_{2} − 1 \\ x_{3} − 1 \end{matrix} )

All the 2nd-order terms are cancelled and we obtain,

7x_{1} + 13x_{2} − 20x_{3} = 14

We now know that it is the plane which discriminates two of these classes ω_{1} and ω_{2}.

3.4 A Higher order Gaussian case

The Equation

g_{i}(x) = ln(p(x|ω_{i}) + ln P (ω_{i}))

(14)

still holds, of course. Now let’s recall that the Gaussoian pdf is

p(x|ω) = \frac{1}{(2π)^{d/2}|Σ|^{1/2}} exp\{−\frac{1}{2}(x − μ)^{t}Σ^{−1}(x − μ)\}  (15)

and as such

g_{i}(x) = −\frac{1}{2}(x − μ_{i})^{t}Σ_{i}^{−1}(x − μ_{i}) − \frac{d}{2} ln(2π) − \frac{1}{2}|Σ_{i}| + ln P (ω_{i})  (16)

We know take a look at cases which simplify situation more or less.

• When Σ_{i} = σ^{2}I In this case, it’s easy to guess samples fall in equal diameter hypershpers. Note, first of all |Σ_{i}| = σ^{2d} and Σ_{i}^{−1} = (1/σ^{2})I. So, we assume g_{i}(x) here to be

g_{i}(x) = −\frac{∥x − μ_{i}∥^{2}}{2σ^{2}} + ln P (ω_{i})  (17)

or, equivalently

g_{i}(x) = −\frac{1}{2σ^{2}} (x^{t}x − 2μ_{i}^{t}x + μ_{i}^{t}μ_{i}) + ln P (ω_{i})  (18)

Neglecting the terms those no affecting to the relation g_{i}(x) − g_{j}(x) = 0 our g_{i}(x) is now

g_{i}(x) = \frac{1}{σ^{2}} μ_{i}^{t}x − \frac{1}{2σ_{i}^{2}} μ_{i}^{t}μ_{i} + ln P (ω_{i})  (19)

Then g_{i}(x) − g_{j}(x) = 0 leads to

\frac{1}{σ^{2}} (μ_{i} − μ_{i})^{t}x − \frac{1}{2σ_{i}^{2}} (∥μ_{i}∥^{2} − ∥μ_{j}∥^{2}) + ln \frac{P (ω_{i})}{P (ω_{j})} = 0  (20)

In this case our classification rule will be

Rule 5 (Minimum Distance Classification) Measure Euclidean distance ∥x−μ∥ for ∀i, then classify x to the class whose mean is nearest to x.

If we carefully modify Eq. (18) we will obtain

w · (x − x_{0})  (21)

Excersize 2 Derive the Eq. (21) specifying w and x_{0}.

Eq. (21) is the equation which can be interpret as

“A hyperplane through x_{0} perpendicular to vecw.”

• When all Σ_{i} = Σ

This condition implies that the patterns in each of both classes distribute like hyperelipsoid. Now that our discriminat function is

g_{i}(x) = −\frac{1}{2}(x − μ)^{t}Σ^{−1}(x − μ)

We again obtain

w · (x − x_{0}) = 0

where w and x_{0} are

w = Σ^{−1}(μ_{i} − μ_{j})  (22)

and

x_{0} = \frac{1}{2}(μ_{i} + μ_{j}) − \frac{ln P (ω_{i})/P (ω_{j})}{(μ_{i} − μ_{j})^{t}Σ^{−1}(μ_{i} − μ_{j})}  (23)

Notice here that w is no more parpendicular to the direction between μ_{i} and μ_{j}.

(Pattern Recognition) 15 Excersize 3 Derive w and x_{0} above. So we modify the avove rule to Rule 6 (Classification by Mahalanobis distance) Assign x to ω_{i} in which Mahalanobis distance from μ_{i} is minimum for ∀i. Yes! This Mahalanobis distance between a and b is defined as

(a − b)^{t}Σ^{−1}(a − b)  (24)

The final example in this sub-section is more general 2-dimensional case, but (artificially) devised so that calculations won’t become very complicated. We now assume μ_{1} = (0, 0) and μ_{2} = (1, 0), and we both classes share the same Σ:

(7) Σ_{1} = ( \begin{matrix}1.1 & 0.3 \\ 0.3 & 1.9 \end{matrix} ) , Σ_{2} = ( \begin{matrix}1.1 & 0.3 \\ 0.3 & 1.9 \end{matrix} )

• When all Σ_{i}’s are arbitrary When no such restriction as above to simplify situation, the discriminant function is

g_{i}(x) = −\frac{1}{2}(x − μ)^{t}Σ_{i}^{−1}(x − μ) − \frac{d}{2} ln 2π − \frac{1}{2} ln |Σ_{i}| + lnP (ω_{i})

Only the term we can neglect now is (d/2) ln 2π. We now apply the identity

(x − y)^{t}A(x − y) = x^{t}Ax − 2(Ay)^{t}x + y^{t}Ay.

Then, we get the following renewed discriminant function

g_{i}(x) = x^{t}W_{i}x + w_{i}^{t}x + w_{i0}  (25)

where

W_{i} = −\frac{1}{2}Σ_{i}^{−1}

w_{i} = Σ_{i}^{−1}μ_{i}

w_{i0} = −\frac{1}{2}μ_{i}^{t}Σ_{i}^{−1}μ_{i} − \frac{1}{2} ln |Σ_{i}| + ln P (ω_{i}).

Hence, g_{i}(x) − g_{j}(x) = 0 leads us to a hyper quadratic form. Or, if you want, we can express it as

(a_{1}x_{1} + a_{2}x_{2} + ⋯ + a_{n}x_{n})(b_{1}x_{1} + b_{2}x_{2} + ⋯ + b_{n}x_{n}) = const.

Namely, the border is either of (i) Hyper-planes; (ii) a pair of hyper-planes; (iii) hypersphere; (iv) yyper-ellipsoid; (v) hyper paraboloid; (vi) hyper-hyperboloid.

3.4.1 2-D cases revisit

4 Parameter Estimation

Goal is to know p(ω_{i}|x). Assume now that we know the density functin for some reason to believe. What we don’t know is the parameter of the density function.

To get a bird’s eye view, we name a few of 1-D examples, besides Gaussian, of such density functions which need to be specified by some parameters.

• Uiform-distribution,

– p(x) = 1/θ ... if x > 0 otherwise 0

• Exponential

– f(x) = θ exp(−θx) ... if x > 0 otherwise 0

• Rayleigh (See Figure below.)

– f(x) = 2θx exp(−θx^{2}) ... if x > 0 otherwise 0

• Poisson

– f(x) = (θ^{x}/x!) exp(−θ) ... for x = 0, 1, 2, ⋯

• Bernoulli

– f(x) = θ^{x}(1 − θ)^{1−x} ... for x = 0, 1

• Binomial

– f(x) = (m!/x!(m − x)! · θ^{x}(1 − θ)^{m−x} ... for x = 0, 1, ⋯, m

Figure 4: Rayleigh distribution function with four different values of θ.

All we don’t know is its parameters to specify the function. But we have an information of prior probability p(θ) and training sample give us p(θ|D) which we expect to have a sharp peak at the true θ.

4.1 Maximum Likelihood Estimation

Before entering this topic, try the following case where we have two classes each of which has only 4 training samples.

Example 1 Assuming in a 2-dimensional space, what if we have a pair of 4 samples from each of two classes? The patterns we have are (8, 3), (4, 3), (6, 2), (6, 4) from ω_{1}0 and (0, 3), (−2, 1), (−4, 3), (−2, 5) from ω_{2}. Try to guess the border between two classes, and then classify a new point (5, 3), for example.

We now further more simplify the situation.

Example 2 Our data is now x_{1} = 3, x_{2} = 8, x_{3} = 2 and x_{4} = 5. Guess what distribution function of these 4 data follows?

This might be a Gaussian distribution but just a brief look at it suggests it more like a Rayleigh distribution.

Figure 5: A cloud of 200 points each with different three different N[μ, Σ].

Excersize 4 (1) We now assume Gaussian with σ = 1 with μ being unknown. Then estimate μ from the same 4 data, i.e. x_{1} = 3, x_{2} = 8, x_{3} = 2 and x_{4} = 5, in the same way as above. (2) Next, create 10, 20, 100 data randamly from N(1, 3), and plot p(D|μ) as a function of μ in each of the three cases.

Rule 7 (An assamption of independence of data) Take θ which maximizes

P (D|θ) = ∏_{k=1}^{n} P (x_{k}|θ)  (26)

In short, choose θ which maximizes

P (D|θ) = ∏_{k=1}^{n} p(x_{k}|θ)  (27)

Then let’s calculate our previous example of one-dimensional four data D = {2, 3, 5, 8} by changign θ from 0.00 to 1.00 with an interval of 0.01. The results are shown the left-most graph of the Fig. 4.1. It might be interesting what if we have more data to estimate. The same procedures are made with the number of data 10, 20, 100 and the results are shown in the same Figure. Important thing is is

“The more data we have, the sharper the graph becomes.”

Figure 6: A cloud of 200 points each with different three different N[μ, Σ].

Figure 7: A cloud of 200 points each with different three different N[μ, Σ].

(Pattern Recognition) 20

Figure 8: A cloud of 200 points each with different three different N[μ, Σ].

4.2 Bayesian parameter estimation

Since we view here the parameter θ as random variables with an information of prior distribution p(θ), we view P (θ|D) as a function of θ. As before, we assume

P (D|θ) = ∏_{k=1}^{n} p(x_{k}|θ)  (28)

where we denote the training data

D_{n} = {x_{1}, x_{2}, ⋯, x_{n}}

which implies

P (D_{n}|θ) = P (x_{n}|θ)P (D_{n−1}|θ)

Therefore the Baysean Formula

p(θ|D) ≈ P (D|θ)P (θ).  (29)

leads us like

P (θ|D_{n}) = P (D_{n})|θ)P (θ)
= P (x_{n}|θ)P (D_{n−1})|θ)P (θ)
= P (x_{n}|θ)P (θ|D_{n−1})

5 Non-paremetric density estimation

Our assumption: so far are

(1) density function p(x|ω) is known

(2) high dimensional density function can be represented as the product of one-dimensional function

Now let’s make a more realistic assamption, ”We don’t know the distribution of samples, i.e., we don’t know the form of the density function. Let’s estimate p(x|ω) from our sample patterns. Or, let’s estimate P (ω|x) directly from our sample patterns. The typical example is Nearest neighbor rule.

6 Unknown Density function Estimation

One of the most elementary formula in probability theory tells us the probability that x will fall in region R will be

P = ∫_{R} p(x)dx  (30)

Recall, if it’s in 1-dimensional case, the probability that x will fall in the region [x_{1}, x_{2}] will be

P = ∫_{x_{1}}^{x_{2}} p(x)dx

The probability of k samples out of n samples of {x_{1}, x_{2}, ⋯, x_{n}} will fall in the region R is

\binom{n}{k}P^{k}(1 − P)^{n−k}  (31)

So, we can assume

k = nP  (32)

holds. On the other hand, Eq. (30) can be

P = p(x) = V  (33)

where V is the volume of region R. This is true on condition that V is small enough. If you want to think of in 1-dimensional space this is

p(x) · (x_{2} − x_{1})

on condition that x_{2} − x_{1} ≈ 0. Hence, we now have a relation

p(x) = \frac{k/n}{V}.  (34)

Excersize 5 Calculate the Eq (31)above in each case of, say, n = 4, 20, 100 and plot the probability as a function of k/n assuming real P is, say, 0.6. Make sure the larger the value of n the sharper the peak.

Now in order to estimate the density p(x) at x, let’s creat a sequence of region R_{1}, R_{2}, R_{3}, ⋯ each of which is a region where estimation is made with n samples.

Now our fundamental Equation becomes

p_{n}(x) = \frac{k_{n}/n}{V_{n}}  (35)

The condition for this to converge is

lim_{n→∞} k_{n} = ∞  (36)

lim_{n→∞} V_{n} = 0  (37)

lim_{n→∞} k_{n}/n = 0  (38)

To realize these conditons we have basically two approaches. In the one approach, we shring V − n as n, the number of training samples, increases, for example, according to V_{n} = V_{0}/\sqrt{n}. In the other approach, we enlarge V_{n} as n increases, for example k_{n} = k_{0}\sqrt{n}. The former is called Parzen-window approach and the latter k_{n} Nearest Neighbor approach, both of which we are going to exploring in the next two subsections.

6.1 Parzen-Window classifier

First of all, we define new function

φ(u) = \{ \begin{matrix}1 & if − \frac{1}{2} < u < \frac{1}{2} \\ 0 & if otherwise \end{matrix}  (39)

Then we get

φ(x − x_{i}/h_{n}) = \{ \begin{matrix}1 & if x_{i} falls within the hypercube \\ 0 & if otherwise \end{matrix}  (40)

where the hypercube is locate such that its center at x and the size of all hedges is h_{n}. Here we can put k_{n} as

k_{n} = ∑_{i=1}^{n} φ(x − x_{i}/h_{n})  (41)

It’s important to note that this is not a function of x at this moment. Now recalling our goal is to obtain the density p(x) at x, we are happy to obtain the following equation.

p_{n}(x) = \frac{1}{n} ∑_{i=1}^{n} \frac{1}{V_{n}} φ(\frac{x − x_{i}}{h_{n}})  (42)

Excersize 6 Assumming p(x) = U(0, 1), try to find what happens if wu use π_{n}(x) as Parzen window function. Draw p_{n}(x) in each case of h_{n} = 1, 1/4, 1/16.

6.1.1 A Neural Network Installation

Figure 9: A cloud of 200 points each with different three different N[μ, Σ].

Figure 10: A cloud of 200 points each with different three different N[μ, Σ].

6.2 Nearest-Neighbor classifier

7 Baesian Blief Network

We now assume we have N variables like our previous example of salmon.

Salmonfemale, male Season spring, summer, autumn, winter Locationsea, river, lake Lengthshort, midium, long

Or in more general form

A(a_{1}, a_{2}, ⋯)

B(b_{1}, b_{2}, ⋯)

C(c_{1}, c_{2}, ⋯)

and we also know, for example,

p(a|b)

p(b|c)

etc... then we can fepresent this by a kind of network like:

We call this network Bayesian Blief Network. Assumption here is we know all ... The aim is

• To classify a new data x.

• To know the probability of an unknown variable from other known variables.

We now take x = (x_{1}, x_{2}, ⋯) as a belief of x_{1}, x_{2}, ⋯ under evidences e = (e_{1}, e_{2}, ⋯) where some of the e_{i} belong to parents variable and the others to children. Then the elementary calculation is

p(x|eall) = p(echildren|x) · p(x|eparents)  (43)

where

p(echildren|x) = p(e_{c_{1}} ⋯ e_{c_{n}}|x) = p(e_{c_{1}}|x) ⋯ p(e_{c_{1}}|x) = Π_{i}p(e_{c_{i}}|x)  (44)

and

p(x|eparents) = p(x|e_{p_{1}}, e_{p_{2}}, ⋯) = ∑ p(x|p_{1}, p_{2}, ⋯)p(p_{1}, p_{2}, ⋯ |e_{p_{1}}, e_{p_{2}}, ⋯) (45)

where

p(p_{1}, p_{2}, ⋯ |e_{p_{1}}, e_{p_{2}}, ⋯) = p(p_{1}|e_{1})· (p_{2}|e_{2}) ⋯  (46)

Like

p(x_{1}) = p(x_{1}|a_{1}b_{1})p(a_{1})p(a_{2}) + ⋯

For example, in our example of salmon, assume we know all

\begin{aligned}& female & male \\ winter & 0.9 & 0.1 \\ spring & 0.3 & 0.7 \\ summer & 0.4 & 0.6 \\ autumn & 0.8 & 0.2 \end{aligned}  (47)

Example 3

(1) A new salmon was caught in winter, at the lake. The size is long and very thin. Then, what is the probability that it is a female salmon?

(2) A new salmon we have is short, thick, and cought in a river. Then what is the most likely season when it was caught?

(3) We have a long, thin salmon caught in winter. What is the probability that it is from sea.

(4) It is caught between autumn and winter from a lake, and we have no information as for the length of the fish, but this is medium thickness. Is the salmon male?

(5) Short and thick which season you guess that the salmon was caught?

(6) Long, thick from a lake what season is mostlikely?

8 Hidden Markov Model

8.1 Markov Model

We now assume that pattern is a sequence of states x(t), something like

x(1), x(2), x(3), ⋯, x(T)

Here we assume time is descrete like 1, 2, 3, ⋯, T. Let’s denote a sequece of states whose length is T as x^{T} Imagine

x^{7} = x_{1}, x_{6}, x_{3}, x_{3}, x_{1}, x_{2}, x_{4}

An example of states is a whether. Think of x_{1} is fine whether, x_{2} is cloudy day, x_{3} is a rainy day, and so on. Assume we know the transition probability from any state to any state, like

P (x_{j}(t + 1)|x_{i}(t)) = a_{ij}.  (48)

We call a full set of these a_{ij} model. When we denote this full set of a_{ij} as θ, the probability of above example of sequence x^{7} under the model θ is expressed as

P (x^{7}|θ) = a_{16} · a_{63} · a_{33} · a_{31} · a_{12} · a_{24}.  (49)

Our goal here is to find a model which best explains training patterns of a known class, and later, a test pattern is classified by the model that has the highest posterior probability P (ω|x).

8.2 Hidden Markov Model

Assume now the state x(t) is not observable, but emits visible symbol y(t). For example,

y_{5}, y_{3}, y_{2}, y_{2}, y_{7}, y_{2}, y_{3}  (50)

If we know

P (y_{k}(t)|x_{j}(t)) = b_{jk}  (51)

Where ∑_{j} a_{ij} = 1 for all i and ∑_{j} b_{jk} = 1 for all j. This is called a Hidden Markov Model because a sequence x(t) is hidden to observers.

Then we have three different problems.

• Evaluation Problem

– Evaluation of probabilitiy of a particular sequence of visible state.

• Decoding Problem

– Assuming we know all of a_{ij} and b_{jk}, determinatin of the most likely sequence of hidden states for a given observation of visible sequence.

• Learning Problem

– Estimation of a_{ij} and b_{jk} from a given set of training obserbations of visible sequences.

9 Classification by Fuzzy Logic

9.1 Basics

Fuzzy logic is an extention of classical set theory. While in the classical set theory an element is either belong to a set or not, in fuzzy logic we use a concept of degree of belongingness with degree of membership value ranging from 0 to 1. For example, how we are sure that x is A is expressed with a function called membership function μ_{A}(x).

9.1.1 Examples of membership function

Do you like cold beer? However “cold” is not a very clear idea. Is 10^{∘}C is cald? What about −5^{∘}C?

In Fuzzy logic, the idea of “Beer is cald,” is expressed with a membership function like

Figure 11: A membership function which expresses Beer is cold.

Figure 12: Very cold, cold, not-so cold, worm

Yet another form of membership function is Cauchy Bell-like one which is given by

μ(x) = \frac{1}{1 + (x − 12)^{2}}  (52)

as shown in the figure below.

For example this could be a membership function for

All real number close to 12.

The other option will be, of course Gaussian membership function.

Figure 13: Trapezoid on corner, trapezoid, triangle singleton, traiangle on corner

Figure 14: Yet another membership function – Cauchy’s belllike function.

9.2 Basic Arethmetic

AuB AcupB Abar

9.2.1 AND & OR

When two membership function have a same domain (which is sometimes called universe of dicourse) An example (comparison with traditional set theory) close to 5 and close to 7 close to 5 or close to 7 not close to 5 1) A ∪ B 2) A ∩ B 3)

Figure 15:

9.3 when x and y are defiened on different domain

young and tall given membership of each 15-35, 170-180 e.g.

how we represent 1) 3-d 2) gfray scale 3) descretize -¿ matrix 175 180 185 190 15 20 25 30 35 Figure 16: Figure 17:

Figure 18: Yet another membership function – Cauchy’s belllike function.

Figure 19:

Standard definition

μ_{AandB} = min(μ_{A}, μ_{B})  (53)

μ_{AorB} = max(μ_{A}, μ_{B})  (54)

A and B  A or B
Yet another standard  μ_{A} · μ_{B}  μ_{A} + μ_{B} − μ_{A} · μ_{B}
Lukasievicz’s  max(0, μ_{A} + μ_{B} − 1)  min(1, μ_{A} + μ_{B})
Einstein’s  μ_{A} · μ_{B}/(2 − (μ_{A} + μ_{B}) + μ_{A} · μ_{B}) (μ_{A} + μ_{B})/(1 + μ_{A} · μ_{B})

9.4 Fuzzy expression of IF-THEN rule

It’s important to express IF-THEN relation in Fuzzy logic, because one of the goals of fuzzy logic is , in short, to obtain y = f(x) from a set of fuzzy rule. Here we see two such expression.

Mamdani’s definition

μ_{A⇒B} = min(μ_{A}, μ_{B})  (55)

Larsen’s definition

μ_{A⇒B} = μ_{A} · μ_{B}  (56)

9.5 Let’s visualize IF-THEN rules

One precondition one rule – simplest case

Let’s see an example

IF x = low THEN y = weak.

Just imagine here, for example, x is a speed of the care I now am driving, and y is strentht of break. We now assume membership function of as follows.

Figure 20:

Excersize 7 By using Mamdani’s expression

μRule(x1, x2) = min(μLow(x1), μWeak(x2))

Express the membersip fuction μRule(x1, x2).

9.5.1 When precondition is multiple

Let’s take an example

IF x_{1} = low and x_{2} = long THEN y = midium.

In this example, imagine, for example, x_{1} is a speed of the care I now am driving, x_{2} is a distance between my car and the car in front, and y is strength of break as before. Now μ_{R}(x1, x2) = min(μ_{low}(x1), mu_{long}(x2), mu_{medium}(y)) would be defiend on 3-D space (x1, x2, y) Hence we connot observe the shape of the function. Only when we give x1, and x2 a specific value μ(x1, x2, y) vs y canbe plotted on 1-D axis of y.

When we have two rules

Example will be

Rule 1: IF x = low THEN y = weak.
Rule 2: IF x = midium THEN y = midium.

Again, all what we can do is only to plot μ by giving specific value of x Then μ_{R}1(y), and μ_{R}2(y) can be seen as before. By interpreting we have to follow

Rule 1 or Rule 2

and using Mamdani’ definition the membership function is

μRule-1 or Rule-2 = max(μRule-1(x_{1}), μRule-2(x_{2}))

Now we plott μ(y) on y axis.

But why this makes us happy?

9.6 Fuzzy Classification

9.6.1 With a set of fixed membership functions Assume our rules are like

IF x_{1} = (S_{1} ∪ S_{5}) ∩ x_{2} = (S_{3}) ∩ ⋯ x_{4} = (S_{2} ∪ S_{4})

THEN class is ω_{j}.

We now think of evolving these rules with chromosom being like ((10010)(11000)(00001)(00110))

9.6.2 Takagi Sugeno Model Assume we have H rules. The k-th rule out of these H is like

R_{k} : IF (x_{1} is A_{1}^{k}) AND (x_{2} is A_{2}^{k}) AND ⋯ AND (x_{n} is A_{n}^{k})

THEN y_{1} is ν_{1}^{k}, ⋯, y_{m} is ν_{m}^{k}.

where A_{i}^{k} is fuzzy linguistic expression of i-th attribute like medium slow and the membership function is

μ_{i}^{k}(x_{i}) = exp\{−\frac{(x_{i} − w_{i}^{k})^{2}}{σ_{i}^{k}} \}.  (57)

Then if a new sample x^{0} is given, y_{j}^{0} is known to be given as

y_{j}^{0} = \frac{∑_{k=1}^{H} μ_{k}(x^{0}) · ν_{j}^{k}}{∑_{k=1}^{H} μ_{k}(x^{0})}  (58)

where

μ_{k}(x^{0}) = ∏_{i=1}^{n} μ_{i}^{k}(x_{i}^{0}).  (59)

APPENDIX

I. Bayes formula in another scenario

· Three prisoners (A, B, and C) are in a prison.

· A knows that the two out of the three are to be executed tomorrow, and the rest becomes free. · A thought either one of B or C is sure to be executed. · Then, A asked a guard “even if you tell me which of B and C is executed, that will not give me any information as for me. So please tell it to me.” · The guard answers that C will. ⇒ data D · Now, A knows one of A or B is sure to be free.

Do you guess probability p(A|D) = 1/2? If this is correct, then the answer of the guard had given an information as for A, since probability p(A) = 1/3. You agree that

p(A) = p(B) = p(C) = 1/3.

Then, try to apply Bayesian rule, i.e., obtain the conditional probability of the data “C will be executed” under the condition that “A will be free tomorrow” And in the same way for B and C. They are:

p(D|A) = 1/2.
p(D|B) = 1.
p(D|C) = 0.

In conclusion:

p(A|D) = \frac{p(D|A)p(A)}{p(D|A)p(A) + p(D|B)p(B) + p(D|C)p(C)} = 1/3.

This shows probability did not change after the information!

• Now it’s clear that

– p(A) and so on are to be called ⋆ a priori probabilities;

– and p(A|D) and so on to be ⋆ a posteriori probabilities.

• In the same way,

– p(ω_{i}) is called ⋆ a priori probability^{3};

– p(ω_{i}|x)

⋆ a posteriori probabilities.

Furthermore

– p(x) is called ⋆ p.d.f. of x

– p(x|ω_{i})

⋆ class-conditional p.d.f.^{4} · which describes the distribution of the feature vectors x in each of the classes ω_{i}.

3Usually given, but if unknown, it can be estimated as N_{i}/N where N_{i} is the number of training samples which belong to class ω_{i}, and N is the total number of training samples

4This is also estimated from training data which will be explained later more in detail.

II. Quadratic form in 2-dimensional space

You might be interested, first of all, in how points scattered are influenced by values in Σ, that is, σ_{1}^{2}, σ_{2}^{2}, and σ_{12} = σ_{21}. Let’s observe here three different cases of Σ when μ = (0, 0).

(1) Σ_{1} = ( \begin{matrix}0.20 & 0 \\ 0 & 0.20 \end{matrix} ) (2) Σ_{2} = ( \begin{matrix}0.1 & 0 \\ 0 & 0.9 \end{matrix} ) (3) Σ_{3} = ( \begin{matrix}0.5 & 0.3 \\ 0.3 & 0.2 \end{matrix} )

Figure 21: A cloud of 200 points each with different three different N[μ, Σ].

(Pattern Recognition) 36

III. How to calculate inverse of 3-dimensional matrix.

We now try to calculate the invers of the following 3-D matrix A which appeared in the subsection 3.3.

A = ( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0.1 & 0.3 & −0.1 \\ 0.1 & −0.1 & 0.3 \end{matrix} )

We use a relation Ax = I where x = (x, y, z)^{T} and I is identity matrix, i.e.,

( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0.1 & 0.3 & −0.1 \\ 0.1 & −0.1 & 0.3 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} )

It remains identical if we multiply {2nd-raw} by 3 and subtract the {1st-raw}, i.e.,

( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0 & 0.8 & −0.4 \\ 0.1 & −0.1 & 0.3 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}1 & 0 & 0 \\ −1 & 3 & 0 \\ 0 & 0 & 1 \end{matrix} )

In the same way, but this time, we multiply {3rd-raw} by 3 and subtract the {1st-raw}.

( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0 & 0.8 & −0.4 \\ 0 & −0.4 & 0.8 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}1 & 0 & 0 \\ −1 & 3 & 0 \\ −1 & 0 & 3 \end{matrix} )

Then, e.g., multiply the {1st-raw} by 8 and then subtract the {2nd-raw}:

( \begin{matrix}2.4 & 0 & 1.2 \\ 0 & 0.8 & −0.4 \\ 0 & −0.4 & 0.8 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}9 & −3 & 0 \\ −1 & 3 & 0 \\ 0 & 0 & 3 \end{matrix} )

Multiply the {3rd-raw} by 2 and then add the {2nd-raw}:

( \begin{matrix}2.4 & 0 & 1.2 \\ 0 & 0.8 & −0.4 \\ 0 & 0 & 1.2 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}9 & −3 & 0 \\ −1 & 3 & 0 \\ −1 & 3 & 6 \end{matrix} )

Subtract {3rd-raw} from the {1st-raw}:

( \begin{matrix}2.4 & 0 & 0 \\ 0 & 0.8 & −0.4 \\ 0 & 0 & 1.2 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}10 & −6 & −6 \\ −1 & 3 & 0 \\ −1 & 3 & 6 \end{matrix} )

Multiply the {2nd-raw} by 3 then add the {3rd-raw}:

( \begin{matrix}2.4 & 0 & 0 \\ 0 & 2.4 & 0 \\ 0 & 0 & 1.2 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}10 & −6 & −6 \\ −4 & 12 & 6 \\ −1 & 3 & 6 \end{matrix} )

Finally, devide the {1st-raw} by 2.4, devide the {2nd-raw} by 2.4, and devide the {3rdraw} by 1.2, we obtain,

( \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ) ( \begin{matrix}x \\ y \\ z \end{matrix} ) = \frac{1}{30} ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} )

Now we know the right-hand-side is the invers of A because the equation implies Ix = B and it holds AIx = AB, that is, Ax = AB. Hence AB = I which means B = A^{−1}.

To make it sure, calculate and find

( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0.1 & 0.3 & −0.1 \\ 0.1 & −0.1 & 0.3 \end{matrix} ) × \frac{1}{30} ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} )

= \frac{1}{6} ( \begin{matrix}25 & −15 & −15 \\ −10 & 30 & 15 \\ −5 & 15 & 30 \end{matrix} ) × ( \begin{matrix}0.3 & 0.1 & 0.1 \\ 0.1 & 0.3 & −0.1 \\ 0.1 & −0.1 & 0.3 \end{matrix} ) = ( \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} )

Therefore

A^{−1} = \frac{1}{30} ( \begin{matrix}5 & −3 & −3 \\ −2 & 6 & 3 \\ −1 & 3 & 6 \end{matrix} )