Attrasoft

All Attrasoft products are Y2K Compliant!

5. Examples 
5.1 Character Recognition
5.1.1 Character Generation 
5.1.2 19x19 Character Recognition (1,023 Classes) 
5.1.3 32 by 32 Character Recognition
5.1.4 19x19 Character Recognition (4,095 classes) 
5.1.5 50x50 and 100x100 Image Recognition 
5.2 Shifter 
5.2.1 Shifter Problem 
5.2.2 Single Shifter 
5.2.3 Double Shifter 
5.3 Nursery School Database
5.4 Mathematical Probabilities and Functions 

5.   Examples

5.1   Character Recognition

There are five character recognition problems:

5 by 7 character recognition
8 by 8 character recognition
19 by 19 character recognition, 1023 classes
19 by 19 character recognition, 4095 classes
32 by 32 character recognition

Figure 9. Example Menu.

The parameters are:

Projects          5x7       8x8       19x19(1023)       32x32
# Neurons       46        74         1387                   1152
# Class            11        10         1023                   127
# Training        46        50         1023                   127

Below, we will discuss the 19x19 project with 1,023 and 4,095 classes. There is a problem here, not for PolyNet, but for the users: nobody can learn and memorize 1,000 characters in a few seconds. This turns out not to be a problem; we can introduce a mathematical way to generate the characters. By looking at a character, you can calculate its classification instead of memorize them.

The command "Decimal/1N1C Classification" is fast than the command "Decimal/Classification". Please use "Decimal/1N1C Classification" for this section.

5.1.1   Character Generation

We get a hint of generating these characters from a digital watch: all digits can be generated from the following frame:

11111111 
1            1
1            1
1            1
1            1
1            1
1            1
1            1
1            1
11111111 
1            1
1            1
1            1
1            1
11111111 

The above pattern is an '8'. The above frame will generate 127 patterns. Some of them are not distinguishable; for example, you can find six '1' in the above frame ( Left 1, Right 1, Left up 1, Left bottom 1, Right up 1, and Right bottom 1). There are seven line segments, called A, B, C, D, E, F, G:

      A 
B             D 
      C 
E             F 
      G 

Each character can be represented by a seven-bit string: gfedcba. For example, 

0000001 means that segment A is present, the rest are not there;
0000011 means segments A and B are present; 
. . . ;
1111111 means all segments are there, which is an '8'. 

These characters are ordered by binary numbers from 0000001 to 1111111, given 127 classes.

To generate more characters, a larger frame has to be used. We will choose the following frame: 

11111111 11111111
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
11111111 11111111
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
11111111 11111111

There are 12 line segments, giving more than 4000 patterns. 

5.1.2   19x19 Character Recognition (1,023 Classes)

The 1,023 characters are generated by the following basic lattice:

11111111 11111111
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
11111111 11111111
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
1              1             1
11111111 11111111

There are 12 line segments in the lattice. Depending on whether these line segments are there or not, we can generate about 4,000 characters. The line segments are labeled by A, B, C, ... L as follows:

      A           H
B           D           I
      C           J
E           F           K
     G            L

Each character can be represented by a 12-bit string: lkjihgfedcba. For example,

00000 00000 01 means segment A is present; the rest are not there;
00000 00000 11 means segments A and B are present;
. . . ;
11111 11111 11 means all segments are there. 

These characters are ordered by binary numbers from 00000 00000 01 to 1111 11111 11, giving 4,095 classes.

In the following, the line-segments K and L will not be used. These line-segments are basically straight lines with 8 bits. In the above frame, segments A, C, G, . . . look alike. To distinguish the line segments, we will add an extra bit to these lines. The definitions of these line segments are:

A: 11111111
                   1
---------------------------------------------

C:               1
     11111111
--------------------------------------------

G:       1
     11111111

--------------------------------------------

H: 11111111
     1
--------------------------------------------

J: 1
    11111111

--------------------------------------------

B:      11           D          11           E:      1
         1                          1                      1
         1                          1                      1
         1                          1                      1
         1                          1                      1
         1                          1                      1
         1                          1                      1
         1                          1                      11
....

The definition of the rest of the line-segments can be found in the training file. The training, and recognition files are generated by clicking: "Example/19x19 character". This command will take a few seconds. 

The class assignment for each character depends on the value of the binary string. For example, 

00000 00000 01 is class 1;
00000 00000 11 is class 3;
... ...

The formula to calculate class is:

class = A + B*2 + C*4 + D*8 + E* 16 + F*32 + G* 64 + H* 128 + I*256 + J * 512 

The above formula means:

If line segment A is in the character, add 1, else add 0; 
If line segment B is in the character, add 2, else add 0;
... ...

Example Several artificial characters are given below:

Class 1: 11111111
                           1
---------------------------------------------

Class 4               1
            11111111

--------------------------------------------

Class 128:                1
                  11111111

--------------------------------------------

Class 256 11111111
                 1

--------------------------------------------

Class 3 (Two segments, A and B, form class 3):

11111111
11 
1 
1 
1 
1 
1 
1 
1 

--------------------------------------------

Class 7:

11111111
11 
1 
1 
1 
1 
1 
1 
1 1
11111111

The training file contains all 1,023 classes. The neural net is organized accordingly: there are 1,026 class-neurons and 19 * 19 = 361 pattern-neurons, giving a total of 1,026 + 361 = 1,387 neurons. The reason to choose 1,026 neurons instead of 1,023 is that the output-vector will be printed 19-digits per line and 1,387 neurons fit into 73 lines. 

The training file has the 1,023 classes. The training time is about 5 seconds (400 MHz Pentium). PolyNet will randomly generate a few characters with some noise and ask the net to recognize it. The 1N1C recognition time is 1- 1.5 second per character (400 MHz Pentium). The recognition rate is 100%. You can regenerate other recognition files by clicking "Examples/19x19 Character recognition" again.

Note, a neural network can learn 1,000 characters in 5 seconds, faster than the human brain. You can argue these characters are special, but we choose them because it is easy for humans. It does not make any difference to PolyNet as to what set of characters are used. Try it and you will convince yourself.

5.1.3   32 by 32 Character Recognition

There are 127 classes:

class = A + B*2 + C*4 + D*8 + E*16 + F*32 + G*64.

There are 127 classes. The neural net is organized accordingly: there are 128 class-neurons and 32 * 32 = 1,024 pattern-vector, giving a total of 1,024 + 128 = 1,152 neurons. The reason to choose 128 class-neurons instead of 127 is that the output-vector will be printed 32-digits per line and 1,152 neurons fit into 36 lines. 

The training file has the 127 classes. PolyNet will randomly generate a few characters and asks the net to recognize it. The recognition rate is 100%. You can regenerate other recognition files by clicking "Examples/32x32 Character recognition" again.

5.1.4   19x19 Character Recognition (4,095 classes)

This is the same as problem as the earlier 19x19 problem, however, the number of classes is increased to 4,095.

Note here, a neural network can learn 4,000 characters in 20 seconds, faster than the human brains. You can argue these characters are special, but we choose them because it is easy for humans. It does not make any difference to PolyNet as to what set of characters are used.

5.1.5   50x50 and 100x100 Image Recognition

Attrasoft provides many software for jpg/gif image recognition. They are:

• Internet ImageFinder 3.3
• ImageFinder for Windows 95/98. 3.4
• ImageClassifier, 3.1
• ImageHunt, 3.5 (Internet Search Engine)

To run the 50x50 example, click "Example/50 by 50 Binary Image Recognition". The data files are img50x50.trn and img50x50.rec. Images of a face, a rabbit, a boat, . . . are used to train the neural net. Users can add their own data to the end of the data files. Make sure you check the data file format before running. To check the format, click "Data/Test". 

To run the 100x100 example, click "Example/100 by 100 Binary Image Recognition". The data files are img100x100.trn and img100x100.rec. Users can add their own data to the end of the data files. Make sure you check the data file format before running. To check the format, click "Data/Test". 

5.2   Shifter

5.2.1   Shifter Problem

All you do to solve a shifter-problem is to click a few buttons; however, first we will explain the problem. 

The shifter problem generates a pattern, shifts the pattern in one of many possible ways and asks a neural network to identify the shifting. 

This is obviously synthetic data instead of real data, such as a person's signature. The shifter problem is chosen because:

(1) the number of neurons is arbitrary, allowing you to work with many different sizes of neural networks; 
(2) the amount of data is arbitrary, allowing you to study the relation between the amount of training data and the results;
(3) this is a very hard problem due to the large sample space. 

Example 4-digit Shifter problem.

Four possible movements are listed as follows:

Class:           Pattern:      Comment
100           1100 1001      Left Shift, Wrap-around
001           1100 0110      Right Shift
010           1100 1100      No Shift
020           2200 2200      No Shift for 2's
003           3300 0330      Right Shift for 3's
111           1111 0000      Not Valid

This shifter has 27 classes: 100, 200, 300, ... 010, 020, 030, ..., 001, 002, ..., 009.

When you shift a pattern once, it is a single shifter; when you shift a pattern twice, it is a double shifter. Shifter data is used to train the neural network. Once training is finished, the network has to solve two types of problems: identify a shift or complete a shift.

5.2.2   Single Shifter

To generate training data from PolyNet, click: 

Example/Shifter/Data.

This command will also open the training and recognition files and set up the neural network. The default values (for the shifter problem) are: 

15-digit Shifter (33 neurons), 
500 training-patterns, 
50 recognition-patterns.

Example/Shifter/Parameter.

* 
shifter, Training Data File 
*

33

010    000001000000010
          000001000000010 

700    777707707070700
          777077070707007
... ...

In the first training pattern, 

010                               means no shift; 
000001000000010       is a 15-digit pattern; and
000001000000010       is generated by 'no shift'.

Now that the network is initialized and the data is generated, it is ready to be trained and tested. Click 'C'-Button (Classification). This command takes no time.

If you click 'D'-button (Distribution), the Boltzmann Machine will present you with a distribution, which is a set of possibilities and the probability for each possibility. The one with the largest number is the preferred answer. The following is an example of the output for a shifter problem: 

xxx
          001101000101111
          100110100010111
Distribution begins: 
100 2
001 50
End of the Distribution 
001
          001101000101111
          100110100010111 50 1

The recognition pattern is printed first. This is followed by all possible solutions and a relative probability for each possibility. Finally, 001 ( Right Shift ) is the answer and it will be printed again with the original input data.

The probabilities printed next to each possibility are relative probabilities. In the first example, the probabilities for 100, 010, and 001 are 2/52, 0, and 50/52, respectively. The probabilities are not normalized because they carry other information: the larger the number is, the more reliable the answer is.

In this example, 40 or so recognition-patterns are recognized. If you are satisfied, copy the unrecognized patterns to the training file and run again (with xxx being replaced properly). Then you should get 100% correct classification. Alternatively, generate more training data by clicking:

Example/Shifter/Parameter,

Click commands "Decimal/Classification" or "Decimal/Distribution", you can expect three possibilities for each recognition pattern:

Correct classification: this is usually the case when the Boltzmann Machine assigns one predominate probability to one of the output possibilities. 

Incorrect classification: this is usually the case when: the Boltzmann Machine assigns a predominate probability to more than one configuration; or the relative probability is "small". It is hard to define "small" here because it depends on the problem, but for a given problem, an experienced user will know what is "small".

No classification: this is usually the case when the Boltzmann Machine can find little correlation in the test pattern, based on the current training. In such a case, "No classification " is printed in the output data file.

PolyNet can generate the shifter problem for you at various sizes. Try the shifter problem with different numbers of neurons to get a feeling for it. You need this feeling to determine how many training-patterns are required for a given problem. 

5.2.3   Double Shifter

If you shift a pattern twice, you will get a double shifter. Here are two examples:

010001
          100100001000000
          100100001000000
          010010000100000

The possible shifts are:

LL      100100
NL      010100 
RL      001100 
LN      100010 
NN      010010 
RN      001010 
LR      100001 
NR      010001 
RR      001001 
 

Where RL means Right-Left shifts. A recognition pattern looks like this:

xxxxxx
          100000000001011
          000000000010111
          000000000010111

The output data file for "Decimal/Distribution" looks like this:

xxxxxx
          100000000001011
          000000000010111
          000000000010111
Distribution begins:
100100      260
010100      128
010010      32
010001      32
001100      80
001001      32
End of the Distribution 
100100
100000000001011
000000000010111
000000000010111 260 1

The output data file contains the original vector, plus several possible shifts. Finally, the answer is printed. Each possibility is associated with a number: 

for class 100100, the number is 260; 
for class 010100, the number is 128; 
for class 010010, the number is 32;
. . . 

The predominate number here is 260, making the preferred answer 100100 (the left-left shift), which is the correct answer. These numbers are relative probabilities. Note, the probabilities printed next to each possibility are relative probabilities. The probabilities are not normalized because they carry other information: the larger the number is, the more reliable the answer is.

5.3   Nursery School Database

1. Sources:

M. Olave, V. Rajkovic, M. Bohanec: An application for admission in public school systems. In (I. Th. M. Snellen and W. B. H. J. van de Donk and J.-P. Baquiast, editors) Expert Systems in Public Administration, pages 145-160. Elsevier Science Publishers (North Holland), 1989.

2. Information

Number of Instances: 12960 (instances completely cover the attribute space)

• Number of Attributes: 8 + 1
• Class Distribution (number of instances per class)

3. Attribute Values:

parents             usual(0), pretentious(1), great_pret(2)
has_nurs           proper(0), less_proper(1), improper(2), 
                                critical(3),very_crit(4)
form                 complete(0), completed(1), incomplete(2), foster(3)
children            1, 2, 3, more(5)
housing             convenient(0), less_conv(1), critical(2)
finance              convenient(0), inconv(1)
social                 non-prob(0), slightly_prob(1), problematic(2)
health                 recommended(0), priority(1), not_recom(2)
Class 

The last 20 rows will be used for the recognition file. Below are the first 10 instances:

0 0 0 1 0 0 0 0 4
0 0 0 1 0 0 0 1 2
0 0 0 1 0 0 0 2 5
0 0 0 1 0 0 1 0 4
0 0 0 1 0 0 1 1 2
0 0 0 1 0 0 1 2 5
0 0 0 1 0 0 2 0 2
0 0 0 1 0 0 2 1 2
0 0 0 1 0 0 2 2 5
0 0 0 1 0 1 0 0 3

Below are the last 20 rows:

2 4 3 5 1 1 2 1 x
2 4 3 5 1 1 2 2 x
2 4 3 5 2 0 0 0 x
2 4 3 5 2 0 0 1 x
2 4 3 5 2 0 0 2 x
2 4 3 5 2 0 1 0 x
2 4 3 5 2 0 1 1 x
2 4 3 5 2 0 1 2 x
2 4 3 5 2 0 2 0 x
2 4 3 5 2 0 2 1 x
2 4 3 5 2 0 2 2 x
2 4 3 5 2 1 0 0 x
2 4 3 5 2 1 0 1 x
2 4 3 5 2 1 0 2 x
2 4 3 5 2 1 1 0 x
2 4 3 5 2 1 1 1 x
2 4 3 5 2 1 1 2 x
2 4 3 5 2 1 2 0 x
2 4 3 5 2 1 2 1 x
2 4 3 5 2 1 2 2 x

The correct answers are 

1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5.

5. Results

Click 'C'-button and get:

================= Beginning ==================

24351121x
243511211 6470 1

24351122x
243511225 5850 1

24352000x
243520002 971 1

... ...

=================== End ==========================

6. Analysis

The correct answers are 

1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5.

The predicted answers are:

1, 5, 2, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5, 1, 1, 5.

95% predictions are correct, the other 5% are very close (1 was identified as 2).
 

5.4   Mathematical Probabilities and Functions

All files used in this section are in your software package. Let a mapping be defined by:
 

y1 = ( x1 + x2 + x3 ) Mod 7; 

y2 = x1; 

y3 = x2 Mod 5; 60%
y3 = ( x2 + 1 ) Mod 5; 40% 

y4 = ( x4 + 1 ) Mod 5; 60% 
y4 = ( x4 + 2 ) Mod 5; 40% 

y5 = x4 Mod 3.

Note that by design, each input, (x1, x2, x3, x4, x5), can have 4 different answers. All four answers are correct and each answer has certain probability to appear. The parameters of this problem are:

9999 rows can generated in this way: 

1 0 0 0 0                 1 1 0 1 0
1 1 0 1 0                 2 1 2 3 1
2 1 2 3 1                 5 2 1 0 0
5 2 1 0 0                 1 5 3 2 0
1 5 3 2 0                 2 1 0 4 2
2 1 0 4 2                 3 2 2 1 1
3 2 2 1 1                 0 3 3 2 1
... 

The 9999 rows of data are divided into two parts: the first 9994 rows will be used for training; and the last five rows will be used for testing. The last 5 rows are:

Input Output

6 6 1 3 1                 6 6 1 4 0
6 6 1 4 0                 6 6 1 0 1
6 6 1 0 1                 6 6 1 1 0
6 6 1 1 0                 6 6 1 3 1
6 6 1 3 1                 6 6 1 4 0

To run this problem:

66131xxxxx
6613166110 912 1

66140xxxxx
6614066101 1280 1

... 

Predicted                 correct Answer

6 6 1 1 0                 6 6 1 4 0
6 6 1 0 1                 6 6 1 0 1
6 6 1 1 0                 6 6 1 1 0
6 6 1 2 1                 6 6 1 3 1
6 6 1 1 0                 6 6 1 4 0

The prediction is very close to the correct answers.

There is another mathematics function problem in the PolyNet. Let a 3-variable function be defined by:

y1 = ( x1 + x2 + x3 ) Mod 7; 
y2 = x1; 
y3 = x2 Mod 5; 

We will generate the data as follows: assume initially that (x1, x2, x3) is in state (1 0 0), then it will generate (1 1 0). Now we will use (x1, x2, x3) = (1 1 0) and it will generate (2 1 1), ... ...

The parameters of this problem are:

• 6 variables (3 input: x1, x2, x3; and 3 output: y1, y2, y3);
• 69 instances;
• 245 classes 

Online Order or if you prefer, Online Fax order: 
PolyNet for Windows 95/98 
Version 4.0 ($124.99 + $5 US Shipping and Handling)

Mail Order: $129.99 (S&H included)

PolyNet 4.0
Attrasoft, Attn.: Gina
P. O. Box 13051
Savannah, GA. 31406, USA

webmaster@attrasoft.com