About Attrasoft TransApplet

TransApplet is a .Net Class Library that enables System Integrators, Solution Developers, and Individuals to quickly add Image Recognition capability to their products and services.�

Software Requirements

The software requirements are:

Microsoft Visual Studio .Net, or
C#.Net, or
VB.Net

�Installing the Software

Copying the �CD:\transapplet80� to the C-driver will complete the installation.

Information & Support

Attrasoft TransApplet

Attrasoft

P. O. Box 13051

Savannah, GA, 31406

USA

http://attrasoft.com

imagefinder@attrasoft.com (Email Subject: Attrasoft)

Phone:� (912) 484-1717

� Attrasoft 1998 - 2011

License Agreement

THIS LICENSE AGREEMENT ("AGREEMENT") IS BETWEEN YOU, THE END USER, AND Attrasoft. IT GOVERNS THE USE OF THE SOFTWARE PROGRAM AND DOCUMENTATION KNOWN AS Attrasoft TransApplet (THE "PRODUCT"). IF YOU USE THE PRODUCT, THEN YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE NOT WILLING TO BE BOUND BY THE TERMS OF THIS AGREEMENT, PROMPTLY RETURN THIS PACKAGE TO THE PLACE OF PURCHASE WITH A COPY OF THE RECEIPT, AND YOUR LICENSE FEE WILL BE REFUNDED.

Attrasoft licenses use of the PRODUCT, which may be manufactured and distributed by Attrasoft or by a third party (either, the "Manufacturer"). You, the end-user, assume responsibility for the selection of the PRODUCT to achieve your intended results, and for its installation and subsequent use.

GRANT OF LICENSE

Attrasoft hereby grants you a non-exclusive license to use the PRODUCT in object code form only, upon the terms and conditions contained in this Agreement.

You may:

1. Use the PRODUCT on the number of workstations for which you have purchased PRODUCT licenses. The workstations must be owned, leased or otherwise controlled by you, whether in a network or other configuration.

2. Create a quantity of backup copies of the PRODUCT, in any machine-readable or printed form, equal to the number of PRODUCT licenses you have purchased.

3. Transfer the PRODUCT and your rights under this Agreement to another party if the other party agrees to accept the terms and conditions of this Agreement. If you transfer the PRODUCT, you must, at the same time, either transfer all copies of PRODUCT to the same party, or destroy any copies not transferred. You must immediately notify Attrasoft of the transfer.

4. Print out one copy of the Product documentation from the Attrasoft program, TransApplet, for each license purchased. If you print out any part of the Product documentation from the Attrasoft program, TransApplet, you must reproduce and include all the copyright notices that appear in the documentation on any such copy of the documentation.

You May Not:

1. Use or copy the PRODUCT, in whole or in part, except as expressly provided in this Agreement.

2. Use the PRODUCT concurrently on more than the number of workstations for which you have purchased licenses.

3. Copy, rent, distribute, sell, license or sub-license, or otherwise transfer the PRODUCT or this license, in whole or in part, to another party, except as specifically set forth above.

4. Incorporate the PRODUCT or any portion of the PRODUCT into, or use the PRODUCT, or any portion of the PRODUCT to develop, other software without a license from Attrasoft, or otherwise modify or create a derivative work from the PRODUCT without a license from Attrasoft.

5. Reverse engineer, decompile, or disassemble the PRODUCT.

To use the PRODUCT as described in Sections 2 or 4 above, or for any other use not specifically set forth above, additional licensing from Attrasoft is required. For further information, please contact Attrasoft at:

Attrasoft, Inc.��

Phone: (912) 484-1717

gina@attrasoft.com

PROPRIETARY RIGHTS

This Agreement gives you ownership only of the physical Attrasoft program, TransApplet, on which the PRODUCT is stored, but not of the PRODUCT itself. You acknowledge that Attrasoft owns all rights, title, and interest in the PRODUCT, and that you will acquire no rights in the PRODUCT through your use of it. You agree that you will take no action that interferes with Attrasoft's rights in the PRODUCT.

TERMS

This Agreement is effective until terminated. You may terminate it at any time by destroying the PRODUCT together with all copies and documentation in any form. This Agreement will also terminate automatically and without notice from Attrasoft if you fail to comply with any term or condition of this Agreement. You agree upon such termination to destroy the PRODUCT and all copies of the PRODUCT.

DISCLAIMER; LIMITED WARRANTY

EXCEPT AS PROVIDED BELOW, THE PRODUCT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PRODUCT IS WITH YOU. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. NEITHER ATTRASOFT NOR MANUFACTURER WARRANTS THAT THE FUNCTIONS CONTAINED IN THE PRODUCT WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE PRODUCT WILL BE UNINTERRUPTED OR ERROR-FREE. However, where Attrasoft is the Manufacturer, Attrasoft warrants that the Attrasoft program, TransApplet, on which the software is furnished will be free from defects in materials and workmanship under normal use for a period of ninety (90) days from the date of delivery as evidenced by a copy of your receipt.

LIMITATION OF REMEDIES

Where Attrasoft is the Manufacturer, Manufacturer�s entire liability and your exclusive remedy shall be:

1. The replacement of the Attrasoft program, TransApplet, not meeting the Limited Warranty, which is returned to Manufacturer with a copy of your receipt.

2. If Manufacturer is unable to deliver replacement Attrasoft program, TransApplet, which is free of defects in materials or workmanship, you may terminate this Agreement by returning the PRODUCT and a copy of your receipt to the place of purchase, and your money will be refunded. Where Attrasoft is not the Manufacturer, Attrasoft shall have no liability to replace or refund, and you agree to look to Manufacturer to meet the obligations described above.

LIMITATION OF LIABILITY

IN NO EVENT WILL ATTRASOFT OR MANUFACTURER BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO, ANY LOST PROFITS, LOST SAVINGS, OR OTHER INDIRECT, SPECIAL, EXEMPLARY, INCIDENTAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF THE USE OR INABILITY TO USE THIS PRODUCT, EVEN IF ATTRASOFT OR MANUFACTURER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. FURTHER, IN NO EVENT WILL ATTRASOFT OR MANUFACTURER

BE LIABLE FOR ANY CLAIM BY ANY OTHER PARTY ARISING OUT OF YOUR USE OF THE PRODUCT. SOME JURISDICTIONS DO NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE ABOVE LIMITATION OR EXCLUSION MAY NOT APPLY TO YOU.

TRADEMARKS

Attrasoft is a trademark of Attrasoft, Inc. Microsoft, C# logo are registered trademarks of Microsoft Corporation. No rights, license, or interest in such trademarks is granted hereunder.

U.S. GOVERNMENT RESTRICTED RIGHTS IN DATA

This computer software product and documentation are provided with Restricted Rights. Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013, or subparagraphs (c)(1) and (2) of the Commercial Computer Software - Restricted Rights at 48 CFR 52.227-19, as applicable. Contractor/Manufacturer is Attrasoft, Inc.

EXPORT CONTROLS

You agree not to export or re-export the PRODUCT, directly or indirectly, to any countries, end-users or for any end uses that are restricted by U.S. export laws and regulations, without first obtaining permission to do so as required by the U.S. Department of Commerce's Bureau of Industry and Security, or other appropriate government agency. These restrictions change from time to time. If you have any questions regarding your obligations under U.S. export regulations, you should contact the Bureau of Industry and Security, U.S. Department of Commerce, Exporter

Counseling Division, Washington D.C. (202) 482-4811, http://www.bis.doc.gov.

About Attrasoft TransApplet. 1

Software Requirements.. 1

Installing the Software.. 2

Information & Support. 2

License Agreement. 3

TABLE OF CONTENTS.. 6

1.�� Introduction.. 13

1.1�� What is TransApplet?. 13

1.2�� Software Requirements. 13

1.3�� Installing the TransApplet 13

1.4�� Attrasoft TransApplet Class Library. 14

1.5�� Linking the TransApplet 14

2.� Image Recognition Overview... 16

2.1�� Attrasoft Image Recognition Basics. 16

2.2�� Image Recognition Internal Structures. 16

3.� TransApplet Overview... 17

3.1�� TransApplet� API. 17

3.2�� User Interface. 17

3.3�� Input & Display. 17

3.4�� ImageFinderLite. 18

3.5�� Signature Recognition. 18

3.6�� Whole Image Matching. 18

3.7�� Dynamic Library. 18

3.8�� Image Segment Matching. 18

3.9�� Videos. 18

3.10�� Image Processing. 19

3.11�� Customized Software. 19

4.�� API 20

4.1�� Image Signature. 20

4.2�� Signature Filter. 22

4.3�� Matching Results. 25

4.4�� Whole Image Matching. 27

4.5�� ImageLibrary. 28

4.6�� Matching Engine. 30

4.7�� NeuralNet Filter. 32

4.8�� Other API. 33

5.�� User Interface.. 34

5.1�� How to Create a Project 34

5.2�� How to Create Menus. 36

5.3�� Link to Class Library. 37

5.4�� Declare Objects. 38

6.�� Input. 39

6.1�� Class Library Name. 39

6.2�� Class Library Overview.. 40

6.3�� Set Up the Chapter Project 41

6.4�� Link to Class Library. 42

6.5�� Key Segment 43

6.6�� Search Source Button. 45

7.�� Image Display.. 47

7.1�� Class Library Name. 47

7.2�� Implementing H, >, and < Buttons. 48

7.3�� Test 50

7.4�� Output� Images. 50

8.�� ImageFinderLite.. 51

8.1�� ImageFinderLite. 51

8.2�� Interface. 51

8.3�� Programming Example. 53

8.4�� Codes for Each of the Steps. 55

8.5�� Test Run. 56

9.�� Image Signatures.. 57

9.1�� Signature Menu. 57

9.2�� API. 58

9.3�� TransApplet Objects. 61

9.4�� Key Signature. 62

9.5�� Signature File Concepts. 64

9.6�� Signature File Implementation. 65

9.7�� Examples. 65

10.�� Whole Image Matching.. 67

10.1�� Whole Image Matching Menu. 67

10.2�� NeuralFilter API. 67

10.3�� N:N Matching Implementation. 68

10.4�� 1:N� Matching Implementation. 69

11.�� Binary Search.. 72

11.1�� Signatures. 73

11.2�� Search. 76

12.�� Dynamic Library.. 80

12.1�� Dynamic Library Menu. 80

12.2�� Dynamic Library API. 81

12.3�� Creating A Master Library. 82

12.4�� Load Dynamic Library. 82

12.5�� Library M:N Matching. 83

12.6�� Library 1:N Matching. 85

12.7�� Update Implementation. 86

13.�� Image Variations.. 88

13.1�� Variation Signatures. 88

13.2�� Segments. 90

14.�� Sub-Image Search.. 93

14.1�� Sub-Images and Variations. 93

14.2�� Search Process. 96

14.3�� Learn by Example. 96

14.4�� Implementation. 100

15.�� Fast Sub-Image Matching.. 102

15.1�� Fast Menu. 102

15.2�� NeuralNet Filter API. 105

16.�� Video Input. 106

16.1�� Class Library Name. 106

16.2�� Class Library Overview.. 107

16.3�� Set Up the Chapter Project 107

16.5�� Key Segment 108

16.6�� Search Source Button. 109

17.�� Whole Video Recognition.. 110

17.1�� Project� Introduction. 110

17.2�� Example. 113

17.3�� Implementation. 114

18.�� Video Frame Recognition.. 116

18.1�� How to Use VideoFinder. 116

18.2�� Implementation. 119

19.�� Video Object Recognition.. 121

19.1�� FrameFinder Functions. 121

19.2�� Beginner Menu. 121

19.3�� User Interface. 122

19.4�� Implementation. 124

20.�� Image Processing Concept. 125

20.1�� Filters. 125

20.2�� Image Preprocessing & Processing. 125

20.3�� Normalization. 126

20.4�� Neural Filter and Neural Net Filter. 127

20.5�� Parameter Object 127

21.�� Image Preprocessing.. 128

21.1�� Image Preprocessing Filter. 129

21.2�� PreProcessing API. 129

21.3�� Enter Parameters. 131

21.4�� Cut Off the Border Areas. 132

21.5�� Impose a Mask. 132

21.6�� Speed Up the Computation. 133

21.7�� Skip the Empty Border by Content Percent 134

22.�� Image Processing.. 136

22.1�� Good & Bad. 136

22.2�� Processing API. 138

22.3�� Set Image Processing Filters. 140

22.4�� First Two Settings. 141

22.5�� Chapter Projects. 142

23.�� Normalization.. 143

23.1�� Class Library Name. 143

23.2�� Class Library Overview.. 144

23.3�� Link to Class Library. 145

23.4�� Parameters. 146

24.�� Parameter Class.. 148

24.1�� Pushing Images Through Filters. 148

24.2�� Predefined Objects. 149

24.3�� Grouping Parameters Together. 150

24.4�� Chapter Project 154

24.5�� TransApplet Objects. 158

24.6�� Selecting Filters. 159

24.7�� Set Filter Parameters. 160

25.�� Parameters.. 163

25.1�� Overview.. 163

25.2�� Image Preprocessing. 163

25.3�� Image Processing. 165

25.3.1�� Edge Filters. 165

25.3.2�� Threshold Filters. 168

25.3.3�� Clean-Up Filters. 169

25.4�� Normalization Filter. 169

25.5�� Unsupervised Filterand BioFilter. 171

25.6�� Neural Filters. 173

25.7�� NeuralNet Filter. 174

25.7.1�� Symmetry. 175

25.7.2�� Translation Type. 176

25.7.3�� Scaling Type. 176

25.7.4�� Rotation Type. 176

25.7.5�� Area of Interest (AOI) 177

25.7.6�� Blurring. 177

25.7.7�� Sensitivity. 178

25.7.8�� Internal/External Weight Cut 178

25.7.9�� Segment Size. 179

25.7.10�� Image Type. 179

25.7.11�� Use BioFilter & Use Neural Filter 180

25.7.12�� Auto Segment 180

25.7.13�� Summary. 180

26. ImageFinder Support Service Packages.. 182

26.1�� Basic Annual Maintenance & Support Package. 182

26.2�� Process. 183

26.3�� What is a Feasibility Study?. 183

26.4�� Customization� Services. 184

� �

1.�� Introduction

Attrasoft TransApplet for Windows is a .Net Class Library that enables the addition of Image Recognition capability to products & services.� It can be used for:

Image Verification (1:1 Matching);
Image Identification (1:N Matching);
Image Search or Retrieval (1:N Matching);
Multiple Verification or Identification (N:N and N:M Matching); and
Video Matching

1.1�� What is TransApplet?

Attrasoft TransApplet for Windows is a .Net Class Library that enables the addition of Image Recognition capability to products & services.�

Before you continue, you should be familiar with the Attrasoft ImageFinder for two reasons:

Learn the structure of the Attrasoft Image Recognition Approach;
Learn the ImageFinder software.

The TransApplet will introduce the class library, �Attrasoft.TransApplet80�. This library is used in the ImageFinder. As a teaching tool, the TransApplet will show you how to build an ImageFinder via C# .Net. Therefore, if you have not learned the ImageFinder, you should start with the ImageFinder now.

The entire ImageFinder C# project is in the software. It is located in the ImageFinder subfolder. You can run the ImageFinder from there.

1.2�� Software Requirements

The software requirements are:

Microsoft Visual Studio .Net, or
C#.Net, or
VB.Net

In addition, we assume users understand C# or VB programming.

1.3�� Installing the TransApplet

Copying the �CD:\transapplet80� to the C-driver will complete the installation. After copying, you should have both the Library Files and the Chapter Example.

1.4�� Attrasoft TransApplet Class Library

The TransApplet library has many *.dll files:

C:\transapplet80\xyz80.dll

�

Each Library file has one main library, which has the same name as the file name:

Attrasoft.TransApplet80.xyz80

Each Library has one main class, which has the same name as the file name:

Attrasoft.TransApplet80.xyz80.xyz80

The advantage of making

class-name = library-name = file-name

is that you will know what to use. If you go to a file xyz80.dll, you will use Library, �Attrasoft.TransApplet80.xyz80�, and you will use class, xyz80. There is no guessing which class you should choose if the Library has more than one class.

The disadvantage is that the compiler will not be able to distinguish between the class-name and the Library-name unless you use the full name.

Xyz80 alone is not understandable by the C# compiler. To declare an object, you must use the full path:

Attrasoft.TransApplet80.xyz80.xyz80 �� myObject80;

Each chapter has a project. Many chapters may share the same project. These projects provide an integration source code so you can cut and paste to produce your own application.

1.5�� Linking the TransApplet

To link to these Libraries:

Start C# project;
Right click References in Solution Explorer and select Add Reference;
Click Browse Tab;
Select �c:\transapplet80\*.dll� files;
Click the �OK� button.

2.� Image Recognition Overview

2.1�� Attrasoft Image Recognition Basics

The Attrasoft Image Recognition Approach has two basic phases:

Whole Images
Sub-Images

In matching whole images, a unique image signature is computed for each image. Different images will have different signatures. The Signature Matching is based on these image signatures. This matching can easily reach the speed of 100,000 to 1,000,000 whole images per second and is very fast.

In the Image Segment matching, image signatures may or may not be used. If no image signatures are used, the recognition speed is much slower.

Image Matching is further divided into filters. It is these filters that will perform the image matching tasks.

2.2�� Image Recognition Internal Structures

An image recognition application is roughly divided into:

Level 5: User Interface;

Level 4: Data Management;

Level 3: Image-Matching Layer;

Level 2: Scanner, Camera, and their Drivers;

Level 1: PC with Windows.

The TransApplet Library will be used to implement the Image-Matching layer. Stand-alone software, like the Attrasoft ImageFinder, is roughly divided into 3 layers:

Level 3: User Interface;

Level 2: Image-Matching Layer;

Level 1: PC with Windows.

3.� TransApplet Overview

ImageFinder for Windows is an off-the-shelf Application Software that enables System Integrators, Solution Developers, and Individuals to quickly test their own Image Recognition ideas.

VideoFinder for Windows is an off-the-shelf Application Software that enables System Integrators, Solution Developers, and Individuals to quickly test their own video Recognition ideas.

TransApplet is .Net Class Library that enables System Integrators, Solution Developers, and Individuals to quickly add Image Recognition capability to their products and services.

This User�s Guide will show you how to build software like the ImageFinder using C# .Net. If you have not learned how use to the ImageFinder, you should start with the ImageFinder now. �The entire ImageFinder project is located in the ImageFinder sub-folder.

The TransApplet consists of a set of class libraries. Each chapter will introduce one of these libraries as you continue to build the �ImageFinder�. Some minor graphical user interface issues will be ignored here because it is already complicated enough.

3.1�� TransApplet �API

Chapter 4, API, describes the API (Application Programming Interface) of the �Attrasoft.TransApplet80� Class Library.

3.2�� User Interface

Chapter 5, User Interface, briefly describes how to set up the forms and the menu bar for the ImageFinder, and also how to link to class libraries.

3.3�� Input & Display

Chapter 6, Input, briefly describes how to enter data to the ImageFinder. The input images are further divided into:

Key image � the sample image the ImageFinder will match against;
Search Source � a list of previously stored images to be searched.

An image path specifies the training image. A search-folder specifies the Search Source.

Chapter 7 will also briefly describe how to display the images. The Image Display will play the following role:

Display Key Segment;
Display the first image in the search source;
Display the next image and the previous image in the search source.
Display the first and next image in the matching results.
Display processed images (how an image is viewed by the software).

3.4�� ImageFinderLite

Chapter 8, ImageFinderLite, briefly describes how to build the software, ImageFinderLite.

3.5�� Signature Recognition

Chapter 9, Image Signatures, will describe how to generate image signatures.

3.6�� Whole Image Matching

Chapter 10, Whole Image Matching, will describe the steps for whole image matching.

Chapter 11, Binary Search, will describe the steps for binary whole image matching.

3.7�� Dynamic Library

Chapter 12, Dynamic Library, introduces the dynamic library used in the ImageFinder. Dynamic Library allows the master library to be updated via insertion, deletion, and update.

3.8�� Image Segment Matching

Chapter 13, Image Variation, will describe how to generate image signatures for some simple variations, such as rotation and flips.

Chapter 14, Sub-Image Matching, introduces the sub-image matching used in the ImageFinder.

Chapter 15, Fast Sub-Image Matching, gives a more options for matching sub-images.

3.9�� Videos

Chapter 16, Video Input, briefly describes how to enter video data to the ImageFinder.

Chapter 17, Whole Video Matching, introduces matching with a whole video as a unit.

Chapter 18, Video Frame Matching, introduces matching with a video frame as a unit.

Chapter 19, Video Object Matching, introduces matching with an object inside a video frame as a unit.

3.10�� Image Processing

Chapter 20, Image Preprocessing Concept, chapter 21, Image Preprocessing and chapter 22, Image Processing, introduces the image processing process required for the ImageFinder.

Chapter 23, Normalization, introduces image representation required for the ImageFinder.

Chapter 24 and Chapter 25 discuss the internal parameters in the ImageFinder.

3.11�� Customized Software

Chapter 26, Development Process, introduces the Image Recognition Implementation Process using the TransApplet.

4.�� API

This chapter will introduce the interfaces of the important classes and important functions. The interface for other classes will be introduced in the later chapters as they are used.

4.1�� Image Signature

Whole Image Matching is done through Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature.

This section introduces the Image Signature interface. An Image Signature consists of the following attributes:

Name (For example, xyz.jpg)

Path (c:\abc\)

Attribute 1

Attribute 2

�

The interface for Image Signature is:

public interface I_ImageSignature

�� {

�� int getStatus ();

�� string getID();

�� string getImageName ();

�� string getImagePath();

�� string getAbsolutePath ();

�� int getNumberOfAttributes ();

�� int [] getSignatureAttributes();

�� int getSignatureAttributes (int index);

�� string toString ();

�� }

The following table lists the functions.

Function	Descriptions	Comments
int getStatus ()	Returns the status associated with the signature. Output: 1: signature ready. 0: signature not ready. -1: no image -2: image segmentation specification error -3: other error.
string getID()	Returns the ID associated with the signature. Output: image ID.
string getImageName ()	Returns the image name associated with the signature. Output: image name.
string getImagePath()	Returns the image path associated with the signature. Output: image path.
string getAbsolutePath ()	Returns the absolute path associated with the signature. Output: image absolute path.
int getNumberOfAttributes ()	Returns the number of attributes associated with the signature. Output: number of attributes.
int [] getSignatureAttributes()	Returns the attribute array associated with the signature. Output: attribute array.
int getSignatureAttributes() (int index)	Returns the attribute associated with the input index. Input: index. Output: attribute associated with the input index.
string toString ()	Returns the entire image signature as a string with fields separated by Tab.

4.2�� Signature Filter

Signature filter computes signatures in the TransApplet; the input is an image and the output is a signature.

The filter has two classes: Signature Generator and Signature Filter. The interface for Signature Generator is:

public interface I_SignatureGenerator

�� {

�� bool setSignatureFilter (int x);

�� int getSignatureFilter ();

�� ImageSignature getSignature (Bitmap b);

�� int getSignatureFilterSize ();

�� string [] getSignatureFilterNames();

�� }//class

The following table lists the functions.

Function	Descriptions	Comments
ImageSignature getSignature (Bitmap b)	Gets the signature of the input image. Input: �� Bitmap b Output: �� Signature.
int getSignatureFilterSize ()	Gets the number of attributes the library of signatures.
bool setSignatureFilter (int x) int getSignatureFilter ()	Selects a Signature Generator.

The interface for Signature Filter is:

public interface I_SignatureFilter

�� {

�� bool setSignatureFilter (int x);

�� int getSignatureFilter ();

�� string [] getSignatureFilterNames();

�� ImageSignature getSignature (string imagePath, string ID);

�� ImageSignature getSignature (string imagePath);

�� ImageSignature getSignature (Bitmap b, string ID);

�� ImageSignature getSignature (Bitmap b);

�� ImageSignature getSignature

( string imagePath, string ID,int x, int y, int w, int h);

�� ImageSignature getSignature

( Bitmap bImg, string path, string name, string ID,

int x, int y, int w, int h);

��

�� bool getLibrary ( string [] imageAbsolutePath, string fileName );

�� bool getLibrary

( string [] imageAbsolutePath, string [] IDs, string fileName );

��

�� bool getSegmentLibrary ( string [] imageAbsolutePath,

�� string [] IDs,

�� string [] xs,

�� string [] ys,

�� string [] ws,

�� string [] hs,

�� string fileName );

��

�� bool getSegmentLibrary ( string [] imageAbsolutePath,

�� string [] IDs,

�� int [] xs,

�� int [] ys,

�� int [] ws,

�� int [] hs,

�� string fileName );��

�� }//class

The following table lists the functions.

Function	Descriptions	Comments
bool setSignatureFilter (int x) int getSignatureFilter ()	Selects a Signature filter.
string [] getSignatureFilterNames();	Gets a list of Signature filter names.
ImageSignature getSignature (string imagePath, string ID); �� ImageSignature getSignature (string imagePath); ImageSignature getSignature (Bitmap b, string ID); ImageSignature getSignature (Bitmap b);	Gets the Signature of the input image.
ImageSignature getSignature ( string imagePath, string ID,int x, int y, int w, int h); ImageSignature getSignature ( Bitmap bImg, string path, string name, string ID, int x, int y, int w, int h);	Gets the Signature of the input image segment. Input: �� string imagePath, or �� Bitmap bImg �� string ID �� int x, int y, int w, int h. Output: �� Signature.
bool getLibrary �( string [] imageAbsolutePath, string fileName ); bool getLibrary ( string [] imageAbsolutePath, string [] IDs, string fileName ); �� bool getSegmentLibrary ( string [] imageAbsolutePath, string [] IDs, string [] xs, string [] ys, string [] ws, string [] hs, string fileName ); �� bool getSegmentLibrary ( string [] imageAbsolutePath, string [] IDs, int [] xs, int [] ys, int [] ws, int [] hs, string fileName );	Generates a Signature library from all images in string [] imageAbsolutePath and produces a file that contains all signatures. Input: �� string []�� imageAbsolutePath Output: �� A text file that contains the library of signatures.

4.3�� Matching Results

Attrasoft ImageFinder matches whole images or image segments. The ImageFinder can be used for:

Image Verification �� (1:1 Matching);
Image Identification �� (1:N Matching);
Image Search or Retrieval �� (1:N Matching); �
Multiple Matching �� (N:N and N:M Matching); and
Video Matching�� (1:1, 1:N, M:N).

The results for N:N Matching always goes to a file. The results for 1:1 and 1:N Matching go to a data structure called Results_1N. The interface for class, Results_1N, is:

public interface I_Results_1N

�� {

�� bool getStatus ();

��

�� int getNumberOfMatches();

�� string getImageID (int i) ;

�� string [] getImageID () ;

�� string [] getImageID_N ( int N );

��

�� string getScore(int i);

�� string [] getScore();

�� string [] getScore_N ( int N );

�� string getImageName (int i) ;

�� string [] getImageName () ;

�� string [] getImageName_N ( int N ) ;

�� string getImagePath (int i) ;

�� string []getImagePath () ;

�� string [] getImagePath_N ( int N ) ;

�� string getX(int i);

�� string [] getX();

�� string [] getX_N ( int N );

�� string getY(int i);

�� string [] getY();

�� string [] getY_N ( int N );

�� string getW(int i);

�� string [] getW();

�� string [] getW_N ( int N );

�� string getH(int i);

�� string [] getH();

�� string [] getH_N ( int N );

�� Results_1N sort ();

�� string toString ();

�� }

The following table lists the functions.

Functions	Descriptions	Comments
int getStatus ()	Returns the status of the current signature comparison: > 0: OK; < 0: Error.
int getNumberOfMatches()	Returns the number of matches of the current signature comparison.
string getImageID (int i) string [] getImageID () string [] getImageID_N ( int N )	Returns the matching IDs of the current signature.
string getScore(int i) string [] getScore() string [] getScore_N ( int N )	Returns the matching scores of the current signature.
string getImageName (int i) string [] getImageName () string [] getImageName_N ( int N )	Returns the matching names of the current signature.
string getImagePath (int i) string []getImagePath () string [] getImagePath_N �( int N )	Returns the matching paths of the current signature.
string getX(int i) string [] getX() string [] getX_N ( int N ) string getY(int i) string [] getY() string [] getY_N ( int N ) string getW(int i) string [] getW() string [] getW_N ( int N ) string getH(int i) string [] getH() string [] getH_N ( int N )	Returns the matching (x, y, w, h) of the current signature.
string toString ()	Returns the entire results as a string with fields separated by Tab.

4.4�� Whole Image Matching

NeuralFilter also matches two whole images. The interface for NeuralFilter is:

public interface I_NeuralFilter

�� {

bool training ( string a1_txt, string match_txt);

�� Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

�� ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

�� Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

�� Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(string path1, string path2);

�� Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(Bitmap left, Bitmap right);

��

�� Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

�� (Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,

�string a1File, string b1File);

�� Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(string keyuPath, string a1File, string b1File);

�� Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Bitmap keyImage, string a1File, string b1File);

��

�� bool findMatchNN (string a1File, string b1File);

�� bool findMatchNM (string a1File, string a2File, string b1File);

�� }

The following table lists the functions.

Functions	Descriptions
bool training ( string a1_txt, string match_txt)	Trains the BioFilter.
Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 �( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1, Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2); Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (string path1, string path2); Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (Bitmap left, Bitmap right);	Makes a 1:1 matching
Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,� string a1File, string b1File); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (string keyuPath, string a1File, string b1File) Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (Bitmap keyImage, string a1File, string b1File)	Makes a 1:N matching
bool findMatchNN (string a1File, string b1File); bool findMatchNM (string a1File, string a2File, string b1File);	Matches all image signatures in file a1File against all image signatures in a1File or a2File and saves the results to b1File.

4.5�� ImageLibrary

Dynamic Library allows the library (N images in a 1:N Matching) to be updated via insertion, deletion, and update.

The interface for Image Library filter is:

public interface I_ImageLibrary

�� {

�� string getLibraryID ();

�� bool setLibraryID (string x);

�� bool load ();

�� bool load ( string fileName);

�� bool load ( string fileName1,� string fileName2);

�� bool print ( );

�� bool clear();

�� bool backup ();

�� bool backup (string fileName);

�� bool addSignature (ImageSignature sig);

��

�� bool deleteSignature (ImageSignature sig);

�� bool deleteSignature (string ID);

�� bool replaceSignature (ImageSignature sig);

�� bool mergeLibrary (string libFile1, string libFile2, string outputLib);

�� }

The following table lists the functions.

Functions	Descriptions	Comments
int getLibraryID ()	Gets the Library ID (optional).
void setLibraryID (string x)	Sets the Library ID (optional).
bool load ()	Loads the default master library, lib1.txt.
bool load ( string fileName)	Loads master library specified by fileName.
bool load ( string fileName1,� string fileName2)	Loads two libraries specified by fileName1 and fileName2.
bool clear()	Clears the current library from RAM only.
bool backup ()	Saves current library to the default file, lib1_bk.txt.
bool backup (string fileName)	Saves current library to a back file.
bool addSignature (ImageSignature sig)	Adds a signature to a loaded image library in RAM.
bool deleteSignature (ImageSignature sig)	Deletes a signature from a loaded image library in RAM.
bool deleteSignature (string ID)	Deletes a signature from a loaded image library in RAM.	�
bool replaceSignature (ImageSignature sig)	Replaces a signature from a loaded image library in RAM.
bool mergeLibrary (string libFile1, string libFile2, string outputLib)	Merges two signature libraries into a single library. Input: �� string libFile1 �� string libFile2 �� string outputLib Output: �� A text file that contains the library of signatures from both input libraries.

4.6�� Matching Engine

The interface for the Neural Filter Matching Engine is:

public interface I_MatchingEngine

�� {

�� int getLowCut();

�� void setLowCut(int x);

��

�� int getHighCut();

�� void setHighCut(int x);

��

�� void setNeuralOutputType(int x);

�� int getNeuralOutputType();

�� void setThreshold(int x);

�� int getThreshold();

�� void setSensitivity(int x);

�� int getSensitivity();

�� void setBlurring(int x);

�� int getBlurring();

��

�� void setUseRelativeScore(int x);

�� int getUseRelativeScore();

�� bool setLibrary

(Attrasoft.TransApplet80.ImageLibrary80.ImageLibrary lib);

��

�� bool setSignature

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig);

bool setNFTraining

(Attrasoft.TransApplet80.MatchingEngineTraining80.MatchingEngineTraining80 a_1);

��

bool setBFTraining

(Attrasoft.TransApplet80.MatchingEngineTrainingBF80.MatchingEngineTrainingBF80 bf80_1);

�� Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11

�� ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

�� Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

��

�� Attrasoft.TransApplet80.Results_1N.Results_1N� findMatch ();

�� Attrasoft.TransApplet80.Results_1N.Results_1N findMatch

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig);

�� }

The following table lists the functions.

Functions	Descriptions	Comments
int getLowCut() void setLowCut(int x) �� int getHighCut() void setHighCut(int x) �� void setNeuralOutputType (int x) int getNeuralOutputType() void setThreshold(int x) int getThreshold() void setSensitivity (int x) int getSensitivity() void setBlurring (int x) int getBlurring() �� void setUseRelativeScore (int x) int getUseRelativeScore()	Gets and Sets matching engine parameters.
bool setLibrary (ImageLibrary lib)	Sets the library to be used in matching signatures.
bool setSignature (ImageSignature sig)	Sets the signature for matching.
bool setNFTraining (MatchingEngineTraining80 a_1) bool setBFTraining (MatchingEngineTrainingBF80 bf80_1)	Sets the training objects for matching.
Results_1N findMatch11 (ImageSignature sig1, ImageSignature sig2);	Gets 1:1 match results for sig1 and sig2.
Results_1N� findMatch ()	Gets the match results for the signature, sig, specified by function, setSignature (ImageSignature sig) Output: �� Results_1N.
Results_1N findMatch (ImageSignature sig)	Gets the match results for the signature, sig. Input: �� ImageSignature sig; Output: �� Results_1N.

4.7�� NeuralNet Filter

The NeuralNet Filter matches a segment of an image(s). The interface for the NeuralNet Filter Matching Engine is:

public interface I_NeuralNetFilter

�� {��

�� bool train (Bitmap img);

�� bool train (string sPath);

�� bool train (Bitmap img, int x, int y, int w, int h);

�� bool train (string sPath, int x, int y, int w, int h);

�� bool retrain (Bitmap img);

�� bool retrain (string sPath);

�� bool retrain (Bitmap img, int x, int y, int w, int h);

�� bool retrain (string sPath, int x, int y, int w, int h);

�� Attrasoft.TransApplet80.Results_1N.Results_1N

�� findMatch11 ( Bitmap img1);

�� Attrasoft.TransApplet80.Results_1N.Results_1N

�� findMatch11 ( string path1);

�� Attrasoft.TransApplet80.Results_1N.Results_1N

�� findMatch1N ( string [] fileList);

�� bool findMatchNN (string [] fileList, string c1File);

�� bool findMatchNM (string [] keyList, string [] libraryList, string c1File);

}

The following table lists the functions.

Functions	Descriptions
bool train (Bitmap img) bool train (string sPath) bool train (Bitmap img, int x, int y, int w, int h) bool train (string sPath, int x, int y, int w, int h)	Trains the neural net.
bool retrain (Bitmap img) bool retrain (string sPath) bool retrain (Bitmap img, int x, int y, int w, int h) bool retrain (string sPath, int x, int y, int w, int h)	Retrains the neural net.
Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11 ( Bitmap img1); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11 ( string path1); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N ( string [] fileList);	Makes a 1:N Matching.
bool findMatchNN (string [] fileList, string c1File) bool findMatchNM (string [] keyList, string [] libraryList, string c1File);	Makes a N:N Matching and N:M Matching.

4.8�� Other API

We will introduce other API�s in later chapters as we encounter them.

5.�� User Interface

This User�s Guide is written as if you will create your own version of the ImageFinder. We will do a step-by-step instruction for implementing an �ImageFinder�. Eventually, you will have your own �ImageFinder�.

This chapter briefly describes how to set up the form and menu bar for the ImageFinder. We will use C# .Net as the implementation language. We will build the user interface form with a Menu Bar and Input buttons.

5.1�� How to Create a Project

To create a new C# project:

Start the Visual Studio .Net, (see Figure 5.1).
Click File � New Project command. The New Project dialog box is displayed (Figure 5.2).
Language and Templates: Highlight the Visual C# project folder in the Project Type list to display the templates that are available for C#. Then, highlight the Windows Application template (Figure 5.2).
Location and Name: Enter a name for the project and select the location for the project. A folder with the same name as the project is automatically added to the location you specify. We will use chap4 as the project name.
Click the �OK� button in Figure 5.2 to start the new project (Figure 5.3).

Now go to the property window and set (Figure 5.4):

Form1.text to �ImageFinder�;
Resize Form1.

Figure 5.1� Create C# Project.

Figure 5.2� Select Project Templates.

Figure 5.3� Form1.

Figure 5.4� Set Form1.Size.

5.2�� How to Create Menus

To add a menu bar to a form:

Add a �MenuStrip� or �MainMenu� control to a form from the toolbox. The control will appear in the component Design Tray at the bottom of the Form Design window. The menu of a form is determined by the Menu property.

To add items to the menu, click the �MenuStrip� or �MainMenu� control and click wherever it says �Type Here� in the menu designer. Then, type the text of the menu item and press the Enter key. Additional entry areas appear below and to the right of the entry.

To edit the property for each menu item, right-click the menu and select the Property command.

5.3�� Link to Class Library

Figure 5.5�� References in the Solution Explorer.

In this section, XYZ denotes any class located in c:\transapplet80\.

To use the class library, �Attrasoft.TransApplet80.XYZ�:

(1) Add a �using� statement:

using Attrasoft.TransApplet80.XYZ;

(2) Right click References and select Add reference in the Solution Explorer (Figure 5.5);

(3) Use the Browse button to find �XYZ.dll�; highlight the library under �Selected Components� and click the �OK� button.

Now the class library, �Attrasoft.TransApple80.XYZ�, is ready to be used. To make sure the link is successful, you should see XYZ under References in the Solution Explorer.

5.4�� Declare Objects

Each Library has one main class, which has the same name as the Library-name:

Attrasoft.TransApplet80.xyz80.xyz80.

�

The advantage of naming the class-name as the library-name is that you will know what to use, i.e.

Attrasoft.TransApplet80.xyz80.xyz80.

There is no guessing which class you should choose if the Library has more than one class.

The disadvantage is the compiler will not be able to distinguish between the class-name and the Library-name unless you use the full name.

To declare all TransApplet objects using the full class-name like this:

public Attrasoft.TransApplet80.xyz80.xyz80� myObj80

�� = new Attrasoft.TransApplet80. xyz80.xyz80 ();

6.�� Input

This chapter describes how to enter data to the ImageFinder. The input images are further divided into:

Key
Search Source.

An Image Path specifies the training image. A Search-Folder specifies the Search Source.

This chapter will introduce several class libraries,

Attrasoft.TransApplet80.InputModule80,

Attrasoft.TransApplet80.InputList80,

Attrasoft.TransApplet80.GUI.

The chapter project is in �.\ transapplet80.chap6�.

6.1�� Class Library Name

The class library is:

Attrasoft.TransApplet80.InputModule80.

The main class in this library will be:

Attrasoft.TransApplet80.InputModule80. InputModule.

The interface is:

public interface I_ InputModule80

�� {

�//Input:

�� bool setDirList(string sInput);

�� bool setSubDirList(string sInput);

�� bool setFileList(string sInput);

�� bool setFileSegmentList(string sInput);

�� bool setFileSegmentKeyList(string sInput);

�� bool setAccessList(string sInput, string sSQL);

�� bool setAccessSegmentList(string sInput, string sSQL);

��

//Output:��

�� Attrasoft .TransApplet80 .InputList80 .InputList� getList();��

�� }

This class has two parts:

Input;
Output.

The input can be:

Folder path
File path containing images
File path containing image segments
Access file containing images
Access file containing image segments
�

Output is an object of the following class:

public class InputList

�� {

�� public int status = 0;

�� public string[] imageAbsoultePath = null;

�� public string[] imageID = null;

�� public string[] imageName = null;

�� public string[] imagePath = null;

�� public string[] imageX = null;

�� public string[] imageY = null;

�� public string[] imageW = null;

�� public string[] imageH = null;

�� public string[] key0 = null;

�� public string[] key1 = null;

��

�� }

6.2�� Class Library Overview

The functions in classes are listed in the following table.

Function	Description
string [] setDirList ( string sInput)	To get a string list of the absolute paths of all images in directory, set the input directory, sInput.
string [] setSubDirList ( string sInput)	To get a string list of the absolute paths of all images in all sub-directories, set the input directory, sInput.
string [] setFileList ( string sInput)	To get a string list of the absolute paths of all images in file, set the input file, sInput.
string []� getFileSegmentList ( string sInput)	To get a string list of the absolute paths of all image segments in file, set the input file, sInput.
string [] getAccessList ( string sInput, string sSQL)	To get a string list of the absolute paths of all images in access file, sInput, specified by a SQL statement, sSQL, set the two input variables.
string []� getAccessSegmentList ( string sInput, string sSQL)	To get a string list of the absolute paths of all image segments in the access file, sInput, specified by a SQL statement, sSQL, set the two variables.
Attrasoft .TransApplet80 .InputList80 .InputList� getList()	After the input is specified, get the output.

6.3�� Set Up the Chapter Project

Figure 6.1� Chapter 6 Project.

We will add the following controls to our form:

��

Key Button, Key Text Box

Use the �Key� button to display the selected key image and print the image path in the key text box. For example, click the button, go to the directory, "C:\transapplet80\ex_label", and then select an image. The selected image will be shown in the ImageFinder to indicate the specification is successful and the key-image is ready for training or retraining.

Source Button, Source Text Box

Use the �Source� button to select a search source and display the source in the text box.

Clear Button

Use the �Clear� button to clear the rich text box.

Rich Text Box

�� Use the Rich Text Box to display messages.

Key Picture Box

Use the Key Picture Box to display key images.

Source Picture Box

Use the Source Picture Box to display images in the search source.

OpenFileDialog

�� Use the OpenFileDialog to select a key image.

FolderBrowserDialog

�� Use the FolderBrowserDialog to select a source path.

After adding these controls, the form will look like Figure 6.1.

6.4�� Link to Class Library

Figure 6.2� Link to Class library.

To include the class library to the project,

Right click References and select Add Reference in the Solution Explorer;
Browse to find �InputModule80.dll� in �c:\transapplet80�;
Highlight it and click the �OK� button (Figure 6.2).

To declare an object, please use the full path for class:

internal Attrasoft.TransApplet80.InputModule80.InputModule

inputModuleSource;��

inputModuleSource

= new Attrasoft.TransApplet80.InputModule80.InputModule(this.richTextBox1);

Now the object is ready to use. In addition, we will add several class libraries:

�� Helper80;

�� GUI;

�� InputList80.

6.5�� Key Segment

We will add a class, �Input�, in the project. We will implement the input buttons in this class. In this way, the main class is simpler. We will declare an object, input, as follows:

Input input;

�� private void Form1_Load(object sender, System.EventArgs e)

�� {

�� h80 = new Attrasoft.TransApplet80.Helper80.Helper80(this.richTextBox1);

�� inputModuleSource

= new Attrasoft.TransApplet80.InputModule80.InputModule(this.richTextBox1);

�� guiSource = new Attrasoft.TransApplet80.GUI.GUI_Key(this.pictureBox2, h80);

�� guiResults = new Attrasoft.TransApplet80.GUI.GUI_Results(this.pictureBox3, h80);

�� input = new Input(this);

��

�� }

The helper80 object is used to print a message:

void appendText (string s)

�� {

�� f.h80 .appendText ( s);

�� }

�� void appendLine (string s)

�� {

�� appendText ( s +"\n");

�� }

We can double click the Key Segment buttons to complete the programming:

private void button1_Click(object sender, System.EventArgs e)

�� {

�� input.keySegment();

�� }

which in turn, calls a function:

�� public bool keySegment()

�� {

�� if ( f.openFileDialog1.ShowDialog () != DialogResult.OK )

�� return false;

�� string fileName = f.openFileDialog1.FileName;

�� try

�� {

�� bTrain =new Bitmap (fileName);

�� f.pictureBox1.Image = bTrain;

�� f.textBox1.Text = fileName ;

�� f.richTextBox1.AppendText

( "Display Key Segment:\n " + fileName +"\n");

�� }

�� catch

�� {

�� appendText ("Invalid key image !\n");

�� return false;

�� }

�� return true;

�� }//keySegment()

The first section of code opens a file dialog so the user can select a file:

if ( f.openFileDialog1.ShowDialog () != DialogResult.OK )

�� return false;

The next section of code gets the selected key:

string fileName = f.openFileDialog1.FileName;

The last section of code creates an image and displays the image:

bTrain =new Bitmap (fileName);

�� f.pictureBox1.Image = bTrain;

�� f.textBox1.Text = fileName ;

�� f.richTextBox1.AppendText ( "Display Key Segment:\n " + fileName +"\n");

6.6�� Search Source Button

Now, we implement the �Source� button:

private void button2_Click(object sender, System.EventArgs e)

�� {

�� input.source( );

�� }

which in turn, calls the following function in the Input class:

�� internal bool source()

�� {

�� try

�� {

�� if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

�� return false;

�� string cDir = f.folderBrowserDialog1.SelectedPath;

�� f.textBox2.Text = cDir;

�� f.inputModuleSource.setSubDirList(cDir);

�� f.guiSource.setInput(f.inputModuleSource.getList());

�� if (!f.guiSource.getStatus())

�� {

�� f.appendText("Data Source fails!\n");

�� return false;

�� }

�� f.guiSource.setLists();

�� }

�� catch (Exception e)

�� {

�� f.appendText(e.ToString() + "\n");

�� return false;

�� }

�� return this.f.guiSource.Button_First(false);

�� }

The first section of code gets the current search directory via the folder browser dialog:

�� if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

�� return false;

�� string cDir = f.folderBrowserDialog1.SelectedPath;

�� f.textBox2.Text = cDir;

The next section enters the selected folder to the InputModule object:

�� f.inputModuleSource.setSubDirList(cDir);

The third section gets the string list of the image paths:

f.inputModuleSource.getList().

And passes this list to the display section:

�f.guiSource.setInput(f.inputModuleSource.getList());

The last section displays the first images:

�� f.guiSource.setLists();

f.guiSource.Button_First(false);

To test, run the software and click the first image in folder, �c:\transapplet80\ex_label�.

7.�� Image Display

This chapter describes how to display images in a search source. To display these search images, three buttons will do:

First Button

Use the �First� button to display the first image in the search source.

> (Next) Button

Use the �Next� button to display the Next image in the search source.

< (Previous) Button

Use the �Previous� button to display the Previous image in the search source.

After adding the three buttons, the form looks like Figure 7.1.

Figure 7.1� First, Next (>), and Previous (<) Buttons.

7.1�� Class Library Name

The class library is:

Attrasoft.TransApplet80.GUI.

It has three libraries:

new Attrasoft.TransApplet80.GUI.GUI_Key

new Attrasoft.TransApplet80.GUI.GUI_Source

Attrasoft.TransApplet80.GUI.GUI_Results

They are designed for:

�� Key;

�� Source;

�� Results.

GUI_Key is the base class of GUI_Source.

GUI_Source is the base class of GUI_Results.

At this point, GUI_Key and GUI_Source are the same.

All three class libraries function in a similar way.

The GUI objects are:

�� guiSource = new Attrasoft.TransApplet80.GUI.GUI_Key(this.pictureBox2, h80);

�� guiResults = new Attrasoft.TransApplet80.GUI.GUI_Results(this.pictureBox3, h80);

The functions in this class are:

Function	Description
Bool setInput ( ImageList80)	Sets an input list of image paths, image ID, �.
void setList ( )	Sets the class ready for the first button, next button, and previous button.
bool getStatus ()	Gets the status.
Bool Button_First(bool);	Displays the first image.
Bool Button_Next(bool);	Displays the next image.
Bool Button_Previous(bool);	Displays the previous image.

The sample codes are:

Input

�� f.guiSource.setInput(f.inputModuleSource.getList());

Computation

�� f.guiSource.setLists();

Display

f.guiSource.Button_First(false);

f.guiSource.Button_Next(false);

f.guiSource.Button_Previous(false);

7.2�� Implementing H, >, and < Buttons

In this section, we will implement the First, Next, and Previous buttons. To see images in a search source, start with the �First� button and then, keep clicking the �Next� button to display the next image; eventually, all of the images in the search source will be shown.

The code for the three buttons consists of three parts:

Populating GUI Object;
Computation to get the object ready;
First, Next, Previous buttons.

First of all, the object has to be populated. This is accomplished by the source button in the last chapter:

�� f.guiSource.setInput(f.inputModuleSource.getList());

�� if (!f.guiSource.getStatus())

�� {

�� f.appendText("Data Source fails!\n");

�� return false;

�� }

�

�Then we will set up the object:

�� f.guiSource.setLists();

Finally, the code for the First, Next, and Previous buttons are:

�� private void button3_Click(object sender, EventArgs e)

�� {

�� bool b = this.guiSource.Button_First(true);

�� if (!b)

�� setText(guiSource.getMessage());

�� }

�� private void button5_Click(object sender, EventArgs e)

�� {

�� bool b = guiSource.Button_Next();

�� if (!b)

�� setText(guiSource.getMessage());

�� }

�� private void button6_Click(object sender, EventArgs e)

�� {

�� bool b = guiSource.Button_Previous();

�� if (!b)

�� setText(guiSource.getMessage());

�� }

7.3�� Test

This section tests the project:

(1) Run the project;

(2) Click the �Key� button, go to �c:\transapplet80\ex_wheel\�, and select a key image;

(3) Click the �Source� button, go to �c:\transapplet80\ex_wheel\�, and select an image.

(4) Click the �First� button;

(5) Click the �Next� button several times;

(6) Click the �Previous� button several times.

7.4�� Output �Images

In the last a few sections, we demonstrated how to display Input Source images. After a matching, the output images will be displayed in a similar fashion.

At the end of a 1:1 or 1:N matching, a Result object will be produced. This object will be entered into GUI_Result object:

guiResults.outputResults(results_1N, softwareDir);��

To display result images: ��

guiResults.Button_First(false);

guiResults.Button_Next(false);

guiResults.Button_Previous(false);

8.�� ImageFinderLite

In this chapter, we will briefly describe how to build a quick and dirty image recognition application. In particular, we will discuss how to identify a full image.

8.1�� ImageFinderLite

Attrasoft.TransApplet80.ImageFinderLite is a component Attrasoft TransApplet. To use the ImageFinderLite object, you will need 5 steps:

Step 1. Set the Folder that contains the images you want to search through.

Step 2. Create a Image Signature Library from the folder.

Step 3. Load the Signature Library into the ImageFinderLite object.

Step 4. Get a key image file name, which will be matched against the library.

Step 5. Make a 1:N search.

8.2� �Interface

The ImageFinderLite interface is given below:

public interface� I_ImageFinderLite

�� {

�� //variables:

�� //1. Foldername (string)

�� //2. Library Name (string)

�� //3. KeyImage (string)

�� //4. Signature (Attrasoft object)

�� //5. Result (Attrasoft object)

�� //step 1 Set Library Folder

�� bool setFolder(string folderName);

�� string getFolder();

�� //step 2 Create Image Signature Library

�� bool computeSignatureLibrary();

�� bool setLibraryName(string a1File);

�� string getLibraryName();

�� //setp 3 Load Signature Library

�� bool loadLibrary();

�� //step 4 Get key file name

�� bool setKeyImage (string key);

�� string getKeyImage();

�� //step 5 search

�� bool� search ( );

�� bool setOutputFileName (string b1File);

�� string getOutputFileName ( );

}

Step 1. Set the Folder that contains the images you want to search through.

bool setFolder(string folderName)

string getFolder()

Use setFolder (string folderName) to set the folder that contains the images you want to search through; use getFolder() to see the current setting.

Step 2. Create Image Signature Library from the folder.

bool computeSignatureLibrary()

Use computeSignatureLibrary() to get a file that contains the image signatures from the specified folder.

bool setLibraryName(string a1File)

string getLibraryName()

Use setFolder setLibraryName(string a1File) to set the image signature file name that contains the images you want to search through. The default file is �.\data\a1.txt�, where �.� is the application folder. Use getLibraryName() to see the current setting.

Step 3. Load the Signature Library into the ImageFinderLite object.

bool loadLibrary();

Use loadLibrary() to load the Signature Library into the ImageFinderLite object.

Step 4. Get a key image file name, which will be matched against the library.

�

bool setKeyImage (string key)

string getKeyImage()

Use setKeyImage (string key) to set the unknown image that you want to search; use getKeyImage() to see the current setting.

Step 5. Make a 1:N search.

bool� search ( )

Use search ( ) to make an 1:N search.

bool setOutputFileName (string b1File)

string getOutputFileName ( )

Use setOutputFileName (string b1File) to set the output file name; the default name is �.\data\b1.txt�, where �.� is the application folder.� Use getOutputFileName ( ) to see the current setting.

8.3�� Programming Example

Now we will show a programming example.

1. Create a User Interface in Figure 8.1.

Figure 8.1 A User Interface, which contains the five steps.

2. Link to the Library

Figure 8.2 Add the class library; see the highlighted areas in the Solution Explorer.

3. Initial Files

Apart from linking the class libraries, there are several data files required by the class libraries. You must provide these data files in order for the class libraries to function properly. You will need to copy �.\data\*.*� from the CD into your project folder. These files must be placed in the �.\data\� sub-folder of your project

4. ImageFinderLite Object

Add an ImageFinderLite object:

Attrasoft.TransApplet80.ImageFinderLite.ImageFinderLite ifl;

��

Create the object in the form constructor:

��

public Form1()

{

�InitializeComponent();

ifl = new Attrasoft.TransApplet80.ImageFinderLite.ImageFinderLite

( this.richTextBox1 );

}

Figure 8.3 Add the ImageFinderLite object; see the highlighted areas in the Solution Explorer.

5. Following the 5 steps:

Step 1. Set the Folder that contains the images you want to search through.

Step 2. Create the Image Signature Library from the folder.

Step 3. Load the Signature Library into the ImageFinderLite object.

Step 4. Get a key image file name, which will be matched against the library.

Step 5. Make a 1:N search.

The details are given in the next section.

8.4� �Codes for Each of the Steps

Step 1. Set the Folder that contains the images you want to search through.

private void button1_Click(object sender, EventArgs e)

�� {

�� if (folderBrowserDialog1.ShowDialog() != DialogResult.OK)

�� return ;

�� string cDir = folderBrowserDialog1.SelectedPath;

�� textBox1.Text = cDir;

�� ifl .setFolder ( cDir );

�� }

Step 2. Create the Image Signature Library from the folder.

private void button2_Click(object sender, EventArgs e)

�� {

�� bool b = ifl.computeSignatureLibrary();

�� }

Step 3. Load the Signature Library into the ImageFinderLite object.

private void button3_Click(object sender, EventArgs e)

�� {

�� if (openFileDialog1.ShowDialog() != DialogResult.OK)

�� return;

�� string fileName = openFileDialog1.FileName;

�� textBox2.Text = fileName;

�� ifl.setKeyImage(fileName);

�� }

Step 4. Get a key image file name, which will be matched against the library.

�� private void button4_Click(object sender, EventArgs e)

�� {

�� bool b = ifl.loadLibrary ();

�� }

Step 5. Make a 1:N search.

�� private void button5_Click(object sender, EventArgs e)

�� {

�� ifl.search () ;

�� }

8.5� �Test Run

Run the software to get Figure 8.1; then

� Click Button 1 to set the Folder that contains the images you want to search through.

� Click Button 2 to create the Image Signature Library from the folder.

� Click Button 3 to load the Signature Library into the ImageFinderLite object.

� Click Button 4 to get a key image file name, which will be matched against the library.

� Click Button 5 to Make a 1:N search.

The results are given in the text window in Figure 8.1 and in a text file, �.\data\b1.txt�.

9.�� Image Signatures

Image Matching is implemented through something called Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature.

In this project, we will use Signature Menu Items to compute. The input is an image and the output is a signature. We will show how to use the Image Signatures in image matching in the next chapter. This chapter introduces how to compute signatures only, which will take an input image and produce a signature.

The chapter project is located at:

C:\transapplet80\imagefinder\.

The executable file is located at:

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

We also call this folder �.\�. Most of the functions will be implemented in two classes:

� MainMenuToAPI

� GUIRelated

Class, MainMenuToAPI, will implement all TransApplet related functions. This class will declare an object of �Attrasoft.TransApplet80.EntryPoint80.EntryPoint80�, which contains all of the required objects.

Class, GUIRelated, will implement all functions related to GUI (Graphical User Interface), such as input/output, display images, messaging, �

The basic object is:

Attrasoft.TransApplet80.SignaturFilter80.SigantureFilter80.

Starting from this chapter, we will use a new object from this class:

Attrasoft.TransApplet80.EntryPoint80.EntryPoint80

This class has a list of objects; one of the objects is the SignatureFilter80.

9.1�� Signature Menu

First of all, we will create the menu shown in Figure 9.1 and create the menu items under the Signature Menu.

Since the project is already implemented, you can simply open the project in:

.\imagefinder\,

and see the implementation of the menu items. We will explain the main functions in each menu item.

Figure 9.1 Signature Menu.

9.2�� API

The basic idea is:

Image Signature = SignatureFilter. getSiganture (image);

As we can see, there are two objects here, Signature and Signature Filter. The Signature functions are:

Function	Descriptions	Comments
int getStatus ()	Returns the status associated with the signature. Output: 1: signature ready. 0: signature not ready. -1: no image -2: image segmentation specification error -3: other error.
string getID()	Returns the ID associated with the signature. Output: image ID.
string getImageName ()	Returns the image name associated with the signature. Output: image name.
string getImagePath()	Returns the image path associated with the signature. Output: image path.
string getAbsolutePath ()	Returns the absolute path associated with the signature. Output: image absolute path.
int getNumberOfAttributes ()	Returns the number of attributes associated with the signature. Output: number of attributes.
int [] getSignatureAttributes()	Returns the attribute array associated with the signature. Output: attribute array.
int int [] getSignatureAttributes() (int index)	Returns the attribute associated with the input index. Input: index. Output: attribute associated with the input index.
string toString ()	Returns the entire image signature as a string with fields separated by Tab.

The Signature Filter functions are:

Function	Descriptions	Comments
bool setSignatureFilter (int x) int getSignatureFilter ()	Selects a Signature filter.
string [] getSignatureFilterNames();	Gets a list of Signature filter names.
ImageSignature getSignature (string imagePath, string ID); �� ImageSignature getSignature (string imagePath); ImageSignature getSignature (Bitmap b, string ID); ImageSignature getSignature (Bitmap b);	Gets the Signature of the input image.
ImageSignature getSignature ( string imagePath, string ID,int x, int y, int w, int h); ImageSignature getSignature ( Bitmap bImg, string path, string name, string ID, int x, int y, int w, int h);	Gets the Signature of the input image segment. Input: �� string imagePath, or �� Bitmap bImg �� string ID �� int x, int y, int w, int h. Output: �� Signature.
bool getLibrary �( string [] imageAbsolutePath, string fileName ); bool getLibrary ( string [] imageAbsolutePath, string [] IDs, string fileName ); �� bool getSegmentLibrary ( string [] imageAbsolutePath, string [] IDs, string [] xs, string [] ys, string [] ws, string [] hs, string fileName ); �� bool getSegmentLibrary ( string [] imageAbsolutePath, string [] IDs, int [] xs, int [] ys, int [] ws, int [] hs, string fileName ); bool getKeywordLibrary( string [] imageAbsolutePath, string [] IDs, int [] xs, int [] ys, int [] ws, int [] hs, string filename, string [] key0, string [] key1, � );	Generates a Signature library from all images in string [] imageAbsolutePath and produces a file that contains all signatures. Input: �� string []�� imageAbsolutePath Output: �� A text file that contains the library of signatures.

9.3�� TransApplet Objects

The first thing we will do is to create all of the objects required for the ImageFinder. As we discussed earlier in this chapter, all of the objects required for the ImageFinder project is grouped into a single class. All we have to do is to create an object of this class and we will call it script.

We will create a new class, MainMenuToAPI. We will do all of the menu item implementation in this class.

internal Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 script;

MainMenu f;

public MainMenuToAPI(MainMenu f1)

{

try

�� {

�� f = f1;

�� script =

new Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 (f.richTextBox1);

}

�� catch (Exception e )

�� {

�� f.richTextBox1.Text = e.ToString () +"\n";

�� }

9.4�� Key Signature

To compute the signature of a key image:

Click the �Key� button to select a key image or sample image;
Click the �Signature/1:1 First Signature� to compute the signature.

Example. Key Signature.

Click the �Key� button to select �.\ex_wheel\2067(1).jpg�;
Click the �Signature/1:1 First Signature� to compute the signature.

Figure 9.2� Click �Signature/Left Signature (Key)�.

The Signature looks like:

�� 3055�� 29�� 33�� 30355� 255�� 18711� 195�� 1�� 1��

In 1:1 Matching or 1:N Matching, the signatures are computed behind the screen; you do not need to use the signature directly. The menu item, �Signature/1:1 First Signature�, shows you the mechanism behind the screen so you can see what a signature looks like.

Now, we will implement these two menu items. Double click these menu items and you will see:

private void menuItem2_Click(object sender, System.EventArgs e)

�� {

�� mainMenuToAPI.signature_Key_Signature ( textBox1.Text );

�� }

Here, mainMenuToAPI is an object, which will implement all functions. The main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool signature_Key_Signature ( string key)

�� {

�� if ( ! System.IO .File .Exists (key) )

�� {

�� appendText ( "Please enter a valid Key!\n");

�� return false;

�� }

�� int x =0, y = 0, w =0, h = 0;

script.imageSignature

= script.signatureFilter.getSignature (key, "1of1N",x, y, w, h );

�� if ( script.imageSignature != null )

�� {

�� setLine ("The Signature for the left picture box:");

�� appendLine (script.imageSignature.toString () );

�� return true;

�� }

�� else

�� return false;

�� }

The following section of code simply makes sure the key image exists:

if ( ! System.IO .File .Exists (key) )

�� {

�� appendText ( "Please enter a valid Key!\n");

�� return false;

�� }

The following section of code computes the signature of the key segment:

script.ImageSignature

�� = script.signatureFilter.getSignature (key, "1 of 1:N", x, y, w, h );��

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, �script.ImageSignature� is the image signature object; and script.signatureFilter is the signature filter object. The parameter, "1 of 1:N", is the ID, which we arbitrarily assigned to the key image. The following section of code prints the signature of the key segment in the text window:

if ( script.imageSignature != null )

�� {

�� setLine ("The Signature for the left picture box:");

�� appendLine (script.imageSignature.toString () );

�� return true;

�� }

�� else

�� return false;

9.5�� Signature File Concepts

In 1:N Matching, the Key image will be matched against all images in a directory. The key signature is computed first, as we demonstrated in the last section. Then this signature will be matched against all signatures in the N-signature file. The N-signature file has a fixed name, �.\data\a1.txt�.

In N:M Matching, the N-signature file is a1.txt, and the M-signature file is a2.txt. The two files, a1.txt and a2.txt, are computed first. Then all of the signatures in a1.txt will be matched against all of the signatures in a2.txt. The M-signature file has a fixed name, �.\data\a2.txt�.

There are 2 menu items:

Signature/1:N Signature (.\data\a1.txt)
Signature/N:M Signature (.\data\a2.txt)

These commands compute the signatures for all images in a folder, including all of its subfolders. The only difference between the two commands is where to save the signatures.

Signature/1:N Signature (.\data\a1.txt)

Menu item, �Signature/1:N Signature (.\data\a1.txt)�, computes the signatures for all images in a directory and saves the signatures to a1.txt, the N-Signature file.�

Signature/N:M Signature (.\data\a2.txt)

Menu item, Signature/N:M Signature (.\data\a2.txt), computes the signatures for all images in a directory and saves the signatures to a2.txt, the M-Signature file.

9.6�� Signature File Implementation

Now, we will implement menu item, Signature/1:N Signature (.\data\a1.txt). Double click this menu item and enter:

private void button1_Click(object sender, EventArgs e)

�� {

�� this.mainMenuToAPI.signature_Search_source

�� (dataDir + "a1.txt", SIGNATURE_LIMIT);

�� }

Again, mainMenuToAPI is an object, which will implement all functions. The main form simply links menu items to functions in the mainMenuToAPI object. The implementation of function, signature_Search_source, is:

�� f.mainMenuToAPI.script.signatureFilter.getKeywordLibrary

�� ( f.guiSource.libraryList.imageAbsoultePath, f.guiSource.libraryList.imageID,

�� f.guiSource.libraryList.imageX, f.guiSource.libraryList.imageY,

�� f.guiSource.libraryList.imageW, f.guiSource.libraryList.imageH,

�� outputFile,

�� f.guiSource.libraryList.key0, f.guiSource.libraryList.key1,

�� f.guiSource.libraryList.key2, f.guiSource.libraryList.key3,

�� f.guiSource.libraryList.key4, f.guiSource.libraryList.key5,

�� f.guiSource.libraryList.key6, f.guiSource.libraryList.key7,

�� f.guiSource.libraryList.key8, f.guiSource.libraryList.key9

�� );

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, �script.signatureFilter� is the image signature filter object. The parameter, �f.gui.imageAbsoultePath�, is a string list that has the absolute paths of all source images to be matched. After the completion of signature computation, the results will be stored in a file specified by the parameter.

9.7�� Examples

Example. Compute N-Signatures:

Click the �Source� button;
Select folder, ./ex_wheel/, here �./� means the folder where the ImageFinder is located;
Click �Signature/N Signature (.\data\a1.txt)� to compute image signatures;
Click �Signature/Open a1.txt� to see the results.

10.�� Whole Image Matching

The NeuralFilter matches whole images. The chapter project is located at:

C:\transapplet80\imagefinder\.

The executable file is located at:

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

We also call this folder �.\�.

10.1�� Whole Image Matching Menu

The NeuralFilter is responsible for whole image matching. The �Whole Image� menu items are given in Figure 10.1.

Figure 10.1� NeuralFilter Menu.

10.2�� NeuralFilter API

The following table lists the NeuralFilter functions.

Functions	Descriptions
bool training ( string a1_txt, string match_txt)	Trains the NeuralFilter.
Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 �( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1, Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2); Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (string path1, string path2); Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (Bitmap left, Bitmap right);	Makes a 1:1 matching.
Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,� string a1File, string b1File); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (string keyuPath, string a1File, string b1File) Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (Bitmap keyImage, string a1File, string b1File)	Makes a 1:N matching.
bool findMatchNN (string a1File, string b1File); bool findMatchNM (string a1File, string a2File, string b1File);	Matches all image signatures in file a1File against all image signatures in a1File or a2File and saves the results to b1File.

10.3�� N:N Matching Implementation

Double click menu item �Whole Imager/N:N Matching (a1.txt vs. a1.txt)� and you will see:

�� private void toolStripMenuItem21_Click (object sender, System.EventArgs e)

�� {

�� mainMenuToAPI.neuralFilter_Matching_NToM�

�� ( dataDir + "a1.txt", dataDir + "a1.txt", dataDir + "b1.txt" );

�� }

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

Public bool neuralFilter_Matching_NToM

�� ( string a1File, string a2File, string b1File)

�� {

�� try{

�� script.neuralFilter.findMatchNM ( a1File, a2File, b1File);

�� }

�� catch (Exception e)

�� {

�� appendText (� e.ToString () + "\n");

�� return false;

�� }

�� return true;

}

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, script.neuralFilter is the NeuralFilter object. The parameters, (a1File, a2File, b1File), indicate all signatures in file, a1File, will be matched against all signatures in file, a2File. After completion of the matching, the results will be stored in a file specified by the last parameter, b1File.

10.4�� 1:N� Matching Implementation

Double click menu item �Whole Image/1:N Matching (Key vs. a1.tx)�� and you will see:

mainMenuToAPI .neuralFilter_Matching_1ToN�

�� ( textBox1.Text , dataDir + "a1.txt",� dataDir + "b1.txt" );

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool neuralFilter_Matching_1ToN

�� ( string key, string a1File, string b1File)

�� {

�� bool b = false;

�� if ( ! System.IO .File .Exists (key) )

�� {

�� appendText ( "Please enter a valid Key!\n");

�� return false;

�� }

�� b = this.signature_Key_Signature (key);

�� if ( ! b )

�� {

�� appendText ( "Key Signature computation fails!\n");

�� return false;

�� }

�� try

�� {

�� script.results_1N = script.neuralFilter.findMatch1N

( script.imageSignature,� a1File,� b1File);

�� }

�� catch (Exception e)

�� {

�� appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

�� return false;

�� }

�� if ( script.results_1N == null )

�� {

�� appendText ( "NeuralFilter 1:N Matching fails!\n" );

�� return false;

�� }

�� if ( script.results_1N.getStatus () )

�� {

�� setText ( script.results_1N.toString () + "\n");

�� appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

�� }

�� else

�� {

�� appendText ( "No Match!\n");

�� return false;

�� }

�� return createFile (key, b1File);

�� }

The following code simply makes sure the key image exists:

if ( ! System.IO .File .Exists (key) )

�� {

�� appendText ( "Please enter a valid Key!\n");

�� return false;

�� }

The next section of code computes the key signature:

�� b = this.signature_Key_Signature (key);

The next section code makes a 1:N Match, the results are in script.results_1N:

try

�� {

�� script.results_1N = script.neuralFilter.findMatch1N

( script.imageSignature,� a1File,� b1File);

�� }

�� catch (Exception e)

�� {

�� appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

�� return false;

�� }

The final section of code prints the 1:N Matching results:

if ( script.results_1N.getStatus () )

�� {

�� setText ( script.results_1N.toString () + "\n");

�� appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

�� }

�� else

�� {

�� appendText ( "No Match!\n");

�� return false;

�� }

In section 7.4, we have already discussed how to display the matching results from the result object, script.results_1N. At the end of a 1:1 or 1:N matching, a Result object will be produced. This object will be entered into GUI_Result object

guiResults.outputResults(script.results_1N, softwareDir);��

To disply the first, next, and previous result images, call: ��

guiResults.Button_First(false);

guiResults.Button_Next(false);

guiResults.Button_Previous(false);

11.�� Binary Search

For a large amount of images, say above 1 million images, binary search is much faster. However, binary search requires additional steps, signature sorting and signature management. The �Binary Search� menu is given in Figure 11.1.

The chapter project is located at:

C:\transapplet80\imagefinder\.

The executable file is located at:

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

We also call this folder �.\�. Run the project to get the figure below.

Figure 11.1 Binary Search Menu Items.

The search steps are:

1. Select a folder (Click the �1� Button in the Toolbar, see Figure 11.1)

2. Get Library Signatures (Click the �2� button in the Toolbar, see Figure 11.1)

3. Sort Signature (see Figure 11.1).

4. Select a Key image (Click the �3� Button in the Toolbar, see Figure 11.1).

5. Search (There are several options in Figure 11.1).

11.1�� Signatures

For binary search, the signature file is sort1.txt. Unlike sequential search, which allows many signature files to be loaded via a library, the sorted signatures will use one single file, so you have to manage this file.

To get a signature file sort1.txt:

1. Select a folder (Click the �1� Button in the Toolbar, see Figure 11.1).

2. Get Library Signatures (Click the �2� button in the Toolbar, see Figure 11.1).

3. Click menu item, �Binary Search/Create sort1.txt from a1.txt�.

If you have several signature files, you have to merge them:

Create sort1.txt;
Copy sort1.txt to sort2.txt;
Create sort1.txt with new data;
Merge sort1.txt and sort2.txt to sort3.txt.

The merger will be done by clicking �Binary Search/Merge sort1.txt, sort2.txt to sort3.txt�.

Sorting a signature file takes two steps:

Create an index file from a signature file;
Sort signatures from the signature file and index file.

Click the menu item, �Binary Search/Create sort1.txt from a1.txt�, you will see:

private void createSort1txtFromA1txtToolStripMenuItem_Click

(object sender, EventArgs e)

�� {

�� mainMenuToAPI.Sort_CreateSort1Froma1

�� (dataDir + "a1.txt",

�� dataDir + "index1.txt",

�� dataDir + "sort1.txt");

�� }

Which will call the following function.

internal bool Sort_CreateSort1Froma1

(string a1_txt, string index1_txt, string sort1_txt)

�� {

�� bool bTmp = Sort_indexing( a1_txt,� index1_txt);

�� if ( ! bTmp )

�� {

�� appendText("Indexing Fails!\n");

�� return false;

�� }

�� else

�� {

�� appendText("Indexing Completed!\n");

��

�� }

�� bTmp = Sort_sortitngFile(a1_txt, index1_txt, sort1_txt);

�� if (!bTmp)

�� {

�� appendText("sort1.txt failed!\n");

�� return false;

�� }

�� else

�� {

�� appendText("sort1.txt created!\n");

��

�� }

�� return true;

�� }

The first step is to create an index file:

�� private bool Sort_indexing(string a1_txt, string index1_txt)

�� {

�� bool b;

�� try

�� {

�� b = script.se.setInputFile(a1_txt );

�� if (!b)

�� {

�� appendText("Input File fails: " + a1_txt +"\n");

�� return false;

�� }

�� b = script.se.setOutputIndexFile(index1_txt);

�� if (!b)

�� {

�� appendText("Cannot create Index File" + index1_txt + "\n");

�� return false ;

�� }

�� b = script.se.indexing();

�� if (!b)

�� {

�� appendText("Indexing fails\n");

�� return false;

�� }

�� catch (Exception ex)

�� {

�� appendText(ex.ToString());

�� return false;

�� }

�� return true;

�� }

This function use an SortEngine object, script.se. It first assign the input image signature file, and assign output index file, then it creates an sorted index file by:

�� b = script.se.indexing();

The next step is to sort the signature file,

� �� private bool Sort_sortitngFile(string a1_txt, string index1_txt, string sort1_txt)

�� {

�� bool b;

�� try

�� {

�� b = script.se.setInputFile(a1_txt);

�� if (!b)

�� {

�� appendText("Input File fails: " + a1_txt + "\n");

�� return false;

�� }

�� b = script.se.setOutputIndexFile(index1_txt);

�� if (!b)

�� {

�� appendText("Cannot create Index File" + index1_txt + "\n");

�� return false;

�� }

�� b = script.se.setOutputLibraryFile(sort1_txt);

�� if (!b)

�� {

�� appendText("Cannot create library File" + sort1_txt + "\n");

�� return false;

�� }

�� b = script.se.sorting();

�� if (!b)

�� {

�� appendText("Sorting fails\n");

�� return false;

�� }

�� appendText("Output file : " + sort1_txt + "\n");

�� }

�� catch (Exception ex)

�� {

�� appendText(ex.ToString());

�� return false;

�� }

�� return true;

�� }

This function will first specify:

�input signature file � file containing image signautres;

�index file � file containing sorted index; and

output signature file � file to be produced with all signatures sorted.

The main statement in this function is:

�� b = script.se.sorting();

At this point, a sorted signature file is created and saved in the output signature file.

11.2�� Search

To make a search:

Select a Key image (Click the �3� Button in the Toolbar, see Figure 11.1).
Search.

There are several search options:

Binary Search/1:N Search (key vs. sort1.txt)
Binary Search/1:N Search (10%)
Binary Search/1:N Search (5%)
Binary Search/1:N Search (3%)
Binary Search/1:N Search (1%)

Assume you have 1 million images, the first command above will search 1 million images; the second command will search 100 thousand images; the third command will search 50 thousand images; the fourth command will search 30 thousand images; and the last command will search 10 thousand images. By definition, the Binary Search will only ignore the parts that are not likely to yield a match.

Assume you use Sequential Search (Chapter 4) for 1 million images; you will search through 1 million images. Then you use the �Binary Search/1:N Search (1%)� for the same 1 million images, provided the image signatures are sorted, you will search through only 10 thousand images. The difference is a factor of 100.

Binary Search is more restrictive than 1:N whole image matching; so the two image search results do not always match. In general, Binary Search generates less results than 1:N while image matching.

All of these menu items use the same functions, they only differ in one parameter. Click menu item, �Binary Search/1:N Search (10%)�,

� private void nSearch10ToolStripMenuItem_Click(object sender, EventArgs e)

�� {

�� int percent = 10;

�� bool bTmp = this.mainMenuToAPI.Sort_Match_1N_KeyVsSort1

�� (textBox1.Text,

�� dataDir + "sort1.txt",

�� dataDir + "b1.txt", percent);

�� if (!bTmp)

�� return;

�� guiResults.outputResults(mainMenuToAPI.script.results_1N, softwareDir);

�� guiResults.Button_First(false);��

�� }

This function first callS a function for 1:N matching, giving the results in mainMenuToAPI.script.results_1N. Then it passes this result to an image display object,�� guiResults. The last line simply displays the first image in the results:

guiResults.Button_First (false);

The function below does the 1:N matching:

�internal bool� Sort_Match_1N_KeyVsSort1

�� (string key, string sort1File, string b1File, int percent)

�� {

�� bool b = false;

�� if (!System.IO.File.Exists(key))

�� {

�� appendText("Please enter a valid Key!\n");

�� return false;

�� }

�� b = this.signature_Key_Signature(key);

�� if (!b)

�� {

�� appendText("Key Signature computation fails!\n");

�� return false;

�� }

��

�� try

�� {

�� script.results_1N = script.neuralFilterSorted .findMatch1N

(script.imageSignature, sort1File, b1File, percent);

�� }

�� catch (Exception e)

�� {

�� appendText("Quick 1:N Matching fails:\n" + e.ToString() + "\n");

�� return false;

�� }

��

�� if (script.results_1N == null)

�� {

�� appendText("NeuralFilter 1:N Matching fails!\n");

�� return false;

�� }

�� if (script.results_1N.getStatus())

�� {

�� appendText(script.results_1N.toString() + "\n");

�� appendText("" + script.results_1N.getNumberOfMatches() + " matches!\n");

�� }

�� else

�� {

�� appendText("No Match!\n");

�� return false;

�� }

��

�� return createFile(key, b1File);

�� }

The code here is similar to the sequential 1:N matching in the last chapter. The matching is done by:

�� script.results_1N = script.neuralFilterSorted .findMatch1N

(script.imageSignature, sort1File, b1File, percent);

The parameters are:

script.imageSignature � signature of a new image;

sort1File � signature file;

b1File � output file;

percent � percent used in the search.

12.�� Dynamic Library

In the last a few chapters, we introduced 1:N Match and N:N Match, where N is fixed. This chapter will introduce the Dynamic Library where N can be updated via:

insertion,
deletion, and
replacement.

The chapter project is located at:

c:\transapplet\80\imagefinder\.

12.1�� Dynamic Library Menu

Figure 12.1 and Figure 12.2 shows the dynamic library menu.

Figure 12.1� Dynamic Library Menu.

Figure 12.2� Dynamic Library/Binary Search Menu.

12.2�� Dynamic Library API

The following table lists the dynamic library functions.

Functions	Descriptions	Comments
int getLibraryID ()	Gets the Library ID (optional).
void setLibraryID (string x)	Sets the Library ID (optional).
bool load ()	Loads the default master library, lib1.txt.
bool load ( string fileName)	Loads master library specified by fileName.
bool load ( string fileName1,� string fileName2)	Loads two libraries specified by fileName1 and fileName2.
bool clear()	Clears the current library from RAM only.
bool backup ()	Saves current library to the default file, lib1_bk.txt.
bool backup (string fileName)	Saves current library to a back file.
bool addSignature (ImageSignature sig)	Adds a signature to a loaded image library in RAM.
bool deleteSignature (ImageSignature sig)	Deletes a signature from a loaded image library in RAM.
bool deleteSignature (string ID)	Deletes a signature from a loaded image library in RAM.	�
bool replaceSignature (ImageSignature sig)	Replaces a signature from a loaded image library in RAM.
bool mergeLibrary (string libFile1, string libFile2, string outputLib)	Merges two signature libraries into a single library. Input: �� string libFile1 �� string libFile2 �� string outputLib Output: �� A text file that contains the library of signatures from both input libraries.

12.3�� Creating A Master Library

Up to this point when we have done 1:N or N:N Matching, the N-images have been fixed. The Dynamic Library allows you to update the N-images, including inserting, deleting, and replacing signatures. We now create the master library. The master library file is lib1.txt.

Assuming we have created the signature file, a1.txt. To use the Dynamic Library, you have to create a library file lib1.txt. To create a library file, open a1.txt and save it to lib1.txt, or click:

��Library/Maintenance/Create lib1.txt from a1.txt�.

12.4�� Load Dynamic Library

The dynamic library uses file, lib1.txt, as a master library file. Before we can use the dynamic library, this file must be loaded into the dynamic library. To load the library file, click �Library/Maintenance/Load lib1.txt�.

Double click menu item �Library/Maintenance/Load lib1.txt� and you will see:

mainMenuToAPI.library_loadLib (dataDir +"lib1.txt" );

��

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool library_loadLib�� ( string lib1_txt)

�� {

�� if (script.imageLibrary == null )

�� {

�� appendText ( "Unable to find a library !\n");

�� return false;

�� }

�� try

�� {

�� bool b = script.imageLibrary.load� ( lib1_txt);

�� if ( b )

�� {

�� appendText ("Library 1 loaded!\n");

��

�� }

}

�� catch (Exception ee )

�� {

�� appendText ( ee.ToString () + "\n" );

�� return false;

�� }

�� return true;

�� }

The following statement loads the library:

bool b = script.imageLibrary.load� ( lib1_txt);

12.5�� Library M:N Matching

In an N:M Matching, a1.txt will be matched against the loaded library. Click menu item �Library/ N:M Matching (a1.txt vs. lib1.txt)� menu item to complete a N:M Match.

Double click menu item, Library/ N:M Matching (a1.txt vs. lib1.txt), and enter:

�� bool b = mainMenuToAPI.library_Matching_N_M

�� ( dataDir + "a1.txt",� dataDir + "b1.txt" );

��

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool library_Matching_N_M�� ( string a1_txt,�� string b1_txt)

�� {

�� if ( script.imageLibrary == null )

�� {

�� appendText ("Dynamic Library not available!\n");

�� return false;

�� }

�� bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

�� if (!b)

�� {

�� appendText ("Dynamic Library Assignment Fails!\n");

�� return false;

�� }

�� try

�� {

�� script.neuralFilter.findMatchNN ( a1_txt,� b1_txt);

�� }

�� catch (Exception e)

�� {

�� appendText (� e.ToString () + "\n");

�� return false;

�� }

�� return true;

�� }

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, �script.imageLibrary� is the dynamic library object.

The following statement assigns the dynamic library to the NeuralFilter object, �script.NeuralFilter�:

bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

�� if (!b)

�� {

�� appendText ("Dynamic Library Assignment Fails!\n");

�� return false;

�� }

Once the NeuralFilter object obtains this library, it will match images against this master library by the following statement:

�� script.neuralFilter.findMatchNN ( a1_txt,� b1_txt);

12.6�� Library 1:N Matching

Double click menu item �Library/ 1:N Matching (Key vs. lib1.txt)� and enter:

bool b = this.mainMenuToAPI.library_Matching_1_N�

�� (textBox1.Text� , dataDir + "b1.txt" );

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool library_Matching_1_N( string key, string b1File)

�� {

if ( script.imageLibrary == null )

�� {

�� appendText ("Dynamic Library not available!\n");

�� return false;

�� }

�� bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

�� if (!b)

�� {

�� appendText ("Dynamic Library Assignment Fails!\n");

�� return false;

�� }

b = false;

�� if ( ! System.IO .File .Exists (key) )

�� {

�� appendText ( "Please enter a valid Key!\n");

�� return false;

�� }

�� b = this.signature_Key_Signature (key);

�� if ( ! b )

�� {

�� appendText ( "Key Signature computation fails!\n");

�� return false;

�� }

�� try

�� {

�� script.results_1N = script.neuralFilter.findMatch1N

�( script.imageSignature);

�� }

�� catch (Exception e)

�� {

�� appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

�� return false;

�� }

�� if ( script.results_1N == null )

�� {

�� appendText ( "NeuralFilter 1:N Matching fails!\n" );

�� return false;

�� }

�� if ( script.results_1N.getStatus () )

�� {

�� setText ( script.results_1N.toString () + "\n");

�� appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

�� }

�� else

�� {

�� appendText ( "No Match!\n");

�� return false;

�� }

}

The only difference between this 1:N Matching and the NeuralFilter 1:N Matching introduced earlier is the following statement:

bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

In a normal 1:N Matching, the neural filter gets the library from a1.txt. This statement assigns the dynamic library, �script.imageLibrary�, to the NeuralFilter object, �script.NeuralFilter�.

12.7�� Update Implementation

Double click menu item �Library/Maintenance/Add (Key)� and enter:

�� if ( ! System.IO .File .Exists (textBox1.Text� ) )

�� {

�� this.mainMenuToAPI.appendText�

("File does not exist: \n" + textBox1.Text + "\n");

�� return;

�� }

�� this.mainMenuToAPI.library_add (textBox1.Text� );��

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

public bool library_add ( string sImage)

�� {

�� script.imageSignature

�� = script.signatureFilter.getSignature (sImage);

�� if (script.imageSignature == null )

�� {

�� this.setLine ("Image Signature computation fails!");

�� return false;

�� }

�� if (script.imageSignature.getStatus () <= 0 )

�� {

�� this.setLine ("Image Signature computation fails!");

�� return false;

�� }

�� bool b =� script.imageLibrary.addSignature (script.imageSignature);

�� return b;

�� }

The script object contains all of the objects required for the ImageFinder project. In particular, �script.imageLibrary� is the dynamic library object. The first section of the code computes the image signature. The next section of code checks the validity of the signature. The following statement inserts the signature to the dynamic library:

bool b =� script.imageLibrary.addSignature (script.imageSignature);

The menu item Delete and the menu item Replace can be implemented in a similar fashion and the codes are provided in the project.

13.�� Image Variations

The chapter project is located at:

c:\transapplet\80\imagefinder\.

13.1�� Variation Signatures

��

Figure 13.1 A change beyond the tolerance of a single image signature.

Figure 13.2 Signature/Variation Signatures Menu.

Image Matching is done through something called Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature. A single image signature will take care of a certain amount of variation. To increase the variations, additional signatures will be added, which is the purpose of the Variation Signatures menu.

An image can be changed, for example, a rotation of 90 degrees to the left. As the image is changed, the signature will change accordingly. In general, this change will not be maintained inside a signature, i.e. a small variation allowed inside a single signature will not take care of large variations like 90 degree rotations.

The �Signature/Variation Signatures� menu item (Figure 13.2) provides a list of menu items to compute signatures. The net result is that an image will have multiple image signatures.

The �Signature/Variation Image� menu item (Figure 13.3) will compute various variations of images from the key image and will deposit these images into �.\tmp\� folder.� To use this menu item, you must first select a key image. This allows a user to see what the variations are.

Figure 13.3 Signature/Variation Images Menu.

This menu supports the following variations:

Signatures/Variation Signatures/Scale (100, 90, 80, �)

�� This menu item generates 7 image signatures:

�� 100%, 90%, 80%, 70%, 60%, 50%, 40%.

�� Here 90% is an image that has 5% border region removed.��

Signatures/ Variation Signatures/Rotation (90, 180, 270)

�� This menu item generates 4 image signatures:

�� 0, 90, 180, 270.

�� Here 90 is an image that has a 90 degree rotation.

Signatures/ Variation Signatures/Flip (X, Y,� XY)

�� This menu item generates 4 image signatures:

�� 0, X, Y, XY.

�� Here X is an image that has flipped around x axis.

Signatures/ Variation Signatures/Flip Rotation (16 variations)

�� This menu item generates 16 image signatures:

�� {0, 90, 180, 270} ^ {0, X, Y, XY}.

�� Here (90, X) is an image that has 90 degree rotation and flipped around x axis.

Signatures/ Variation Signatures/Stretch (4 variations)

�� This menu item generates 5 image signatures:

�� (0, 0), (120%, 100%), (80%, 100%), (100%, 120%), (100%, 80%).

�� Here (120%, 100%) are stretch percentages along the x and y directions.

��

Signatures/ Variation Signatures/Resize�

�� This menu item generates 5 image signatures:

�� 100, 150, 200, 300, 400.

�� Here 150 increases the image size by 50% in terms of pixel counts in each direction.

The implementation for menu item, Signatures/Variation Signatures/Scale (100, 90, 80, �), is:

� �� script.signatureFilterSym .getKeywordLibrary_Scale

�� (f.guiSource.libraryList.imageAbsolutePath, f.guiSource.libraryList.imageID,

�� f.guiSource.libraryList.imageX, f.guiSource.libraryList.imageY,

�� f.guiSource.libraryList.imageW, f.guiSource.libraryList.imageH,

�� outputFile,

�� f.guiSource.libraryList.key0, f.guiSource.libraryList.key1,

�� f.guiSource.libraryList.key2, f.guiSource.libraryList.key3,

�� f.guiSource.libraryList.key4, f.guiSource.libraryList.key5,

�� f.guiSource.libraryList.key6, f.guiSource.libraryList.key7,

�� f.guiSource.libraryList.key8, f.guiSource.libraryList.key9,

�� f.mainMenuToAPI.script.tranAppletPara .signatureFilter_LibraryScaleType

�� );

13.2�� Segments

A collection of features is grouped into a signature. One image has one signature. The signature matches whole images against whole images, i.e. it does not look inside the image.

An image can be many logical segments. As the focus changes from a whole image to an image segment inside the image (Figure 13.3), the signature will change accordingly. In general, this change will not be maintained inside a basic signature.

The �Signature/Segment Signatures� menu item attempts to partially address this image segmentation problem. The menu item will look at some segments inside an image. On average, it will look at six segments.

The �Signature/Segment Images� menu item (Figure 13.2) computes signatures for image segments. The net result is that an image will have multiple image signatures, one for each segment.

The �Signature/Segment Images� menu item (Figure 13.3) will compute segments from the key image and will deposit them into �.\tmp\� folder.� To use this menu item, you must first select a key image. This allows a user to see what the segments are.

��

Figure 13.4 Image Segmentations.

The object, imgSigFilterSeg, used for computing segment signatures, is an instance of the following class:

Attrasoft.TransApplet80.ImageSignatureFilterSegment80.ImageSignatureFilterSegment80� imgSigFilterSeg;

The implementation of the menu item, Signature/Segment Signatures, is:

imgSigFilterSeg.getSegmentLibrary_Modelbased(sourceFolder, outputFile);

This function will convert images in parameter, sourceFolder, into image signatures and save them to a file, outputFile.

14.�� Sub-Image Search

The chapter project is located at:

c:\transapplet80\imagefinder\.

The sub-image matching requires a different set of parameters. There are two different settings:

Setting/Search Sub-Images

Setting/Search Whole Images

To match sub-images, click menu item, �Setting/Search Sub-Images�. To switch back to the original setting for whole images, click: �Setting/Search Whole Images�.

Figure 14.1 Setting Change.

14.1�� Sub-Images and Variations

Starting from this chapter, we will search inside an image. We will deal with the problems of:

Sub-image
Variations.

The number of sub-images is very large, measured by exponential functions of the number of pixels. The number of variations is also very large:

�

Translated segments;
Rotated segments;
Scaled segments;
Rotated and Scaled segments;
Brighter or Darker segments;

� ...

All of the above simultaneously.

We will not use image signatures because of the large amount of variations. This will significantly slow down the search speed.

Because the variations for searching inside an image are huge, we might impose some limits on where to search. There are several approaches:

Automatic Segmentation;
Enumerative.

Automatic segmentation deploys certain algorithms to look at a small number of segment locations inside the image. There are two menu items to address this approach (Chapter 13):

The �Signature/Segment Signatures� menu item (Figure 13.2) computes signatures for a selected number of image segments. The net result is that an image will have multiple image signatures, one for each segment.

The �Signature/Segment Image� menu item (Figure 13.3) will compute segment images from the key image and will deposit them into �.\tmp\� folder.� To use this menu item, you must first select a key image. This allows a user to see what the segments are.

The advantages of automatic segmentation are that it will dramatically reduce the number of search locations. The disadvantages of segmentation algorithms are that it will not be able to address some human cropped images which cut right through the middle of a segment.

The enumerative approaches are not restricted by the automatic segmentation algorithm; however, the variations for enumerative sub-image searching are huge; some sort of limits on where to search will have to be imposed.

Figure 14.2 Two parameters.

Figure 14.3 Sub-Image Menu.

14.2�� Search Process

In this chapter, we will use Enumerative search by imposing these two conditions:

Translation invariance only;
No scale, rotation, �, invariance.

In each search, there are two parameters N x M (Figure 14.2), which specify an array of sub-images to search. The software will search via an �N by M� grid for each unknown image for the specified key image. The key will not be rotated or scaled.

Initially, you will need three steps for an image search:

1. Select a Key;

2. Select a folder;

3. Search.

To see the results:

1. Click the Results Home button (Figure 14.3);

2. Click the Results �=>� button (Figure 14.3).

Starting from the second search, you do not have to select the folder again because it is already selected; each search has only two steps:

1. Select a Key;

3. Search;

1. Select a Key;

3. Search;

�

Note:

Step 1, selecting a Key can be completed by dragging & dropping an image to the first text box;

Step 2, select a folder, which can be completed by dragging & dropping a folder to the second text box.

14.3�� Learn by Example

��

Figure 14.4�� 1:1 Matching.

The first example (see Figure 14.4) is to match the key,

.\ ex_object1\key.jpg,

against the image,

.\ ex_object1\CIMG4643_mid_shelf1.JPG.

The example will take three clicks:

1. Click the �Key� button in the Toolbar, and select .\ ex_object1\ key.jpg. You can also drag & drop .\ ex_object1\key.jpg to the first text box.

Figure 14.5 Step 1, selecting a key. After selecting a key image, the image will be displayed in the Key Panel.

2. Click the �Source� button in the Toolbar and select the folder, .\ ex_object1\.

You can also drag & drop the folder, .\ ex_object1\, to the second text box; the use �=>� button to select image, .\ ex_object1\CIMG4643_mid_shelf1.JP, see Figure 14.6.

Figure 14.6� Step 2, selecting a folder. After selecting a folder, the first image will be displayed in the Source Panel (middle picture box). The list of image files will be displayed in the text window. The Home button goes to the first image; the �=>� button goes to the next image; and the �<=�� button� goes to the previous image. All sub-folders will be included.

Figure 14.7� Search the N x M grid of sub-images. For example, setting them to 30 x 5 will result in searching 150 sub-images.

3. Before we make a search, you will specify how to search via an �N by M� grid, i.e. how many sub-images we will look at for the specified key image. The default value is a 10 by 10 grid.

To make a 1:1 search,

� Enter 30 and 5 in Figure 14.7.

� Click the �Sub-Image/1:1 Matching� menu item in Figure 14.3.

You will get:

ID�� Name�� Path�� Score�� X�� Y�� W�� H

CIMG4643_mid_shelf1.JPG_146� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 84�� 366�� 34�� 73�� 184

CIMG4643_mid_shelf1.JPG_134� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 84�� 197�� 34�� 73�� 184

CIMG4643_mid_shelf1.JPG_123� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 83�� 42�� 34�� 73�� 184

CIMG4643_mid_shelf1.JPG_110� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 83�� 282�� 25�� 73�� 184

CIMG4643_mid_shelf1.JPG_117� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 81�� 380�� 25�� 73�� 184

CIMG4643_mid_shelf1.JPG_100� CIMG4643_mid_shelf1.JPG� .\ex_object1\�� 80�� 141�� 25�� 73�� 184

Note, if you do not get this, click menu item, �Setting/Search Sub-Images� and then try again.

Figure 14.8 Search the selected folder. Images that matched the �key� will be displayed in the Results Panel. Use the Home, =>, and <=buttons to see matched images. The Home button goes to the first image; the �=>� button goes to the next image; and the �<=�� button� goes to the previous image.

Figure 14.9 The matched score and coordinates, (x, y, w, h), are given in a text file: �.\data\c.txt�.

To see the first matched image, click the Home button in the Results Panel (see Figure 14.8). To see the next matched image, click the �=>� button in the Results Panel. The key and the matched images are displayed side-by-side.

14.4�� Implementation

The basic idea is to cut a segment of a source image, which has the same size as the key image. These two images are matched via image signatures. This process is repeated for various locations.

The basic function is:

r80 = script.smallSegment.findmatch11_NeuralFilter_LargeImage

(b1, b2, theID, thePath, theName, f.pictureBox4);

Where:

findmatch11_NeuralFilter_LargeImage (�) is the function;
B1 is the key bitmap;
B2 is the source bitmap;
theID is the image ID used in the output file;
thePath is the image Path used in the output file;
f.pictureBox4 displays each sub-image of B2 that is matched against Bitmap, B1.

15. ��Fast Sub-Image Matching

The sub-image matching requires a different set of parameters. There are two different settings:

Setting/Search Sub-Images

Setting/Search Whole Images

To match sub-images, click menu item, �Setting/Search Sub Images�. To switch back to the original setting for whole images, click: �Setting/Search Whole Images�.

The objective for the last chapter menu is to find a cropped segment at all cost, provided there are no scaling factors. This can take a long time if the N x M search grid is set to be very large.

The first objective of this chapter is to find a cropped segment much faster.

The second objective is to add scale invariance and rotation invariance.

Figure 15.1 �Fast� menu items.

15.1�� Fast Menu

The objective of the �Fast� menu items is to find a cropped segment from an image within 1 second.

The �Fast� menu items provide a much faster search than the menu items of the last chapter, but these menu items are less accurate than those in the last chapter. It is a trade-off between accuracy and computation time. Figure 15.1 shows the menu items under this menu.

Example. We will match (see Figure 15.4 from last chapter) the key,

.\ ex_object1\key.jpg,

against the image,

.\ ex_object1\CIMG4643_mid_shelf1.JPG.

Figure 15.2 Fast match results.

Before starting this example, click menu item, �Setting/Search Sub Images�.

The example will take three clicks:

1. Click the �Key� button, and select .\ ex_object1\key.jpg. You can also drag & drop .\ex_object1\key.jpg to the first text box.

2. Click the �Source� button and select the folder, .\ ex_object1\. You can also drag & drop the folder, .\ ex_object1\, to the second text box.

3. Fast Search.

Click Fast/1:1 Matching and see the result in Figure 15.2.

The menu items are given below:

Sub Image/Fast 1:1

Use �Sub Image/Fast 1:1� to make a 1:1 Matching.

Sub Image/Fast 1:N (Same size)

Use �Sub Image/Fast 1:N (Same size)� to make a 1:N Matching; it will search through all images in a folder for the specified cropped image. This command is faster than �Sub Image/1:N (Different size)�, but it requires all images in the search folder to have the same pixel size.

Sub Image/Fast 1:N (Different size)

Use �Sub Image/Fast 1:N (Different size)� to make a 1:N Matching; it will search through all images in a folder for the specified cropped image. This command is slower than �Sub Image/1:N (Same size)�, but it does not requires all images in the search folder to have the same pixel size.

To implement these menu items, the object is:

f.mainMenuToAPI.script.neuralNetFilter

This object will need to learn the sample bitmap, b:

try

�� {

�� f.mainMenuToAPI.script.neuralNetFilter.setSymmetry(3);

�� f.mainMenuToAPI.script.neuralNetFilter.train(b, right.Width, right.Height);

�� return true;

��

�� }

�� catch (Exception ex)

�� {

�� f.mainMenuToAPI.appendText(ex.ToString () + "\n");

�� return false;

�� }

After that, it will make a fast 1:N matching:

f.mainMenuToAPI.script.results_1N

�� = f.mainMenuToAPI.script.neuralnet2.findMatch1N

�� (f.guiSource .libraryList .imageAbsolutePath, matchingCode);

15.2�� NeuralNet Filter API

The following table lists the NeuralNet Filter functions.

Functions	Descriptions
bool train (Bitmap img) bool train (string sPath) bool train (Bitmap img, int x, int y, int w, int h) bool train (string sPath, int x, int y, int w, int h)	Trains the neural net, either use whole image or image segement (x, y, w, h).
bool train (Bitmap �b, int w, int h)	Place an image b into a white image w pixels by h pixels, the train the neural net.
bool retrain (Bitmap img) bool retrain (string sPath) bool retrain (Bitmap img, int x, int y, int w, int h) bool retrain (string sPath, int x, int y, int w, int h)	Retrains the neural net.
Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11 ( Bitmap img1); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11 ( string path1); Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N ( string [] fileList);	Makes a 1:1 and 1:N Matching.
bool findMatchNN (string [] fileList, string c1File) bool findMatchNM (string [] keyList, string [] libraryList, string c1File);	Makes a N:N Matching and N:M Matching.

16.�� Video Input

Video recognition largely mirrors the image recognition. Identify a whole video is similar to identify a whole image. This chapter describes how to enter video data to the VideoFinderLite. The input videos are further divided into:

Key
Search Source

A Key-folder specifies the sample videos. A Search-Folder specifies the Search Source. The chapter project is in �.\videofinderlite�. The video input programming mirrors the image input.

16.1�� Class Library Name

The class library is:

Attrasoft.TransApplet80.InputModuleVideo.

The main class in this library will be:

Attrasoft.TransApplet80.InputModuleVideo .InputModuleVideo.

The objects for key and source are:

internal Attrasoft.TransApplet80.InputModuleVideo .InputModuleVideo� inputModuleKey;

internal Attrasoft.TransApplet80.InputModuleVideo.InputModuleVideo inputModuleSource;

The interface is:

public interface I_ InputModule80

�� {

�� Attrasoft .TransApplet80 .InputList80 .InputList� getList( string);��

�� }

The input is a folder path. Output is an object of the following class:

public class VideoInputList

�� {

�� public int status = 0;

��

�� public string[] imageAbsoultePath = null;

�� public string[] VideoID = null;

�� public string[] VideoName = null;

�� public string[] VideoPath = null;

�� public string[] key0 = null;

�� public string[] key1 = null;

�� public string[] key2 = null;

�� public string[] key3 = null;

�� public string[] key4 = null;

�� }

16.2�� Class Library Overview

The functions in classes are listed in the following table.

Function	Description
Attrasoft .TransApplet80 . VideoInputList. VideoInputList� getList( string folder)	After input folder is specified, get the output object.

16.3�� Set Up the Chapter Project

Figure 16.1� Chapter Project.

The project has the following controls:

��

Key Button, Key Text Box

Use the �Key� button to display the selected key video folder.

Source Button, Source Text Box

Use the �Source� button to select a search video folder.

Clear Button

Use the �Clear� button to clear the rich text box.

Rich Text Box

�� Use Rich Text Box to display messages.

Key Picture Box

Use the Key Picture Box to display key images.

Source Picture Box

Use the Source Picture Box to display images in the search source.

OpenFileDialog

�� Use the OpenFileDialog to select a key image.

FolderBrowserDialog

�� Use the FolderBrowserDialog to select a source path.

After adding these controls, the form will look like Figure 16.1.

16.5�� Key Segment

We can double click the Key Segment buttons to complete the programming:

public bool Button_Keys()

�� {

�� try

�� {

�� if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

�� return false;

�� string cDir = f.folderBrowserDialog1.SelectedPath;

�� f.textBox1.Text = cDir;

�� f.guiKey.setInput(f.inputModuleKey.getList(cDir));

�� if (!f.guiKey.getStatus())

�� {

�� f.appendText("Data Source fails!\n");

�� return false;

�� }

�� f.guiKey.setLists();

�� }

�� catch (Exception e)

�� {

�� f.appendText(e.ToString() + "\n");

�� return false;

�� }

�� return this.f.guiKey.Button_First(false);

�� }

The first section of code opens a file dialog so the user can select a file:

�� if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

�� return false;

�� string cDir = f.folderBrowserDialog1.SelectedPath;

The next section enters the selected folder to the InputModuleKey object and gets the string list of the image paths:

�� f.inputModuleKey.getList(cDir);

Then it passes this list to display object:

�� f.guiKey.setInput(f.inputModuleKey.getList(cDir));

The last section displays the first images:

�� f.guiKey.setLists();

�� f.guiKey.Button_First(false);

The video input process is basically the same as image input process.

16.6�� Search Source Button

The search source button is very similar to the Key button. The code is in the project. The explanation is identical to image source input.

17.�� Whole Video Recognition

Figure 17.1 Step 1, Select Library (Source button).

We will continue the .\VideoFinderLite project from the last chapter.

17.1�� Project �Introduction

Initially, the operation will consist of the following six steps for video matching:

��

�� 1. Select Library ("1..." button)

�� 2. Get Library fingerprints ("2..." button)

�� 3. Load Library ("3..." button)

�� 4. Select Key ("4..." button)

�� 5. Create Key fingerprint ("5..." button)

�� 6. 1:N Matching ("6..." button)

Later, you do not have to create a library each time:

��

�� 3. Load Library ("3..." button)

�� 4. Select Key ("4..." button)

�� 5. Create Key fingerprint ("5..." button)

�� 6. 1:N Matching ("6..." button)

��

The procedure is:

1. Select Library ("1..." button)

A video clip is converted into a video clip fingerprint. A set of video clip fingerprints forms a library. A set of video clips will be placed in a directory, and the directory path will be entered into the VideoFinderLite, for example, �c:\abc\def\�.

This step selects a video clip directory. Dragging & dropping a folder into the second text box will complete this step. This step can also be implemented by clicking:

� �Search/1. Select A Folder� menu item, OR

� "1. Folder" button.

The video clips in the selected directory will be displayed in the second picture box (labeled �Lib(rary)�). Use the �Play� buttons to play the video clip. Make sure to use �Stop� to stop the video clip.

Use the �Home� button to select the First video in the directory; use the �=>� button to select the next video clip; and use the �<-� button to select the previous video clip.

2. Create Fingerprint Library ("2..." button)

This step converts all video clips in the selected directory into fingerprints and saves them to the file, �a1.txt�. This step is implemented by clicking:

o �Search/2. Create Fingerprint Library (.\a1.txt)� menu item, OR

o �2. Library� button.

Figure 17.2 Step 3, Select Key (Key button).

3. Load Library ("3..." button)

Before matching, you will need to select a library and load it to the VideoFinderLite. This step is implemented by clicking:

o �Search/3. Load Fingerprint Library� menu item, OR

o �3. Load� menu item.

4. Select Key ("4..." button)

A key is a newly obtained video clip. The video clip is converted into a fingerprint. To select a key, drag & drop a folder into the first text box and select a file in this key folder. Again, a key is a video in a folder, a folder path is the parameter to be entered, not a video file path. Once the folder is entered, you will use � and � buttons to select a file. This step can also be implemented by clicking:

� �Search/4. Select Key� menu item, OR

� �4. Key� button.

The video clips in the selected directory will be displayed in the first picture box (labeled �Key�). Use the �Play� buttons to play the video clip. Make sure to use �Stop� to stop the video clip.

Use the �Home� button to select the First video in the directory; use the � button to select the next video clip; and use the � button to select the previous video clip.

5. Create Key fingerprint ("5..." button)

This step converts a key video clip into a fingerprint. This step is implemented by clicking:

� �Search/5. Key Fingerprinting� menu item, OR

� �5. Finprt� button.

Figure 17.3 Step 6, 1:N Matching.

6. 1:N Matching ("6..." button)

�1:N Matching� (read as 1 to N matching) compares �1� (i.e., the new video clip) with �N� (i.e., the previously stored video clips).� This step is implemented by clicking:

� �Search/6. Search� menu item, OR

� �Search� button.

The 1:N matching results are also displayed in the third picture box (labeled �Results�). Use the �Play� buttons to play the video clip. Make sure to use �Stop� to stop the video clip.

Use the �Home� button to select the First video in the directory; use the �=>� button to select the next video clip; and use the �<=� button to select the previous video clip.

17.2 ��Example

In this section, we will use the �Friends� Video Clips. The stored data used in this section is in folder,

.\ test0 \

where �.\� is the folder where the software is installed. There are five video clips in this folder. The 1:N match compares a newly captured fingerprint with all video fingerprints in a1.txt.

��

1. Select Library ("1..." button)

Drag & drop �.\test0\� folder into the second text box.

2. Get Library fingerprints ("2..." button)

Click the �2 �� button; the results will be saved to �.\data\ a1.txt� automatically.

3. Load Library ("3..." button)

Click the �3 �� button to load the library.

4. Select Key ("4..." button)

Drag & drop �.\test0\� folder into the first text box. By default, the first file in this folder is chosen; you can select any file in this folder as a search key.

5. Create Key fingerprint ("5..." button)

Click the �5 �� button; the key fingerprint will be saved internally. You can also save the results by clicking Match/Key Fingerprinting (key.txt); followed by clicking �Match/1:N Matching (Key.txt : Library)� in step 6.

6. 1:N Matching ("6..." button)

Click the �6 �� button to make a 1:N match, see Figure 17.4.

17.3�� Implementation

Out of the six steps:

��

�� 1. Select Library ("1..." button)

�� 2. Get Library fingerprints ("2..." button)

�� 3. Load Library ("3..." button)

�� 4. Select Key ("4..." button)

�� 5. Create Key fingerprint ("5..." button)

�� 6. 1:N Matching ("6..." button)

We have covered step 1 and step 4.

The Input objects are:

internal Attrasoft.TransApplet80.InputModuleVideo .InputModuleVideo� inputModuleKey;

internal Attrasoft.TransApplet80.InputModuleVideo.InputModuleVideo inputModuleSource;

The display objects are:

�� internal Attrasoft.TransApplet80.VideoGUI.GUI_Key guiKey;

�� internal Attrasoft.TransApplet80.VideoGUI.GUI_Key guiSource;

�� internal Attrasoft.TransApplet80.VideoGUI.GUI_Results guiResults;

��

All computations are done by the following objects:

�� internal Attrasoft.TransApplet80.VideoEntryPoint80.VideoEntryPoint80 script;

Get Library fingerprints ("2..." button)

�

�f.script.createFingerprintLibrary

�� (f.gui.imageAbsoultePath, f.dataDir + "a1.txt");

Load Library ("3..." button)

f.script.loadLibrary(sTmp);

Create Key fingerprint ("5..." button)

�� f.script.getKeyFingerprint(sTmp);

1:N Matching ("6..." button)

�� f.script.videoResult = f.script.findMatch1N(f.script.videoFingerPrint);

18.�� Video Frame Recognition

A video frame is an image, so matching video frames is similar to whole image matching. A key video will produce an image-frame signature file, key.txt. Videos in a source will produce a signature file, lib1.txt.

A 1:N video frame matching, i.e. matching a video frame against all video frames of a set of videos, �is converted into a N:M image matching.

The chapter project is in .\videofinder folder.

Figure 18.1� VideoFinder.

18.1�� How to Use VideoFinder

Initially, you will need the following steps for video matching:

0. Initialization (Only Once after starting the software)

1. Select Library (Source Button)

2. Get Library Signatures (source.txt)

3. Select Key (Key Button)

4. Get Key Signatures (key.txt)

5. Create Library File (lib1.txt)

6. Load Library (lib1.txt)

7. 1:N Matching (Match/1:N Match menu item)

8. First Results (2nd F Button)

9. Next Results (2nd > Button)

Later, you do not have to create a library each time:

0. Initialization (Only Once After Starting the Software)

6. Load Library (lib1.txt)

3. Select Key (Key Button)

4. Get Key Signatures (key.txt)

7. 1:N Matching (Match/1:N Match menu item)

3. Select Key (Key Button)

4. Get Key Signatures (key.txt)

7. 1:N Matching (Match/1:N Match menu item)

�

0. �Initialization (Only Once After Starting the Software)

This step initializes the software. This step is implemented by clicking:

� �Beginner/0. Initialization (Only Once)� menu item, OR

� �Init� button.

1. Select Library (Source Button)

A video clip is converted into a video clip signature. A set of video clip signatures forms a library. A set of video clips will be placed in a directory and the directory path (for example, c:\abc\def\) will be entered into the VideoFinder.

This step selects a video clip directory. This step is implemented by clicking:

� �Beginner/1. Select Library (Source Button)� menu item, OR

� �Source� button.

The video clips in the selected directory will be displayed in the second picture box.

Use the �Play� and �Stop� buttons to play the video clip.

Use the �F� button to select the First video in the directory; use the �>� button to select the next video clip; and use the �<� button to select the previous video clip.

2. Get Library Signatures (source.txt)

This step converts all video clips in the selected directory into signatures and saves them to the file, �source.txt�. This step is implemented by clicking:

� �Beginner/2. Get Library Signatures (source.txt)� menu item, OR

� �Signature/Library Signatures� menu item.

3. Select Key (Key Button)

A key is a newly obtained video clip. The video clip is converted into a signature. The key (for example, c:\abc\def\ghi.avi) will be entered into the VideoFinder.

This step selects a key. This step is implemented by clicking:

� �Beginner/3. Select Key (Key Button)� menu item, OR

� �Key� button.

The selected key will be displayed in the first picture box. Use the �Play� and �Stop� buttons to play the video clip.

4. Get Key Signatures (key.txt)

This step converts a key video clip into a signature and saves them to the file, �key.txt�. This step is implemented by clicking:

� �Beginner/4. Get Key Signatures (key.txt)� menu item, OR

� �Signature/Key Signature� menu item.

5. Create Library File (lib1.txt)

Before matching, you will need to select a library and load it to the VideoFinder. The library to be loaded is fixed to the file, lib1.txt. We have created a library file, �source.txt�; now we will simply copy it to the loading-library file, �lib1.txt�.

This step is implemented by clicking:

� �Beginner/5. Create Library File (lib1.txt)� menu item, OR

� �Library/Copy source.txt to lib1.txt� menu item.

6. Load Library (lib1.txt)

Before matching, you will need to select a library and load it to the VideoFinder. This step is implemented by clicking:

� �Beginner/6. Load Library (lib1.txt)� menu item, OR

� �Library/Load (lib1.txt)� menu item.

7. 1:N Matching� (Match/1:N Match menu item)

�1:N Matching� (read as 1 to N matching) compares �1� (i.e., the new video clip) with �N� (i.e., the previously stored video clips).� This step is implemented by clicking:

� �Beginner/7. 1:N Matching� menu item, OR

� �Match/1:N Matching� menu item.

8. First Results (2^nd F Button)

The 1:N matching results are also displayed in the second picture box. To select the first retrieved video clip, use:

� �Beginner/8. First Results (2nd F Button)� menu item, OR

� second �F� button.

Use the �Play� and �Stop� buttons to play the video clip.

9. Next Results (2^nd > Button)

The 1:N matching results are also displayed in the third picture box. To select the next retrieved video clip, use:

�

� �Beginner/9. Next Results (2nd > Button)� menu item, OR

� Second �>� button.

Use the �Play� and �Stop� buttons to play the video clip.

18.2�� Implementation

1. Select Library (Source Button)

The implementation is similar to the last chapter.

2. Get Library Signatures (source.txt)

The basic implementation algorithm is:

loop through all videos

{

For each video, loop through all frame

{

�� For each frame, get an image signatures

� �� Add image signature to a source library.

}

Backup the in-memory library to a1.txt.

3. Select Key (Key Button)

The implementation is similar to the last chapter.

4. Get Key Signatures (key.txt)

The basic implementation is:

Loop through all frames

{

�� For each frame, get an image signatures

� �� Add image signature to a key library

}

Backup the in-memory library to key.txt.

5. Create Library File (lib1.txt)

Copy a1.txt to lib1.txt.

6. Load Library (lib1.txt)

Load library file, lib1.txt.

7. 1:N Matching (Match/1:N Match menu item)

Make an N:M matching with key.txt and lib1.txt.

The details of the implementation are given in the project.

19.�� Video Object Recognition

A video frame is an image, so matching an object within video frames is similar to sub-image matching. The chapter project is in .\framefinder folder.

19.1 ��FrameFinder Functions

Attrasoft FrameFinder searches video clips for an object specified by a key image. Attrasoft FrameFinder also searches live video for an object specified by a key image.

Figure 19.1 Attrasoft FrameFinder searches video clips for an object specified by a key image.

19.2�� Beginner Menu

You can operate the software from the �Beginner� menu alone.

Figure 19.2 The �Beginner� Menu and its menu items.

The matching between a sample image, called a key, and previously stored video clips is implemented directly without signatures.�

You can have as many keys as you want. Before matching, you will need to select a key using the �Key� button and select a folder containing video clips using the �Source� button; then, you are ready to search.

19.3�� User Interface

The FrameFinder has three video panels: �Key�, �Source�, and �Results�.

Key:

A key is an image, which specifies an object. A key is selected via the �Key� button.

Source:�

A Source is a directory of video clips. Source videos are selected via the �Source� button. Use the �Play� and the �Stop� button below the label �Source� to play a video clip. Use the �F�, �>�, and �<� button to select a video clip in the library: here �F� stands for First; �>� stands for Next; and �<� stands for previous.

�

Results:

Results consist of two sections: matched videos and matched frames. Use the �Play� and the �Stop� button below the label �Results� to play the matched video clip. Use the �VF�, �V>�, and �V<� button (Video First, Video Next, Video Previous) to select a video clip in the matched collections. Use the �IF�, �I>�, and �I<� button (Image First, Image Next, Image Previous) to display matched image frames.

Figure 19.3 Three Video Panels: �Key�, �Source�, and �Results�. The Key image is labeled by the �Key� label.

Figure 19.4 Three Video Panels: �Key�, �Source�, and �Results�. The Source video panel is labeled by the �Source� label.

19.4�� Implementation

The basic implementation is:

�

loop through all videos in a search folder

{

For each video, loop through all frame

{

�� For each frame, get an image

� �� 1:1 matching between the key image and the current images.

}

�

The details of the implementation are given in the project.

20.�� Image Processing Concept

20.1�� Filters

For the ImageFinder, the Image Matching is divided into:

Image Preprocessing
Image Processing
Normalization
Signatures
Signature Recognition
Image Segment Recognition

One or more filters further implement each step:

Image Preprocessing

�� Preprocessing Filter

Image Processing

�� Edge Filter

�� Threshold Filter

�� Clean Up Filter

Normalization

�� Reduction Filter

Signature

�� Signature Filter

Feature Recognition

�� Unsupervised Filter

�� BioFilter

�� NeuralFilter

Image Segment Recognition

�� Neural Net Filter

�� Neural Net II Filter

The image matching software design is implemented to push an image through all these filters and at the end of pushing, the matching results will be obtained.

20.2�� Image Preprocessing & Processing

Attrasoft ImageFinder learns an image in a way similar to human eyes:

Ignore the background;
Focus on an object in the image.

The Image Preprocessing is implemented by one filter, the Image Preprocessing Filter. Three filters implement image processing:

� Edge Filters;

� Threshold Filters; and

� Clean-Up Filters.

The Image Preprocessing and the Image Processing processes are not unique; there are many options available. Some are better than others.

The principle of choosing the Image Preprocessing and the Image Processing filters is to make the sample objects stand out, otherwise, change the options.

Image Preprocessing has the following functions:

Cut off the border areas;

Impose a mask;

Speed up the computation;

Skip the empty border area;

Image Processing has the following filters:

The Edge Filters attempt to exaggerate the main features a user is looking for.

The Threshold Filters attempt to suppress the background.

The Clean-Up Filters will smooth the resulting image to reduce recognition error.

20.3�� Normalization

The Normalization Filter will prepare the images for the underlying image matching engine, which is a 100x100 array of neurons.

While any size of neural net can be used, when coming to a particular application, a decision has to be made. The ImageFinder uses 6 different sizes:

10,000 neurons,
8,100 neurons,
6,400 neurons,
4,900 neurons, or
2,500 neurons.

Later in the multi-layered design, the number of neurons can be much larger. The Normalization Filter, also called Reduction Filter, will connect the images to various neural networks.

20.4�� Neural Filter and Neural Net Filter

The ability to learn is a fundamental trait of intelligence. Neural Nets are deployed in both Signature Matching and Segment Matching. Neural Nets can learn from examples. There are two basic phases in image recognition:

� Training; and

� Recognition.

In the training phase, data is imposed upon a neural network to force the network to remember the pattern of training data. A neural network can remember the training data pattern by adjusting its internal synaptic connections.

In the recognition phase, the neural network, based on its internal synaptic connections, will determine whether the newly captured image matches the sample image.

Neural Filter is used for whole image matching. It is trained with 10,000 pairs of random images.

Neural Net Filter is used for the sub-image matching.

20.5�� Parameter Object

Implementing image matching software via the TransApplet means pushing an image through these many filters:

1. Preprocessing Filter

2. Edge Filter

3. Threshold Filter

4. Clean-Up Filter

5. Reduction Filter

6. Signature Filter

7. Unsupervised Filter

8. BioFilter

9. NeuralFilter

10. Neural Net Filter

11. Neural Net Filter II

Each filter is an object (See their API in earlier chapters). Each filter will have many parameters and these filters together can have many parameters. It will be easier, from the programming point of view, to group all of the objects together and to group all of the parameters together into an object.

21.�� Image Preprocessing

To get a good image matching, two things are important:

To control the number of variations.
To make the area of interest stands out and to make the rest as little as possible.

The chapter project is in �.\ transapplet80.chap21�.

Figure 21.1 Chapter project.

Figure 21.2 Image Preprocessing Parameters.

21.1�� Image Preprocessing Filter

Start the chapter project, you will see Figure 21.1.

To set the Image Preprocessing Filter parameters, click the Parameter button in Figure 21.1; then click the Parameter button again, and you will see Figure 21.2. You will set the Image Preprocessing parameters on Figure 21.2. The �OK� button in Figure 21.2 enters the parameters to the Preprocessing filter.

To see the effect of the Image Preprocessing and Image Processing filters, you have to select a key image via the Key button. The image will be displayed on the left. Then click the �Preprocessing Image� button to apply the preprocessing filter and see the processed image on the right.

21.2�� PreProcessing API

The Image Preprocessing interface is:

public interface I_ImagePreProcessing80

�� {

�� int getMaskX();

�� void setMaskX(int x);

�� int getMaskY();

�� void setMaskY(int x);

�� int getMaskW();

�� void setMaskW(int x);

�� int getMaskH();

�� void setMaskH(int x);

�� int getMaskType();

�� void setMaskType(int x);

�� int getBorderCut();

�� void setBorderCut(int x);

�� int getStickShift();

�� void setStickShift(int x);

�� void setSkipEmptySpaceType(int x);

�� int getSkipEmptySpaceType();

�� void setSkipEmptySpacePercentage(int x);

�� int getSkipEmptySpacePercentage();

��

�� void setSkipEmptySpaceThresholdFilter(int x);

�� int getSkipEmptySpaceThresholdFilter();

�� void setSkipEmptySpaceEdgeFilter(int x);

�� int getSkipEmptySpaceEdgeFilter();

�� Bitmap getPreProcessingImageBW (Bitmap bImg);

�� Bitmap getPreProcessingImageColor (Bitmap bImg);

��

�� }

The following table lists the functions.

Functions	Description
int getMaskX(); void setMaskX(int x); int getMaskY(); void setMaskY(int x); int getMaskW(); void setMaskW(int x); int getMaskH(); void setMaskH(int x); int getMaskType(); void setMaskType(int x);	Sets and gets mask (x, y, w, h) and mask type.
int getBorderCut(); void setBorderCut(int x);	Sets and gets Border Cut.
int getStickShift(); void setStickShift(int x);	Sets and gets Stick Shift.
void setSkipEmptySpaceType(int x); int getSkipEmptySpaceType(); void setSkipEmptySpacePercentage(int x); int getSkipEmptySpacePercentage(); �� void setSkipEmptySpaceThresholdFilter(int x); int getSkipEmptySpaceThresholdFilter(); void setSkipEmptySpaceEdgeFilter(int x); int getSkipEmptySpaceEdgeFilter();	Sets and gets parameters for Skip Empty Space.
Bitmap getPreProcessingImageBW (Bitmap bImg); Bitmap getPreProcessingImageColor (Bitmap bImg);	Get preprocessed image from an input image, bImg.

To add an image preprocessing object, add:

Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 ipre80;

private void Form1_Load(object sender, System.EventArgs e)

�� {

ipre80 = new Attrasoft.TransApplet80.ImagePreProcessing80

�.ImagePreProcessing80 (richTextBox1);

�� richTextBox1.Clear ();

�� }

21.3�� Enter Parameters

Clicking the �OK� button on Figure 21.2 will enter the parameters to the Image Pre-processing filter. The �OK� button is implemented as follows:

private void button1_Click(object sender, System.EventArgs e)

�� {

�� try

�� {

�� ipre80.setMaskX (int.Parse (textBox1.Text)) ;

�� ipre80.setMaskY� (int.Parse (textBox2.Text )) ;

�� ipre80.setMaskW� (int.Parse (textBox3.Text )) ;

�� ipre80.setMaskH� (int.Parse (textBox4.Text )) ;

�� ipre80.setMaskType� (int.Parse (textBox11.Text )) ;

�� ipre80.setBorderCut�� (int.Parse (textBox5.Text )) ;

��

�� ipre80.setStickShift�� (int.Parse (textBox6.Text )) ;

�� ipre80.setSkipEmptySpaceType�� (int.Parse (textBox7.Text )) ;

�� ipre80.setSkipEmptySpacePercentage�� (int.Parse (textBox8.Text )) ;

�� ipre80.setSkipEmptySpaceThresholdFilter�� (int.Parse (textBox9.Text )) ;

�� ipre80.setSkipEmptySpaceEdgeFilter�� (int.Parse (textBox10.Text )) ;

��

�� this.Close();

�� }

�� catch

�� {

�� MessageBox.Show ("Please enter valid integers", "Entry Error");

�� }

21.4�� Cut Off the Border Areas

Let us assume we want to cut off 10% of the border, click the �Parameter� button. You will see Figure 21.2. Enter 10 to the first textbox in Figure 21.2. Click the �Key� button and select an image from �c:\transapplet80\ex_wheel\�. Click the �Preprocessing Image� button and you will have Figure 21.3. You will see that the second image is the first image with 10% of the border cut off.

Figure 21.3 Five Percent Border Cut.

The implementation of the �Preprocessing Image� button is:

private void button11_Click(object sender, System.EventArgs e)

�� {

�� if (b1 == null)

�� {

�� richTextBox1.AppendText ("Please select an image");

�� return;

�� }

��

�� b2 = ipre80.getPreProcessingImageColor� (b1);

�� pictureBox2.Image = b2;

�� richTextBox1.AppendText

(" (W, H) = " + b2.Width + " " + b2.Height + "\n");

�� }

21.5�� Impose a Mask

Let us assume we want to cut off 20% of the border on top, 20% on the left, 10% on the right, and 10% on the bottom, then enter (20, 20, 70, 70, 2) to the second row textboxes in Figure 21.2. Click the �Preprocessing Image� button and we will have Figure 21.4.

In Figure 21.4, you will see the second image is the first image with a mask (20%, 20%, 70%, 70%). Here 2 in (20, 20, 70, 70, 2) means in percentage rather than pixels. You will see that the second image is the first image with the following cut: cut off 20 % of the border on top, 20% on the left, 10% on the right, and 10% on the bottom.

Figure 21.4� Mask (20, 20, 70, 70, 2).

Now, if we use pixels rather than percentage, enter (20, 20, 70, 70, 1) to the second row textboxes in Figure 21.2 and we will have Figure 21.5.

Figure 21.5� Mask (20, 20, 70, 70, 1).

21.6�� Speed Up the Computation

To speed up the computation, set the parameter �Stick Shift� in Figure 21.2 between 0 and 5, with 0 being the slowest and 5 being the fastest.

21.7�� Skip the Empty Border by Content Percent

To get a good image matching, two things are important:

To control the number of variations.
To make the area of interest stands out and to make the rest as little as possible.

Skipping empty border can reduce the number of variations. The last function for the Preprocessing filter is to skip border by content percent, not pixel percent.

The �Skip Empty Border� field in Figure 21.2 specifies the type:

0 �� No skip;

�� 1 �� Skip the white empty border space;

�� 2 �� Skip the black empty border space;

�� 3 �� Skip x percent of the contents on the white background space;

�� 4 �� Skip x percent of the contents on the black background space;

5 �� Skip empty border space on the user defined Threshold Filter;

�� 6 �� Skip x percent of the contents on the user defined Threshold/Edge Filters.

We will discuss Threshold/Edge Filters in the next chapter, Image Processing.

Example 1. Set �Skip Empty Border� = 2, means skipping the black empty space on the edge. The result is shown in 21.6; you will see that the second image is the first image without a black border.

Figure 21.6� Skipping the black empty space.

Example 2. Set �Skip Empty Border� = 4, means skipping the black empty space on the edge. Set �Skip percent� = 10. The result is shown in Figure 21.7; you will see that the second image is the first image with 10% of the contents cut on each side.

Figure 21.7� Skipping 10% of the contents on the black background.

22.�� Image Processing

Attrasoft ImageFinder learns an image in a way similar to human eyes:

Ignore the background;
Focus on an object in the image.

The Image Preprocessing in the last chapter and the Image Processing in this chapter prepare the image for the ImageFinder.

The Image Processing process is not unique; there are many options available. Some are better than others. Image Processing can make it or break it. For many problems like fingerprints, palm prints, �,� special image processing filters will be required.

22.1�� Good & Bad

The principle of choosing the Image Preprocessing and Image Processing filters is to make the sample objects stand out, otherwise change the options.

Do not make too many things stand out, i.e. as long as the area of interest stands out, the rest should show as little as possible.

Figure 22.1� Bad Image Processing Example.

Figure 22.2� Good Image Processing Example.

In Figure 22.1, the first image is the selected key image. The objective is to identify the logo, �2004 Davis Cup ��.

Figure 22.3 Image Processing Setting.

First of all, let us use the setting:

Edge Filter = 2

Threshold Filter = 1

Clean-Up Filter = 2

The second image in Figure 22.1 is a preprocessed image; and the third image in Figure 22.1 is the processed image. This is a bad example of image processing, because the sample object, �2004 Davis Cup ��, does not stand out.

Now, we will try a different setting:

Edge Filter = 0

Threshold Filter = 1

Clean-Up Filter = 16

The third image in Figure 22.2 is the processed image. This is a good example of image processing, because the sample object, �2004 Davis Cup ��, does stand out.

22.2�� Processing API

The Image Processing interface is:

public interface I_ImageProcessing80

�� {

�� string [] getEdgeFilterNames();

�� int getEdgeFilterDefaultIndex();

�� string [] getThresholdFilterNames();

�� int getThresholdFilterDefaultIndex();

�� string [] getCleanUpFilterNames();

�� int getCleanUpFilterDefaultIndex();

�� int getDoubleImageProcessing ();

�� void setDoubleImageProcessing (int x);

�� bool setImage (Bitmap sImageName );

�� bool imageProcess();

�� void setEdgeFilter(int x);

�� int getEdgeFilter();

�� bool getEdgeFiltersStatus();

�� void setThresholdFilter (int x);

�� int getThresholdFilter ();

�� bool getThresholdFiltersStatus();

�� void setCleanUpFilter (int x);

�� int getCleanUpFilter ();

�� bool getCleanUpFilterStatus();

�� void setR1(int x);

�� int getR1();

�� void setR2(int x);

�� int getR2();

�� void setR3(int x);

�� int getR3();

�� void setG1(int x);

�� int getG1();

�� void setG2(int x);

�� int getG2();

�� void setG3(int x);

�� int getG3();

�� void setB1(int x);

�� int getB1();

�� void setB2(int x);

�� int getB2();

�� void setB3(int x);

�� int getB3();

�� }//image processing

The following table lists the functions.

Functions	Description
string [] getEdgeFilterNames(); int getEdgeFilterDefaultIndex(); string [] getThresholdFilterNames(); int getThresholdFilterDefaultIndex(); string [] getCleanUpFilterNames(); int getCleanUpFilterDefaultIndex();	Gets the basic info about Edge filter, Threshold filter, and Clean-Up filter.
int getDoubleImageProcessing (); void setDoubleImageProcessing (int x);	Sets and gets the �Double Processing� parameter.
bool setImage (Bitmap sImageName );	Sets the input image for the Image Processing filters, which is also the output image; i.e. the image processing is applied to the image directly.
void setEdgeFilter(int x); int getEdgeFilter(); bool getEdgeFiltersStatus(); void setThresholdFilter (int x); int getThresholdFilter (); bool getThresholdFiltersStatus(); void setCleanUpFilter (int x); int getCleanUpFilter (); bool getCleanUpFilterStatus();	Sets and gets Image Processing filters.
bool imageProcess();	Apply the image-processing filter to the input image. The input image is also the output image, because the filter is applied to the image directly.
void setR1(int x); int getR1(); void setR2(int x); int getR2(); void setR3(int x); int getR3(); void setG1(int x); int getG1(); void setG2(int x); int getG2(); void setG3(int x); int getG3(); void setB1(int x); int getB1(); void setB2(int x); int getB2(); void setB3(int x); int getB3();	Sets and gets Threshold Filter parameters.

To add an image-processing object, add:

Attrasoft.TransApplet80.ImageProcessing80.ImageProcessing80 ip80;

private void Form1_Load(object sender, System.EventArgs e)

�� {

�� ipre80 = new Attrasoft.TransApplet80.ImagePreProcessing80 .

ImagePreProcessing80 (richTextBox1);

�� ip80 = new Attrasoft.TransApplet80.ImageProcessing80.ImageProcessing80 ();

�� richTextBox1.Clear ();

�� }

22.3�� Set Image Processing Filters

The image processing will be applied to all images before recognition. As far as the operation is concerned, this means setting three filters:

Edge Filters;

Threshold Filters; and

Clean-Up Filters.

In Figure 22.3,

to select an Edge Filter, click the Edge Filter Drop Down List, which is the first List in the red box;
to select a Threshold Filter, click the Threshold Filter Drop Down List, which is the second List;
to select a Clean-Up Filter, click the Clean-Up Filter Drop Down List, which is the third List.

The Edge Filters attempt to exaggerate the main features a user is looking for.

The Threshold Filters attempt to suppress the background.

The Clean-Up Filters will smooth the resulting image to reduce recognition error.

The default setting in the ImageFinder is:

Edge Filter = 2

Threshold Filter = 1

Clean-Up Filter = 2

The code for the three Drop Down Lists is:

private void comboBox1_SelectedIndexChanged(object sender, System.EventArgs e)

�� {

�� f.ip80.setEdgeFilter (comboBox1.SelectedIndex);

�� f.richTextBox1.AppendText ( "Set EdgeFilter " +f.ip80.getEdgeFilter () + ".\n" ) ;

�� }

private void comboBox2_SelectedIndexChanged(object sender, System.EventArgs e)

�� {

�� f.ip80.setThresholdFilter (comboBox2.SelectedIndex );

�� f.richTextBox1.AppendText ( "Set ThresholdFilter "

+comboBox2.SelectedIndex + ".\n" );

�� }

private void comboBox3_SelectedIndexChanged(object sender, System.EventArgs e)

�� {

�� f.ip80.setCleanUpFilter ( comboBox3.SelectedIndex );

�� f.richTextBox1.AppendText� ( "Set CleanUpFilter "

+comboBox3.SelectedIndex + ".\n" );

�� }

22.4�� First Two Settings

The default setting should be:

Edge Filter = 2 or �Sobel 2�.

Threshold Filter =1 or� �Dark Background 128�.

Clean-Up Filter = 2

Your second choice should be:

Edge Filter = 0

Threshold Filter =5

Clean-Up Filter = 2

22.5�� Chapter Projects

Figure 22.4�� Chapter Project.

To see how image processing works, click the �Parameter� button in Figure 22.4 to get Figure 22.3. In Figure 22.3, set the:

Edge Filter = 2 or �Sobel 2�.

Threshold Filter =1 or� �Dark Background 128�.

Clean-Up Filter = 2

You should see the following message in the text window:

Set EdgeFilter 2.

Set ThresholdFilter 1.

Set CleanUpFilter 1.

Display Key Segment!

Click the �Key� button and select image, �C:\transapplet80\ex_label\L01008gi-020501.jpg�. Click the �Image Processing� button and Figure 22.4 will show the results of image processing.�

23.�� Normalization

This chapter introduces a filter, the Normalization filter or Reduction Filter. From the user�s and programmer�s point of view, it means the setting of several parameters. This filter, together with the Image PreProcessing Filter and Image Processing Filters, will be passed to later layers as parameters.

If you are not interested in the details, you can skip this chapter.

The Normalization filter will prepare the images for the underlying NeuralNet filters. The neural net deployed in the ImageFinder, by default, is a 100x100 array of neurons. While any size of ABM neural net can be used, when coming to a particular application, a decision has to be made. The ImageFinder uses 6 different sizes:

� 10,000 neurons,

� 8,100 neurons,

� 6,400 neurons,

� 4,900 neurons, or

� 2,500 neurons.

23.1�� Class Library Name

The class library is:

�� Attrasoft.TransApplet80.ReductionFilter80.

The class in this library will be:

�� ReductionFilter80.

The interface, which will be used by ReductionFilter80, is:

��

�� public interface I_ReductionFilter80

�� {

�� string [] getReductionFilterNames();

�� int getReductionFilterDefaultIndex();

�� void setReductionFilter(int x);

�� int getReductionFilter();

�� bool getReductionFilterStatus();

�� int getNeuralNetWAndH ( );

�� void setNeuralNetWAndH (int i );

��

�� void setSegmentCut(int x);

�� int getSegmentCut();

��

�� void setSizeCut(int x);

�� int getSizeCut();

�� void setBorderCut(int x);

�� int getBorderCut();

�� void setLookAtX(int x);

�� int getLookAtX();

�� void setLookAtY(int x);

�� int getLookAtY();

�� void setLookAtXLength(int x);

�� int getLookAtXLength();

�� void setLookAtYLength(int x);

�� int getLookAtYLength();

�� string getMessage();

�� string getInfomation();

�� string toString();

�� void setImage (Bitmap b );

�� int [] getPixels ();

�� int[] getArray ();

�� int getArrayWidth();

�� int getArrayHeight();

�� }

23.2�� Class Library Overview

This class library will address a single filter in the Normalization layer, the Reduction Filter.

The Reduction Filter selections are:

string [] getReductionFilterNames();

�� int getReductionFilterDefaultIndex();

To set the Reduction Filter, use:

�� void setReductionFilter(int x);

int getReductionFilter();

�� bool getReductionFilterStatus();

To set the Reduction Filter parameters, use:

void setSegmentCut(int x);

int getSegmentCut();

void setSizeCut(int x);

int getSizeCut();

void setBorderCut(int x);

int getBorderCut();

void setLookAtX(int x);

int getLookAtX();

void setLookAtY(int x);

int getLookAtY();

void setLookAtXLength(int x);

int getLookAtXLength();

void setLookAtYLength(int x);

int getLookAtYLength();

string getMessage();

String getInfomation();

String toString();

To process an image via the Reduction Filter, specify the image via:

�� void setImage (Bitmap b );

To use the filter, call:

�� int[] getArray ();

�� int getArrayWidth();

�� int getArrayHeight();

23.3�� Link to Class Library

To include the class library in the project,

Right click References and select Add Reference in the Solution Explorer;
Browse to find �ReductionFilter80.dll� in �c:\transapplet80\�;
Highlight it and click the �OK� button.

To use the class library, add:

using Attrasoft.TransApplet80.ReductionFilter80;

To declare an object, add:

public Attrasoft.TransApplet80.ReductionFilter80.ReductionFilter80 rd80

= new Attrasoft.TransApplet80.ReductionFilter80.ReductionFilter80 ();

Now ReductionFilter80, rd80, is ready to use.

23.4�� Parameters

The Reduction Filter parameters are:

�Segment-Cut Button

Use the �Segment-Cut� button to shape the segment considered by the ImageFinder. The Segment-Cut parameter ranges from 0 to 12. This parameter deals with the edges of segments in the images. The larger this parameter is, the smaller the segment the ImageFinder will use. The possible settings of this parameter in the user interface are: 0, 1, 2, .., and 12.

Size-Cut Button

Use the "Size-Cut" button to limit the dimensions of images to be searched. In some applications, the users only want to search images of certain dimensions and ignore other images. The dimension setting ranges from 0 to 9.

� If the setting is 0, this parameter will be ignored.

� If the parameter is 1, then the longest edge of the image to be considered must be at least 100, but less than 199.

� If the parameter is 2, then the longest edge of the image to be considered must be at least 200, but less than 299, �

Border Cut

Use the �Border-Cut� button to ignore the sections of the image near the borders. The Border-Cut parameter ranges from 0 (no cut) to 9 (18% border cut). The possible settings in the user interface are: 0, 1, 2, .., and 9. Assume an image is (0,0; 1,1); setting Border-Cut to 1 means the ImageFinder will look at the section (0.02, 0.02; 0.98, 0.98); setting Border-Cut to 2 means the ImageFinder will look at the section (0.04, 0.04; 0.96, 0.96); � .

Look-At Area

The �Look-At Area� is the area the ImageFinder will use. A 100x100 window specifies a whole image. In the Integer-Reduction, the actual area can be less than 100x100.� The Look-At Area is specified by 4 numbers:

�� (x, y, w, h)

(x, y) are the coordinates of the upper-left corner and (w, h) are the width and height of the Look-At Window.

�

24.�� Parameter Class

ImageFinder has many parameters. For the ImageFinder, the Image Matching is divided into:

Image Preprocessing
Image Processing
Normalization
Signatures
Signature Recognition
Image Segment Recognition

One or more filters further implements each step:

Image Preprocessing

�� Preprocessing Filter

Image Processing

�� Edge Filter

�� Threshold Filter

�� Clean-Up Filter

Normalization

�� Reduction Filter

Signature

�� Signature Filter

Signature Recognition

�� Unsupervised Filter

�� BioFilter

�� NeuralFilter

Image Segment Recognition

�� Neural Net Filter

The image-matching software design, based on the TransApplet, is implemented to push an image through all of these filters. At the end of this pushing, the matching results will be obtained.

24.1�� Pushing Images Through Filters

How to push an image through the following filters:

1. Preprocessing Filter

2. Edge Filter

3. Threshold Filter

4. Clean-Up Filter

5. Reduction Filter

6. Signature Filter

7. Unsupervised Filter

8. BioFilter

9. NeuralFilter

10. Neural Net Filter

11. �

The implementation is as follows: the second filter will use the first filter as a parameter; the third filter will use the first and second filters as parameters; �

Each filter is an object (See their API in earlier chapters). Each filter will have many parameters and all filters can have many parameters.

It will be easier, from the programming point of view, to group all of the objects together and to group all of the parameters together.

� Section 24.2 will group all of the objects together; and

� Section 24.3 will group all of the parameters together.

24.2�� Predefined Objects

You can either declare objects yourself or use the predefined objects in your project. There is one class in the TransApplet:

�� Attrasoft.TransApplet80.EntryPoint80,

that has all of the objects required for the ImageFinder. In particular, the objects are:

public Attrasoft.TransApplet80.ImageSignature80.ImageSignature imageSignature;

public Attrasoft.TransApplet80.ImageSignature80.ImageSignature imageSignature1;

public Attrasoft.TransApplet80.Results_1N.Results_1N results_1N;

public Attrasoft.TransApplet80.ImagePacket80.ImagePacket80 imagePacket ;

public Attrasoft.TransApplet80.TransAppletParameters.TransAppletParameters� transAppletPara;

�

public Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 imagePreProcessingFilter ;

public Attrasoft.TransApplet80.ImageProcessing80.ImageProcessing80 imageProcessingFilter ;

public Attrasoft.TransApplet80.ReductionFilter80.ReductionFilter80 reductionFilter ;

public Attrasoft.TransApplet80.ImageSignatureFilter80.ImageSignatureFilter signatureFilter;

public Attrasoft.TransApplet80.UnsupervisedFilter80 .UnsupervisedFilter unsupervisedFilter;

public Attrasoft.TransApplet80.BioFilter80.BioFilter bioFilter;

public Attrasoft.TransApplet80.NeuralFilter80.NeuralFilter neuralFilter;

public Attrasoft.TransApplet80.ImageLibrary80.ImageLibrary imageLibrary;

public Attrasoft.TransApplet80.NeuralNet80.NeuralNet80 neuralNetFilter;

public Attrasoft.TransApplet80.Counting80 .Counting80 counting;

�

To use these objects, simply declare an object of the following class:

Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 script;

In the constructor, add:

script = new Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 (richTextBox1);

Now, you can use all of the objects:

Objects	Descriptions
Script. ImageSignature	Image Signature
Script. signatureFilter	Signature Filter
Script. bioFilter	BioFilter
Script. unsupervisedFilter	Unsupervised Filter
�	�

24.3�� Grouping Parameters Together

The class that groups all parameters together is called TransAppletParameters. The class interface is:

public interface TransAppletParameters�

�� {

�� public bool setPreProcessingFilterData

�� ( Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 ipre80);

�� public bool getPreProcessingFilterData

�� ( Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 ipre80);

�� public bool setImageProcessingFilterData

�� ( Attrasoft.TransApplet80.ImageProcessing80.ImageProcessing80 ip80);

�� public bool getImageProcessingFilterData

�� ( Attrasoft.TransApplet80.ImageProcessing80.ImageProcessing80 ip80);

�� public bool setReductionFilterData

�� ( Attrasoft.TransApplet80.ReductionFilter80 .ReductionFilter80 rd80);

�� public bool getReductionFilterData

�� ( Attrasoft.TransApplet80.ReductionFilter80 .ReductionFilter80 rd80);

��

�� public bool setSignatureFilterData

�� ( Attrasoft.TransApplet80.ImageSignatureFilter80.ImageSignatureFilter sig80);

�� public bool getSignatureFilterData

�� ( Attrasoft.TransApplet80.ImageSignatureFilter80.ImageSignatureFilter sig80);

�� public bool setUnsupervisedFilterData

�� ( Attrasoft.TransApplet80.UnsupervisedFilter80 .UnsupervisedFilter� usf80);

�� public bool getUnsupervisedFilterData

�� ( Attrasoft.TransApplet80.UnsupervisedFilter80 .UnsupervisedFilter� usf80);��

��

�� public bool setBioFilterData

�� ( Attrasoft.TransApplet80.BioFilter80.BioFilter� bf80);

public bool getBioFilterData

�� ( Attrasoft.TransApplet80.BioFilter80 .BioFilter�� bf80);

��

�� public bool setNeuralFilterData

�� ( Attrasoft.TransApplet80.NeuralFilter80.NeuralFilter nf80);

�� public bool getNeuralFilterData

�� ( Attrasoft.TransApplet80.NeuralFilter80 .NeuralFilter� nf80);

��

�� public bool setNeuralNetData

�� ( Attrasoft.TransApplet80.NeuralNet80 .NeuralNet80�� nn80);

�� public bool getNeuralNetData

�� ( Attrasoft.TransApplet80.NeuralNet80 .NeuralNet80�� nn80);

�� }

Before we explain the functions, let us first see the base class, which contains all of the parameters:

public class TransAppletParametersData

�� {

�� //[ImageFinder 7.0]

�� public int executionCode =0;

�� //[Input]

�� public string keyImage_absolutePath="None";

�� public string keyImage_segmentX="0";

�� public string keyImage_segmentY="0";

�� public string keyImage_segmentXlength="0";

�� public string keyImage_segmentYLength="0";

�� public string searchSource_Name="None";

�� public string searchSource_Type = "0";

�� public string searchSource_sqlStatement ="NA";

�� //[Image PreProcessing Filters]

�� public int imagePreProcessing_BorderCut =0;

�� public int imagePreProcessing_MaskX =0;

�� public int imagePreProcessing_MaskY =0;

�� public int imagePreProcessing_MaskW =0;

�� public int imagePreProcessing_MaskH = 0;

�� public int imagePreProcessing_MaskType = 0;

�� public int imagePreProcessing_StickShift = 0;

�� public int imagePreProcessing_SkipEmptyBorderType� =0;

�� public int imagePreProcessing_SkipPercent =0;

�� public int imagePreProcessing_SkipEdgeFilter = 0;

�� public int imagePreProcessing_SkipThresholdFilter = 0;

��

�� //[Image Processing Filters]

�� public int imageProcessing_edgeFilter=2;

�� public int imageProcessing_thresholdFilter=1;

�� public int imageProcessing_cleanUpFilter=2;

�� public int imageProcessing_doubleProcessing=0;

�� public int imageProcessing_R1=0;

�� public int imageProcessing_R2=128;

�� public int imageProcessing_R3=2;

�� public int imageProcessing_G1=0;

�� public int imageProcessing_G2=128;

�� public int imageProcessing_G3=2;

�� public int imageProcessing_B1=0;

�� public int imageProcessing_B2=128;

�� public int imageProcessing_B3=2;

�� //[Reduction Filter]

�� public int reduction_Filter=0;

�� public int reduction_SegmentCut=0;

�� public int reduction_SizeCut=0;

�� public int reduction_BorderCut=0;

�� public int reduction_LookAtX=0;

�� public int reduction_LookAtY=0;

�� public int reduction_LookAtXLength=0;

�� public int reduction_LookAtYLength=0;

��

�� //[SignatureFilter]

�� public int signatureFilter=0;

��

�� //[UnsupervisedFilter]

�� public int UnsupervisedFilterFilter=0;

�� public int UnsupervisedFilter_Percent=20;

�� public int UnsupervisedFilter_Mode=0;

�� public int UnsupervisedFilter_CutOff=0;

�� public int UnsupervisedFilter_OutputFileType=0;

�� public int UnsupervisedFilter_ShowFile=1;

�� public int UnsupervisedFilter_Blurring=2;

�� public int UnsupervisedFilter_Sensitivity=4;

�� public int UnsupervisedFilter_UseRelativeScore=0;

�� public int UnsupervisedFilter_ShowScore = 1 ;

�� public int UnsupervisedFilter_AutoSegment=0;

��

�� //[BioFilter]

�� public int bioFilter=0;

�� public int bioFilter_Percent=20;

�� public int bioFilter_Mode=0;

�� public int bioFilter_CutOff=0;

�� public int bioFilter_OutputFileType=0;

�� public int bioFilter_ShowFile=1;

�� public int bioFilter_Blurring=2;

�� public int bioFilter_Sensitivity=4;

�� public int bioFilter_UseRelativeScore=0;

�� public int bioFilter_ShowScore = 1 ;

�� public int bioFilter_AutoSegment=0;

��

�� //[NeuralFilter]

�� public int neuralFilter=2;

�� public int neuralFilter_Percent=20;

�� public int neuralFilter_Mode=0;

�� public int neuralFilter_Size=2;

�� public int neuralFilter_CutOff=0;

�� public int neuralFilter_OutputFileType=0;

�� public int neuralFilter_ShowFile=1;

�� public int neuralFilter_Blurring=2;

�� public int neuralFilter_Sensitivity=4;

�� public int neuralFilter_UseRelativeScore=0;

�� public int neuralFilter_ShowScore = 1 ;

�� public int neuralFilter_AutoSegment=0;

��

�� //[Neural Net]

�� public int neuralNetFilter=0;

�� public int neuralNetFilter_symmetry=3;

�� public int neuralNetFilter_rotationType=0;

�� public int neuralNetFilter_translationType=0;

�� public int neuralNetFilter_scalingType=0;

�� public int neuralNetFilter_sensitivity=50;

�� public int neuralNetFilter_blurring=10;

�� public int neuralNetFilter_internalWeightCut=100;

�� public int neuralNetFilter_externalWeightCut=0;

�� public int neuralNetFilter_segmentSize=0;

�� public int neuralNetFilter_imageType=1;

�� public int neuralNetFilter_fileDisplayType=0;

�� public int neuralNetFilter_autoSegment=0;

�� public int neuralNetFilter_neuralNetMode=0;

�� }

The base class, TransAppletParametersData, holds all parameters for the TransApplet. The main class, TransAppletParameters, has many pairs of functions like this:

public bool setPreProcessingFilterData

�� (Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 ipre80);

�� public bool getPreProcessingFilterData

�� (Attrasoft.TransApplet80.ImagePreProcessing80.ImagePreProcessing80 ipre80);

The first function, setPreProcessingFilterData, passes the parameters in the TransAppletParameters object to a filter.

The second function, getPreProcessingFilterData, gets all of the parameters in a filter and stores them into the current TransAppletParameters object.

In this way, we get and set all of the parameters of a filter together. We also treat all of the filters exactly the same way.

24.4�� Chapter Project

The chapter project is located at c:\transapplet80\imagefinder\. Starting the project and you will see Figure 24.1

In Figure 24.1, click the �Para� in Figure 24.1, which will open a form, see Figure 24.2. This form will handle all of the parameters used in the TransApplet. It also links to many other forms used to specify the parameters. Each of these forms specifies parameters for one filter.

Figure 24.1�� The main form.

Figure 24.2� The Parameters form.

Figure 24.3� Image Preprocessing Parameter Form.

Figure 24.4� Image Processing Parameter Form.

Figure 24.5� Normalization Parameter Form.

Figure 24.6� Unsupervised Filter Parameter Form.

Figure 24.7� Neural Filter Parameter Form.

Figure 24.8� NeuralNet Filter Parameter Form.

Parameter Form

Implements all of the TransApplet parameters. This form will link to many other forms used to specify parameters. Each of these forms specifies parameters for one filter. Figure 24.2 shows the form.

Image Preprocessing Parameter Form

Implements all of the Image Preprocessing parameters. Figure 24.3 shows the form.

Image Processing Parameter Form

Implements all of the Image Processing parameters. Figure 24.4 shows the form.

Normalization Parameter Form

Implements all of the Normalization parameters. Figure 24.5 shows the form.

Unsupervised Filter Parameter Form

Implements all of the Unsupervised Filter parameters. Figure 24.6 shows the form.

BioFilter Parameter Form

Implements all of the BioFilter parameters. The BioFilter Parameter Form looks like the Unsupervised Filter parameters form in Figure 24.6.

NeuralFilter Parameter Form

Implements all of the NeuralFilter parameters. Figure 24.7 shows the form.

NeuralNet Filter Parameter Form

Implements all of the NeuralNet Filter parameters. Figure 24.8 shows the form.

24.5�� TransApplet Objects

The first thing we will do is to create all of the objects required for the ImageFinder. As we discussed earlier in this chapter, all of the objects required for the ImageFinder project is grouped into a single class. All we have to do is to create an object of this class and we will call it script. We will create this object in class, MainMenuToAPI. The object is called script.

internal Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 script;

MainMenu f;

public MainMenuToAPI(MainMenu f1)

{

try

�� {

�� f = f1;

�� script = new Attrasoft.TransApplet80.EntryPoint80.EntryPoint80

�(f.richTextBox1);

�� }

�� catch (Exception e )

�� {

�� f.richTextBox1.Text = e.ToString () +"\n";

�� }

24.6�� Selecting Filters

The �Parameter� button in Figure 24.1 opens the Parameter Form in Figure 24.2, as follows:

private void button10_Click(object sender, System.EventArgs e)

�� {

�� Parameters paForm = new Parameters (this.mainMenuToAPI .script );

�� paForm.ShowDialog ();

�� }

Once in the Parameter Form in Figure 24.2, you can set the filters directly. To set the filter parameters, you have to open the following additional forms:

Image Preprocessing Parameters Form

Image Processing Parameters Form

Normalization Parameters Form

Unsupervised Filter Parameters Form

BioFilter Parameters Form

NeuralFilter Parameters Form

NeuralNet Filter Parameters Form

There are two parts related to the parameters:

Selecting Filters
Setting Filter Parameters

Selecting filters are done in the Parameter Form in Figure 24.2.�

Setting filter parameters requires opening another form in Figures 24.3 through 24.8; �it is in these forms that the filter parameters are set.

We now discuss how to set each filter:

Image Preprocessing Filter

There is only one Image Preprocessing Filter, so you do not have to set this filter.

Image Processing Filters

There are three image processing filters: Edge Filters, Threshold Filters, and Clean-Up Filters. You can select these filters from the Drop Down Lists in Figure 24.2.

The Edge Filter is implemented as follows:

void comboBox1_SelectedIndexChanged(object sender, System.EventArgs e)

{

script.transAppletPara.imageProcessing_edgeFilter =

�� comboBox1.SelectedIndex;

script.transAppletPara.setImageProcessingFilterData

�� (script.imageProcessingFilter );��

richTextBox1.AppendText� ( "Set EdgeFilter "

�+script.transAppletPara.imageProcessing_edgeFilter + ".\n" ) ;

�� }

The first line gets the new setting from the Combo Box (Drop Down List).

The second line assigns the new data to the Image Processing filter object.

The third line prints a message.

The Threshold Filter and the Clean-Up Filter are handled in a similar fashion.

Normalization Filter

You can select the Normalization filters from the Drop Down List in Figure 24.2. The Normalization Filter is implemented as follows:

void comboBox4_SelectedIndexChanged(object sender, System.EventArgs e)

{

script.transAppletPara.reduction_Filter

�� =� comboBox4.SelectedIndex ;

script.transAppletPara.setReductionFilterData

�� (script.reductionFilter� );

richTextBox1.AppendText ( "Set Redeuction Filter "

+comboBox4.SelectedIndex + ": " +� comboBox4.SelectedItem + "\n" );

}

The Unsupervised Filter selection, BioFilter selection, NeuralFilter selection, and NeuralNet Filter selection will be implemented in a similar fashion.

24.7�� Set Filter Parameters

In the last section, we have shown that you can select a filter in the Parameter Form shown in Figure 24.2. After selecting a filter, to set the filter parameters, you have to open the following additional forms and specify the parameter:

Image Preprocessing Parameter Form

Image Processing Parameter Form

Normalization Parameter Form

Unsupervised Filter Parameter Form

BioFilter Parameter Form

NeuralFilter Parameter Form

NeuralNet Filter Parameter Form

We will use the Image Preprocessing Parameter Form in Figure 24.3 to show the implementation. The �OK� button in Figure 24.3 is implemented as follows:

private void button1_Click(object sender, System.EventArgs e)

{

try

{

script.transAppletPara.imagePreProcessing_MaskX = int.Parse (textBox1.Text) ;

script.transAppletPara.imagePreProcessing_MaskY = int.Parse (textBox2.Text ) ;

script.transAppletPara.imagePreProcessing_MaskW = int.Parse (textBox3.Text ) ;

script.transAppletPara.imagePreProcessing_MaskH =� int.Parse (textBox4.Text ) ;

script.transAppletPara.imagePreProcessing_MaskType

= int.Parse (textBox11.Text );

script.transAppletPara.imagePreProcessing_BorderCut

=�� int.Parse (textBox5.Text ) ;

script.transAppletPara.imagePreProcessing_StickShift�

=�� int.Parse (textBox6.Text ));

script.transAppletPara.imagePreProcessing_SkipEmptyBorderType

=�� int.Parse (textBox7.Text ) ;

script.transAppletPara.imagePreProcessing_SkipPercent

=�� int.Parse (textBox8.Text ) ;

script.transAppletPara.imagePreProcessing_SkipThresholdFilter��

=�� int.Parse (textBox9.Text ) ;

script.transAppletPara.imagePreProcessing_SkipEdgeFilter��

=�� int.Parse (textBox10.Text ) ;

script.transAppletPara.setPreProcessingFilterData

�(script.imagePreProcessingFilter );

��

this.Close();

}

catch

{

MessageBox.Show ("Please enter valid integers", "Entry Error");

}

The first section takes the input data from the text boxes and stores them in object, ��script.transAppletPara.

The second section,

script.transAppletPara.setPreProcessingFilterData (script.imagePreProcessingFilter );

passes the data in object, �script.transAppletPara�, to the Image Preprocessing filter,

script.imagePreProcessingFilter.

In a similar fashion, we can implement the following parameter forms:

Image Processing Parameter Form

Normalization Parameter Form

Unsupervised Filter Parameter Form

BioFilter Parameter Form

NeuralFilter Parameter Form

NeuralNet Filter Parameter Form

25.�� Parameters

This chapter will describe the parameters in the ImageFinder. The chapter project is in .\framefinder folder.

25.1�� Overview

The ImageFinder for Windows has many open parameters for the user to adjust for their particular image type. Click the �Parameter� button; you will see Figure 25.1, where you can set the parameters.

Figure 25.1� Parameter Window.

25.2�� Image Preprocessing

Figure 25.2� Image Preprocessing.

Border Cut

Use the �Border Cut� parameter to cut the border areas. Enter N to the first textbox in Figure 25.2, then N% near the border will be cut off.

Mask

Use the �Mask� parameter to impose a mask on the input images. Enter (x, y, w, h, 2) to the second row in Figure 25.2; then a mask in percent value will be imposed on the input images. Enter (x, y, w, h, 1) to the second row in Figure 25.2; then a mask in pixel value will be imposed on the input images.

Stick Shift

Use the �Stick Shift� parameter to speed up the computation.� Set �Stick Shift� in Figure 25.2 between 0 and 5, with 0 being the slowest and 5 being the fastest.

Skip Empty Board

Skip Percent

Skip Threshold Filter

Skip Edge Filter

Use these parameters to skip the border area by cutting off N% percent of the contents.

The �Skip Empty Border� parameter in Figure 25.2 specifies the type:

0 �� No skip;

�� 1 �� Skip the white empty border space;

�� 2 �� Skip the black empty border space;

�� 3 �� Skip x percent of the contents on the white background space;

�� 4 �� Skip x percent of the contents on the black background space;

5 �� Skip empty border space on user defined Threshold Filter;

6 Skip x percent of the contents on user defined Threshold/Edge Filters.

Use the �Skip Percent� parameter to specify the percentage of content you want to cut off for Options 3, 4, and 6.

Options 1, 2, and 5 use the default setting, which is 2%. Use the �Skip Threshold Filter� and �Skip Edge Filter� to set the Edge Filter and Threshold Filter, respectively.

25.3�� Image Processing

Figure 25.3� Image Processing.

25.3.1 ��Edge Filters

Edge Filters extract and enhance edges & contours in an image by expressing intensity differences (gradients) between neighboring pixels as an intensity value. The basic variables are the differences between the top and bottom rows, the differences between the left and right columns, and the differences between the center point and its neighbors.

Edge Filters have the following selections:

Code�� Meaning

0�� No Edge Filter

1�� Sobel 1 (Prewitt)

2�� Sobel 2 (Sobel)

3�� Sobel 3

4�� Sobel 4

5�� Gradient

6�� Gradient, 45�

7�� Sobel 1, 45�

8�� Sobel 1, - 45�

9�� Laplacian 4

10�� CD 11

11�� FD 11

12�� FD 9

13�� FD 7

14 FD 13

15 Laplacian 5

16�� Laplacian 8

17�� Laplacian 9

18�� Laplacian 16

19�� Laplacian 17

All other filters have to be ordered in a Customized Version.

These names really do not make any sense to common users; the best way to figure out what these filters are, is to select a training image and try each of the filters. In general, these filters require the �Dark Background 125� Threshold Filter.

If you do not want to know the details, please skip the rest of this section. The details will be given below so you will know how to order a customized filter:

Sobel 1:

�� -1� 0� 1�� -1� -1� -1

�� -1� 0� 1�� 0�� 0�� 0

�� -1� 0� 1�� 1�� 1�� 1

Sobel 2:

�� -1� 0� 1�� -1� -2� -1

�� -2� 0� 2�� 0�� 0�� 0

�� -1� 0� 1�� 1�� 2�� 1

Sobel 3:

�� -1� 0� 1�� -1� -3� -1

�� -3� 0� 3�� 0�� 0�� 0

�� -1� 0� 1�� 1�� 3�� 1

Sobel 4:

�� -1� 0� 1�� -1� -4� -1

�� -4� 0� 4�� 0�� 0�� 0

�� -1� 0� 1�� 1�� 4�� 1

Gradient:

�� 0� 0� 0�� 0� -1�� 0

�� -1� 0� 1�� 0�� 0�� 0

�� 0� 0� 0�� 0�� 1�� 0

Gradient,� 45�

�� 0� 0� 1�� -1� 0�� 0

�� 0� 0� 0�� 0�� 0�� 0

�� -1� 0� 0�� 0�� 0�� 1

Sobel 1,� 45�

�� 0� 1� 1�� 1�� 1�� 0

�� -1� 0� 1�� 1�� 0�� -1

�� -1 -1� 0�� 0� -1� -1

Sobel 2,� 45�

�� 0� 1� 2�� 2�� 1�� 0

�� -1� 0� 1�� 1�� 0�� -1

�� -2 -1� 0�� 0� -1� -2

Laplacian 4

�� 0� -1� 0��

�� -1�� 4� -1��

�� 0�� -1� 0��

Laplacian 5

�� 0� -1� 0��

�� -1�� 5� -1��

0� -1� 0��

Laplacian 8

�� -1� -1� -1��

�� -1�� 8� -1��

�� -1� -1� -1

Laplacian 9

�� -1� -1� -1��

�� -1�� 9� -1��

�� -1� -1� -1

Laplacian 16

�� 0�� 0�� -1�� 0�� 0

�� 0� -1�� -2� -1�� 0��

�� -1� -2� 16 -2� -1��

�� 0�� -1� -2� -1�� 0

�� 0�� 0�� -1�� 0�� 0

Laplacian 17

�� 0�� 0�� -1�� 0�� 0

�� 0� -1�� -2� -1�� 0��

�� -1� -2� 17 -2 �-1��

�� 0�� -1� -2� -1�� 0

�� 0�� 0�� -1�� 0�� 0

25.3.2�� Threshold Filters

After Edge Filters, the Threshold Filter will be applied to the images. Choose these two filters where the sample objects stand out, otherwise change the filters.

If no filter in this version fits your problem, a Customized Filter has to be built. DO NOT make too many things stand out, i.e. as long as the area of interest stands out, the rest should show as little as possible.

Once you make a selection, the objects in the images are black and the background is white (like a book: white paper, black print). You should make the black area as small as possible, as long as it covers the key-segment(s). Otherwise, switch to a different background.

Figure 25.4 �Threshold Filter Parameters.

There are 30 Threshold filters in the ImageFinder. A few filters, including the average-filter and the customized-filter, allow you to specify any color range. Color is specified by three separate colors: Color = (red, green, blue). Each of the colors ranges from 0 to 255.� (0, 0, 0) is black;� (255, 255, 255) is white.

You should choose a filter where the sample object(s) stand out. You may want to know the meaning of the filters; example, "Light Background 128" means:

�

� �RGB Average in 0� � 127 �� objects; and

� �RGB Average in 128 - 255� � background.

To Summarize:

� Choose an Edge Filter and a Threshold Filter where the sample object(s) stand out;

� Choose an Edge Filter and a Threshold Filter where the black area is as small as possible, as long as it covers the key-segment(s).

25.3.3�� Clean-Up Filters

Clean-Up Filters will clear noise off the image, but it will take more computation time.

25.4�� Normalization Filter

The Normalization Filter connects the image to the underlying neural nets. Let the underlying neural net be 100x100: if an image is larger than 100x100, say 350x230, then this image will be reduced to 100x100 or smaller.

When reducing images, a scaling factor can be introduced easily. Although scaling symmetry can compensate for this scaling factor, scaling symmetry is computationally expensive.

It is important to know that the Reduction Filter will match the selected underlying neural net, therefore, the behavior of the Reduction Filter not only depends on the selection of this filter itself, but also depends on the NeuralNet Filter chosen.

Figure 25.5� Selecting Reduction Filter.

There are several ways to reduce images:

Integer,
Real, or
All images are reduced by the same amount.

Integer Reduction

Images are reduced by an integer factor to maximally fit 100x100 without distortion. For example, a 350x230 image will be reduced to 87x57.

Real Reduction

Images are reduced by a real number to maximally fit 100x100 without distortion. For example, a 350x230 image will be reduced to 100x65.

Within each type of reduction, there are 3 more settings. Assume a 3x3 pixel array is reduced to 1 pixel,

Avg: Assign the average of the 3x3 pixel array to the new pixel;
Max: Assign the maximum of the 3x3 pixel array to the new pixel; or
Min: Assign the minimum of the 3x3 pixel array to the new pixel.

To select the Reduction Filter, use the fourth drop down list. The Reduction Filter has seven parameters.

Segment Cut

This parameter deals with the edges of the segments in the images. The Segment Cut parameter ranges from 0 to 12. The larger this parameter is, the smaller the segment the ImageFinder will use. The possible settings in the user interface are: 0, 1, 2, .., and 12.

Size Cut

In some applications, the users only want to search images of certain dimensions and ignore other images. An example is given below:

In this example, the two stamps belong to two different classes based on the image dimension alone.

The Size Cut parameter ranges from 0 to 9.

� If the setting is 0, this parameter will be ignored.

� If the parameter is 1, then the longest edge of the image to be considered must be at least 100, but less than 199.

� If the parameter is 2, then the longest edge of the image to be considered must be at least 200, but less than 299; �

Border Cut

The Border Cut parameter ranges from 0 (no cut) to 9 (18% border cut). For some images (see the picture below), you might want to get rid of the sections of images close to the borders. To get rid of the border section, use the Border Cut.

The possible settings in the user interface are: 0, 1, 2, .., and 9.

Assume an image is (0,0; 1,1),

� setting Border Cut to 1 means the ImageFinder will look at the section (0.02, 0.02; 0.98, 0.98);

� setting Border Cut to 2 means the ImageFinder will look at the section (0.04, 0.04; 0.96, 0.96); � .�

Look-At Area

The Look-At Area is the area the ImageFinder will use in a matching operation. A 100 x 100 window specifies a whole image. If an integer Reduction Filter is used, the actual area can be less than 100x100.�

Four numbers specify the Look-At Area:

�� (x, y, w, h)

(x, y) are the coordinates of the upper-left corner and (w, h) are the width and height of the Look-At window.

To use this Look-At window, enter (x, y, w, h) to the 4 text boxes.

25.5�� Unsupervised Filter and BioFilter

The Unsupervised Filter and the BioFilter have similar parameters, so we have combined these two filters together.

Figure 25.6� Unsupervised Filter or BioFilter Parameter.

Fault Tolerance Scale

Use this parameter to control the amount of output. This parameter ranges from 0 to 100. The larger this number is, the more matches you will get. To set this parameter, enter a number between 0 and 100 to the text box.

Blurring

Sensitivity

Threshold

The result of image comparison is a "score", indicating the degree to which a match exists. This score is then compared to a pre-set Threshold to determine whether or not to declare a match. This parameter sets the threshold. To decide what threshold to use, you should make a test run first and look at the scores. Matching images have higher scores; unmatched images have lower scores. Select a threshold to separate these two groups. There will be a few images in the middle, representing both groups. Under these circumstances, the threshold selection depends on your application.

Relative Score

Use the relative score to set the range of matching score between 0 and 100.

Show File

This parameter is set to 1 by default, which will show the output file. If this parameter is set to 0, then output file will not be shown.

25.6�� Neural Filters

Figure 25.7� �NeuralFilter Parameter.

Fault Tolerance Scale

Blurring

Sensitivity

Threshold

Relative Score

Use the relative score to set the range of matching scores between 0 and 100.

Neural Filter Opening

This parameter controls the amount of output. This parameter has 5 settings:

Very Large

Large

Normal

Small

Very Small

Large openings will allow more output than small openings. To set the parameter, keep clicking the button; the setting will switch from one to the next each time you click the button.

Show File

This parameter is set to 1 by default, which will show the output file. If this parameter is set to 0, then output file will not be shown.

25.7�� NeuralNet Filter

Figure 25.8� �Neural Net Parameter.

The available NeuralNet filters are:

100x100�� (Most Accurate)
90x90
80x80
70x70
60x60
50x50�� (Least Accurate)

Let the speed of a 100x100 filter be a base, then the overall speed for:

90x90 filter is 1 times faster;
80x80 filter is 1.6 times faster;
70x70 filter is 2.7 times faster;
60x60 filter is 5 times faster; and
50x50 filter is 10 times faster.

The NeuralNet Filter has many parameters. The following sections will explain these parameters.

25.7.1�� Symmetry

Symmetry or Invariance means similarity under certain types of changes. For example, considering two images, one with a face in the middle and the other with the face moved to the edge; we say these two images are similar because of the face.

�

The symmetry defines "similar images". The Attrasoft ImageFinder supports five symmetry settings:

� No symmetry (0);

� Translation symmetry (3);

� Scaling symmetry (4);

� Rotation symmetry (5); and

� Rotation & Scaling symmetries (6).

The numbers are the codes in the batch file. Currently, Scaling symmetry and Oblique symmetry are the same.

Other symmetries, or combination of symmetries, can be built for Customized Orders.

A customized Attrasoft ImageFinder can implement any symmetry (or combination of symmetries), which can be described by mathematics.

However, symmetries are computationally expensive.

Every symmetry setting has the Translation symmetry, except "No Symmetry".� In addition, each of the above settings support:

� Intensity symmetry.

Symmetries are computationally expensive, meaning it will take a longer time to do the job. You should use them only when they are required.

To set the Symmetry, keep clicking the �Symmetry� button; the setting will switch from one to the next each time you click the button. The default setting in this version is Translation symmetry.

For example, it seems that Stamp Recognition requires Translation and Rotation symmetries. But because the edges of a stamp can be detected easily, the stamp can be rotated and shifted to a fixed position where the horizontal side is longer than the vertical side. All you need to do is recognize a stamp or an upside-down stamp. Therefore, Stamp Recognition does not really require Translation and Rotation symmetries.

25.7.2�� Translation Type

The Translation Type defines the accuracy of the Translation symmetry. �

The Translation Type settings (and their codes) are:

� Most Accurate (0);

� Accurate (1); and

� Least Accurate (2).

To set the Translation Type, keep clicking the �T Type� button; the setting will switch from one to the next each time you click the button. The default setting is 0, the most accurate setting.

25.7.3�� Scaling Type

The Scaling Type defines the accuracy of the Scaling symmetry.�

The Scaling Type settings (and their codes) are:

� Least Accurate (0);

� Accurate (1);

� Accurate (2); and

� Most Accurate (3).

To set the Scaling Type, keep clicking the �S Type� button; the setting will switch from one to the next each time you click the button. The default setting is 0, the least accurate setting.

25.7.4�� Rotation Type

The Rotation Type defines the accuracy of the Rotation symmetry.�

The Rotation Type settings (and their codes) are:

� 360� rotation, least accurate (0);

� -5� to 5� rotation (1);

� -10� to 10� rotation (2);

� 360� rotation, accurate (3);

� 360� rotation, more accurate (4);

� 360� rotation, most accurate (5).

To set the Rotation Type, keep clicking the �R Type� button; the setting will switch from one to the next each time you click the button. The default setting is 360� rotation, the least accurate setting (0).

25.7.5�� Area of Interest (AOI)

Selecting an image segment is very important for training.

Use image segments for searching similar images.
Only use the whole image for exact matches.

Training requires an "Area of Interest" (AOI) or� "Set Focus", which selects a key-segment. If an AOI is not chosen, the whole image is the AOI.� Four numbers specify AOI: the upper-left corner coordinates, and the length & width. Once the segment specification is successful, a box will cover the selected area. When you look at the training image, if the selected area is not what you want, just re-select the area again and click the �Segment� button.

The default setting is the whole image; the code is (x y w h) = (0 0 0 0).�� (0000) means ignore the segment. The units are pixels.

There are two situations where you should create a new sample image out of a sample segment:

� You repeatedly use an image segment;

� The image segment is not a rectangle; say a polygon.

The Windows Paint program will help you to create an image from a segment.

When you create an image segment, please do not change the original image size.

For example, if your image is 512x512 and you want create a segment of 400x200, please paste the 400x200 segment into a 512x512 empty image.

25.7.6�� Blurring

Blurring is one of the most important search parameters and the first parameter you should adjust.

Blurring compensates for minor image changes, which are not visible to human eyes. For example, if you use software to compress an image, to change the intensity of an image, or to translate, scale, or rotate an image, the image will be distorted a bit at the pixel level. You have to set �Blurring� to compensate for this.

The Blurring setting ranges from 0 to 50. The default setting is 10. You should set the parameters in the following order:

Blurring, Internal Weight Cut, Sensitivity, External Weight Cut.

To Summarize:

When a search yields no results, increase Blurring;
When a search yields too many results, decrease Blurring.

25.7.7�� Sensitivity

The Sensitivity parameter ranges from 0 (least sensitive) to 100 (most sensitive).

To search small segment(s), use high sensitivity search.
To search large segment(s), use low sensitivity search.
The higher this parameter is, the more results you will get.

The Sensitivity parameter ranges from 0 to 100. The default is 50.

To Summarize:

When a search yields no results, increase sensitivity;
When a search yields too much result, decrease sensitivity.

�

25.7.8�� Internal/External Weight Cut

You can set the "Internal Weight Cut" (Internal Cut) or "External Weight Cut" (External Cut) to list only those retrieved images with scores or weights greater than a certain value (called Threshold).

It is better to give no answer than a wrong answer.

Assume you are searching images and all similar images have weights ranging from 1,000 to 10,000. It is possible that some other images pop up with weights ranging from 10 to 100. To eliminate these images, you can set the �External Weight Cut� to 1,000.

The Internal Cut plays a similar role as the External Cut. There are two differences between these two cuts:

� The Internal Cut ranges from 0 to 99; the External Cut can be any number;

� The Internal Cut stops the images from coming out, whereas the External Cut can bring the eliminated images back if you set the External Cut to 0. You might need to see the eliminated images sometimes for the purpose of adjusting the parameters.

To Summarize:

� Set the �Internal Cut� or �External Cut� to eliminate errors.

25.7.9�� Segment Size

The ImageFinder is currently tuned to search for large image segments (size of the whole image). It can look for small segments via the "Small Segment" setting; however, only Translation symmetry is supported for small segments.

A Customized Version can be ordered for other symmetries.

To search large segments, use setting 0.

To search small segments, use setting 1.�

For example:

If a sample segment is one quarter of the sample image, it is a large segment.
If the segment is 1/20 of the sample image, it is a small segment.

Currently, "S Segment" only supports Translation symmetry. If you need Rotation or/and Scaling symmetry, please use "L Segment".�

Other symmetries can be added in a Customized Version.

25.7.10�� Image Type

There are BW and Color images. For each of them, there are �sum-search�, �maximum-search�, and �average-search�. This generates 6 image types:

BW Sum
BW Max
BW Avg
Color Sum
Color Max
Color Avg

"BW Sum� is like an integration of function f (x).

"BW Max� is like a maximum value of f (x); and

"BW Avg� is the average of the above two.

"Color Sum� is like an integration of function f (x).

"Color Max� is like a maximum value of f (x); and

"Color Avg� is the average of the above two.

To set the image type, keep clicking the Image Type button; the setting will switch from one to the next each time you click the Image Type button.

25.7.11�� Use BioFilter & Use Neural Filter

These two parameters, �Use BioFilter� and �Use NeuralFilter�, will decide whether the BioFilter and NeuralFilter will be used before the NeuralNet Filter is used to eliminate some images.

25.7.12�� Auto Segment

The training segment can be specified in two ways:

Manual Specification

Automatic Specification

The default is Manual Specification. In this setting the segment will be specified by the four text boxes (x, y, w, h), as we discussed earlier.

If you do not want to pick up a training segment, then let the ImageFinder pick up the segment for you by using the Automatic Specification. This parameter has several settings:

NO Auto Segment

Very Large Segment

Large Segment

Medium Segment

25.7.13�� Summary

The NeuralNet Filter is hard to use because it has so many parameters. Not all parameters are equal. We divide the parameters into two groups. The beginners should use only parameters in the first group. Note that:

The most important parameters for Training are Image Processing, AOI, Symmetry, and Segment Cut.
The most important parameters for Matching are Blurring, and Sensitivity.

In a typical search, you will set these parameters and leave other parameters with default values. These are the 7 parameters you should focus on first:

Training (3 parameters):

Segment: selecting �Large AutoSeg 3� so the ImageFinder will select a training segment for you at the beginning.

� Symmetries

Segment Cut (in the Reduction Filter)

Matching (4 parameters):

Sensitivity
Blurring
External Weight Cut
Internal Weight Cut

Ignore the rest of the parameters at the beginning.

26. ImageFinder Support Service Packages

Attrasoft TransApplet Services are designed to accelerate a company's path to deploy Image Recognition Solutions. Our structured Service Offerings help you to develop your products/services with the ImageFinder as a component.

26.1�� Basic Annual Maintenance & Support Package

The Basic Annual Maintenance & Support package can be purchased. ��

ImageFinder Support Service Packages are designed to help a person/company to understand the process of integrating Image Recognition technology into their product/service. From a large set of possible paths, a person/company must plan the most realistic execution to ensure the success of the project.

The TransApplet Annual Support covers:

� Pointing to the right development path and shorten the learning curve for developers.

� Explaining all of the code in this User�s Guide.

� Preventing you from following a wrong path, thus saving you from an unnecessary waste of resources, or even failure of the project.

The TransApplet Annual Support does not cover:

� Explaining all of the code not in this User�s Guide, but in the project.

There are huge amount of codes in the project; unless the codes are explicitly listed in this User�s Guide, the explanation of those programming codes will have additional fees.

26.2�� Process

�

Image Recognition Solution Development Procedure is:

�

(1) Set up the system (you can do that);

(2) Collect the data (you can do that);

(3) Preliminary Assessment via our off-the-shelf software, the ImageFinder.

� You should get Identification Rates ranging from 60% to 89%.

� The best rate, one of our customers (without any customization) was able to obtain, was an 89% Identification Rate.

� The off-the-shelf ImageFinder has 100 open parameters for users to adjust, which is the reason customers are only able to achieve Identification Rates ranging from 60% to 89%.�

� The ImageFinder itself has 3,000+ internal parameters which users have no access to. Customization is the process of adjusting these 3,000+ internal parameters for your specific image type. This is where the high degree of accuracy (95% to 99.9%) for your specific image type is obtained.

�(4) Feasibility Project via Customized Stand-alone Software.

ATTRASOFT will develop stand-alone software, which will address the special needs / requirements of your application.
This will allow you to show your upper management the practicality of a larger project.

(5) Programming Library in .Net.

(6) System Integration to your Production Line.

(7) Licensing, Annual Maintenance & Support.

One-time License:�� 2% of the project.

Annual License:� �� 0.6% of the project.

Annual Maintenance:�� 20% of the project.

�

26.3�� What is a Feasibility Study?

Attrasoft Image Recognition technology offers a suite of products,

ImageFinder (off-the-shelf),
VideoFinder (off-the-shelf),
TransApplet (Library version of the ImageFinder),
Customized-ImageFinder,
Customized-TransApplet,

that enables the success of projects a person/company will develop with Attrasoft�s software� components.

In general, a Feasibility Study is very valuable because experience from many past projects will be naturally deployed into your project. The very first question we encountered, and you will be asked when you justify your project to your company, is �Can this be done?� Very often, the answer is more than a simple "Yes". You will want to see it works on a small scale.

26.4�� Customization �Services

If the off-the-shelf product satisfies your needs, your path will be:

Set up the system (you can do that);

Collect the data (you can do that);

Purchase the ImageFinder and make a Preliminary Assessment;

Purchase TransApplet;

System Integration Support;

License and Annual Support.

The most likely path you will go through, which requires some customization, is:

Set up the system (you can do that);

Collect the data (you can do that);

Purchase the ImageFinder and make a Preliminary Assessment;

Purchase Customization of the standalone software;

Purchase Customized TransApplet;

System Integration;

License and Annual Support.

Customization deals with problems like:

Reducing the operation Complexity;

Speed Optimization;

Internal Structure Optimization;

Graphical User Interface Customization;

Database Interface;

Web Interface;

Video Formats;

New Image Preprocessing Filters;

Customized Filters;

Programming Library;

Specific Symmetries or Combination of Symmetries;

Attrasoft can implement any symmetry (or combination of symmetries) which can be described by mathematics;

Further refinement for small image segments;

Fine Tuning of the Parameters;

Digital Image Database (Combine ImageFinder with Database);

Internet Image Search Engines;

Multi-layers of image matching;

Other Specific Needs.