About Attrasoft TransApplet

 

TransApplet is a .Net Class Library that enables System Integrators, Solution Developers, and Individuals to quickly add Image Recognition capability to their products and services. 

 

 

Software Requirements

 

The software requirements are:

 

 


 

 Installing the Software

 

Copying the “CD:\transapplet80” to the C-driver will complete the installation.

 

 

Information & Support

 

Attrasoft TransApplet

Attrasoft

P. O. Box 13051

Savannah, GA, 31406

USA

 

http://attrasoft.com

 

imagefinder@attrasoft.com (Email Subject: Attrasoft)

 

Phone:  (912) 484-1717

 

© Attrasoft 1998 - 2011

 

 


License Agreement

THIS LICENSE AGREEMENT ("AGREEMENT") IS BETWEEN YOU, THE END USER, AND Attrasoft. IT GOVERNS THE USE OF THE SOFTWARE PROGRAM AND DOCUMENTATION KNOWN AS Attrasoft TransApplet (THE "PRODUCT"). IF YOU USE THE PRODUCT, THEN YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU ARE NOT WILLING TO BE BOUND BY THE TERMS OF THIS AGREEMENT, PROMPTLY RETURN THIS PACKAGE TO THE PLACE OF PURCHASE WITH A COPY OF THE RECEIPT, AND YOUR LICENSE FEE WILL BE REFUNDED.

 

Attrasoft licenses use of the PRODUCT, which may be manufactured and distributed by Attrasoft or by a third party (either, the "Manufacturer"). You, the end-user, assume responsibility for the selection of the PRODUCT to achieve your intended results, and for its installation and subsequent use.

 

GRANT OF LICENSE

Attrasoft hereby grants you a non-exclusive license to use the PRODUCT in object code form only, upon the terms and conditions contained in this Agreement.

You may:

1. Use the PRODUCT on the number of workstations for which you have purchased PRODUCT licenses. The workstations must be owned, leased or otherwise controlled by you, whether in a network or other configuration.

2. Create a quantity of backup copies of the PRODUCT, in any machine-readable or printed form, equal to the number of PRODUCT licenses you have purchased.

3. Transfer the PRODUCT and your rights under this Agreement to another party if the other party agrees to accept the terms and conditions of this Agreement. If you transfer the PRODUCT, you must, at the same time, either transfer all copies of PRODUCT to the same party, or destroy any copies not transferred. You must immediately notify Attrasoft of the transfer.

4. Print out one copy of the Product documentation from the Attrasoft program, TransApplet, for each license purchased. If you print out any part of the Product documentation from the Attrasoft program, TransApplet, you must reproduce and include all the copyright notices that appear in the documentation on any such copy of the documentation.

 

You May Not:

1. Use or copy the PRODUCT, in whole or in part, except as expressly provided in this Agreement.

2. Use the PRODUCT concurrently on more than the number of workstations for which you have purchased licenses.

3. Copy, rent, distribute, sell, license or sub-license, or otherwise transfer the PRODUCT or this license, in whole or in part, to another party, except as specifically set forth above.

4. Incorporate the PRODUCT or any portion of the PRODUCT into, or use the PRODUCT, or any portion of the PRODUCT to develop, other software without a license from Attrasoft, or otherwise modify or create a derivative work from the PRODUCT without a license from Attrasoft.

5. Reverse engineer, decompile, or disassemble the PRODUCT.

 

To use the PRODUCT as described in Sections 2 or 4 above, or for any other use not specifically set forth above, additional licensing from Attrasoft is required. For further information, please contact Attrasoft at:

Attrasoft, Inc.     

Phone: (912) 484-1717

gina@attrasoft.com

 

PROPRIETARY RIGHTS

This Agreement gives you ownership only of the physical Attrasoft program, TransApplet, on which the PRODUCT is stored, but not of the PRODUCT itself. You acknowledge that Attrasoft owns all rights, title, and interest in the PRODUCT, and that you will acquire no rights in the PRODUCT through your use of it. You agree that you will take no action that interferes with Attrasoft's rights in the PRODUCT.

 

TERMS

This Agreement is effective until terminated. You may terminate it at any time by destroying the PRODUCT together with all copies and documentation in any form. This Agreement will also terminate automatically and without notice from Attrasoft if you fail to comply with any term or condition of this Agreement. You agree upon such termination to destroy the PRODUCT and all copies of the PRODUCT.

 

DISCLAIMER; LIMITED WARRANTY

EXCEPT AS PROVIDED BELOW, THE PRODUCT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PRODUCT IS WITH YOU. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO THE ABOVE EXCLUSION MAY NOT APPLY TO YOU. NEITHER ATTRASOFT NOR MANUFACTURER WARRANTS THAT THE FUNCTIONS CONTAINED IN THE PRODUCT WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION OF THE PRODUCT WILL BE UNINTERRUPTED OR ERROR-FREE. However, where Attrasoft is the Manufacturer, Attrasoft warrants that the Attrasoft program, TransApplet, on which the software is furnished will be free from defects in materials and workmanship under normal use for a period of ninety (90) days from the date of delivery as evidenced by a copy of your receipt.

 

LIMITATION OF REMEDIES

Where Attrasoft is the Manufacturer, Manufacturer’s entire liability and your exclusive remedy shall be:

1. The replacement of the Attrasoft program, TransApplet, not meeting the Limited Warranty, which is returned to Manufacturer with a copy of your receipt.

2. If Manufacturer is unable to deliver replacement Attrasoft program, TransApplet, which is free of defects in materials or workmanship, you may terminate this Agreement by returning the PRODUCT and a copy of your receipt to the place of purchase, and your money will be refunded. Where Attrasoft is not the Manufacturer, Attrasoft shall have no liability to replace or refund, and you agree to look to Manufacturer to meet the obligations described above.

 

LIMITATION OF LIABILITY

IN NO EVENT WILL ATTRASOFT OR MANUFACTURER BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO, ANY LOST PROFITS, LOST SAVINGS, OR OTHER INDIRECT, SPECIAL, EXEMPLARY, INCIDENTAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF THE USE OR INABILITY TO USE THIS PRODUCT, EVEN IF ATTRASOFT OR MANUFACTURER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. FURTHER, IN NO EVENT WILL ATTRASOFT OR MANUFACTURER

BE LIABLE FOR ANY CLAIM BY ANY OTHER PARTY ARISING OUT OF YOUR USE OF THE PRODUCT. SOME JURISDICTIONS DO NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THE ABOVE LIMITATION OR EXCLUSION MAY NOT APPLY TO YOU.

 

TRADEMARKS

Attrasoft is a trademark of Attrasoft, Inc. Microsoft, C# logo are registered trademarks of Microsoft Corporation. No rights, license, or interest in such trademarks is granted hereunder.

 

U.S. GOVERNMENT RESTRICTED RIGHTS IN DATA

This computer software product and documentation are provided with Restricted Rights. Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013, or subparagraphs (c)(1) and (2) of the Commercial Computer Software - Restricted Rights at 48 CFR 52.227-19, as applicable. Contractor/Manufacturer is Attrasoft, Inc.

 

EXPORT CONTROLS

You agree not to export or re-export the PRODUCT, directly or indirectly, to any countries, end-users or for any end uses that are restricted by U.S. export laws and regulations, without first obtaining permission to do so as required by the U.S. Department of Commerce's Bureau of Industry and Security, or other appropriate government agency. These restrictions change from time to time. If you have any questions regarding your obligations under U.S. export regulations, you should contact the Bureau of Industry and Security, U.S. Department of Commerce, Exporter

Counseling Division, Washington D.C. (202) 482-4811, http://www.bis.doc.gov.

 

 


TABLE OF CONTENTS

About Attrasoft TransApplet. 1

Software Requirements.. 1

Installing the Software.. 2

Information & Support. 2

License Agreement. 3

TABLE OF CONTENTS.. 6

1.   Introduction.. 13

1.1   What is TransApplet?. 13

1.2   Software Requirements. 13

1.3   Installing the TransApplet 13

1.4   Attrasoft TransApplet Class Library. 14

1.5   Linking the TransApplet 14

2.  Image Recognition Overview... 16

2.1   Attrasoft Image Recognition Basics. 16

2.2   Image Recognition Internal Structures. 16

3.  TransApplet Overview... 17

3.1   TransApplet  API. 17

3.2   User Interface. 17

3.3   Input & Display. 17

3.4   ImageFinderLite. 18

3.5   Signature Recognition. 18

3.6   Whole Image Matching. 18

3.7   Dynamic Library. 18

3.8   Image Segment Matching. 18

3.9   Videos. 18

3.10   Image Processing. 19

3.11   Customized Software. 19

4.   API 20

4.1   Image Signature. 20

4.2   Signature Filter. 22

4.3   Matching Results. 25

4.4   Whole Image Matching. 27

4.5   ImageLibrary. 28

4.6   Matching Engine. 30

4.7   NeuralNet Filter. 32

4.8   Other API. 33

5.    User Interface.. 34

5.1   How to Create a Project 34

5.2   How to Create Menus. 36

5.3   Link to Class Library. 37

5.4   Declare Objects. 38

6.   Input. 39

6.1   Class Library Name. 39

6.2   Class Library Overview.. 40

6.3   Set Up the Chapter Project 41

6.4   Link to Class Library. 42

6.5   Key Segment 43

6.6   Search Source Button. 45

7.   Image Display.. 47

7.1   Class Library Name. 47

7.2   Implementing H, >, and < Buttons. 48

7.3   Test 50

7.4   Output  Images. 50

8.   ImageFinderLite.. 51

8.1   ImageFinderLite. 51

8.2   Interface. 51

8.3   Programming Example. 53

8.4   Codes for Each of the Steps. 55

8.5   Test Run. 56

9.   Image Signatures.. 57

9.1   Signature Menu. 57

9.2   API. 58

9.3   TransApplet Objects. 61

9.4   Key Signature. 62

9.5   Signature File Concepts. 64

9.6   Signature File Implementation. 65

9.7   Examples. 65

10.   Whole Image Matching.. 67

10.1   Whole Image Matching Menu. 67

10.2   NeuralFilter API. 67

10.3   N:N Matching Implementation. 68

10.4   1:N  Matching Implementation. 69

11.   Binary Search.. 72

11.1   Signatures. 73

11.2   Search. 76

12.   Dynamic Library.. 80

12.1   Dynamic Library Menu. 80

12.2   Dynamic Library API. 81

12.3   Creating A Master Library. 82

12.4   Load Dynamic Library. 82

12.5   Library M:N Matching. 83

12.6   Library 1:N Matching. 85

12.7   Update Implementation. 86

13.   Image Variations.. 88

13.1   Variation Signatures. 88

13.2   Segments. 90

14.   Sub-Image Search.. 93

14.1   Sub-Images and Variations. 93

14.2   Search Process. 96

14.3   Learn by Example. 96

14.4   Implementation. 100

15.   Fast Sub-Image Matching.. 102

15.1   Fast Menu. 102

15.2   NeuralNet Filter API. 105

16.   Video Input. 106

16.1   Class Library Name. 106

16.2   Class Library Overview.. 107

16.3   Set Up the Chapter Project 107

16.5   Key Segment 108

16.6   Search Source Button. 109

17.   Whole Video Recognition.. 110

17.1   Project  Introduction. 110

17.2   Example. 113

17.3   Implementation. 114

18.   Video Frame Recognition.. 116

18.1   How to Use VideoFinder. 116

18.2   Implementation. 119

19.   Video Object Recognition.. 121

19.1   FrameFinder Functions. 121

19.2   Beginner Menu. 121

19.3   User Interface. 122

19.4   Implementation. 124

20.   Image Processing Concept. 125

20.1   Filters. 125

20.2   Image Preprocessing & Processing. 125

20.3   Normalization. 126

20.4   Neural Filter and Neural Net Filter. 127

20.5   Parameter Object 127

21.   Image Preprocessing.. 128

21.1   Image Preprocessing Filter. 129

21.2   PreProcessing API. 129

21.3   Enter Parameters. 131

21.4   Cut Off the Border Areas. 132

21.5   Impose a Mask. 132

21.6   Speed Up the Computation. 133

21.7   Skip the Empty Border by Content Percent 134

22.   Image Processing.. 136

22.1   Good & Bad. 136

22.2   Processing API. 138

22.3   Set Image Processing Filters. 140

22.4   First Two Settings. 141

22.5   Chapter Projects. 142

23.   Normalization.. 143

23.1   Class Library Name. 143

23.2   Class Library Overview.. 144

23.3   Link to Class Library. 145

23.4   Parameters. 146

24.   Parameter Class.. 148

24.1   Pushing Images Through Filters. 148

24.2   Predefined Objects. 149

24.3   Grouping Parameters Together. 150

24.4   Chapter Project 154

24.5   TransApplet Objects. 158

24.6   Selecting Filters. 159

24.7   Set Filter Parameters. 160

25.   Parameters.. 163

25.1   Overview.. 163

25.2   Image Preprocessing. 163

25.3   Image Processing. 165

25.3.1   Edge Filters. 165

25.3.2   Threshold Filters. 168

25.3.3   Clean-Up Filters. 169

25.4   Normalization Filter. 169

25.5   Unsupervised Filterand BioFilter. 171

25.6   Neural Filters. 173

25.7   NeuralNet Filter. 174

25.7.1   Symmetry. 175

25.7.2   Translation Type. 176

25.7.3   Scaling Type. 176

25.7.4   Rotation Type. 176

25.7.5   Area of Interest (AOI) 177

25.7.6   Blurring. 177

25.7.7   Sensitivity. 178

25.7.8   Internal/External Weight Cut 178

25.7.9   Segment Size. 179

25.7.10   Image Type. 179

25.7.11   Use BioFilter & Use Neural Filter 180

25.7.12   Auto Segment 180

25.7.13   Summary. 180

26. ImageFinder Support Service Packages.. 182

26.1   Basic Annual Maintenance & Support Package. 182

26.2   Process. 183

26.3   What is a Feasibility Study?. 183

26.4   Customization  Services. 184

   


 

1.   Introduction

 

Attrasoft TransApplet for Windows is a .Net Class Library that enables the addition of Image Recognition capability to products & services.  It can be used for:

 

 

1.1   What is TransApplet?

 

Attrasoft TransApplet for Windows is a .Net Class Library that enables the addition of Image Recognition capability to products & services. 

 

Before you continue, you should be familiar with the Attrasoft ImageFinder for two reasons:

 

 

The TransApplet will introduce the class library, “Attrasoft.TransApplet80”. This library is used in the ImageFinder. As a teaching tool, the TransApplet will show you how to build an ImageFinder via C# .Net. Therefore, if you have not learned the ImageFinder, you should start with the ImageFinder now.

 

The entire ImageFinder C# project is in the software. It is located in the ImageFinder subfolder. You can run the ImageFinder from there.

 

1.2   Software Requirements

 

The software requirements are:

 

 

In addition, we assume users understand C# or VB programming.

 

1.3   Installing the TransApplet

 

Copying the “CD:\transapplet80” to the C-driver will complete the installation. After copying, you should have both the Library Files and the Chapter Example.

 

1.4   Attrasoft TransApplet Class Library

 

The TransApplet library has many *.dll files:

 

C:\transapplet80\xyz80.dll

 

Each Library file has one main library, which has the same name as the file name:

 

Attrasoft.TransApplet80.xyz80

 

Each Library has one main class, which has the same name as the file name:

 

Attrasoft.TransApplet80.xyz80.xyz80

 

The advantage of making

 

class-name = library-name = file-name

 

is that you will know what to use. If you go to a file xyz80.dll, you will use Library, “Attrasoft.TransApplet80.xyz80”, and you will use class, xyz80. There is no guessing which class you should choose if the Library has more than one class.

 

The disadvantage is that the compiler will not be able to distinguish between the class-name and the Library-name unless you use the full name.

 

Xyz80 alone is not understandable by the C# compiler. To declare an object, you must use the full path:

 

Attrasoft.TransApplet80.xyz80.xyz80           myObject80;

 

Each chapter has a project. Many chapters may share the same project. These projects provide an integration source code so you can cut and paste to produce your own application.

 

1.5   Linking the TransApplet

 

To link to these Libraries:

 

 


 

2.  Image Recognition Overview

2.1   Attrasoft Image Recognition Basics

 

The Attrasoft Image Recognition Approach has two basic phases:

 

 

In matching whole images, a unique image signature is computed for each image. Different images will have different signatures. The Signature Matching is based on these image signatures. This matching can easily reach the speed of 100,000 to 1,000,000 whole images per second and is very fast.

 

In the Image Segment matching, image signatures may or may not be used. If no image signatures are used, the recognition speed is much slower.

 

Image Matching is further divided into filters. It is these filters that will perform the image matching tasks.

 

2.2   Image Recognition Internal Structures

 

An image recognition application is roughly divided into:

 

Level 5: User Interface;

Level 4: Data Management;

Level 3: Image-Matching Layer;

Level 2: Scanner, Camera, and their Drivers;

Level 1: PC with Windows.

 

The TransApplet Library will be used to implement the Image-Matching layer. Stand-alone software, like the Attrasoft ImageFinder, is roughly divided into 3 layers:

 

Level 3: User Interface;

Level 2: Image-Matching Layer;

Level 1: PC with Windows.

 

 

 


3.  TransApplet Overview

ImageFinder for Windows is an off-the-shelf Application Software that enables System Integrators, Solution Developers, and Individuals to quickly test their own Image Recognition ideas.

 

VideoFinder for Windows is an off-the-shelf Application Software that enables System Integrators, Solution Developers, and Individuals to quickly test their own video Recognition ideas.

 

TransApplet is .Net Class Library that enables System Integrators, Solution Developers, and Individuals to quickly add Image Recognition capability to their products and services.

 

This User’s Guide will show you how to build software like the ImageFinder using C# .Net. If you have not learned how use to the ImageFinder, you should start with the ImageFinder now.  The entire ImageFinder project is located in the ImageFinder sub-folder.

 

The TransApplet consists of a set of class libraries. Each chapter will introduce one of these libraries as you continue to build the “ImageFinder”. Some minor graphical user interface issues will be ignored here because it is already complicated enough.

 

3.1   TransApplet  API

 

Chapter 4, API, describes the API (Application Programming Interface) of the “Attrasoft.TransApplet80” Class Library.

 

3.2   User Interface

 

Chapter 5, User Interface, briefly describes how to set up the forms and the menu bar for the ImageFinder, and also how to link to class libraries.

 

3.3   Input & Display

 

Chapter 6, Input, briefly describes how to enter data to the ImageFinder. The input images are further divided into:

 

 

An image path specifies the training image. A search-folder specifies the Search Source.

 

Chapter 7 will also briefly describe how to display the images. The Image Display will play the following role:

 

 

3.4   ImageFinderLite

 

Chapter 8, ImageFinderLite, briefly describes how to build the software, ImageFinderLite.

 

3.5   Signature Recognition

 

Chapter 9, Image Signatures, will describe how to generate image signatures.

 

3.6   Whole Image Matching

 

Chapter 10, Whole Image Matching, will describe the steps for whole image matching.

 

Chapter 11, Binary Search, will describe the steps for binary whole image matching.

 

3.7   Dynamic Library

 

Chapter 12, Dynamic Library, introduces the dynamic library used in the ImageFinder. Dynamic Library allows the master library to be updated via insertion, deletion, and update.

 

3.8   Image Segment Matching

 

Chapter 13, Image Variation, will describe how to generate image signatures for some simple variations, such as rotation and flips.

 

Chapter 14, Sub-Image Matching, introduces the sub-image matching used in the ImageFinder.

 

Chapter 15, Fast Sub-Image Matching, gives a more options for matching sub-images.

 

3.9   Videos

 

Chapter 16, Video Input, briefly describes how to enter video data to the ImageFinder.

 

Chapter 17, Whole Video Matching, introduces matching with a whole video as a unit.

 

Chapter 18, Video Frame Matching, introduces matching with a video frame as a unit.

 

Chapter 19, Video Object Matching, introduces matching with an object inside a video frame as a unit.

 

 

3.10   Image Processing

 

Chapter 20, Image Preprocessing Concept, chapter 21, Image Preprocessing and chapter 22, Image Processing, introduces the image processing process required for the ImageFinder.

 

Chapter 23, Normalization, introduces image representation required for the ImageFinder.

 

Chapter 24 and Chapter 25 discuss the internal parameters in the ImageFinder.

 

3.11   Customized Software

 

Chapter 26, Development Process, introduces the Image Recognition Implementation Process using the TransApplet.


 

 

4.   API

 

This chapter will introduce the interfaces of the important classes and important functions. The interface for other classes will be introduced in the later chapters as they are used.

 

4.1   Image Signature

 

Whole Image Matching is done through Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature.

 

This section introduces the Image Signature interface. An Image Signature consists of the following attributes:

 

ID

Name (For example, xyz.jpg)

Path (c:\abc\)

Attribute 1

Attribute 2

 

The interface for Image Signature is:

 

public interface I_ImageSignature

            {

                        int getStatus ();

                        string getID();

                        string getImageName ();

                        string getImagePath();

                        string getAbsolutePath ();

                        int getNumberOfAttributes ();

                        int [] getSignatureAttributes();

                        int getSignatureAttributes (int index);

                        string toString ();

            }

 

The following table lists the functions.

 

Function

Descriptions

Comments

int getStatus ()

Returns the status associated with the signature.

Output:

1: signature ready.

0: signature not ready.

-1: no image

-2: image segmentation specification error

-3: other error.

 

 

 

 

 

string getID()

Returns the ID associated with the signature.

Output: image ID.

 

string getImageName ()

Returns the image name associated with the signature.

Output: image name.

 

string getImagePath()

Returns the image path associated with the signature.

Output: image path.

 

string getAbsolutePath ()

Returns the absolute path associated with the signature.

Output: image absolute path.

 

 

int getNumberOfAttributes ()

Returns the number of attributes associated with the signature.

Output: number of attributes.

 

int [] getSignatureAttributes()

Returns the attribute array associated with the signature.

Output: attribute array.

 

 

int getSignatureAttributes() (int index)

Returns the attribute associated with the input index.

Input: index.

Output: attribute associated with the input index.

 

 

string toString ()

Returns the entire image signature as a string with fields separated by Tab.

 

 

4.2   Signature Filter

 

Signature filter computes signatures in the TransApplet; the input is an image and the output is a signature.

 

The filter has two classes: Signature Generator and Signature Filter. The interface for Signature Generator is:

 

public interface I_SignatureGenerator

            {

                        bool setSignatureFilter (int x);

                        int getSignatureFilter ();

 

                        ImageSignature getSignature (Bitmap b);

                        int getSignatureFilterSize ();

 

                        string [] getSignatureFilterNames();

            }//class

 

The following table lists the functions.

 

Function

Descriptions

Comments

ImageSignature getSignature (Bitmap b)

Gets the signature of the input image.
Input:

   Bitmap b

Output:

   Signature.

 

int getSignatureFilterSize ()

Gets the number of attributes the library of signatures.

 

bool setSignatureFilter (int x)

 

int getSignatureFilter ()

 

Selects a Signature Generator.

 

 

The interface for Signature Filter is:

 

public interface I_SignatureFilter

            {

                        bool setSignatureFilter (int x);

                        int getSignatureFilter ();

 

                        string [] getSignatureFilterNames();

 

                        ImageSignature getSignature (string imagePath, string ID);

                        ImageSignature getSignature (string imagePath);

 

                        ImageSignature getSignature (Bitmap b, string ID);

                        ImageSignature getSignature (Bitmap b);

 

                        ImageSignature getSignature

( string imagePath, string ID,int x, int y, int w, int h);

                        ImageSignature getSignature

( Bitmap bImg, string path, string name, string ID,

int x, int y, int w, int h);

                       

                        bool getLibrary ( string [] imageAbsolutePath, string fileName );

                        bool getLibrary

( string [] imageAbsolutePath, string [] IDs, string fileName );

                       

                        bool getSegmentLibrary ( string [] imageAbsolutePath,

                                    string [] IDs,

                                    string [] xs,

                                    string [] ys,

                                    string [] ws,

                                    string [] hs,

                                    string fileName );

                       

                        bool getSegmentLibrary ( string [] imageAbsolutePath,

                                    string [] IDs,

                                    int [] xs,

                                    int [] ys,

                                    int [] ws,

                                    int [] hs,

                                    string fileName );       

            }//class

 

The following table lists the functions.

 

Function

Descriptions

Comments

bool setSignatureFilter (int x)

 

int getSignatureFilter ()

 

Selects a Signature filter.

 

string [] getSignatureFilterNames();

Gets a list of Signature filter names.

 

ImageSignature getSignature (string imagePath, string ID);

           

ImageSignature getSignature (string imagePath);

 

ImageSignature getSignature (Bitmap b, string ID);

 

ImageSignature getSignature (Bitmap b);

Gets the Signature of the input image.

 

ImageSignature getSignature

( string imagePath, string ID,int x, int y, int w, int h);

 

ImageSignature getSignature

( Bitmap bImg, string path, string name, string ID,

int x, int y, int w, int h);

Gets the Signature of the input image segment.


Input:

   string imagePath, or

   Bitmap bImg

   string ID

   int x, int y, int w, int h.

Output:

   Signature.

 

bool getLibrary

 ( string [] imageAbsolutePath,

string fileName );

 

bool getLibrary

( string [] imageAbsolutePath,

string [] IDs,

string fileName );

                       

bool getSegmentLibrary ( string [] imageAbsolutePath,

string [] IDs,

string [] xs,

string [] ys,

string [] ws,

string [] hs,

string fileName );

                       

bool getSegmentLibrary ( string [] imageAbsolutePath,

string [] IDs,

int [] xs,

int [] ys,

int [] ws,

int [] hs,

string fileName );

Generates a Signature library from all images in string [] imageAbsolutePath and produces a file that contains all signatures.

 

Input:

   string []   imageAbsolutePath

 

Output:

   A text file that contains the library of signatures.

 

 

4.3   Matching Results

 

Attrasoft ImageFinder matches whole images or image segments. The ImageFinder can be used for:

 

 

The results for N:N Matching always goes to a file. The results for 1:1 and 1:N Matching go to a data structure called Results_1N. The interface for class, Results_1N, is:

 

public interface I_Results_1N

                        {

                                    bool getStatus ();

                       

                                    int getNumberOfMatches();

 

                                    string getImageID (int i) ;

                                    string [] getImageID () ;

                                    string [] getImageID_N ( int N );

                                   

                                    string getScore(int i);

                                    string [] getScore();

                                    string [] getScore_N ( int N );

 

                                    string getImageName (int i) ;

                                    string [] getImageName () ;

                                    string [] getImageName_N ( int N ) ;

 

                                    string getImagePath (int i) ;

                                    string []getImagePath () ;

                                    string [] getImagePath_N ( int N ) ;

 

                                    string getX(int i);

                                    string [] getX();

                                    string [] getX_N ( int N );

 

                                    string getY(int i);

                                    string [] getY();

                                    string [] getY_N ( int N );

 

                                    string getW(int i);

                                    string [] getW();

                                    string [] getW_N ( int N );

 

                                    string getH(int i);

                                    string [] getH();

                                    string [] getH_N ( int N );

 

                                    Results_1N sort ();

 

                                    string toString ();

                        }

 

The following table lists the functions.

 

Functions

Descriptions

Comments

int getStatus ()

Returns the status of the current signature comparison:

> 0: OK;

< 0: Error.

 

int getNumberOfMatches()

Returns the number of matches of the current signature comparison.

 

string getImageID (int i) string [] getImageID ()

string [] getImageID_N

( int N )

Returns the matching IDs of the current signature.

 

string getScore(int i)

string [] getScore()

string [] getScore_N

( int N )

Returns the matching scores of the current signature.

 

string getImageName

(int i)

string [] getImageName ()

string [] getImageName_N ( int N )

Returns the matching names of the current signature.

 

string getImagePath (int i)

string []getImagePath ()

string [] getImagePath_N

 ( int N )

Returns the matching paths of the current signature.

 

string getX(int i)

string [] getX()

string [] getX_N ( int N )

string getY(int i)

string [] getY()

string [] getY_N ( int N )

string getW(int i)

string [] getW()

string [] getW_N ( int N )

string getH(int i)

string [] getH()

string [] getH_N ( int N )

Returns the matching (x, y, w, h) of the current signature.

 

string toString ()

Returns the entire results as a string with fields separated by Tab.

 

 

4.4   Whole Image Matching

 

NeuralFilter also matches two whole images. The interface for NeuralFilter is:

 

public interface I_NeuralFilter

            {

bool training ( string a1_txt, string match_txt);

 

                        Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

                                    ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

                                    Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

 

                        Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(string path1, string path2);

                        Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(Bitmap left, Bitmap right);

                       

                        Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

                                    (Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,

 string a1File, string b1File);

                        Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(string keyuPath, string a1File, string b1File);

                        Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Bitmap keyImage, string a1File, string b1File);

                       

                        bool findMatchNN (string a1File, string b1File);

                        bool findMatchNM (string a1File, string a2File, string b1File);

            }

 

 

The following table lists the functions.

 

Functions

Descriptions

bool training ( string a1_txt, string match_txt)

Trains the BioFilter.

Attrasoft.TransApplet80.Results_1N .Results_1N

findMatch11

 ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

 

Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(string path1, string path2);

 

Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (Bitmap left, Bitmap right);

Makes a 1:1 matching

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,  string a1File, string b1File);

 

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (string keyuPath, string a1File, string b1File)

 

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Bitmap keyImage, string a1File, string b1File)

Makes a 1:N matching

bool findMatchNN (string a1File, string b1File);

bool findMatchNM (string a1File, string a2File, string b1File);

Matches all image signatures in file a1File against all image signatures in a1File or a2File and saves the results to b1File.

 

4.5   ImageLibrary

 

Dynamic Library allows the library (N images in a 1:N Matching) to be updated via insertion, deletion, and update.

 

The interface for Image Library filter is:

 

public interface I_ImageLibrary

            {

                        string getLibraryID ();

                        bool setLibraryID (string x);

 

                        bool load ();

                        bool load ( string fileName);

                        bool load ( string fileName1,  string fileName2);

 

                        bool print ( );

 

                        bool clear();

 

                        bool backup ();

                        bool backup (string fileName);

 

                        bool addSignature (ImageSignature sig);

                       

                        bool deleteSignature (ImageSignature sig);

                        bool deleteSignature (string ID);

                        bool replaceSignature (ImageSignature sig);

                        bool mergeLibrary (string libFile1, string libFile2, string outputLib);

 

            }

 

The following table lists the functions.

 

Functions

Descriptions

Comments

int getLibraryID ()

Gets the Library ID (optional).

 

void setLibraryID (string x)

Sets the Library ID (optional).

 

bool load ()

Loads the default master library, lib1.txt.

 

bool load ( string fileName)

Loads master library specified by fileName.

 

bool load ( string fileName1,  string fileName2)

Loads two libraries specified by fileName1 and fileName2.

 

bool clear()

Clears the current library from RAM only.

 

bool backup ()

Saves current library to the default file, lib1_bk.txt.

 

bool backup (string fileName)

Saves current library to a back file.

 

bool addSignature (ImageSignature sig)

 

Adds a signature to a loaded image library in RAM.

 

bool deleteSignature (ImageSignature sig)

 

Deletes a signature from a loaded image library in RAM.

 

bool deleteSignature (string ID)

 

Deletes a signature from a loaded image library in RAM.

 

bool replaceSignature (ImageSignature sig)

Replaces a signature from a loaded image library in RAM.

 

bool mergeLibrary

(string libFile1, string libFile2, string outputLib)

Merges two signature libraries into a single library.

Input:

   string libFile1

   string libFile2

   string outputLib

Output:

   A text file that contains the library of signatures from both input libraries.

 

 

4.6   Matching Engine

 

The interface for the Neural Filter Matching Engine is:

 

public interface I_MatchingEngine

            {

                         int getLowCut();

                         void setLowCut(int x);

                         

                        int getHighCut();

                         void setHighCut(int x);

                       

                        void setNeuralOutputType(int x);

                         int getNeuralOutputType();

 

                         void setThreshold(int x);

                         int getThreshold();

 

                          void setSensitivity(int x);

                          int getSensitivity();

 

                          void setBlurring(int x);

                          int getBlurring();

                       

                          void setUseRelativeScore(int x);

                          int getUseRelativeScore();

 

                        bool setLibrary

(Attrasoft.TransApplet80.ImageLibrary80.ImageLibrary lib);

                       

                        bool setSignature

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig);

 

bool setNFTraining

(Attrasoft.TransApplet80.MatchingEngineTraining80.MatchingEngineTraining80 a_1);

                       

bool setBFTraining

(Attrasoft.TransApplet80.MatchingEngineTrainingBF80.MatchingEngineTrainingBF80 bf80_1);

 

                        Attrasoft.TransApplet80.Results_1N.Results_1N findMatch11

                                    ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

                                    Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

                       

 

                        Attrasoft.TransApplet80.Results_1N.Results_1N  findMatch ();

                        Attrasoft.TransApplet80.Results_1N.Results_1N findMatch

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig);

            }

 

The following table lists the functions.

 

Functions

Descriptions

Comments

int getLowCut()

void setLowCut(int x)

                         

int getHighCut()

void setHighCut(int x)

                       

void setNeuralOutputType

(int x)

int getNeuralOutputType()

 

void setThreshold(int x)

int getThreshold()

 

void setSensitivity (int x)

int getSensitivity()

 

void setBlurring (int x)

int getBlurring()

                       

void setUseRelativeScore

(int x)

int getUseRelativeScore()

Gets and Sets matching engine parameters.

 

bool setLibrary

(ImageLibrary lib)

Sets the library to be used in matching signatures.

 

bool setSignature

(ImageSignature sig)

Sets the signature for matching.

 

bool setNFTraining

(MatchingEngineTraining80 a_1)

 

bool setBFTraining (MatchingEngineTrainingBF80 bf80_1)

Sets the training objects for matching.

 

Results_1N findMatch11

(ImageSignature sig1,

ImageSignature sig2);

Gets 1:1 match results for sig1 and sig2.

 

Results_1N  findMatch ()

Gets the match results for the signature, sig, specified by function, setSignature

(ImageSignature sig)

Output:

   Results_1N.

 

Results_1N findMatch (ImageSignature sig)

Gets the match results for the signature, sig.

Input:

   ImageSignature sig;

Output:

   Results_1N.

 

 

 

4.7   NeuralNet Filter

 

The NeuralNet Filter matches a segment of an image(s). The interface for the NeuralNet Filter Matching Engine is:

 

public interface I_NeuralNetFilter

            {                               

                        bool train (Bitmap img);

                        bool train (string sPath);

                        bool train (Bitmap img, int x, int y, int w, int h);

                        bool train (string sPath, int x, int y, int w, int h);

 

                        bool retrain (Bitmap img);

                        bool retrain (string sPath);

                        bool retrain (Bitmap img, int x, int y, int w, int h);

                        bool retrain (string sPath, int x, int y, int w, int h);

 

                        Attrasoft.TransApplet80.Results_1N.Results_1N

                                    findMatch11 ( Bitmap img1);

                        Attrasoft.TransApplet80.Results_1N.Results_1N

                                    findMatch11 ( string path1);

                        Attrasoft.TransApplet80.Results_1N.Results_1N

                                    findMatch1N ( string [] fileList);

 

                        bool findMatchNN (string [] fileList, string c1File);

                        bool findMatchNM (string [] keyList, string [] libraryList, string c1File);

}

 

The following table lists the functions.

 

Functions

Descriptions

bool train (Bitmap img)

bool train (string sPath)

bool train (Bitmap img, int x, int y, int w, int h)

bool train (string sPath, int x, int y, int w, int h)

Trains the neural net.

bool retrain (Bitmap img)

bool retrain (string sPath)

bool retrain (Bitmap img, int x, int y, int w, int h)

bool retrain (string sPath, int x, int y, int w, int h)

Retrains the neural net.

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch11 ( Bitmap img1);

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch11 ( string path1);

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch1N ( string [] fileList);

Makes a 1:N Matching.

 

bool findMatchNN (string [] fileList, string c1File)

 

bool findMatchNM (string [] keyList, string [] libraryList, string c1File);

Makes a N:N Matching and N:M Matching.

 

4.8   Other API

 

We will introduce other API’s in later chapters as we encounter them.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5.    User Interface

 

This User’s Guide is written as if you will create your own version of the ImageFinder. We will do a step-by-step instruction for implementing an “ImageFinder”. Eventually, you will have your own “ImageFinder”.

 

This chapter briefly describes how to set up the form and menu bar for the ImageFinder. We will use C# .Net as the implementation language. We will build the user interface form with a Menu Bar and Input buttons.

 

5.1   How to Create a Project

 

To create a new C# project:

 

  1. Start the Visual Studio .Net, (see Figure 5.1).
  2. Click File č New Project command. The New Project dialog box is displayed (Figure 5.2).
  3. Language and Templates: Highlight the Visual C# project folder in the Project Type list to display the templates that are available for C#. Then, highlight the Windows Application template (Figure 5.2).
  4. Location and Name: Enter a name for the project and select the location for the project. A folder with the same name as the project is automatically added to the location you specify. We will use chap4 as the project name.
  5. Click the ‘OK’ button in Figure 5.2 to start the new project (Figure 5.3).

 

Now go to the property window and set (Figure 5.4):

 

 

 

 

Figure 5.1  Create C# Project.

 

 

Figure 5.2  Select Project Templates.

 

 

 

Figure 5.3  Form1.

 

 

Figure 5.4  Set Form1.Size.

 

5.2   How to Create Menus


To add a menu bar to a form:

  1. Add a “MenuStrip” or “MainMenu” control to a form from the toolbox. The control will appear in the component Design Tray at the bottom of the Form Design window. The menu of a form is determined by the Menu property.

 

  1. To add items to the menu, click the “MenuStrip” or “MainMenu” control and click wherever it says “Type Here” in the menu designer. Then, type the text of the menu item and press the Enter key. Additional entry areas appear below and to the right of the entry.

 

  1. To edit the property for each menu item, right-click the menu and select the Property command.

 

5.3   Link to Class Library

 

 

 

Figure 5.5   References in the Solution Explorer.

 

In this section, XYZ denotes any class located in c:\transapplet80\.

To use the class library, “Attrasoft.TransApplet80.XYZ”:

 

(1) Add a “using” statement:

using Attrasoft.TransApplet80.XYZ;

 

(2) Right click References and select Add reference in the Solution Explorer (Figure 5.5);

(3) Use the Browse button to find “XYZ.dll”; highlight the library under “Selected Components” and click the “OK” button.

 

Now the class library, “Attrasoft.TransApple80.XYZ”, is ready to be used. To make sure the link is successful, you should see XYZ under References in the Solution Explorer.

 

5.4   Declare Objects

 

Each Library has one main class, which has the same name as the Library-name:

 

Attrasoft.TransApplet80.xyz80.xyz80.

 

The advantage of naming the class-name as the library-name is that you will know what to use, i.e.

 

Attrasoft.TransApplet80.xyz80.xyz80.

 

There is no guessing which class you should choose if the Library has more than one class.

 

The disadvantage is the compiler will not be able to distinguish between the class-name and the Library-name unless you use the full name.

 

To declare all TransApplet objects using the full class-name like this:

 

public Attrasoft.TransApplet80.xyz80.xyz80  myObj80

            = new Attrasoft.TransApplet80. xyz80.xyz80 ();

 


6.   Input

 

This chapter describes how to enter data to the ImageFinder. The input images are further divided into:

 

 

An Image Path specifies the training image. A Search-Folder specifies the Search Source.

 

This chapter will introduce several class libraries,

 

Attrasoft.TransApplet80.InputModule80,

Attrasoft.TransApplet80.InputList80,

Attrasoft.TransApplet80.GUI.

 

The chapter project is in “.\ transapplet80.chap6”.

 

6.1   Class Library Name

 

The class library is:

 

Attrasoft.TransApplet80.InputModule80.

 

The main class in this library will be:

 

Attrasoft.TransApplet80.InputModule80. InputModule.

 

The interface is:

 

public interface I_ InputModule80

            {

 //Input:

    bool setDirList(string sInput);

        bool setSubDirList(string sInput);

        bool setFileList(string sInput);

        bool setFileSegmentList(string sInput);

        bool setFileSegmentKeyList(string sInput);

        bool setAccessList(string sInput, string sSQL);

        bool setAccessSegmentList(string sInput, string sSQL);

  

//Output:    

        Attrasoft .TransApplet80 .InputList80 .InputList  getList();                

            }

This class has two parts:

 

 

The input can be:

 

 

Output is an object of the following class:

 

public class InputList

    {

        public int status = 0;

        public string[] imageAbsoultePath = null;

 

        public string[] imageID = null;

        public string[] imageName = null;

        public string[] imagePath = null;

 

        public string[] imageX = null;

        public string[] imageY = null;

        public string[] imageW = null;

        public string[] imageH = null;

 

        public string[] key0 = null;

        public string[] key1 = null;

        

    }

6.2   Class Library Overview

 

The functions in classes are listed in the following table.

 

Function

Description

string [] setDirList ( string sInput)

To get a string list of the absolute paths of all images in directory, set the input directory, sInput.

string [] setSubDirList ( string sInput)

To get a string list of the absolute paths of all images in all sub-directories, set the input directory, sInput.

string [] setFileList ( string sInput)

To get a string list of the absolute paths of all images in file, set the input file, sInput.

string []  getFileSegmentList

( string sInput)

To get a string list of the absolute paths of all image segments in file, set the input file, sInput.

string [] getAccessList

( string sInput, string sSQL)

To get a string list of the absolute paths of all images in access file, sInput, specified by a SQL statement, sSQL, set the two input variables.

string []  getAccessSegmentList

( string sInput, string sSQL)

To get a string list of the absolute paths of all image segments in the access file, sInput, specified by a SQL statement, sSQL, set the two variables.

Attrasoft .TransApplet80 .InputList80 .InputList  getList()

After the input is specified, get the output.

 

 

6.3   Set Up the Chapter Project

 

 

Figure 6.1  Chapter 6 Project.

 

We will add the following controls to our form:

           

Key Button, Key Text Box

Use the “Key” button to display the selected key image and print the image path in the key text box. For example, click the button, go to the directory, "C:\transapplet80\ex_label", and then select an image. The selected image will be shown in the ImageFinder to indicate the specification is successful and the key-image is ready for training or retraining.

 

Source Button, Source Text Box

Use the “Source” button to select a search source and display the source in the text box.

 

Clear Button

Use the “Clear” button to clear the rich text box.

 

Rich Text Box

            Use the Rich Text Box to display messages.

 

Key Picture Box

Use the Key Picture Box to display key images.

 

Source Picture Box

Use the Source Picture Box to display images in the search source.

 

OpenFileDialog

        Use the OpenFileDialog to select a key image.

 

FolderBrowserDialog

        Use the FolderBrowserDialog to select a source path.

 

After adding these controls, the form will look like Figure 6.1.

 

6.4   Link to Class Library

 

Figure 6.2  Link to Class library.

 

To include the class library to the project,

 

 

To declare an object, please use the full path for class:

 

internal Attrasoft.TransApplet80.InputModule80.InputModule

inputModuleSource;  

 

inputModuleSource

= new Attrasoft.TransApplet80.InputModule80.InputModule(this.richTextBox1);

 

Now the object is ready to use. In addition, we will add several class libraries:

 

            Helper80;

            GUI;

            InputList80.

 

6.5   Key Segment

 

We will add a class, “Input”, in the project. We will implement the input buttons in this class. In this way, the main class is simpler. We will declare an object, input, as follows:

 

Input input;

 

      private void Form1_Load(object sender, System.EventArgs e)

            {

            h80 = new Attrasoft.TransApplet80.Helper80.Helper80(this.richTextBox1);

 

            inputModuleSource

= new Attrasoft.TransApplet80.InputModule80.InputModule(this.richTextBox1);

 

            guiSource = new Attrasoft.TransApplet80.GUI.GUI_Key(this.pictureBox2, h80);

            guiResults = new Attrasoft.TransApplet80.GUI.GUI_Results(this.pictureBox3, h80);

 

            input = new Input(this);

                       

            }

 

The helper80 object is used to print a message:

 

void appendText (string s)

                        {

                                    f.h80 .appendText ( s);

                        }

 

            void appendLine (string s)

                        {

                                    appendText ( s +"\n");

                        }

 

We can double click the Key Segment buttons to complete the programming:

 

private void button1_Click(object sender, System.EventArgs e)

                        {

                                    input.keySegment();

                        }

 

which in turn, calls a function:

 

            public bool keySegment()

                        {

                                    if ( f.openFileDialog1.ShowDialog () != DialogResult.OK )

                                                return false;

 

                                    string fileName = f.openFileDialog1.FileName;

                                    try

                                    {

                                                bTrain =new Bitmap (fileName);

                                                f.pictureBox1.Image = bTrain;

                                                f.textBox1.Text = fileName ;

                                                f.richTextBox1.AppendText

( "Display Key Segment:\n " + fileName +"\n");

                                    }

                                    catch

                                    {

                                                appendText ("Invalid key image !\n");

                                                return false;

                                    }

                                    return true;

                        }//keySegment()

 

The first section of code opens a file dialog so the user can select a file:

 

if ( f.openFileDialog1.ShowDialog () != DialogResult.OK )

                        return false;

 

The next section of code gets the selected key:

 

string fileName = f.openFileDialog1.FileName;

 

The last section of code creates an image and displays the image:

 

bTrain =new Bitmap (fileName);

            f.pictureBox1.Image = bTrain;

            f.textBox1.Text = fileName ;

            f.richTextBox1.AppendText ( "Display Key Segment:\n " + fileName +"\n");

 

6.6   Search Source Button

 

Now, we implement the “Source” button:

 

private void button2_Click(object sender, System.EventArgs e)

                        {

                                    input.source( );

                        }

 

which in turn, calls the following function in the Input class:

 

         internal bool source()

        {

            try

            {

               if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

                    return false;

                string cDir = f.folderBrowserDialog1.SelectedPath;

                f.textBox2.Text = cDir;

 

                f.inputModuleSource.setSubDirList(cDir);

 

                f.guiSource.setInput(f.inputModuleSource.getList());

                if (!f.guiSource.getStatus())

                {

                    f.appendText("Data Source fails!\n");

                    return false;

                }

                f.guiSource.setLists();

            }

            catch (Exception e)

            {

                f.appendText(e.ToString() + "\n");

                return false;

            }

            return this.f.guiSource.Button_First(false);

        }

 

 

The first section of code gets the current search directory via the folder browser dialog:

 

               if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

                    return false;

                string cDir = f.folderBrowserDialog1.SelectedPath;

                f.textBox2.Text = cDir;

 

The next section enters the selected folder to the InputModule object:

 

                f.inputModuleSource.setSubDirList(cDir);

 

The third section gets the string list of the image paths:

 

f.inputModuleSource.getList().

 

And passes this list to the display section:

 

 f.guiSource.setInput(f.inputModuleSource.getList());

 

The last section displays the first images:

 

            f.guiSource.setLists();

f.guiSource.Button_First(false);

 

To test, run the software and click the first image in folder, “c:\transapplet80\ex_label”.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

7.   Image Display

 

This chapter describes how to display images in a search source. To display these search images, three buttons will do:

 

First Button

Use the “First” button to display the first image in the search source.

 

> (Next) Button

Use the “Next” button to display the Next image in the search source.

 

< (Previous) Button

Use the “Previous” button to display the Previous image in the search source.

 

After adding the three buttons, the form looks like Figure 7.1.

 

 

Figure 7.1  First, Next (>), and Previous (<) Buttons.

 

7.1   Class Library Name

 

The class library is:

 

Attrasoft.TransApplet80.GUI.

 

It has three libraries:

 

new Attrasoft.TransApplet80.GUI.GUI_Key

new Attrasoft.TransApplet80.GUI.GUI_Source

Attrasoft.TransApplet80.GUI.GUI_Results

 

They are designed for:

 

            Key;

            Source;

            Results.

 

GUI_Key is the base class of GUI_Source.

GUI_Source is the base class of GUI_Results.

At this point, GUI_Key and GUI_Source are the same.

All three class libraries function in a similar way.

 

The GUI objects are:

 

            guiSource = new Attrasoft.TransApplet80.GUI.GUI_Key(this.pictureBox2, h80);

            guiResults = new Attrasoft.TransApplet80.GUI.GUI_Results(this.pictureBox3, h80);

 

The functions in this class are:

 

Function

Description

Bool setInput ( ImageList80)

Sets an input list of image paths, image ID, ….

void setList ( )

Sets the class ready for the first button, next button, and previous button.

bool getStatus ()

Gets the status.

Bool Button_First(bool);

Displays the first image.

Bool Button_Next(bool);

Displays the next image.

Bool Button_Previous(bool);

Displays the previous image.

 

The sample codes are:

 

Input

            f.guiSource.setInput(f.inputModuleSource.getList());

 

Computation

            f.guiSource.setLists();

 

Display

f.guiSource.Button_First(false);

f.guiSource.Button_Next(false);

f.guiSource.Button_Previous(false);

 

7.2   Implementing H, >, and < Buttons

 

In this section, we will implement the First, Next, and Previous buttons. To see images in a search source, start with the “First” button and then, keep clicking the “Next” button to display the next image; eventually, all of the images in the search source will be shown.

 

The code for the three buttons consists of three parts:

 

 

First of all, the object has to be populated. This is accomplished by the source button in the last chapter:

 

                f.guiSource.setInput(f.inputModuleSource.getList());

                if (!f.guiSource.getStatus())

                {

                    f.appendText("Data Source fails!\n");

                    return false;

                }

 

 Then we will set up the object:

 

              f.guiSource.setLists();

 

Finally, the code for the First, Next, and Previous buttons are:

 

        private void button3_Click(object sender, EventArgs e)

        {

            bool b = this.guiSource.Button_First(true);

            if (!b)

                setText(guiSource.getMessage());

        }

 

        private void button5_Click(object sender, EventArgs e)

        {

            bool b = guiSource.Button_Next();

            if (!b)

                setText(guiSource.getMessage());

        }

 

        private void button6_Click(object sender, EventArgs e)

        {

            bool b = guiSource.Button_Previous();

            if (!b)

                setText(guiSource.getMessage());

        }

7.3   Test

 

This section tests the project:

 

(1)               Run the project;

(2)               Click the “Key” button, go to “c:\transapplet80\ex_wheel\”, and select a key image;

(3)               Click the “Source” button, go to “c:\transapplet80\ex_wheel\”, and select an image.

(4)               Click the “First” button;

(5)               Click the “Next” button several times;

(6)               Click the “Previous” button several times.

 

7.4   Output  Images

 

In the last a few sections, we demonstrated how to display Input Source images. After a matching, the output images will be displayed in a similar fashion.

 

At the end of a 1:1 or 1:N matching, a Result object will be produced. This object will be entered into GUI_Result object:

 

guiResults.outputResults(results_1N, softwareDir);              

 

To display result images:   

 

guiResults.Button_First(false);

guiResults.Button_Next(false);

guiResults.Button_Previous(false);

 

 


8.   ImageFinderLite

 

In this chapter, we will briefly describe how to build a quick and dirty image recognition application. In particular, we will discuss how to identify a full image.

 

8.1   ImageFinderLite

 

Attrasoft.TransApplet80.ImageFinderLite is a component Attrasoft TransApplet. To use the ImageFinderLite object, you will need 5 steps:

 

Step 1. Set the Folder that contains the images you want to search through.

Step 2. Create a Image Signature Library from the folder.

Step 3. Load the Signature Library into the ImageFinderLite object.

Step 4. Get a key image file name, which will be matched against the library.

Step 5. Make a 1:N search.

 

8.2   Interface

 

The ImageFinderLite interface is given below:

 

public interface  I_ImageFinderLite

    {

        //variables:

        //1. Foldername (string)

        //2. Library Name (string)

        //3. KeyImage (string)

        //4. Signature (Attrasoft object)

        //5. Result (Attrasoft object)

 

        //step 1 Set Library Folder

        bool setFolder(string folderName);

        string getFolder();

 

        //step 2 Create Image Signature Library

        bool computeSignatureLibrary();

        bool setLibraryName(string a1File);

        string getLibraryName();

 

        //setp 3 Load Signature Library

        bool loadLibrary();

 

        //step 4 Get key file name

        bool setKeyImage (string key);

        string getKeyImage();

 

        //step 5 search

        bool  search ( );

        bool setOutputFileName (string b1File);

        string getOutputFileName ( );

}

 

Step 1. Set the Folder that contains the images you want to search through.

 

bool setFolder(string folderName)

string getFolder()

 

Use setFolder (string folderName) to set the folder that contains the images you want to search through; use getFolder() to see the current setting.

 

 

 

Step 2. Create Image Signature Library from the folder.

 

bool computeSignatureLibrary()

 

Use computeSignatureLibrary() to get a file that contains the image signatures from the specified folder.

 

bool setLibraryName(string a1File)

string getLibraryName()

 

Use setFolder setLibraryName(string a1File) to set the image signature file name that contains the images you want to search through. The default file is “.\data\a1.txt”, where “.” is the application folder. Use getLibraryName() to see the current setting.

 

 

 

 

Step 3. Load the Signature Library into the ImageFinderLite object.

 

bool loadLibrary();

 

Use loadLibrary() to load the Signature Library into the ImageFinderLite object.

 

 

 

Step 4. Get a key image file name, which will be matched against the library.

 

bool setKeyImage (string key)

string getKeyImage()

 

Use setKeyImage (string key) to set the unknown image that you want to search; use getKeyImage() to see the current setting.

 

 

 

Step 5. Make a 1:N search.

 

bool  search ( )

 

Use search ( ) to make an 1:N search.

 

bool setOutputFileName (string b1File)

string getOutputFileName ( )

 

Use setOutputFileName (string b1File) to set the output file name; the default name is “.\data\b1.txt”, where “.” is the application folder.  Use getOutputFileName ( ) to see the current setting.

 

 

8.3   Programming Example

 

Now we will show a programming example.

 

1. Create a User Interface in Figure 8.1.

 

 

Figure 8.1 A User Interface, which contains the five steps.

 

2. Link to the Library

 

 

 

Figure 8.2 Add the class library; see the highlighted areas in the Solution Explorer.

 

3. Initial Files

 

Apart from linking the class libraries, there are several data files required by the class libraries. You must provide these data files in order for the class libraries to function properly. You will need to copy “.\data\*.*” from the CD into your project folder. These files must be placed in the “.\data\” sub-folder of your project

 

4. ImageFinderLite Object

Add an ImageFinderLite object:

 

Attrasoft.TransApplet80.ImageFinderLite.ImageFinderLite ifl;

  

Create the object in the form constructor:

        

public Form1()

{

 InitializeComponent();

 

ifl = new Attrasoft.TransApplet80.ImageFinderLite.ImageFinderLite

( this.richTextBox1 );

}

 

 

Figure 8.3 Add the ImageFinderLite object; see the highlighted areas in the Solution Explorer.

 

5. Following the 5 steps:

Step 1. Set the Folder that contains the images you want to search through.

Step 2. Create the Image Signature Library from the folder.

Step 3. Load the Signature Library into the ImageFinderLite object.

Step 4. Get a key image file name, which will be matched against the library.

Step 5. Make a 1:N search.

 

The details are given in the next section.

 

8.4   Codes for Each of the Steps

 

Step 1. Set the Folder that contains the images you want to search through.

 

private void button1_Click(object sender, EventArgs e)

        {

            if (folderBrowserDialog1.ShowDialog() != DialogResult.OK)

                return ;

            string cDir = folderBrowserDialog1.SelectedPath;

            textBox1.Text = cDir;

 

            ifl .setFolder ( cDir );

        }

 

Step 2. Create the Image Signature Library from the folder.

 

private void button2_Click(object sender, EventArgs e)

        {

            bool b = ifl.computeSignatureLibrary();

        }

 

Step 3. Load the Signature Library into the ImageFinderLite object.

 

private void button3_Click(object sender, EventArgs e)

        {

               if (openFileDialog1.ShowDialog() != DialogResult.OK)

                    return;

                string fileName = openFileDialog1.FileName;

                textBox2.Text = fileName;

                ifl.setKeyImage(fileName);

        }

 

Step 4. Get a key image file name, which will be matched against the library.

 

    private void button4_Click(object sender, EventArgs e)

        {

            bool b = ifl.loadLibrary ();

        }

 

Step 5. Make a 1:N search.

 

   private void button5_Click(object sender, EventArgs e)

        {

            ifl.search () ;

        }

 

8.5   Test Run

 

Run the software to get Figure 8.1; then

 

·         Click Button 1 to set the Folder that contains the images you want to search through.

·         Click Button 2 to create the Image Signature Library from the folder.

·         Click Button 3 to load the Signature Library into the ImageFinderLite object.

·         Click Button 4 to get a key image file name, which will be matched against the library.

·         Click Button 5 to Make a 1:N search.

The results are given in the text window in Figure 8.1 and in a text file, “.\data\b1.txt”.

9.   Image Signatures

 

Image Matching is implemented through something called Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature.

 

In this project, we will use Signature Menu Items to compute. The input is an image and the output is a signature. We will show how to use the Image Signatures in image matching in the next chapter. This chapter introduces how to compute signatures only, which will take an input image and produce a signature.

 

The chapter project is located at:

 

C:\transapplet80\imagefinder\.

 

The executable file is located at:

 

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

 

We also call this folder “.\”. Most of the functions will be implemented in two classes:

 

·         MainMenuToAPI

·         GUIRelated

 

Class, MainMenuToAPI, will implement all TransApplet related functions. This class will declare an object of “Attrasoft.TransApplet80.EntryPoint80.EntryPoint80”, which contains all of the required objects.

 

Class, GUIRelated, will implement all functions related to GUI (Graphical User Interface), such as input/output, display images, messaging, …

 

The basic object is:

 

Attrasoft.TransApplet80.SignaturFilter80.SigantureFilter80.

 

Starting from this chapter, we will use a new object from this class:

 

Attrasoft.TransApplet80.EntryPoint80.EntryPoint80

 

This class has a list of objects; one of the objects is the SignatureFilter80.

 

9.1   Signature Menu

 

First of all, we will create the menu shown in Figure 9.1 and create the menu items under the Signature Menu.

 

Since the project is already implemented, you can simply open the project in:

 

.\imagefinder\,

 

and see the implementation of the menu items. We will explain the main functions in each menu item.

 

 

Figure 9.1 Signature Menu.

 

9.2   API

 

The basic idea is:

 

Image Signature = SignatureFilter. getSiganture (image);


As we can see, there are two objects here, Signature and Signature Filter. The Signature functions are:

 

 

Function

Descriptions

Comments

int getStatus ()

Returns the status associated with the signature.

Output:

1: signature ready.

0: signature not ready.

-1: no image

-2: image segmentation specification error

-3: other error.

 

 

 

 

 

string getID()

Returns the ID associated with the signature.

Output: image ID.

 

string getImageName ()

Returns the image name associated with the signature.

Output: image name.

 

string getImagePath()

Returns the image path associated with the signature.

Output: image path.

 

string getAbsolutePath ()

Returns the absolute path associated with the signature.

Output: image absolute path.

 

 

int getNumberOfAttributes ()

Returns the number of attributes associated with the signature.

Output: number of attributes.

 

int [] getSignatureAttributes()

Returns the attribute array associated with the signature.

Output: attribute array.

 

 

int int [] getSignatureAttributes() (int index)

Returns the attribute associated with the input index.

Input: index.

Output: attribute associated with the input index.

 

 

string toString ()

Returns the entire image signature as a string with fields separated by Tab.

 

 

 

 

The Signature Filter functions are:

 

 

Function

Descriptions

Comments

bool setSignatureFilter (int x)

 

int getSignatureFilter ()

 

Selects a Signature filter.

 

string [] getSignatureFilterNames();

Gets a list of Signature filter names.

 

ImageSignature getSignature (string imagePath, string ID);

           

ImageSignature getSignature (string imagePath);

 

ImageSignature getSignature (Bitmap b, string ID);

 

ImageSignature getSignature (Bitmap b);

Gets the Signature of the input image.

 

ImageSignature getSignature

( string imagePath, string ID,int x, int y, int w, int h);

 

ImageSignature getSignature

( Bitmap bImg, string path, string name, string ID,

int x, int y, int w, int h);

Gets the Signature of the input image segment.


Input:

   string imagePath, or

   Bitmap bImg

   string ID

   int x, int y, int w, int h.

Output:

   Signature.

 

bool getLibrary

 ( string [] imageAbsolutePath,

string fileName );

 

bool getLibrary

( string [] imageAbsolutePath,

string [] IDs,

string fileName );

                       

bool getSegmentLibrary ( string [] imageAbsolutePath,

string [] IDs,

string [] xs,

string [] ys,

string [] ws,

string [] hs,

string fileName );

                       

bool getSegmentLibrary ( string [] imageAbsolutePath,

string [] IDs,

int [] xs,

int [] ys,

int [] ws,

int [] hs,

string fileName );

 

bool getKeywordLibrary( string [] imageAbsolutePath,

string [] IDs,

int [] xs,

int [] ys,

int [] ws,

int [] hs,

string filename,

string [] key0,

string [] key1,

… );

 

 

Generates a Signature library from all images in string [] imageAbsolutePath and produces a file that contains all signatures.

 

Input:

   string []   imageAbsolutePath

Output:

   A text file that contains the library of signatures.

 

 

 

9.3   TransApplet Objects

 

The first thing we will do is to create all of the objects required for the ImageFinder. As we discussed earlier in this chapter, all of the objects required for the ImageFinder project is grouped into a single class. All we have to do is to create an object of this class and we will call it script.

 

We will create a new class, MainMenuToAPI. We will do all of the menu item implementation in this class.

 

internal Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 script;

MainMenu f;

public MainMenuToAPI(MainMenu f1)

{

try

            {

            f = f1;

            script =

new Attrasoft.TransApplet80.EntryPoint80.EntryPoint80 (f.richTextBox1);

}

            catch (Exception e )

            {

                        f.richTextBox1.Text = e.ToString () +"\n";

            }

            }

 

9.4   Key Signature

 

To compute the signature of a key image:

 

 

Example. Key Signature.

 

 

 

Figure 9.2  Click “Signature/Left Signature (Key)”.

 

 

The Signature looks like:

 

            3055    29        33        30355  255      18711  195      1          1         

 

In 1:1 Matching or 1:N Matching, the signatures are computed behind the screen; you do not need to use the signature directly. The menu item, “Signature/1:1 First Signature”, shows you the mechanism behind the screen so you can see what a signature looks like.

 

Now, we will implement these two menu items. Double click these menu items and you will see:

 

private void menuItem2_Click(object sender, System.EventArgs e)

            {

            mainMenuToAPI.signature_Key_Signature ( textBox1.Text );

            }

 

Here, mainMenuToAPI is an object, which will implement all functions. The main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool signature_Key_Signature ( string key)

                        {

                                    if ( ! System.IO .File .Exists (key) )

                                    {

                                                appendText ( "Please enter a valid Key!\n");

                                                return false;

                                    }

 

                                    int x =0, y = 0, w =0, h = 0;

script.imageSignature

= script.signatureFilter.getSignature (key, "1of1N",x, y, w, h );

 

                                    if ( script.imageSignature != null )

                                    {

                                                setLine ("The Signature for the left picture box:");

                                                appendLine (script.imageSignature.toString () );

                                                return true;

                                    }

                                    else

                                                return false;

                        }

 

The following section of code simply makes sure the key image exists:

 

if ( ! System.IO .File .Exists (key) )

                        {

                                    appendText ( "Please enter a valid Key!\n");

                                    return false;

                        }

 

The following section of code computes the signature of the key segment:

 

script.ImageSignature

                        = script.signatureFilter.getSignature (key, "1 of 1:N", x, y, w, h );               

 

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, “script.ImageSignature” is the image signature object; and script.signatureFilter is the signature filter object. The parameter, "1 of 1:N", is the ID, which we arbitrarily assigned to the key image. The following section of code prints the signature of the key segment in the text window:

 

if ( script.imageSignature != null )

            {

                        setLine ("The Signature for the left picture box:");

                        appendLine (script.imageSignature.toString () );

                        return true;

            }

            else

                        return false;

 

9.5   Signature File Concepts

 

In 1:N Matching, the Key image will be matched against all images in a directory. The key signature is computed first, as we demonstrated in the last section. Then this signature will be matched against all signatures in the N-signature file. The N-signature file has a fixed name, “.\data\a1.txt”.

 

In N:M Matching, the N-signature file is a1.txt, and the M-signature file is a2.txt. The two files, a1.txt and a2.txt, are computed first. Then all of the signatures in a1.txt will be matched against all of the signatures in a2.txt. The M-signature file has a fixed name, “.\data\a2.txt”.

 

There are 2 menu items:

 

 

These commands compute the signatures for all images in a folder, including all of its subfolders. The only difference between the two commands is where to save the signatures.

 

Signature/1:N Signature (.\data\a1.txt)

Menu item, “Signature/1:N Signature (.\data\a1.txt)”, computes the signatures for all images in a directory and saves the signatures to a1.txt, the N-Signature file. 

 

Signature/N:M Signature (.\data\a2.txt)

Menu item, Signature/N:M Signature (.\data\a2.txt), computes the signatures for all images in a directory and saves the signatures to a2.txt, the M-Signature file.

 

9.6   Signature File Implementation

 

Now, we will implement menu item, Signature/1:N Signature (.\data\a1.txt). Double click this menu item and enter:

 

private void button1_Click(object sender, EventArgs e)

            {

            this.mainMenuToAPI.signature_Search_source

                (dataDir + "a1.txt", SIGNATURE_LIMIT);

            }

 

Again, mainMenuToAPI is an object, which will implement all functions. The main form simply links menu items to functions in the mainMenuToAPI object. The implementation of function, signature_Search_source, is:

 

             f.mainMenuToAPI.script.signatureFilter.getKeywordLibrary

                        ( f.guiSource.libraryList.imageAbsoultePath, f.guiSource.libraryList.imageID,

                        f.guiSource.libraryList.imageX, f.guiSource.libraryList.imageY,

                        f.guiSource.libraryList.imageW, f.guiSource.libraryList.imageH,

                        outputFile,

                        f.guiSource.libraryList.key0, f.guiSource.libraryList.key1,

                        f.guiSource.libraryList.key2, f.guiSource.libraryList.key3,

                        f.guiSource.libraryList.key4, f.guiSource.libraryList.key5,

                        f.guiSource.libraryList.key6, f.guiSource.libraryList.key7,

                        f.guiSource.libraryList.key8, f.guiSource.libraryList.key9

                        );

 

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, “script.signatureFilter” is the image signature filter object. The parameter, “f.gui.imageAbsoultePath”, is a string list that has the absolute paths of all source images to be matched. After the completion of signature computation, the results will be stored in a file specified by the parameter.

 

9.7   Examples

 

Example. Compute N-Signatures:

10.   Whole Image Matching

 

The NeuralFilter matches whole images. The chapter project is located at:

 

C:\transapplet80\imagefinder\.

 

The executable file is located at:

 

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

 

We also call this folder “.\”.

 

10.1   Whole Image Matching Menu

 

The NeuralFilter is responsible for whole image matching. The “Whole Image” menu items are given in Figure 10.1.

 

 

 

Figure 10.1  NeuralFilter Menu.

 

10.2   NeuralFilter API

 

The following table lists the NeuralFilter functions.

 

Functions

Descriptions

bool training ( string a1_txt, string match_txt)

Trains the NeuralFilter.

Attrasoft.TransApplet80.Results_1N .Results_1N

findMatch11

 ( Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig1,

Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig2);

 

Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11

(string path1, string path2);

 

Attrasoft.TransApplet80.Results_1N .Results_1N findMatch11 (Bitmap left, Bitmap right);

Makes a 1:1 matching.

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Attrasoft.TransApplet80.ImageSignature80.ImageSignature sig,  string a1File, string b1File);

 

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N (string keyuPath, string a1File, string b1File)

 

Attrasoft.TransApplet80.Results_1N.Results_1N findMatch1N

(Bitmap keyImage, string a1File, string b1File)

Makes a 1:N matching.

bool findMatchNN (string a1File, string b1File);

bool findMatchNM (string a1File, string a2File, string b1File);

Matches all image signatures in file a1File against all image signatures in a1File or a2File and saves the results to b1File.

 

 

10.3   N:N Matching Implementation

 

Double click menu item “Whole Imager/N:N Matching (a1.txt vs. a1.txt)” and you will see:

 

            private void toolStripMenuItem21_Click (object sender, System.EventArgs e)

                        {

                            mainMenuToAPI.neuralFilter_Matching_NToM 

                                    ( dataDir + "a1.txt", dataDir + "a1.txt", dataDir + "b1.txt" );

                        }

 

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

Public bool neuralFilter_Matching_NToM

                        ( string a1File, string a2File, string b1File)

                        {

                                    try{

                                    script.neuralFilter.findMatchNM ( a1File, a2File, b1File);

                                    }

                                    catch (Exception e)

                                    {

                                                appendText (  e.ToString () + "\n");

                                                return false;

                                    }

                                    return true;

}

 

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, script.neuralFilter is the NeuralFilter object. The parameters, (a1File, a2File, b1File), indicate all signatures in file, a1File, will be matched against all signatures in file, a2File. After completion of the matching, the results will be stored in a file specified by the last parameter, b1File.

 

10.4   1:N  Matching Implementation

 

Double click menu item “Whole Image/1:N Matching (Key vs. a1.tx)”  and you will see:

 

mainMenuToAPI .neuralFilter_Matching_1ToN 

                        ( textBox1.Text , dataDir + "a1.txt",  dataDir + "b1.txt" );

 

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool neuralFilter_Matching_1ToN

                        ( string key, string a1File, string b1File)

                        {

                        bool b = false;

 

                        if ( ! System.IO .File .Exists (key) )

                                    {

                                                appendText ( "Please enter a valid Key!\n");

                                                return false;

                                    }

 

                        b = this.signature_Key_Signature (key);

                                    if ( ! b )

                                    {

                                                appendText ( "Key Signature computation fails!\n");

                                                return false;

                                    }

 

                        try

                                    {

                                    script.results_1N = script.neuralFilter.findMatch1N

( script.imageSignature,  a1File,  b1File);

                                    }

                                    catch (Exception e)

                                    {

                                                appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

                                                return false;

                                    }

                        if ( script.results_1N == null )

                                    {

                                                appendText ( "NeuralFilter 1:N Matching fails!\n" );

                                                return false;

                                    }

 

                        if ( script.results_1N.getStatus () )

                                    {

                                    setText ( script.results_1N.toString () + "\n");

                                    appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

                                    }

                        else

                                    {

                                                appendText ( "No Match!\n");

                                                return false;

                                    }

 

                                    return createFile (key, b1File);

                        }

 

The following code simply makes sure the key image exists:

 

if ( ! System.IO .File .Exists (key) )

                                    {

                                                appendText ( "Please enter a valid Key!\n");

                                                return false;

                                    }

 

The next section of code computes the key signature:

 

            b = this.signature_Key_Signature (key);

 

The next section code makes a 1:N Match, the results are in script.results_1N:

 

try

                                    {

                                    script.results_1N = script.neuralFilter.findMatch1N

( script.imageSignature,  a1File,  b1File);

                                    }

                                    catch (Exception e)

                                    {

                                                appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

                                                return false;

                                    }

 

The final section of code prints the 1:N Matching results:

 

if ( script.results_1N.getStatus () )

                                    {

                                    setText ( script.results_1N.toString () + "\n");

                                    appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

                                    }

                        else

                                    {

                                                appendText ( "No Match!\n");

                                                return false;

                                    }

 

In section 7.4, we have already discussed how to display the matching results from the result object, script.results_1N. At the end of a 1:1 or 1:N matching, a Result object will be produced. This object will be entered into GUI_Result object

 

guiResults.outputResults(script.results_1N, softwareDir);              

 

To disply the first, next, and previous result images, call:   

 

guiResults.Button_First(false);

guiResults.Button_Next(false);

guiResults.Button_Previous(false);

 

 

 

 


11.   Binary Search

 

For a large amount of images, say above 1 million images, binary search is much faster. However, binary search requires additional steps, signature sorting and signature management. The “Binary Search” menu is given in Figure 11.1.

 

The chapter project is located at:

 

C:\transapplet80\imagefinder\.

 

The executable file is located at:

 

c:\transapplet80\imagefinder\ imagefinder Lite\bin\Release\.

 

We also call this folder “.\”. Run the project to get the figure below.

 

 

 

Figure 11.1 Binary Search Menu Items.

 

The search steps are:

 

1. Select a folder (Click the “1” Button in the Toolbar, see Figure 11.1)

2. Get Library Signatures (Click the “2” button in the Toolbar, see Figure 11.1)

3. Sort Signature (see Figure 11.1).

4. Select a Key image (Click the “3” Button in the Toolbar, see Figure 11.1).

5. Search (There are several options in Figure 11.1).

 

11.1   Signatures

 

For binary search, the signature file is sort1.txt. Unlike sequential search, which allows many signature files to be loaded via a library, the sorted signatures will use one single file, so you have to manage this file.

 

To get a signature file sort1.txt:

 

1. Select a folder (Click the “1” Button in the Toolbar, see Figure 11.1).

2. Get Library Signatures (Click the “2” button in the Toolbar, see Figure 11.1).

3. Click menu item, “Binary Search/Create sort1.txt from a1.txt”.

 

If you have several signature files, you have to merge them:

 

 

The merger will be done by clicking “Binary Search/Merge sort1.txt, sort2.txt to sort3.txt”.

 

Sorting a signature file takes two steps:

 

 

Click the menu item, “Binary Search/Create sort1.txt from a1.txt”, you will see:

 

private void createSort1txtFromA1txtToolStripMenuItem_Click

(object sender, EventArgs e)

        {

                mainMenuToAPI.Sort_CreateSort1Froma1

                (dataDir + "a1.txt",

                dataDir + "index1.txt",

                dataDir + "sort1.txt");

 

        }

 

Which will call the following function.

 

internal bool Sort_CreateSort1Froma1

(string a1_txt, string index1_txt, string sort1_txt)

        {

            bool bTmp = Sort_indexing( a1_txt,  index1_txt);

            if ( ! bTmp )

            {

                appendText("Indexing Fails!\n");

                return false;

            }

            else

            {

                appendText("Indexing Completed!\n");

               

            }

 

            bTmp = Sort_sortitngFile(a1_txt, index1_txt, sort1_txt);

            if (!bTmp)

            {

 

                appendText("sort1.txt failed!\n");

                return false;

            }

            else

            {

                appendText("sort1.txt created!\n");

               

            }

            return true;

        }

 

 

The first step is to create an index file:

 

        private bool Sort_indexing(string a1_txt, string index1_txt)

        {

            bool b;

            try

            {

                b = script.se.setInputFile(a1_txt );

                if (!b)

                {

                    appendText("Input File fails: " + a1_txt +"\n");

                    return false;

                }

 

                b = script.se.setOutputIndexFile(index1_txt);

                if (!b)

                {

                    appendText("Cannot create Index File" + index1_txt + "\n");

                    return false ;

                }

 

                b = script.se.indexing();

                if (!b)

                {

                    appendText("Indexing fails\n");

                    return false;

                }

            }

            catch (Exception ex)

            {

                appendText(ex.ToString());

                return false;

            }

            return true;

 

        }

 

This function use an SortEngine object, script.se. It first assign the input image signature file, and assign output index file, then it creates an sorted index file by:

 

                b = script.se.indexing();

 

The next step is to sort the signature file,

 

            private bool Sort_sortitngFile(string a1_txt, string index1_txt, string sort1_txt)

            {

            bool b;

            try

            {

                b = script.se.setInputFile(a1_txt);

                if (!b)

                {

                    appendText("Input File fails: " + a1_txt + "\n");

                    return false;

                }

                b = script.se.setOutputIndexFile(index1_txt);

                if (!b)

                {

                    appendText("Cannot create Index File" + index1_txt + "\n");

                    return false;

                }

                b = script.se.setOutputLibraryFile(sort1_txt);

                if (!b)

                {

                    appendText("Cannot create library File" + sort1_txt + "\n");

                    return false;

                }

                b = script.se.sorting();

                if (!b)

                {

                    appendText("Sorting fails\n");

                    return false;

                }

                appendText("Output file : " + sort1_txt + "\n");

            }

            catch (Exception ex)

            {

                appendText(ex.ToString());

                return false;

            }

            return true;

            }

 

This function will first specify:

 

 input signature file – file containing image signautres;

 index file – file containing sorted index; and

output signature file – file to be produced with all signatures sorted.

 

The main statement in this function is:

 

                b = script.se.sorting();

 

At this point, a sorted signature file is created and saved in the output signature file.

 

11.2   Search

 

To make a search:

 

 

There are several search options:

 

 

Assume you have 1 million images, the first command above will search 1 million images; the second command will search 100 thousand images; the third command will search 50 thousand images; the fourth command will search 30 thousand images; and the last command will search 10 thousand images. By definition, the Binary Search will only ignore the parts that are not likely to yield a match.

 

Assume you use Sequential Search (Chapter 4) for 1 million images; you will search through 1 million images. Then you use the “Binary Search/1:N Search (1%)” for the same 1 million images, provided the image signatures are sorted, you will search through only 10 thousand images. The difference is a factor of 100.

 

Binary Search is more restrictive than 1:N whole image matching; so the two image search results do not always match. In general, Binary Search generates less results than 1:N while image matching.

 

All of these menu items use the same functions, they only differ in one parameter. Click menu item, “Binary Search/1:N Search (10%)”,

 

  private void nSearch10ToolStripMenuItem_Click(object sender, EventArgs e)

        {

            int percent = 10;

 

            bool bTmp = this.mainMenuToAPI.Sort_Match_1N_KeyVsSort1

                (textBox1.Text,

                 dataDir + "sort1.txt",

                dataDir + "b1.txt", percent);

 

            if (!bTmp)

                return;

 

            guiResults.outputResults(mainMenuToAPI.script.results_1N, softwareDir);

            guiResults.Button_First(false);        

        }

 

This function first callS a function for 1:N matching, giving the results in mainMenuToAPI.script.results_1N. Then it passes this result to an image display object,            guiResults. The last line simply displays the first image in the results:

 

guiResults.Button_First (false);

 

The function below does the 1:N matching:

 

 internal bool  Sort_Match_1N_KeyVsSort1

             (string key, string sort1File, string b1File, int percent)

         {

             bool b = false;

             if (!System.IO.File.Exists(key))

             {

                 appendText("Please enter a valid Key!\n");

                 return false;

             }

 

             b = this.signature_Key_Signature(key);

             if (!b)

             {

                 appendText("Key Signature computation fails!\n");

                 return false;

             }

            

             try

             {

                 script.results_1N = script.neuralFilterSorted .findMatch1N

(script.imageSignature, sort1File, b1File, percent);

             }

             catch (Exception e)

             {

                 appendText("Quick 1:N Matching fails:\n" + e.ToString() + "\n");

                 return false;

             }

             

             if (script.results_1N == null)

             {

                 appendText("NeuralFilter 1:N Matching fails!\n");

                 return false;

             }

 

             if (script.results_1N.getStatus())

             {

                 appendText(script.results_1N.toString() + "\n");

                 appendText("" + script.results_1N.getNumberOfMatches() + " matches!\n");

             }

             else

             {

                 appendText("No Match!\n");

                 return false;

             }

            

             return createFile(key, b1File);

         }

 

The code here is similar to the sequential 1:N matching in the last chapter. The matching is done by:

              script.results_1N = script.neuralFilterSorted .findMatch1N

(script.imageSignature, sort1File, b1File, percent);

 

The parameters are:

 

script.imageSignature – signature of a new image;

sort1File – signature file;

b1File – output file;

percent – percent used in the search.

 


 

12.   Dynamic Library

 

In the last a few chapters, we introduced 1:N Match and N:N Match, where N is fixed. This chapter will introduce the Dynamic Library where N can be updated via:

 

 

The chapter project is located at:

 

c:\transapplet\80\imagefinder\.

 

12.1   Dynamic Library Menu

 

Figure 12.1 and Figure 12.2 shows the dynamic library menu.

 

 

Figure 12.1  Dynamic Library Menu.

 

 

 

Figure 12.2  Dynamic Library/Binary Search Menu.

 

12.2   Dynamic Library API

 

The following table lists the dynamic library functions.

 

Functions

Descriptions

Comments

int getLibraryID ()

Gets the Library ID (optional).

 

void setLibraryID (string x)

Sets the Library ID (optional).

 

bool load ()

Loads the default master library, lib1.txt.

 

bool load ( string fileName)

Loads master library specified by fileName.

 

bool load ( string fileName1,  string fileName2)

Loads two libraries specified by fileName1 and fileName2.

 

bool clear()

Clears the current library from RAM only.

 

bool backup ()

Saves current library to the default file, lib1_bk.txt.

 

bool backup (string fileName)

Saves current library to a back file.

 

bool addSignature (ImageSignature sig)

 

Adds a signature to a loaded image library in RAM.

 

bool deleteSignature (ImageSignature sig)

 

Deletes a signature from a loaded image library in RAM.

 

bool deleteSignature (string ID)

 

Deletes a signature from a loaded image library in RAM.

 

bool replaceSignature (ImageSignature sig)

Replaces a signature from a loaded image library in RAM.

 

bool mergeLibrary

(string libFile1, string libFile2, string outputLib)

Merges two signature libraries into a single library.

Input:

   string libFile1

   string libFile2

   string outputLib

Output:

   A text file that contains the library of signatures from both input libraries.

 

 

12.3   Creating A Master Library

 

Up to this point when we have done 1:N or N:N Matching, the N-images have been fixed. The Dynamic Library allows you to update the N-images, including inserting, deleting, and replacing signatures. We now create the master library. The master library file is lib1.txt.

 

Assuming we have created the signature file, a1.txt. To use the Dynamic Library, you have to create a library file lib1.txt. To create a library file, open a1.txt and save it to lib1.txt, or click:

 

 “Library/Maintenance/Create lib1.txt from a1.txt”.

 

12.4   Load Dynamic Library

 

The dynamic library uses file, lib1.txt, as a master library file. Before we can use the dynamic library, this file must be loaded into the dynamic library. To load the library file, click “Library/Maintenance/Load lib1.txt”.

 

Double click menu item “Library/Maintenance/Load lib1.txt” and you will see:

 

mainMenuToAPI.library_loadLib (dataDir +"lib1.txt" );

                       

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool library_loadLib     ( string lib1_txt)

                        {

                        if (script.imageLibrary == null )

                                    {

                                                appendText ( "Unable to find a library !\n");

                                                return false;

                                    }

                        try

                                    {

                                                bool b = script.imageLibrary.load  ( lib1_txt);

                                                if ( b )

                                                {

                                                            appendText ("Library 1 loaded!\n");

                                               

                                                }

}

                        catch (Exception ee )

                                    {

                                                appendText ( ee.ToString () + "\n" );

                                                return false;

                                    }

                                    return true;

                        }

 

The following statement loads the library:

 

bool b = script.imageLibrary.load  ( lib1_txt);

 

12.5   Library M:N Matching

 

In an N:M Matching, a1.txt will be matched against the loaded library. Click menu item “Library/ N:M Matching (a1.txt vs. lib1.txt)” menu item to complete a N:M Match.

 

Double click menu item, Library/ N:M Matching (a1.txt vs. lib1.txt), and enter:

 

            bool b = mainMenuToAPI.library_Matching_N_M

                                    ( dataDir + "a1.txt",  dataDir + "b1.txt" );

                       

 

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool library_Matching_N_M   ( string a1_txt,    string b1_txt)

                        {

                        if ( script.imageLibrary == null )

                        {

                                    appendText ("Dynamic Library not available!\n");

                                    return false;

                        }

 

                        bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

                        if (!b)

                        {

                                    appendText ("Dynamic Library Assignment Fails!\n");

                                    return false;

                        }

                        try

                        {

 

                                    script.neuralFilter.findMatchNN ( a1_txt,  b1_txt);

                        }

                        catch (Exception e)

                        {

                                    appendText (  e.ToString () + "\n");

                                    return false;

                        }

                        return true;

                        }

 

In this statement, the script object contains all of the objects required for the ImageFinder project. In particular, “script.imageLibrary” is the dynamic library object.

 

The following statement assigns the dynamic library to the NeuralFilter object, “script.NeuralFilter”:

 

bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

                        if (!b)

                        {

                                    appendText ("Dynamic Library Assignment Fails!\n");

                                    return false;

                        }

 

Once the NeuralFilter object obtains this library, it will match images against this master library by the following statement:

 

            script.neuralFilter.findMatchNN ( a1_txt,  b1_txt);

 

12.6   Library 1:N Matching

 

Double click menu item “Library/ 1:N Matching (Key vs. lib1.txt)” and enter:

 

bool b = this.mainMenuToAPI.library_Matching_1_N 

                                    (textBox1.Text  , dataDir + "b1.txt" );

 

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool library_Matching_1_N( string key, string b1File)

                        {

if ( script.imageLibrary == null )

                        {

                                    appendText ("Dynamic Library not available!\n");

                                    return false;

                        }

                        bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

                        if (!b)

                        {

                                    appendText ("Dynamic Library Assignment Fails!\n");

                                    return false;

                        }

b = false;

 

                        if ( ! System.IO .File .Exists (key) )

                        {

                                    appendText ( "Please enter a valid Key!\n");

                                    return false;

                        }

 

                        b = this.signature_Key_Signature (key);

                        if ( ! b )

                        {

                                    appendText ( "Key Signature computation fails!\n");

                                    return false;

                        }

 

                        try

                        {

                        script.results_1N = script.neuralFilter.findMatch1N

 ( script.imageSignature);

                        }

                        catch (Exception e)

                        {

                                    appendText ( "NeuralFilter 1:N Matching fails:\n"

+ e.ToString () + "\n");

                                    return false;

                        }

                        if ( script.results_1N == null )

                        {

                                    appendText ( "NeuralFilter 1:N Matching fails!\n" );

                                    return false;

                        }

 

                        if ( script.results_1N.getStatus () )

                        {

                                    setText ( script.results_1N.toString () + "\n");

                                    appendText ("" + script.results_1N.getNumberOfMatches ()

+" matches!\n");

                        }

                        else

                        {

                                    appendText ( "No Match!\n");

                                    return false;

                        }

}

 

The only difference between this 1:N Matching and the NeuralFilter 1:N Matching introduced earlier is the following statement:

 

bool b = script.neuralFilter .setLibrary ( script.imageLibrary );

 

In a normal 1:N Matching, the neural filter gets the library from a1.txt. This statement assigns the dynamic library, “script.imageLibrary”, to the NeuralFilter object, “script.NeuralFilter”.

 

12.7   Update Implementation

 

Double click menu item “Library/Maintenance/Add (Key)” and enter:

 

            if ( ! System.IO .File .Exists (textBox1.Text  ) )

                        {

                                    this.mainMenuToAPI.appendText 

("File does not exist: \n" + textBox1.Text + "\n");

                                    return;

                        }

                        this.mainMenuToAPI.library_add (textBox1.Text  );           

 

Here, mainMenuToAPI is an object, which will implement all functions. As we discussed earlier, the main form simply links menu items to functions in the mainMenuToAPI object. The implementation is:

 

public bool library_add ( string sImage)

                        {

                        script.imageSignature

                                    = script.signatureFilter.getSignature (sImage);

 

                        if (script.imageSignature == null )

                        {

                                    this.setLine ("Image Signature computation fails!");

                                    return false;

                        }

 

                        if (script.imageSignature.getStatus () <= 0 )

                        {

                                    this.setLine ("Image Signature computation fails!");

                                    return false;

                        }

                        bool b =  script.imageLibrary.addSignature (script.imageSignature);

                        return b;

                        }

 

The script object contains all of the objects required for the ImageFinder project. In particular, “script.imageLibrary” is the dynamic library object. The first section of the code computes the image signature. The next section of code checks the validity of the signature. The following statement inserts the signature to the dynamic library:

 

bool b =  script.imageLibrary.addSignature (script.imageSignature);

 

The menu item Delete and the menu item Replace can be implemented in a similar fashion and the codes are provided in the project.

 


13.   Image Variations

 

The chapter project is located at:

 

c:\transapplet\80\imagefinder\.

13.1   Variation Signatures

 

               

 

Figure 13.1 A change beyond the tolerance of a single image signature.

 

 

Figure 13.2 Signature/Variation Signatures Menu.

 

Image Matching is done through something called Image Signature. An image has a set of computed values called features. A collection of features is grouped into a signature. A single image signature will take care of a certain amount of variation. To increase the variations, additional signatures will be added, which is the purpose of the Variation Signatures menu.

 

An image can be changed, for example, a rotation of 90 degrees to the left. As the image is changed, the signature will change accordingly. In general, this change will not be maintained inside a signature, i.e. a small variation allowed inside a single signature will not take care of large variations like 90 degree rotations.

 

The “Signature/Variation Signatures” menu item (Figure 13.2) provides a list of menu items to compute signatures. The net result is that an image will have multiple image signatures.

 

The “Signature/Variation Image” menu item (Figure 13.3) will compute various variations of images from the key image and will deposit these images into “.\tmp\” folder.  To use this menu item, you must first select a key image. This allows a user to see what the variations are.

 

 

 

 

Figure 13.3 Signature/Variation Images Menu.

 

 

 

 

This menu supports the following variations:

 

Signatures/Variation Signatures/Scale (100, 90, 80, …)

 

            This menu item generates 7 image signatures:

            100%, 90%, 80%, 70%, 60%, 50%, 40%.

            Here 90% is an image that has 5% border region removed.     

 

Signatures/ Variation Signatures/Rotation (90, 180, 270)

 

            This menu item generates 4 image signatures:

            0, 90, 180, 270.

            Here 90 is an image that has a 90 degree rotation.

 

Signatures/ Variation Signatures/Flip (X, Y,  XY)

 

            This menu item generates 4 image signatures:

            0, X, Y, XY.

            Here X is an image that has flipped around x axis.

 

Signatures/ Variation Signatures/Flip Rotation (16 variations)

 

            This menu item generates 16 image signatures:

            {0, 90, 180, 270} ^ {0, X, Y, XY}.

            Here (90, X) is an image that has 90 degree rotation and flipped around x axis.

 

Signatures/ Variation Signatures/Stretch (4 variations)

 

            This menu item generates 5 image signatures:

            (0, 0), (120%, 100%), (80%, 100%), (100%, 120%), (100%, 80%).

            Here (120%, 100%) are stretch percentages along the x and y directions.

        

Signatures/ Variation Signatures/Resize 

 

            This menu item generates 5 image signatures:

            100, 150, 200, 300, 400.

            Here 150 increases the image size by 50% in terms of pixel counts in each direction.

 

The implementation for menu item, Signatures/Variation Signatures/Scale (100, 90, 80, …), is:

 

            script.signatureFilterSym .getKeywordLibrary_Scale

                    (f.guiSource.libraryList.imageAbsolutePath, f.guiSource.libraryList.imageID,

                    f.guiSource.libraryList.imageX, f.guiSource.libraryList.imageY,

                    f.guiSource.libraryList.imageW, f.guiSource.libraryList.imageH,

                    outputFile,

                    f.guiSource.libraryList.key0, f.guiSource.libraryList.key1,

                    f.guiSource.libraryList.key2, f.guiSource.libraryList.key3,

                    f.guiSource.libraryList.key4, f.guiSource.libraryList.key5,

                    f.guiSource.libraryList.key6, f.guiSource.libraryList.key7,

                    f.guiSource.libraryList.key8, f.guiSource.libraryList.key9,

                    f.mainMenuToAPI.script.tranAppletPara .signatureFilter_LibraryScaleType

                    );

13.2   Segments

 

A collection of features is grouped into a signature. One image has one signature. The signature matches whole images against whole images, i.e. it does not look inside the image.

 

An image can be many logical segments. As the focus changes from a whole image to an image segment inside the image (Figure 13.3), the signature will change accordingly. In general, this change will not be maintained inside a basic signature.

 

The “Signature/Segment Signatures” menu item attempts to partially address this image segmentation problem. The menu item will look at some segments inside an image. On average, it will look at six segments.

 

The “Signature/Segment Images” menu item (Figure 13.2) computes signatures for image segments. The net result is that an image will have multiple image signatures, one for each segment.

 

The “Signature/Segment Images” menu item (Figure 13.3) will compute segments from the key image and will deposit them into “.\tmp\” folder.  To use this menu item, you must first select a key image. This allows a user to see what the segments are.

 

 

        

 

 

Figure 13.4 Image Segmentations.

 

The object, imgSigFilterSeg, used for computing segment signatures, is an instance of the following class:

 

Attrasoft.TransApplet80.ImageSignatureFilterSegment80.ImageSignatureFilterSegment80  imgSigFilterSeg;

 

The implementation of the menu item, Signature/Segment Signatures, is:

 

imgSigFilterSeg.getSegmentLibrary_Modelbased(sourceFolder, outputFile);

 

This function will convert images in parameter, sourceFolder, into image signatures and save them to a file, outputFile.


 

14.   Sub-Image Search

 

The chapter project is located at:

c:\transapplet80\imagefinder\.

 

The sub-image matching requires a different set of parameters. There are two different settings:

 

Setting/Search Sub-Images

Setting/Search Whole Images

 

To match sub-images, click menu item, “Setting/Search Sub-Images”. To switch back to the original setting for whole images, click: “Setting/Search Whole Images”.

 

 

Figure 14.1 Setting Change.

 

14.1   Sub-Images and Variations

 

Starting from this chapter, we will search inside an image. We will deal with the problems of:

 

 

The number of sub-images is very large, measured by exponential functions of the number of pixels. The number of variations is also very large:

 

·         ...

 

We will not use image signatures because of the large amount of variations. This will significantly slow down the search speed.

 

Because the variations for searching inside an image are huge, we might impose some limits on where to search. There are several approaches:

 

 

Automatic segmentation deploys certain algorithms to look at a small number of segment locations inside the image. There are two menu items to address this approach (Chapter 13):

 

The “Signature/Segment Signatures” menu item (Figure 13.2) computes signatures for a selected number of image segments. The net result is that an image will have multiple image signatures, one for each segment.

 

The “Signature/Segment Image” menu item (Figure 13.3) will compute segment images from the key image and will deposit them into “.\tmp\” folder.  To use this menu item, you must first select a key image. This allows a user to see what the segments are.

 

The advantages of automatic segmentation are that it will dramatically reduce the number of search locations. The disadvantages of segmentation algorithms are that it will not be able to address some human cropped images which cut right through the middle of a segment.

 

The enumerative approaches are not restricted by the automatic segmentation algorithm; however, the variations for enumerative sub-image searching are huge; some sort of limits on where to search will have to be imposed.

 

 

Figure 14.2 Two parameters.

 

 

Figure 14.3 Sub-Image Menu.

 

 

 

14.2   Search Process

 

In this chapter, we will use Enumerative search by imposing these two conditions:

 

 

In each search, there are two parameters N x M (Figure 14.2), which specify an array of sub-images to search. The software will search via an “N by M” grid for each unknown image for the specified key image. The key will not be rotated or scaled.

 

Initially, you will need three steps for an image search:

 

1.      Select a Key;

2.      Select a folder;

3.      Search.

 

To see the results:

 

1. Click the Results Home button (Figure 14.3);

2. Click the Results “=>” button (Figure 14.3).

 

Starting from the second search, you do not have to select the folder again because it is already selected; each search has only two steps:

 

1. Select a Key;

3. Search;

1. Select a Key;

3. Search;

 

Note:

Step 1, selecting a Key can be completed by dragging & dropping an image to the first text box;

Step 2, select a folder, which can be completed by dragging & dropping a folder to the second text box.

 

14.3   Learn by Example

 

 

key.JPG     CIMG4643_mid_shelf1.JPG

 

Figure 14.4    1:1 Matching.

 

 

The first example (see Figure 14.4) is to match the key,

.\ ex_object1\key.jpg,

 

against the image,

.\ ex_object1\CIMG4643_mid_shelf1.JPG.

 

The example will take three clicks:

 

1.      Click the “Key” button in the Toolbar, and select .\ ex_object1\ key.jpg. You can also drag & drop .\ ex_object1\key.jpg to the first text box.

 

 

Figure 14.5 Step 1, selecting a key. After selecting a key image, the image will be displayed in the Key Panel.

 

2.      Click the “Source” button in the Toolbar and select the folder, .\ ex_object1\.

 

You can also drag & drop the folder, .\ ex_object1\, to the second text box; the use “=>’ button to select image, .\ ex_object1\CIMG4643_mid_shelf1.JP, see Figure 14.6.

 

 

Figure 14.6  Step 2, selecting a folder. After selecting a folder, the first image will be displayed in the Source Panel (middle picture box). The list of image files will be displayed in the text window. The Home button goes to the first image; the “=>” button goes to the next image; and the “<=  button  goes to the previous image. All sub-folders will be included.

 

 

Figure 14.7  Search the N x M grid of sub-images. For example, setting them to 30 x 5 will result in searching 150 sub-images.

 

3.      Before we make a search, you will specify how to search via an “N by M” grid, i.e. how many sub-images we will look at for the specified key image. The default value is a 10 by 10 grid.

 

To make a 1:1 search,

 

·         Enter 30 and 5 in Figure 14.7.

·         Click the “Sub-Image/1:1 Matching” menu item in Figure 14.3.

 

You will get:

 

ID    Name       Path         Score       X             Y             W            H

CIMG4643_mid_shelf1.JPG_146  CIMG4643_mid_shelf1.JPG  .\ex_object1\            84            366          34            73            184

CIMG4643_mid_shelf1.JPG_134  CIMG4643_mid_shelf1.JPG  .\ex_object1\            84            197          34            73            184

CIMG4643_mid_shelf1.JPG_123  CIMG4643_mid_shelf1.JPG  .\ex_object1\            83            42            34            73            184

CIMG4643_mid_shelf1.JPG_110  CIMG4643_mid_shelf1.JPG  .\ex_object1\            83            282          25            73            184

CIMG4643_mid_shelf1.JPG_117  CIMG4643_mid_shelf1.JPG  .\ex_object1\            81            380          25            73            184

CIMG4643_mid_shelf1.JPG_100  CIMG4643_mid_shelf1.JPG  .\ex_object1\            80            141          25            73            184

 

Note, if you do not get this, click menu item, “Setting/Search Sub-Images” and then try again.

 

 

Figure 14.8 Search the selected folder. Images that matched the “key” will be displayed in the Results Panel. Use the Home, =>, and <=buttons to see matched images. The Home button goes to the first image; the “=>” button goes to the next image; and the “<=  button  goes to the previous image.

 

 

 

Figure 14.9 The matched score and coordinates, (x, y, w, h), are given in a text file: “.\data\c.txt”.

To see the first matched image, click the Home button in the Results Panel (see Figure 14.8). To see the next matched image, click the “=>” button in the Results Panel. The key and the matched images are displayed side-by-side.

 

14.4   Implementation

 

The basic idea is to cut a segment of a source image, which has the same size as the key image. These two images are matched via image signatures. This process is repeated for various locations.

 

The basic function is:

 

r80 = script.smallSegment.findmatch11_NeuralFilter_LargeImage

(b1, b2, theID, thePath, theName, f.pictureBox4);

 

Where:

 

 

 


15.   Fast Sub-Image Matching

 

The sub-image matching requires a different set of parameters. There are two different settings:

 

Setting/Search Sub-Images

Setting/Search Whole Images

 

To match sub-images, click menu item, “Setting/Search Sub Images”. To switch back to the original setting for whole images, click: “Setting/Search Whole Images”.

 

The objective for the last chapter menu is to find a cropped segment at all cost, provided there are no scaling factors. This can take a long time if the N x M search grid is set to be very large.

 

The first objective of this chapter is to find a cropped segment much faster.

The second objective is to add scale invariance and rotation invariance.

 

 

 

Figure 15.1 “Fast” menu items.

 

15.1   Fast Menu

 

The objective of the “Fast” menu items is to find a cropped segment from an image within 1 second.

 

The “Fast” menu items provide a much faster search than the menu items of the last chapter, but these menu items are less accurate than those in the last chapter. It is a trade-off between accuracy and computation time. Figure 15.1 shows the menu items under this menu.

 

Example. We will match (see Figure 15.4 from last chapter) the key,

 

.\ ex_object1\key.jpg,

 

against the image,

 

.\ ex_object1\CIMG4643_mid_shelf1.JPG.

 

 

 

Figure 15.2 Fast match results.

 

Before starting this example, click menu item, “Setting/Search Sub Images”.

 

The example will take three clicks:

 

1.      Click the “Key” button, and select .\ ex_object1\key.jpg. You can also drag & drop .\ex_object1\key.jpg to the first text box.

 

2.      Click the “Source” button and select the folder, .\ ex_object1\. You can also drag & drop the folder, .\ ex_object1\, to the second text box.

 

3.      Fast Search.

Click Fast/1:1 Matching and see the result in Figure 15.2.

 

The menu items are given below:

 

Sub Image/Fast 1:1

Use “Sub Image/Fast 1:1” to make a 1:1 Matching.

 

Sub Image/Fast 1:N (Same size)

Use “Sub Image/Fast 1:N (Same size)” to make a 1:N Matching; it will search through all images in a folder for the specified cropped image. This command is faster than “Sub Image/1:N (Different size)”, but it requires all images in the search folder to have the same pixel size.

 

Sub Image/Fast 1:N (Different size)

Use “Sub Image/Fast 1:N (Different size)” to make a 1:N Matching; it will search through all images in a folder for the specified cropped image. This command is slower than “Sub Image/1:N (Same size)”, but it does not requires all images in the search folder to have the same pixel size.

 

To implement these menu items, the object is:

 

f.mainMenuToAPI.script.neuralNetFilter

 

This object will need to learn the sample bitmap, b:

 

try

            {

                f.mainMenuToAPI.script.neuralNetFilter.setSymmetry(3);

                f.mainMenuToAPI.script.neuralNetFilter.train(b, right.Width, right.Height);

                return true;

               

            }

            catch (Exception ex)

            {

                f.mainMenuToAPI.appendText(ex.ToString () + "\n");

                return false;

            }

 

After that, it will make a fast 1:N matching:

 

f.mainMenuToAPI.script.results_1N

                    = f.mainMenuToAPI.script.neuralnet2.findMatch1N

                    (f.guiSource .libraryList .imageAbsolutePath, matchingCode);


15.2   NeuralNet Filter API

 

The following table lists the NeuralNet Filter functions.

 

Functions

Descriptions

bool train (Bitmap img)

bool train (string sPath)

bool train (Bitmap img, int x, int y, int w, int h)

bool train (string sPath, int x, int y, int w, int h)

Trains the neural net, either use whole image or image segement (x, y, w, h).

bool train (Bitmap  b, int w, int h)

Place an image b into a white image w pixels by h pixels, the train the neural net.

bool retrain (Bitmap img)

bool retrain (string sPath)

bool retrain (Bitmap img, int x, int y, int w, int h)

bool retrain (string sPath, int x, int y, int w, int h)

Retrains the neural net.

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch11 ( Bitmap img1);

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch11 ( string path1);

Attrasoft.TransApplet80.Results_1N.Results_1N

findMatch1N ( string [] fileList);

Makes a 1:1 and 1:N Matching.

 

bool findMatchNN (string [] fileList, string c1File)

 

bool findMatchNM (string [] keyList, string [] libraryList, string c1File);

Makes a N:N Matching and N:M Matching.

 


16.   Video Input

 

Video recognition largely mirrors the image recognition. Identify a whole video is similar to identify a whole image. This chapter describes how to enter video data to the VideoFinderLite. The input videos are further divided into:

 

 

A Key-folder specifies the sample videos. A Search-Folder specifies the Search Source. The chapter project is in “.\videofinderlite”. The video input programming mirrors the image input.

 

16.1   Class Library Name

 

The class library is:

 

Attrasoft.TransApplet80.InputModuleVideo.

 

The main class in this library will be:

 

Attrasoft.TransApplet80.InputModuleVideo .InputModuleVideo.

 

The objects for key and source are:

 

internal Attrasoft.TransApplet80.InputModuleVideo .InputModuleVideo  inputModuleKey;

internal Attrasoft.TransApplet80.InputModuleVideo.InputModuleVideo inputModuleSource;

 

The interface is:

 

public interface I_ InputModule80

            {

         Attrasoft .TransApplet80 .InputList80 .InputList  getList( string);                 

            }

 

The input is a folder path. Output is an object of the following class:

 

public class VideoInputList

    {

       public int status = 0;

     

        public string[] imageAbsoultePath = null;

 

        public string[] VideoID = null;

        public string[] VideoName = null;

        public string[] VideoPath = null;

 

        public string[] key0 = null;

        public string[] key1 = null;

        public string[] key2 = null;

        public string[] key3 = null;

        public string[] key4 = null;

   }

 

16.2   Class Library Overview

 

The functions in classes are listed in the following table.

 

Function

Description

Attrasoft .TransApplet80 . VideoInputList. VideoInputList  getList( string folder)

After input folder is specified, get the output object.

 

 

16.3   Set Up the Chapter Project

 

 

Figure 16.1  Chapter Project.

 

The project has the following controls:

           

Key Button, Key Text Box

Use the “Key” button to display the selected key video folder.

 

Source Button, Source Text Box

Use the “Source” button to select a search video folder.

 

Clear Button

Use the “Clear” button to clear the rich text box.

 

Rich Text Box

            Use Rich Text Box to display messages.

 

Key Picture Box

Use the Key Picture Box to display key images.

 

Source Picture Box

Use the Source Picture Box to display images in the search source.

 

OpenFileDialog

            Use the OpenFileDialog to select a key image.

 

FolderBrowserDialog

            Use the FolderBrowserDialog to select a source path.

 

After adding these controls, the form will look like Figure 16.1.

 

16.5   Key Segment

 

We can double click the Key Segment buttons to complete the programming:

 

public bool Button_Keys()

        {

            try

            {

                if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

                    return false;

                string cDir = f.folderBrowserDialog1.SelectedPath;

                f.textBox1.Text = cDir;

 

   f.guiKey.setInput(f.inputModuleKey.getList(cDir));

                if (!f.guiKey.getStatus())

                {

 

                    f.appendText("Data Source fails!\n");

                    return false;

                }

 

                f.guiKey.setLists();

            }

            catch (Exception e)

            {

                f.appendText(e.ToString() + "\n");

                return false;

            }

            return this.f.guiKey.Button_First(false);

        }

 

The first section of code opens a file dialog so the user can select a file:

 

                if (f.folderBrowserDialog1.ShowDialog() != DialogResult.OK)

                    return false;

                string cDir = f.folderBrowserDialog1.SelectedPath;

 

The next section enters the selected folder to the InputModuleKey object and gets the string list of the image paths: