Applied Multimedia Database Systems

Face Recognition And The Querying Of Multimedia Databases

ABSTRACT

Information technology and computer-based security systems are things that have now become increasingly more popular and effective, but when combining the two technologies together to try and create automated database driven identification programs, many problems have arisen and the advantages have often been hidden behind clouds of flaws and therefore doubt.

Accurate face capture and recognition through the querying of multimedia databases could be the key to creating successful automated security and identification systems.

In this report I will explore the methods of computer based face recognition, and the use of multimedia databases, and I will try and conclude whether or not it is possible to successfully create the previously mentioned systems, and I will try and give you an incite on how they might work.

INTRODUCTION

Before we can go any further, we need to sample what is already available, how it is being used and whether or not it has been successful or not.

The Current State Of Computer Based Face Recognition

Various software packages are available that have the ability to recognise faces through the use of image capturing devices and by querying a database with them. Applications such as FaceIT, ID-2000 and Identix are all commercial products for this area, and ID-2000 SDK is a development kit for producing customised face recognition software. But none of them have been completely successful and most security/identification systems, including the Police Departments, still have not chosen to run this software to assist identification.
Also all of the above face recognition systems never seemed to mention the ability to search through the database for characteristics such as skin colour, hair colour, gender, age, weight, or many other things, that we as humans, have the ability to recognise or determine when we look at people (Kottas. J. and Dawson. B., 2000).

The Current State Of Multimedia Databases

Multimedia databases are not to be confused with standard databases, a multimedia database stores information and media ‘internally’ whereas a ‘normal’ database can only store textual or numerical data.

Multimedia databases, although available, are still in their infancy and are very unpopular. Jasmine II is the most successful ‘true’ multimedia database system to date, allowing all types of different media to be stored internally within the database and allowing queries to be based upon that data.

The main problems with multimedia databases include their size, processing needs, and their complex configuration, but none of these problems should concern large organisations that use powerful mainframe computers and could easily benefit from their use.

FACE RECOGNITION

We as humans have throughout history relied on face recognition for survival, we have a feature based approach to it, where facial features, such as the eyes, mouth and nose, are interpreted without assuming any knowledge of general face structure (Manjunath B.S., 1992). Being the most popular natural means of biometric identification, we have no problem identifying one person from another, but only until recently has this area been seen as a science and not just a natural area (ADMS, 2002). So just how do humans recognise and identify people and how can computers be just as accurate with no true intelligence?

How Faces Are Identified And Recognised

Firstly there is a difference between recognising a face and identifying a face, to identify someone usually means you have background knowledge on that person, whereas recognising someone means that you have just seen them before. Face identification is an area that is prone to error much more than face recognition, as much more brain processing is required and much more memory need to be retrieved (Martinez Y., [A]).

The beliefs and theories on how faces are actually identified or recognised revolve around the fact that peoples faces are constructed of complex organic shapes that cannot be translated into simple geometric shapes, therefore making them harder to confuse and more distinguishable. Other factors such as colour, size and texture can also be taken into account, and when combined with hair styles and colour, facial hair, scars, markings and permanent jewellery or accessories (e.g. glasses), pretty much everyone, except identical twins, have completely individual faces and appearances.

 Computerised Face Recognition

Computerised face recognition is a technology that has only become available over the past decade or so. The methods used are usually combinations of complex algorithms that take place on an image of a face to try and find a unique ‘code’ for querying predefined ‘codes’ of face images stored in a database.
The ‘codes’ are usually strings that define a net made up of a series of points that have been extracted from the most distinctive areas of the face image. Other computer based face recognition techniques include drawing a series of vector based lines that follow the distinctive areas, or use light intensity of the reflection from features on the face image.
A typical face recognition program that is commercially available is Visionics FaceIT; this uses biometric information to create a series of unique ‘nodal points’ (Visionics term for points on an image that are distinctive). The program is designed to work through CCTV systems, and using algorithms it automatically detects head ‘shapes’ and determines the heads position, size and pose, the face needs to be turned within 35° of the camera in order to carry on from this point. From here the image of the face is normalised (rotated to an appropriate size and pose) and then represented into a hopefully unique ‘code’ (faceprint) that is used to query a database of faceprints that have already been recorded (Bonsor K., 1998-2002).

A human face has about 80 ‘nodal points’ in order for most face recognition software to work it requires around a minimum of 20 ‘nodal points’, but better/more accurate results are obviously achieved with more.

Practical Examples Of Computer Based Face Recognition Techniques

The key to all face recognition software is how the software stores the data, and how it is ‘trained’ to interpret information and query it. Below are examples of how images could be processed so that they could be queried from within a database.

1.	2.	3.	4.	5.	6.	7.

1. The Original Image

Current facial recognition software stores both the image and textual data based upon that image for querying (or it finds this at run time for each image). So it needs to be interpreted in some way to make it easier to be queried.

2. Gif Compression Using 9 Colours

Although the example shown is a raster image, it does show distinctive lines that would be present should it have been traced as a relatively high quality vector image. This vector based image could be queried using lines to represent the shape of features such as jaw-lines, eyebrows, mouth and nose.

3. Gif Compression Using 3 Colours

Again a raster based image has been used to represent a mid to low quality vector image. This image could be queried in the same way as number two, but would not find features such as the jaw line or nose due to the lack of quality. This shows that the more data that is saved or ‘preserved’, the more ways it can be queried and more precisely queried.

4. Gif Compression Using 2 Colours

This is the final raster image that has been used to represent a vector image, this time it is very low quality. This example shows that when images are automatically vector traced the computer can often recognise areas that would not be detected by the human eye. This image would be very difficult to query using vector based lines to represent features, as the lines that are shown in the example are not what the user would draw to query it e.g. there are no distinctive eyebrow lines, and the mouth has shading which would cause conflicts.

5. Nodal Points

In the example above some nodal points have been detected by hand, but with computer based software the program would need to be ‘trained’ on how to detect them through the use of various image manipulation algorithms. These nodal points cannot be queried in any way, but when the distances between them are calculated they can be queried using a…

6. Mesh Or ‘Face Print’

The example shows a mesh created by joining together some of the nodal points, with computer generated meshes, all of the chosen nodal point distances would be used to query facial images or ‘face prints’ in a database, in the example only a few of the nodal points are connected to prevent the diagram from getting too complex.

7. 3D Mesh

3D meshes are the way forward with face recognition technology; they do not rely on images of faces being within certain angles, as the head is complete. Most software packages that create 3D meshes from photographed heads/faces tend to require a minimum of two photographs (a side view and a front view) and they use nodal points of the images to work out the dimensions of the head/face. Other software packages use a number of photographs taken from various angles and use outlines of the head and reference points of the environment to construct a 3D mesh.

These 3D meshes can/could be queried using both vector lines and nodal points, and should/would return more successful results when used correctly.

Other Form Of Biometrics

Face recognition isn’t the only way that humans can be identified through biological characteristics. Fingerprints for example are a very common form of identification, and have been used quite successfully for a number of years. Other forms include hand geometry, retina scans, iris scans, and voice recognition. Each of these forms of biometric identification have varying ease of use, and levels of error incidence, accuracy, user-acceptance, required security and long-term stability. The more accurate methods tend to be the ones that are the most technical. But the most important factor when related to identification of a single person, through biometric methods, is the amount of crossovers that can occur.
The most accurate by far is the retinal scan with a crossover of one in every ten million or more, second the iris scan with a cross over of one in one hundred and thirty one thousand. The others are fingerprints, hand geometry and voice recognition with crossovers of one in five hundred, one in five hundred and one in fifty respectively. (AMDS, 2002) Face recognition cannot be analysed for crossovers due to the nature of the source.

MULTIMEDIA DATABASES

Multimedia databases, as previously mentioned, have the ability to store, index, and query images, sound, video and other forms of media. This can be incredibly useful for all types of application, so, before we discuss face recognition and multimedia databases, let’s look at how different, less complex forms of media can be queried.

How Images Can Be Queried In Multimedia Databases

Various techniques have been researched into how images can be defined from one another and then queried in a multimedia database, but very few variations between them seem to exist. Query by content, query by example, similarity retrieval, sketch retrieval, colour histograms, texture analysis, and orientation analysis are all methods that were mentioned in a report by Jacobs C., Finkelstien A., and Salesin D.  (1992), and methods such as shape comparison and edge definition seem to fit into these areas.

Many of these querying methods work by defining differences in colour from pixel to pixel, and then from each group of similar pixels to another group of similar pixels. Often complex algorithms are used in order to decide if the values that need to be present are available on the image being queried, and this can take some time; this is why many ‘image querying database’ type products store a result of a mathematical algorithm that has already been performed, and compare the query to that as it uses much processing power. This technique of only storing the mathematical data of an image limits the number of ways that an image can be then queried, as not all information is present.

With all image querying the tolerances are often very ‘loose’, as the processes performed to retrieve matches are not very exact. This will therefore often bring back unwanted results as the criteria for them has been slightly misinterpreted, or with the tolerance applied has allowed images that should not have been shown to pass through the query.

COMBINING THE TECHNOLOGIES

Now that we have a brief idea of how images are recognised within databases, and of how faces are recognised by both humans and computers, we can discuss how multimedia databases can be used for the identification of people. It’s fairly safe to say that with the more different ways you can query an image, the better the chance of you getting a positive result. Also the more types of information you have stored within the database, on the people you want to identify, the more queries you can apply to minimise negative results.

Querying Multimedia Databases Using Biometric Identification

If we take the example of a Police suspect identification system, which revolves around the use of a multimedia database that contains passport style photographs of suspects, along with finger-prints and personal information. We can assume that querying the records in the database would be the sole purpose of the program, in order to find potential suspects etc.

The user of this system would probably have a photograph of a suspect and would want to query the database in as many ways as possible with it, in order to minimise the number of potential matches, therefore combinations of face recognition methods, image querying methods and text based queries would need to be used.

As previously mentioned, if you only store the mathematical description of a facial image that is based on one facial recognition algorithm, you can only query that image in so many ways (as used in FaceIT). So in order for this database to be queried multiple times, using a variety of face recognition or image querying algorithms, the image must remain in its original raster based form therefore massively increasing the processing time. However if the results of all available algorithms were stored alongside the image, this would cut down the processing time, but massively increase the size of the database. Some kind of balance would need to be found; maybe by only storing the results of the most frequently used face recognition algorithms, and then querying using the other methods after, when fewer records would need to be processed in a much more time consuming way.

Querying Multimedia Databases Without A Sample Photograph

If we continue using the example above, except this time the user does not have an image of the suspect, and is working from memory. Then the database would need to be queried using much less face recognition methods, and more standard image querying techniques.
If the user has a description of someone in front of them, and a police-artists sketch, and they need to minimise the potential number of matches so that the victim can identify who the culprit was, then the user would have to query the database initially through text based queries, defining descriptive terms such as height, weight, age and skin colour. Then the user could begin to query biometric information about the face of the culprit, such as location of scars or moles and the proportions between each of the main facial features, such as eyes, nose and mouth. Finally they could use the artists sketch to query the database for vector based lines, similar to those drawn by the artist, defining the culprits chin line, eyebrow shape and nose.

Theories On How Querying Images Using Face Recognition Could Be Improved

The biggest way in which the current facial recognition systems could be improved is the use of detailed 3D meshes of peoples heads instead of 2D photographs of the people stored in the database. This would give much more information about the people, and would allow queries to take place that didn’t require the sample image of the persons face to be within 35° of the camera, 90° stills or even more could be useful, as the profile of the head could be used instead of, or as well as, the facial features themselves.

This 3D facial technology is currently been developed and tested by various software and security companies, and should increase the accuracy of facial recognition systems.

CONCLUSION

The current use of both multimedia database systems and the use of face recognition are both hampered by their hardware requirements. Methods have been produced that work quite successfully, but the number of systems that can effectively use the technology beneficially, are few. As the specification of computers double every eighteen months or so, it should not be long before the hardware required to run these system are both affordable and readily available.

More Effective Use Of Multimedia Databases And Face Recognition

Currently facial recognition system are only being used at airports, large chain casinos, by the police and by the military, they could be more effectively used through commercial involvement. Banks for example could use this technology to confirm identity and attractions could use it to survey who their most frequent visitors are.

This technology could also be used wirelessly, and bring massive advantages to services such as the emergency services and various commercial businesses. A small handset that takes a picture and sends it off to a central server for processing, returning the results within seconds would be extremely useful to paramedics for identifying people that are unconscious. The police could ensure that suspects aren’t initially lying about there identity, and roadside recovery could ensure that people are members even if they have forgotten their card or documentation. The uses and benefits for this are potentially endless and they would be virtually error free and without the need for ‘secondary’ identification (such as cards, tags or fobs).

Benefits And Negatives Of Using The Technologies

The benefits of these face recognition systems have mostly already been explained, but there are various ethical problems and other negatives that need to be addressed.

The first problem is that people often don’t trust new technology, so when implementing these systems, people need to be informed on how they work and what they do, and this can be of quite an expense. The next is the data protection act; all of the information stored within the database must follow the data protection act, which in itself introduces new technicalities and therefore more expense. The final problem that I will mention is the fact that people often change the way they look, put on or lose weight, and grow old, this would therefore introduce a whole new set of problems with the system and make it generally less trusted.

The Future Of This Technology

Face recognition and biometric identification in general will be used more and more, and multimedia databases are the only way, in my opinion, that all of this information can be stored and queried successfully from one central source. Lots of money has already been spent on research in these areas, and development of many relatively low-scale applications has taken place.

REFERENCES AND BIBIOGRAPHY

ADMS. (1992)
Face Recognition [online]
Available from: http://www.admsyst.com/face/face.htm
Continuing on: http://www.admsyst.com/comparison_bio.htm
[Accessed 18th October 2002]

Bonsor K., (1998-2002)
How Facial Recognition Systems Work [online]
Available from: http://www.howstuffworks.com/facial-recognition.htm/printable
[Accessed 18th October 2002]

Jacobs C., Finkelstien A., and Salesin D.  (1992)
Fast Multi-resolution Image Querying [online]
Available from: http://grail.cs.washington.edu/projects/query/mrquery.pdf
[Accessed 18th October 2002]

Kottas. J. and Dawson. B. (2000)
The Miros TrueFace Recognition Technology [online]
An abstract passage written on a handout for a presentation by The Boston Chapter of the IEEE’s Robotics and Automation Society.
Available from: http://www.ccs.neu.edu/home/cleary/ieee/abstracts/2000/0002.miros.txt
[Accessed 18th October 2002]

Manjunath B.S. (1992)
A Feature Based Approach To Face Recognition [online].
Available from: http://vision.ece.ucsb.edu/publications/92CVPR.pdf
[Accessed 18th October 2002]

Martinez Y. (A)
Psychology 398: Face Recognition [online]
Available from: http://www.cgl.uwaterloo.ca/~pjolicoe/398/handouts/pdf/face_recognition2.pdf
[Accessed 18th October 2002]