Automatic Caption Localization for Photographs on World Wide Web Pages

Bibliographic description
ROWE, Neil C.; FEW, Brian. Automatic Caption Localization for Photographs on World Wide Web Pages [on line]. Monterey: Department of Computer Science, U. S. Naval Postgraduate School, 1998. Available on: http://www.nps.navy.mil/Content/CS/ncrowe/marie/webpics.html"

Dublin Core
Title : Automatic Caption Localization for Photographs on World Wide Web Pages
Creator : Neil C. Rowe, Brian Frew
Subject : photograph retrieval / Photograph indexing / World Wide Web
Description : "Pictures, especially photographs, are one of the most valuable resources available on the Internet through the popular World Wide Web. Unlike text, most photographs are valuable primary sources of real-world data. Unlike conventional copy technology, photographs on the Web maintain their quality under transmission. Interest has increased recently in multimedia technology as its speed has improved by hardware and software refinements; photographs also benefit from these advances. This has meant that multimedia resources on the Web have grown quickly. For these reasons, indexing and retrieval of photographs is becoming increasingly critical."
Publisher : Department of Computer Science, U. S. Naval Postgraduate School, Monterey
Date : 1998
Type : Article
Format : HTML
Identifier : http://www.nps.navy.mil/Content/CS/ncrowe/marie/webpics.html
Source: http://www.nps.edu/
Language : En
Relation : -
Coverage : USA
Rights : U. S. Naval Postgraduate School

Abstract
"A variety of software tools index text of the World Wide Web, but little attention has been paid to the many photographs. We explore the indirect method of locating for indexing the likely explicit and implicit captions of photographs. We use multimodal clues including the specific words used, the syntax, the surrounding layout of the Web page, and the general appearance of the associated image. Our MARIE-3 system thus avoids full image processing and full natural-language processing, but shows a surprising degree of success. Experiments with a semi-random set of Web pages showed 41% recall with 41% precision for the task of distinguishing captions from other text, and 70% recall with 30% precision. This is much better than chance since actual captions were only 1.4% of the text on pages with photographs."

The Subject Analysis of Images: Past, Present and Future

Bibliographic description
WARDEN, Ginger; DUNBAR, Denise; WANCZYCKI, Catherine; O'HANLEY, Suanne. The Subject Analysis of Images: Past, Present and Future [on line]. University of British Columbia School of Library, 27th March 2002. Available on:
http://www.slais.ubc.ca/people/students/student-projects/C_Wanczycki/libr517/homepage.html

Dublin Core
Title : The Subject Analysis of Images: Past, Present and Future
Creator : Ginger Warden, Denise Dunbar, Catherine Wanczycki, Suanne O'Hanley
Subject : image collection / image classification / thesaurus / image indexing
Description : "The Art and Architecture Thesaurus (AAT) is a structured vocabulary that can be used to improve access to art, architecture, and material culture."
Publisher : University of British Columbia School of Library
Date : 2002-03-27
Type : Web site
Format : HTML
Identifier : http://www.slais.ubc.ca/people/students/student-projects/C_Wanczycki/libr517/homepage.html
Source: http://www.slais.ubc.ca/
Language : En
Relation : -
Coverage : UK
Rights : No

Extract
"Image collections exist for many purposes: medicine (ultrasounds, CAT scans), architecture (building plans), geography (aerial photos, maps), art (paintings, cartoons), business (trademarks), history (photographs). Some image collections are very large. The Getty Institute's Photo Study Collection, for example, has over two million photographs. Indexing collections of this size can be extremely time consuming, and unlike text, images cannot be searched by keyword. Many automatic indexing systems have been developed, but what computers can currently extract from images are "mostly low-level features" (Rui, 1999) like color, shape, and texture. Research on the information needs of users, and on human perception of images may, in time, contribute the knowledge needed to produce the most precise and efficient retrieval systems possible.

In the meantime, librarians contending with image collections have to make decisions about how best to provide access to them. Currently, there is no universal consensus in libraries. In a survey of 58 libraries in the U.K., (Graham, 1999) the clear majority of respondents employed in-house methods of classifying and indexing their collections, rather than relying on publicized schemes, such as the AAT (Art and Architecture Thesaurus), LCTGM (Library of Congress Thesaurus for Graphic Materials), and LCSH (Library of Congress Subject Headings). This is likely the result of tradition. Curators of image collections were left to their own devices for most of the century, insofar as subject headings for images went, while LCSH concentrated on primarily text-based materials. Many different thesauri were developed by individuals or groups of individuals to deal with particular collections but efforts to create a universally acceptable indexing language for images has only been a point of interest in the past 30 years or so, with the increasing volume of available images and the desire for increased resource-sharing between institutions.

The AAT and LCTGM are presently the two most widely accepted vocabularies for use with image collections. Their development, structure and scope are the main focus of this website. Subject headings from each are applied to several types of images by way of example. We also look to the past and future of subject access to images by surveying both the methods librarians have used in the past (and are still using today to some extent) and the methods that are currently being developed (and to some extent already in place)."

IPTC Standard

Bibliographic description
The IPTC-NAA standards [on line]. Controlled Vocabulary. Available on:
http://www.controlledvocabulary.com/imagedatabases/iptc_naa.html

Dublin Core
Title : The IPTC-NAA standards
Creator : ?
Subject : metadata / IPTC / image description / image database
Description : "A controlled vocabulary can be useful in describing images and information when organizing and classifying content for image databases."
Publisher : Controlled Vocabulary
Date : ?
Type : Article
Format : HTML
Identifier : http://www.controlledvocabulary.com/imagedatabases/iptc_naa.html
Source : http://www.controlledvocabulary.com/
Language : En
Relation : -
Coverage : ?
Rights : -

Extract
Each image file can be saved using Adobe Photoshop with this text information embedded within the file. Anyone that's worked around newspapers, with digital images or image databases for a while has probably heard the acronyms IPTC or IPTC-NAA tossed around, usually when discussing the use of the File Info feature of photoshop. But few understand what they mean or stand for. The short story is that IPTC, the International Press Telecommunications Council, was one of the groups responsible for encouraging the standards necessary to“marry” the text information describing an image with the image data itself. The NAA is the Newspaper Association of America (formerly ANPA), and they also have been responsible for developing standards for exchanging information between news operations, including information used to describe images. [...]
Standards regarding metadata for news images have evolved over time, beginning in the 1970's when some were first issued as“guidelines.” However, most of these efforts were regional in nature, and focused on text. As news organizations moved from manual typewriters to CRTs (Cathode Ray Tubes) and VDTs (Video Display Terminals) these standards were revised and became more specific. Only later, as the world embraced the web, did the standards begin to address multimedia content.
In 1979, the International Press Telecommunications Council (IPTC) approved its first news exchange standard IPTC 7901. This provided metadata and content in plain text only; the only delimiters allowed were spaces and line breaks...

Real-Time Computerized Annotation of Pictures

Bibliographic description
LI, Jia; Z.WANG, James. Real-Time Computerized Annotation of Pictures [on line]. The Pennsylvania State University, University Park, 25 July 2006. Available on: http://infolab.stanford.edu/~wangz/project/imsearch/ALIP/ACMMM06/li06.pdf

Dublin Core
Title : Real-Time Computerized Annotation of Pictures
Creator : Jia Li and James Z. Wang
Subject : digital picture / indexing / automatic indexing
Description : An article about automated annotation of digital pictures and the web site ALIPR (Automatic Linguistic Indexing of Pictures).
Publisher : http://infolab.stanford.edu/
Date : 2006-07-25
Type : article
Format : PDF
Identifier : http://infolab.stanford.edu/~wangz/project/imsearch/ALIP/ACMMM06/li06.pdf
Source : http://infolab.stanford.edu/
Language : En
Relation : http://www.alipr.com/, http://wang.ist.psu.edu/docs/home.shtml
Coverage : USA
Rights : ACM Multimedia Conference

Abstract
Automated annotation of digital pictures has been a highly challenging problem for computer scientists since the invention of computers. The capability of annotating pictures by computers can lead to breakthroughs in a wide range of applications including Web image search, online picture-sharing communities, and scientific experiments. In our work, by advancing statistical modeling and optimization techniques, we can train computers about hundreds of semantic concepts using example pictures from each concept. The ALIPR (Automatic Linguistic Indexing of Pictures - Real Time) system of fully automatic and high speed annotation for online pictures has been constructed. Thousands of pictures from an Internet photo-sharing site, unrelated to the source of those pictures used in the training process, have been tested. The experimental results show that a single computer processor can suggest annotation terms in real-time and with good accuracy.

Tribune company photo archiving task force : keys words (enhancement terms), and photo type words for digital photo archives

Bibliographic description
SLA NEWS DIVISION. Tribune company photo archiving task force : keys words (enhancement terms), and photo type words for digital photo archives, usage guidelines and alternatives [on line]. SLA News division, July 1995. Available on: http://www.ibiblio.org/slanews/conferences/sla1998/tribusage.html

Dublin Core
Title : Tribune company photo archiving task force : keys words (enhancement terms), and photo type words fer digital photo archives, usage guidelines and alternatives
Creator : SLA News division
Subject : Photography / Indexing / Keywords / Index terms / Digital photograph
Description : It's a keywords list for indexing press photos
Publisher : ibiblio.org
Date : 1995-07
Type : Guide
Format : HTML
Identifier : http://www.ibiblio.org/slanews/conferences/sla1998/tribusage.html
Source : ibiblio.org
Language : En
Relation : 1998 SLA News Division Preliminary Program
Coverage : USA
Rights : No

Extract
"ABUSE
ABORTION
ACCIDENT
ADVERTISING
AGRICULTURE - see also GARDEN, FARM, RANCH.
AIDS - the disease, not implements that assist. Use with DISEASE.
not air force; use MILITARY.
AIR
AIRCRAFT - use for helicopters, commercial airliners, bombers, spy planes, and inflatable transportation vehicles, such as weather balloons, the Goodyear blimp, etc. See also SATELLITE, SPACECRAFT.
ALCOHOL - see also BEVERAGE, WINE.
not alien; use IMMIGRANT, MIGRANT, REFUGEE, TRAVEL.
not amusement park; use ATTRACTION.
ANATOMY - see also BODY, NUDITY.
ANIMAL
ANNIVERSARY
ANTIQUE
APARTMENT
APPLIANCE - see also EQUIPMENT.
ARCHAEOLOGY
not archery; use ** in supplemental category field.
not archive; use LIBRARY or MUSEUM.
ARCHITECTURE
not army; use MILITARY.
not arms or armament; use WEAPON.
not arms control; use WEAPON and CONTROL.
ARREST
ART - a photo of a painting, collage, etc.; see also SCULPTURE.
ASSASSINATION
ASTRONAUT
ASTRONOMY - includes asteroids, comets, galaxies, planets (except earth), stars, and other celestial objects. See also ECLIPSE, MOON, SUN.
ATHLETE - use to distinguish between persons with same or similar name.
ATTRACTION - A place or event to which a tourist might want to go.
AUCTION
AUDIO - use with EQUIPMENT for audio equipment.
AUTO
AUTUMN
AVALANCHE
AWARD - includes medal, trophy, certificate, honorary degree, national honor.

BABY - includes last trimester of pregnancy through age 2; Age 3 to teens use CHILD; then TEENAGER or JUVENILE, as appropriate.
not ballet; use DANCE.
BALLOON - toy; use AIRCRAFT for inflatable air transportation vehicles and weather balloons.
not band; use MUSIC and GROUP.
BANK - includes depository, S&L, credit union, etc.
BAR - see also RESTAURANT..."

Image classification for content-based indexing

Bibliographic description
VAILAYA, Aditya; FIGUEIREDO, Mario A . T; JAIN, Anil K; ZHANG, Hong-Jiang. Image classification for content-based indexing [on line]. IEEE EXPLORE, january 2001. Available on: http://citeseer.ist.psu.edu/correct/686126

Dublin Core
Title : Image classification for content-based indexing
Creator : Aditya Vailaya, Mario A. T. Figueiredo, Anil K. Jain, Hong-Jiang Zhang
Subject : Image classification / Content-based / Indexing
Description : "Grouping images into (semantically) meaningful categories using low-level visual features is a challenging and important problem in content-based image retrieval."
Publisher : IEEE EXPLORE
Date : 2001-01
Type : article
Format : HTML
Identifier : http://citeseer.ist.psu.edu/correct/686126
Source : http://citeseer.ist.psu.edu/
Language : En
Relation : http://citeseer.ist.psu.edu/nrelated/1894793/686126
Coverage : USA
Rights : Copyright 2001 IEEE

Extract
"Grouping images into (semantically) meaningful categories using low-level visual features is a challenging and important problem in content-based image retrieval. Using binary Bayesian classifiers, we attempt to capture high-level concepts from low-level image features under the constraint that the test image does belong to one of the classes. Specifically, we consider the hierarchical classification of vacation images; at the highest level, images are classified as indoor or outdoor; outdoor images are further classified as city or landscape; finally, a subset of landscape images is classified into sunset, forest, and mountain classes. We demonstrate that a small vector quantizer (whose optimal size is selected using a modified MDL criterion) can be used to model the class-conditional densities of the features, required by the Bayesian methodology. The classifiers have been designed and evaluated on a database of 6931 vacation photographs. Our system achieved a classification accuracy of 90.5% for indoor/outdoor, 95.3% for city/landscape, 96.6% for sunset/forest and mountain, and 96% for forest/mountain classification problems. We further develop a learning method to incrementally train the classifiers as additional data become available. We also show preliminary results for feature reduction using clustering techniques. Our goal is to combine multiple two-class classifiers into a single hierarchical classifier."

Text and Photo : database enhancement terms

Bibliographic description:
WILLEN BROWN, Stéphanie. Text and Photo : database enhancement terms [on line]. Special Libraries Association News Division, 1st september 2001. Available on: http://www.ibiblio.org/slanews/archiving/terms/index.htm

Dublin Core:
Title : Text and Photo : database enhancement terms
Creator : Stephanie Willen Brown
Subject : enhancement terms / indexation / newspaper
Description : Links to database enhancement terms used in newspaper libraries across the United States.
Publisher : www.ibiblio.fr
Date : 2001-09-04 (last updated)
Type : Text
Format : HTML
Identifier : http://www.ibiblio.org/slanews/archiving/terms/index.htm
Source : http://www.ibiblio.org
Language : En
Relation : -
Coverage : USA
Rights : Amy Disch


Extract

"These pages provide links to database enhancement terms used in newspaper libraries across the United States.
Newspaper librarians use enhancement terms to assist in the full-text retrieval of news articles and photos from large databases.
Enhancement terms are also known as keywords or subjects.
Chicago Tribune
Tribune Company Photo Archiving Task Force
Detroit Free Press
The News & Observer, Raleigh, N.C.
Sacramento Bee
St. Petersburg Times
Spokane Spokesman-Review 
Springfield (MA) Union-News "

Indexing Photographs

Bibliographic reference
WILLEN BROWN, Stephanie. Indexing photographs [on line]. Springfield (Mass.): Union-News and Sunday Republican, Visual Edge '98 Archive Program, 4th September 2001. Available on: http://www.ibiblio.org/slanews/archiving/VE98/presentation.htm


Dublin Core
Title : Indexing photographs
Creator : Stephanie Willen Brown
Subject : newspaper photograph / indexing / keyword / free text
Description : It's about "newspaper photograph indexing, the use of keywords or free text to accomplish the indexing, and several examples of indexed newspaper pictures."
Publisher : http://www.ibiblio.org/
Date : 2001-12-04
Type : Conference
Format : HTML
Identifier : http://www.ibiblio.org/slanews/archiving/VE98/presentation.htm
Source : http://www.ibiblio.org/
Language : En
Relation : http://www.ibiblio.org/slanews/archiving/VE98/indexing.htm
Coverage : USA
Rights : Amy Disch


Extract
"Introduction
It is essential to have several ways to define the activity a photograph describes because describing the exact meaning of a picture is very difficult. When indexing works of art or music, this is especially tough, because only terms added by a librarian can be used to search the database. Newspaper photograph databases are easier to search because the cutline field is a wonderful source of information. The cutline usually has most of the information relevant to the image, including names of subjects, location, and a description of the activity.

In this presentation, I will (briefly!) discuss newspaper photograph indexing, the use of keywords or free text, and review several examples of indexed newspaper pictures. I have pulled 18 photographs from the Springfield Union-News to demonstrate how both keywords and free text indexing might be applied to a variety of situations.

Different Kinds of Searching
Searching the cutlines of hundreds of thousands of photographs to retrieve one of the principal of a grammar school would be relatively easy: the searcher – librarian, reporter, editor, photographer – would simply type in the name and a small number of photographs would likely be retrieved (assuming the person is not a trouble maker or married to a prominent figure). However, searching for an appropriate photograph of the mayor of the dominant city in a newspaper’s coverage area, Springfield in our case, would likely retrieve hundreds of images.

And what if you needed photographs of golf courses to accompany a story about the proliferation of golf courses in your area? You would have to remember the names of all the golf courses in the area and do a complicated, nested search. — Franconia, Crestview, the Orchards — You might forget the name of one or two of them, or you might retrieve photographs of events taking place at the "19th hole" of an area course.

A further curve thrown into the photo indexing mix is that different types of people — photographers, librarians, editors, and the general public — will need to search the database. Each group will need to retrieve different kinds of pictures:

o photographers might want to see if a particular scene had already been shot; (images of Taste of Holyoke, for example)
librarians would be looking for a photograph of a prominent politician;
Living/Arts editors might want a picture of last year’s big event (Shriner’s auto show) for an advance of this year’s event;
the general public might want a photograph of little Janey playing field hockey.

Indexing is the Answer
Applying subject terms to each photograph will greatly aid in retrieval. Ideally, each image will be assigned two to five subjects, addressing the central news aspect of the picture. This results in a richer description of a photograph. Indexers should add words that are not in the cutline to enhance the value of the subject field. A picture of a defendant in a murder trial, for example, would be assigned the keywords Murder and Trial. Either one used by itself is not enough to describe a photo of a defendant charged of murder who is on the witness stand, though both are correct and useful. But together, they accurately express the concepts demonstrated in the photograph.

Keywording vs. Free text
We will discuss two different ways of adding subject ideas to a photograph database: keywording and free text. I’ll describe, and show examples, of each.

"Keywording" refers to adding terms from a controlled vocabulary to a database of photographs to aid later retrieval. The most important component of keywording is the notion of a controlled vocabulary, a specific set of words from which index terms can be taken. Keywording is the a traditional means of indexing photographs, and is taught as Indexing in library school. A photograph of children on a swing, for example, might be given the keywords CHILD; PLAYING; and SUMMER.

"Free text" indexing, on the other hand, does not rely on a controlled vocabulary. Instead, it is more like free association: the indexer looks at the photograph and uses her imagination to describe what it is "about." A photograph of children on a swing, for example, could be "about" a hot summer afternoon; children or kids; swinging or playing; brothers, perhaps, if the children are related; smiling or laughing; and having fun."