I am a PhD student and a researcher in the Department of
Electronics and Computer Technology at the University of
Granada. I am working on a natural language dialogue system
that aims to answer product orders and questions of clients
in fast-food restaurants. It may be considered a rule-based
expert system whose behaviour is determined from a recorded
dialogue corpus obtained at a real restaurant. The system is
quite developed at the moment, though it needs some improvement
to enhance the level of understanding and naturalness.
I would like to get information about the available evaluation
methods of such a system, as well as information about the
evaluation of natural language dialogue systems in general (used
techniques, bibliography, web sites, etc.).
In order to provide more information, I enclose a short abstract
about the system I am working on.
--- Abstract ----
The system goal is to simulate the restaurant-clerk behaviour. It
must be able to provide information and ask client questions
similarly to how a human clerk does. In addition we
want it to process spontaneous voiced-speech, which at a
linguistic level means to consider phenomena such as unnecessary
word repetition, grammatical order change, anaphora, discordances,
context information, grammatical mistakes, etc. We also expect a
learning ability for the system to allow new information (foods,
drinks, ingredients, etc.) acquisition from client interaction.
The basis for the system development is as follows:
- Unnecessary information in client utterance: Usually,
not all words in a sentence are necessary to obtain its semantic
interpretation, which can be achieved from meaning words only
(keywords). To obtain such interpretation, the system uses
keywords and a keyword lattice analysis. This analysis is carried
out by means of syntactic and semantic rules. From dialogue corpus
we found out that clients usually use a small number of words in
their utterances (communication client-clerk tends to be
telegram-like), therefore a system dictionary can be size-reduced.
- Use of a small number of patterns: Clients tend to communicate
using a small number of patterns to order products, ask questions,
or modify previous product orders. Using these patterns the system
can extract most semantic meanings from clients' utterances. In case
the meaning cannot be obtained, clients are asked to help the system
understanding process or to repeat the utterance input differently.
The system is a compound of several modules: Input Interface, Control
Module, Memory Module, Restaurant-product Knowledge Base, Lexicon, and
Output Interface.
At the moment the system takes about 30.000 C++ code lines. Its inputs
and outputs are natural language text sentences.
Its Input interface is well developed but still needs to define some
syntactic and semantic rules, since now only product orders and
questions are carried out.
We are about to start the Modification Module set up. This module
will
be activated when the desire of modification of previous orders is
detected in client input.
Also, the Learning Module needs to be started. This module will be
activated when "possible" unknown foods, ingredients, drinks, etc.
are detected in client input. These new products will be learnt, so
they could be recognized the next time they appear in client
sentences.
The Natural Language Generator needs improvement to enhance the
expression power, though at the moment, the system can build
both syntactically and semantically right sentences, in a very
natural fashion, by using pronouns and context information available
at the moment of the natural language generation.
The system uses a graphic interface that now is useful but simple. In
future we would like to improve it by including product-pictures and
graphics of the "artificial" restaurant-clerk face, in order to
improve a friendly communication.
We think the integration of the system in a voice-controlled
response system represents its best application. To
do so, it would need a speech-to-text interface that
provides a text-word sequence from client voice. A text-to-speech
interface should transform the system output into synthesized voice.
Theoretically the whole system could be part of an
automatic front-end dialogue system for clients in restaurants,
or for those at home who use telephone for ordering.
--- End of Abstract ------
I do not know if this short abstract would be enough for you to
get an idea of the system, so in case you need any further
information, or in case you have any comment or remark, please
let me know.
I look forward to hearing from you soon. Thanks again.
Sincerely,
Ramon Lopez-Cozar Delgado
Electronics and Computer Technology Dept.
University of Granada
18071 Granada
Spain.