These are the unexpurgated reviews that we received on our CHI submission about the Barney hack. Basically, they fall into two groups. One set of reviewers argue that the paper is a joke, and therefore should not be published. The other set argue that the paper is not a joke, and so should be published. I think both of these sets of people were wrong.
Of course, it was actually published. Presenting it with a straight face proved in the end to be the biggest challenge, especially after I had spiced it up with some imaginary scenarios of Major HCI Figures using our system in their everyday work. Our summer intern, who had done most of the work after all, was originally due to present it, but bailed at the last minute. I can't blame him.
There was always going to be some dispute about this paper. I learned later that we were fortunate enough to have an advocate on the program committee who had delighted in the bizarre nature of the paper and had fought hard to get it accepted for publication. I shall not name him here but he recruited me the following year to help him push through another paper in a similar vein. He opened his recruitment pitch with, "You owe me a karmic debt". And so I did.
Dear Colleague,
Thank you very much for submitting the following paper to CHI 99:
SWEETPEA: Software Tools for Programmable Embodied Agents
We are happy to inform you that this paper has been accepted for inclusion
in the CHI Technical Program. As an accepted paper, your work will be
published in the CHI Conference Proceedings. You will shortly be receiving
instructions for preparing and formating your paper for publication.
Attached to this email you will find a set of comments from the paper's
reviewers, together with the meta-review prepared by a member of the
Program
Committee. These reviews contain detailed comments on your paper that you
may find useful when revising it for inclusion in the conference
proceedings. Note that the meta-review is primarily a summary of the other
reviews, sometimes including a recommendation to the Program Committee on
how to interpret the reviewers' scores. We hope that you will respond to
reviewers' suggestions for improvement. Feel free to contact us if you
have
any questions about preparing your final contribution to the proceedings,
or
about your presentation at the conference.
It is expected that authors with accepted papers will attend the CHI
Conference and present their papers. Advance Programs will be mailed in
the
beginning of January, informing you when your paper is scheduled for
presentation. We urge you to register for the conference early to secure
the pre-registration discount.
Thank you for your interest in CHI 99. We are looking forward to meeting
you
in Pittsburgh.
Kate Ehrlich and William Newman
CHI 99 Papers Co-Chairs
chi99-papers@acm.org
----------
Reviews for Paper #:99092
Paper Title: SWEETPEA: Software Tools for Programmable
Embodied Agents
NOTE: Some punctuation characters (i.e., ' represented as ? ) may have
reproduced incorrectly in the reviews.
We apologize for any confusion this causes.
----------
Document ID: 99092-0 META-REVIEW
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
The authors discuss how devices like the Actimates "Barney" can be adapted
as
control and information-providing devices. This required
reverse-engineering
the operation of the device, learning how to control it for new purposes,
and
-- most importantly -- thinking carefully about what it means for such a
device
to play an interactive role with people, and how this affects how we think
about computing.
2. Select the phrase that best describes the paper's contribution:
Design for interactive system
3. Review of the paper:
This is and will be a very controversial paper. Reviewers split very
cleanly
on this paper: Half saw it as a very significant investigation into how
external, portable, character-like devices can play a role (literally!) in
our
interactive/computational life; the other half saw the paper as an
almost-trivial writeup of some people fiddling around with a toy.
I feel very strongly that the paper should be accepted. It is certainly a
non-traditional paper that addresses a non-traditional topic with
non-traditional technology. But I believe the authors have done it
carefully
and well. They have not simply fiddled around with a toy, but have thought
carefully about notions of role and character, and how even a simple device
like that studied here can offer insights into computing. This is not to
say
that I expect we will soon all have Barneys sitting next to our machines.
But
I do expect that future systems will be able to learn a great deal from the
work here, which is where the direct benefit will come from. And it
certainly
won't be a boring presentation....
4. Comment on the paper's written presentation:
The paper is very well written; the occasional bits of humor don't detract
from
its content (or, at least, it didn't for me). I do agree with the points
of
some reviewers that too much space is devoted to the reverse-engineering
aspects of the project, and that the "Barbie" discussion could be safely
omitted. Figures are fine; international readers should have no problems.
5. Provide any other comments that may be useful to the authors.
----------
Document ID: 99092-1
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
The paper describes an experiment in using a commercial toy (a Barney doll)
as
an interaction device.
2. Select the phrase that best describes the paper's contribution:
Experience: including design briefing / case history
3. Review of the paper:
This has to be one of the funniest papers I have ever read and I certainly
do
not believe CHI should reject a paper merely because it treated a subject
humorously. However, I felt there were some more serious problems with the
paper.
In particular, I found the work not particularly significant in that it
reports
on one summer's worth of experimentation with one toy. There is a laudable
attempt to produce general principles and guidelines from this experience
but I
would rather see a broader base of experience before reaching for such
guidelines.
Finally, I find the author(s) use of the toy in a work/office context to be
inappropriate. Their idea of using "PEAs" (Programmable Embodied Agents)
follows in a tradition of using plush toys that goes back at least to
Druin,
but these uses have been educational and entertainment-oriented. One of
the
important principles of graspable user interfaces is that the form of the
interface tool accords with the task for which it is to be used. Mapping
the
waving of Barney's arms to the status of a copier is arbitrary at best.
4. Comment on the paper's written presentation:
The presentation is quite good and the humor helps a good deal.
5. Provide any other comments that may be useful to the authors.
6. Rating of Paper's acceptability (1 is low, 5 is high): 2
7. Rate your expertise in the topic area of this paper (1 is low, 5 is
high): 4
----------
Document ID: 99092-2
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
This paper is part reverse engineering of the ActiMates Barney (a big hit
at
CHI?98 and promising to have a continued presence at CHI?99), part
exploration
of bleeding edge HCI issues like character-based interfaces and embodied
interaction, and part sheer fun. The authors have deconstructed Barney?s
API
in order to use its capabilities to express other applications,
particularly as
proxies for or awareness of system activity or other users. The
contribution
of the paper is the exploration of the use of embodied interaction through
a
character, and (at least for the authors) the demonstration that "off the
shelf" technology can be adapted to support new applications.
2. Select the phrase that best describes the paper's contribution:
Tool for interactive system design/development
3. Review of the paper:
This work addresses one of the most interesting new areas of HCI ? namely
character-based interfaces and embodied interaction. The authors provide
an
excellent, if short, review of related work, including ubiquitous computing
and
Ishii?s tangible bits work at MIT. The "lessons learned" from the
literature
review (last paragraph in the Related Work section) is the most valuable
conceptual contribution of the paper, and it would be great to have a
fuller
discussion of it, particularly in the context of the framework presented
later
in the paper. To that end, the discussion of Barbie might be dropped,
since it
is not needed to understand the rest of the paper (or at least condensed
into a
paragraph, since Barbie is not studied or discussed further). The
discussion
of how embodied interactions and character-based interfaces are different
from
"computer" interfaces, and what kinds of messages they are particularly
well-suited for is one of the main benefits others can gain from this work.
The authors stress (
over-stress, I believe) the value of the demonstration that these kinds of
applications can be built from "off the shelf" components (i.e., by
hacking, or
is it back-hacking? Barney), but it is not clear how satisfactory their
Barney
behaviors are, particularly without the luxury of arbitrary speech.
Without a
fairly complete ability to program Barney (wrt the behaviors he has), it
seems
difficult to argue that it is possible to consider it a development
platform,
and it tends to make the applications the authors create seem a bit like
"stupid Barney tricks." But they are not just that ? at least, to the
extent
that they constitute a meaningful testbed for exploring ideas about
embodied
interfaces & characters, and the authors discuss their applications in
these
terms. So although I think the paper is acceptable the way it currently
stands, I think it would be much stronger and make a more significant
contribution to the CHI community if these aspects were better highlighted
in
the discussion, and better i
ntegrated with the literature mentioned in the introduction as related
work.
For example, all of the sections at the end elaborating the framework
should be
discussed together and summarized at the end ? I read through them all, but
did
not come away with any unified idea about what had been discovered about
character-based interfaces and/or embodied interaction.
The work is original and valid.
4. Comment on the paper's written presentation:
The writing is clear and appropriate for an international audience. The
figures are gorgeous; too bad they can?t be reproduced in color (it would
not
be worth taking them out of context to get them in color). I don?t get the
"SWEETPEA" in the title, though ? ok, she is a character in Popeye, but
that
won?t be understood by many non-Americans. PEA is programmable embodied
agents
? is SWEET supposed to be "software tools"?? The acronym/term is never
used in
the paper. I would suggest that the title just be "software tools for
programmable embodied agents" (or feel free to preface it by "Tangible
Stuffed
Bits" ? see below ;-)
5. Provide any other comments that may be useful to the authors.
I thoroughly enjoyed reading this paper, and am sure it will
be well-received
by the CHI?99 audience. By the end, I was envisioning all the beanie
babies,
teletubbies, and other stuffed characters my colleagues and I keep atop our
monitors and on our desks coming to life in a veritable "tangible stuffed
bits"
universe. But rather than recoil in horror, it reveals something of what
the
future of billions of (physically) detached, networked objects might hold.
I
think this is the kind of paper that will entertain as it broadens thought;
perhaps not the most erudite of HCI work, but definitely making a
contribution.
6. Rating of Paper's acceptability (1 is low, 5 is high): 4
7. Rate your expertise in the topic area of this paper (1 is low, 5 is
high): 3
----------
Document ID: 99092-3
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
This paper describes the use of various "multimedia" children's toys
as user interface components.
2. Select the phrase that best describes the paper's contribution:
Experience: including design briefing / case history
3. Review of the paper:
This is a sort of interesting paper but it is really in the form of a
product announcement. There appears to be no science here and very
little engineering. I got the feeling that this was the description
of some out-of-hours activity stimulated by the desire to make a toy
work in a way not originally intended by the manufacturers.
A real problem I have with the work is that most of it appears to be
reverse engineering how to make the product behave as desired instead
of as intended.
At the end of the day the phrase is very much "So what?"
4. Comment on the paper's written presentation:
Essentially OK except that the author sometimes uses first person
singular and sometime first person plural. This really irritates
since the implication is that for part of the work there was a team
but I do not think this is the case.
5. Provide any other comments that may be useful to the authors.
6. Rating of Paper's acceptability (1 is low, 5 is high): 2
7. Rate your expertise in the topic area of this paper (1 is low, 5 is
high): 3
----------
Document ID: 99092-4
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
The paper explores the use of new, networked toys (e.g., Talk-With-Me
Barbie,
Actimates Barney) as forms of computational interaction.
2. Select the phrase that best describes the paper's contribution:
Interaction technique
3. Review of the paper:
I have to say I couldn't tell if this was serious or not. The idea of
using
that annoying little dinosaur in office contexts had me questioning the
author's sanity for a bit. But I decided that it probably was for real.
Contribution. Pretty clear to me that there's something here. Using
existing
toy devices to provide additional i/o channels is an interesting tweak on
the
some of the tangible/ambient media work pioneered by Hiroshi Ishii and
others.
Unlike Ishii, the authors propose exploiting features of well-known
characters
using off-the-shelf toys. Although we are seeing more techniques that
exploit
physical objects as i/o devices, i think there's something important about
the
use of well known characters. Plus the use of these technologies in office
settings vs. playrooms.
Having said that, I'll try to provide some suggestions to the authors. My
main
complaint is the overemphasis on reverse engineering Barney. A huge chunk
of
the paper talks about how the designers had to interpret Barney's MIDI
stream.
For one, I suspect they could have asked Erik Strommen at Microsoft for the
protocol. Two, I could care less about its protocol. Every toy will have
its
own protocol until some set of "toy networking" standards are developed, so
it
just doesn't matter. What *does* matter are the types of interaction that
you
can squeeze out of the purple one. Other reviewers may disagree with me on
this one. But I feel that whole "Exploring Barney's Technology" section
could
be eliminated. Or replaced by the sentence, "We had to spend time
understanding the communications protocol before we could develop an API."
Same goes for the talk about LPC and Barney-speech. I don't care about
these
issues, I only care about what you can do with the current implementation.
The other complaint is the (in my opinion) huge assumption that people like
to
communicate with autonomous characters in computing environments. The
authors
give the example of the Microsoft Paper Clip agent as an interactive proxy.
While they don't say whether the paper clip man is a successful example or
not,
I know I've never met *anyone* who actually likes that feature. I mean
*nobody*.
Ben Schneiderman has posed many arguments against the use of
anthropomorphic
characters, yet his pleas are almost always ignored by those working on
character-based agents.
Whether or not people really want talking dinosaurs, it is at least useful
to
acknowledge that there is some dispute over this. Instead of boldly
stating,
"these characters embody a stronger sense of identity and agency than can
be
conveyed by a typical graphical interface. As a result, then, they can
serve
better as channels for carrying particular sorts of information ... (p. 2,
related work section)" It just seems like a jump to imagine that a winking
Barney gives me any more context than a dialog box stating, "i messed up
your
print job." I'm being grumpy about this point, but I think it's fair to
state
the arguments against anthropomorphic agents.
The paper does a nice job of spelling out the various applications that
have
been developed with Barney. The real issue for me is the lack of any
evaluations - formal or informal - of these applications. I'd be happy
with
knowing how people in the authors' workspace felt about Barney at the print
queue or flagging incoming mail. Do people really see this type of
technology
as useful? Whether anthropomorphic or not, it would be useful to get a
sense
for reactions to computing that lives outside the monitor and keyboard.
This
paper does what much ubiquitous computing research falls prey to: nice
technology, where's the impact? So while I think the work is interesting,
it
does nothing to advance my understanding of how such devices could be
deployed
in my workspace.
4. Comment on the paper's written presentation:
generally well written. argument comes across loud and clear. i loved the
figures... as long as international readers understand
"super-dee-duper"(tm), i
think the paper satisfies that point as well.
5. Provide any other comments that may be useful to the authors.
I griped a lot, but I do think the paper adds a new
dimension to the ubiquitous
computing paradigm. Again, one day I'm sure someone will take the time to
actually think about what these things do for people, but until then...
So I weigh towards acceptance of the paper. But if informal feedback from
users could be added, I'd feel even stronger about the work.
6. Rating of Paper's acceptability (1 is low, 5 is high): 4
7. Rate your expertise in the topic area of this paper (1 is low, 5 is
high): 5
----------
Document ID: 99092-5
1. Briefly summarize the paper in your own words, making sure
to identify the paper's contribution to the field of HCI:
Identifies interactive devices entitled Programmable Embodied Agents and
describes a quick study of features of an prototype PEA Interactive
Barney?,
resulting in an ?Interaction Framework?.
2. Select the phrase that best describes the paper's contribution:
Interaction technique
3. Review of the paper:
Concise introduction clearly stating area of discussion and aims of the
paper.
The Related Work section is not very comprehensive: the last paragraph is
not
justified in the context of related previous work in this area. In
addition, it
has a limited set of references relying on past CHI papers as a mainstay
and
the author does not take into account non-US work or work in the filed of
Intelligent Agents (other than by Laurel) as I would have expected. The
Technical Basis section is overlong with too much detail on interaction
protocols: this part of the paper is more suitable for a technical report
rather than a CHI submission. It would have been much more appropriate to
focus
on details of the software infrastructure: the figure given (Fig. 2) is
fairly
meaningless without explanation and claims that the applications are
?lightweight, portable and seamless? are not justified. More information on
the
widget set would have been useful and made this paper more interesting and
innovative: there is enough informa
tion to whet the appetite but not enough for a comprehensive understanding
of
what the author?s aims were. Again, Figure 4 is not explained at all well
and
the framework is not introduced or described clearly. The Channel/Person
category especially hard to understand: simply referring the reader to
other
references is not sufficient. In all, the work, as the author states, was a
?brief exploration? of a field which merits more detailed and careful
investigation and a more carefully reasoned and thoughtful presentation.
Contribution - This is a contribution to a field which may well prove to be
a
very promising one in terms of interaction techniques and new designs.
However
this is obviously initial exploratory work, presented and written (I would
guess) very quickly and needs a much more careful and well-designed study
with
all assumptions clearly stated and rationale suitably justified. This paper
does not do that but may be of interest as the basis of future studies.
Validity - Very little to comment on in terms of rationale: work presented
seems to be mostly in terms of ?:these are my experiences?.
Originality - Original but with a limited base.
4. Comment on the paper's written presentation:
Written presentation - Sometimes well and clearly written but lacking in
consistency. The text does become dense in places (Related Work?s last
paragraphs for example) and with a number of throw-way lines or a ?chatty?
style in others. In addition, the references in the paper are not in CHI
standard format.
5. Provide any other comments that may be useful to the authors.
6. Rating of Paper's acceptability (1 is low, 5 is high): 2
7. Rate your expertise in the topic area of this paper (1 is low, 5 is
high): 3