trystan.org

BBC news reports the AI agent at http://www.elbot.com/ convinced 3 out of 12 human investigators that it was “indistinguishable from them.[1]” In the experimental context, the participants don’t know if they’re text chatting with a human or the AI agent. If enough people exposed to the agent are tricked into believing they’ve been chatting with a human then the agent is said to have “passed the turing test.” This particular agent doesn’t appear to make any significant advances beyond others I’ve toyed with; they are really still toys.

Although I haven’t personally dug into the code for this critter, or similar programs, it seems likely that it uses traditional “Chomsky” grammar analysis techniques which would explain the apparent mastery of lexical structure but very limited power over semantics. It is very easy to design questions that expose the agent’s lack of any grasp of meaning. In fact, when asked “What is 2 minus 2?” it replied “…1.” When asked very abstract questions such as “What is the difference between objectivity and subjectivity?” It would typically respond with an attempt at humor that had no contextual connection at all. A more convincing response would simply be “I don’t know.” At least this would leave the possibility open that it was a young person or someone who never studied science or philosophy. Unfortunately, after only a few questions one is left with two possibilities. You’re talking to a program or an idiot that just happens to have perfect lexical structure, flawless spelling and never makes typos.

In the end, I’m not very surprised at the responses of the agent. I’m more surprised that 3 people actually deemed it “indistinguishable from a human respondent.” Although, I’m not exactly sure how that’s defined. I’m skeptical that a generative grammar type approach alone will ever produce a passable “conversation agent.” Regardless of whether language is emergent or nativist, there needs to be additional formal abstractions built to encapsulate some notion of understanding. For the nativist camp, I think this means building something atop formal grammars. For the emergent camp, this may mean realizing generative grammars within a connectionist context, such as a neural network. Having a layer underneath the grammar may provide insight on how to evolve the notion of “understanding.”

[1]: http://news.bbc.co.uk/2/hi/uk_news/magazine/7670050.stm

Macintosh computers and OS X have a reputation for usability, Linux does not, and Windows tends to sit somewhere in between in most people eyes. I run all three operating systems. The Ubuntu flavor of Linux being my primary workstation OS and Mac/OS X being my mobile solution. I run Windows XP and Vista on a VM on both Linux and OS X. Today, I was rather shocked when I plugged in my new Kodak camera into my Macbook and it was not auto-detected but it was on Ubuntu Linux! Then I removed the SD card and put it in a jump drive, still no action on OS X but Ubuntu auto-mounted it again and gave me a view of the file system.

What is the world coming to? Could it be that the open source community is actually surpassing Apple in terms of usability? Well, probably not quite yet in terms of the bigger picture but I remember when Linux was too imature to be a viable system for mainstream computing without a dual boot with some other OS. Certainly, that has not been the case for a couple years now. At this point, it is evident that strides are being made in the “it just works” department as well; an area where Linux was weak historically. Onward to glory!