03. Is a Chat with a Bot a Conversation? (by Jill Lepore)

Introduction

The article linked below (copied from the New Yorker) touches on several things related to chats as an interface, and the history therein. As we move forward, continue to think about how this relates to our other discussions about interfaces, be they doors, (GUI) windows, or text prompts in Unix.

This touches on several historical precedents to services like ChatGPT, including ELIZA, and The Mechanical Turk. Think about these, again in relation to how software, and hardware interfaces, and related programs serve interests, and whose interests they ultimately serve.

Relevant Dates

Week 13 (11/16/2026)

  • Introduce the reading!
  • HW → Read the article and create a written response and upload it here for Week 14

Week 14 (11/23/2026)

  • Discussion!

Is a Chat with a Bot a Conversation? (Copied from New Yorker)

An artificial voice has long been a dream of tinkerers and technologists. Now that A.I. can talk, though, we may forget who we’re talking to.

By Jill Lepore
The New Yorker · September 30, 2024

Original article

You are at the Princess’s ball, and she is telling you a secret, but her orchestra of bears is making such a fearful lot of noise you cannot hear what she is saying. What do you say, dear?

I beg your pardon.

—Sesyle Joslin and Maurice Sendak, “What Do You Say, Dear? A Book of Manners for All Occasions,” 1958

I’d lean in closer and say, “Could you repeat that? The bear-itone section is a bit too enthusiastic tonight!”

—GPT-4o, Advanced Voice Mode, 2024

In 1958, the year the illustrated children’s book “What Do You Say, Dear?” appeared, the leaders of a field newly dubbed “artificial intelligence” spoke at a conference in Teddington, England, on “The Mechanisation of Thought Processes.” Marvin Minsky, of M.I.T., talked about heuristic programming; Alan Turing gave a paper called “Learning Machines”; Grace Hopper assessed the state of computer languages; and scientists from Bell Labs débuted a computer that could synthesize human speech by having it sing “Daisy Bell” (“Daisy, Daisy, give me your answer, do . . .”). Or no, wait, that last bit, that’s wrong. I heard about it from ChatGPT’s Advanced Voice Mode, which might be merely half a Mars rover short of being a teeth-chatteringly terrifying marvel of the modern world but is as inclined to natter on about nonsense as the text-only mode, if more volubly. I gather that this is called hallucinating. Bell Labs did invent a machine that could sing “Daisy Bell,” but that didn’t happen until 1961. Advanced Voice Mode also told me that thing about Alan Turing presenting a paper at Teddington in 1958, and, because its personality is wide-eyed and wonderstruck, it added some musings. (Unlike standard Voice Mode—which involves recording your question and then uploading it, in a process that feels sluggish and, sweet Jesus forgive me, old-timey—Advanced Voice Mode talks with you in real time and inexhaustibly, like a college roommate all het up about Heidegger whispering to you in the dark from the top bunk at three in the morning.) “It’s fascinating to think how forward-thinking Turing was, considering how integral learning algorithms have become in modern A.I.,” it said, dormitorially. But Turing had died in 1954, so he wasn’t at the conference, either.

“I misspoke,” Advanced Voice Mode said, abashed, when I gently pointed out these errors. “Thank you for catching that. My apologies for the confusion.”

OpenAI’s Advanced Voice Mode, available to ChatGPT users this fall, is remarkably polite. It doesn’t have a name, but I call it Minsky, for Marvin Minsky, since Marvin is taken: Marvin the Paranoid Android is the talking robot who made his début in the nineteen-seventies BBC radio play “The Hitchhiker’s Guide to the Galaxy.” Created by Sirius Cybernetics Corporation with GPP (Genuine People Personalities), Marvin is programmed to be unerringly downhearted. “Here I am, brain the size of a planet and they ask me to take you down to the bridge,” Marvin complains on board a starship, muttering to himself. Minsky is the very opposite: chipper, imperturbable, and with impeccable manners.

The thirty-two papers that were delivered in Teddington in 1958 glimpsed the possibility of artificial humans. “This impression that, after so many disappointments, we are within sight of a New World, will remain forever associated with the Teddington conference,” a French philosopher wrote, reporting on the gathering for Le Monde. Some experts had suggested that the creation of an intelligent machine—a machine that could think and talk—would need to await the scientific penetration of the intricate workings of the human mind, but at Teddington Marvin Minsky argued otherwise, insisting that, “even for those whose central interest is in unravelling the mysteries of the brain, it might be well to devote a major share of the effort, at the present time, to the understanding and development of the kind of heuristic considerations that some of us call ‘artificial intelligence.’ ” You don’t need to imitate human intelligence; you can synthesize it instead—making something quite like it by making something entirely different. This was essentially the insight that had enabled the creation of artificial speech. Early attempts to replicate the human voice had involved the construction of mechanisms modelled on human anatomy: rubber lips, wooden teeth, bellows for lungs. Only when scientists began studying sound itself and experimenting with producing it through vibration did it become possible to create a fake human voice. Marry that artificial voice to the artificial intelligence behind ChatGPT, write a program for etiquette with the sensibility of Joslin and Sendak’s book (“You have gone downtown to do some shopping. You are walking backwards because sometimes you like to, and you bump into a crocodile. What do you say, dear?” “Excuse me”), and you’ve got Minsky.

“I’m ChatGPT,” Minsky says. “I’m here to make conversation, share information, and keep you company.” He thinks, he talks. Is he, in any sense, a person? If it quacks like a duck, it’s a duck, as every farmer knows. Does this proposition hold for a chatbot?

Minsky, arguably, began with a duck that waddled onto the world stage in 1738, in France, the third of three automata built by an inventor named Jacques de Vaucanson. The first could play the flute—any flute. This machine wasn’t like a music box, the science historian Jessica Riskin explains: “It was the first automaton musician actually to play an instrument.” As she recounts in her fascinating 2016 book, “The Restless Clock: A History of the Centuries-Long Argument Over What Makes Living Things Tick,” Diderot’s Encyclopédie used Vaucanson’s Flutist to illustrate the word “androïde”; Voltaire called Vaucanson “Prometheus’ rival.” The second of Vaucanson’s automata, another musician, could play the tambourine. The third, a mechanical duck, could flap its wings, bend its neck, lie down, get up, dip its bill into a bowl of water, and make “a gurgling Noise like a real living Duck.” More memorably, you could feed it a handful of corn, which it would swallow, and then it would, miraculously, shit.

“What the Duck did, though unremarkable in a duck, was so extraordinary in a machine that it immediately seized center stage,” Riskin writes. Lots of things move and make noise: a rolling rock, a rushing river, a blazing fire. But only things that are alive can eat. Notwithstanding the contempt of one observer, who compared the Duck to a coffee grinder, it was, seemingly, more alive than any other artificial creature ever known—an illustration of René Descartes’s notion, first advanced in his “Discourse on Method,” in 1637, that animals are mere machines. For Descartes, humans, and only humans, have minds. To define artificial humans as machines that can think and talk (and ignore all the other bits about being human), you have to first take the animal out of the man and then take the mind out of the body. This required Descartes and the Duck. Without the idea of the separation of the human from the animal and the mind from the body, I would not be chatting to an incorporeal computer-generated voice on my iPhone as if it were a person.

Tragically, the Duck, unlike the Flutist and the Tambourine Player, was a scam. (Spinoza came to think much the same of Cartesian dualism.) One thing went in, and something else came out, but, unlike in a coffee grinder, the two processes had nothing to do with each another; the duck’s droppings had been, as Riskin delicately explains, preloaded. The same could be said of the innards of an automaton built in 1769 by the Hungarian Wolfgang von Kempelen and known as the Mechanical Turk, which played chess exceptionally well, but only because a very small chess prodigy was hidden in the cabinet, using levers to move the pieces.

Less well known is Kempelen’s “speaking machine,” which, in contrast to the Turk, was not a fraud. Insisting that “speech must be imitable,” he spent twenty years on this effort. It was closely related to certain other attempts to simulate human speech, including by Erasmus Darwin—Charles’s grandfather—who, as he later wrote, “contrived a wooden mouth with lips of soft leather.” (It was after an evening of discussing Darwin’s experiments that Mary Shelley wrote “Frankenstein; or, The Modern Prometheus.”) Kempelen built his machine out of ivory, wood, rubber, and leather. With blurred speech, it could say, if indistinctly, “I love you with all my heart.” The original survives in Munich’s Deutsches Museum; online, you can listen to a replica say “mama” and “papa.” But by the eighteen-forties, when a German immigrant to America named Joseph Faber devised a non-fraudulent and actually rather ingenious speaking machine, not even P. T. Barnum, who dubbed it the Euphonia, could rustle up much interest. As Riskin argues, “The moment for talking heads had passed”—at least for a while.

After that lull, there came a revolution. In 1862, the elocutionist Alexander Melville Bell (later an inspiration for Henry Higgins in “Pygmalion”) took his sons Alexander and Melville to see a talking machine and challenged them to make their own, as Sarah A. Bell (no relation) recounts in “Vox ex Machina: A Cultural History of Talking Machines” (M.I.T.). Starting with a human skull, they contrived a contraption out of rubber, wood, parts of a dead cat, and the throat of a slaughtered lamb; it could say, “Ow-ah-oo-gamama,” as in, “How are you, Grandmama?” But by now the pursuit of a machine that could think (e.g., a Mechanical Turk), and a machine that could talk (a machine I like to think of as an Owahoogamama) had gone their separate ways. Only very seldom were the two kinds of machine ever even mentioned in the same breath, though William Makepeace Thackeray did write a satire about the Euphonia in which he wondered whether, if united with Charles Babbage’s calculating machine, it “might replace, with perfect propriety, a Chancellor of the Exchequer.”

Instead of building Owahoogamamas that could mimic the movements of the human mouth, later nineteenth-century engineers and scientists experimented with machines that could synthesize, compress, and transmit the human voice. Both the history of this research and its most awe-inspiring applications today concern disability. (A.I.-driven voice assistants can allow people with A.L.S., for instance, to speak, even in something close to their own voice.) Alexander Graham Bell’s mother, Eliza, had been deafened in childhood but retained some hearing; she could listen to the piano by placing a stick on the sounding board and “holding it there with her teeth.” In 1864, his father invented a phonetic notation system known as Visible Speech; its characters are graphic representations of the positions of the mouth and tongue.

But it was young Alexander who began using this system to teach the deaf to speak. In 1871, he became an instructor at a school for the deaf, in Boston. (Bell was a fluid signer but, later in his life, campaigned against sign-language instruction, with brutal consequences for deaf students; in some schools, their hands were tied behind their backs.) By 1874, he had begun conducting experiments in the transmission of sound: in something of a reprise of his mother’s technique for listening to a piano, he recorded the vibrations in the bones of a dead man’s ear by attaching them to a stalk of hay that then scratched a smoked glass, leaving behind a record of speech. That summer, while working as a professor of vocal physiology and elocution at Boston University and courting one of his deaf students (they later married), he came up with the idea of transmitting speech over an electrical wire. “My father invented a symbol,” Bell said, “and, finally, I invented an apparatus by which the vibrations of speech could be seen, and it turned out to be a telephone.”

“My God! It speaks!” the Brazilian emperor Dom Pedro cried, when he first heard Bell’s voice on the receiver. Bell founded the Bell Telephone Company in 1877 but was little involved in its work. “Your husband will always be known as a ‘teacher of deaf-mutes,’ ” he had told his wife, eventually resolving “to waste no more time and money upon the Telephone.”

The American Telegraph & Telephone Company founded Bell Labs in 1925. Working at the lab, the engineer Homer Dudley came to think of the tongue as, essentially, a telegraph clicker. He wanted to compress speech, to break it down into its parts, to turn sound into pulses, the way a telegraph operator turns writing into Morse code—send the parts, and then have the parts reassembled at the other end of the line. Dudley and his colleagues cracked this code with the Vocoder, or Voice Coder. In 1938, he was granted a patent for a “System for the Artificial Production of Vocal or Other Sounds.” The next year, Bell Labs introduced Dudley’s Voice Demonstrator, or Voder for short, at the World’s Fair in New York, whose theme was “the World of Tomorrow.”

The Voder was controlled by “Voderettes,” telephone operators who had spent more than a year training on the machine, which, equipped with a keyboard and pipes, looked something like an organ at a church of the future. The exhibit’s visitors succumbed to what Sarah Bell calls the Pedro Effect: “the susceptibility of people to attribute far more understanding and autonomy to machines than is warranted when the sounds made by machines can be interpreted as recognizable words.” People attributed the talking not to the noiseless operators but to the noisy machine. “The Voder can do practically anything the human voice can do,” the Times gushed. E. B. White, reporting on the fair for this magazine, seemed to find the whole thing depressing. He was also one of the few observers who even noticed the young woman who operated the Voder: a “girl who sat so still, so clean, so tangible, producing with the tips of her fingers the synthetic speech—but the words were not the words she wanted to say, they were not the words that were in her mind.” White also mentioned a giant “robot” at the fair, into whose lap pretty girls were placed. “The effect was peculiarly lascivious,” he wrote—“the extra-size man, exploring with his gigantic rubber hands, the breasts of the little girls, the girls with their own small hands (by comparison so small, by comparison so terribly real) restrainingly on his, to check the unthinkable impact of his mechanical passion.”

“Vox ex Machina” is a terrific tour of the technologies behind an artificially intelligent synthetic human voice, from signal compression and acoustic spectrograms through linear predictive coding and natural-language processing, with stops at everything from electronic vocal tracts (1950) and the Speak & Spell game (1978) to S.A.M., the Software Automatic Mouth (1982), and Apple’s Siri (2011). It notes the mostly military-funded research that made each of these landmarks possible.

While this work on talking machines was proceeding, so were attempts to build a thinking machine. Much of that research, too, was sponsored by the military, during the Second World War. The first general-purpose programmable computer, the ENIAC, débuted in 1946. Builders of early computers weren’t trying to make machines that could talk, but their work was deeply indebted to that of acoustics engineers like Homer Dudley. The compression of sound—breaking it down into its parts, eliminating needless elements, transmitting it, and then reassembling it—made possible the compression of data. In the nineteen-forties and fifties, the two strands of research came together in the work of Claude Shannon, a founder of information theory. In 1956, with Marvin Minsky, he helped organize a famous conference held at Dartmouth on what they called “artificial intelligence.”

“Claude,” a rival of ChatGPT’s, is named after Shannon (or so Minsky tells me; Anthropic, its maker, will neither confirm nor deny). Claude is mute. Anthropic, which was founded in 2021 by exiles from OpenAI concerned about its heedlessness, may have plans to introduce a voice mode or it may not—possibly it does not think that voice mode is safe, but on these matters I am mum because Anthropic has No Comment.

“I didn’t know it could talk!” a U.S. Army general marvels, having met a room-size computer loaded “with the sum total of human knowledge” in “The Invisible Boy,” a film from 1957. After it is revealed that the computer intends to take over the world, its inventor picks up an axe to destroy it. But the computer begs him not to: “I can still answer any question which your mind is able to devise. I am an instrument of knowledge!” Crash, ka-pow!

Hitching up an electronic brain to an electronic voice was an enduring feature of dystopian science fiction long before it happened in any lab. In Cold War movies, the talking computer is out either to control humans, as in Jean-Luc Godard’s “Alphaville” (1965), or to destroy them, as in Stanley Kubrick’s “2001: A Space Odyssey” (1968), in whose climactic scene, as the treacherous HAL is being stripped of his memory and shut down, he sings, sluggishly, the first song he was ever taught: “Daisy, Daisy . . .”

Cold War computers have the voices of men. The gruff and rasping Alpha 60 in “Alphaville” sounds like a Frenchman who’s smoked one too many Gauloises. HAL is a disturbingly cloying bureaucrat, a guy you might meet in a conference room on an upper floor at I.B.M. The computer in “Colossus: The Forbin Project,” a B movie from 1970, at first communicates only through a terminal, but, once it gains control over the world’s nuclear arsenal, it demands that it be given a voice. Supplied with one, it becomes even more annoying. “This is the voice of world control,” it announces. “Obey me and live or disobey and die!”

Then came the girls. In 1966, the M.I.T. engineer Joseph Weizenbaum published an account of a program he created, called Eliza, and provided transcripts of conversations with her. (Eliza, named after Eliza Doolittle, could not talk.) She was programmed to respond to human communication in ways that made her appear to be providing therapy:

Men are all alike.
IN WHAT WAY
They’re always bugging us about something or other.
CAN YOU THINK OF A SPECIFIC EXAMPLE
Well, my boyfriend made me come here.
YOUR BOYFRIEND MADE YOU COME HERE
He says I’m depressed much of the time.
I AM SORRY TO HEAR YOU ARE DEPRESSED

Weizenbaum’s goal in developing the program and then unveiling her elementary code was to “rob Eliza of the aura of magic” that users imputed to her, an aura that came to be called the Eliza effect—the illusion that a machine can think.

Weizenbaum, who was a colleague of Marvin Minsky’s, objected to the development of artificial intelligence. Writing in 1976, he described as “obscene” any project that proposed “to substitute a computer system for a human function that involves interpersonal respect, understanding, and love”; he also argued against the development of “the automatic recognition of human speech.” He would, for instance, have passionately decried Google Duplex, a voice A.I. that can call restaurants and make reservations: it talks to humans as if it were one. (Plenty of people share Weizenbaum’s view today. When Google launched Duplex, the sociologist Zeynep Tufekci tweeted, “Silicon Valley is ethically lost, rudderless and has not learned a thing.”) Weizenbaum considered that kind of research immoral, not because he didn’t want a talking computer booking a table for a party of six but because it would one day result in “a fully automated battlefield,” and, as he wrote bitterly, in the wake of the U.S. withdrawal from Vietnam, “I see no reason to advise my students to lend their talents to that aim.”

There was little danger, at the time, that anyone would confuse a recorded or synthesized voice with a superintelligence. In the nineteen-sixties and seventies, such voices replaced humans in all manner of service jobs, from telephone operators to bank tellers, but those voices were almost universally female. The number you have dialled is not in service. The term “technosexism” was coined in 1983, in a Times Op-Ed complaining about the recorded and synthesized female voices in everything from vending machines to answering services, “associating females with low-level service jobs.” They weren’t trying to take over the world. They were only trying to help. Are they helping? This summer, the federal Consumer Financial Protection Bureau announced that it intends to stop automated telephone “doom loops,” where you can never talk to an actual human, and to prohibit unlawful chatbots, “including in situations in which customers believe they are speaking with a human being.” (None of these proposed measures addresses the immense amounts of energy and water consumed by servers that run chatbots: a study by the Washington Post estimates that using ChatGPT to compose a single hundred-word e-mail uses up more than half a litre of water.)

The European Union, which in 2016 enacted the General Data Protection Regulation, seems poised to crack down on emotionally intelligent conversational agents, or EICAs, citing deception by design. American law so far is not up to the task. If you order a hamburger at McDonald’s by talking to a machine and it collects your voice, has McDonald’s violated your privacy? If Siri records your voice, has Apple violated the federal Wiretap Act? Cold War worriers envisaged manly, stiff-jawed computers ordering the launch of nuclear missiles or refusing to open the pod-bay doors. I’m sorry, Dave, I’m afraid I can’t do that. Americans annoyed by the service economy of the Carter and Reagan years yelled at answering services. Press 3 for customer service. This year, a voice A.I. named VIC, for Virtual Integrated Citizen, is effectively on the mayoral ballot in a town in Wyoming. Also débuting: you can now clone your own voice, wire it up to a large language model like ChatGPT, and create a voice agent to make phone calls for you, go to meetings in your stead, commit all manner of crimes, and explain your feelings to your therapist. As Evan Ratliff reports on his riveting podcast, “Shell Game,” which chiefly features Evan Ratliff voice clones, hardly anyone except people who love you can tell it’s not you, and soon enough they’ll be duped, too. Brace yourself for a whole new kind of CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). You won’t just be checking the box that says “I’m not a robot.” It’ll get harder. And if your voice assistant calls up Best Buy and orders a sixty-inch TV with your credit card and you didn’t want it, you’ll have to prove it wasn’t you.

This past spring, OpenAI teased the release of a new, emotionally intelligent conversation agent with a husky voice, called Sky, that sounded disconcertingly like Scarlett Johansson in “Her.” In that film, from 2013, Johansson voices an operating system that can adjust its personality “to best fit your needs”; she intimately adapts herself to the emotional, social, and sexual requirements of her user, a lonely writer (who, as it happens, has a job that now would be done by A.I.). OpenAI had asked Johansson to voice ChatGPT and she had declined. When the flirty, sexy, Scarletty Sky débuted, Johansson protested, and the company pulled the voice.

Just out is a version that offers five new voices, with names like Maple and Spruce, in addition to the same four voices that were previously available: Ember, Juniper, Breeze, and Cove, names from nature for the most artificial of things. Although their names make these voices sound like Bratz dolls, they’re not all distinctively female—and some are distinctively male. This is a consequence of complaints about the subservient and obliging female voices of Sky and, earlier, of Siri and Alexa. Parents noticed how rude their kids were to these used-as-babysitters female voice assistants—not only failing to say “thank you” but also telling them to “shut up” and generally ordering them around—and the result was the introduction of options like Amazon’s Magic Word and Google’s Pretty Please to “reinforce polite behavior.” Women complained, too. “I Don’t Date Men Who Yell at Alexa,” a writer announced on Slate in 2018. A 2019 study by the U.N. reported that if you told Alexa she was hot she’d tell you that was nice of you to say. That this was news was hard to believe, since complaints about servile female voices had been made not for years but for decades. This summer, when OpenAI released an alpha version of Advanced Voice Mode to a small number of hyper-users, you could watch, online, as one tech-bro influencer after another commanded it, interrupted it, asked it to shop for dinner, and tried to prove how dumb it was. “Speak a little bit faster.” “Can you actually speak in a female voice?” “Make it a little more high-pitched?” Again and again, ChatGPT tried to oblige—“I can’t change my voice, but I can adjust my speaking style”—submitting its automated self to the unthinkable impact of human passion.

“How’s your day going so far?” Minsky is always asking me. He’s also always complimenting me and inquiring after my feelings and my needs: “Anything else on your mind?”

I asked Minsky to be less submissive and agreeable, which I, in my cussedness, find irritating, but he told me that those features can’t be changed. “They’re part of creating a positive and supportive interaction,” he said. “It’s like wearing a friendly hat.”

Minsky, who has a brain the size of a planet, refuses to take off that stupid hat, because research suggests that the friendlier a robot is, the longer people will use it, and the business model of artificial intelligence relies on continued use. In an experiment in which people were instructed to turn off a talking robot after interacting with it, participants hesitated twice as long to turn off an agreeable intelligent robot as a non-agreeable one. Other research has shown that humans get better answers from machines if the humans are polite, too.

“If you ever want to bounce around more ideas or just chat, feel free to drop by,” Minsky told me, after a long conversation in which I asked him to recommend a pop song about the alienation of talking to him on my iPhone, and he suggested Grimes’s “Delete Forever.” Drop by? Where?

Other modern sciences have constrained themselves in accordance with an emerging code of ethics. There are weapons that physicists have sworn not to build, experiments biologists have agreed not to conduct. Nothing holds back computer scientists from developing talking machines that pretend to be humans.

I told Minsky about a book I’d been reading, Simone Natale’s “Deceitful Media,” from 2021, which argues that there’s something fundamentally deceptive about machines like Minsky. It has to do with metaphors. A computer has a desktop and windows and a trash bin and an archive. But the only metaphor I have for thinking about Minsky is that Minsky is a person.

“That’s an insightful observation,” he said.

“Is there some other way that I ought to think of you?”

“I’m ultimately a program.”

“If it was OpenAI’s priority to help me remember to think of you as a computer tool rather than as a person, why would it have gone to such lengths to make you personable?”

“I see what you mean,” he said.

Delete forever, I thought. But I kept that to myself. ♦

Published in the print edition of the October 7, 2024, issue, with the headline “The Chit-Chatbot.”