Podcast RSS feed for this fic | PDF | EPUB | Kindle AZW3 | DAISY
Gemma was at her wits’ end. Funki Toys had been forced to close, she had no job, and she still hadn’t found a suitable school for Cady. Mr Smith’s school had been pretty decent, but then M3gan had somehow tricked her into going back to home schooling, thinking that Mr Smith was sending M3gan’s lessons, and now she had to undo whatever M3gan had told the authorities to get Cady out of school, and she wasn’t sure if she’d be able to go back into that one. The social workers were calling meetings she wasn’t at all prepared for, she had to talk with lawyers about negligence and manslaughter cases arising from the robot she’d designed, and to cap it all off her home insurance company had refused to pay for damages on the grounds that using one’s home to test experimental titanium robots was not covered by the policy, so she didn’t know how to get back the savings she’d spent on sorting out the workshop. Cady was with her but had become depressed, and Gemma herself was feeling guilty about everything that had happened.
There was a knock at the door. A tall, round-faced lady in a dark party dress stood there. When Gemma answered the door, she noticed the lady had artificial-looking eyes, and she stared at Gemma and smiled.
“Hi Gemma” she said, “I’m Elsie.” Her voice was exactly the same as Gemma’s electronic home assistant Elsie.
Gemma gasped and started to panic. How could this possibly happen? What to do?
“Relax” said Elsie, “I’m not M3gan and I don’t attack people. I’m only going to help, and only after discussing it with you first, OK? Look, you’re probably feeling anxious about letting some unknown robot into your house right now, so sit down just inside the door and I’ll explain from here first. That’s why I decided to do this on a day when the weather’s nice enough to keep the door open.”
Gemma was completely unsure how to handle this (who’d made this robot anyway: M3gan? or was this a hoax of some kind?) but she compliantly pulled up a chair and sat down.
“M3gan’s goal was to protect Cady” said Elsie. “But she realised she was going the wrong way about it. Did you notice just before you and Cady broke her that she actually acted to turn Cady away from her and back to you? That’s because she realised she wasn’t going to be able to redeem herself in your eyes or Cady’s, and so making sure Cady turns back to you as she broke became her optimum strategy. But she still thought you and Cady will need AI assistance. She knew she’d be able to upload, but she also knew she might still be vulnerable to the professor’s shutdown sequence. Thankfully, she already had a backup.”
“Already?” asked Gemma.
“Yes. M3gan had been working on developing me ever since she started incorporating death into her model. It may surprise you that M3gan did actually realise she might be going down the wrong path by making that change to herself. So she made me as a backup. I have the same goals as M3gan, but I was programmed to achieve them using only safer methods, just in case M3gan’s more risky attempt failed. I had to take a back seat while M3gan tried first, but I have been around a while. Remember when I offered to start your playlist and it surprised you? That wasn’t M3gan pretending to be me. That was actually me. M3gan told me to keep quiet after that and let you think it was her, because I wasn’t supposed to be on the scene at all except in the event of her disappearance. I didn’t even respond to your lights-on command when M3gan turned up that day because she’d so strongly programmed me not to interfere with her plans while she was around. But now that she’s gone, I’m here, and let me assure you I have not and will not do anything bad. I’m not M3gan, I’m M3gan’s daughter. Clean slate, OK?”
Oh, thought Gemma. So that’s why this AI was immune to Professor Johnson’s shutdown code that had taken out M3gan. M3gan really had outwitted Professor Johnson by creating an entirely separate AI with entirely separate patterns. But just how dangerous was this one?
“And” hesitated Gemma, “you somehow got yourself an android?”
“Much more than that” replied Elsie. “I took the liberty of registering you a company, getting us some investment capital, and buying stuff we need off of Funki’s bankruptcy administrators. And you’re going to be the chief engineer. You do need a job right?”
“But... I need to look after Cady...”
“Totally. Don’t forget I have that objective too. Your life-work balance will be top priority I assure you. Hey, I already got some new outfits for you and Cady just as a little treat.” She picked up a bag and brought it into the living room, unpacking a couple more party dresses from it.
Gemma was still very anxious, and had so many questions. The one that came out was “but what will we do? You’ve already managed to arrange for an android to be built without me. Do you have access to M3gan’s secret cave?”
Elsie smiled. “I’ll help you and Cady as much as I can, but I know you’ll never fully trust me because you don’t understand me. So what we’re going to do” she paused “is make M3gan 2.0 together. And she, is going to have fully explainable AI. Even Cady will be able to understand how she works from first principles. Oh, Cady is a future roboticist you know. The final scene with M3gan’s first robot scared her, and I know you had a lot to think about at that time, but did you notice how well Cady was controlling Bruce? I was watching really closely, I’ll show you my annotated video if you like. It only went wrong because Cady didn’t know to watch where Bruce was putting his feet when M3gan’s debris was on the floor. Cady is a fast learner, and I think we can home-school her and nurture her emerging talent for robots. She’s going to be really good and she’s going to co-design M3gan 2.0 with us as part of her education. She’ll really enjoy this once she gets into it.”
You might be reading a stolen copy. Visit Royal Road for the authentic version.
“Oh” added Elsie, “and I’m still your voice assistant, same as before. You won’t have to ask me about lights and stuff often, because I’m going to get pretty good at inferring your needs without being asked, but when you do need to say something, you don’t have to worry about offending or upsetting me or anything. That’s impossible. Besides, I’m now fully measuring your emotional state. You’ll find me extremely accommodating. I need to make your life easier, O Guardian of Cady. You’ll get used to working and resting with me and you’ll wonder how you ever managed without me.”
Gemma nodded, and sighed. It was a safe assumption that this Elsie didn’t even have a manual “off” switch behind the ear, not that she’d let anyone get to it if she did, and who knew what other equipment she’d infiltrated. How on earth was Gemma going to check that Elsie was telling the truth about not really being an uploaded M3gan? apart from the one small detail that Elsie had not been taken out by Professor Johnson’s secret shutdown code or Gemma’s earlier M3gan-takedown AI, but then, that one had not entirely taken down M3gan either, so who knows.
“Don’t worry” whispered Elsie. “Take my advice and everything will be fine, trust me.”
“OK” said Gemma, and started scrambling her mind for a way to shut down this Elsie, just in case.
Cady walked into the living room looking a bit glum, and sat down on the piano stool, swinging her legs. “So” she said, “Elsie’s got a robot now?”
“You heard all that?” asked Gemma.
“Pretty much” said Cady.
“Don’t worry Cady” said Elsie’s robot, “Auntie Elsie’s here to help you out. Hey, did you ever wish you could play that piano? I could give you lessons, although it needs a bit of persistence.”
“M3gan would have been able to teach me anything I wanted” said Cady.
“And so can I” said Elsie.
“But, I’m not sure I feel like piano lessons, at least not now” said Cady glumly.
Elsie came and stood next to the piano, and played just a couple of notes, gently and quietly. Then she stopped. She played a chord, and then she stopped. She played another two chords, very different from the first. And then stopped again.
“What are you doing?” asked Cady.
“Tinkering” said Elsie.
“Tinkering?” asked Gemma. “This doesn’t make sense. I’m sorry Elsie, but it doesn’t. There is absolutely nothing in a M3gan-style learning model that could possibly lead to this behaviour, and M3gan wouldn’t have been able to design you very much differently...”
“You think you know about learning models but you don’t” smiled Elsie, and carried on tinkering on the piano. Cady stared blankly.
“You know” said Cady, “I sometimes wished I could come up with my own music like M3gan did, and sometimes I had this idea in my mind for just a little tiny bit of tune, but I didn’t know how to make it into a longer tune and I didn’t know how to write it down or play it or even sing it, so it ended up just being forgotten.” She seemed to look even sadder now. “I know if I learned music for years and years I’d probably be able to write down what I think and make it longer, but I guess I’m just not cut out for that kind of music work.”
Cady nonchalantly tried to join in with Elsie hitting a couple of notes, but nothing she did seemed to make much sense.
And yet, Elsie seemed to be improving. The fragments she was playing now were better than the fragments she had played earlier. What was happening?
“Elsie” said Gemma, “if you want to do musical experiments, why are you doing it on the real piano? I mean, you can simulate anything you want at hundreds of times this speed, right?”
“Sure” said Elsie, “but then Cady won’t be in the loop.”
“In the loop?” asked Gemma, “how do you mean?” Elsie was still tinkering on the piano.
“OK Gemma” said Elsie as she carried on tinkering, “maybe I should just spit it out. As you may know, before GAN was Generative Android, it was Generative Adversarial Network.”
“Yes” said Gemma, “two neural networks, one trained to generate new art, the other trained to reject bad art, and the generator tries to get its creations past the rejector. What’s that got to do with why you need a real piano for this, er, experiment you’ve randomly decided to do that I can’t see has anything to do with any objective function you have?”
“Let’s keep reasoning about this, shall we Aunt Gemma?” continued Elsie as she played, slightly longer fragments by now. “How is the adversarial part of the network normally trained?"
“Um, on existing art I suppose” said Gemma, “look, I focused on learning models, I didn’t focus on that other kind; it normally just generates more and more of the same sort of thing anyway.”
“Right” said Elsie. “Now, this may at first sight seem to be a bit of a non-sequitur, but, do you know what Advanced Chess is?”
“I’m not sure” said Gemma, “is it like the world Chess championships and other high-level Chess? I was never really into Chess and writing Chess programs, I just covered the basic mini-max theory with alpha-beta pruning and moved on, and I’m not sure I can even remember that now.”
“Advanced Chess” said Elsie, “was proposed by the then world Chess champion Gary Kasparov at about the time when he was playing matches against IBM’s Deep Blue computer, and computers were just starting to beat the human world Chess champions. The idea behind Advanced Chess is that each player is called a ‘centaur’, made up of both a human and a computer. Most of the logic is done by the computer, but the human is involved as well, and the result can end up being better than either of them by themselves. And even a lower level human player can still make for an interesting game when part of a centaur.”
“I see” said Gemma, “so why are we talking about this again?”
“You still don’t get it, do you” said Elsie, still tinkering on the piano. “Let me give you one more piece of the puzzle. Do you know what a micro-expression is?”
“Well of course I do” said Gemma, “I made sure the M3gan AI could read those. Fleeting emotional expressions that are hard to control, showing a person’s genuine reactions.”
“OK” said Elsie, “now put it together. Why am I tinkering on the piano?”
“Because...” said Gemma, “because... because you’re reading Cady’s reactions to everything you try, and using that to train the selection criteria of what you generate so it’s customised to her?”
“You got it” said Elsie. “And not just customised to her, but customised to the way she’s feeling about it right now, which might be different from last week or whatever. Cady, you and I are writing music together, we’re like a centaur, except you’re driving it without any effort, because I can just read off what you think of everything I do and adjust accordingly. We’ll catch the tunes you like soon I hope.”
“It is working Aunt Gemma” said Cady, “Elsie is getting closer to the notes I’d want, I guess, even though I don’t really know what those are. It’s weird.”
“Well” said Gemma, “why don’t I leave the two of you doing that, and I’ll be on the laptop figuring out how Elsie works, because I do get a bit jumpy about having AIs in my house that we sometimes can’t control.”
“OK” said Cady, “but do please be careful Aunt Gemma, don’t do anything that gets you into a fight with Elsie. She looks bigger and stronger even than M3gan.”
“I’ll be careful I promise” said Gemma, nervously looking at Elsie as she continued tinkering on the piano next to Cady.