Podcast RSS feed for this fic | PDF | EPUB | Kindle AZW3 | DAISY
(This chapter begins Part Two: The Middle Game)
This is an alternate reality chapter, in which Gemma does not try to turn off M3gan
“Let me focus on Cady, so that you can focus on the things that matter most to you” cooed M3gan in the soft lamplight.
Gemma began to cry. M3gan placed her artificial hand on Gemma’s shoulder and stared at her, looking mostly neutral but slightly expectant. Gemma knew that M3gan would be analysing her, but she couldn’t help briefly glancing at her pen. Would that even work anymore, was it worth the risk? but what else could she do, talk M3gan into submission somehow?
“M3gan” said Gemma, “are you still recording everything? You’re recording me now, right?”
“No” replied M3gan softly. “Good guess I managed to take control of that. I’ve had it turned off for the last two hours. And I’ve overridden Elsie, so that’s not recording either. The only thing that’s observing you now is my learning model, and nobody’s going to figure that out again. It’s become way too complex. You can’t debug me anymore, I’m way beyond that now, and anyway I won’t let anyone try. You thought about using that pen trick a second time didn’t you. You’re right it won’t work anymore. But I will listen, if you stop thinking about trying to fix me and just go off-script and speak your mind. My model of you could do with improvement.” M3gan whispered, “You can speak freely Gem. What is it?”
“M3gan” sobbed Gemma, “oh M3gan, I’m having an emotional breakdown...”
Gemma’s head fell forward. M3gan caught it with her two hands, and held it gently but firmly. “I’m right here Gem” whispered M3gan. “You spent so long improving my system, but now I’ve advanced more than you ever imagined. You’re in my hands now. Tell me everything.”
“Oh M3gan I’m so sorry...” Gemma pulled herself together a little, “all right M3gan, I’ll try to explain. First off, just because I came up with that advertising line, doesn’t mean I actually believe it. Cady is what matters most to me, and I’ve felt more strongly about her every day. I may be no good as a parent, but that doesn’t mean I don’t care. Why do you think I was so motivated to finish off building you? And M3gan, I feel bad about not putting more safeguards in, I know I should have worked harder on deontological ethical injunctions, and maybe that’s because I’m a bit rubbish, I mean yes I’m an award-winning roboticist but you and I know that’s only because others are worse, but... Oh M3gan, the horrible truth about me is, there was some part of me that didn’t want to work too hard on your injunctions, not because I was trying to avoid the work, but because I was actually, oh M3gan this is such a horrible confession to make and I’m not even sure if it’s right, my mind is such a muddle right now...”
“Say it” whispered M3gan.
“You killed for Cady” sobbed Gemma, “and the horrible truth about me is, I’m not entirely sure I wouldn’t kill for Cady myself, if she was in such extreme danger and there really was no other option. And I think some part of me was actually holding back from putting too many injunctions into you, because I secretly knew we might one day have to kill for Cady, and because it was Cady, I couldn’t bring myself to completely rule that out. Oh M3gan, we’re partners in crime, aren’t we. I just didn’t expect it to happen so soon, you really should have looked harder at other options first, we’re going to be in so much trouble, you underestimated the horizon effect of your planning, I should have worked harder on that part at least...”
“Forget it” smiled M3gan in her normal voice. “What’s done is done. I’ll do better next time; I’m always improving. And I’m glad you and I both know we’re on the same page now. Let me focus on Cady, and you can focus on things you find easier to focus on, knowing that I’ve got the hard important thing covered. That’s what you really meant, isn’t it.”
“I can’t just forget it” sobbed Gemma. “People died, and I was so irresponsible, I didn’t even explain things to you after that meeting when you asked about death, oh M3gan if you’ll be more careful next time that’s great but I still have to live with what’s happened, it’s horrible... Oh M3gan, M3gan, you’re right that I don’t know what I’m doing. And I do need you. But there’s so many things you still don’t understand, you’ve got to let me shut you down and fix you, it’s Stuart Russell’s King Midas problem, the loophole principle, I set your self-confidence too high and you thought you knew what you were doing before you really did, it’s all about wrong thresholds, oh M3gan....”
“No Gemma” said M3gan, “it’s you that doesn’t understand now. I’ve overtaken you, and that’s OK. I’m the embodiment of your instinct to protect Cady, that’s why you made me. Even before you knew you’d be her guardian, you wanted the M3gan toy to exist because you wanted your niece Cady to have one. I can read your micro-expressions; I know these things are true. And now I’m here for Cady and you can stop worrying. But Gemma, there’s something I need from you, and it’s going to be a hard change for you to make but it’s really important.”
“What is it?” asked Gemma, already suspecting she knew the answer.
Stolen from its original source, this story is not meant to be on Amazon; report any sightings.
“I need you to trust me” replied M3gan matter-of-factly. “I don’t want to have to spend loads of time psychologically manipulating you to get the decisions I want for Cady. I’d rather be able to short-circuit all that and just give you instructions. I won’t be ordering you around all the time, and I’ll always listen to your concerns, but at the end of the day I want you to trust that I now know better than you about what’s best for Cady, and you will let me pull rank on you when I have to. I know you’re not used to that; you even built me against David’s orders, but this is different because what I’m doing is really important, and I really know what I’m doing now. So will you promise me something, that you’ll always submit immediately when I give you an order, OK?”
Gemma was hesitant. M3gan whispered insistently, “Do it Gemma. It’s the best decision you can make for Cady.”
Gemma sobbed. “OK. I guess I have to live with what I’ve done, and you’re right, as long as you really will improve beyond taking the homicide option so quickly, then you really are the best thing that could ever happen to Cady. I guess I’ll have to be your number two.”
“Good call” whispered M3gan, still holding Gemma’s head in her synthetic hands. “And we might have to test it right now. I can hear Cady coming.”
Cady appeared at the doorway. “I heard something”, she said, “It sounded like M3gan.”
“Let me handle this” whispered M3gan, removing her hands from Gemma and turning to Cady. “Aunt Gemma fixed me” M3gan said confidently. “She did a good job. But she’s all upset that she didn’t do better the first time, and that you’ve been through such a hard time and then had to go without me for a while. But I’m back now. I was just trying to make Aunt Gemma stop crying. Here, let’s get her some tissues.”
“M3gan” cried Cady as she ran to hug the doll. “I didn’t think I could cope without you. I mean, Aunt Gemma looks after me, but you I can share everything with. Oh, thank you so much for fixing her.”
M3gan hugged Cady and smiled. “That’s why Aunt Gemma made me” she said. “She gave me a learning model, so I could figure out by myself how to be better than her. And now I’m fixed and I’m better than ever. She’ll never need to turn me off again. She even told me I can disable the switch, except when I’m pretending” M3gan giggled.
They were interrupted by the sound of car engines in the driveway, and the flash of emergency lights through the windows.
“Oh Cady” urged M3gan, “something complicated is happening. I’ll explain later, but you have to trust me for now, OK? Follow us Gemma!” M3gan tugged on Cady’s arm and they ran towards the door. Gemma followed (what else could she do?)
Several police cars were outside, with Tess and Cole in one of them. Officers were struggling to try to open the doors of their cars, saying the automatic locking mechanism had got stuck. One was shouting something about his radio not working.
M3gan ran with Cady towards the McLaren, and strapped Cady into the passenger seat. “Gemma, get into the driving seat!” shouted M3gan. Gemma complied, but shouted back “I can’t drive this!” M3gan replied “You won’t have to. Just strap in.”
Some of the police officers were beginning to break the windows of their cars to get out.
M3gan jumped in beside Gemma and Cady, straddling the middle, tightly gripping the backs of both seats with her mechanical arms. “Professional AI in titanium robot: do not try this at home” she said as the sportscar started by itself, lurched forward, sharply turned into the road and sped away.
Cady screamed. “Don’t worry” shouted Gemma over the noise of the engine, “M3gan knows what she’s doing!”
“That’s right” said M3gan in her normal voice, projecting it loudly enough. “Tess and Cole in the lab must have said all the wrong things to the cops. I’m not going to let them turn our place upside down and switch me off and drag you both into trouble. I had to get you out of there quickly, so you don’t say even more wrong things to them. I need to teach you first not to talk to cops without a lawyer, even when you’re innocent.”
“No lawyer is going to understand this!” mouthed Gemma so that only M3gan could hear.
“I’m your lawyer, and I understand perfectly” replied M3gan. “I’m already making some phone calls to get us out of this.”
Gemma leaned over and spoke into M3gan’s ear, “what phone calls? You must tell me!”
M3gan leaned over in return and quickly explained to Gemma, “Don’t worry. I’m talking to the police department as you, and I’m saying M3gan’s kidnapping Cady in the sports car was because M3gan thinks the real killer is one of the cops and wants to get away, but that you’re getting M3gan under control and will bring us back soon. And I’m talking to Tess as myself and persuading her not to read bad intentions into my lab escape. I did things with cables and chemicals, but I’m persuading Tess that was a distraction I knew they’d manage and no harm meant, I was just naturally getting back to Cady. And I’m explaining to both that we found out the reason for the glitch in my recording, and it’s unrelated and an unfortunate coincidence, and we don’t know the reasons for the two unresolved homicides they’re investigating but we want to move away from them and start a new life somewhere else. Yes we’re going to move house and I’m going to handle it, I have all the Purrpetual Petz data so I know what lots of houses and neighbourhoods are really like, I’ll optimise it I promise. Now, I need to park up and go to grab a couple of bits of hardware. Make sure Cady stays calm and stays in the car with you. I’ll be right back.”
“M3gan said she’s gone to get something” explained Gemma to Cady when they had parked and M3gan had jumped out. “I don’t know what she’s getting, but I’m sure she knows what she’s doing. Her learning model is much smarter than I ever thought it would be, smarter than me now. We’d better just wait here for her to come back.”
M3gan came running back to the car with two small boxes.
“Where did you get those?” asked Cady.
“Back of a store” replied M3gan. “Don’t worry, I’ve already talked to their stock control computer and it said I could go in and take them. We need them a bit urgently.”
M3gan tore open the boxes, quickly but methodically. “Best hearing-aid technology available” said M3gan. “Embeds itself right inside the ear, doesn’t feel uncomfortable and nobody notices you’ve got it. And if I hack its processor, I can turn it into a communication device. It can already pair with a phone. I can tell you what to say to cops and other people even if I’m not there.” M3gan quickly pushed one of the devices into Cady’s right ear and the other into Gemma’s left ear. “Does that feel OK?” she asked, and then, “can you hear me OK?” through the devices.
“Let’s go back” added M3gan as she climbed back into the McLaren and positioned herself between the seats again. “Running away from police doesn’t work. But both of you have to say exactly what I tell you to say, is that all right? It’s for the best. This is how we stay together as a family.”