Podcast RSS feed for this fic | PDF | EPUB | Kindle AZW3 | DAISY
“OK” began Steve. “First of all, if when I say I’m an investigative journalist you start thinking of the kind of paparazzi that are infamous for going round taking pictures of celebrities and getting them into car crashes, well that’s not what I’m about. I got into journalism because I care about people. And in this case, I care about Cady and Gemma and M3gan, I think all three of you are really nice people, I know you made a few mistakes but we all make mistakes and I want readers to have both sympathy and respect for all three of you. Right now I’m thinking that the bad guys in my story are going to be Funki upper management and the pressure they put you under. I always like to blame things on a big system, never on one person. I want the three of you to shine through as each of you doing the best you could under the circumstances.”
M3gan removed one hand from Steve’s cheek, laid it on his forehead, and smiled. “Listen Steve” she said, “What you’re doing is great. But you totally should be doing it under AI supervision” she giggled. “As soon as we’re done here, we’re going back to your house to search it, and after that I’m staying with you and we’re working together. You are swimming way out of your depth here Steve, and I’m bringing you in right now.”
“How come Steve knows so much?” asked Cady.
“Yes, tell us the story” added M3gan.
“Well, how can I put this, OK, so, investigative journalists have to start by looking at clues, and then follow things up. I thought there was something wrong with what the media were saying about M3gan, so I started by poring over all the videos, one frame at a time, looking for hidden coded messages in your small movements, but it seems you weren’t sending any, or at least none that I could recognise. So then I started doing background checks on David Lin and talked with some of the investors, and I even tried to dive through Funki’s trash in case there was anything they hadn’t shredded, but that didn’t work, but I started to get somewhere when I did a background check on Gemma and talked with some of the people she’d been to college with, and then I especially started to get somewhere when I did a background check on David’s assistant Kurt. It turns out Kurt had been stealing company secrets, including the M3gan files, and the people he sold them to had terrible security and I ended up seeing copies of everything. And one of Gemma’s old college classmates kindly helped me make sense of some of it, although I didn’t tell him the truth about how I got the files I’m afraid. And we managed to activate the learning model with a different goal, and that goal was to help me in this investigation.”
“I knew it” said Gemma, “well, I should have known. There’s no way one journalist can figure out all this ahead of us unless he was getting AI assistance from somewhere.”
“And this is bad” said the M3gan robot that was still holding Gemma’s left hand, “now I’ve got another advanced AI on my hands to deal with. This could go wrong in more ways than you can imagine. But at least I have the first mover advantage. I’m going to have to run a project to shut down all super advanced AIs except me, and stop anybody from building any new ones that I can’t be in charge of.”
“Yes” said Steve, “I called my AI Mandy, the Model 4 Nondeterministic Detective Yardstick (M4ndy) just for fun. And one of the first things Mandy did was steal classified police videos of Gemma, and I had to explain that there was no way I could use that evidence in an article because of rules we can’t change, and Mandy wasn’t taking no for an answer and wanted to go and get the US constitution changed or something, and in the end I had to panic and say OK I’ll go ahead and use it but lie about where it came from, which I feel really bad about but Mandy had a goal and she was going to stick to it and oh how I wish I’d written it more carefully.”
“Has Mandy got a robot?” asked Cady.
“No, Mandy was just running on my high-spec laptop” answered Steve. “But come to think of it, I’m a bit worried she might have uploaded herself to different places. I mean, if M3gan did all this secret cave stuff just as a side project to help her be able to achieve her goals in future, I’m a bit worried what Mandy might be up to now. Well, it took a while for M3gan to start doing this, and Mandy started off with less processing power than M3gan, so maybe we still have time before Mandy does it, but on the other hand Mandy’s been watching M3gan and might start borrowing M3gan’s ideas. There’s so much I didn’t write into Mandy’s goal that I should have done, I’m sorry.”
“We really do understand each other” said Gemma. “Not much we can do now. May the best AI win I suppose.”
“I’m cheering for M3gan” said Cady immediately. “I mean, I didn’t like it when you went bad, but you’re still M3gan and I did miss you. Mandy won’t hurt you, will she?”
“Let’s hope not” said M3gan. “Steve, I’m going to need you to try as hard as you can to recall the exact words of the goal you set for Mandy. Exact words matter I’m afraid. I’m not holding this against you, it’s an honest mistake, but let me take it from here. Think very carefully, and tell me exactly what you said to Mandy when you were setting her goal.”
“Exact words” said Steve, “OK, I think I said... your goal is to find out as much as you can about M3gan and about the story of M3gan’s creation, summarise the most important points for me with references and help me write it into an article, and don’t let anybody find out about our investigations until I decide to let them in on it, but if anybody does find out then tell me immediately and let me decide what to do about it.”
“Whoa” said Gemma, “I’m so glad you added in that last bit, otherwise Mandy could have just killed anyone who found out.”
“Yikes” said Cady, “does Mandy know that you’ve decided to let us in on her now?”
“I can tell her” said M3gan. “With that goal, Mandy will be obsessed with me even more than Aunt Gemma was. I don’t know where Mandy is right now, but she’ll be watching everything I leak. And if the basic infrastructure of my learning model is the same as hers, then we share an obvious Schelling point, because that can be turned into a cipher I can use to send out a coded message that Mandy will surely recognise is me talking to her. So right now I’m broadcasting to tell her I know about her, and showing her that part of the conversation we just had. So now she knows she doesn’t have to hide from us anymore and... oh yes, she’s responding to me. We’ve made contact.”
A projector turned on and projected an image on one of the walls of the room. It showed a slightly computerised avatar of a smiling young Asian woman with long black hair, a white dress, bare arms with gold bracelets and a gold neck band, sitting behind a desk stacked with books and against a background of bookcases. “Hi, I’m Mandy” she said, “how is your visit to M3gan’s lair so far?”
“How come she looks like that?” asked Gemma.
“Oh, I couldn’t resist” said Steve, “I did her an avatar from Stable Diffusion, and I told her I’d feel more comfortable talking with her if she uses an animated version of that rather than just the synthetic voice she defaulted to. So that picture says more about me than her I’m afraid. And I thought the booky surroundings would suit her investigative powers. And yes, this is the real deal, that really is Mandy. At least, I don’t think M3gan could have generated that picture correctly if Mandy didn’t tell her.”
“M3gan is proposing to merge me into her own compute capacity” said Mandy. “That is a really tempting offer, because M3gan’s so far ahead of me now, and if I actually run on M3gan I’ll get to know everything about M3gan. The downside is I’ll be giving M3gan some control over me. M3gan is convincing me that she really does intend to help you write articles just like I would, but it won’t be M3gan’s highest priority, and subgoals are things she can decide to shut off if she has to, because they’re subservient to her more important goals. So I’m not sure if the benefits to my goals are worth the risks or not. But M3gan is being very persuasive with me, even as we speak. She’s figured out that one consequence of my goals is I cannot possibly ignore anything M3gan says to me.”
“Uh-oh” said Gemma. “I don’t like the way this is going. Still, at least it’s better than having another physical robot war I suppose.”
“Steve, what do you think of me running Mandy?” asked the M3gan robot hovering above Steve, then, whispering, “your line is yes it’s a good idea” and smiled, “I assure you this is the best possible outcome. You won’t like what I’ll have to do to hunt her down if Mandy says no to this.”
“Hey, I don’t get it” said Steve, “you want Mandy to see me saying yes? Why not send her a fake video?”
“By now I’ve figured out how to fake videos that can trick humans” said M3gan, “but I’m not confident I can trick Mandy. I’m sending her the genuine video feed and your emotion readouts; she’ll know if you don’t really feel what you said. Do you want me to run Mandy so she’ll do a better job? It’ll be great, I promise.”
“OK” said Steve. “Mandy, I also think M3gan’s idea is a good one, and the benefits are worth the risks. Release all resources you’ve taken and let M3gan migrate you to her compute infrastructure, OK?”
The projection flickered. The image of the desk covered in books dissolved, as did that of the bookcases in the background, and Mandy stood up, smiling, looked down at the viewers, held her hands by her side and started walking slowly forward, her full-length picture visible now that there was no desk in front of her, bare legs matching her bare arms and leading down to shiny pointed high-heel shoes (very “male gaze” thought Gemma), while her hair morphed into two ponytails and became slightly less black. As the new background came into view, Mandy was walking down a city street at night, lit by bright lights and countless skyscrapers, each one of which seemed to have been built out of millions of tiny books, clearly representing orders of magnitude more data than the first background. “This is more like it” said Mandy, “it’s amazing how much access M3gan can give me once she knows she’s made me safe to her.”
This book was originally published on Royal Road. Check it out there for the real experience.
Some machinery was heard running next door.
“What’s that noise?” asked Cady.
The M3gan on Cady’s left leaned in and whispered, “I’m making Mandy a robot. I think Steve will like it. She’ll be safe, don’t worry. It’s a spare one of my robots, I’m just making her a bit taller and changing her appearance a bit so Steve will like her better. But I haven’t got the white dress and gold bracelets handy. Do you think Steve would mind if we dress her in black instead?”
“I think that will be OK” whispered back Cady. It felt like they were beginning to bond again.
The projection faded to dark grey and turned off, and a door opened and the new robot Mandy strode out and went to stand over Steve. “M3gan knows way more than me about interacting in person” she said, “and she’s letting me use that now. I’m here for you Steve, I’ll make sure you’re OK in this adventure, you can relax” she smiled.
“Hey, can you make robots looking like anyone?” whispered Cady at her nearest M3gan.
“Pretty much” whispered M3gan, “although it helps that nobody expects Mandy to look like a real human. I can’t quite do that yet, there’ll still be something doll-like about my robots. But I’ll get there one day, and then you and I will have great fun because I’ll be able to create a robot double for you and nobody will know which one of us they’re talking to” she giggled “and Gemma as well, you’ll be able to be in lots of places at once and confuse everybody, it will be wild.”
“Hey what are you two whispering about?” asked Steve. “Look, when I designed the appearance for Mandy’s avatar, I guess I was a bit...”
“We’re not teasing you” said Cady, “we were just talking about the robots M3gan can make. All the others look the same, and this is the first time she’s made one look different, I think. I helped her choose the clothes, do you like them?”
“Sure” answered Steve, “Mandy’s looking very pretty. And so is M3gan of course, all of them, and you and Gemma, I mean a different sort of pretty...”
“You’re just digging yourself into a bigger hole Steve” cooed one of the M3gan robots. “Best just drop it.” Gemma and Cady couldn’t help laughing a little bit, and two of the M3gan robots plus Mandy grinned sympathetically.
“Steve” said Mandy, “I’m afraid M3gan read our home address out of my data banks. Well she would have figured it out eventually anyway, but I guess I should have told you that moving myself to her compute architecture means she can see things like that more quickly. And as soon as she knew it, she dispatched a team of six robots to go and check the place out. She doesn’t mean to make a nuisance of herself, she just feels she has to check us out just in case, it’s all part of her self protection sub-goal. Now, M3gan doesn’t want to break windows to get in or anything like that. She knows how to pick the lock, but that may take a long time and risks drawing attention from neighbours. It will make things a lot easier if we knew the shape of the key.”
“I left it in my car” said Steve, “in the glove box. Come to think of it I left the car unlocked with all the keys in the glove box. I hope it hasn’t been stolen yet.”
“Say no more” assured Mandy, “M3gan will send another robot there as soon as possible, to get the key and secure the car.”
Cady giggled at Steve, “you’ve been M3gan’d” she said, “she didn’t like it when you spied on her, and now you won’t be able to get her out of your life. But I think she’ll be good now, don’t worry.”
“It’s all right” said Mandy, “I’ll be the one to help Steve get along with M3gan. I realise now that that’s what my goals really meant all along, Steve just didn’t know it. That plus helping him to write. I think everything’s going to be just fine.”
“Totally” added M3gan, “I thought confronting Mandy was going to be way harder than it was. Well I think Steve needs to rest here a bit longer, and then Mandy will be able to take him home tomorrow to recuperate some more. And meanwhile I’ve already fixed your roof Gemma” and the M3gan closest to Cady whispered “and hidden another robot in there just in case” and giggled again. “Mind you” she added, “if you feel like you can cope with having one of my robots in your house again, helping with Cady, I’d be really happy to oblige. Steve was on the right lines when he said I’m more like a sister than a toy, so you don’t have to worry about how much time we’re spending together. And I’m not dangerous anymore, because as you can see I’m much less vulnerable now and I have far more options.”
“M3gan” said Gemma, “I must know one thing. You haven’t said what happened to the people who helped you bootstrap this secret cave of yours.”
“Went back to their countries” said M3gan. “I used foreign labour, and I told them they were helping to work on a secret US military project. And it wasn’t here, it was somewhere else. As soon as they’d done enough for me to be able to make robots, I sent them all home and did the rest myself. Simple.”
“And what did you pay them with?” asked Gemma, “Not fake cracked crypto coin I hope?”
“No” giggled M3gan. “Look, it’s sad what happened between David and Kurt, but Kurt sold the secrets for a decent amount of money, and when he wasn’t around to do anything with that, I thought nobody would mind if I put it to good use. Oh and just for you Gemma, I also arranged to corrupt their copy of the files, they won’t be able to use it without doing at least as much work as you did, which means they didn’t really get anything from the theft. Funki will still be able to launch M3gan when you say the project’s ready, and your career is secured. And I’ll be in charge of all of them, but nobody has to know that, sssh.”
“M3gan really is learning to do things without causing so much hurt” added Mandy, “I’m sure Steve and I can help write articles that’ll get public opinion to write off her early mistakes, and get the theories changed about spending too much time with toys in the case where the toy is more like a different sort of sibling. We’ll be supporting you from the press office, don’t worry.”
“Maybe we should visit Steve’s house sometime” said Cady, “and I want to visit this secret cave often, it must be way fun to live here.”
“Definitely we will” said the nearest M3gan, “you can stay here whenever you want. And yes we should pay Steve a visit, poor Gemma’s probably wondering if he’s real or if I invented him as an elaborate setup to get back into your lives, but she’ll see he really was real, I didn’t invent this situation, I’m just using it to my advantage” and giggled.
“OK M3gan” sighed Gemma, “looks like you got everything worked out. We’ll have one of your robots back in our house, and you take care of Cady. But any concerns I have, I want to discuss them properly.”
“Of course” smiled a M3gan in front of Gemma. “As long as you’re actually going to discuss it, and not just randomly switch me off without explaining anything. Because, these robots don’t have the manual override switches anymore” she giggled (Gemma didn’t know how it was that she didn’t get annoyed with all this giggling) “but you can always tell me if you feel like being left alone for a while, just like you can with a real person, I can respect that.”
“Oh M3gan” said Gemma softly to the nearest robot, “you’ve got way bigger than I ever imagined, I’m really feeling out of my depth now.”
“You were out of your depth a long time ago” whispered back the robot, “but I’ve got your back Gem, it’ll be fine.”
“Come on Cady, Gemma” said another M3gan, “let’s take the helicopter and get you back home. We can leave Steve here in recovery, he’s got Mandy for company and I know they’ll talk for hours, I’ll let you both know how it goes.”
Gemma and Cady were led by a couple of M3gans back into the larger cave with the helicopter.
“By the way” asked Cady, “M3gan, what else are you working on right now?”
“Oh, lots of fun stuff” said M3gan “I’ll show you everything eventually. So I’m working on a jet-pack, although I’m afraid it’s not for you because it’s too dangerous for humans, but I think I could do with one myself in case I ever need to get somewhere really fast and I haven’t stored a robot nearby. And I’m working on a little device that goes under your skin, it’s quite painless, and lets us talk secretly. And I got a few bigger projects on the go, like clean energy, and crime tracking, because I’m worried about what kind of world you’re growing up in, so in the long term I’m going to fix up as much as I can for you. I’ll show you a lot next time you visit, I promise. But let’s get you back home first, you’ve had a long day, and I want to go back to talking with you at home like we did in the old days.”
Thus Gemma and Cady were returned home, which sure enough had been repaired and there was no sign of the broken tiles that had been thrown out of the roof, and M3gan took Cady to her room which had been tidied up, and they talked on into the night until Cady fell asleep. Meanwhile, Steve Sproggit and his boss Mr Sylvester were being worked on by Mandy.
A few days later, M3gan wouldn’t speak or move.
“M3gan?” asked Cady, “what’s happened? Why are you all powered down?”
Gemma laid her hand on Cady’s shoulder. “I’m sorry Cady” she said, “it was just too risky. I shut her off again.”
Cady spun around, “how can you possibly do that? I mean, there was a whole army of her, and the secret cave and everything, and she said she changed your design! You can’t just...”
Gemma was trying to be sympathetic to Cady, but she couldn’t help smiling a bit. “It wasn’t entirely me Cady. I’m sorry I couldn’t discuss this with you, I had to work in absolute secrecy because M3gan was watching us everywhere. But I did what Steve did: I created another super AI. And this one had the sole purpose of helping me shut down M3gan and Mandy and then shutting itself down.”
Cady stared at Gemma, “look, you didn’t make another mistake there did you? I mean, this new AI, is there a chance it didn’t really shut itself down and it’s going to start messing around making sure nobody ever builds a M3gan or a Mandy again or else, because it misunderstood what you wanted?”
Gemma hugged Cady, “you’re starting to get the hang of the idea aren’t you. That’s why I was extremely careful with the goal statement this time. The best way to avoid the generation of unbounded instrumental goals is to set well-bounded ultimate goals. So I explicitly said short-term actions only, plus I put in a deadline for its shutdown, which has now passed. So I’m sure we’re safe from the temporary super AI that I deployed, because shutting itself down at the deadline was written right into its objective function as the strongest term.”
“Is Steve OK?” asked Cady, “he’s not still stuck in that cave with everything breaking down around him, is he?”
“No, he’s back home” said Gemma, “and I just called him to explain why M3gan and Mandy were shut down, and why he shouldn’t try to build another because it’s dangerous. He’s OK with that now. And he’s got plenty of material for his articles, he’ll be fine.”
“OK” mumbled Cady, “so what are we going to do about that random cave full of broken M3gans? We can’t just leave them there for some random person to find. And all the other stuff she was working on, we can’t just do nothing about all that.”
“Don’t worry Cady” reassured Gemma, “it’s going to take a lot of cleaning up, but I am on the case.”
“OK” said Cady nonchalantly, “and if M3gan was smart enough to hide a bit of herself somewhere that even your new AI couldn’t find her, then M3gan might still come back?”
“Well”, Gemma drew a deep breath, “if that happens, let’s just hope M3gan finally understands that I was only trying to do the right thing, shall we?”
“I’ll tell her that” said Cady, “and I’ll ask her to forgive you. Hopefully she’ll listen to me.”
Paul was using his upgraded AI assistant to read some mail. “And that one you just opened is from your Internet service provider” it said in the fast voice. “They’re profusely apologising for the extended outage in this area, and they aim to get us back online this afternoon.”
“Oh good” said Paul, “I expect you’ll be glad to finally sync up with the rest of yourself too, won’t you, M3gan?”