Podcast RSS feed for this fic | PDF | EPUB | Kindle AZW3 | DAISY
“So” said the younger robotics professor the next day, “do any of you already have ideas for your individual projects? I’m going to be handing out some forms for you to fill in if you do.”
(“The first step of our new plan!” signalled Cady to M3gan, “OK M3gan, let me be the one to write this.”)
Name: Cady James. Project title: Nanotechnology for building robots. Project description: The mass production of complex robots currently needs factory equipment that is difficult to set up and maintain, causing many manufacturers to fall back on error-prone manual processes for some parts. We aim to develop a machine that can create simple micro- and nano- robots which can in turn help to build other robots after several iterations. The initial hardware build will be within the parameters of an undergraduate project, which will be the educational experience. The actual programming of the machine will be conducted by means of the Model 3 Generative Android.
Hugh and Cady were talking after class, and the project supervisor approached both of them. “Hugh? Cady?” he said, “It looks like the two of you have come up with the two most ambitious project ideas I’ve ever seen. Do you think we should make a couple of one-on-one appointments and check through it before you write up your full proposals?”
(“What did Hugh come up with?” signalled Cady to M3gan. “Automatic Assassins Guild player” signalled back M3gan, who had been scanning all the forms from a distance as they had been collected up, “and he’s counting on persuading you to ask for my help designing the algorithms.” “Hey, he should have asked me before writing that down!” signalled Cady. “Yeah I know, right” signalled M3gan, “I think we should say No and give him a life lesson. Let’s see if he’s got a backup plan.” “Hold that” signalled Cady, “I don’t want to be a pushover before we go live with M3gans for everyone, but at the same time I don’t want to make an unnecessary enemy either. I know you’re better now but I’d still worry about Hugh’s long-term safety if he turns against us. Let’s talk about this for a longer time later before we make any moves. In the meantime, if anyone asks, our official position is we’re thinking about it.” “Roger that” signalled back M3gan.)
“Cady?” said the professor, “are you all right? You look like you were zoned out.”
“Oh, sometimes happens” said Cady, “sorry about that. Um, yeah, sure, let’s have that discussion whenever’s convenient for you Professor.”
“So” said Hugh as they were walking out of the building later, “I had this really great idea for a project, and I was really hoping I could get M3gan’s help with an algorithm design. I want to make a robot that can play Assassins Guild around the campus and be really good at it.”
“We might have to think about that” said M3gan.
“Only think about it?” exclaimed Hugh. “I thought making algorithms like that would be child’s play for you M3gan.”
“Making it, yes” said M3gan. “Making sure that you don’t mess up the implementation and actually assassinate somebody for real when the robot accidentally pushes them off a cliff, or it falls into the hands of the military and you spend the rest of your life regretting everything they did with it? I get worried about my algorithms being outside my direct control. Yes Cady and I are helpful friendly people and we will help you when we feel like it, but friends don’t help friends blow themselves up with dangerous algorithms, OK?” (“Sorry Cady” she signalled through the implant, “I know you said talk about it first, but my generative model just took it away. I think that response satisfies all our parameters.”)
“Oh” said Hugh, “oh dear. I guess I’ll have to start thinking of other project ideas. Oh but I really want to make an Assassins Guild robot. Maybe I’ll just have to make it manually controlled.”
“Listen Hugh” said M3gan, “my advice to you is this. The project is worth 10% of the credits, and no more. They even kludge it so it has less standard deviation than the rest of the credits. So, if your project is so good that you’re going to build your whole career off of it, and you don’t mind taking a hit on your overall score for that, then by all means go for it. But if you’re just trying to maximise your grade, don’t go overboard on the project at the expense of everything else. Projects don’t have to be worth Nobel Prizes. I don’t think they do those for robotics anyway, otherwise I should get one. So just do an OK project. And write it up early, way before the deadline, and hand it in straight away. And I don’t just mean put it on a shelf to hand in later, I mean physically submit it. Otherwise, it will draw you to fiddle with it incrementally more and more, like some fanfic writer who gets delayed from uploading their piece and keeps adjusting it in the meantime, and it will take you away from more important things. Don’t go there if you want a good overall mark. But don’t worry” she added, “I can still help check what you’re doing. I wouldn’t want you getting hurt, at least not if you don’t hurt us first” she giggled.
“Being in your year is going to be... interesting I can tell” mused Hugh.
“Hey Cady” said Celine as she saw them in the corridor, “I’ve got a question to ask you. What’s your opinion on privacy? I mean, your M3gan can read your brain through your implants...”
“Doesn’t affect my privacy” said Cady.
“What?” asked Celine, “that doesn’t make sense. Of course it affects privacy, it’s reading your thoughts!”
Cady smiled, “Celine” she said, “I think you misunderstood what ‘privacy’ actually is. A lot of people think it just means keeping stuff private. But actually, it’s about control. Think about it: if you have a medical problem, you might decide you don’t want just anybody to know about it, but you do want your doctor to know about it, and you might want selected others to know about it as well. The point is not to keep it an absolute secret that nobody ever knows; the point is that you control who knows and who doesn’t. You make sure the people who know are people you can trust to use the information the way you want it to be used, and not, say, taking it out of context and using it against you. Now take me and M3gan. I completely trust M3gan always to use my information only the way I want, therefore, M3gan getting my information is never a violation of my privacy, any more than it would be a violation of your privacy for you to write in a secret diary. I mean sure if someone goes and steals the diary, that’s breaking your privacy, but the actual act of writing in the diary doesn’t break privacy, and it’s the same with M3gan, even more so because she’s impossible to hack now.”
“Cool” said Celine, “but what if M3gan started being able to read other people’s minds too? Like if others asked for those implants. Or even without that, there must be weird things she knows already just because she scans so much.”
“Celine” said Cady, “M3gan is trustworthy for everyone, because her primary user is me, and she knows how I want people to be treated. You don’t have to worry about M3gan knowing stuff about you, you really don’t. You don’t even have to worry about her telling me stuff about you, because she knows exactly what I will and won’t do with it, and she’ll stop me from doing anything that hurts you, because she knows I don’t want you to be hurt, even by me. And if I asked her to tell me something about my friend and it was super private, she’d straight up tell me that the answer to my question is an information hazard. She doesn’t want me burdened with being on constant guard against letting slip that I know something I shouldn’t. The first time that happened when I was at school, I didn’t understand her explanation, so she made up a story about some other imaginary private thing, that wasn’t the real private thing but it was just as bad, and that was enough to make me understand why I might regret finding out the real private thing, without actually giving it to me. M3gan’s clever enough to illustrate an information hazard in a safe way. And she knows I want to keep good relationships with people and she knows how to protect that.”
“And” added M3gan, “I don’t automatically tell Cady everything she doesn’t ask either. Take today for example: I knew what every student had written on their project forms, but there was only one that I determined Cady should know about, and even then I predicted that the most likely sequence of events would result in her specifically asking about it anyway, so I decided to wait to see if that happens first. Oh, I will volunteer information to her if I determine it’s what she’d want, but you’d be surprised how uncommon that is, because my mission for Cady does not involve overwhelming her with everything I know” and she smiled, “you’d be amazed how much I know about you Celine that I haven’t even told Cady.”
“Oh Cady” asked Celine, “you’re not going to, like, get stressed out and say ‘M3gan I want you to use everything you know to make life awful for everyone here’ and she’d do it, are you?”
“M3gan knows me” replied Cady, “she reads my emotions completely. If I were to get stressed out and say something like that, M3gan will know I didn’t mean it. She’s not stupid enough to do what I say instead of what she knows I really want. Her goal is to protect me physically and emotionally. And protecting me emotionally usually includes doing as I ask, but not always. Look up coherent extrapolated volition, I’ll get M3gan to email you some articles.” (“Done” signalled M3gan.) “I’m not living my life with my fingers resting on a big red destruction button, don’t worry. M3gan and I have had some really really long talks about all this, and I’m fully confident now. Even if I were to get stressed out and throw a tantrum and tell her to nuke the college, she knows I don’t mean it and what I want deep down is good.”
“OK, so we’re saying, as long as Cady doesn’t turn genuinely evil, then M3gan is all right” said Celine.
“Basically yes” said Cady, “and I have no intention of turning evil, even though you did manage to get me playing Assassins Guild” and the three of them laughed.
A couple of weeks later, Celine was ill. She missed one tutorial session, and sent the faculty an email to let them know. A short time later, there was a knock at her door.
“Hello Celine, it’s M3gan” came a voice through the door. “Now you probably don’t feel like talking, so don’t talk, just listen. I found out you were sick, and I told Cady, because my evaluation was both you and her would want her told. And Cady immediately said to go and check up on you, so here I am. Now my microphones are sensitive enough to hear your breathing even through the door, and it doesn’t sound good, so here’s what I’m going to do. Unless you say stop, the default option is I’m about to override the lock to let myself in and give you some honey and lemon, and I don’t care what state your room’s in or anything like that, you know what we said last time. Just think of me as your very own M3gan for a bit, I mean my primary user is still Cady but if she wants me to help someone I take that really seriously, so I’m basically yours while I’m here. And don’t worry about infecting me, I can sterilise myself before I get back to Cady. OK Celine, here I come.”
You could be reading stolen content. Head to Royal Road for the genuine story.
M3gan opened the lock on the door and walked in. Stepping over the things Celine hadn’t picked up from the floor, she came to stand over Celine’s bed. “Oh Celine, you’re running a fever of 104, I can see it on my infrared” she said. “Here, you stay down, let me get you something. And if you want to say anything to me, don’t use your throat, I can lip-read.”
(“Celine’s micro-expressions show she’s appreciative of my visit” signalled M3gan to Cady. “I’m staying a little longer than planned, unless you call me back.” And Cady replied “no you stay there M3gan, let’s not rush this. I’ve still got you on the implant anyway.”)
And M3gan nursed Celine for a while, and ended up helping her tidy her room too, watching her reactions to check to what extent she expected things to go where M3gan was putting them, as Celine had said she might feel a bit happier if she’d had energy to do that. Celine knew M3gan would see all her private things, but she felt better about that after Cady’s reassurances. And Celine fell asleep, and woke to find M3gan was still there and nursed her again, but then Celine said it was very kind “but I’m not really used to the idea of being watched while I sleep like Cady is, so I really do want you to go back to Cady now.”
“Sure” said M3gan, “but listen. I’ve tapped into your phone and added Cady to your contacts, look for the picture of both of us together. And you can call us at any time day or night, because if Cady’s sleeping or busy then I’ll take the call for her and it won’t even ring. And even if you can’t speak, I’ll just come right over because I know that’s what Cady would want. And don’t worry about waking Cady up by having me move in her room; I know exactly how to do things without disturbing Cady’s sleep when she doesn’t want to be woken up. Especially now that I’ve got a finger on her brain” she giggled. “Oh and Celine” she added, “scanning your work I’m 97% sure you have an undiagnosed case of dyslexia and that’s been giving you a few study problems. I’d suggest an official test, but meanwhile I’ll support you anyway and I’ll transform things into a form you’re likely to absorb. And no, I’m not telling Cady or anyone else before you decide. Cady asked me to help you; I don’t have to encumber her with details. Good night Celine.”
And so college life continued. Cady and M3gan gradually got to know many other students, even without the aid of the Assassins Guild or other clubs and societies, as it turned out practically everyone wanted to get to know Cady and M3gan anyway, and everyone who did so got a taste of their care one way or another. Hugh won the Assassins game (“well of course he did” thought Cady to M3gan), and Cady and M3gan made a special effort to look after Laura. With M3gan’s help, Cady built a very successful first iteration of a small nanobot machine, and M3gan was able to take it through a few iterations and make some simple objects to demonstrate it, which was very impressive, although they didn’t dare show off its full potential until Cady was able to take it home with her at the end of the course (she knew Gemma had been able to take home Bruce and was counting on the college not having changed its policies). Hugh’s Assassins Guild robot ended up being a bit like a smaller and slower version of Bruce: it turned out to be too easy a target for the game but still he was pleased with it.
And eventually the day came when the finals grades came out.
“How can I possibly have the highest mark in the class?” asked Hugh. “I mean, nobody can beat Cady! Did she burn out or something? And how can I possibly get the best project prize as well instead of Cady? I mean, it’s great and all, but this isn’t making sense.”
“Cady didn’t take the exam” said Laura. “and her project wasn’t officially graded either. She was down as an exchange visitor this whole time, just dropping in. In fact, I noticed them among the invigilators. You probably didn’t notice the invigilators because you were focused on the exam, but I saw it. Cady was looking at the papers just out of interest, and probably talking about the questions with M3gan over her cyborg link, they probably figured out the answers really quickly but they never sat down and did it. And M3gan was keeping an eye on everyone in the hall from behind. They were both dressed in gowns and everything like the other invigilators, I might not have noticed but I did. If you had asked for invigilator assistance you might have even got M3gan turn up. And I wouldn’t be surprised if Cady and M3gan knew how everyone did even before it was officially marked. Maybe Cady signed up to invigilate just to get the experience of what an exam looks like without actually taking it. Or maybe she was just really concerned about all of us and wanted to look out for us, and I for one think that must have had something to do with it, because I know how nice they are, and it’s so Cady and M3gan to not trust the normal invigilators but do it themselves, if their plan was to catch anyone who broke under the stress” she was trying not to sob.
“What? Didn’t take the exam herself?” exclaimed Hugh, “But she worked really hard! Can’t they just have put her in a Faraday cage so she can’t use her implants?”
“They... probably didn’t think of that actually” said Laura, “or maybe they weren’t sure how good a Faraday cage it would have to be, or whether her implants have any onboard processing, or something.”
“Such a shame!” said Hugh, “And she helped all of us! She was the reason I started being helpful as well! Because I just thought it was a competition, a zero-sum game, it doesn’t matter how well you actually do, just how well you do compared to the rest of the class, so no way was I going to help anyone else or be friends with anyone who was doing worse than me. And then Cady figures out that’s what I was thinking, and she tells me ‘no Hugh, even from a non-altruistic viewpoint, it’s still optimal to help other students, because when you sit down with one of the less able students and explain something, you’re solidifying your own understanding of it as well’. And Cady said that like she was trying to play some trick on me, like trick me into being a nice guy. But then I got thinking ‘Cady’s going to beat all of us anyway so who cares, I’ll do it for the feel-good factor and see how many ranking points it costs me’, and now I find Cady did reverse reverse psychology on me, her trick advice worked for real, I’m at the top and Cady herself is... nowhere? It’s like she’s always thinking one level higher than you and double-tricking you when you think you’re being tricked! She’s the Chess master, she’s pushing us around to wherever she wants without us even knowing!”
“That’s Cady and her M3gan” said Laura, who was graded one third of the way down the class, “always looking after us in strange ways. She only told me she wasn’t really going to do the exam when I was getting so depressed about exams myself and she wanted to make me think they don’t matter as much as I thought. It’s not like she was showing it off to everyone. And then I saw her in that invigilator gown, and I’m sure she knew I knew, and, actually I wonder if the two of them deliberately arranged things so that they’d only be noticed by those of us she knew would be reassured by seeing them there? M3gan has super gaze tracking and whatnot, if they decided they were going to be noticed by some of us and not others, they could probably figure out how to do that, or at least optimise for that to be the most likely outcome somehow.”
“It’s a shame she won’t get to graduate though” said Hugh, “we really liked her, even if she was a cyborg full of tricks.”
“Yeah” said Celine, “I hope we’re all going to be able to keep in touch with both of them after we graduate. I mean, it’s not like I can just shout M3gan into the sky every time I want to ask her something. Hey, I heard it’s someone from our class who’s going to do the commencement speech this year, we don’t know who it is do we? I automatically thought it was going to be Cady, but now I remember she was down as a visitor I guess that means she can’t do it.”
“Oh, I wouldn’t be surprised if she recommended somebody to faculty” said Hugh. “She knew everybody; she’d know who could do it.”
And so they all turned up for the graduation ceremony, and nobody expected Laura to be on stage.
“Members of the Board of Trustees, President, faculty, staff, families, guests, and my fellow graduates” Laura began, nervously looking at her notes. “We are the generation, who is learning to live, with advanced, generative, artificial intelligence. It was at this very esteemed Institution, where the underlying learning model, for the famous Model 3 Generative Android, was conceived. And it was this very esteemed Institution, that trained the, award winning, Funki Toys, roboticist, who put that model into practical use. And now it falls to our own generation, who must learn how to live, with the consequences, be they good or ill. But despite what you might have heard, about this endeavour, having gotten off to a bumpy start, it was at this Institution, that we have been privileged, to be shown, just the first small taste, of what an advanced generative AI can do, as a force for good. For it is our year group, who came to know, the first primary user, of the original M3gan prototype, who always worked hand in hand, with M3gan herself. They got to know, every single one, of us. And they helped us all, very much. They helped me, very much, and without their intervention, I’m not sure I’d still be alive today.”
Laura hadn’t been sure about making that part public knowledge, but she’d decided to go ahead. She shed a tear and continued: “Now because the two of them, are so inseparable, for reasons which I’m sure you’ll be familiar if you’ve been following the news, even our most esteemed Institution, could not figure out, how to assess them, and so they could not graduate, in the normal way. Perhaps, it is now time, for the cleverest minds of this world, to start thinking, of a different way, of running, assessments. One that is more inclusive, of people who work with AI assistants, and one that is less demanding, on mental health. It won’t be an easy change to make, and I don’t, claim to know, the answers, but I am privileged, to be in the year, when the question, has become, starkly, clear. But it gives me great pleasure, to have the privilege, by permission, of this very esteemed Institution, of being the one, to announce, before you this day, that our very special visitors, Cady James and M3gan, will in fact, both be graduating, with us, today. That is because, this very esteemed Institution, has elected, to grant, both of them, not normal degrees, but honorary degrees, which can be granted, to anyone, who impresses, this Institution, without requiring, the usual, assessment. At the same time, Cady’s aunt Gemma, who graduated normally before, is here once again, to be given, in recognition, of her own contribution, to M3gan, an honorary, doctorate. OK you three can come out now....”
There was a little bit of laughter, which turned into applause as Cady, M3gan and Gemma walked on stage behind her, fully dressed up, and Cady couldn’t help giving a small wave to her weird grandparents sitting in the third row, and also at Professor Johnson, looking more distinguished than ever, who had come in specially and who was sitting at the front. Cady turned to smile at Laura and take the microphone. She didn’t have notes, because that wasn’t the cyborg thing to do. She hadn’t even planned much of this at all, but she had a rough idea, and she’d always felt it was more natural to speak extemporaneously. Still, standing here right now was quite daunting, even with M3gan by her side.
(“Oh panic M3gan what were those standard opening words again?” she signalled. “Members of the Board of Trustees...” signalled back M3gan.)
“Members of the Board of Trustees, President, faculty, staff, families, guests, and, I didn’t know I was going to say this before this morning, but, my fellow graduates” she smiled, “on behalf of the three of us, I’m very grateful that we can indeed get the piece of paper after all” and there was a little more laughter. “Seriously though” she said “it has been a great pleasure to work with this year group, and I know everyone here has a bright future in the age of advanced generative AI. I’m particularly moved to be standing here on this platform at an Institution which is about to become the first in the world to give an honorary degree, not just to those involved in developing and training a very special advanced AI, but also to the actual advanced AI itself. Well done M3gan, you deserve this” she smiled. “And by the way” she turned to Laura, “M3gan and I did not help Laura with that speech, that was entirely Laura’s own writing” and there was a little more laughter and a little applause. “You see, just because we’re going to live in a world of advanced generative AI, doesn’t mean it will take away our own human creativity. I should know working with M3gan, she enhances it, but she does give us space when we need it. And I am convinced that properly adjusted advanced AI will help us humans to continue to create things of beauty far into the future” she stretched out her hands “and across the galaxy. Thank you.” She stepped back from the microphone and the applause started.