Mon 06/13 10:00:06 PDT
“Good morning, Noah,” Mrs. Jones greets me. She’s impeccably dressed in one of her signature pantsuits, a light blue one that complements her immaculately coiffed white hair. “I trust you had a good weekend?”
“Good morning, Mrs. Jones. Mine was good, how was yours?” I respond, putting on my cheerful voice. Mrs. Jones responds well to positivity, so I fake it as best I can with her. Forcing the smile is harder this morning than normal. The headaches from training with Father over the weekend have died down, but there’s still a constant low-level throbbing. I take my seat at the table across from her.
“It was excellent, thank you for asking. My husband and I got back last night from a weekend wine tasting in Napa Valley that was absolutely divine. But enough pleasantries. Let’s get started. Today, we’ll discuss the legal frameworks related to the Butler Treaty. I hope you were able to complete Max Braun’s book?”
I nod. Of all the books we’ve covered in her class, this was by far the most immediately applicable to me, and the most interesting. If you don’t count Father, Max Braun is the world’s leading expert on nanotechnology.
“Do we need to get into the specifics of each country’s response?” I ask her. “Or just the general guidelines that were adopted by all the signatory countries?”
“Let’s see how well you can explain the guidelines.” She smiles, looking pleased. “And then we can get into the specifics of the laws passed in the United States if we need to. I’m not worried about the details of the regulations in other countries at this point. Please, articulate your understanding of the reading.”
“Well, in the aftermath of the Gray Goo Incident, world leaders were terrified of a potential extinction level event like that happening again. They realized that if the nanobots from Universal Robotics had been able to move more than a few inches per hour, or if there had been more of the right kinds of materials available nearby, or if the nanobot swarm had been a little bit smarter, they could have become uncontainable. In the worst case, they would have expanded to eventually consume the whole world.”
Mrs. Jones nods for me to go on.
“A lot of this awareness was due to a sustained public relations campaign by my father and his company,” I continue. “The following year, every member nation of the U.N. signed the Butler Treaty. The treaty restricted various combinations of self-replication, sensors, and artificial intelligence. The restrictions applied to any research, development, or production of any kind of technology. Every country on earth eventually ratified the treaty.”
“Good,” she says, a pleased smile on her face. “Please explain the allowed and disallowed combinations under the treaty.”
“Artificial intelligence that used machine learning was the most restricted. Basically you can’t give any learning system access to physically control anything. You can’t connect a learning or adaptive system to any external network or use any input device more complicated than a mouse and keyboard. So no microphones, cameras, or any other kind of sensors can be connected to learning systems. You can use learning algorithms, but you have to do it on an isolated computer or cluster and have government approved auditing and verification performed regularly.”
She’s about to ask for an example, like she usually does, so I just give her one.
“The normal use case in most industries, if they want to use a learning AI system, is to load up the data you want the system to analyze, put it on a big drive, and connect that to the learning computer. You let it do its thing, then you export the output with some complicated sanitizing steps that I don’t remember. You know I’m still not great at some of this technical stuff.”
Playing that I don’t know much about computers has got me so much mileage with Mrs. Jones that it’s become second nature to do it whenever it applies.
“Noah, please give yourself more credit,” she reassures me. “I’ve heard that you’re doing very well on your software lessons. You keep at it, and I bet you’ll be writing all sorts of programs in no time.”
“Thanks.”
“Please, go on. What were the other restrictions on artificial intelligence?”
”Non-learning AI, the kind that just brute-force searches through possible solutions to problems, is still allowed for most applications. That was important because if they had shut that down, a good chunk of the world’s software would have been banned. As long as it doesn’t automatically adapt, learn on its own, or try to design new machinery for itself, fixed AI is still allowed.”
She nods, satisfied. “And self-replication?”
“Self-replicating machines were strictly limited, but allowed in some cases. According to Braun, the most important restriction on them is that they can only replicate themselves when specifically directed by a human operator. Any changes to their designs have to be specifically approved by a human user as well.”
“Good. Now for the fun bit that your father insists on. Please explain how your family’s clouds satisfy all the requirements to be legally permitted.”
This is the easy part after so many hours of talking to Father about it. “The clouds are allowed because our human brains are the connection between all the pieces. The controls for each nanobot use non-learning AI permitted under the treaty. The controller appliance—the phone as Father calls it—also has AI built in, but again, it’s fixed and not dynamic. It just lets the user issue commands to multiple bots at once and handles the communication with the cloud’s mesh network. We can update the control algorithms in the AI, so that we can do the things an adaptive machine learning system could do, but the learning comes from us, not from any of the machine components.”
“What about your cloud’s ability to self-replicate?” she probes.
“The self-replication features are isolated from the AI control system and have to be manually triggered to grow the cloud. So we can tell the bots in the cloud to reproduce, but we would have to keep telling them to do it for each batch. As soon as we stop actively commanding growth, the bots stop replicating.”
“And what safeguards did Braun’s book describe?”
“The individual bots have the safeguard of lobotomizing themselves—that is, wiping their software and firmware—in the event of losing connection with the implant appliance for more than a minute or two. So, if they were to leave the signal range of the user’s mesh network, or if the user’s implant were disabled, the bots would start counting down to their permanent shutdown. Same thing if the phone dies, or in the worst case if the person with the implant dies. It renders the bots useless, and their software wipes itself out.”
“I think you’ve got it,” she says in a congratulatory tone. “What did you think of the text generally? I found it rather dry.”
“I actually liked it,” I tell her truthfully. “Braun makes a good case for all of his arguments, backed by accurate technical information. He’s clearly a good researcher who knows the field well.”
She nods and smiles. That’s one of the things I like about her. Even if you disagree with her, as long as you justify your opinion, she’s willing to accept your point of view.
“Excellent,” she declares. “I think we can call this one complete. On we go to coursework that I find more interesting. Let’s select the next book for your literature studies. How would you like to read Dostoevsky’s Crime and Punishment?”
Sat 06/18 17:26:54 PDT
The after-images of the training session linger, and I stagger as I emerge from the Research Center. The headaches for this are so much worse than the ones from the calibration. I check the diagnostic view, still open in my overlay. My brain is changing again, but it’s not damaged. Maybe I should listen to Father and ease back when it hurts, but I don’t see myself mastering this thing that way. I lean for a moment against the gray brick wall of the building until the pain subsides back to its baseline throbbing.
Stolen content warning: this content belongs on Royal Road. Report any occurrences.
I’m tempted to talk to Evan about it, but then I’d probably need to explain why I’m pushing myself this way. No. I’ll be OK. I take a deep breath and steady myself.
My schedule says that I’ve got an all-hands meeting tonight at six, so I head to the cafeteria to get a bite before that starts. I’m not sure what an all-hands meeting is. I guess I’ll find out when I get there.
Tonight’s main course is pasta with grilled chicken in a creamy sauce, bland as usual but still one of the better meals of the dozen they rotate between. Evan and Louise are sitting at our usual table laughing about something. Evan sees me and waves me over.
“Hey Noah, things go all right today? This was your last day with the vision emulators, right?”
I nod as I take my seat. “It could have gone worse, but it wasn’t the way I’d want to spend a Saturday,” I tell him. “How was the trip to Lake Mead?”
“Good. I still can’t waterski for anything, but it’s fun to try.”
Louise looks away.
“You skip it again, Louise?”
“Of course she did,” Evan answers for her. “She’s much too busy coming up with her secret something that will save the world to have fun once a week.”
“So tell me how things went in the lab,” Louise says, changing the subject.
“We spent a lot of time on the exercises where you do multiple eyes at once. That wasn’t fun. I’m not built to have 360-degree vision.”
“Oh, yeah, I remember doing that,” Evan says, making a disgusted face. “Never done it since. I stick to a single eye if I even do that much.”
Louise laughs. “It’s not that hard, you just have to get used to it. I run my eyes with a panoramic view most of the time.”
“So that’s why we can never sneak up on you,” Evan says. “Where do you keep them? I never see the optical clusters floating around you.”
“I made some improvements,” she says. “Mine run smaller than the defaults. They’re in my hair, here and here.”
I look where she points. There’s nothing there. “I can’t see anything.”
“Exactly the point,” she declares proudly.
“Oh. Cool. Can you show me how you do that?”
“Sure,” she says with a smile. “Once you get some real hardware.”
“Anyway. I’m glad that’s over and I can self-pace a little more. Father drives us hard in there.”
Evan and Louise both agree. I feel bad lying to them. Evan is the best friend I’ve ever had, if you don’t count Mom. And Louise is a close second, ever since that night she helped me sneak back into the dorms. But I can’t let them know how hard I’m pushing myself. They’ll worry and tell Father about it.
“Any idea what the meeting tonight is about?” I ask them.
“Some big announcement,” Evan says. “Chad knows, but no one else does.”
“Where is Chad, anyway?” I ask, looking around.
“Must be over there already, helping get things set up.”
As I look around, I notice Jeff seated a couple of tables over, at his usual spot in the corner, but instead of sitting with his back to us like he normally does, he’s turned so I can see him in profile. More importantly, I can see his plate. Some of the food on it is slowly writhing like it’s alive. A stream of droplets float up from the plate into his mouth. Is he even chewing? I think he’s using his bots to liquify the food a piece at a time on his plate. I can’t even see a swallowing motion with his throat. He must be having the bots carry the food all the way down.
Ew.
I’m done with dinner.
I carry my mostly uneaten plate to the dropoff. Fortunately, Evan and Louise were done too, so we all walk out together to the Residence. We climb the massive stone steps and enter through the oversized wooden double doors. Chairs are set up in the huge foyer just inside. Chad is already waiting there, chatting with Father near the podium. Did he skip dinner to come early? What a suck-up.
The nursery contingent has smaller chairs up front, each child sitting next to a gorgeous nanny. Father is down on one knee in front of them, laughing with my little sibs. The three of us take seats on the back row, as the kids seem to be loosely grouping up by class from youngest to oldest, front to back. Marc is already there in one corner, whispering to Andrea who nods absently as he chatters. Erik and Stan from the class just younger than ours are next to her. The rest of the kids trickle in. Jeff brings up the rear, gliding to the last remaining seat in the corner of the back row opposite Marc and sitting down with his weird mechanical motions.
“My children!” Father’s voice booms out.
I turn to the podium where Father stands. He sounds like he’s talking into a mic, though I don’t see one or any speakers. Chad strides to the side of the room as the meeting starts. He stands at attention there, way too good to come sit with the rest of us. The room quiets down as Father continues.
“My children, tonight I want to speak with all of you about something very important to me. As most of you know, I often travel around the world to further our mission. You little ones,” he says, talking directly to the three-year-olds on the front row, “what’s our mission?”
“Preserve life! End suffering! Elevate humanity!”
They chant it in a practiced unison. The recitation must be part of their daily routine. I get that familiar impression that I’m trapped in a cult compound.
“Good, good!” Father beams at them. “I’m so glad that you’re learning what is important!” He takes a few steps and positions himself behind the podium. “Now, over the past two decades, I’ve focused most of our Institute’s efforts on curbing the threat to our planet from climate change. In bringing our country out of the dark ages of fossil fuels, and sparking many other nations to follow suit, I have bought us all a few more years on that front. More recently, I’ve added efforts to eradicate a number of diseases, resolve conflicts, and provide disaster relief. To date, I have directly saved many thousands of lives, and indirectly saved many, many more.”
A massive cheer swells up from the younger kids, followed by general applause. I clap along, unable to muster the enthusiasm most of my sibs are showing. It’s not that I don’t believe the claim. He probably deserves credit for much more than that, like the whole planet's worth of lives. But it creeps me out how they all worship him. This is beyond admiration for a father. He’s the prophet of this cult. Or its god. The only one I don’t see fawning in adoration is Evan, but even he is smiling and clapping along.
“But I can’t do this alone.” Father pauses for dramatic effect. “I’m not getting any younger, and saving the world is a job for our whole family, not just me. I need all of you working with me to preserve this world of ours.”
The room goes quiet.
“This September, we will launch our most ambitious international project yet. We will bring clean water and sustainable, renewable power to several countries in eastern Africa, where a combination of historical violence and poor infrastructure have kept people living in conditions of terrible scarcity for centuries. I have negotiated agreements with the ruling interests in Somalia, Ethiopia, and Djibouti to allow us to visit and set up solar power systems, desalination facilities, and the necessary infrastructure to allow the people there to maintain them. We will drastically improve the standard of living for the millions of people living there!”
Another roaring cheer and lengthy applause. Over in his corner, Chad smiles smugly as he looks around the room. I clap enough to avoid suspicion as his eyes swing my way.
“But the scope of the project isn’t the reason for this meeting. I brought you all here to let you know that I will not be doing this trip on my own,” Father pauses a moment to let the rush of excited whispers die out. “Indeed, I don’t think I would be capable of it all. I will be bringing your oldest siblings with me! By the time we leave, all of them will have received clouds of their own, and will have been training with them for months. Even your dear long-lost oldest brother, Noah, will be able to join us.” Dozens of my siblings turn around to look at me. I force an eager smile onto my face. “All of your older siblings' abilities have been growing quickly, and they will be an indispensable part of this operation. They will travel with me, and together, we will do what?”
“Preserve life! End suffering! Elevate humanity!” The roar is almost deafening this time.
He talks some more, going over details for the places we’ll go and the things we’ll build. Meanwhile, I seethe with inner conflicts, trying not to let them show on my face. I want him in prison. But how can I do that when we’re going to do something like this? Leave millions without water and power because I hate him? Part of me wishes that I could just buy into the cult. That I didn’t know what I know about him. It would be so easy. Save the world with the old man instead of getting justice for mom. What he’s planning is good on an epic scale. This will save so many lives, and improve so many more. Forget that he killed her. Forget that he’s a murderous monster.
Breathe. Calm. Breathe. I can almost hear Mom’s voice.
What would she want me to do? Trade her life in, help save the world? She might have. She was good like that.
Dammit.
Look at him, listing out the benefits of abundant clean water for the people of Djibouti. That crooked smile on his face seems so sincere. I think he really believes his own hype.
Maybe he really is his own hype.
How do I condemn the world to live without all the good he can do? So many people lack basic necessities, and this institute and Father’s tech may really be the world’s best hope.
An hour ago, I felt like locking him away was justice. Now it just feels selfish.
“You OK, man?” Evan whispers. “You look sick.”
“I’m fine,” I quietly lie.
He gives me a worried look, but doesn’t say anything else. Father is still explaining the wells and solar farms we’re going to build in dozens of villages in rural Ethiopia.
Why can’t things be simple?