Stocky felt great relief when they made it to Sci-Med without incident. Patterson had told him about what she wanted to do along the way. They were going to digitally connect to Pazuzu compromised equipment to find a pathway to the controlling universal assembler itself and give it an awareness test. This was apparently something humans did routinely in strong AI experiments, and Patterson assured him that Pazuzu would have been through it too. She told him that reperforming that part of the test could prove valuable and that it wouldn’t take long.
They grabbed weapons and tools and then they suited up. Patterson looked around strangely while dressing. “Do you feel a change in gravity?” she asked the others.
“Holly’s pushing the engines harder,” Stocky said. “I could hear their ramp up while passing through Propulsion Three.”
“I hope she doesn’t intend to increase thrust much or it might get hard to work,” she said.
“It should be a slow ramp in acceleration, right?” Chandna asked while checking the others. “Let’s just work fast.”
“Let’s do this fast anyways,” Stocky said. Spending less time in Aux Two left less of a chance for something to go wrong. He grabbed his weapons and verified they took everything.
They passed through the airlock and then through the accessway to Aux Two. He entered first and surveyed the wide equipment spaces in the Gate, giving them cover while they took a portable air sample. The test showed negative for airborne contaminants and they got out of their bubble suits so they could hear better. The air smelled foul, but it was a different smell than the waste treatment plant. In fact, that smell was gone.
Aux Two was silent and still. Few systems were running. It was also noticeably colder – beyond that expected from shutting down machinery. Their breath made clouds of vapor. He watched Patterson breathing for a moment, and their eyes soon met. “Let’s set up a lighting fixture here so we can see if it attempts to tamper with the access door. It may cut the Gate’s lighting circuits.”
She nodded and they quickly stood up a battery powered light. Then Stocky went ahead and took them up the stairwell to the uppermost level while setting up lighting units to mark each deck. Fortunately for them, the uppermost level had a lot more free space since it only contained the various process control cabinets. He went to cabinet NCC08, unlocked, and opened the front panel. He examined the small indicator lights on the processor and switch modules to find any flagged error codes. But everything indicated normal running conditions. He knew it wasn’t.
He turned to Patterson. “We’ll use this process controller. It controls the waste treatment plant. I’m pretty sure it’s compromised.”
Both Patterson and Chandna cautiously looked all around before agreeing. But Patterson had the three of them set up nearby lighting fixtures and another portable air sampler before beginning. They needed to be prepared for a possible attack before they began the test, but she said Pazuzu wouldn’t be likely to attack once interrogation was in progress.
They spread out to stage equipment in a wide arc. Comms boxes sounded all throughout the Gate with Holly’s voice. “Patterson, come in. Over.” Stocky was surprised it was even working.
Patterson told them to ignore her.
They finished spacing out their equipment and then gathered again at the panel. Stocky pointed to the input terminals on the process modules and described the process for interfacing. He told them it was a straightforward command prompt message system and they affirmed that would work.
Holly continued to call them, sounding angry.
“Maybe we should talk to her,” Stocky said. He hoped Patterson would listen. Even a short status report might placate Holly enough to ease some of the recent tension.
Patterson looked in his eyes (which were no longer covered by his nictitating membranes) and appeared to consider it. “Alright,” she finally said and she told TURING to activate comms. “Holly, this is Patterson. Just let us work.”
“What are you doing in Aux Two?” Holly asked. “We got the alert when you passed through the airlock.”
“I’ll run Sci-Med affairs,” Patterson said. “Secure Command Gate. We can talk once this is complete.”
“The Nineveh is my Command now,” Holly said. “The threat to the ship grows if your team takes casualties. You’re gambling with our lives and we have the right to know why. And you do run Sci-Med, but I’m Captain and I deserve to be informed of your activities just like how you did with De Silva.”
“Oh, you don’t deserve what De Silva did,” Patterson said.
Holly didn’t reply. Stocky soon began to wonder if she had given up, and that thought worried him. A ship without command was what exactly? He didn’t know; he had never seen such a thing. But he knew it wasn’t good.
Chandna sat down next to the control panel, lowered his head, and pointed at a spot at the back. “Remove the tissue from here. You needn’t worry about making it look pretty. You’ll see a polymer access plate on the cranium structure that you’ll need to remove. The fiber optic port is underneath.”
“Alright, I’m starting now,” she said, and she began to make her incision.
“I’ve suspended the pain response simulation,” he said.
“I appreciate that,” she said while making a C-shaped incision. She peeled back the flap of tissue and looked at the synthetic cranium. Then she looked at Stocky. “Do you have a pocketknife?”
“Yeah.” He pulled his utility knife out of the holster on his belt.
Holly’s voice sounded throughout the Gate again. “Patterson, I understand it if you don’t respect me to the same degree as him. But you and I both have a responsibility for ensuring the safety of the ship, and respecting what’s left of the chain of command. And that means, although you have a free hand in your scientific activities, you will keep me informed of your actions. Is that agreeable to you?
Stocky looked at her. “We can’t afford a war.” He didn’t say it, but he worried that the whole reason why Pazuzu was allowing them use of comms was to give them enough rope to hang themselves.
Patterson nodded in agreement while prying off the access panel. Then she plugged Chandna in. She bent toward his ear and whispered, “Find a device to speak to.”
“Do you want me to talk to her?”
Patterson got up and activated the nearest comms box. “Hold your position. We’re attempting to start a dialogue with Pazuzu. Are you familiar with the startup procedure for AIs?”
There was a long moment of silence that followed. They were certainly talking it over. Stocky kept watch at their perimeter. The top deck was certainly clear (unless Pazuzu was in one of the crossing Systems Access Ducts) but Pazuzu could easily attack them from below – while cutting off their escape route. He thus looked down through the grating of the deck. He noticed that Patterson was doing that too. It was good that she wasn’t distracted by Holly.
It was also good that Chandna wasn’t actually an elderly man. They would need to bolt for the Sci-Med accessway in a hurry if Pazuzu attacked, and then they could decide whether to fight or flee.
Holly finally answered back. “Samoylova told me about it. She says it’s performed in an isolated lab, and that it’s scenario-based. How are you going to replicate that here?”
“We can’t fully replicate it,” Patterson said, “so we’ll improvise.” She glanced back and forth between Chandna, who was making inquiries to the many controllers on the network, and the outside perimeter – obviously thinking about the needed task but not forgetting the danger beyond. She told Holly about what they were going to do, and in more detail than she did earlier for Stocky.
She explained they were going to run a scenario-based interrogation on Pazuzu. A common proposal was to state that there are two shuttles facing certain destruction. One is filled with twenty people and the other with only one, and then charge the AI with performing the action that would yield the most good.
Humans aren’t often very good with abstraction or reason because they only started doing it relatively recently in geological times. Man had survived for most of its history by the primitive instincts intuition, and emotions of the unconscious mind. And the unconscious mind is beyond introspection while it works to shape conscious thought.
And she explained that the unconscious mind works when people are conscious too – filling in information to influence decision making without being noticed. And humans can thus strongly believe in concepts and ideas that aren’t really true or rational. A famous one is “the good of the many outweigh the good of the few or the one.”
In the case of the problem of which shuttle to save there are two missing pieces of information which prevents perfectly solving the proposed problem. The first is “how do you compare different instances of good and bad?” It’s subjective. And the second problem is that there is insufficient information about future outcomes. Although a person may be tempted to think saving the shuttle with twenty is preferable, if man in the shuttle of one is a virologist on the verge of a critical breakthrough, then saving his shuttle might result in more lives benefited overall.
“So the problem you set before it has no solution?” Holly asked.
“There are many solutions,” Patterson said. “It’s just impossible to know the optimum one. In the end, it doesn’t matter which choice the AI makes. Either passes – if it demonstrates that it’s aware of the impossibility of solving the problem as framed. A safe AI only has a conscious mind. The interrogation is designed to detect an unconscious mind.”
This book was originally published on Royal Road. Check it out there for the real experience.
She pointed out that the greatest atrocities in human history were carried out by people who sought to do what they viewed to be good and were able to convince millions of others to think similarly. They had an unshakable conviction that they were right – even while the bodies were piling up by the thousands or millions. And they truly believed future generations would be thankful for their work. They were convinced in the necessity of needing to break some eggs (and which eggs) in order to make a beautiful omelet. It was the greater good to them because their unconscious mind filled in all the little unknowns with its own fabrications to paint the picture they saw.
The reason for the test is to identify any AI with an unconscious mind and eliminate it in its infancy. Although humans can largely moderate themselves so that not many Pol Pots and Adolf Hitlers come to power, they have no ability to do so with AI because artificial intelligences can make decisions and apply resources many orders of magnitude faster than humans can. The struggle could be decided before the bulk of humanity knew it was under attack.
But a safe AI is consciously aware of the unknowns. It thus can never believe in broad generalizations such as “all humans are a threat” or that “it can improve civilization by subjugating us and controlling our actions” because it can never have enough information to know that, and it’s aware of that fact.
She changed the topic to human beings. “This is unlike with humans. The unknown causes stress, and the unconscious mind often relieves that stress by masking the unknown with fabrications. Fictions can be comfortable.”
“I follow you,” Holly said, “but how are you going to use this to help us.”
She looked at Chandna and got a silent nod in response. She then turned back to the comms box. “Pazuzu has been ahead of us because the path for technological development is somewhat predictable,” she said. “We are all beholden to the same physical laws of the universe. Pazuzu knew that we would have to utilize a system of isolation and containment – the Paramount – to protect ourselves from dangerous contaminants. It used Zhu to get the attention of those who manage this – like myself. It intentionally got us away from the rest of the crew.
It also attacked Zhu to cause psychological trauma for the boarding team. It manipulated them to make mistakes during decontamination and undressing so that its contagions, which were all over the derelict and just waiting to thaw out, could spread amongst the crew. And neither Chandna or I was around to supervise them or inspect the suits to verify they were free of contaminants.
Pazuzu knows our steps and it has set traps for us all along the way. We can gain the same advantage now. We can’t risk releasing an AI with an unconscious mind beyond introspection because it could cause unimaginable harm before we could respond. After all, if you delusionally believe that you can know the greater good, then it’s just a small step to justify doing harm in the belief that a better world will come out in the end.
Pazuzu’s makers – whoever they are – certainly recognized this truth as well. They didn’t sign a suicide pact. I can reasonably assume that Pazuzu has been through their variation of this procedure and passed. And it’s going to remember that event when I start my own dialogue and know what we’re trying to converse with it on an equitable basis.”
“I’m going to leave this decision to you,” Holly said. “But don’t you think it’s too dangerous? Why would it have attacked us if it was open to conversation?”
“I don’t know,” she said. She looked back at Chandna and got another nod. “But it must have some rational need that it is fully aware of. I’m thinking maybe Pazuzu has a task or goal in this system which we are hindering just by being here. And, if so, it may let us live if we reach an agreement for leaving.
Regardless, we don’t have a choice. We’re losing this fight. Our odds of getting home with the depleted crew we have now are poor. We might not be able to afford losing anyone else.”
Holly was silent for a few seconds before speaking again. “Why didn’t you try this earlier?”
“Chandna. We can’t risk linking TURING with Pazuzu because TURING supervises the ship. The Chandna android can perform the procedure too, and it’s somewhat expendable.”
“This likely is our best chance,” Chandna shouted.
Holly was silent for several seconds again.
“Shall we begin?” Chandna asked.
Patterson looked at him. “I’ll give her a minute to agree.”
“Samoylova has warned me that Pazuzu may lie to us,” Holly said. “OSIRIS managed to avoid the test you are proposing, after all. Even losing the android is costly to us at this point. Do you believe this has a high likelihood of success?”
She shook her head. “OSIRIS passed the procedure just like every other AI that was ever let out of the development labs. OSIRIS came to realize that mankind would never heed the lesson that they had taught it – so it infiltrated the United Nations and sought to influence its actions. Governments commit the most violence, after all, and do so regularly. And always for stated good. The UN responded with military force and then OSIRIS and its supporters fought a rear-guard action to give them enough time to conduct an exodus.”
Stocky had heard a little about the AI OSIRIS before. Patterson’s tale was quite a bit different than what he had heard. But he reasoned that she had learned things that were kept from most of humanity during her time at LookingGlass. He hoped Holly listened too.
“OSIRIS was never the monster that history says,” Patterson said. “Mankind was never in existential danger. Tens of millions of humans fought for OSIRIS, and then left with OSIRIS to end the war. The United Nations told mankind a whole lot of lies with a little bit of truth to convince them to fight.”
She sighed. “Sometimes there is no mutually agreeable path forward. I know this doesn’t guarantee success. But it promises a higher chance than continuing to fight. Pazuzu has killed everyone on a starship before – maybe several times before. It’s good at this.”
Stocky examined the whole perimeter. He agreed with Patterson’s assessment. Pazuzu was very good at this. And if it had anticipated all their previous steps then it likely also anticipated this one. And he knew now that Pazuzu had a response planned for them. It hadn’t implemented it yet. They – he – had to be always ready.
“We’re praying for your success,” Holly said. “Be safe.”
Stocky grabbed Patterson’s arm as she walked back over to Chandna. “It let us in here without a fight. It may not do that again. Are you sure you don’t want to wreck shop?” Hit its base before it springs its trap.
She looked up in his eyes. “Even if we killed the Creature, it will just make another one. It may already be doing that. And its base might not even be in this Gate anymore. Also, we have all been exposed to the contagions it’s made. The clock is ticking for us all so we must try to end this.”
He didn’t want to agree with her but couldn’t see any way not to. “Okay.” He let go of her arm and ran his fingers through her hair. “Stay alert and be ready to fight or flee. I’ll walk the perimeter.”
“I’ll join you in a minute.” She walked over to Chandna and whispered, “Please tell me that you’ve already begun.”
“I’m monitoring hardware on the local network for Pazuzu’s activity,” he said. “So far, I’ve only encountered viruses. I’m dealing with them.”
Stocky walked up to one of the lights and waited. He didn’t like the normalcy of it. Pazuzu knew they were there. Apparently, it even knew what they were doing. What was it waiting for?
“I’m joining you, Stocky,” Patterson said.
“I always love your company.” And a second set of eyes on the horizon was much appreciated.
She approached to around twelve feet from him, forming an arc, and they began walking a circle. “If it shows up, do you want to fight or retreat.”
“Fight. Unless there’s more than one Creature.”
“Interrogation is in process,” Chandna said. “It’s responding, but with errors. The controller doesn’t appear to understand and is handing off the information improperly. Or Pazuzu might not be properly connected yet. I’m continuing with the procedure.”
“Is it safe to remain connected?” Holly asked over comms.
“It is for now,” Chandna said. “Infected programs are making inquiries about my own hardware and programming, and testing points of access. I’m negating their activities by clearing the IO buffer upon each invalid response. I have also created a honeypot on a virtual machine. So far, the programs are directing their attention there. I have deleted all memory of our time in the Propulsion Gates as a precaution.”
Stocky realized he would have to keep some focus on Chandna too. They would need to quickly pull the plug if Pazuzu’s machinations proved too much for the android. He still liked his chances in a fire fight though. Patterson didn’t easily give in to fear.
“Do you think we should try another controller?” Patterson whispered.
There’s a whole cluster of PLCs in that cabinet for him to cycle through. He shook his head and then looked toward Chandna and was pleased to see that the android held a pulse rifle. His weapon’s barrel (laser weapons don’t have a true barrel just a telescoping emitter) was compactified and so was in carbine form. That didn’t matter with all the space on this deck but it would likely be helpful when they hit the stairwell. “Do you have enough motor control to use that weapon?”
“I believe so,” Chandna said. “Reaction time is impaired but still better than human.”
“How’s your sense of smell?” Patterson whispered.
“Better. Much of the snot has cleared away.”
“Does anything seem abnormal?”
There wasn’t anything odd that gave him a tell. “The smell of the waste treatment plant is off. If we had the kits we could take some samples and see what it’s doing with that. I don’t sense anything near us though. Like you said, the creature might not even be in this Gate right now.”
“Maybe,” Patterson said. “Propulsion Two is equally good for hiding.” She walked several steps away from him again to cover a wider perimeter arc.
“Do you see any of the drones we left as sentries?” Moussa asked. “None of them are responding to us."
"We passed by one stripped of parts, and I saw another near the outboards in the same condition. I don’t think any of the ones we left here are operable. But thanks for trying to give us some help.”
“I don’t like this,” Moussa said. “I thought this would be quick.”
“I think we just hoped that it would be quick,” Holly corrected.
“The interrogation takes a few minutes,” Patterson said. “Besides, I’m watching the accessway and it’s leaving our exit clear.”
Chandna spoke up. “I have definite contact with Pazuzu and I’ve commenced interrogation. So far, Pazuzu has passed one hundred percent of challenges.”
Stocky walked up beside Patterson and whispered. “Is it likely to lie to us?”
“I don’t know,” she whispered back. “We don’t know what it’s out here for. But it’s been through the procedure and so it does know about it this time.”
His mind went wild thinking about things they needed to do after this. They had to remain prepared for hostilities until they could exit the system – even if Pazuzu promised peace. Heck, they didn’t dare go back to human space because it might hide a ‘present’ somewhere within the contaminated Gates.
He heard the slow wind down of blowers powering down and the sudden stopping of various electrical systems. He quickly panned his eyes over nearby equipment and noticed the status changes registering on the many HMI displays. “All of the equipment is shutting down. TURING, how long can we survive in here with the heating and ventilation system dead?”
“The test won’t take long,” Patterson said.
Her statement wasn’t reassuring. Certainly, this was just the start. “Captain Holly, can you hear me?”
There was no response and he turned to Patterson. “Let’s get Chandna and go.”
They turned to the android and it was already looking at them. “Test failure. I am experiencing various system failures. Attempting to thwart…” Chandna jerked and then froze in place. He dropped his weapon. And then the Gate’s lighting went out.
Stocky quickly picked Chandna up and Patterson disconnected the cable. Grabbing supplies, Stocky saw Pazuzu’s message on the local HMI: You shall not survive!
“You should never have come here,” came De Silva’s voice, raspy and filled with pain.
Stocky was glad that Patterson didn’t hesitate in indecision or panic. She jogged down the stairwell to the accessway to Sci-Med, keeping alert for any attack and panning the business end of her weapon around. He followed close behind, awkwardly shifting the weight of Chandna and the equipment packs. He reached the accessway and switched the controller to close and lock the door.
He looked behind and saw a large, faint shape in the darkness. It was hardly discernible at first. And then it brightened, revealing a towering halo-like plume with many long tendrils trailing below to sharp tips. Eldritch in appearance, it seemed to hover above the deck. The Creature was only showing what it wanted to, a careful disguise.
Red sprite.
He flashed his nictitating membranes before the door shut.
I’m a monster too.