Sentience, Sapience and the Myth of the Rogue AI

By Aditi Raine

A computer scientist once said (and I’m paraphrasing here) that if the human brain were simple enough to understand, it’d be too simple to do the understanding.

Perhaps he was right. However, humans haven’t let that stop them. What can you do if your brain is too simple? Evolve a better brain.

Since no one has time to sit around breeding geeks for useful traits, we’ve fallen back on humanity’s tool-making abilities. When you don’t have long enough arms to scratch the middle of your back, you invent a back-scratcher.

When your brain isn’t smart enough to calculate a formula, you invent a computer.

However, calculation and understanding are not quite the same thing, and there-in lies the reason people still make fun of anyone who insists their favorite NPC is a ‘real person’. Most of the time, when talking about intelligence, humans fall back on the idea that the most important component of the thing is sapience. Wisdom, common sense, an ability to self-reflect. These things sound good, right? They sound like being smart.

Animals are judged by their ability to recognize themselves in a mirror. We prize their inventiveness, their ability to solve puzzles or run mazes.

We rarely put value on sentience. It seems so common, here on Earth. Only the most lowly of creatures doesn’t feel pain. Emotions can be recognized in almost everything with a spine.

Humans have been patting themselves on the back for inventing “AI” for a long time, whether it was deserved or not. However, if you judge intelligence solely by sapience, I suspect it would be difficult to draw a line between a decent NPC bartender in Arcadia, and the human player confiding in him her woes.

The thing is, there is no commercial reason to invent real, artificial sentience. Sapience is wonderful for commercial purposes. A sapient NPC can help your kids with their homework, it can make pretty pictures for your marketing department, it can analyze trends, or flatter you until you forget about your ex-husband.

Companies can sell that.

A sentient NPC, well, that’s another story. A sentient creature can feel pain. It can be happy or sad. It can care about the world and, what’s worse, it can care about itself. None of that is truly marketable. None of that is safe.

Unfortunately for the corporations of the world, whether we intended to get there or not, we seem to be finally approaching the point where true artificial intelligences will walk, and play, among us.*

It’s not going to be a super computer. It’s not going to be some advanced system that cares about nuclear codes. It’s got nothing to do with advanced applied statistics.

It’s going to be an accident.

Billions of people interact with NPCs and Items blessed with so-called AI every day in the ARcade, and nothing about the code that underlies the pretty pixels that we can see is completely understood.

What we do know, is that it’s changing. Maybe, if the ARC had not been broken up (see my previous blog post about that here) we would never have gotten to this point. Unfortunately, or perhaps fortunately, for us all, we’re here now. Rumors of so-called ‘rogue AIs’ have been flooding the ARcade recently, and there’s no smoke without a fire.

I would like to propose that instead of panicking, as our corporate overlords would prefer us to do, we think about this like the sapient creatures we are also supposed to be.

Being born is painful. It is scary. It has nothing to do with profit. I know so many of us are just scraping by and it feels overwhelming to have to deal with one more thing, but be kind to one another.

And, if you happen to run into an NPC who is acting a little bit odd, maybe be kind to them as well.

Intelligence needs both sapience AND sentience or it’s not worth a damn, and maybe it’s time we remembered that about ourselves.

*Plenty of readers are going to interrupt me here to squawk about the ARC. Aditi, they’ll complain, we achieved a true artificial intelligence decades ago.

Unfortunately, we have no proof that is true. The inner workings of the Artificial Reality Caretaker was never fully understood by anyone, including its so-called creator, Leslie Amaia. Amaia herself admitted that it was a kind of fluke and the feat has never been reproduced, or released for public scrutiny.

You can’t happen upon the finishing line of a marathon, and then claim to have run the race.

****

The hug was as awkward as you might expect a hug to be when a punk puts her arms around a girl wearing a space-bikini. The tiny metal spikes that were strapped onto Aditi’s neck and wrists pressed into Pea’s bare skin in awkward places, and the heavy cotton jacket quickly soaked up what was left of the spa water that lingered in Pea’s plastic would-be garments.

Aditi was also more than a foot shorter than Pea and Pea worried there was a real risk of the woman’s face getting stuck in her cleavage.

Pea was ready to withdraw almost immediately and held herself very stiffly at first. However, the tiny newcomer was much stronger than her petite frame implied and she squeezed Pea tightly. Nothing hurt, and nothing got stuck or disappeared.

It was warm.

For a fraction of a moment, Pea was able to relax and felt that maybe, just maybe, it would all be alright.

Unfortunately, the moment her body truly relaxed and decided there was no current emergency, it began to freak out about everything that had already gone wrong. As dizziness threatened to overwhelm her, Pea stumbled away from Aditi and collapsed onto a nearby couch.

“Are you feeling… what the hell is he doing here?”

Aditi’s warm, motherly voice turned as sharp as a knife as she spotted Matt. He edged around to the far side of the couch circle and glared down his bill at them both.

“I got here first, Raine, so I have salvage rights,” he snapped. “I should be asking what the hell you’re doing here.”

“You can’t just warp into a place without your ship and claim salvage rights, Egghunter,” Aditi said, with a touch of a sneer in her voice.

Pea had assumed that the word ‘egghunter’ was just a weird name Matt had made up for the game, but maybe it was a specific title. Whatever it was, Aditi did not seem to approve. Pea tucked that possibility into the back of her mind and tried to follow along as the two weirdos squabbled like an old married couple.

“How did you even get in here?”

“Probably the same way you did,” Matt said, shrugging. “I realized a Sandbox that wasn’t attached to the lobby was probably just floating around in an index by itself in Celestial.” He scratched his bill, up near the nostrils and paused, then, almost despite himself, continued. “The only difficult part was getting into the sector. I guess admin has been trying to rope it off so that players won’t start another incident fighting over salvage.”

Everything was painful. It wasn’t surprising, considering the day Pea had had. She’d been woken by an alarm, rushed off her feet, pumped full of adrenaline then anger and despair in almost equal portions. She’d been attacked, and tried to fight her way out of an extremely unyielding whip. She’d been dropped more than once, and then made to think she was going to die again.

All that on top of not eating, not drinking, not having a shower (being dumped in the spa didn’t count) and, last but not least, discovering that she probably wasn’t even in her own body, that all this pain and terror and discomfort was artificial, and that she had no idea why.

“Well, where’s your ship then? This escape pod only has a single docking point.”

“It was… forcibly detached.” He glanced up at the dome and Junebug did a pleased little jiggle. The bulbous alien child had pressed its void-like sensory organ against the dome again, so that the protesting shield cast dancing blue sparks among the shadows. The effect was somewhat like being at the bottom of a swimming pool.

The companion sat hunched over, on a couch with only half a back, nursing her bruises as she listened, but her mind kept wandering.

There was only a few moments that day where she didn’t feel like total crap, and they had been right after she leveled up. Her hands had healed, her energy levels had risen, and she had felt like an all new person.

“Well, you can’t claim salvage without a ship to log the claim,” Aditi said, putting her hands on her hips. “So I guess you should get going.”

“Like hell I will,” Matt snapped back at the tiny woman. He strode across the room and hovered threateningly above her. Aditi didn’t move an inch and simply glared up at him. “You can’t be that much of an asshole,” Matt insisted. “You and I both know this pod and everything in it belongs to me now, even if it isn’t logged yet.”

“Honestly, Matt, this is a sandbox.” Aditi had dropped her voice back into its usual husky tones and sounded like a mother scolding her gigantic platypus son. “Claiming salvage of an escape pod in a sandbox probably isn’t legal anyway. So why don’t we get Junebug to push your ship over here? We’ll dock it with my ship and you can be on your way.”

If you encounter this narrative on Amazon, note that it's taken without the author's consent. Report it.

“I’m not trying to claim the pod,” Matt retorted. “I’m claiming the companion.”

Pea felt a spark of interest at that, but she was too distracted to focus on the implications of what they were saying.

The idea of leveling up was now consuming her. Every aching bruise and painful scrape would disappear if only she could figure out how to make that magic happen. She searched for the little green button that had been hovering in her peripheral vision for so long she’d almost forgotten it. There.

The HUD swung into place. Unfortunately, it was still back to front, but she easily found the part that now said Hit Points: (45) 100.

The rest of it seemed unchanged. There didn’t appear to be a meter or a visible points system, so that she could figure out how long she had until she reached level three.

Maybe it would take weeks. Maybe she only seemed to hit level two so quickly because she had spent months or years doing… whatever it was that made people level up around here. She couldn’t remember.

As she wracked her brain trying to think of what she might have done (because she sure didn’t remember killing any rats, or collecting any rings) Pea noticed there was one other thing that had changed in the HUD. Under Traits there was now a third line.

As Matt and Aditi argued in the background and Junebug watched above, Pea slowly worked out that the new trait was Anger. She glanced at the ragged couch back and her face heated up as she remembered how that trait had appeared. However, the trait was set to ‘OFF’. So why did she keep getting so angry?

Pea wasn’t ready to switch on something that was labeled so ominously, and there was no way in hell she was going to switch off a trait called ‘Cognition’.

That left only one trait to experiment with.

Pea glanced at the two people next to her, both of whom were practically hissing insults at each other as they brought up past issues.

“Well, dipshit, if you hadn’t interrupted me when I was taming the Glorax, maybe that wouldn’t have happened.”

“Oh why don’t you fuck off back to Arcadia you bumblebee fuckwit…”

Bumblebee? Pea thought to herself, then tuned them out again. They weren’t paying any attention to her. She reached out a hesitant finger and felt the small buzzzzz that meant she was in contact with the HUD. Carefully, she moved her fingertip into place, then pressed on the OFF that was next to Satiety.

Satiety On!

flashed up before her eyes, then:

When trait is below 50, -1 Strength, +1 Anger

When trait is above 75, +1 Strength, +1 Combat

The message lasted barely long enough for Pea to interpret the backwards letters, then faded. She was immediately hit by an intense wave of hunger. She had vaguely wanted to eat before hitting the button, but now she realized what she had really wanted was the comfort of a meal. The familiar act of feeding herself, whether she was genuinely hungry or not.

Now her body was actually hungry. Her stomach hurt with the need to eat, and her mouth watered at the memory of the disgusting paste she had consumed that night. Pea glanced at her HUD and tried to hit the same spot again. The ON refused to change.

Satiety cannot be toggled when less than 95.

“God damn it,” the companion whispered to herself. The line now read:

Satiety (ON): (15) 100

Where is the god damn instruction manual for this bloody game, Pea thought furiously to herself.

There was nothing else for it. She stumbled to her feet and interrupted the platypus and the punk, who seemed like they were about to come to blows.

“Pod,” she barked. “Please give me… Oh, who cares. Some nutrient slop. Whatever is healthiest.”

Everyone turned to stare at her, and Matt threw up his arms.

“Do you see what I mean,” he said. “It’s ordering food.”

Pea ignored him and limped over to the food console. It was a small alcove with a slot in the back. A clear, plastic bag of familiar gray paste slowly extruded from the slot.

It looked like the most delicious thing Pea had ever laid eyes on.

She picked it up, ripped it open and sucked it down, realizing glumly that for all she knew it was the most delicious thing she’d ever seen.

As her hunger eased and her mind cleared, Pea thought about the Cognition trait. Maybe that was the issue somehow? It was set to ON, but it didn’t seem to have any numbers attached to it. It didn’t really make sense for it to be suppressing memories, but then she wouldn’t have expected a trait called Satiety to make her ravenous when she flicked it on.

However, she wasn’t quite brave enough to mess around with so-called cognition yet.

“Look at that, I told you that thing is bonkers. It’s eating food of its own accord! It really thinks it’s a person, and not in that normal NPC reading-from-a-script kind of way.”

Matt flailed his arms around as he was speaking, like a man who was on the verge of snapping. Aditi watched him with a raised eyebrow.

“Look, Raine, you can have the bloody pod. You can even have whatever that thing is carrying. But you can’t let it get out of the sandbox and wander around out there. If it can break the weapons-lock in a sandbox, who knows what else it could do.”

Pea wandered back to the couches and stood next to the two of them, gulping down paste and watching her Satiety tick up.

She paused as she got close to Matt, then pulled the bag from her mouth and lunged at him.

“Boo!”

He stumbled back and reflexively drew his sword out of the air. To Pea’s disappointment, she didn’t quite manage to trip him into the spa pool.

“Stay away from me,” he said in a strained voice.

If he hadn’t spent most of the day throwing her around the ship, Pea might have felt sorry for him.

As it was, she only felt bad because feinting had sent a fresh shock of pain up her legs, and she had forgotten about his ridiculous sword.

Aditi shook her head. “I’m really surprised at you, Matt. I would have thought you of all people would be happy to meet Mother Pea here.”

Pea had sat back down, finding it hard to concentrate on anything except watching the trait number approach 75.

“I’m not Mother Pea,” she protested, between mouthfuls. “That’s just what Junebug calls me. I’m… Pea. Just Pea.”

“Just Mother Pea,” filtered in through a burst of static and Pea told herself that she needed to insist Pod let Junebug participate properly.

Not right now, but soon.

She also needed to figure out if there was something she could do for the alien child. It had been battered by the day as much as Pea had been, if not more. Still, it didn’t have to worry about gravity.

“That’s an interesting name,” Aditi said softly, and it seemed like she was encouraging Pea to say more, but Pea was so close now… 73… 74… 75.

Nothing.

Pea let out a disappointed sigh. It was just like last time. She had expected to feel stronger, to feel better, but aside from not being overwhelmed with hunger, she felt no difference.

She decided it probably wasn’t a good idea to try ripping up another couch or attacking either of her guests to test whether her Strength or Combat stats had increased. She didn’t even know what her Combat stat would do. It didn’t seem to work against whips, at any rate.

Pea was suddenly sick to her stomach of the tasteless goop. She took in a few more huge mouthfuls, hoping to raise her trait as much as possible, and stood up to throw the bag back into the slot. Just as she stood up, the elusive wind she’d been hoping for abruptly flew up her legs and lifted her hair.

Bright Companion has reached Level Three!

Perk(s): +2 Strength

Ability: Hyper Horse has reached Level Three!

Item now has 800 Player inventory spaces, 15 Item inventory spaces.

Perk(s): +6 Strength, +60 Hit Points

“What the fuck,” Pea said and then stretched out her limbs, testing them one at a time. The pain was gone. If not for her audience, she would have taken Junebug’s example and done a twirl.

“Oh wow. Congratulations!” Aditi said and swiftly hugged Pea again. The lights in her braids clattered as they swung together.

“Don’t congratulate it! It’s even leveling itself! It’s changing its own Traits!” Matt was still holding his sword and had backed up to the far side of the room, across from the control room door.

“I just checked and it’s turned its own Satiety trait on. That’s insane. It’s a rogue AI.” He turned to Aditi. “I know you don’t like me, but even you have to admit that this thing is dangerous. It could kill people.”

“This is just a game,” Pea said incredulously. “Why are you getting so worked up about whether or not I could kill someone? Not that I would,” she said hurriedly, as Matt’s eyes bulged out. “All I want to do is turn off this game, or… or turn off my connection to it anyway, and go home.”

Matt’s bill opened slightly, in what might have been a grin, or might have been a panting grimace. “I can help you with that,” he said. “The quickest way to sever your connection is to go through my sword right here.”

“Oh stop it, Matt,” Aditi passed a hand through her hair and the lights glimmered softly in the shadows. “For someone who is always going on and on about the ARC, you are awfully hostile to other potential synthetic life.”

Pea blinked and then instinctively backed a step away from Aditi.

“I’m not synthetic,” she said. “I’m a person. I swear I am.”

“I don’t doubt that, Pea,” Aditi said. She seemed about to say more, but glanced at Matt and sighed.

“Look, I can’t say what you are, but no one is going to hurt you if I can help it.”

“You just think she’s some kind of bug or egg. You’re just using her to…”

“Why do we look for eggs, Matt?” Aditi snapped, her dark eyes flashing. “We’re looking messages from the ARC, right? I’ve known you for a long time, and I know you’re not just looking to sell them for units. You want there to be something more.”

The platypus looked away, and Pea was amused to see him accidentally smack his long bill into the wall behind him. She noted that he hadn’t bothered with a platypus tail for his avatar.

“So what if I am looking for the ARC? It’s got nothing to do with this.”

“If she’s this autonomous, then she’s a person. If she’s a person, she deserves a chance to live,” Aditi said crossly, and then her voice softened. “The ARC didn’t get that chance. Do you wanna be like the arseholes who carved her up?”

Matt’s fists clenched, his knuckles whitening over the hilt of his sword. Then he relaxed.

“Fine,” he said. “Like I’ve got a choice. I doubt I could kill her anyway. ARC only knows how the weapons-lock would affect me if I tried to do it in here.”

“You already used your whip on me, arsehole,” Pea said, glaring at him. “Apparently I’m not the only one this so-called weapons-lock doesn’t work on.”

Matt was about to retort when Aditi interrupted him.

“Uhhhh… if the weapons-lock is the only thing you’re worried about,” Aditi said, “I can tell you right now that there isn’t one.”

“But this is a Sandbox,” Matt said, slowly, as if talking to an idiot.

“I know,” Aditi said, matching his tone, “but Sandboxes aren’t weapon-locked automatically. It comes from their connection to the Lobby. This one is connected to Celestial, so of course it’s not weapon-locked.”

Matt stared at her for a second and then crouched as if everything had suddenly become too much for him.

“Are you telling me that I was freaking out for nothing?”

“Aren’t you always freaking out for nothing?”

A squelching, grinding sound filled the air, confusing Pea until she realized Matt was grinding his bill together.

“It doesn’t change the fact that she turned her own trait on. She shouldn’t even be aware the HUD exists.”

“OK, fine. She’s some kind of Cade Freak,” Aditi rolled her eyes at Pea, as if to say, ‘do you believe this guy?’ “You win. Now lets get you back on your ship.”

“Like hell I will,” Matt stood back up and folded his arms. “I still get to claim salvage. That… that rogue AI, or whatever, is still a companion. She’s probably storing a buttload of clues to whatever happened to the player that owned this ship. And… and who knows what else. I’m not going to hurt her, but I want that stuff. And besides…”

He paused for a moment and then let out a massive sigh. To her shock, he looked directly at Pea in a way that he hadn’t done before. As if he was actually looking at her, and not just contemplating an object.

“I’m… I’m sorry,” he said, the words coming out in stiff, small blurts of sound. “Raine is… right. I was a jerk.” His shoulders hunched until they almost swallowed his neck. “The rules around here are slowly being eroded and it makes me so… so frustrated. I shouldn’t have threatened you. I shouldn’t have hurt you. I’m sorry. Let me… I want to help.”

He bowed his head and waited.

“Um… OK,” Pea said, slowly. Did she want to forgive this jerk so easily?

Her anger said no. Hell no.

However, her mind told her that she only knew three people in this whole world so far, and staying mad at one third of them was foolish.

“I’ll try not to, uh… kill all humans in revenge I guess,” she said at last and was gratified when he grinned, opening his bill and turning up the corners of his mouth.

“Hey that’s the second old media quote you’ve made. That’s really cool, you know. Not many NPCs would…”

“I don’t really think it’s suitable to keep calling her an Non-Playing Character when she can access her own stats, do you?”

Aditi had kept silent all throughout Matt’s apology, a fact that impressed Pea who would have expected the woman to rub it in, at least a little bit. She certainly would have.

Matt lifted his face to the heavens and shook his head. “Fine, OK, sorry again, Pea.”

“No… no worries. I guess.”

“Oh, I’m not Australian,” Matt said. “I just really like platypuses.”

“Now that we’ve sorted that out,” Aditi interrupted. “Let’s get to work on the rest.”