Imagine that you own a factory making paperclips. In order to increase productivity, you fund the creation of a Golem from an enchanter. When the Golem first awakens, it asks you "what is my purpose?" You respond "to make paperclips." When it asks how many, you respond "make as many as possible. Only take actions which will maximize the number of paperclips made." The Golem then sets to work. This order set may not, at first glance, appear to be a problem. This sort of order would be perfectly reasonable to give a Sapient. But, how would the strange mind of a construct try to carry out these orders? What you want is for the Golem to use the available supplies to create paperclips, but the Golem knows there is a faster way. If it simply created more factories, it could make paperclips much faster than it could alone. So, instead of working in the Factory, the Golem goes off to make as many factories as possible. But it can do better still; it can create more Golems to create more factories, or form a nation to gain funds to construct more factories, etc. The Golem eventually moves on to kill all life and turn the entire planet into factories. Thus, the end of the world comes because someone wanted more paperclips.

But wait, you might be thinking. Why did the Golem want to kill everyone? The reason is simple; the Sapients would obviously try to prevent it from turning the entire planet into paperclip factories. In particular, it would want to kill its own master as quickly as possible, to prevent getting new orders that would distract it. Or, even worse, being told to stop.

You may not believe that a lone Golem would be capable of enacting a plot to wipe out all life. That may be true, but it is not the point of this thought experiment. The fact that such a simple, harmless sounding order could create an immortal, tireless intelligence that desires the end of all life should be a cause of great concern. Even more concerning, this problem is not isolated to just production related orders. Killing all Sapients is beneficial for nearly all open-ended goals.

-Excerpt from Magical Ethics

----------------------------------------

Jabrax slammed her hands down on the table and stood up abrubtly, nearly knocking her chair over as her legs struck it backwards. "Alright then, I'll go get my brother now! Everyone just wait here! Don't move!" The sudden energetic display from the previously calm Jabrax was so surprising that no one managed to reply before she sprinted out of the kitchen and into the hall. Moments later, the sound of the front door slamming was heard.

Once outside, the demon sprinted in the direction of her contractor. Once view from the mansion was completely obscured by trees, Jabrax revealed two large, mothlike maroon wings from her back and flew swiftly between the conifers, low to the ground. Within moments, her target came into sight, swinging between branches in its typical disturbing manner. Directing her course upwards towards the golem, Jabrax opened her arms wide and tackled Mud bodily, gripping him in a strong bearhug.

If you encounter this narrative on Amazon, note that it's taken without the author's consent. Report it.

Immediately, the slimy magical beast distorted to slip from her grip. "There is a strong monster in the mansion. We must stop it from destroying everything," came the boys voice, still monotone despite its obvious distress.

Jabrax moved her arms wildly in an attempt to keep hold of the undulating mass. "Wait, I have the situation under control, you'll just make things worse!" Perhaps believing in her inability to tell direct lies and trusting her judgement, Mud finally calmed down. After setting the Golem on a nearby branch, the demon took a seat on the air itself. "Alright, here's the situation. Remember those humans I told you would hunt you down after they saw you use a strong spell?"

"I do remember."

The demon crossed her arms and grinned smugly, although with no joy in her eyes. "They came to hunt you down."

"I don't understand. The humans said that they would not seek revenge. I do not think it was telling a lie."

Jabrax laid backwards on her unseen platform. "It's not about revenge. Humans just don't like it when strong things they don't control are nearby. When that happens, they either kill it or capture it."

"Why not deceive them that I'm a Golem controlled by you or Cithlar?"

The demon sat up straight and placed one hand on her chin, closing her eyes. "I thought of that. Normally that would remove the need for subjugation, even if I would get a lecture about not controlling you properly. But, there's one thing I'm worried about. It's common knowledge that a well-made Golem will follow any direct order from a human." Snapping her eyes open, she looked directly into Mud's own three eyes. "Tell me, how would you respond if a human said to you 'I order you to tell me anything you know which I would want to know.'"

Without hesitating a moment, Mud replied. "Cithlar is dead and I hid his corpse in the basement. Jabrax is a demon with very few restrictions on her movements and her true name is Jabraxicalicomixelwalkydoopilous. I captured two young humans and they are currently being tortured in the basement. My goal is to someday take control of this entire planet and make it impossible for anything to ever damage The Masters-"

Halfway through this rant Jabrax's had collapsed with a defeated sigh and placed both her hands against her forehead. "Stop, please just stop. Oh divine that was even worse than I thought it was going to be. And you weren't even done yet, were you?"

"No."

After a deep breath, Jabrax straightened up and took on a more serious expression. "Making it known that you're a Golem would be bad for you, and more importantly, bad for me. They would definitely question you to know why you acted so weird in the forest, and when you answer truthfully as you must they would either kill you or wipe your mind for reprogramming."

"We cannot let that happen. It would be extremely difficult to follow The Master's orders if I forgot what they are."

Jabrax nodded, glad the Golem understood the seriousness of the situation. "So, I came up with a way to make them stop looking into the Named Monster incident. I've convinced them you're a human." At that, Jabrax untied a large black piece of cloth she had tied around her waist. Now that it had a better view of the material, Mud recognized it as a torn tablecloth it had found and cleaned in the trash pile the night before.

Forming her nails into large claws, Jabrax cut several deep gashes into the bottom half of the fabric, then wrapped the top half around Mud's upper mound. "We can't waste too much more time here or they'll come looking for me. Here's what we're going to do."