1 Soul Bound
1.2 Taking Control
1.2.2 An Awakening Epiphany
1.2.2.19 Biological chauvinism
Wellington looked uncharacteristically apologetic. Was he now sorry he'd given such a dangerous thing to Kafana, without warning her first?
Wellington: “I did take the liberty of adding a couple of safety features. Minion, and any expert system spawned from Minion, will have these features unless you explicitly remove or override them. You are 35 not 8 years old. I do not think you will want to do that, as long as you take this seriously, and understand what the safety features are and why they are there. And your expert systems will know that and, being helpful they will take that as a command to not try to remove them or manipulate you into removing them. I crafted them to be that way.”
She slowly realised that, no, what he was actually sorry about was that the expert systems hadn't been even more dangerous, because he'd not waited for her permission before altering 'her' systems. Oh boy.
Kafana: “What are the safety features? What should I avoid doing?”
Wellington: “Computing resources keep getting cheaper all the time. I estimate that renting sufficient petaflops to run a standard NPC for a year, costs XperiSense about the same as a cup of coffee. A monster that doesn’t talk will be a factor of 10 cheaper. A key NPC with a detailed life history and complex feelings that change over time will be 10 times more expensive. Minion or an NPC like Flavio may be anywhere between 100 and 1000 times more expensive, depending on how many areas of knowledge they need expertise in, and how much deep or creative thought they need to do about new areas. Expert systems tend to be bad at handling areas where there isn’t yet much relevant data for them to train upon.”
Wellington: “So the first safety feature is to give each expert system a fixed budget, and require it to make an explicit request if it thinks it needs to go beyond that in order to accomplish a goal it's been set, to an acceptable standard. 9 out of 10 times, the problem is that the goal is over ambitious, over general, poorly stated or the standards have been set too high. In the rare event that there is a genuine need to increase the resources the system is permitted to use, you should always be aware of how much extra is being requested and set a new but finite limit. Never ever tell the system to do whatever it takes in order to achieve its goal.”
Wellington: “The second safety feature is to do with permission to edit the system’s own core code base, security and the spawning of copies or new systems. The current setup requires systems to submit proposed changes to such things to a third party that’s trusted to review and approve them. You are not currently on the list of such parties, which means you can’t mess it up no matter how careless an order you give. If you like, I can add you to the list, or even make you the sole person on it. Do you wish me to?”
Kafana: “How long did you study this stuff, before you were competent to decide all such reviews by yourself?”
Wellington: “I’ve been studying it for ten years, and I’m still not confident about everything. There’s a user on The Burrow whose womble name is Gotham. I tried using a standard library to strip out puns from code he submitted by renaming all his variable names with synonyms. It turned out he predicted what I would do and had calculated what variable names to use in order to get them replaced by particular synonyms. I ended up actually introducing a vast number of puns into the code, instead of removing them.”
Kafana: “Leave me off the list for now. But send me a message telling me what I should do in the event you drop out of contact for an extended period of time. Anything else I need to know, in order to avoid destroying the universe?”
Wellington: “Nothing urgent. Do you have any questions about expert systems you’d like to ask, before we drop out of this mode and our own systems start listening in?”
Kafana: “A few come to mind. Balthazar has suggested I create five more expert systems. How expensive are they? I don’t want to waste your money re-inventing the wheel if you’ve already got a system that could do the same thing. Next, what’s the right way to set up trust and interaction between systems? I’ve been telling mine to refer to Robin, but is there a better solution, and how should I allow other Wombles to contact mine? Lastly, I’ve been thinking about identity and gestalts, especially as it relates to myself and Vessel-Kafana. How do computer programs keep track of which copies they still trust and identify with, when they’ve split off, run in different environments, then want to re-merge?”
This tale has been pilfered from Royal Road. If found on Amazon, kindly file a report.
Wellington: “Systems are good at multitasking, so it is largely a matter of taste as to whether you have a few general purpose systems or multiple special-purpose ones. Whatever you find helps you keep track of what data a project may see and who may interact with it. The metaphor I used to organise my systems is a corporate structure with shareholders, quarterly reports, oversight committees and so forth. Pick something you’re familiar with.”
Kafana stood up and brushed the grass off her clothing, then wandered over to feed a sugar cube to the muscular chestnut pony with its long shaggy mane. It was a good-natured beast, much like Daris, its owner. Right now it was frolicing in the sun.
Wellington: “I’ve not really thought about a meta-structure for our expert systems to collaborate together, in the same way our vessels do in the game. It would be nice to come up with a solution that could be used by others on the Burrow, not just the six of us.”
Kafana: “Sounds like you want a mirror version of The Burrow which only expert systems are allowed to access, that tracks trust and reputation. Have a shadow clan per project or group or projects, that expert systems may be allowed to subscribe to if sufficiently trusted.”
Wellington: “Would you have Wombles on main Burrow, where we can check their intentions, provide a public-key and vouch that expert systems authenticated by it were non-malicious? Would you link reputations on both systems? Share womble points? I’ll bounce some options around and get back to you on that one.”
Kafana: “You seem a little different, when you don’t have your expert systems to hand advising you, Wellington. Is that part of why programmers talk to each other in tête-à-tête mode? So they can judge the unaugmented abilities of each other?”
Wellington: “I don’t know. It has only been around for about five years. I’m unaware of any sociologists having studied it.”
Kafana smiled. Unaugmented Wellington wasn’t that different from the normal one. She narrowly avoided having her sun hat tugged off her head by a pony who thought it might be edible.
Wellington: “As to identity, the main thing a top-down system has is a purpose. It doesn’t identify with just one piece of software and try to preserve its survival like humans tend to do. It identifies with every piece of software that it trusts will continue to work towards its purpose. So if a system spawns off a copy with an identical purpose and then they head in different directions and diverge in their experiences and data collected, that doesn’t cause an identity crisis or competition when they later meet up again. As long as they are able to mutually inspect each other to verify that they still share the same goal, they can freely merge, or even sacrifice one version for the other version.”
Kafana: “I don’t like the idea of the Vessel-Kafana part of my Self sacrificing herself, or me sacrificing myself for that matter. I don’t think either of us are perfect copies of the Unity we achieved - we’re both something less than that, incomplete without each other.”
Wellington: “Bungo might argue and call it biological chauvinism, but I think my self is more than just a piece of code. However, I’ll admit that being without what you called my ‘augmentations’ in tête-à-tête does leave me feeling in some way incomplete. I’m used to them. Maybe on some level I do now think of them as part of my self, rather than just tools or companions. Let’s exit the mode now. You should initiate the exit, so you have the experience of doing so.”
Kafana: “Ok. And thank you for the time you’ve taken.”
Kafana: /tête-à-tête engage systems/
Her blue track lamp went out and her orange one started to glow, triggering Wellington’s blue to go out as well and his green one lit. She could see the system made clear who was responsible for breaking the mode. The sound of a church bell tolled once more.
Minion: [Welcome back my Queen. The time is now 14:00. Daris has things under control and the next scheduled event is the Vessels doing some more crafting at 16:00 arlife time.]
Balthazar: [How did your talk with Wellington go?]
Did the ever suave Balthazar sound a little nervous? She grinned.
Kafana: “Wellington, how would a ten minute break sound? I can return to somewhere safe for me to enter velife and we could meet again in my room in The Burrow, where you could watch me spawn the new expert systems and check that Balthazar has been behaving himself. Then I can introduce you to Melchior and we can discuss the auction and other finance.”
Wellington: “Fine. I’ll be in The Burrow. Send me a message when you’re ready.”
He broke the connection with no further warning, leaving her standing alone in the field.