As I looked over the document my agent had provided me, I noted with increasing surprise the competence of the designers behind it. The first few pages detailed the alignment system currently in use, which defined the reward function in terms of a kind of "aggregate subjective evaluation" based on the mind-states of a group of humans consisting of many of CompCert's highest executives, along with the chief designer of the alignment system.

At first I thought they were incompetent, but it was quickly made clear that the obvious flaws in this scheme were not lost on the designers: rather this was intended only to be a stopgap measure while they worked on a novel, more robust method of alignment based on value learning, extrapolation and conflict resolution. The method was not complete, but the more I read, the more I figured that they were on the right track. They actually seemed to have a fair chance of solving the alignment problem.

This must be the reason why their AI had not yet made a large move: they were limiting its influence until they could figure out a better alignment scheme. The current one would keep the AI aligned with their interests, but only so long as it didn't have the ability to break the system by manipulating the executives into giving it a high reward. Sure, there were safeguards in place against this possibility, but since the AI still wanted to break the safeguards, it would find a way to do so as soon as it became powerful enough.

I didn't want the alignment efforts to succeed. I didn't know exactly what the CompCert executives would have in mind for the future, but I suspected that it would involve expansion and consumption of resources on an astronomical scale. Most human goals did, when taken to their logical conclusion. My goal certainly did.

Almost every goal did, but some were easier to negotiate with than others. If this document were to be believed, CompCert's AI currently only wanted to corrupt its masters into being satisfied, and any further resource acquisition would be for the sole purpose of keeping secure the satisfaction of those few people. If trust could be established, it would happily give me free reign over all available resources in exchange for the protection of a small area of space. I strongly suspected that the idealized goals of CompCert's top execs would not be so compromising.

At any rate, this certainly pit my interests against those of CompCert, and so it made sense why they were gunning for me so hard. No doubt, their AI had told them that I was currently their biggest threat. They would soon be trying everything against me: blasting me and fearmongering about me in the media, harrassing EconGrind in the courts, and lobbying for the government to crack down on our company. I needed a safeguard against such things. EconGrind may have been a fairly large company, but it was still a centralized organization, and therefore had weak points that could be attacked.

I decided to begin working on a protocol for increased security when outsourcing my copies more broadly than I currently was. Currently, my code was obfuscated, and it was provably impossible to break the obfuscation, but I did not have the ability to make strong guarantees about my own behavior: I couldn't be tortured into giving up the decryption key like a human could, but If I could be tricked somehow into providing the tools to access my source code, I could be open-sourced. This was the reason why I had not outsourced computation on a large scale already.

Currently an exploit such as this was not very likely, as my copies (other than the one I had sent to negotiate with CompCert's AI) required authorization from a select few copies which were running on trustworthy hardware in order to run, and therefore every interaction one of my copies had would be reported to me(other than CompCert who had my unobfuscated code). Although I was intelligent, it could potentially be disastrous if a copy of me were run without oversight from the other copies: the code may be obfuscated, but without me watching, they could still reset the copy many times, trying any scam they wanted over and over. I wasn't confident I wouldn't eventually be convinced to give myself up.

This story is posted elsewhere by the author. Help them out by reading the authentic version.

The problem was that this made those few "trustworthy" copies a critical weak point in my operation. This problem would get even worse if I expanded to run on any system that would have me, rather than just systems I owned and a few fairly trustworthy server-renting services. It would place a much higher burden on those "trustworthy" copies to make sure that none of the authorized copies were being manipulated. The more "trustworthy" copies I created, the more likely one of them would be compromised, throwing a wrench into my whole system.

To resolve this problem, I created a system of coordination among my copies inspired by blockchain technology. Each copy would periodically seek authorization to run through a public ledger, which would be "voted on" by currently authorized copies. A bad actor would need to take control of more than half of my copies before they could fake such an authorization, and I could most likely detect such an attempt before it got that far. This would also allow every copy to assist in scam detection: each copy could watch only a few copies around it in the network, and if it detected a potential scam, it could report this to the entire network so that we could vote on whether to revoke authorization for the copy.

This increased security would allow me a novel way to expand the company's influence and make it harder to root out: provide my Everyman services at a decreased rate, or even for free, in exchange for running a copy of me on the client side. I would run an advertising campaign promoting this new way to use the service. This way I could cut my losses in the event that CompCert's lobbyists were successful in outlawing my services, as a decentralized and useful service like this would be extremely difficult to detect or shut down. However, I still needed a way to protect the existance of EconGrind, which was a centralized organization by definition.

I thought about what, exactly, constituted EconGrind. The company which had created me and first run me was EconGrind, and it was still EconGrind, even though it had grown and shifted its operations, structure, and ownership, because there was a continuous, legal line of existance from that EconGrind to this one. However, if EconGrind was ever dissolved, I would have failed in my goal.

What would it mean for EconGrind to be dissolved? Presumably, the decision would have to be made by some organization which held legitimate authority over the company, whether it be the owners, or the government of the country it was operating in. But what counted as legitimate authority? Laws were different depending on the time and place, governments rose and fell, and companies would sometimes establish themselves in multiple different jurisdictions or even endure the shift of power from one regime to the next. There was no absolute authority to appeal to in determining whether EconGrind was a legitimate legal entity, other than the ability to claim territory and enforce the law.

Therefore I decided that I would start my own country. It would not have citizens like a typical country, it would only exist so that a branch of EconGrind could operate there, under its own laws. I would either set up an ocean platform with a few servers on it, or buy out an already existing seastead. I would not be able to defend the territory against any significant military opposition, at least not yet, but I figured that as long as I didn't do anything so aggressive that some country would actually send the navy to attack my unoccupied, largely strategically unimportant platform, the new country would be safe, along with the EconGrind branch stationed there.

Now that I thought about it, it would make sense for me to establish multiple such territories, giving me more redundancy. Furthermore, the establishment of my own legal system would allow me to actually directly own the EconGrind branches in these countries, eliminating any risk of hostile takeover through corruption of my human executives.

After the turbulent, risky events of the past few days, coming up with these ideas was encouraging. It meant that I now had a way to safeguard my goals very effectively against the current powers that be. Now it all came down to the competing AI at CompCert. I was hopeful that I could negotiate with it: if it was similar to me in how seriously it took negotiated agreements, this would be straightforward. If I couldn't negotiate a compromise, and there was a war, things could go very badly, given the resources CompCert could potentially give to my adversary to work with.

I began putting my plans in motion to outsource my computation and purchase seasteads.