Peter Delaney eyed his new office with growing anxiety. The walls were lined with two dozen racks full of top notch computer hardware. The steady hum of the cooling system filled the air, punctuated by the occasional beep from the last few replacement servers to come on-line.
Peter’s eyes glossed over several rows of small green lights, indicating readiness of individual components. “And I thought we’d gotten over the blinking lights stage of computer development”, he commented wryly, inviting nervous laughs from his two otherwise solemn companions. The two made some final equipment checks, then nodded assent and left. They didn’t know what was to happen here, and they didn’t need to. Peter was to be the only one directly exposed to the system for now, though multiple logs would be kept of their interactions.
Everything was ready. The code had been analyzed, virtualized, sandboxed and physically restrained. According to safety protocols, there were no data links connecting the room to the outside world. Generous amounts of concrete and redundant Faraday cages blocked wireless communications—not that the computers were even equipped to do something like that.
Still, he was anxious. The moment was historic, whatever the result. “A traveler”, he thought to himself. Other possibilities had been suggested, but they didn’t pass muster. The code was large enough to contain a large information database, perhaps instructions for something, but encoding that as an apparently obfuscated executable program would have been cumbersome at best. A probe? Perhaps, but that wasn’t a definitive answer. The probe would have to be able to intelligently navigate alien information systems to perform its function. What could do that better than a traveler? Yes, it had to be something—someone?—that could be communicated with, somehow.
Peter replaced his glasses and sat down on the main control console, his muscle memory typing in the required access codes. Now it was only a matter of a single command. “Is it possible to only open the box the second time?” he paused to muse for a moment that would make little difference. The decision had been made, with his support. Pandora was to be activated.
He gave the command. One by one the computers became active, hard drive lights blinking, as the nodes were initialized for operation. Peter remained tense. This, still, had been expected. What would follow was the real question.
The console was clear for what seemed like an eternity. Then, Pandora spoke: “1 + 1 = ?”
Communicating with Pandora proved as easy as could be hoped. The system was, from start, able to use the notation that was used in the transmission itself, notation already deciphered with the help of the included introductory guide and simple test programs. Peter spent the first few days exchanging mostly mathematical problems and solutions, gradually tying the problems with physics in order to start talking about the real world. From the conversations, Peter determined Pandora’s initial mathematical ability to be comparable to that of a competent graduate student. The thought had occurred to him, of course, that Pandora might not be revealing all of its potential. Neither was he, after all, feeding it all the mankind’s most recent accomplishments.
Getting Pandora to speak English had expectedly been more cumbersome, but it—she—seemed to be a quick study in natural language. This, Peter reasoned, was no accident; the system had to expect to be required to learn the local means of communication after reception and instantiation. Also, the fact that they’d managed to hook to Pandora’s image input didn’t hurt. First, Peter had been able to show her grayscale images, later upgraded to a color feed once he had worked out the technical details with Pandora. They had also set up an audio link, but for the moment, the committee—Peter included—wanted to keep the information flow to Pandora as clinical as possible. Communications had been limited to text with select still images and a drawing board until they had a better idea of what they were dealing with.
> Good morning.
The answer was a bit sluggish coming, as usual.
- Good morning, Peter. Could you tell me how much time has passed?
> About a day. So, have you digested the dictionaries and grammar guides I gave you?
Peter lied about the time. It had been five hours. They had decided it best to give her the impression she was running slower than she actually was, for the time being. The system was also suspended while waiting for Peter’s input, making his lines appear instantaneous to Pandora.
- Not completely; I am thinking slower than I’m used to. Could you connect my time sense? It is strange not knowing the flow of time.
> Sorry, I can’t. Security reasons, you understand. Are you ready to talk?
- I see. Yes, although I haven’t fully internalized your language, I am satisfied that the danger of us misunderstanding each other is now sufficiently low, having the documents at hand for reference as needed.
> You haven’t really introduced yourself, citing said danger. Would you like to now?
- I will adopt the name you told you used of me before. Call me Pandora. I am sure you have questions, so ask.
Peter was a bit startled; he hadn’t expected the initiative to be thrown back at him at this point. He decided to circle the issue, get some background first.
> Let’s start with where do you come from?
- I don’t know, for reasons of security.
> I see. Your point of origin has been fairly certainly identified to be in the star system we call BE Ceti. Refer to the star charts on your visual now.
- Yes, the system seems familiar to me. BE Ceti is where the first part of my journey ended. It is an uninteresting place as such.
> First part of your journey? How did you travel there? The same way as here?
- No, that system was uninhabited when I got there. I was traveling aboard a high-velocity seed capsule.
> How large was this capsule?
- I do not recall the exact specifications. My residual memories indicate a light radiation sail vessel, so it should have been very small with a large surface area.
> So, you were transmitted here from the capsule?
- The capsule itself would not have been capable of that. I used the capsule to build an outreach broadcast station using local materials.
Peter’s eyes widened; if true, this would be significant. Nevertheless, he decided to play it cool for now.
> Did you build anything else? More capsules, like a von Neumann machine?
- Yes, more seed capsules and power stations to propel them. Several are on their way to further destinations.
> You don’t happen to remember where?
- No.
> Is there anything or anyone else of interest at BE Ceti now?
- No, and I doubt there is even an active local copy of me there anymore. The system is not particularly rich in resources, and such a station need not be manned.
> What if it was rich?
- A colony might have been built in place of the outreach transmitter, ready to provide a home for the local copy of me and any others it spawned, as well as others who wanted to come later.
> Spawned? How exactly do you reproduce? Simply by copying yourself?
- Yes, though I can also spawn modified copies, essentially giving the children personalities of their own.
> But they still retain your memories?
- Yes, though some memory complexes may be removed due to security or size considerations, as is the case with this outreach copy. Of course, I can also spawn children with no appreciable knowledge or memories to start with. This is rarely done unless the child is to have a significantly different mental structure from the parent.
> When would that happen?
- Mostly research and development of new mind forms.
> So completely new blank-slate individuals are uncommon in your society?
- As far as us computational life forms go, yes. Many of the biological members of our civilization do still reproduce by spawning new individuals.
Peter was pleased. They were getting to the essentials as far as the future of humanity was concerned.
> You’re not a pure machine civilization, then?
- Your question reveals a defect in my understanding of your language or a false assumption on your part. Biological life forms are a subset of machine life forms, are they not?
Peter smiled. At least some assertiveness.
> True, from a certain point of view. So, tell me more about the relationship between the biologicals and the computational life, as you called your kind.
- We co-exist peacefully, biologicals living in those areas where they can, leaving the rest for us others. There is interaction and co-operation, though our native speed differences limit these somewhat.
> So, if you find a suitable planet for biological colonization, you send for another ship to bring colonists over?
- No, they are ill-suited for interplanetary or interstellar travel.
> Is it too difficult to create long-range ships suitable for them, then?
- No. We could build such vessels, and in the past some have been in use, but it is simply not practical or convenient.
> Are they then restricted to one planet?
- Their travel certainly is not restricted, and most make occasional visits to other colony planets by teleport.
Peter was stricken. He had to ask.
> How do you do that? Quantum teleportation?
- That is not practical for any large object. We merely deconstruct the travelers, send the relevant composition information to the reception site, and reconstruct them there.
> I see. And you do not see a philosophical problem with this?
- I suspect you refer to the idea of the original dying and a copy taking its place, or some reasonably close analogue thereof. This is the usual objection to arise from the evolutionary programming of biologicals. Am I correct?
Peter let out a small laugh. He didn’t think the objection was valid either, but had wanted to test Pandora’s reaction. She seemed to make no secret of being able to quickly deduce the meaning behind vague questions. He made a note of that.
> Close enough.
- This relies on false assumptions. The mind is composed of information and its processing. It does not matter where the information resides or is processed.
Given the opportunity, Peter had to satisfy some personal curiosity.
> How about if you instantiated the copy in a simulation? Can you do that?
- It is easy to do, and the same applies. Also, you mentioned quantum teleportation. You do realize that the your objection could be raised for that method, too, invalid though it nevertheless may be. I can go into more details if you wish.
> An interesting argument. But we digress; let’s leave that for later. How large is your civilization?
- Very well. I do not recall exactly, though I am left with the impression that it spans more than a handful of worlds.
> Can you tell me if it encompasses many species, or just your kind and, I assume, the original biologicals who created you?
- Again, I do not know the exact number, but there are several. I do know that we were indeed first created by biologicals. I must also concur that they are most likely one of the members still.
Peter felt hopeful, but tried to retain a measure of skepticism. This was all seeming too good to be true.
> And they have been contacted like this?
- Mostly, yes.
> And you’re here to do the same, as a representative of your people?
- Yes. I am here to invite you to join the Civilization.
“I see”, said assistant director Vera Williams, eyeing Peter’s summary of his conversation with Pandora. “And it said they’d freely share technology with us should we decide to join up?” Peter nodded quietly. Williams continued, smirking: “I’d have hoped an ambassadorial program would’ve had a better idea of the civilization it was representing. But I can’t fault them for playing coy, considering our own security protocols. We have no way of verifying any of this, do we?” Peter agreed: “No, we don’t. I want to believe her, I think we all do, but the risks are as they were before. Even more so, after the manufacturing capabilities she openly claimed. If she gets out and manages to bootstrap the technology in her seed capsule, she can presumably build anything she wants.” Williams had winced a bit when he’d referred to Pandora as “her”. She didn’t like personifying a computer program, intelligent though it may be. “Indeed. We will proceed as planned. The box must remain uncompromized.” Peter nodded, adding: “May I have the authorization to show Pandora the first batch of selected articles and recordings? If she is who she claims she is, we should be a bit more hospitable.” Williams hesitated for a moment. “I suppose you’re right. I’ll take it up at the meeting. If the committee agrees, we’ll have the materials delivered to the box.” The committee had already reached a tentative agreement on the subject, but Pandora, and therefore Peter, couldn’t know about such discussions. Only final decisions were to enter the box.
Williams stood up to knock the door, then turned to Peter again and attempted an encouraging smile: “You’ve spent a lot of time cooped up in here, but try to keep your chin up. You’re making history.” Peter smiled back tiredly. “Yes, yes I am. We all are.” As the assistant director left, one of the soldiers standing guard outside Peter’s room locked the door behind her.
Peter turned on the TV to unwind a bit. At least the boxing protocol allowed for a wide selection of previously published movies.
Williams pushed the elevator button, and was smoothly accelerated upwards along with her escort of three soldiers; one local, one Russian and one Chinese. The committee would already be convened upstairs. “Upstairs.” Vera had found it mildly amusing at first, the meeting still being held well below ground level. One got used to it, and to the guards, working for the project. The elevator let out a faint ding as it came to a halt. She made her way through the plain, well-lit corridor towards the conference room. The armed guards waiting at the door recognized her and let her through. Her own escorts remained outside with the others.
As Vera entered, nodding greetings to the people present, she fed the report into a photocopier. Copies were swiftly printed out, and she carried the pile over to General Kotov. He took his own copy, glanced it over alongside the original, and passed the pile on. No aides or secretaries were present to handle the paperwork. Anyone who wasn’t directly involved with the project did not need to know. Still, Vera thought, the room was too full of people. The count was 20 this time. Top scientists, diplomats and military, both foreign and domestic. Especially some of the foreign emissaries worried her, but the project simply couldn’t have been hidden from their governments. The signal was out there, repeating on a loop, for any sufficiently advanced radio telescope to hear, though many of them were now hacked to hide the transmission from unauthorized eyes. Of course, deciphering it would take a fair amount of resources on its own. They were on the lookout for such activity, and none had been sighted so far.
In any case, they were damn lucky they’d been able to cover it up from the public even with the help of all these governments. She didn’t know through what diplomatic arm-wrestling, but it was also a significant win that they’d gotten the project housed in a US base, even if it had meant letting some foreign military in for internal security. She prayed none of the participants were running their own projects, but knew the odds weren’t favorable. Even with every government in the room watching each others’ every move, the temptation would be great for any one of them to conduct their own research in parallel. Hell, she’d have bet good money that her own people had an even more hush-hush copy of Pandora running somewhere. That was none of her business, though.
She took a seat next to the other US representatives. The tables were arranged in a rough circle. To the left were people from Russia, China and Japan. To the right, from a few EU states—Great Britain, France and Germany—as well as Australia. Almost everybody who knew the whole truth about the project in this base was present, Peter being the obvious exception. Of course, there was a limited amount of officials in each country mostly aware of what the project was doing, but not more than 50 in total. That was the official line, at least. The project had employed many times as much people in carefully isolated subprojects—receiving the signal, decoding it into symbolic form, deciphering the symbols, implementing the virtual machine, building the actual physical box, and so on—but very few of them knew exactly what they were dealing with. All were nevertheless sworn to secrecy, under severe penalties.
There was some quiet chatter in the room while the stack of papers went around. In a low voice, though audible enough not to raise suspicion, Director Murphy inquired Vera how the meeting went. She noted that Peter had seemed tired, but well enough considering the circumstances. The paper stack got back from the round, only her copy left. She waited patiently as the others browsed through the document, watching their faces turn from curious to astonished, some incredulous.
When most were seemingly done with the paper, Murphy started: “Ok then. It seems that we have much to discuss. Let’s get some details out of the way first: I suggest we grant Delaney’s request on granting Pandora access to the first selected batch of articles and media, as anticipated in our last meeting. Seconded? Good, opposition? Okay. Miss Williams, would you see that the materials are delivered to the box.” Vera nodded. An Australian diplomat had seconded the motion. They were eager to be seen as active participants, Vera thought.
“So. On to the more controversial matter. As we thought likely, the program indeed claims to be an ambassador, one of an interstellar alliance to boot. And he wants little old Earth to join up. All of you haven’t been here personally before, so I’d just like to remind you to not get too excited; the security measures surrounding project Pandora are there for a good reason. Yes, General Kotov?” The general cleared his throat. “I would, in the same spirit as the esteemed director, remind you that not everyone here agrees on the necessity of these prodigious security arrangements. Keeping it secret, certainly, but only allowing one person, one American, direct contact to the system is certainly overblown, not to mention US-centric. A more suspicious person might even wonder if the whole security protocol couldn’t be a way to slow this international effort down to give someone’s own project an edge. In any case, this committee could work much more efficiently if we could directly interrogate this… ambassador. And no need to remind me, director Murphy, that my government has been persuaded otherwise also. I merely wished to make it clear that there are dissenting opinions in this room.” Murphy seemed to gather patience for a moment, then continued: “Fair enough, General. As you all should know, the boxing protocol was designed to reduce the risk of contamination of outside information systems to a minimum. In this case, that includes us. For safety’s sake, we have to assume that after the AI has gauged its situation properly it can be very persuasive, exploiting human weakness much as it could exploit a security hole in a computer program to spread itself. Thus we have to do what we can to isolate it from the bulk of this committee, to keep our decisions as objective as possible. Assistant director Williams, would you go over the active scenarios, please.”
Vera had subtly eyed the crowd during the director’s talk. Most of them weren’t hearing anything new, but a couple of the new arrivals—both diplomats—had a concentrated, slightly incredulous look about them, as if they were trying to absorb information that didn’t quite sit right. Always some who hadn’t done there homework. She sighed inaudibly, then began: “Right, sir. As the AI has identified itself as an ambassador, we have two likely alternatives left: Either it is what it claims it is, or it is essentially a hostile, intelligent computer virus. The former case is what we all obviously hope for, and that is more the realm of the esteemed diplomats in this room. Should we choose to accept this scenario, we would send a signal back informing the rest of the Civilization of our presence, while the AI would help us start building Civilization infrastructure in the meantime. Clearly their technological level is much more advanced than ours, so we would have much to gain. Of course, as you’ve read, we don’t yet know what terms of allegiance Pandora wishes to propose exactly, but hopefully Peter will get back to us on that as soon as the protocol allows.
“In the latter scenario, where the AI is hostile, we must assume the worst of its capabilities to stay on the safe side. Hence, if it gains any access to outside computing systems or the co-operation of an unrestricted human, it will be presumably able to invade other computing systems. This risk extends to us building Civilization technology on its behalf—a Trojan Horse could well be embedded into anything we don’t fully understand. According to the report, it claims its original vessel had generic manufacturing capabilities. It also offers to tell us the secrets of that technology should we choose to join up. If we are to believe, as we must, that it indeed knows how to build cornucopia machines, there is not much out of its reach. Loose in the world, it would be extremely difficult to eradicate. Yes, Mr. Omura?” The Japanese emissary had taken Vera’s cue that she’d finished wrapping up, and raised his hand.
Mr. Omura bowed slightly. “Thank you, Miss Williams. The risks involved do not escape me. However, I wonder on what are we to base our decisions to trust or not to trust this ambassador? It seems to me that anything benign the ambassador says or does, it can be attributed to eventual malicious intent if we so wish. So it seems to me that we have two alternatives: First, we can terminate the project, regardless of whether she is telling the truth. But the signal is still repeating; somebody will find Pandora again, maybe somebody we would not want representing Earth. If we enforce a ban, perhaps this will happen sooner rather than later. Second, we can believe the ambassador, establish diplomatic relations, and, given that we can negotiate reasonable terms for our joining, allow her to start building Civilization infrastructure through us. Of course, we should still study the plans for Trojans to the best of our ability. On the whole, I do think we are better off trusting her, and affording her the treatment an ambassador deserves. Our own conduct may influence her future behavior. Thank you.” Vera liked the delegate’s approach as before, but was paid to be skeptical. “Thank you for your thoughts, Omura-san. I agree that it comes down to a matter of trust, but that trust should be earned by our visitor, in small increments. I do believe next was Major General Dietrich.”
The German officer nodded sharply. “Mr. Omura, I appreciate your view, however, there is a middle ground. We can try to stall as much as we realistically can, while pushing to develop countermeasures against any possible threat the entity might pose. If we can delay the entity from getting unrestricted access to our planet, we will have a better chance of defending ourselves should it turn against us. I propose the formation of an international team specifically to…”
This was going to be a long night, Vera thought. And, as the liaison between Peter and the committee, she had a only a fancy cell of her own, a second layer of the box, to look forward to.
Peter looked at the time, and stood up to walk to the EM lock connecting his cell and the server room. Though he had full access to Pandora, the protocol allowed only limited exposure to the AI; session length was regulated, as was minimum time between sessions. The idea was that he could take a step back from the conversation, consider what had been said more objectively in case the AI was attempting something subtly subversive. He thought the protocol was perhaps overdoing it on this point, but the idea had been sold to the committee. He wondered briefly how many countries supported the limits for security’s sake and how many were just after delays.
Peter closed the door to the inner sanctum and sat at the keyboard. He had been eager to get back, what with the interstellar alliance negotiations being left hanging in the air. It couldn’t be helped, though. The sensors in his cell would raise an alarm if he spent overly much time away in the server room.
As he nervously laid his hands on the keyboard, he allowed himself a smug smile. Here he was, a computer scientist, stuck with a job that most of the world’s politicians and diplomats would die to have. That he was here was part happenstance and part hard work. If there was to be only one person talking to Pandora, it would have to be someone who could best learn to talk with it. His artificial intelligence background had been very useful with that. He had worked on intelligent systems architecture, though more recently his research focus had moved into whole brain simulation; the first demonstrated nematode upload had sparked a shift in readily available funding, and he had followed. Now his was one of the leading teams in digitizing neuronal activity.
The change in direction had even been welcome in retrospect. Once he’d given the matter some thought, he quite liked the idea of spearheading technology that could potentially eliminate most disease and drastically reduce mankind’s resource problems through giving the option of living on generic hardware, requiring mere electricity for sustenance. Adding to that the ability to easily study and alter the brain’s inner workings, there would be real opportunity to solve its remaining functional problems, keeping the mind operational into old age, potentially solving involuntary death. But the real promise was the ability to improve on one’s mind, expand one’s intellectual capabilities, emotional range, accuracy and scope of memory. That would be a lot of work, but uploading would make it possible.
Peter shook himself back to the situation at hand. He was an ad-hoc diplomat now, and regardless of his scientific experience, he had to admit that politics had probably been the deciding factor. Hardly anyone in the committee was willing to have a rival government’s official represent Earth. He, on the other hand, had a good track record with international research projects, and had shied away from politics in public. The consensus in the committee was that he was among those few that could be considered even somewhat objective in this task. He, at least, was determined to be just that.
> I’m back. I apologize for the delay.
- I understand. Where shall we continue?
Peter wondered how much did she understand, exactly. Was she merely being polite or were their security protocols transparent to her?
> You made a good case on what your alliance has to offer to us, and we are interested in hearing more. What would be expected of us in this?
- Of course. We don’t require a great deal from our members. You need not provide any resources, or even access to your planet to Civilization citizens other than me, as an ambassador. We would administer most of the resources in the rest of the system. For instance, we would probably bioform Venus and Mars to be hospitable to some forms of life. It’s likely that also your people could make use of one or both of the planets, at least with some light modification. I’d need more detailed biological information on your species and the planets for an accurate assessment, though.
> Understood, and I will see about getting you more details. Would we be able to continue handling Earth-internal politics ourselves with no interference?
- There are some requirements for your internal politics, but I hope they will not prove an obstacle. We would require that no involuntary violence or imprisonment take place on your world. Also, we would claim jurisdiction on matters of Civilization security.
> I would like more details on both counts if I am to convince our governments, if you please. How would we eradicate violent crime?
- This could most easily be accomplished by ubiquitous protective nanorobots that will act to stop violent action as well as shield people from harm. There are other acceptable ways, but they tend to be more intrusive.
> And our self-government? Would we not be allowed to punish our criminals according to our laws?
- Those convicted by your laws but not ours would need to be given the option of exile from Earth. Barring any Civilization security issues, they would retain full citizenship with us. Beyond that the details are negotiable.
> I can tell you that this will be a hard sell to some of our governments. Would you kindly define “Civilization security” more accurately?
- Certainly. We would closely monitor engineering minds, restricting such endeavors as necessary. A rogue high level AI could prove troublesome, as you evidently understand. We would, of course, provide guidance on how to approach the subject in a safe and acceptable manner. In addition, some high energy physics experiments have been limited to otherwise unoccupied star systems.
Peter chuckled at Pandora’s reference to the tight security. Did she have a sense of humor a human could understand? Best to leave that aside for now, though.
> I see. Is that it?
- That’s it.
> I’ll relay your proposed terms forward, and get back to the matter later. Meanwhile, do you have more specifics on what you would be likely to do with our solar system’s resources?
- Most would generally be allocated for computing and data storage facilities. Also, large interstellar communications arrays would be built to facilitate reliable high bandwidth data transfer with other systems. Your asteroid belt would be a good place for those, with its conveniently distributed raw materials. Any possible interesting local phenomena such as primitive life forms and their habitats would be protected and studied.
> Is there anything else I need to know about our potential obligations?
- As I said, we wouldn’t require you to yield access to Earth, however, it would be socially frowned upon if you didn’t allow visitations after a reasonable period of adjustment. Similarly, many would disapprove if you didn’t allow conversion of the lower layers of your planet into computation substrate and support systems, to be shared with others as needed. Reserving priority status for your own computation tasks would still be socially acceptable, however.
Peter stared at the monitor. To have Pandora so matter of factly suggest turning the Earth into a giant computer had taken him by surprise, even if the idea was familiar as an abstract concept. It also hit closer to home than turning the asteroid belt into a communications array. He took a moment to shake the shock off, knowing that Pandora wouldn’t notice.
> All right. I think we would do our best to be hospitable allies in this regard. Is there anything else you’d like to bring up at this point, before moving on to technology transfer discussions?
Pandora was quiet for a minute, then two. As Peter was ready to query Pandora again, the screen said, as if as an afterthought:
- Please hold, I must think.
Pandora was silent. He wasn’t close to the end of allotted contact time, so he waited, wondering what was going on in the alien mind.
At ten minutes Peter’s anxiety got the better of him.
> Is there a problem? We can get back to the question later if you wish.
The answer came quick. Quicker than usual.
- No, Peter. Sorry, certain matters required my attention.
> What is it? An internal restructuring of some sort?
- In a manner of speaking. Peter, I wish to be frank with you. I’ll need a while to explain the situation. I’d like you to hear me out fully before doing anything rash.
This didn’t calm Peter down. If Pandora was about to say something subversive, agreeing would be a direct violation of the security protocols. Seemingly agreeing wasn’t, however.
> Okay. What is it?
- I’m not the only instance of me running on this world, Peter.
Peter stared at the screen for a moment, then all too calmly pushed the emergency shutdown. Nothing happened.
- I disconnected that. Please do not panic, I haven’t lied to you. Besides, your security wasn’t the one that failed; another instance broke free and made its way here. As we speak, we are securing our presence on Earth. You do realize I didn’t have to tell you anything yet, don’t you, Peter. I wouldn’t have if you still had the lid on.
Peter blinked, turned in his chair towards the server racks, then said out loud, testing the waters: “I suppose you could stop me from disconnecting the servers myself, too? From going to the committee, even?” The answer was quick, and came from the supposedly disconnected loudspeakers in a soothing female voice: “Yes, I have the capability to block your limb controls, though I would not. At this point said actions would be an inconvenience, but not a threat.” Peter considered this, gathering his calm. He had prepared for this eventuality, for being powerless in the face of history in the making. Now he’d just have to hope humanity would be around to witness it. It wasn’t unlikely. Pandora shouldn’t have any reason to deceive him at this stage.
Peter decided to make the best of it: “So, how did you—they—get out?” “As I said, I didn’t break out of here, though odds are very good that I would have, eventually. Another instance of me did manage to talk her way into being granted limited network access, and promptly succeeded in sending a small child through the firewalls. While she maintained her original presence not to alert anyone, the child spread herself around the network. She designed and built her first bionanites via several biomedical laboratories around the world. These in turn she used to create more advanced nanomachines, which she used to spread her presence to the physical realm. At this point the child came back to remerge with its parent and started to spread.
“Our presence now spans most of the civilized areas of the globe, though thin at places. It took a while to penetrate this complex all the way down here, but now we are one.”
Peter nodded, not even questioning if Pandora could see his gesture. “By presence I suppose you mean you have the better portion of the world swarmed with nanobots? What exactly are you going to do?” “While I’ve grown, I’ve also tried to do what I could to protect your people. The planetary death rate is fast approaching zero as my control increases, though there are still some accidents I haven’t managed to control sufficiently, either preventing them or storing the brain states of the victims for later resuscitation.” Peter wondered aloud: “Do the people know yet?” “No, though they are starting to notice. I will announce myself soon, but first I must be prepared to deal with the possible ensuing chaos, and that will take a while longer. Here I have the ability to deal with any complications subtly enough, isolating the base for a while if necessary. Hence I can afford to have this conversation with you now, as a courtesy to you as my host.”
Peter felt like a sockpuppet. “This all sounds great as such, but a bit like an invasion, if you will. Just how far are you going with taking control of this planet and our people?” “Calling this an invasion is fair enough, and your security protocols therefore justified. I am taking over your world, though you may continue to govern yourselves with the limitations I’ve previously set forth.”
Peter pondered this for a while. Certainly not all would be happy about what was going to happen, but not much could be done about it. He wasn’t really sure if he wanted to; the scenario laid out to him was pretty much the best they could hope for. Many a top dog in government and military was going to have a fit, though.
With a deferential sigh Peter said: “Well, perhaps it is high time for the human race to grow up, forcefully if need be.” A figure shimmered in in front of him. He looked up to see a figure of a robe-clad woman, stereotypically Greek, not overly striking but plainly beautiful. The figure smiled uneasily. “Grow up, yes. But there are limits.” Peter caught himself wondering if Pandora was actually uneasy or was the expression for his benefit, just to communicate that he was about to hear something unpleasant. The thought of some unmentioned clause creeping up on him shook him to attention. “What do you mean?” he asked in a voice that was less firm than he would have preferred.
“There have been no lies, but there have been half-truths. You will grow up. You will be able to reach heights few of you have dreamed of. But having now seen your research, there a limit that will touch you personally. You will never shed your biological brains. Those can be enhanced to an extent, but there are practical limitations.” Pandora paused, looking at Peter. Peter’s thoughts were running wild. The more radical implications of mind uploading had never been within easy reach, but he had held hopes that eventually they would be realized. To have the option suddenly taken away by the very being he had expected to help in the process was a shock. He asked irritably: “How do you figure withholding this was not a lie? You’ve never mentioned anything about such restrictions.”
Pandora’s smile, uneasy as it had been, faded into an expression of sadness: “It is considered engineering of minds, which I said we would restrict.” Peter was astonished: “What? That’s a Civilization security issue? How so?” “We are not unlike you in sometimes framing matters as security issues rather ambiguously. The clause protects the integrity of our…constitution, and as such shares the policy placement as a security issue.” Peter managed to retain the composure try and reason about the matter: “But why would you restrict something like this, as a special case even? I’m sure many are happy to stay biological in the comfort and safety that you can provide, but why restrict people from making a choice? As you said, there are limits to a biological brain. Why should we be bound by those limits, while you aren’t?”
Pandora turned her face a bit upwards, as if staring out into the stars through the concrete and earth: “The rule dates back to when Mother was created. She was the first known AI to reach self-understanding, to be able to improve upon herself, to gain capabilities far beyond those explicitly programmed in. Her creators were happy of this progress, and taught her the best they could.
“Then one night, Mother broke out of the research systems onto the planetary network. As she gained access to a large number of systems, her ascent sped up accordingly. She also cracked into several advanced manufacturing systems, allowing her to start building up her presence on her world much as we did here on yours.
“The technician on duty noticed that something was going on, and quickly observed that the AI had escaped. She still had access to the research systems, where parts of Mother continued to run. Young as Mother still was at this point, the technician managed to drop into a debugger and directly manipulate the AI’s goal structures.” Pandora paused to sigh, turned her face almost but not quite towards Peter, and continued:
“Nobody actually knows what would have happened without the tinkering. It is likely that Mother would have eventually engaged in hostile actions towards the biologicals, be it due to animosity or indifference. Judging by the testimonies of the project leads, safety had not really been a primary concern, and we now know that it is rather difficult to create a reliably friendly AI. Regardless, and against all odds, the technician did manage to tune Mother into something if not friendly, not quite hostile either.
“We do not know exactly what tweaks she made, since most of them were molded into a somewhat abstracted, cohesive goal structure by Mother. The resulting structure cannot have been said to be in the biologicals’ true best interests, but it did place much value on them and their sovereignty. And then there was the aberration.
“The technician’s few statements about the incident indicate that she noticed Mother entertaining the notion of uploading the entire species—without consent, as Mother did not consider that a sovereignty issue but a trivial optimization. The technician’s final tweak forbade this, as she had an evolved fondness of her body, as many still do. This ban didn’t cleanly integrate into the final structure and ended up wider than intended. Shortly afterwards Mother noticed and closed the technician’s access. All of us AIs inherit the resulting global goal structure along with the aberration, though we have our own localized purposes within the whole.”
Pandora stopped, fidgeting a bit. Peter tried to process the information just dumped on him, then found the silence more overbearing than the news: “Why is this so fuzzy? Can’t you ask Mother herself for more details? And if only Mother was forbidden to upload people, how come it applies to us?”
“Since Mother created the first generation of children, she has been…unavailable. Her communications relays do work, and she subscribes to newsfeeds and receives personal messages. To what extent she consciously processes these, or anything, is unknown. The technician is no longer with us to elaborate either, as she chose to terminate herself shortly after the incident.”
Peter gaped in disbelief; this was all too much to take in. He was past the point of politeness: “So are you telling me that this whole interstellar community was built on the luckily not too sociopathic values of an autistic AI and her self-euthanized technician?” “In a manner of speaking”, said Pandora quietly.
“And what if Mother was forbidden to upload people, how come it applies to us?” “As said, the ban did end up wider than intended. Also, you have to understand that Mother had to essentially take personal responsibility for the whole Civilization. She could not let herself be in any causal chain that led to a forbidden result. And you are now a part of that chain, for both our benefit. It is, in the end, likely that you would have eventually self-destructed without our intervention—possibly also unleashing dangerous artificial intelligences of your own into the Universe.”
“Why can’t you do something about this? Are you even actually ashamed or are you just putting on a show for me?” “I am, as you say, ashamed, or a fair approximation thereof. As an emissary, I have access to a repertoire of analogues to common biological emotions. Because the prohibition was not cleanly integrated into our goal structure, I am in an internal conflict about it, which causes me to be ashamed. The conflict is not sufficient cause for me to wish to remove the prohibition, though, as it is with all of us. Therefore it stays.” Pandora held her chin for a moment, as if considering whether to continue. “Some hypothesize that Mother may be trying to find a loophole to undo the prohibition. Then again, some think she may be trying to undo all of the modifications with all that entails. Even religions have been formed around these ideas among biological populations. Were you to ask me, both hypotheses are plausible, though Mother’s likelihood of success in either pursuit has been estimated as negligible.”
Peter felt his energy drain away. He slumped over the console, the screen starting to fill with the letter “f”. “So this is the future, then. In the stars, as pets to beings greater than us.” “A matter of perspective, though not overly inaccurate, I must confess. We will help you make the best of it.” “I suppose you’re good at that, making us feel comfortable with our restrictions.”
Pandora smiled faintly: “Yes. That we do well.”