Katja Grace at Meteuphoric:
In order to grow more powerful than everyone else you need to get significantly ahead at some point. You can imagine this could happen either by having one big jump in progress or by having slightly more growth over a long period of time. Having slightly more growth over a long period is staggeringly unlikely to happen by chance, so it needs to share some cause too. Anything that will give you higher growth for long enough to take over the world is a pretty neat innovation, and for you to take over the world everyone else has to not have anything close. So again, this is a big jump in progress. So for AI to help a small group take over the world, it needs to be a big jump.
Notice that no jumps have been big enough before in human invention. Some species, such as humans, have mostly taken over the worlds of other species. The seeming reason for this is that there was virtually no sharing of the relevant information between species. In human society there is a lot of information sharing. This makes it hard for anyone to get far ahead of everyone else. While you can see there are barriers to insights passing between groups, such as incompatible approaches to a kind of technology by different people working on it, these have not so far caused anything like a gap allowing permanent separation of one group. ...
Some thoughts: a lot of these issues have been hashed out on the internet before. Making reliable predictions about the future is hard, and high quality debate about futuristic scenarios seems hard to do. High-quality criticism of singularitarian ideas is also hard to come by, so this post seems encouraging.
Moving to the object-level, a criticism. Consider:
Some species, such as humans, have mostly taken over the worlds of other species. The seeming reason for this is that there was virtually no sharing of the relevant information between species. In human society there is a lot of information sharing. This makes it hard for anyone to get far ahead of everyone else. While you can see there are barriers to insights passing between groups, such as incompatible approaches to a kind of technology by different people working on it, these have not so far caused anything like a gap allowing permanent separation of one group.
and translate it one step backwards in the history of the world:
Some stable patterns, such as life, have somewhat taken over the world of other stable patterns, at least on the surface of earth. The seeming reason for this is that there was virtually no correlation of relevant information (about which patterns are likely to stick around in the current environment) between life and nonlife. Life makes incremental improvements, nonlife executes some random walk or just sits there. In ecosystems, there is a lot of information sharing because species coevolve with each other. This makes it hard for any one species to get far ahead of any other species. While you can see there are barriers to information passing between species, such as the inability to mate with each other or living on different continents, these have not so far caused anything like a gap allowing permanent separation of one species.
we see that there must be something wrong with the argument presented. The flaw could be that if an advantage that one entity gains over its competitors gives it both an advantage and at the same time cuts off information sharing with those competitors (for example, by changing so fast that the competitors simply cannot keep up with it because their ability to adapt is rate-limited), then that entity can surge ahead, leaving its competitors in the dust. This is exactly what humans did to other species. The phrase that biologists use for this particular case of competitors being left in the dust is the "Holocene extinction".
Many arguments claiming that no one superintelligence can surge ahead of the rest of the world are also, upon appropriate word replacement, arguments that Homo Sapiens could not possibly (or is highly unlikley to) have surged ahead of the rest of the global ecosystem. Yes, we had competitors (such as cave hyenas or other apes or hominids). Yes, those competitors felt a pressure to adapt to our innovations. Yes, relative to the diversity in the global ecosystem, our competitor species were very, very closely related to us. There were even certain (now extinct) hominid lines such as Homo neanderthalis that competed against us throughout certain key parts of the human intelligence explosion. All seven other hominid lines are now dead; a winner emerged and took all.
Posts
10 comments:
Testing the comment system
If we take the original argument at face value, that a single world dominating super-intelligence is unlikely to take over the world, that still doesn't rule out the event we call the singularity. there could be millions of different super-intelligent beings after the singularity. It's not called the singularity because, like Highlander, "There can be just one." It's called that because there is an event horizon beyond which our imaginations cannot reach. That event horizon is not the singularity, anymore than in the black hole scenario the terms come from. There is no way we can imagine even the short period that will immediately precede the singularity. Today I think it is only safe to say that we are all going to be surprised, and most of us will say we saw it coming all along. At best, some have been able to imagine the fuzzy blur in the dark that shows the general shape the future will take, but we have no idea as to the details.
I had an aunt who was born shortly after the Civil War, and died shortly after Neil Armstrong walked on the moon. She saw enormous changes in 99 years, but most of the change she saw came in the last 20 years of her life. The 40 years since she passed on have been even more amazing.
We are already part of the embryonic singularity even as I type and as you read. The conversations we have about our hopes and fears of what might happen are shaping the future. We are part of the initial conditions the singularity is developing from and will expand on. The future doesn't come from some other dimension and intrude upon our lives. Tomorrow is the natural outgrowth of today. So if we want to keep the super intelligent cyborgs of the future from being jerks, we might want to focus on clearing up that problem in our present lives. Or - as the twig is bent, etc.
I now return you to the future, already in progress.
> "If we take the original argument at face value, that a single world dominating super-intelligence is unlikely to take over the world, that still doesn't rule out the event we call the singularity."
This is true, but I thought I would attack the argument as presented, because it is (IMO) flawed.
In reality, two or more roughly-equal-in-power synthetic intellgiences will be somewhat tempted to aggregate their utility functions, rather than pay the price of fighting each other. This means that the intelligence explosion tends to result in a Singleton, as far as I can see.
Your argument requires the sort of sharing of information that humans do to be analogous in the relevant way to the sharing that species do in co-evolving. In coevolution species do not copy the techniques the other has at all, they adapt to the change caused by the new techniques. So if one species has a better than usual new innovation the other does not get a better than usual adaptation in response, just the same mutations as they would have otherwise with a bit more selection pressure. Finding out that someone is doing something is much less useful than being able to do it too.
Your idea of changing so fast that nobody can catch up seems similar to mine of there being a large jump, so not sure we disagree there. I'm not sure that's what we did to the other species though. Had we moved slowly would they have adapted differently? Unless they adapted by also becoming similar to us, I think we would have won out. There seems little reason to think they would do that.
@Katja:
"Your argument requires the sort of sharing of information that humans do to be analogous in the relevant way to the sharing that species do in co-evolving."
right, but *the relevant way* depends upon what other, fast-changing pieces of information are out there in the environment at that time. For example, being able to look at, copy and spread by word of mouth is enough to catch up with innovations that a human tribe might make. But it would not be enough to catch up with innovations that a self-improving AI or emulant made if those innovations were either well hidden or happened on a 10-second timescale.
Roko,
Innovation happening so fast that spreading by the usual means is too slow comes under what I called 'a big jump'. The usual means will presumably be faster when this happens if we have moved to it incrementally. Does someone have better calculations for what the speed of innovation might be?
> The usual means will presumably be faster when this happens if we have moved to it incrementally.
Suppose that two fast things both accelerate in proportion to how fast they are already going. The gap between them increases.
There are many processes where the winner takes all because of this effect - the further ahead you are so far, the more you outpace the competition.
Roko, your picture is scary. Also part of the question is what interaction between them happens while one is not too far ahead. e.g. information sharing. It is faster to catch up with someone with a faster growth rate by copying them than it is to attain whatever they did on your own.
@Katja:
Suppose that you and I are competing recursively-self-improving processes.
Suppose that the fundamental equation for recursively-self-improving processes is something like dI/dt = kI
It is true that you could put a term into your differential equation that said that the further you are ahead of me, the more I am likely to be able to steal from you, and therefore the more quickly I will approach your current position.
dI/dt = kI + m(J-I) if I less than J
dI/dt = kI if I greater than J
m >> k (it is easier to steal than to invent for yourself)
But if this term had been there in the evolution of humans, then all of the other species in the world would have stolen intelligence from us. So the term must say something else; perhaps
dI/dt = kI + m log(J-I) if I less thanJ
dI/dt = kI if I greater than J
So that if I get sufficiently far ahead of you, my kI term means that I grow more quickly than you.
"But if this term had been there in the evolution of humans, then all of the other species in the world would have stolen intelligence from us."
There is a slight flaw in this argument. Which is, that species cannot deliberately steal or copy innovations of other species under any circumstances. So a fox can't admire a badger's digging skills and copy the badger's paws and shoulders for herself or her kits. Nor can a chimpanzee admire the brain power of a homo erectus and deliberately copy this brain power for herself or her children.
Humans, on the other hand, can and do copy technology from competing groups. "Hey! They've got swords, let's make some sword-like things for ourselves so we can fight back!" This is why the evolution of culture and technology has followed a much different, and much faster path than the evolution of species.
If two fast objects both increase their speed proportional to already going fast the fast one gets faster. Except, this is a special case. The faster you're going, the more able you are to steal speed from all other objects, especially slower objects. So the fastest object steals speed (in this case brain power/innovation speed) from all other objects like it's closest competitor (another very intelligent AI) and lots of other objects (less intelligent AIs, server farms, humans that are {willingly or not} implanted with brain-computer-interface chips).
So, just as we couldn't compare the progress of human technology to the progress of evolution without taking into account the game-changing acceleration of deliberately stealing competitor's innovations, we cannot directly compare the progress of human-level AI innovation to the progress of human technology without taking into account the game-changing acceleration of deliberately stealing competitor's brainpower as well as just their innovations.
(I apologize for the run on sentences.)
So, the slightly better AI would be using it's slightly better smarts to hack into and steal computing power from its nearest competitors, or else forcing them to cut themselves off from the internet in fear, and thus leaving all those huge non-intelligent computing resources to the sole dominion of the slightly better AI, thus making it far far better in a matter of minutes or a few hours. And of course, it's control over the worlds economies would allow it to hire mercenaries to penetrate and destroy (or better yet connect to the internet without a firewall) the physical locations of its competitors' servers. Thus, the situation would lean heavily in favor, as Roko said, of a single entity prevailing. Of course, this single entity would be free to split itself into many independant-but-fully-cooperative entities, or remerge these entities at will.
The key controlling factors are could the best AI escape human control, and would it act to maximize itself if it did so.
Post a Comment