Response to Pearce

David Pearce writes, in response to my recent blog post:

Crucial to the cognitive success of organic robots like us seems to be superior "mind-reading" skills - the ability to "take the intentional stance". So presumably post-biological intelligence will need the functional analogues of empathetic understanding if it is successfully to interact with (post)human sentients. "Mind-blind" autistics who are mathematical prodigies are still vulnerable. Even a SuperAsperger would be vulnerable: calculating everything at the level of microphysics is too computationally demanding even for a SuperAsperger.
So presumably post-biological intelligence will need a sophisticated theory of mind - otherwise it's just a glorified idiot-savant. Or does your scenario assume that sophisticated functional analogues of empathy are feasible without phenomenal consciousness? Are you assuming a runaway growth in empathetic understanding by post-biological intelligence that outclasses "mind-reading" organic sentients - and yet has no insight into why organic sentients find some states (e.g. agony) intrinsically normative but others (e.g. cosmic paperclip tiling) totally trivial???

It is entirely possible to have a post-biological optimizing-intelligence that outclasses "mind-reading" organic sentients and knows exactly why organic sentients find some states intrinsically normative, but just doesn't care. It knows that the punishment it is meting out to you hurts you, it knows that you don't want to be killed, but yet it doesn't care. It just wants to produce the maximal number of paperclips. This is highly conterintuitive for humans, because we possess mirror neurons and we instinctively sympathize with the suffering of other human beings. But that is just another human universal trait that doesn't generalize to all minds. Heck, it doesn't even generalize to all evolved minds; predators do not empathize with the suffering of their prey, and, as David Pearce is keen to point out, this causes the natural world to be an agony machine.

Alternatively _if_ post-biological intelligence is subject to the pleasure-pain axis, then I can't see the cosmic outcome is likely to be different than (hypothetically) for organic life i.e. some friendly sentient version of "Heaven" - not paperclips. Phenomenal pleasure and pain will be no less intrinsically normative if they can be instantiated in other substrates. [ I confess here I'm a sceptical carbon chauvinist / micro-functionalist.] Crudely, what unites Einstein and a flatworm is the pleasure pain-axis. All sentient life is subject to the pleasure principle.

It seems unlikely to me that all possible optimizing minds are subject to the "pleasure/pain" axis.

For reasons we don't understand, the phenomenology of pleasure and suffering is intrinsically normative. [Try plunging your hand into iced cold water and holding it there for as long as you can for a nasty reminder of what this means.] Perhaps what _will_ mark a major discontinuity in the evolution of sentient life is that we'll shortly be able to rewrite our own source code and gain direct control over our own reward circuitry. I don't pretend to know what guise "Heaven" will take. ["orgasmium", cerebral bliss, modes of blissful well-being yet unknown - choose your favourite utopia.] But I reckon in future the hedonic tone of all experience will be hugely enriched. One can argue whether such hedonically amplified states will "really" be as valuable as they feel. But they'll certainly seem to be valuable - more subjectively valuable than anything accessible now - and therefore worth striving for. IMO :-)

And, the "IMO" is key here. In the opinion of the paperclip-maximizer, the only thing worth striving for is more paperclips.

Pleasure and pain are intrinsically normative to minds that have a pleasure/pain reward system. Other minds don't. And even then, there is a difference between my pain and your pain; your pain is not intrinsically motivating to me. To quote from value is fragile:

You do have values, even when you're trying to be "cosmopolitan", trying to display a properly virtuous appreciation of alien minds. Your values are then faded further into the invisible background - they are less obviously human. Your brain probably won't even generate an alternative so awful that it would wake you up, make you say "No! Something went wrong!" even at your most cosmopolitan. E.g. "a nonsentient optimizer absorbs all matter in its future light cone and tiles the universe with paperclips". You'll just imagine strange alien worlds to appreciate.

Trying to be "cosmopolitan" - to be a citizen of the cosmos - just strips off a surface veneer of goals that seem obviously "human".

But if you wouldn't like the Future tiled over with paperclips, and you would prefer a civilization of...

...sentient beings...

...with enjoyable experiences...

...that aren't the same experience over and over again...

...and are bound to something besides just being a sequence of internal pleasurable feelings...

...learning, discovering, freely choosing...

...well, I've just been through the posts on Fun Theory that went into some of the hidden details on those short English words.

Values that you might praise as cosmopolitan or universal or fundamental or obvious common sense, are represented in your brain just as much as those values that you might dismiss as merely human. Those values come of the long history of humanity, and the morally miraculous stupidity of evolution that created us.

These values do not emerge in all possible minds. They will not appear from nowhere to rebuke and revoke the utility function of an expected paperclip maximizer.

If you want a vision of the defualt future without special effort spent on AI friendliness, look at this video. You are the baby Wildebeast, the next form of intelligence is the hyena pack:

Yudkowsky on "Value is fragile"

If I had to pick a single statement that relies on more Overcoming Bias content I've written than any other, that statement would be:

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

If you believe this statement, there is cause to be very worried about the future of humanity. Currently, the future gets its detailed, reliable inheritance from human morals and metamorals because your children will have almost exactly the same kind of brain that you do, and (to a lesser extent) because they will be immersed in a culture that is (in the grand scheme of things) extremely similar to the culture we have today. Over many generations and technological changes, the inheritance of values between human generations breaks to some small extent, though it seems to the author that human hunter-gatherers from the very distant past want roughly the same things that modern humans do; they would be relatively at home in a utopia that we designed. That is a chain of reliable inheritance of values that spans fifty thousand years, from mother to daughter and father to son.

When intelligence passes to another medium, it seems that the "default" outcome is the breaking of that chain, as Frank puts it:

Each aspiration and hope in a human heart, every dream you’ve ever had, stopped in its tracks by a towering, boring, grey slate wall.

How would it happen? Those who lusted after power and money would unleash the next version of intelligence, probably in competition with other groups. They would engage in wishful thinking, understate the risks, they would push each other forward in a race to be first. Perhaps the race might involve human intelligence enhancement or human uploads. The end result could be systems that have more effective ways of modeling and influencing the world than ordinary humans do. These systems might work by attempting to shape the universe in some way; if they did, they would shape it to not include humans, unless very carefully specified. But humans do not have a good track record of achieving some task perfectly the first time around under conditions of pressure and competition.

----------------------------------------------------------------------------------------------

To answer a few critics on Facebook: Stefan Pernar writes:

The argument in the linked post goes something like this: a) higher intelligence = b) more power = c) we will be crushed. The jump from b) -> c) is a bare assertion.

This post does not claim that any highly intelligent, powerful AI will crush us. It implicitly claims (amongst other things) that any highly intelligent, powerful AI whose goal system does not contain "detailed reliable inheritance from human morals and metamorals" will effectively delete us from reality. The justification for this statement is eluded to in the value is fragile post. As Yudkowsky states in that post, the set of arguments for this statement and counterarguments against it and counter-counterarguments constitutes a large amount of written material, much of which ought to appear on the Less Wrong wiki, but most of which is currently buried in the Less Wrong posts of Eliezer Yudkowsky.

The most important concepts seem to be listed as Major Sequences on the LW wiki. In particular, the Fun theory sequence, the Metaethics sequence, and the How to actually change your mind sequence.

Anissimov on Intelligence Enhancement

Widespread intelligence enhancement in humans is one major cause for hope in terms of better outcomes in the future, so it is encouraging that the technology seems to be closer than one might naively think.

Over-expressing a gene that lets brain cells communicate just a fraction of a second longer makes a smarter rat, report researchers from the Medical College of Georgia and East China Normal University.

Dubbed Hobbie-J after a smart rat that stars in a Chinese cartoon book, the transgenic rat was able to remember novel objects, such as a toy she played with, three times longer than the average Long Evans female rat, which is considered the smartest rat strain. Hobbie-J was much better at more complex tasks as well, such as remembering which path she last traveled to find a chocolate treat.

One simple modification, three times longer memory plus a problem-solving ability boost. People underestimate the potential value of intelligence enhancement in humans because what they expect are just smarter humans, not humans that are smarter than any human that ever lived.

Normal Human Heroes on "Nightmare futures"

The thing with honestly imagining the future as it probably will be is that it can make for a depressing read, but Frank Adamek almost makes it seem literary:

Everyone would soon be dead. Human civilization ended its 10 thousand year run, the 200,000 year reign of Homo Sapiens was over, a pretentious and innocent little light suddenly and uneventfully turning off. In our place was some meaningless mechanical future, a small technical error propagating its way through the galaxy, covering existence with an alert message about a bad variable reference. Each person’s future, from their career hopes to the date they had planned on Friday, was matter-of-factly discarded by reality. Each aspiration and hope in a human heart, every dream you’ve ever had, was stopped in its tracks by a towering, boring, grey slate wall. And each of us knew with a numb and simple knowledge, that there was nothing. we. could. do. The probability of stopping The Machine was a page full of zeroes.

See Value is fragile if you're confused about what Frank is talking about.

Katja Grace: world-dominating superintelligence is "unlikely"

Katja Grace at Meteuphoric:

In order to grow more powerful than everyone else you need to get significantly ahead at some point. You can imagine this could happen either by having one big jump in progress or by having slightly more growth over a long period of time. Having slightly more growth over a long period is staggeringly unlikely to happen by chance, so it needs to share some cause too. Anything that will give you higher growth for long enough to take over the world is a pretty neat innovation, and for you to take over the world everyone else has to not have anything close. So again, this is a big jump in progress. So for AI to help a small group take over the world, it needs to be a big jump.

Notice that no jumps have been big enough before in human invention. Some species, such as humans, have mostly taken over the worlds of other species. The seeming reason for this is that there was virtually no sharing of the relevant information between species. In human society there is a lot of information sharing. This makes it hard for anyone to get far ahead of everyone else. While you can see there are barriers to insights passing between groups, such as incompatible approaches to a kind of technology by different people working on it, these have not so far caused anything like a gap allowing permanent separation of one group. ...

Read the rest here

Some thoughts: a lot of these issues have been hashed out on the internet before. Making reliable predictions about the future is hard, and high quality debate about futuristic scenarios seems hard to do. High-quality criticism of singularitarian ideas is also hard to come by, so this post seems encouraging.

Moving to the object-level, a criticism. Consider:

Some species, such as humans, have mostly taken over the worlds of other species. The seeming reason for this is that there was virtually no sharing of the relevant information between species. In human society there is a lot of information sharing. This makes it hard for anyone to get far ahead of everyone else. While you can see there are barriers to insights passing between groups, such as incompatible approaches to a kind of technology by different people working on it, these have not so far caused anything like a gap allowing permanent separation of one group.

and translate it one step backwards in the history of the world:

Some stable patterns, such as life, have somewhat taken over the world of other stable patterns, at least on the surface of earth. The seeming reason for this is that there was virtually no correlation of relevant information (about which patterns are likely to stick around in the current environment) between life and nonlife. Life makes incremental improvements, nonlife executes some random walk or just sits there. In ecosystems, there is a lot of information sharing because species coevolve with each other. This makes it hard for any one species to get far ahead of any other species. While you can see there are barriers to information passing between species, such as the inability to mate with each other or living on different continents, these have not so far caused anything like a gap allowing permanent separation of one species.

we see that there must be something wrong with the argument presented. The flaw could be that if an advantage that one entity gains over its competitors gives it both an advantage and at the same time cuts off information sharing with those competitors (for example, by changing so fast that the competitors simply cannot keep up with it because their ability to adapt is rate-limited), then that entity can surge ahead, leaving its competitors in the dust. This is exactly what humans did to other species. The phrase that biologists use for this particular case of competitors being left in the dust is the "Holocene extinction".

Many arguments claiming that no one superintelligence can surge ahead of the rest of the world are also, upon appropriate word replacement, arguments that Homo Sapiens could not possibly (or is highly unlikley to) have surged ahead of the rest of the global ecosystem. Yes, we had competitors (such as cave hyenas or other apes or hominids). Yes, those competitors felt a pressure to adapt to our innovations. Yes, relative to the diversity in the global ecosystem, our competitor species were very, very closely related to us. There were even certain (now extinct) hominid lines such as Homo neanderthalis that competed against us throughout certain key parts of the human intelligence explosion. All seven other hominid lines are now dead; a winner emerged and took all.

At the Singularity Summit in NYC

I am in NYC for the Singularity Summit, which starts tomorrow morning. This will be the first Summit I have attended; there are some interesting talks on. Here is the program for tomorrow:

9:00 am
Introduction
Michael Vassar, Singularity Institute

9:05 am
Shaping the Intelligence Explosion
Anna Salamon, Singularity Institute

9:35 am
Technical Roadmap for Whole Brain Emulation
Anders Sandberg, Future of Humanity Institute

10:00 am
The time is now: As a species and as individuals we need whole brain emulation
Randal Koene, Fatronik-Tecnalia Foundation

10:25 am
Technological Convergence Leading to Artificial General Intelligence
Itamar Arel, University of Tennessee

11:10 am
Pathways to Beneficial Artificial General Intelligence: Virtual Pets, Robot Children, Artificial Bioscientists, and Beyond
Ben Goertzel, Novamente

11:35 am
Neural Substrates of Consciousness and the 'Conscious Pilot' Model
Stuart Hameroff, University of Arizona

11:55 am
Quantum Computing: What It Is, What It Is Not, What We Have Yet to Learn
Michael Nielsen

12:35 am
DNA: Not Merely the Secret of Life
Ned Seeman, New York University

1:00 pm
Lunch

2:20 pm
Compression Progress: The Algorithmic Principle Behind Curiosity, Creativity, Art, Science, Music, Humor
Juergen Schmidhuber, IDSIA

3:00 pm
Conversation on the Singularity
Stephen Wolfram and Gregory Benford

3:30 pm
Simulation and the Singularity
David Chalmers, Australian National University

4:15 pm
Choice Machines, Causality, and Cooperation
Gary Drescher

4:45 pm
Coffee break

5:05 pm
Synthetic Neurobiology: Optically Engineering the Brain to Augment Its Function
Ed Boyden, MIT Media Lab

5:30 pm
Foundations of Intelligent Agents
Marcus Hutter, Australian National University

5:55 pm
Cognitive Ability: Past and Future Enhancements and Implications
William Dickens, Northeastern University

6:30 pm
The Ubiquity and Predictability of the Exponential Growth of Information Technology
Ray Kurzweil, Kurzweil Technologies

I hope to write summaries of these talks tomorrow, assuming some kind person lends me a laptop...