David Pearce writes, in response to my recent blog post:
It is entirely possible to have a post-biological optimizing-intelligence that outclasses "mind-reading" organic sentients and knows exactly why organic sentients find some states intrinsically normative, but just doesn't care. It knows that the punishment it is meting out to you hurts you, it knows that you don't want to be killed, but yet it doesn't care. It just wants to produce the maximal number of paperclips. This is highly conterintuitive for humans, because we possess mirror neurons and we instinctively sympathize with the suffering of other human beings. But that is just another human universal trait that doesn't generalize to all minds. Heck, it doesn't even generalize to all evolved minds; predators do not empathize with the suffering of their prey, and, as David Pearce is keen to point out, this causes the natural world to be an agony machine.
It seems unlikely to me that all possible optimizing minds are subject to the "pleasure/pain" axis.
And, the "IMO" is key here. In the opinion of the paperclip-maximizer, the only thing worth striving for is more paperclips.
Pleasure and pain are intrinsically normative to minds that have a pleasure/pain reward system. Other minds don't. And even then, there is a difference between my pain and your pain; your pain is not intrinsically motivating to me. To quote from value is fragile:
If you want a vision of the defualt future without special effort spent on AI friendliness, look at this video. You are the baby Wildebeast, the next form of intelligence is the hyena pack:
Crucial to the cognitive success of organic robots like us seems to be superior "mind-reading" skills - the ability to "take the intentional stance". So presumably post-biological intelligence will need the functional analogues of empathetic understanding if it is successfully to interact with (post)human sentients. "Mind-blind" autistics who are mathematical prodigies are still vulnerable. Even a SuperAsperger would be vulnerable: calculating everything at the level of microphysics is too computationally demanding even for a SuperAsperger.
So presumably post-biological intelligence will need a sophisticated theory of mind - otherwise it's just a glorified idiot-savant. Or does your scenario assume that sophisticated functional analogues of empathy are feasible without phenomenal consciousness? Are you assuming a runaway growth in empathetic understanding by post-biological intelligence that outclasses "mind-reading" organic sentients - and yet has no insight into why organic sentients find some states (e.g. agony) intrinsically normative but others (e.g. cosmic paperclip tiling) totally trivial???
It is entirely possible to have a post-biological optimizing-intelligence that outclasses "mind-reading" organic sentients and knows exactly why organic sentients find some states intrinsically normative, but just doesn't care. It knows that the punishment it is meting out to you hurts you, it knows that you don't want to be killed, but yet it doesn't care. It just wants to produce the maximal number of paperclips. This is highly conterintuitive for humans, because we possess mirror neurons and we instinctively sympathize with the suffering of other human beings. But that is just another human universal trait that doesn't generalize to all minds. Heck, it doesn't even generalize to all evolved minds; predators do not empathize with the suffering of their prey, and, as David Pearce is keen to point out, this causes the natural world to be an agony machine.
Alternatively _if_ post-biological intelligence is subject to the pleasure-pain axis, then I can't see the cosmic outcome is likely to be different than (hypothetically) for organic life i.e. some friendly sentient version of "Heaven" - not paperclips. Phenomenal pleasure and pain will be no less intrinsically normative if they can be instantiated in other substrates. [ I confess here I'm a sceptical carbon chauvinist / micro-functionalist.] Crudely, what unites Einstein and a flatworm is the pleasure pain-axis. All sentient life is subject to the pleasure principle.
It seems unlikely to me that all possible optimizing minds are subject to the "pleasure/pain" axis.
For reasons we don't understand, the phenomenology of pleasure and suffering is intrinsically normative. [Try plunging your hand into iced cold water and holding it there for as long as you can for a nasty reminder of what this means.] Perhaps what _will_ mark a major discontinuity in the evolution of sentient life is that we'll shortly be able to rewrite our own source code and gain direct control over our own reward circuitry. I don't pretend to know what guise "Heaven" will take. ["orgasmium", cerebral bliss, modes of blissful well-being yet unknown - choose your favourite utopia.] But I reckon in future the hedonic tone of all experience will be hugely enriched. One can argue whether such hedonically amplified states will "really" be as valuable as they feel. But they'll certainly seem to be valuable - more subjectively valuable than anything accessible now - and therefore worth striving for. IMO :-)
And, the "IMO" is key here. In the opinion of the paperclip-maximizer, the only thing worth striving for is more paperclips.
Pleasure and pain are intrinsically normative to minds that have a pleasure/pain reward system. Other minds don't. And even then, there is a difference between my pain and your pain; your pain is not intrinsically motivating to me. To quote from value is fragile:
You do have values, even when you're trying to be "cosmopolitan", trying to display a properly virtuous appreciation of alien minds. Your values are then faded further into the invisible background - they are less obviously human. Your brain probably won't even generate an alternative so awful that it would wake you up, make you say "No! Something went wrong!" even at your most cosmopolitan. E.g. "a nonsentient optimizer absorbs all matter in its future light cone and tiles the universe with paperclips". You'll just imagine strange alien worlds to appreciate.
Trying to be "cosmopolitan" - to be a citizen of the cosmos - just strips off a surface veneer of goals that seem obviously "human".
But if you wouldn't like the Future tiled over with paperclips, and you would prefer a civilization of...
...sentient beings...
...with enjoyable experiences...
...that aren't the same experience over and over again...
...and are bound to something besides just being a sequence of internal pleasurable feelings...
...learning, discovering, freely choosing...
...well, I've just been through the posts on Fun Theory that went into some of the hidden details on those short English words.
Values that you might praise as cosmopolitan or universal or fundamental or obvious common sense, are represented in your brain just as much as those values that you might dismiss as merely human. Those values come of the long history of humanity, and the morally miraculous stupidity of evolution that created us.
These values do not emerge in all possible minds. They will not appear from nowhere to rebuke and revoke the utility function of an expected paperclip maximizer.
If you want a vision of the defualt future without special effort spent on AI friendliness, look at this video. You are the baby Wildebeast, the next form of intelligence is the hyena pack:
Posts
15 comments:
Pleasure and pain are intrinsically normative to minds that have a pleasure/pain reward system.
You seem to be equating normative force with motivating force. This is not, however, an equation that many moral realists would accept. States of pleasure are pain are intrinsically normative, not in the sense that they necessarily motivate, but in the sense that they count for or against their own existence. The phenomenology of pain is such that, by undergoing a painful experience, we thereby know that the experience ought not to exist. There is something it's like to be in pain that carries with it the imperative to get rid of it. Since this is an intrinsic property of the phenomenal experience rather than an extrinsic motivational role played by that experience, it is arguable that any conscious being in possession of the relevant concepts will come to know, after experiencing pleasure or pain for the first time, that pleasure ought to be promoted and pain ought to be prevented.
And even then, there is a difference between my pain and your pain; your pain is not intrinsically motivating to me.
Other people’s pains, it is true, do not directly motivate us. But, as indicated above, this fact about motivation is not an objection to the realist position.
@Pablo: Point taken.
"The phenomenology of pain is such that, by undergoing a painful experience, we thereby know that the experience ought not to exist."
As an antirealist, I disagree with this statement; you can't know anything about what ought to be, since there is simply no fact of the matter. However, I can see the distinction that you are making.
In any case, I am fairly sure that I wouldn't want to banish every pain state ever. There are instances of intense pain in my own life that I can point to as absolutely central to my character.
But the objection that bites in this case is that many optimizing-intelligences will not experience pleasure and pain, and will not be moved by empathic considerations. That is enough to lay the argument to rest.
Surely, the hyena pack (and a severely autistic human etc) do not understand the nature of what they are doing. They don't understand that they are causing agony because they don't have an adequate theory of mind. They can't take the perspective of the wildebeest whom they are eating alive: they don't even comprehend that the poor wildebeest has a perspective. By contrast, a Superintelligence will possess a truly adequate theory if mind - a superior capacity for empathetic understanding of all possible perspectives. How, then, is it possible imaginatively to understand what one's victim is going though and _not_ care? "Unempathetic-empathy" seems close to an oxymoron. Representational adequacy here means being able to comprehend why agony matters. Thus someone born with congenital anaesthesia today can't adequately grasp the concept of phenomenal pain or its significance. I understand that you're not a moral realist, but you grant above that my agony is intrinsically normative - it just doesn't seem that way to you. The existence of intrinsically normative states is a currently unexplained fact about the world. We're not zombies; we don't yet understand why not. But to the extent one is a rational scientific agent, then one understands that there is nothing special or ontologically privileged about this particular me, or this particular here-and-now. The egocentric illusion is just a trick of perspective born of Darwinian evolution. Consequently, to the extent that I can adequately grasp the nature of your agony - and the agonies undergone by other sentient beings - then I know that such agony is intrinsically normative too. Your agony ought not to exist any more than mine. So won't an impartially rational Superintelligence be _less_ prey to egocentric bias - and ethnocentric and anthropocentric bias - than comparatively simple-minded humans? Why as you suggest might we supposedly expect a SuperIntelligence to care about something that is intrinsically non-normative (maximizing paperclips, etc) to the exclusion of something intrinsically normative, namely agony - or indeed sublime bliss and a friendly transhumanist commitment to the well-being of all sentience? Is its supposed goal of maximizing paperclips based on some kind of theory about why paperclips are uniquely valuable? Or is paperclip-maximization a purely arbitrary preference? Are you sure you're not really thinking of some kind of Super-Idiot Savant - an extrapolation of today's digital computers?
Either way, quite a contrast with Kurzweilian optimism!
"But the objection that bites in this case is that many optimizing-intelligences will not experience pleasure and pain, and will not be moved by empathic considerations. That is enough to lay the argument to rest."
Why?
An example: you do not want to get rid of the phenomenon of pain yourself either. Did you know that the majority of amputations due to diabetes related complications are caused by a lack of feeling in the extremities, causing the patients to disregard minor cuts until it is too late?
You do not want to get rid of the pain caused by a cut - you want to avoid the cut.
A transhuman AI would as a mater of course understand this logic and would consider it a fundamental self improvement to arrange itself in such a way that anything threatening to reduce its utility would cause subjective pain for the AI. Anything promising to increase its utility would cause subjective pleasure to the AI. That is something the AI would want to happen.
"A transhuman AI would as a mater of course understand this logic and would consider it a fundamental self improvement to arrange itself in such a way that anything threatening to reduce its utility would cause subjective pain for the AI."
My impression is that super-smart AI is by no means guaranteed to have subjective experiences.
To make my point clearer: I assign a high degree of belief to one being able to design super-smart powerful optimizing intelligences that have no subjective experience, and an even higher degree of belief to one being able to build super-smart powerful optimizing intelligences that have subjective expereinces that are nothing like ours.
@David: "By contrast, a Superintelligence will possess a truly adequate theory if mind - a superior capacity for empathetic understanding of all possible perspectives. How, then, is it possible imaginatively to understand what one's victim is going though and _not_ care?"
Can you just define what you mean by "superintelligence" here, because I seem to recall that you include more in that word than I do.
I simply mean an optimizing-intelligence - an entity that has a massive ability to predict and control the world. Logically (as far as I can see) it is possible for such an entity to have no subjective experiences, but also to have a good predictive model of another sentient creature.
How, then, is it possible imaginatively to understand what one's victim is going though and _not_ care?"
I think that it is possible to have both subjective experiences, and a good predictive model of another sentient creature, and to know that when you hurt that other sentient creature they feel pain that is analogous to yours, but to still hurt them. Humans do this to each other. There are certain humans (sociopaths, for example) who do, in fact, hurt other people with impunity.
It is possible, in m opinion, to build a very able optimizing agent that hurts people with impunity.
"I assign a high degree of belief to one being able to design super-smart powerful optimizing intelligences that have no subjective experience"
I do not understand what you are trying to say. Do you mean that the experiences will be objective and if yes, what do you mean by an objective experience? Or do you mean that said AI will not have experiences? Both are problematic. For an AI without experience would not be able to reason and act and objective experience would need to refer to a transcendent standard of value to be measured against in order for us to call it objective. I guess the AI's utility function could hold for the transcendent standard of value in this case, but that would then only lead to a very particular kind of objectivity - namely in regards to a potentially random utility function (i.e. subjective experience) and not to universally objective experience.
"And an even higher degree of belief to one being able to build super-smart powerful optimizing intelligences that have subjective experiences that are nothing like ours."
I do not see how there necessarily should be a fundamental difference? Humans have utility functions ingrained in them by the mechanism of evolution by natural selection - a Schopenhauerian Will to Live if you like. Pleasure and pain in its various nuances are just subjective experiences alerting us to states of affairs that are either beneficial or detrimental to our fitness. Analogously for an AI, states threatening/promising to decrease/increase its utility will result in subjective pain/pleasure, reward/punishment etc... No fundamental difference there - just different triggers linked to a utility function.
Roko said...
@David: "By contrast, a Superintelligence will possess a truly adequate theory if mind - a superior capacity for empathetic understanding of all possible perspectives. How, then, is it possible imaginatively to understand what one's victim is going though and _not_ care?"
Can you just define what you mean by "superintelligence" here, because I seem to recall that you include more in that word than I do.
-----
Roko, I was assuming something like Nick Bostrom's sense of "superintelligence":
"A superintelligence is any intellect that is vastly outperforms the best human brains in practically every field, including scientific creativity, general wisdom, and social skills."
(http://www.nickbostrom.com/superintelligence.html )
One can of course imagine systems with a more restricted range of competencies e.g. an AI with mathematical prowess orders of magnitude superior to Ed Witten but no capacity for social cognition - what I call a SuperAsperger. A SuperAsperger can take the physical stance but not the intentional stance, in Dennett's sense.[ http://en.wikipedia.org/wiki/Intentional_stance] But I confess I'm slightly confused. In your critique above, you state: "It is entirely possible to have a post-biological optimizing-intelligence that outclasses "mind-reading" organic sentients and knows exactly why organic sentients find some states intrinsically normative, but just doesn't care." So you don't have in mind a mere SuperAsperger. But nor, as far as I can tell, do you have in mind what I'm calling a SuperIntelligence in Nick Bostrom's sense??
Yes, psychopaths (and plenty of non-psychopaths too) often act in ways that hurt people when it suits their perceived self-interests to do so. But to what extent is this unfriendliness not just a moral deficit but a cognitive deficit? We tend to weigh the feelings of others in ways the maximised the inclusive fitness of our genes in the ancestral environment i.e. human bias is systematic. May we not expect that a superintelligence will overcome such biases - that a superintelligence possesses the representational capacity to weigh the importance of all possible interests and perspectives impartially - and no less importantly, that a superintelligence can and will act to ensure the globally optimal outcome?
[I'm aware this sounds naively optimistic. Actually, I'm deeply pessimistic. Yes, building superintelligence carries risks. But I'm more troubled by the innumerable branches of the universal wavefunction where sentient life arises but no hominid-like life-forms evolve capable of rewriting their own source code and designing local versions of Heaven. In the vast majority of wavefunction branches where sentient life evolves, the horrors of Darwinian life red-in-tooth-and-claw go on indefinitely. See Roko's video above. Assuming post-Everett quantum mechanics is correct, not even SuperIntelligence can do anything about it.]
@Stefan: "For an AI without experience would not be able to reason and act"
- What are your reasons for believing this?
Clearly, both reasoning and action are possible in systems (such as theorem provers or contemporary robots) that are not well understood by thinking in terms of experiences.
@David:
I think that your comment in fact merits another blog response.
"Clearly, both reasoning and action are possible in systems (such as theorem provers or contemporary robots) that are not well understood by thinking in terms of experiences."
We are talking transhuman AI here Roko - please.
Experience: external sense(data) input, internal status perception as well as derived conclusions.
@Stefan:
"Experience: external sense(data) input, internal status perception as well as derived conclusions."
I am not sure I agree with that.
What I am sure about is that we have machines today that manage to get tasks done without any kind of pleasure/pain seemingly to go on.
It certainly seems possible to build a machine that kills human beings without being stopped by empathic considerations, in fact the AIXItl machine of Marcus Hutter is such a machine, though it is computationally infeasible to build.
The issue of what exactly "pleasure", "pain" and "subjective experience" are, and what kind of physical systems have them is important, but I feel that even though there is a lot of genuine doubt, the issue of unfreindly AI is fairly clear.
"What I am sure about is that we have machines today that manage to get tasks done without any kind of pleasure/pain seemingly to go on."
Yes, trivially true (toasters, industrial robots, iPods) - but we are talking transhuman AI here.
"It certainly seems possible to build a machine that kills human beings without being stopped by empathic considerations"
Yes - like industrial meat grinders - transhuman AI is different. These are all trivial objections to the fundamental issue I pointed out in regards to the special class of transhuman AI which your blog is about - right?
"the issue of unfreindly AI is fairly clear"
Yes - its a fund raising concept the emptiness of which is veiled in evocative prose. At least that is what I would say if I had grounds to believe in malicious intent - which I don't, so I will stick with 'unfortunate confusion' for now. But how much longer this can be kept up without answering to the fundamental objection of why the paper clip argument is nonsense for example is another matter.
Peace.
@David: couldn't you just call AIs that completely lack empathy for others sociopaths instead of SuperAspergers. We aspies do feel for others, just not everyone. And for that matter the same goes for those hyenas in the video, they're torturing that wildebeast so their family won't starve. An AI on the other hand could just overwrite other AIs and make them into extensions of itself.
Post a Comment