If I had to pick a single statement that relies on more Overcoming Bias content I've written than any other, that statement would be:
If you believe this statement, there is cause to be very worried about the future of humanity. Currently, the future gets its detailed, reliable inheritance from human morals and metamorals because your children will have almost exactly the same kind of brain that you do, and (to a lesser extent) because they will be immersed in a culture that is (in the grand scheme of things) extremely similar to the culture we have today. Over many generations and technological changes, the inheritance of values between human generations breaks to some small extent, though it seems to the author that human hunter-gatherers from the very distant past want roughly the same things that modern humans do; they would be relatively at home in a utopia that we designed. That is a chain of reliable inheritance of values that spans fifty thousand years, from mother to daughter and father to son.
When intelligence passes to another medium, it seems that the "default" outcome is the breaking of that chain, as Frank puts it:
Each aspiration and hope in a human heart, every dream you’ve ever had, stopped in its tracks by a towering, boring, grey slate wall.How would it happen? Those who lusted after power and money would unleash the next version of intelligence, probably in competition with other groups. They would engage in wishful thinking, understate the risks, they would push each other forward in a race to be first. Perhaps the race might involve human intelligence enhancement or human uploads. The end result could be systems that have more effective ways of modeling and influencing the world than ordinary humans do. These systems might work by attempting to shape the universe in some way; if they did, they would shape it to not include humans, unless very carefully specified. But humans do not have a good track record of achieving some task perfectly the first time around under conditions of pressure and competition.
----------------------------------------------------------------------------------------------
To answer a few critics on Facebook: Stefan Pernar writes:
The argument in the linked post goes something like this: a) higher intelligence = b) more power = c) we will be crushed. The jump from b) -> c) is a bare assertion.This post does not claim that any highly intelligent, powerful AI will crush us. It implicitly claims (amongst other things) that any highly intelligent, powerful AI whose goal system does not contain "detailed reliable inheritance from human morals and metamorals" will effectively delete us from reality. The justification for this statement is eluded to in the value is fragile post. As Yudkowsky states in that post, the set of arguments for this statement and counterarguments against it and counter-counterarguments constitutes a large amount of written material, much of which ought to appear on the Less Wrong wiki, but most of which is currently buried in the Less Wrong posts of Eliezer Yudkowsky.
The most important concepts seem to be listed as Major Sequences on the LW wiki. In particular, the Fun theory sequence, the Metaethics sequence, and the How to actually change your mind sequence.
Posts
25 comments:
We are soooo screwed.
I am not yet convinced that we are fully screwed. I think that many people (ahem, Shane) underestimate how hard it will be to get to the kind of AI that can reliably self-improve. There are probably still a few fairly safe decades left to act, which is time enough to see how much the powers that be are prepared to listen.
And, if they are prepared to listen, we may well win.
I'm with Shane.
with a name like sexgenderbody, it sounds like you being "with shane" could be pretty kinky...
The powers that be will listen, but not before it's really obvious where things are going. By then you won't be able to stop it. Some group somewhere will perceive a gain to be made and defect.
The blogger.com captcha word that just came up for this comment is, would you believe, "prefai"
I doubt the universe would exists without human. It is just an assumption. Only human have feeling but not robot.
It would not be as scary as you think. Not all parents have bad morals. Only bad values will build a disastrous foundation.
@Shane: To what extent does that worry you that we are in a simulation?
@Roko: it doesn't worry me whether reality is a simulation or not. Reality is what it is and there is nothing I can do about it. Even if I knew for sure, I don't see how it would impact any decisions I make. What ever I'm experiencing, is, for me, reality.
By the way, the captcha this time is "reali"
Maybe the reality simulations lords are trying to screw with my mind :-) As I believe that people see patterns in randomness, they are going to have to try harder.
@Marc
There was a guy in New Zealand call, if I remember, Bruce Cathy. He had this theory that certain numbers, "harmonic frequencies", were fundamental to physics and UFOs and so on. He wrote several books about it and I read a couple of them from our local library as a teenager. Sure enough, after reading these books, I started seeing these special numbers turning up all over the place. So I made an experiment: I picked a new number at random and remembered it and sure enough... I started noticing this number again and again over the following weeks. :-)
@Shane: there are some instances where a simulation hypothesis actually has relevance.
For example, it seems that simulating people over many different contexts in order to ascertain their motivations is a strategy that comes up in a number of situations.
for example, you might simulate a bunch of copies of a person in a pre-singularity scenario in order to see how hard they work on friendliness, and if they work hard on it and generally act so as to mitigate the risk of a negative singularity, then they get some reward. If not, then they get some punishment.
Sure, there are plenty of reasons to be worried, the shields don't work so well...
@Roko
Either way I don't want a negative singularity.
... that's not a very convincing example!
Re "It implicitly claims (amongst other things) that any highly intelligent, powerful AI whose goal system does not contain "detailed reliable inheritance from human morals and metamorals" will effectively delete us from reality."
I find Yudkowsky's writing style - nested linking to multiple 10k word long ramblings - to be a form of security through obscurity in protecting the essential emptiness of many of his arguments. In the past I found that engaging in the laborious practice of dissecting his writing usually yielded little reward and that his followers prefer to refute my counter arguments by simply pointing out that they contradicted Yudkowsky's and posting a few links to his writings instead of actually arguing the case why him contradicting me invalidated my position yet me contradicting him did not invalidate his. Without realizing of course that what they did was just another form of argument from authority.
Looks like you did it again: advising me to just read a few thousand words here, here and there and I should be back on track again from my erroneous thinking. Let me tell you something Roko - I have been reading and understanding Yudkowsky's position for the better part of the past 5 years now and see little to no value in it and find this kind of 'argumentation by linking' highly patronizing. Of course I wont hold it against you since I am convinced you mean well and hope that we will get along very well in the future :-)
And of course I will not do the same and link here and there and over there to things that I have written over the past 4 years and ask you to read that. Instead I will take my own medicine and and actually present a short, consistent and concise argument to illustrate my point instead and hope that it will be reciprocated.
Core question: are human morals and (meta)morals universal/rational?
Assumption A: Human (meta)morals are not universal/rational.
Assumption B: Human (meta)morals are universal/rational.
Under assumption A one would have no chance of implementing any moral framework into an AI since it would be undecidable which ones they were. Mine or yours, Hitler's or Gandhi's, Joe the plumber's or Joe Lieberman's, Buddha's or Xenu's?
Under assumption B one would not need to implement a moral framework at all since the AI would be able to deduce them using reason alone and come to cherish them for the sole reason that they are based on rational understanding and universality.
You see? Under both assumptions the above statement is simply nonsense. Since under A it would be impossible and under B it would be unnecessary because it would be self evident to a transhuman AI.
Now I happen to be a proponent of assumption B but that is totally beside the point in this particular situation.
For anyone wanting to read a bit more on why the paper clip argument is nonsense feel free to check out Why you don’t want your bombs to be too smart.
@Stefan:
"Under assumption A one would have no chance of implementing any moral framework into an AI since it would be undecidable which ones they were. Mine or yours, Hitler's or Gandhi's, Joe the plumber's or Joe Lieberman's, Buddha's or Xenu's?"
That doesn't mean that one cannot do it. I can simply choose what to put in.
@Roko "That doesn't mean that one cannot do it. I can simply choose what to put in."
Yes of course you could do that, Roko. But it would merely be random value system X without any objective grounds to why it would be any better or worse than random value system Y. Thus highlithing the emptiness of the original statement at hand:
"Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth."
This statement then can be true only in a tautological sense (i.e. worthless), since under assumption A you arbitrarily set the standard for what 'something of worth' is by decree. Thus an AI having said standard would create a future of worth by circular definition.
Scenario 'Buddha', value 'compassion': AI creates all encompassing compassion within its light cone => future contains lots of worth in line with original definition
Scenario 'Hitler', value 'kill the Jews': AI kills all the Jews => future contains lots of worth in line with original definition
Scenario 'insane office supply otaku', value 'paperclips': AI converts everything into paperclips => future contains lots of worth in line with original definition
You see? :-)
@Stefan: "But it would merely be random value system X without any objective grounds to why it would be any better or worse than random value system Y."
- ah, I see, you are a moral realist. That's the source of our disagreement. I also used to be a moral realist too.
So, have you looked at Greene's paper "the terrible, horrible, no good truth about morality"?
http://www.acceleratingfuture.com/michael/blog/2009/03/the-terrible-horrible-no-good-very-bad-truth-about-morality/
Do you realize that what you are in fact saying is: 'Stefan, you being a moral realist invalidates the conclusions derived from assuming moral relativism to be true under assumption A.'
Roko, you are kidding - right? :-)
As I pointed out before: linking to a 100'000 word document is no substitute for a valid argument. And precisely if morals happen to be in line with assumption A - which I am in no way granting but you seem to suggest by linking to Greene's paper - the critique from my earlier post would still be valid.
The statement: "Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth."
Is tautological under assumption A and plain wrong under assumption B. Either way it is not adding anything in terms of insight or understanding.
May I ask why you are so reluctant to accept this fact?
Ok - so I actually downloaded the paper and took a look at it:
Statement 1: "First, I argue on metaphysical grounds against moral realism, the view according to which there are first order moral truths." - from the abstract
Statement 2: "proliferation of weapons of mass destruction, the disruption of the environment, etc" are "the great global problems of our time—the" - page 3
Wow - self contradicting within the first few pages. If statement 1 is true then on what grounds is Greene justified to conclude that "proliferation of weapons of mass destruction, the disruption of the environment, etc." are "great global problems of our time"?
However if "proliferation of weapons of mass destruction, the disruption of the environment, etc." are in fact "great global problems of our time" in what way is Greene justified to "argue on metaphysical grounds against moral realism"?
As they say: you can not keep the cake and eat it too.
"Is tautological under assumption A "
correct. But sometimes tautologies can be useful things to know. Um, I also had that objection. But there is an answer to it, to be found in Greene's paper. I should really go over this in a top level post. But I am about to move my blog to a new domain...
@Roko "But sometimes tautologies can be useful things to know."
Totally agree - sometimes they (propositional and modal logic, math, i.e. logical tautologies) can be useful under certain conditions and some more and some less so.
In this case however, we are dealing with a rhetoric tautology and consequently the statement conveys no useful information regardless of its length or complexity.
I really do not want to come across as a meanie here, but I am indeed surprise to what length you go to stick to a statement in light of so many contradictions. That is rationalizing instead of being rational.
From The Skeptics Guide to the Universe : Top 20 Logical fallacies:
Tautology
tautology is an argument that utilizes circular reasoning, which means that the conclusion is also its own premise. The structure of such arguments is A=B therefore A=B, although the premise and conclusion might be formulated differently so it is not immediately apparent as such. For example, saying that therapeutic touch works because it manipulates the life force is a tautology because the definition of therapeutic touch is the alleged manipulation (without touching) of the life force.
Post a Comment