On Friday, I had the pleasure of meeting with Professor Richard Jones who (blogs at Soft Machines) at his Sheffield office. This was motivated by a rather tedious blog-comment argument over on accelerating future, mostly between me and Dale Carrico. In that discussion, Richard briefly commented in support of some of the things that Dale said, so I was curious to meet Richard - a prominent scientist and critic of certain (somewhat careless) claims made by proponents of advanced nanotechnology research.
In my meeting with Richard, I introduced him to some ideas from Eliezer's school of thought. In particular, the following two ideas were pretty important, I felt:
1. That the way we evolved from less intelligent primates indicates that humans are roughly as stupid as it is possible to be without being too stupid to have a functioning society. - Richard was initially skeptical about this position, but on further reflection we agreed that the rate at which technological society had developed from the onset of agriculture to today was so fast on an evolutionary timescale that we could not reasonably expect a modern human newborn to have significantly more cognitive ability and potential than a hunter-gatherer newborn from 13,000BC. Another way of putting this is that humans are at or close to the very beginning of the hierarcy of intelligent life, not the end point.
This position has some very important implications:
- It is very likely that, if there is an ultimate limit on how intelligent a system could be in our universe (for example, due to the Bekenstein bound), then it is likely to be a very long way above the human level of intelligence. This is purely a probabilistic argument: it is unlikely that such a bound would be above the minimum required for organized society, but only just above that minimum.
- Many technical problems that seem very hard to us would probably be fairly trivial to intelligences that are genuinely smarter than us, because these intelligences would be able to improve themselves deliberately and would therefore not be bound by the slow evolutionary timescale of human intelligence improvement. We should thus expect smarter than human AI to be very powerful indeed. For example, indefinite life extension of human beings would probably fall into the "trivial" category for smarter than human AI. (Why? Well, mere humans have extended their lifespans by a factor of 3, with a consistent increase of 1 year per decade at the time of writing. Given that humans are, on a cosmic scale of intelligence, idiotically stupid, it seems highly likely that a self-improving smarter than human AI bound only by the laws of physics would be able to solve the problem). Similar comments apply to claims like space colonization, revival of cryonics patients, etc.
3. There is an important role for transhumanist and singularitarian advocacy which is not science, but political activism. The role of such activism is not to distort scientific findings to make "sceince fiction - like" claims seem more plausible, but to influence the order in which science is done, i.e. to influence priority.
For example, suppose $100 million available today can either be spent on studying the many species of parrots that live in the Amazon Basin, or that same $100 million in 2009 can be spent developing safe smarter than human AI. Science itself cannot decide which piece of research to do now - there is no scientific answer to that question. It is necessarily a political question.
And the singularitarian position is that it is better - by the standards of any sane human being - to spend that money on safe smarter than human AI research.
Overall, it was a pleasure to speak with Richard. We agreed on pretty much everything that we discussed - which was a breath of fresh air compared to the rivalrous feel of the Accelerating Future comments thread from which the visit was inspired.
Comment on this post on Facebook [You have to create a Facebook account and add me as a friend for this to work, but by doing so you'll be able to network with other people interested in the future of humanity, and you'll help to inform other people on Facebook about what is going on]
Posts
18 comments:
Roko, I like the energy, enthusiasm, intelligence and other qualities you display in support of an increasingly beneficial increasingly technologically supported future.
And, similar to your flirtation with the prospects for an objective morality, I see you still open to growth beyond the "truth" you presently conceive.
In the same light, I would urge you to try to see beyond Dale's strident and abrasive rhetoric, with all its flourishes redolent of Berkeley and the humanities and consider that he's trying so hard because maybe he *does* have an important point to convey to those who inhabit a different world with a different language and culture.
I see him repeatedly making the same observation, deeply relevant to your effectiveness in the here and now, but achieving much the same impact as an amphibian pointing out to deep sea creatures that they are in fact immersed in and dependent on "water" of which they are unaware, and furthermore, that there's a world above the "water" supporting even greater diversity and growth.
In contrast with Dale, my background is of one steeped in science and engineering and business management, but with a philosophical aspect that allows me to relate to--or alienate--both sides of the fence.
Wishing you ongoing growth, full of surprises.
@ Jef:
Thanks!
So, can you translate Dale's point into science speak?
oh, and I meant to disable this comments section, and intended people to use the "comment on this post on facebook" button. Since I forgot to do this, I'll keep the blogger comments open for this one post, but I'd be interested in people's opinions on the pros/cons of commenting on facebook
"Richard was initially skeptical about this position, but on further reflection we agreed that the rate at which technological society had developed from the onset of agriculture to today was so fast on an evolutionary timescale that we could not reasonably expect a modern human newborn to have significantly more cognitive ability and potential than a hunter-gatherer newborn from 13,000BC."
Roko, this is dead wrong. Many cognitive traits have quite a lot of heritable variation, and in an environment where variation in those traits contributed significantly to fitness you could get significant differences over a few dozen generations, let alone hundreds (intense artificial selection can enhance mouse, dog, or fox intelligence in a handful of generations). If selective pressures on cognitive traits were weak, that might not have happened, but you can't rule it out based on the time scale.
See Cochran and Harpending's "10,000" year explosion, and "Farewell to Alms" for a more detailed treatment of serious hypotheses about significant cognitive changes as recently as the last 1,000 years.
@ Roko: "So, can you translate Dale's point into science speak?"
Yes, but any such message would be incomplete due to the bandwidth being insufficient to convey critical context.
If we were face to face, I could respond to your individual assertions about "reality" (those presently at issue) and often provide you with greater context which, without denying your "truths" as presently perceived, would facilitate your ongoing construction of a more encompassing, more pragmatic "truth" we each would agree would promote greater effectiveness. Lather, rinse, repeat--with the more you learn, the more questions you can ask.
Apologies if that appears vague, mystical or arrogant--it's none of those, but it's ironically and self-referentially symptomatic of the very problem it addresses.
Jef: "If we were face to face, "
- do you skype?
@Roko: "do you skype?"
As a rule, no. And I try to keep my occasional blog communications to within 5 paragraphs.
Really, I don't have the personal bandwidth, given that I'm running a business, writing software, performing research, nurturing a marriage, typically have a stack of over thirty books and hundreds of technical papers to read, and I'm especially challenged to make time for physical exercise...
But if and when you and I can get to the same city for a futurist event or otherwise, it would be fun to spend some hours together.
@Jef: This is unfortunate...
You leave me tantalized that there's something really major I'm missing - but then say that you don't have time to tell me what it is!
Can you at least give me a clue? Even a 500 word incomplete exposition which you could write in 5 minutes would be better than nothing...
Anyway, very best to you with your marriage, business, etc.
@Roko:
It's major in the sense of the "water" in the worldview of the deep sea fish earlier.
It's major in the sense of recognizing when it seems that someone is trying to get to the moon by climbing ever higher trees.
It's major in the sense that belief in a flat earth made it perfectly reasonable to believe in a hell below, and a heaven above.
It's major in the sense that knowing kinds of questions that don't make sense can be just as valuable as knowing answers to questions than can.
It's major in understanding that achieving goals is only a special case of promoting values.
It's major in understanding that while the notion of absolute Truth is incoherent¸ it's true (from within the system) that increasing coherence over increasing context tends toward the good.
And so on...
@Jef: Thanks for your efforts.
You speak in riddles and allegories - I have no idea what any of your last comment actually means in the context of my discussion with Richard or Dale. At least what Dale says is (mostly) coherent but false!
@ Roko:
I tried to give you the most meaningful 500 words I could.
I tried to make each example (allegory, riddle) build upon the preceding one.
What do they all have in common? A lack of appreciation for the importance of context.
The dynamical regularities of the deep sea fish environment, from the point of view of the fish, can be well modeled without reference to the ubiquitous water. In this (admittedly fanciful and idealized) thought experiment, the simplest model would have the water factored out in every case. In real life science as well, it's natural to ignore context, but we shouldn't forget that its always present.
In my former career, I managed teams of engineers, including PH.D.'s, many of whom were inclined to ignore context. They would apply their considerable intelligence to understanding the instrument, and if an anomaly was observed, they tended to look closer and closer into its workings. A fair amount of the time, the problem was not "in" the instrument, but an emergent effect of its interaction with its environment. Temperature, humidity, vibration: Okay, these environmental effects are all mentioned in the manual. But what about a spider walking across a mirrored optic? What about acoustic noise from the meeting room next door, where noisy meetings were held only after 5pm and no one was looking at the instrument running unattended. What about a myriad of complex and subtle operator errors? These are all parts of the environmental context, but tending to be virtually invisible to the most "rational" of my engineers.
I had an exchange with a highly "rational" denizen of the Extropy list a few months ago. He was trying to understand a simple voltage-follower transistor circuit, but couldn't understand why the output, measured across a 1k resistor, was half what he expected. After DAYS of studying the circuit, he was stumped.
The answer was simple, the circuit operated within the context of a load with an impedance of 1k ohms. There was the mysterious voltage divider he couldn't see, right in front of him the whole time.
...
In your essay here, you say: "...indefinite life extension of human beings would probably fall into the "trivial" category for smarter than human AI. (Why? Well, mere humans have extended their lifespans by a factor of 3, with a consistent increase of 1 year per decade at the time of writing."
This appears to be an example of "getting to the moon by climbing ever higher trees." At some point, the context changes. It always does. It must. Any quantitative change, sufficiently extended, becomes a qualitative change implying an "emergent" effect due to context. IN the case of life extension, we observe roughly steady improvement as our technology deals with various disease and disfunction. At some point, the context changes and it's much more about genetics. Further along genetics is not so much an issue but it comes down to information-theoretic concerns of error detection and correction, strategies for robustness via reduncy, backups, spatial and temporal diversity. At some point the context has changed such that the question of "who" or "what" is being "preserved" rises to prominence.
Obvious extrapolations would have us updating the software on our refrigerators eventually. Do you think that is obviously bound to be the case, or might the context change sufficiently to make the question of refrigerators irrelevant?
Belief in a flat earth made it quite reasonable to conceive of hell below and heaven above. Of course, we know that was silly, but within context it was reasonable and serious.
Fast forward to the early 21st century. People talking about the reasonableness of cryonic preservation. Nothing fundamentally wrong with the fundamental technical theory--everyone acknowledges that it's presently high risk and there's a lot more development needed, but nothing visibly insurmountable. But what about context? What, exactly, is it that people intend to "preserve"? Very few recognize the impact and importance of context to the present sense of "self", nor do they appreciate the inevitable effect on the perceived sense of self were it to be migrated with environmental change much more drastic than a fish out of water (with the necessary technical apparatus for breathing, of course.)
And the ready answer, is that they could alternatively "experience" life in a virtual environment (again nothing fundamentally wrong) but now with an assumption of reified "experience" as if it were something that could be transported, measured, valued. If that were the case, why not simply replay just a few seconds of "peak experience" for as long as it could be sustained? Better yet, play a hundred, thousand, a million copies of that experience in parallel, thus maximizing the expected utility? Few see the incoherence of that concept, lacking in appreciation of context to any question of value.
With over twenty years managing Field Service engineers, I observed that the least experienced were generally the most sure of any situation. And in fact their efficiency correlated with their certainty--inversely. They almost had an idea of what to do, were highly action-oriented, and critical of old-timers like myself who would do nothing much more than speak in riddles, asking apparently irrelevant questions, and then somehow know what to ignore. What was left was usually the solution. If not, then enlarge the context.
If the context is already well defined, such as with arithmetic, video games, most school work, then go directly for the answers using your best-known methods.
If the context is open and not well-defined, such as investing in equities, troubleshooting complex systems within a complex environment, or predicting the future, a smarter strategy involves ruling out the improbable before trying to identify the probable.
Many "futurists" tend to get this backwards, again for lack of appreciation of the context.
Achieving goals as only a special case of promoting values:
Over and over again naive but enthusiastic futurists will talk with certainty about "goals" to be realized in an increasingly uncertain future context.
Now I realize that virtually every self-help book, and every course on project management, emphasizes the vital importance of making your goals specific and crystal clear. And they're right, but only to the extent that the context is well-defined (as with well-formed goals for performance or efficiency within business, or school, or perhaps in terms of exercise or weight-loss.) All of these entail a well-defined context.
But then enthusiastic futurists apply these assumptions to their thinking about plausible futures, not realizing tat the future context is inherently uncertain.
Most of the time it's just talk among enthusiasts, so the net consequences are negligible. But to the extent that real resources are going into planning and development for "goals" that can't be defined without a well-defined context, then those resources, and the associated opportunity costs, are wasted.
Much wiser to apply limited resources, in the here and now, promoting evolving values, to create a future yet to be discovered.
I've spent much more time here than I planned, and by rough typing off the top of my head I may initiated a process generating more heat than light. I hope that's not the case.
I've left the last of my "riddles and allegories" unexpanded. Partly because that one would take more time than the others and partly as a test of whether these efforts will be considered worth reciprocating.
@Jef: Thanks. Your comments are most insightful (though there are a few non-sequiturs in there).
But they seem pretty unrelated to the discussions with Dale or Richard Jones.
This appears to be an example of "getting to the moon by climbing ever higher trees." At some point, the context changes. It always does. It must. Any quantitative change, sufficiently extended, becomes a qualitative change implying an "emergent" effect due to context. IN the case of life extension, we observe roughly steady improvement as our technology deals with various disease and disfunction. At some point, the context changes and it's much more about genetics. Further along genetics is not so much an issue but it comes down to information-theoretic concerns of error detection and correction, strategies for robustness via reduncy, backups, spatial and temporal diversity. At some point the context has changed such that the question of "who" or "what" is being "preserved" rises to prominence.Depending on how you choose to interpret "indefinite" this is plausibly a fair point. But really it is tangental to my main argument. The fact that "who" or "what" is being "preserved" rises to prominence is just another argument that smarter than human AI has the ability to satisfy our desires to a very high degree.
Roko, it was a pleasure to meet you too, and I'm sorry the meeting was rather brief - no doubt we would have found more to disagree with had we talked longer.
Let me make a few quick comments on the points you mention. Firstly, and this, I think, gets to the heart of what Dale is saying, it's not at all clear that enough conceptual heavy lifting has been done to support the simple notions of intelligence that your arguments depend on. Secondly, while I agree that in principle you can discuss advanced AI entirely independently of Drexlerian nanotechnology, in practise Kurzweilian projections of the timescales on which brains can be scanned in sufficient detail rely on some kind of radical nanotechnology to emerge that can make practical that which with today's technology is very far out of reach. Finally, you're welcome to think in terms of singularitarianism as political activism that seeks to steer the priorities of science towards its goals, but don't be surprised if I don't agree with your assessment of what those priorities should be.
Post a Comment