Donald Hoffman: The Mathematics of Consciousness

uh well one thing that's come out of cognitive neuroscience is that perception is not simply like in the case of vision just a camera it's not that we take a picture of a pre-existing world rather it turns out that about a third of our brain's cortex is involved in visual perception and that's a lot of processing power for just taking a picture so what seems to be going on is that perception is really an active constructive process we're constructing the depths the colors the shapes the motions the textures that we see and we're doing it very very quickly we construct all of this visual world around us in you know when you open your eyes about it takes 100 milliseconds a tenth of a second or so to construct this whole visual world so we have the feeling that it was there all along because it's so quick so we actually fooled ourselves we're so good at this we think we're seeing the world as it really is we're seeing the world as we construct it so that's one of the big insights from cognitive neuroscience all of the above so cognitive neuroscience has really made it clear that um given the image at the eye for example and i speak about the eye because that's where i do my research is envision the image at the eye is a two-dimensional image so anytime you see in three dimensions there are an infinite number of three-dimensional interpretations that are in principle compatible with the flat image of the eye and that means that in a mathematical sense the problem can't be solved unless you bring in other constraints and so that's where memory or something or like experience and it could be a species memory in terms of things that have been programmed into us evolutionarily into our the the wiring of our brain i'm now speaking in typical cognitive neuroscience terms where they they speak of the brain as doing all of this stuff so the idea would be that built into the synaptic connections in the brain are these constraints that allow us to take this mathematical problem of getting 3d from 2d constrain that problem and actually get unique solutions and also to to get solutions for color and motion and shading and so forth absolutely the world perceived as 3d is is clearly a construct and we actually know the rules well enough that we can in virtual reality situations make you see any 3d construct that we really want you to see because we know the rules that are built into the system so one of the things that we try to do is to understand these rules with such mathematical precision that we can build robotic vision systems um so that computers can see and those are we're pretty successful at that now we can build robots that can drive cars down freeways pretty well without without a driver the construction of our perceptual world takes place in a cascade of neural representations and each one can contribute its own kind of illusions so it's not just really like two levels if you think about it really carefully there's many many levels of neural representation and there's wiring at each level so there are what there's wiring even in the retina where there are the so-called center surround cells that that have lateral inhibition and those contribute to certain visual illusions we have so if you have a sequence of of stripes of gray whether it's really dark and then not so dark and not so dark until you get all the way to white if you look at those you'll see that the edges of those stripes look a little bit brighter one edge and darker at another something called the mock band effect that's probably due to very early primitive processing in the visual system in the retina and maybe in in primary visual cortex but then as you go up the cortex there's two pathways there's the ventral pathway and the dorsal pathway and the ventral pathway itself has several way stations along it and so what you're going to get is at every weigh station there are built into the neural wiring these constraints that we've probably evolved over you know hundreds of thousands of years to have for how we are going to construct color construct edges construct depth construct motion and so forth so i wouldn't say that there's a two-stage boundary here like you know early and cognitive it's more gradual there's the very very primitive and then we see a sequence of representations till you get to what you might call the very high cognitive well there's that's one of the big questions now the standard way that the field looks at this is to say well um you know for in the case of vision for example you don't need eyes to have visual experiences right in fact if you you can close your eyes and if we stimulate your brain you you can have visual phosphine so you can have conscious visual experiences so in some sense your consciousness of vision is not just the process of the eyes and they would then say that that's evidence that it's the brain that's causing your conscious experiences and so what we do have and what the field is really good at right now are literally dozens perhaps even hundreds now of well-defined neural correlates of consciousness so that we where we know that this particular area of the brain has activity that's highly correlated with this particular kind of of conscious experience so for example area v4 of cortex a visual cortex is highly correlated with color perception so if you inhibit that area and the left say the left hemisphere with transcranial magnetic stimulation then you will lose all experience of color in the right visual field you will still experience color in the left visual field so here's a case where we can just put a magnet just touch it to your skull and your conscious experiences changes dramatically and we know where to touch it and what experience you will have so there's clear correlation between the conscious visual experience that you're having and the changes in it and the cortical activity that you're in this case inhibiting and if you do it on the upright hemisphere side then you lose visual you know color in the left visual world and the same is true for motion so there's an area in lateral temporal area that does motion so if you inhibit that with magnetic fields you'll lose the ability to see motion in the right visual field everything is like a stroboscope see a person there there but you don't see the motion in between fortunately when you take the magnet away the motion comes back and the color comes back so we have all these correlates but the question then the deeper question is does the correlation imply causation is it the case that all these very strong correlations and they're there imply that the brain is causing the conscious experiences now most people in the field think yes it's a slam dunk because the evidence is quite clear but the problem we have is that no one has been able to give not only a scientific theory but even any remotely plausible idea about how that could be how could sodium potassium calcium ions running back and forth through holes in neural membranes be my conscious experience of the color green uh or the smell of a rose or the sound of a trumpet with the there are a lot of so-called theories out there and there's you know they'll have all these interesting neural mechanisms like re-entrant thalamocortical loops um collapse of states of microtubules and so forth but then when you say where does the consciousness comes in and they try to say here comes consciousness there's always a gap right there there's a miracle that occurs between the reentrant thalamual cortical loops and the appearance of consciousness and no one has been able to fill that gap in um with the scientific theory not even with a plausible idea so the idea that the brain causes our conscious experiences is widespread most people in cognitive science would say of course but the embarrassing fact is we don't have a single scientific theory and no plausible ideas so i'm actually exploring a different direction you could think of it as a search strategy we're trying to find the solution to this problem how how is consciousness related to brain activity 99 of the researchers are searching in one part of the search space namely how does how could the brain cause consciousness and we're not getting anywhere so at least a couple percent of us should go to the other part of the search space and try the other direction let's try consciousness first and see if it leads to brain activity you know as a product so go the other direction and solve the mind-body problem that way it sounds implausible to my colleagues in cognitive neuroscience but hey as a search strategy it's implausible for us all to look in one part of the search space so let me go over here and look at this other part and we'll see you know who comes up with an answer when i actually go give talks at universities and conferences i present the mathematical model and that raises the level of the conversation if i didn't have that i'm sure i would be dismissed without without you know further ado um but with the mathematics then they have to take it quite seriously so so my burden then is different from the burden of the neuroscientists who say that brain causes consciousness they have to show how the brain and or activities of the brain or properties of neurons or loops of activity cause consciousness without a hand wave and they can't do that my burden is to start with consciousness and without any hand waves get first all of physics so i need to get um you know classical relativity theory i need to get quantum theory and i need to get relativistic quantum theory as a starting point and then hopefully an understanding of what how neurobiology arises out of it so for me what i have to do is first say what i mean by consciousness so i have to have a mathematical model and i've presented one at this conference that i call conscious agent and it's a very simple and for a mathematician my structure is actually quite simple it's just three probability spaces three maps between those spaces technically markovian kernels and then just one counter that counts that counts the number of experiences that you've had it's a very very simple mathematical structure so i propose that that captures that plus then all the combinatorics right that you can get by interacting different conscious agents together gives me a complete model for consciousness so it's an empirical claim and the claim is what i call the the conscious agent thesis that every aspect of consciousness can be represented as some system of interacting conscious agents that's the claim very very precise claim and i present it in the same manner as church touring thesis in the field of computation where turing came up with a model that's an abstract model of computation that's about as simple as my model it's different but it's a very he had i think it's a six or seven tuple turing machine that he defined i have a seven tuple conscious agent that i've defined and then turing said that this simple seven tuple that he defined the turing machine is capable of representing any effective procedure period anything that any computer could do could be done by his turing machine that's the church touring thesis if he's right then the church then the turing machine provides a simple formalism which is universal for all computation and theorems then can be built on that because it's so simple so we get the halting problem and busy beaver problem all these things that are actually solved because of during thesis we get computer science as a science and not just hacking because of this this thesis so i present my thesis the conscious agent thesis in the same spirit here's a fundamental definition of conscious agent the claim is that every aspect of consciousness every aspect can be represented by a single conscious agent or conscious agents in connection in interaction and so it's a it's a it's an empirical thesis in the same sense as touring's it can be falsified if someone can find some feature of consciousness that they show cannot be modeled by this by this formula i'm wrong and i have to either abandon the project or change my formalism i'll probably try to change the formalism first but so then the other thing i have is what i call the thesis of conscious realism which says that the world consists only of conscious agents so i have one definition of conscious agent and one thesis namely that um the world consists only of conscious agents and from that so one definition and one postulate i'm hoping by theorem and proof to get all of quantum physics and and a complete model of consciousness so to solve the mind-body problem in that way um without any hand wave as pure mathematical um consequences of the basic definition and postulate so theorem proof they're improved all the way through and what i presented in the talk this morning was i am able to show that the asymptotic behavior of conscious agents in interaction gives us precisely the same equations the the equations that describe that are mathematically precisely the same equations at one level that describe the wave equation of the quantum free particle i'm able to put them in precise correspondence so i can read off how physical variables like position time energy momentum um are related to precise variables of the asymptotic behavior of conscious agents so the claim is that particles in physics are our representation of asymptotic dynamics of consciousness so it's a the long-term asymptotics means long-term the long-term dynamics of consciousness is what we represent by these simple things particles that's the correspondence so that's a precise mathematical claim and that's the nice thing about having a mathematical formalism these claims are extremely precise and so they're extremely falsifiable that's right the particle would be a representation that a conscious agent uses as a shorthand to represent certain properties of this really complicated interaction of other conscious agents their interactions are quite complicated right but if you can summarize the long-term behavior of those interactions with just a few numbers you get what we call a particle so there literally could be an infinite number of discrete interactions among these agents you can't deal with all that infinite complexity of all those discrete interactions but you can see what is the pattern long term of those interactions and that we can characterize with a handful of numbers and that's what we call a particle in physics is quite interesting um point particles are a problem right because especially if you try to build a theory of quantum gravity as you go down to smaller and smaller spatial scales the energies go to infinity and things blow up so so they've always known that at some point the notion of a particle is going to have to give away um because it blows up on you that's one reason why they've gone to strings and membranes and so forth to to break out of that that that blow up but if you ask where do those come from well they'll they'll say that those are the fundamental entities that strings and membranes are just what is they're the fundamental they are vibrations in an dimensional space for example so they are they are like they i guess they would think of them as um oscillatory patterns or vibrational patterns um in higher dimensional you know objects the various particles are so but they will still have to postulate these are fundamental and that's true of any scientific theory right by the way you can't explain everything you always assume something uh and then you explain something else so you put so to speak all your miracles up front this is what i'm going to assume and and the game of science is to make sure you put all the assumptions up front then after that no more miracles are allowed right and then you have to get everything else without you know a miracle and that's that's the failure of the cognitive neuroscience approach right now to solving the mind body problem what they put up front is um unconscious matter space energy and so forth neurons and by the time they try to get to consciousness they need another miracle to get there i'm hoping to put up my definition of conscious agent as fundamental so i'm just claiming you know this is just the way the universe is it's fundamentally conscious agents live with it that's just the way it is and this is how they interact that's my postulate can i prove it well i can try to disprove it i can't prove it but what i can try to do is to look at the implications of this theory and in the scientific approach of falsifying things i can try to be as explicit as possible about the predictions and then see if i can falsify them so so i make assumptions about conscious agents and just say that's the way it is i don't know why i mean i didn't create this thing but you know maybe i did but um but that's just the way it is and then look at the implications and test them my idea that conscious agents are fundamental and that space and time and physical objects are not fundamental um is is not the same thing as claiming that there is no objective world it's it's claiming that the nature of that objective world is different than what most people think most of us think that the objective world is something that's spatial temporal and has physical objects in it with mass and energy i'm saying that there is an objective world in the sense that there's a world that would exist even if i as a conscious observer weren't here to observe it there i mean if i i mean i'm happy to buy life insurance that means i expect things to exist even after i die so i believe in an objective world in that sense but the objective world i claim is a world of conscious agents and what we call the physical world space and time and physical objects is simply a really dumbed down representation of that and the and the reason why we have that is each conscious agent has only a finite capacity so if i'm conscious age i have a finite capacity for experience and for decision and for action that means that most of the rest of the universe which is all conscious agents interacting i can't represent in its full glory in that sense i'm unconscious of it so you know the neuroscientists will say that 99 of your brain activity is unconscious activity yes that falls right out of this theory because conscious agents have finite capacity so most of the processing of all the agents that constitute what i call me is beyond the consciousness of the agent that's talking to you the agent that's talking to you right now has finite channel capacity so almost everything else in the universe i'm either unconscious of or i have really stupid low level representations of it that we call space time and objects actually one key part of the model is a mathematical thing called a kernel that goes from my experiences to my choice of actions and it's that part of the mathematical model is my precise mathematics to capture what i claim is the ultimate free will of the conversation so each agent in this theory has free will but the best that we can actually do in mathematics to model that is write down something called a markovian kernel which is just a list of probabilities if i have this experience then the probability i'll take this action is this this action and so forth if i have this experience then here's the probabilities for my actions that's the best we can do in mathematical models for the fundamental reality which is not mathematical of ultimate free will absolutely in fact i do i've published in collaboration with other studies fmri studies and eeg studies so my view does not mean that i don't want to do neuroscience i do neuroscience i read neuroscience i think it's extremely valuable we need to get our data wherever we can the question is how do we interpret that data are we trying to interpret it in a simplistic way almost it says that this shows these correlations show that brain activity causes consciousness or do we have to have a more sophisticated interpretation that that says correlation does not imply causation standard thing in statistics correlation does not imply causation and it's really true here brain activity correlated with consciousness does not entail logically that the brain activity causes consciousness that's everybody would have to grant that and then they'd have to grant that they've not had any success at actually getting a theory that goes from brain activity to consciousness so at least give me a chance to try to go the other direction one thing that comes out of this point of view is that we think of space and time and objects as the truth and our perceptions our visual perceptions are the truth but i've been doing evolutionary game simulations which show that true perceptions die or go extinct when they compete with perceptions that are just tuned to fitness so evolution does not favor true perceptions it actually drives them to extinction that's evolutionary game theory and genetic algorithms actually show that true perceptions never even arise they just never even get on the scene so that means that we need to reinterpret how we think about space and time and physical objects and i propose this interface theory of perception that says our perceptions are like the desktop on your computer if you have a computer with a little icon in the lower right hand corner that's blue and rectangular for a file that doesn't mean that the file itself in the computer is blue and rectangular if someone thought that that would mean that they really didn't understand what an interface is about the blue color is not trying to tell you that the file really is blue the shape of the icon is not trying to tell you the true shape of the file in fact color and shape are just the wrong categories for describing the file and yet the interface is useful not because it's true but because it hides the truth and it just guides behaviors that to do things you need to do write a paper edit an image and so forth so the idea is that space and time as we perceive them 3d space and time is our desktop physical objects are just the icons they're species specific icons that we've been shaped by natural selection through evolution to have to allow us simply to survive long enough to reproduce in our niche it's not about truth about having kids and so that's what it's all about we have to take our icons seriously i mean the the big objection to my point of view here would be to say well hoffman if you think that train coming down the track is just an icon on your desktop once you step in front of it and after you're dead and your theory with you will know that there's more to the train than just being an icon and the point is i won't step in front of the train for the same reason i wouldn't take an icon and drag it to the trash can accidentally or you know casually i don't take the icon literally it's not blue the file's not blue and rectangular but i do take it seriously if i drag that icon to the trash can i could lose a year of work similarly we've been shaped by evolution to have symbols like trains snakes cliffs that we better take seriously that's why we were shaped to have them don't take them literally that's a logical error taking them seriously yes but that does not logically entail that we must take them literally and yet most of us have the psychological urge to go from i better take it seriously to of course that means i have to take it literally and that's the mistake