mental imagery & the intuitive appeal of qualia

since i expressed my sympathy for illusionism on twitter (@hakwanlau) a couple of months ago, friends have questioned my loyalty to sanity, and asked if i have become **one of those philosophically-ignorant scientists** who deny that consciousness is a real phenomenon, without ever bothering to understand what such denial even means. i have very much enjoyed the exchanges. :-)

in a previous post i've explained a bit why exactly i see some promise in illuionishm. to me it's all about whether you can have a somewhat plausible positive story about how the illusion comes about. so here it is, as promised. it's a bit long (for a blogpost). i recommend you pair this with a glass of red wine. maybe Pinot Noir. or some cheap Merlot would do too.

***

if you’re lucky like i am, who speaks a non-Indo-European language as a first language, you might have found it hard to explain to people what we mean when we say we study consciousness. worse still are concepts like phenomenal consciousness or qualia. even for native English-speaking folks, these concepts aren’t immediately obvious and intuitive at all. so when people say certain views on consciousness are part of our unshakable intuitions, it is worth asking - whose intuitions? if it takes formal definitions to even introduce these basic concepts, perhaps the relevant ‘intuitions’ are just a direct result of our loading the dice in our definitions. so we need a better, more neutral way to get at this.

for English-speaking common folks, Nagel’s famous phrase often connects. this much i’ve learned from teaching undergrads who aren’t philosophy majors, especially those who have no inclination to ever become so. you can explain to them: you know, there is something it is like to have certain brain processes going on. like when you taste soy sauce, your brain doesn’t just  recognize what it is, there is something it is like for you to have that experience. but for some other brain processes, it just goes on without your noticing it. there is nothing it is like to have those brain processes going on in your head.

in my experience, with some patience, most English-speaking students can get this. but what exactly does the phrase really mean to them? that may be less clear. does it mean there really *is something* it is like to taste soy sauce - like, it is a *thing*? and what is a *thing* anyway? is money a thing? are Wednesdays a thing? certainly we don’t want to say there aren’t ever Wednesdays. but can we define Wednesdays in purely physical terms?

we can get ourselves into all sorts of trouble when we engage in this kind of thinking. instead, i find it more useful to think through what plausibly could actually go through their minds, when my students are asked to think about what it’s like to have certain experiences.

thinking about what it’s like

when we think about what it’s like to taste soy sauce, i take it that we just imagine having that experience. if we succeed in such imagination, we say: yeah there is something it is like to taste soy sauce. if i press you further: so what’s it like? you may say… well, it’s a bit salty, like sea water, but more viscous, with some taste of umami (if you know what it is), a bit almost like seafood, or mushrooms, but not quite. or anyway, it is tastier than sea water. it has more flavor.

that is to say, you compare it with other experiences you summon into your imagination. perhaps this makes sense, becoz, why else would you want to think about your experiences anyway. i suppose our brains aren’t designed for doing philosophy in the first place. usually, when you perceive something and you focus on it, you just end up thinking about things you perceive. you don’t really think about the experience itself. not often anyway. unless, you try to compare it with other experiences: hm… do i like the *taste* of this steamed fish with soy sauce? or would it have been better still if i added more ginger and scallion? how about cilantro? now, that is something worth thinking about. we should all have the brains to do that.

also, it would be a bit weird for people to say: yes there is something it is like to have a certain experience, and yet they can’t say anything at all what it is like in comparison with some other experiences. that experience would have to be really unique. and by pointing out there’s nothing like it, we also get a sense of how strange it must be anyway. 

perhaps this is the whole point in thinking about experiences. we put them in the space of imagination so as to compare them, concurrent experiences and summoned memories alike.

so when we try to get our undergrad students into the topic of consciousness, and ask them to think about whether there is something it is like to have a certain experience, i take it that this is all they do. they imagine having that experience, in the way they usually do when they compare these experiences in their imagination. 

knowing what it is like

in philosophy, of course, we also speak of whether one knows what it is like to have certain experiences. but again, what does it mean? is there really something to know, like a piece of knowledge, like someone’s birthday?

again, why not consider concrete everyday examples. when my senior students are asked: do you know what it is like to stay up drinking all night? the answer is either yes or no. for most people who say yes, some know because they have done it. but some others who haven’t done it may tentatively say, i think i know. that is, maybe they have tried drinking a lot over a long time, maybe during the day. (not that i recommend any of it). and, they may also have stayed up all night before. so they think, well, it’s just both of these experiences added together, no?

but are they correct in thinking that they may know? well, the proof will be in the pudding, like they say. so they may finally try it, and say: ah, this is exactly what i always thought it would be. or they may say: i was so wrong, this is much worse than i thought.

that is, if one is imaginative enough, and if one has experience with some related or similar experiences, one can certainly know what it is like to go through some experience without actually ever having it; David Hume was right when he talked about that ‘missing shade of blue’. to *know* what it is like to have certain experience is just to be able to anticipate having that experience, such that when you have it there will be no surprises. you can also compare that anticipated experience with other experiences in your imagination. at least that’s what we usually mean. and it is a very useful ability to have - for otherwise how else do we know whether we should accept invitations to stay up drinking all night? we’d better be able to anticipate what it would be like.

intrinsic qualia

if the above is more or less correct about how my undergrads think when they are introduced to these concepts…. and assuming there’s a chance some of them may become philosophy professors one day too (!) …. well, then, i get why certain technical theoretical posits related to consciousness may seem plausible to some, so much so that some people say certain things are ‘intuitive’ in the literature.

we can call one such posit intrinsic qualia, which are the private, ineffable, intrinsic properties of experience that are immediately apprehensible. 

so consciousness is the general phenomenon that there is something it is like to be in some mental states. some people call it phenomenal consciousness these days, though that term can also refer to some specific theoretical notion rather than just the general phenomenon. 

assuming we get the general notion through all of these talks of “what it is like”. why would we accept such a specific theoretical posit as intrinsic qualia? well, that’s becoz, given the way we think about what it is like, these qualia may seem harmless enough.

first, of course, some properties of our experience are private. makes total sense. if it is all about my own imagination, how are you ever gonna know mine? 

the important point here is i’m not saying your conscious experience *is* your imagination of that experience. but when you think about your conscious experience, when you think about what it is like to have that experience, perhaps all you do is to imagine having that experience. from there, one may - mistakenly or not - find it *intuitive* to think of properties of imagination as properties of the experience itself.

and then, of course, the content of one’s imagination may seem ineffable too. our language may just not have that fineness of grain. a lot can be going on. it outstrips our limited vocabulary.

and of course, one’s imagination is naturally immediately apprehensible. that’s why we imagine it in the first place. it’s all for our own appreciation. 

the trickiest - and also most critical - is whether some properties of our experience are intrinsic. and what does it even mean? not all philosophers agree on this usage, and in fact some disallow it, but some use the term intrinsic to mean that these properties are not relational, that they cannot be defined in more simple terms. they are what they are. they can’t be characterized fully in terms of something else.

perhaps the following is what people find plausible: since my imagination of experiencing soy sauce is certainly private, mine isn’t necessarily the same as yours. we can’t point to the same bottle of soy sauce and say: hey it’s the same stuff we’re tasting. my imagination of how it tastes like is up in my head, not yours. there must be something unique up there that isn’t necessarily gonna be anything like yours.

but of course, at the end of the day, you two are tasting the same soy sauce. you two are possibly just having the same experience of the same thing. that may be all there is.

but all the same, becoz we think of it in terms of our own imagination, we may well find it plausible that something intrinsic is involved. it’s not just about the soy sauce itself. it’s not the taste buds either. you can do the imagination without either of them being there at the moment. so it must be it’s something more abstract, something special, unique to oneself, up there in the mind itself.

so from there, we may find this notion of intrinsic qualia somewhat plausible. intuitive even, perhaps.

and of course, once we accept qualia as such, all sorts of metaphysical problems arise. if qualia are intrinsic, by definition no functional analysis of it will be possible. we can’t just talk about what our perceptual system represents, coz there is something inside that ultimately matters, in ways that can’t be understood from the outside. a piece of computer software will never have such magical stuff as qualia. something will always be missing.

phenomenal concepts

but what if - just what if - our minds are actually nothing but a piece of software, running on a biologically instantiated computer? a very special piece of software, mind you. so special that we are capable of imagining having certain experiences at will. that is, we can drive our perceptual programs in a purely top-down fashion, even though they are originally meant for bottom-up sensing of external information. but still, it’s really just a piece of software, implemented somehow using some organic stuff. so, there are really no intrinsic qualia as such. but we mistakenly think there are, becoz we can’t really think deeply about our experiences, other than imagining having them. that is, when we are asked to refer to the experience in our thought, we typically just simulate the whole thing, and mistake some properties of the simulation as the properties of the experience itself.

that would be somewhat congruent with two major / popular strategies for addressing some ‘classic’ philosophical problems of consciousness, e.g. the explanatory gap, the so-called Hard Problem, the Knowledge Argument etc. the idea is there may be an epistemic gap, but it doesn’t mean there is a metaphysical one. in other words, we only *think* that there is a problem because of the way we think about consciousness. 

on one such strategy, so some philosophers say this is because when we think about consciousness we deploy phenomenal concepts. these concepts refer to the perceptual states that are actually just physical representations; no magic involved. but when we use these concepts to think about the physical perceptual states, we think about them in terms of subjective experience. from there, we are somehow blocked from seeing that these perceptual states just are the same representations as they are described in physical terms.

many people aren’t sure if we really have these phenomenal concepts, or how they are supposed to work. sure one can think about the same stuff with different concepts. i can think of myself as my parents’ naughtiest child (true), or the author of these words (also true). these are different concepts, which refer to the same thing - me. and sure enough, if i think about myself in terms of my authorship here, i may not realize it is the same person as my parents’ naughtiest child; i may mistakenly think that my sister was the naughtier one. but the thing is, once we are explained in no uncertain terms how the two notions are related, we no longer have trouble connecting them. 

but when it comes to phenomenal concepts, for them to work as they should, they need to be stubbornly opaque. even when we are told that, actually, that red experience is really just these neurons firing, we can never cross that bridge and get to genuinely appreciate that they are the same thing. that part remains somewhat magical.

so perhaps, thinking about this in terms of imagination may help to flesh out what these phenomenal concepts really are. when we think about the experience of tasting soy sauce, we simulate that experience in our imagination. that’s how the so-called phenomenal concepts are deployed. perhaps then it makes sense that simulating having the experience is gonna feel rather different from describing the experience in mechanistic terms. for a start, simulating it actually is an experience in itself. it feels like something. describing it doesn’t feel quite the same.

mental quality in imagery

this may seem to get us into a bit of a funny chicken and egg problem. so thinking about an experience is an experience in itself. but, we were hoping to say that having an experience is no magic - specifically, by magic we mean something intrinsic that can’t be explained in purely functional / software terms. but now, maybe in thinking about an experience, because it feels like something, there is magic after all? 

the answer is no. so let’s focus on vision for a moment. the story i’ve been trying to tell is: when we think about the experience of seeing a cat, we typically create the mental image of a cat. now then, of course there is something it is like to have the mental image of a cat. there need not be magic. one way to go about this may be to say: well, there is something it is like to have experience X just when we can think of what it is like. maybe all that is, like what i said earlier, is that you succeed at imagining having that experience. so i certainly can imagine having the mental image of a cat. when i do so…. i sort of just end up having that mental image again. i can refer to it, compare it with other experiences, etc., so maybe that’s all there is.

another thing is that imagery and conscious perception share common / similar mental qualities. so these qualities may be defined roughly in terms of how a certain stimulus can be distinguished perceptually by the subject against other perceptible stimulus (cf Rosenthal on ‘mental quality space’). so a cat looks the way it does, because it looks more like a small tiger than a squirrel. more like a dog than a piece of rock. more like a human than a tree, and more like a tree than a giant blue triangle, etc. etc. etc. etc. (until you exhaust the whole list of all the things you can possibly see). now then, imagery has a similar quality too. an imagined cat likewise looks more like some of these same imagined things and less like some others. so mental imagery may share a similar structure of mental qualities too. and having these qualities maybe the crux of all this talks of ‘what it is like’ - it’s really about whether you can compare it with other things you see. there’s just no absolute, pure conscious sensation that can’t be compared against anything else.

so we already accepted that there is something it is like to see something, and there’s no magic. well, imagery shares some similar mental qualities. and in both cases, you can think about what it is like to have them, by imagining having them. so perhaps there’s no magic there in either case. mental imagery is likewise just generated by some software implemented in the brain.

one problem is: mental imagery and actual perception are only somewhat similar. but they are also somewhat different in terms of how they feel. they certainly aren’t exactly the same. the question is, how so? my answer has two parts. first of all, when you actually see something you see it as being out there, present in the world. when you imagine seeing it, you don’t. it lacks this sense of presence, or assertoric force, as philosophers sometimes say. i have previously argued that this is part of the phenomenology of seeing. and to account for that difference all you need is software.

in fact, i’d go so far to say when you consciously see a cat you are basically interpreting that mental quality (i.e. the thing that looks more like a small lion than a squirrel, etc. etc.) as reflecting the present state of the world, *meaning* that it does not just reflect your own imagination. when you enjoy the mental imagery, on the other hand, you take that very similar mental quality to reflect your own imagination, *meaning* that it does not reflect the present state of the world. it’s all part of the phenomenology itself. and it may be why we need the phenomenology at all - the two modes of representations need to feel immediately different somehow, for otherwise the confusion would be dire. the implicit linkage between these two modes of representation may be what causes much of the confusion: perception may seem magical because we tend to think of it in terms of imagery, and imagery may in turn seem magical, coz it’s almost as if you’re seeing something consciously. but really it’s just the other side of the same coin. in both cases they just represent the perceptual object. there's no internal magic going on.

there is, of course, another difference between imagery and actual perception. people have suggested that imagery is less vivid, somewhat faint. i think that’s right. but other people have also argued that the notion of vividness isn’t so easy to define. i don’t find it so problematic. we can perhaps define it as how similar it looks with respect to not seeing anything. so a less vivid image of a cat is just more similar to nothing than a more vivid image of a cat. 

this way, we account for why imagery has a lower fineness of grain too. that is, in imagery it seems to be less detailed. when one thinks of a cat, it doesn’t seem to be quite so specific about its color, compared to seeing a real cat. if the cat looks stripey, there’s also the famous observation that we can’t ever really count the number of stripes on the imagined cat. but if the imagined cat is less vivid in the sense that it is more similar to nothing, than a more vivid image of a cat is…. well, then, by rather simple psychophysical principles we can understand why it is not so distinguishable from other images of cats in imagery space. the idea is simple: take two gabor patches or slightly different orientations. as the luminance contrast decreases, they become less distinguishable. so vividness may be a bit like contrast, in that it reflects how strong the signal is, and there is a clear point of zero which refers to the absence of any stimulus. when the signal is tiny, as in imagery, you can’t distinguish things very well. but it still roughly has the same content, just less clear.

illusionism

so we addressed why there is something it is like to engage in mental imagery. there need not be any magic involved, just like there need not be magic involved in normal perception either. and there may be some shared qualities between imagery and actual perception. but does it mean that imagery (or more generally, imagination in any other sensory modality) is the correct way to think about the experience?

earlier i mentioned that there are two major / popular strategies for dealing with those ‘classic’ philosophical problems of consciousness. one is the phenomenal concept strategy. the other is illusionism. the two strategies are often seen as mutually exclusive, or in competition somehow. but i don’t really think that way. they are different, but the difference may be rather subtle.

so on illusionism, when we think about our experiences, we are actually somewhat mistaken. how bad that mistake is … depends on the version of the theory. one version is ultra-strong illusionism, which nobody really believes. it’s just set up as a strawman by its opponents. according to this version, we only think we are conscious, but we never really are. not in any way. but nobody really believes that version of illusionism, not really. 

on weak illusionism, which even some people who openly hate illusionism endorse, we are only mistaken in thinking that there is qualia. but we aren’t wrong in thinking that we are conscious somewhat. 

so, obviously, the question is how we are conscious somewhat, exactly. so we’ve agreed that there’s something it is like to be in some mental states, e.g. seeing a cat. it is so, in the sense that the question of - so what is it like? - make sense. there is something to be said about it. on the other hand, if i ask you: so what is it like to have some of your V1 neurons firing at 10hz, below perceptual threshold? it should draw a blank. there is nothing it is like. not just becoz of the way i posit the question, assuming you know some neuroscience. if i actually stimulate your brain that way, you also wouldn’t be able to tell me what it is like. there is just nothing it is like to be stimulated that way. so, some states are conscious, some aren’t. that’s a meaningful distinction worth keeping.

but the question is, when we think about what it is like to see the cat, are we thinking about the experience itself, really? or are we just engaging in a different activity altogether, i.e. summoning a mental imagery, which shares some aspects but not all aspects with the experience itself? if imagery is a valid way of thinking about the experience, then the phenomenal concept strategy is ok. but if imagery is just a way to think about something close enough to the experience, thereby allowing us to do some thinking about the experience, approximately… well, then, this may be good enough for evolutionary purposes, but you aren’t really thinking about the experience itself. not exactly.

this is not at all to say you are mistaken in thinking that you’re conscious. you think you are and i respect that. who am i to say otherwise. you say there is something it is like to see a cat, and i believe you. there probably is indeed something it is like, as opposed to nothing, as in the case of subthreshold brain stimulation mentioned above. but when you think about what it is like, perhaps you aren’t really just thinking about the experience of seeing a cat - not exactly. you also invoke something else. you’re simulating it, rather than just thinking about it. maybe that’s becoz there’s just no other way for you to think about *it*, other than to simulate it. even when you are seeing the cat right now, if you focus on it you just think about the cat. to think about the experience itself is an awkward thing, and perhaps it can really only be done via something like imagery. but a mental simulation is a complex process. it maybe somewhat similar to the experience itself, but also somewhat different. so you may very well be mistaken about some aspects of it. like all those qualia stuff, which ultimately lead to all of those age-old problems of consciousness.

this is a bit like saying, there’s no way for me to think about the pandemic without getting deeply emotional about it. this is especially the case when i think about the various politicians’ reactions to it, including those who exploit the ‘opportunity’ as a moral free-for-all. but my feeling emotional may have nothing to do with the pandemic itself. it’s just me. but i can’t help thinking about it this way, and this may color my thinking about the pandemic. if i ended up finding it intuitive that there’s something emotional about the virus itself though, i would be very much mistaken.

aphantasia

so we now know why we may be somewhat mistaken in thinking about consciousness - something i called ‘illusionishm’ in a previous blogpost - coz exactly how mistaken we are is…. well it is what it is. it does not deny that there is something it is like to have certain brain processes going on in your head. but it denies qualia. 

and i have given an account of why we may be so mistaken to think there are such things as qualia too, which give us all the troubles. ah well, maybe an account is a bit of an exaggeration. it’s more like a just-so story. but a story is better than no story.

it has not escaped my imagination that the foregoing implies some predictions about aphantasia, the condition where some people may not have any visual imagery. if a theoretical view makes no new predictions, it is somewhat worthless (to me). so take this as a ‘pre-registration’ of the most informal kind: we expect that aphantasics may not have the same intuitions about qualia, specifically in the sensory modality in which they show aphantasia. we are on it. we are the empirical folks after all.

stay tuned.

***

i will try to reply to comments and questions on twitter - https://twitter.com/hakwanlau/status/1253779189527834624?s=20