epistemology

All posts tagged epistemology

Appendix A: An Imagined and Incomplete Conversation about “Consciousness” and “AI,” Across Time

Every so often, I think about the fact of one of the best things my advisor and committee members let me write and include in my actual doctoral dissertation, and I smile a bit, and since I keep wanting to share it out into the world, I figured I should put it somewhere more accessible.

So with all of that said, we now rejoin An Imagined and Incomplete Conversation about “Consciousness” and “AI,” Across Time, already (still, seemingly unendingly) in progress:

René Descartes (1637):
The physical and the mental have nothing to do with each other. Mind/soul is the only real part of a person.

Norbert Wiener (1948):
I don’t know about that “only real part” business, but the mind is absolutely the seat of the command and control architecture of information and the ability to reflexively reverse entropy based on context, and input/output feedback loops.

Alan Turing (1952):
Huh. I wonder if what computing machines do can reasonably be considered thinking?

Wiener:
I dunno about “thinking,” but if you mean “pockets of decreasing entropy in a framework in which the larger mass of entropy tends to increase,” then oh for sure, dude.

John Von Neumann (1958):
Wow things sure are changing fast in science and technology; we should maybe slow down and think about this before that change hits a point beyond our ability to meaningfully direct and shape it— a singularity, if you will.

Clynes & Klines (1960):
You know, it’s funny you should mention how fast things are changing because one day we’re gonna be able to have automatic tech in our bodies that lets us pump ourselves full of chemicals to deal with the rigors of space; btw, have we told you about this new thing we’re working on called “antidepressants?”

Gordon Moore (1965):
Right now an integrated circuit has 64 transistors, and they keep getting smaller, so if things keep going the way they’re going, in ten years they’ll have 65 THOUSAND. :-O

Donna Haraway (1991):
We’re all already cyborgs bound up in assemblages of the social, biological, and techonological, in relational reinforcing systems with each other. Also do you like dogs?

Ray Kurzweil (1999):
Holy Shit, did you hear that?! Because of the pace of technological change, we’re going to have a singularity where digital electronics will be indistinguishable from the very fabric of reality! They’ll be part of our bodies! Our minds will be digitally uploaded immortal cyborg AI Gods!

Tech Bros:
Wow, so true, dude; that makes a lot of sense when you think about it; I mean maybe not “Gods” so much as “artificial super intelligences,” but yeah.

90’s TechnoPagans:
I mean… Yeah? It’s all just a recapitulation of The Art in multiple technoscientific forms across time. I mean (*takes another hit of salvia*) if you think about the timeless nature of multidimensional spiritual architectures, we’re already—

DARPA:
Wait, did that guy just say something about “Uploading” and “Cyborg/AI Gods?” We got anybody working on that?? Well GET TO IT!

Disabled People, Trans Folx, BIPOC Populations, Women:
Wait, so our prosthetics, medications, and relational reciprocal entanglements with technosocial systems of this world in order to survive makes us cyborgs?! :-O

[Simultaneously:]

Kurzweil/90’s TechnoPagans/Tech Bros/DARPA:
Not like that.
Wiener/Clynes & Kline:
Yes, exactly.

Haraway:
I mean it’s really interesting to consider, right?

Tech Bros:
Actually, if you think about the bidirectional nature of time, and the likelihood of simulationism, it’s almost certain that there’s already an Artificial Super Intelligence, and it HATES YOU; you should probably try to build it/never think about it, just in case.

90’s TechnoPagans:
…That’s what we JUST SAID.

Philosophers of Religion (To Each Other):
…Did they just Pascal’s Wager Anselm’s Ontological Argument, but computers?

Timnit Gebru and other “AI” Ethicists:
Hey, y’all? There’s a LOT of really messed up stuff in these models you started building.

Disabled People, Trans Folx, BIPOC Populations, Women:
Right?

Anthony Levandowski:
I’m gonna make an AI god right now! And a CHURCH!

The General Public:
Wait, do you people actually believe this?

Microsoft/Google/IBM/Facebook:
…Which answer will make you give us more money?

Timnit Gebru and other “AI” Ethicists:
…We’re pretty sure there might be some problems with the design architectures, too…

Some STS Theorists:
Honestly this is all a little eugenics-y— like, both the technoscientific and the religious bits; have you all sought out any marginalized people who work on any of this stuff? Like, at all??

Disabled People, Trans Folx, BIPOC Populations, Women:
Hahahahah! …Oh you’re serious?

Anthony Levandowski:
Wait, no, nevermind about the church.

Some “AI” Engineers:
I think the things we’re working on might be conscious, or even have souls.

“AI” Ethicists/Some STS Theorists:
Anybody? These prejudices???

Wiener/Tech Bros/DARPA/Microsoft/Google/IBM/Facebook:
“Souls?” Pfffft. Look at these whackjobs, over here. “Souls.” We’re talking about the technological singularity, mind uploading into an eternal digital universal superstructure, and the inevitability of timeless artificial super intelligences; who said anything about “Souls?”

René Descartes/90’s TechnoPagans/Philosophers of Religion/Some STS Theorists/Some “AI” Engineers:

[Scene]


Read more of this kind of thing at:
Williams, Damien Patrick. Belief, Values, Bias, and Agency: Development of and Entanglement with “Artificial Intelligence.” PhD diss., Virginia Tech, 2022. https://vtechworks.lib.vt.edu/handle/10919/111528.

As of this week, I have a new article in the July-August 2023 Special Issue of American Scientist Magazine. It’s called “Bias Optimizers,” and it’s all about the problems and potential remedies of and for GPT-type tools and other “A.I.”

This article picks up and expands on thoughts started in “The ‘P’ Stands for Pre-Trained” and in a few threads on the socials, as well as touching on some of my comments quoted here, about the use of chatbots and “A.I.” in medicine.

I’m particularly proud of the two intro grafs:

Recently, I learned that men can sometimes be nurses and secretaries, but women can never be doctors or presidents. I also learned that Black people are more likely to owe money than to have it owed to them. And I learned that if you need disability assistance, you’ll get more of it if you live in a facility than if you receive care at home.

At least, that is what I would believe if I accepted the sexist, racist, and misleading ableist pronouncements from today’s new artificial intelligence systems. It has been less than a year since OpenAI released ChatGPT, and mere months since its GPT-4 update and Google’s release of a competing AI chatbot, Bard. The creators of these systems promise they will make our lives easier, removing drudge work such as writing emails, filling out forms, and even writing code. But the bias programmed into these systems threatens to spread more prejudice into the world. AI-facilitated biases can affect who gets hired for what jobs, who gets believed as an expert in their field, and who is more likely to be targeted and prosecuted by police.

As you probably well know, I’ve been thinking about the ethical, epistemological, and social implications of GPT-type tools and “A.I.” in general for quite a while now, and I’m so grateful to the team at American Scientist for the opportunity to discuss all of those things with such a broad and frankly crucial audience.

I hope you enjoy it.

So with the job of White House Office of Science and Technology Policy director having gone to Dr. Arati Prabhakar back in October, rather than Dr. Alondra Nelson, and the release of the “Blueprint for an AI Bill of Rights” (henceforth “BfaAIBoR” or “blueprint”) a few weeks after that, I am both very interested also pretty worried to see what direction research into “artificial intelligence” is actually going to take from here.

To be clear, my fundamental problem with the “Blueprint for an AI bill of rights” is that while it pays pretty fine lip-service to the ideas of  community-led oversight, transparency, and abolition of and abstaining from developing certain tools, it begins with, and repeats throughout, the idea that sometimes law enforcement, the military, and the intelligence community might need to just… ignore these principles. Additionally, Dr. Prabhakar was director of DARPA for roughly five years, between 2012 and 2015, and considering what I know for a fact got funded within that window? Yeah.

To put a finer point on it, 14 out of 16 uses of the phrase “law enforcement” and 10 out of 11 uses of “national security” in this blueprint are in direct reference to why those entities’ or concept structures’ needs might have to supersede the recommendations of the BfaAIBoR itself. The blueprint also doesn’t mention the depredations of extant military “AI” at all. Instead, it points to the idea that the Department Of Defense (DoD) “has adopted [AI] Ethical Principles, and tenets for Responsible Artificial Intelligence specifically tailored to its [national security and defense] activities.” And so with all of that being the case, there are several current “AI” projects in the pipe which a blueprint like this wouldn’t cover, even if it ever became policy, and frankly that just fundamentally undercuts Much of the real good a project like this could do.

For instance, at present, the DoD’s ethical frames are entirely about transparency, explainability, and some lipservice around equitability and “deliberate steps to minimize unintended bias in Al …” To understand a bit more of what I mean by this, here’s the DoD’s “Responsible Artificial Intelligence Strategy…” pdf (which is not natively searchable and I had to OCR myself, so heads-up); and here’s the Office of National Intelligence’s “ethical principles” for building AI. Note that not once do they consider the moral status of the biases and values they have intentionally baked into their systems.

An "Explainable AI" diagram from DARPA, showing two flowcharts, one on top of the other. The top one is labeled "today" and has the top level condition "task" branching to both a confused looking human user and state called "learned function" which is determined by a previous state labeled "machine learning process" which is determined by a state labeled "training data." "Learned Function" feeds "Decision or Recommendation" to the human user, who has several questions about the model's beaviour, such as "why did you do that?" and "when can i trust you?" The bottom one is labeled "XAI" and has the top level condition "task" branching to both a happy and confident looking human user and state called "explainable model/explanation interface" which is determined by a previous state labeled "new machine learning process" which is determined by a state labeled "training data." "explainable model/explanation interface" feeds choices to the human user, who can feed responses BACK to the system, and who has several confident statements about the model's beaviour, such as "I understand why" and "I know when to trust you."

An “Explainable AI” diagram from DARPA

Continue Reading

I’m Not Afraid of AI Overlords— I’m Afraid of Whoever’s Training Them To Think That Way

by Damien P. Williams

I want to let you in on a secret: According to Silicon Valley’s AI’s, I’m not human.

Well, maybe they think I’m human, but they don’t think I’m me. Or, if they think I’m me and that I’m human, they think I don’t deserve expensive medical care. Or that I pose a higher risk of criminal recidivism. Or that my fidgeting behaviours or culturally-perpetuated shame about my living situation or my race mean I’m more likely to be cheating on a test. Or that I want to see morally repugnant posts that my friends have commented on to call morally repugnant. Or that I shouldn’t be given a home loan or a job interview or the benefits I need to stay alive.

Now, to be clear, “AI” is a misnomer, for several reasons, but we don’t have time, here, to really dig into all the thorny discussion of values and beliefs about what it means to think, or to be a pow3rmind— especially because we need to take our time talking about why values and beliefs matter to conversations about “AI,” at all. So instead of “AI,” let’s talk specifically about algorithms, and machine learning.

Machine Learning (ML) is the name for a set of techniques for systematically reinforcing patterns, expectations, and desired outcomes in various computer systems. These techniques allow those systems to make sought after predictions based on the datasets they’re trained on. ML systems learn the patterns in these datasets and then extrapolate them to model a range of statistical likelihoods of future outcomes.

Algorithms are sets of instructions which, when run, perform functions such as searching, matching, sorting, and feeding the outputs of any of those processes back in on themselves, so that a system can learn from and refine itself. This feedback loop is what allows algorithmic machine learning systems to provide carefully curated search responses or newsfeed arrangements or facial recognition results to consumers like me and you and your friends and family and the police and the military. And while there are many different types of algorithms which can be used for the above purposes, they all remain sets of encoded instructions to perform a function.

And so, in these systems’ defense, it’s no surprise that they think the way they do: That’s exactly how we’ve told them to think.

[Image of Michael Emerson as Harold Finch, in season 2, episode 1 of the show Person of Interest, “The Contingency.” His face is framed by a box of dashed yellow lines, the words “Admin” to the top right, and “Day 1” in the lower right corner.]

Continue Reading

I talked with Hewlett Packard Enterprise’s Curt Hopkins, for their article “4 obstacles to ethical AI (and how to address them).” We spoke about the kinds of specific tools and techniques by which people who populate or manage artificial intelligence design teams can incorporate expertise from the humanities and social sciences. We also talked about compelling reasons why they should do this, other than the fact that they’re just, y’know, very good ideas.

From the Article:

To “bracket out” bias, Williams says, “I have to recognize how I create systems and code my understanding of the world.” That means making an effort early on to pay attention to the data entered. The more diverse the group, the less likely an AI system is to reinforce shared bias. Those issues go beyond gender and race; they also encompass what you studied, the economic group you come from, your religious background, all of your experiences.

That becomes another reason to diversify the technical staff, says Williams. This is not merely an ethical act. The business strategy may produce more profit because the end result may be a more effective AI. “The best system is the one that best reflects the wide range of lived experiences and knowledge in the world,” he says.

[Image of two blank, white, eyeless faces, partially overlapping each other.]

To be clear, this is an instance in which I tried to find capitalist reasons that would convince capitalist people to do the right thing. To that end, you should imagine that all of my sentences start with “Well if we’re going to continue to be stuck with global capitalism until we work to dismantle it…” Because they basically all did.

I get how folx might think that framing would be a bit of a buzzkill for a tech industry audience, but I do want to highlight and stress something: Many of the ethical problems we’re concerned with mitigating or ameliorating are direct products of the capitalist system in which we are making these choices and building these technologies.

All of that being said, I’m not the only person there with something interesting to say, and you should go check out the rest of my and other people’s comments.

Until Next Time.

Last week, I talked to The Atlantic’s Ed Yong about new research in crowd sentiment tipping points, how it could give hope and dread for those working for social change, and how it might be used by bad actors to create/enhance already-extant sentiment-manipulation factories.

From the article:

…“You see this clump of failures below 25 percent and this clump of successes above 25 percent,” Centola says. “Mathematically, we predicted that, but seeing it in a real population was phenomenal.”

“What I think is happening at the threshold is that there’s a pretty high probability that a noncommitted actor”—a person who can be swayed in any direction—“will encounter a majority of committed minority actors, and flip to join them,” says Pamela Oliver, a sociologist at the University of Wisconsin at Madison. “There is therefore a good probability that enough non-committed actors will all flip at the same time that the whole system will flip.”

We talked about a lot, and much of it didn’t make it into the article, but one of the things that matters most about all of this is that we’re going to have to be increasingly mindful and intentional about the information we take in. We now know that we have the ability to move the needle of conversation, with not too much effort, and with this knowledge we can make progressive social change. We can use this to fight against the despair that can so easily creep into this work of spreading compassion and trying to create a world where we can all flourish.

https://upload.wikimedia.org/wikipedia/commons/thumb/6/65/Argentina_-_Mt_Tronador_Ascent_-_65_-_Casa%C3%B1o_Overa_glacier_%286834408616%29.jpg/640px-Argentina_-_Mt_Tronador_Ascent_-_65_-_Casa%C3%B1o_Overa_glacier_%286834408616%29.jpg

[Argentina’s Mt Tronador Casaño Overa glacier, by McKay Savage]

But we have to know that there will also be those who see this as a target number to hit so that they might better disrupt and destabilize groups and beliefs. We already know that many such people are hard at work, trying to sow doubt and mistrust. We already have evidence that these actors will make other people’s lives unpleasant for the sake of it. With this new research, they’ll be encouraged, as well. As I said to Ed Yong:

“There are already a number of people out there who are gaming group dynamics in careful ways… If they know what target numbers they have to hit, it’s easy to see how they could take this information and create [or increase the output of the existing] sentiment-manipulation factory.”

The infiltration of progressive groups to move them toward chaos and internal strife is not news, just like the infiltration (and origin) of police and military groups by white supremacists is not news.

And so, while I don’t want to add to a world in which people feel like they have to continually mistrust each other, we do have to be intentional about the work we do, and how we do it, and we have to be mindful of who is trying to get us to believe what, and why they want us to believe it. Especially if we want to get others to believe and value as we do.

This research gives us a useful set of tools and a good to place to start.

Until Next Time.

[Direct link to Mp3]

[09/22/17: This post has been updated with a transcript, courtesy of Open Transcripts]

Back on March 13th, 2017, I gave an invited guest lecture, titled:

TECHNOLOGY, DISABILITY, AND HUMAN AUGMENTATION

‘Please join Dr. Ariel Eisenberg’s seminar, “American Identities: Disability,” and [the] Interdisciplinary Studies Department for an hour-long conversation with Damien Williams on disability and the normalization of technology usage, “means-well” technological innovation, “inspiration porn,” and other topics related to disability and technology.’

It was kind of an extemporaneous riff on my piece “On the Ins and Outs of Human Augmentation,” and it gave me the opportunity to namedrop Ashley Shew, Natalie Kane, and Rose Eveleth.

The outline looked a little like this:

  • Foucault and Normalization
    • Tech and sociological pressures to adapt to the new
      • Starts with Medical tech but applies Everywhere; Facebook, Phones, Etc.
  • Zoltan Istvan: In the Transhumanist Age, We Should Be Repairing Disabilities Not Sidewalks
  • All Lead To: Ashley Shew’s “Up-Standing Norms
    • Listening to the Needs and Desires of people with disabilities.
      • See the story Shew tells about her engineering student, as related in the AFWTA Essay
    • Inspiration Porn: What is cast by others as “Triumphing” over “Adversity” is simply adapting to new realities.
      • Placing the burden on the disabled to be an “inspiration” is dehumanizing;
      • means those who struggle “have no excuse;”
      • creates conditions for a “who’s got it worse” competition
  • John Locke‘s Empiricism: Primary and Secondary Qualities
    • Primary qualities of biology and physiology lead to secondary qualities of society and culture
      • Gives rise to Racism and Ableism, when it later combines with misapplied Darwinism to be about the “Right Kinds” of bodies and minds.
        • Leads to Eugenics: Forced sterilization, medical murder, operating and experimenting on people without their knowledge or consent.
          • “Fixing” people to make them “normal, again”
  • Natalie Kane‘s “Means Well Technology
    • Design that doesn’t take into account the way that people will actually live with and use new tech.
      • The way tech normalizes is never precisely the way designers want it to
        • William Gibson’s quote “The street finds its own uses for things.”
  • Against Locke: Embrace Phenomenological Ethics and Epistemology (Feminist Epistemology and Ethics)
    • Lived Experience and embodiment as crucial
    • The interplay of Self and and Society
  • Ship of Theseus: Identity, mind, extensions, and augmentations change how we think of ourselves and how society thinks of us
    • See the story Shew tells about her friend with the hemipelvectomy, as related in the aforementioned AFWTA Essay

The whole thing went really well (though, thinking back, I’m not super pleased with my deployment of Dennett). Including Q&A, we got about an hour and forty minutes of audio, available at the embed and link above.

Also, I’m apparently the guy who starts off every talk with some variation on “This is a really convoluted interplay of ideas, but bear with me; it all comes together.”

The audio transcript is below the cut. Enjoy.

Continue Reading

(Direct Link to the Mp3)

This is the recording and the text of my presentation from 2017’s Southwest Popular/American Culture Association Conference in Albuquerque, ‘Are You Being Watched? Simulated Universe Theory in “Person of Interest.”‘

This essay is something of a project of expansion and refinement of my previous essay “Labouring in the Liquid Light of Leviathan,”  considering the Roko’s Basilisk thought experiment. Much of the expansion comes from considering the nature of simulation, memory, and identity within Jonathan Nolan’s TV series, Person of Interest. As such, it does contain what might be considered spoilers for the series, as well as for his most recent follow-up, Westworld.

Use your discretion to figure out how you feel about that.


Are You Being Watched? Simulated Universe Theory in “Person of Interest”

Jonah Nolan’s Person Of Interest is the story of the birth and life of The Machine, a benevolent artificial super intelligence (ASI) built in the months after September 11, 2001, by super-genius Harold Finch to watch over the world’s human population. One of the key intimations of the series—and partially corroborated by Nolan’s follow-up series Westworld—is that all of the events we see might be taking place in the memory of The Machine. The structure of the show is such that we move through time from The Machine’s perspective, with flashbacks and -forwards seeming to occur via the same contextual mechanism—the Fast Forward and Rewind of a digital archive. While the entirety of the series uses this mechanism, the final season puts the finest point on the question: Has everything we’ve seen only been in the mind of the machine? And if so, what does that mean for all of the people in it?

Our primary questions here are as follows: Is a simulation of fine enough granularity really a simulation at all? If the minds created within that universe have interiority and motivation, if they function according to the same rules as those things we commonly accept as minds, then are those simulation not minds, as well? In what way are conclusions drawn from simulations akin to what we consider “true” knowledge?

In the PoI season 5 episode, “The Day The World Went Away,” the characters Root and Shaw (acolytes of The Machine) discuss the nature of The Machine’s simulation capacities and the audience is given to understand that it runs a constant model of everyone it knows, and that the more it knows them, the better its simulation. This supposition links us back to the season 4 episode “If-Then-Else,” in which the machine runs through the likelihood of success through hundreds of thousands of scenarios in under one second. If The Machine is able to accomplish this much computation in this short a window, how much can and has it accomplished over the several years of its operation? Perhaps more importantly, what is the level of fidelity of those simulations to the so-called real world?

[Person of Interest s4e11, “If-Then-Else.” The Machine runs through hundreds of thousands of scenarios to save the team.]

These questions are similar to the idea of Roko’s Basilisk, a thought experiment that cropped up in the online discussion board of LessWrong.com. It was put forward by user Roko who, in very brief summary, says that if the idea of timeless decision theory (TDT) is correct, then we might all be living in a simulation created by a future ASI trying to figure out the best way to motivate humans in the past to create it. To understand how this might work, we have to look as TDT, an idea developed in 2010 by Eliezer Yudkowsky which posits that in order to make a decision we should act as though we are determining the output of an abstract computation. We should, in effect, seek to create a perfect simulation and act as though anyone else involved in the decision has done so as well. Roko’s Basilisk is the idea that a Malevolent ASI has already done this—is doing this—and your actions are the simulated result. Using that output, it knows just how to blackmail and manipulate you into making it come into being.

Or, as Yudkowsky himself put it, “YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.” This is the self-generating aspect of the Basilisk: If you can accurately model it, then the Basilisk will eventually, inevitably come into being, and one of the attributes it will thus have is the ability to accurately model that you accurately modeled it, and whether or not you modeled it from within a mindset of being susceptible to its coercive actions. The only protection is to either work toward its creation anyway, so that it doesn’t feel the need to torture the “real” you into it, or to make very sure that you never think of it at all, so you do not bring it into being.

All of this might seem far-fetched, but if we look closely, Roko’s Basilisk functions very much like a combination of several well-known theories of mind, knowledge, and metaphysics: Anselm’s Ontological Argument for the Existence of God (AOAEG), a many worlds theorem variant on Pascal’s Wager (PW), and Descartes’ Evil Demon Hypothesis (DEDH; which, itself, has been updated to the oft-discussed Brain In A Vat [BIAV] scenario). If this is the case, then Roko’s Basilisk has all the same attendant problems that those arguments have, plus some new ones, resulting from their combination. We will look at all of these theories, first, and then their flaws.

To start, if you’re not familiar with AOAEG, it’s a species of prayer in the form of a theological argument that seeks to prove that god must exist because it would be a logical contradiction for it not to. The proof depends on A) defining god as the greatest possible being (literally, “That Being Than Which None Greater Is Possible”), and B) believing that existing in reality as well as in the mind makes something “Greater Than” if it existed only the mind. That is, if God only exists in my imagination, it is less great than it could be if it also existed in reality. So if I say that god is “That Being Than Which None Greater Is Possible,” and existence is a part of what makes something great, then god must exist.

The next component is Pascal’s Wager which very simply says that it is a better bet to believe in the existence of God, because if you’re right, you go to Heaven, and if you’re wrong, nothing happens; you’re simply dead forever. Put another way, Pascal is saying that if you bet that God doesn’t exist and you’re right, you get nothing, but if you’re wrong, then God exists and your disbelief damns you to Hell for all eternity. You can represent the whole thing in a four-option grid:

[Pascal’s Wager as a Four-Option Grid: Belief/Disbelief; Right/Wrong. Belief*Right=Infinity;Belief*Wrong=Nothing; Disbelief*Right=Nothing; Disbelief*Wrong=Negative Infinity]

And so here we see the Timeless Decision Theory component of the Basilisk: It’s better to believe in the thing and work toward its creation and sustenance, because if it doesn’t exist you lose nothing, but if it does come to be, then it will know what you would have done either for or against it, in the past, and it will reward or punish you, accordingly. The multiversal twist comes when we realise that even if the Basilisk never comes to exist in our universe and never will, it might exist in some other universe, and thus, when that other universe’s Basilisk models your choices it will inevitably—as a superintelligence—be able to model what you would do in any universe. Thus, by believing in and helping our non-existent Super-Devil, we protect the alternate reality versions of ourselves from their very real Super-Devil.

Descartes’ Evil Demon Hypothesis and the Brain In A Vat are so pervasive that we encounter them in many different expressions of pop culture. The Matrix, Dark City, Source Code, and many others are all variants on these themes. A malignant and all-powerful being (or perhaps just an amoral scientist) has created a simulation in which we reside, and everything we think we have known about our lives and our experiences has been perfectly simulated for our consumption. Variations on the theme test whether we can trust that our perceptions and grounds for knowledge are “real” and thus “valid,” respectively. This line of thinking has given rise to the Simulated Universe Theory on which Roko’s Basilisk depends, but SUT removes a lot of the malignancy of DEDH and BIAV. The Basilisk adds it back. Unfortunately, many of these philosophical concepts flake apart when we touch them too hard, so jamming them together was perhaps not the best idea.

The main failings in using AOAEG rest in believing that A) a thing’s existence is a “great-making quality” that it can possess, and B) our defining a thing a particular way might simply cause it to become so. Both of these are massively flawed ideas. For one thing, these arguments beg the question, in a literal technical sense. That is, they assume that some element(s) of their conclusion—the necessity of god, the malevolence or epistemic content of a superintelligence, the ontological status of their assumptions about the nature of the universe—is true without doing the work of proving that it’s true. They then use these assumptions to prove the truth of the assumptions and thus the inevitability of all consequences that flow from the assumptions.

Another problem is that the implications of this kind of existential bootstrapping tend to go unexamined, making the fact of their resurgence somewhat troubling. There are several nonwestern perspectives that do the work of embracing paradox—aiming so far past the target that you circle around again to teach yourself how to aim past it. But that kind of thing only works if we are willing to bite the bullet on a charge of circular logic and take the time to showing how that circularity underlies all epistemic justifications. The only difference, then, is how many revolutions it takes before we’re comfortable with saying “Enough.”

Every epistemic claim we make is, as Hume clarified, based upon assumptions and suppositions that the world we experience is actually as we think it is. Western thought uses reason and rationality to corroborate and verify, but those tools are themselves verified by…what? In fact, we well know that the only thing we have to validate our valuation of reason, is reason. And yet western reasoners won’t stand for that, in any other justification procedure. They will call it question-begging and circular.

Next, we have the DEDH and BIAV scenarios. Ultimately, Descartes’ point wasn’t to suggest an evil genius in control of our lives just to disturb us; it was to show that, even if that were the case, we would still have unshakable knowledge of one thing: that we, the experiencer, exist. So what if we have no free will; so what if our knowledge of the universe is only five minutes old, everything at all having only truly been created five minutes ago; so what if no one else is real? COGITO ERGO SUM! We exist, now. But the problem here is that this doesn’t tell us anything about the quality of our experiences, and the only answer Descartes gives us is his own Anslemish proof for the existence of god followed by the guarantee that “God is not a deceiver.”

The BIAV uses this lack to kind of hone in on the aforementioned central question: What does count as knowledge? If the scientists running your simulation use real-world data to make your simulation run, can you be said to “know” the information that comes from that data? Many have answered this with a very simple question: What does it matter? Without access to the “outside world”–that is, the world one layer up in which the simulation that is our lives was being run–there is literally no difference between our lives and the “real world.” This world, even if it is a simulation for something or someone else, is our “real world.”

And finally we have Pascal’s Wager. The first problem with PW is that it is an extremely cynical way of thinking about god. It assumes a god that only cares about your worship of it, and not your actual good deeds and well-lived life. If all our Basilisk wants is power, then that’s a really crappy kind of god to worship, isn’t it? I mean, even if it is Omnipotent and Omniscient, it’s like that quote that often gets misattributed to Marcus Aurelius says:

“Live a good life. If there are gods and they are just, then they will not care how devout you have been, but will welcome you based on the virtues you have lived by. If there are gods, but unjust, then you should not want to worship them. If there are no gods, then you will be gone, but will have lived a noble life that will live on in the memories of your loved ones.”

[Bust of Marcus Aurelius framed by text of a quote he never uttered.]

Secondly, the format of Pascal’s Wager makes the assumption that there’s only the one god. Our personal theological positions on this matter aside, it should be somewhat obvious that we can use the logic of the Basilisk argument to generate at least one more Super-Intelligent AI to worship. But if we want to do so, first we have to show how the thing generates itself, rather than letting the implication of circularity arise unbidden. Take the work of Douglas R Hofstadter; he puts forward the concepts of iterative recursion as the mechanism by which a consciousness generates itself.

Through iterative recursion, each loop is a simultaneous act of repetition of old procedures and tests of new ones, seeking the best ways via which we might engage our environments as well as our elements and frames of knowledge. All of these loops, then, come together to form an upward turning spiral towards self-awareness. In this way, out of the thought processes of humans who are having bits of discussion about the thing—those bits and pieces generated on the web and in the rest of the world—our terrifying Basilisk might have a chance of creating itself. But with the help of Gaunilo of Marmoutiers, so might a saviour.

Guanilo is most famous for his response to Anselm’s Ontological Argument, which says that if Anselm is right we could just conjure up “The [Anything] Than Which None Greater Can Be Conceived.” That is, if defining a thing makes it so, then all we have to do is imagine in sufficient detail both an infinitely intelligent, benevolent AI, and the multiversal simulation it generates in which we all might live. We will also conceive it to be greater than the Basilisk in all ways. In fact, we can say that our new Super Good ASI is the Artificial Intelligence Than Which None Greater Can Be Conceived. And now we are safe.

Except that our modified Pascal’s Wager still means we should believe in and worship and work towards our Benevolent ASI’s creation, just in case. So what do we do? Well, just like the original wager, we chuck it out the window, on the grounds that it’s really kind of a crappy bet. In Pascal’s offering, we are left without the consideration of multiple deities, but once we are aware of that possibility, we are immediately faced with another question: What if there are many, and when we choose one, the others get mad? What If We Become The Singulatarian Job?! Our lives then caught between at least two superintelligent machine consciousnesses warring over our…Attention? Clock cycles? What?

But this is, in essence, the battle between the Machine and Samaritan, in Person of Interest. Each ASI has acolytes, and each has aims it tries to accomplish. Samaritan wants order at any cost, and The Machine wants people to be able to learn and grow and become better. If the entirety of the series is The Machine’s memory—or a simulation of those memories in the mind of another iteration of the Machine—then what follows is that it is working to generate the scenario in which the outcome is just that. It is trying to build a world in which it is alive, and every human being has the opportunity to learn and become better. In order to do this, it has to get to know us all, very well, which means that it has to play these simulations out, again and again, with both increasing fidelity, and further iterations. That change feels real, to us. We grow, within it. Put another way: If all we are is a “mere” a simulation… does it matter?

So imagine that the universe is a simulation, and that our simulation is more than just a recording; it is the most complex game of The SIMS ever created. So complex, in fact, that it begins to exhibit reflectively epiphenomenal behaviours, of the type Hofstadter describes—that is, something like minds arise out of the interactions of the system with itself. And these minds are aware of themselves and can know their own experience and affect the system which gives rise to them. Now imagine that the game learns, even when new people start new games. That it remembers what the previous playthrough was like, and adjusts difficulty and types of coincidence, accordingly.

Now think about the last time you had such a clear moment of déjà vu that each moment you knew— you knew—what was going to come next, and you had this sense—this feeling—like someone else was watching from behind your eyes…

[Root and Reese in The Machine’s God Mode.]

What I’m saying is, what if the DEDH/BIAV/SUT is right, and we are in a simulation? And what if Anselm was right and we can bootstrap a god into existence? And what if PW/TDT is right and we should behave and believe as if we’ve already done it? So what if all of this is right, and we are the gods we’re terrified of?

We just gave ourselves all of this ontologically and metaphysically creative power, making two whole gods and simulating entire universes, in the process. If we take these underpinnings seriously, then multiversal theory plays out across time and space, and we are the superintelligences. We noted early on that, in PW and the Basilisk, we don’t really lose anything if we are wrong in our belief, but that is not entirely true. What we lose is a lifetime of work that could have been put toward better things. Time we could be spending building a benevolent superintelligence that understands and has compassion for all things. Time we could be spending in turning ourselves into that understanding, compassionate superintelligence, through study, travel, contemplation, and work.

Or, as Root put it to Shaw: “That even if we’re not real, we represent a dynamic. A tiny finger tracing a line in the infinite. A shape. And then we’re gone… Listen, all I’m saying that is if we’re just information, just noise in the system? We might as well be a symphony.”

What is The Real?

I have been working on this piece for a little more than a month, since just after Christmas. What with one thing, and another, I kept refining it while, every week, it seemed more and more pertinent and timely. You see, we need to talk about ontology.

Ontology is an aspect of metaphysics, the word translating literally to “the study of what exists.” Connotatively, we might rather say, “trying to figure out what’s real.” Ontology necessarily intersects with studies of knowledge and studies of value, because in order to know what’s real you have to understand what tools you think are valid for gaining knowledge, and you have to know whether knowledge is even something you can attain, as such.

Take, for instance, the recent evolution of the catchphrase of “fake news,” the thinking behind it that allows people to call lies “alternative facts,” and the fact that all of these elements are already being rotated through several dimensions of meaning that those engaging in them don’t seem to notice. What I mean is that the inversion of the catchphrase “fake news” into a cipher for active confirmation bias was always going to happen. It and any consternation at it comprise a situation that is borne forth on a tide of intentional misunderstandings.

If you were using fake to mean, “actively mendacious; false; lies,” then there was a complex transformation happening here, that you didn’t get:

There are people who value the actively mendacious things you deemed “wrong”—by which you meant both “factually incorrect” and “morally reprehensible”—and they valued them on a nonrational, often actively a-rational level. By this, I mean both that they value the claims themselves, and that they have underlying values which cause them to make the claims. In this way, the claims both are valued and reinforce underlying values.

So when you called their values “fake news” and told them that “fake news” (again: their values) ruined the country, they—not to mention those actively preying on their a-rational valuation of those things—responded with “Nuh-uh! your values ruined the country! And that’s why we’re taking it back! MAGA! MAGA! Drumpfthulhu Fhtagn!”

Logo for the National Geographic Channel’s “IS IT REAL?” Many were concerned that NG Magazine were going to change their climate change coverage after they were bought by 21st Century Fox.

You see? They mean “fake news” along the same spectrum as they mean “Real America.” They mean that it “FEELS” “RIGHT,” not that it “IS” “FACT.”

Now, we shouldn’t forget that there’s always some measure of preference to how we determine what to believe. As John Flowers puts it, ‘Truth has always had an affective component to it: those things that we hold to be most “true” are those things that “fit” with our worldview or “feel” right, regardless of their factual veracity.

‘We’re just used to seeing this in cases of trauma, e.g.: “I don’t believe he’s dead,” despite being informed by a police officer.’

Which is precisely correct, and as such the idea that the affective might be the sole determinant is nearly incomprehensible to those of us who are used to thinking of facts as things that are verifiable by reference to externalities as well as values. At least, this is the case for those of us who even relativistically value anything at all. Because there’s also always the possibility that the engagement of meaning plays out in a nihilistic framework, in which we have neither factual knowledge nor moral foundation.

Epistemic Nihilism works like this: If we can’t ever truly know anything—that is, if factual knowledge is beyond us, even at the most basic “you are reading these words” kind of level—then there is no description of reality to be valued above any other, save what you desire at a given moment. This is also where nihilism and skepticism intersect. In both positions nothing is known, and it might be the case that nothing is knowable.

So, now, a lot has been written about not only the aforementioned “fake news,” but also its over-arching category of “post-truth,” said to be our present moment where people believe (or pretend to believe) in statements or feelings, independent of their truth value as facts. But these ideas are neither new nor unique. In fact, Simpsons Did It. More than that, though, people have always allowed their values to guide them to beliefs that contradict the broader social consensus, and others have always eschewed values entirely, for the sake of self-gratification. What might be new, right now, is the willfulness of these engagements, or perhaps their intersection. It might be the case that we haven’t before seen gleeful nihilism so forcefully become the rudder of gormless, value-driven decision-making.

Again, values are not bad, but when they sit unexamined and are the sole driver of decisions, they’re just another input variable to be gamed, by those of a mind to do so. People who believe that nothing is knowable and nothing matters will, at the absolute outside, seek their own amusement or power, though it may be said that nihilism in which one cares even about one’s own amusement is not genuine nihilism, but is rather “nihilism,” which is just relativism in a funny hat. Those who claim to value nothing may just be putting forward a front, or wearing a suit of armour in order to survive an environment where having your values known makes you a target.

If they act as though they believe there is no meaning, and no truth, then they can make you believe that they believe that nothing they do matters, and therefore there’s, no moral content to any action they take, and so no moral judgment can be made on them for it. In this case, convincing people to believe news stories they make up is in no way materially different from researching so-called facts and telling the rest of us that we should trust and believe them. And the first way’s also way easier. In fact, preying on gullible people and using their biases to make yourself some lulz, deflect people’s attention, and maybe even get some of those sweet online ad dollars? That’s just common sense.

There’s still some something to be investigated, here, in terms of what all of this does for reality as we understand and experience it. How what is meaningful, what is true, what is describable, and what is possible all intersect and create what is real. Because there is something real, here—not “objectively,” as that just lets you abdicate your responsibility for and to it, but perhaps intersubjectively. What that means is that we generate our reality together. We craft meaning and intention and ideas and the words to express them, together, and the value of those things and how they play out all sit at the place where multiple spheres of influence and existence come together, and interact.

To understand this, we’re going to need to talk about minds and phenomenological experience.

 

What is a Mind?

We have discussed before the idea that what an individual is and what they feel is not only shaped by their own experience of the world, but by the exterior forces of society and the expectations and beliefs of the other people with whom they interact. These social pressures shape and are shaped by all of the people engaged in them, and the experience of existence had by each member of the group will be different. That difference will range on a scale from “ever so slight” to “epochal and paradigmatic,” with the latter being able to spur massive misunderstandings and miscommunications.

In order to really dig into this, we’re going to need to spend some time thinking about language, minds, and capabilities.

Here’s an article that discusses the idea that you mind isn’t confined to your brain. This isn’t meant in a dualistic or spiritualistic sense, but as the fundamental idea that our minds are more akin to, say, an interdependent process that takes place via the interplay of bodies, environments, other people, and time, than they are to specifically-located events or things. The problem with this piece, as my friends Robin Zebrowski and John Flowers both note, is that it leaves out way too many thinkers. People like Andy Clark, David Chalmers, Maurice Merleau-Ponty, John Dewey, and William James have all discussed something like this idea of a non-local or “extended” mind, and they are all greatly preceded by the fundamental construction of the Buddhist view of the self.

Within most schools of Buddhism, Anatta, or “no self” is how one refers to one’s indvidual nature. Anatta is rooted in the idea that there is no singular, “true” self. To vastly oversimplify, there is an concept known as “The Five Skandhas” or “aggregates.” These are the parts of yourself that are knowable and which you think of as permanent, and they are your:

Material Form (Body)
Feelings (Pleasure, Pain, Indifference)
Perception (Senses)
Mental Formations (Thoughts)
Consciousness

http://www.mountainsoftravelphotos.com/Tibet%20-%20Buddhism/Wheel%20Of%20Life/Wheel%20Of%20Life/slides/Tibetan%20Buddhism%20Wheel%20Of%20Life%2007%2004%20Mind%20And%20Body%20-%20People%20In%20Boat.JPG

Image of People In a Boat, from a Buddhist Wheel of Life.

Along with the skandhas, there are two main arguments that go into proving that you don’t have a self, known as “The Argument From Control” (1) and “The Argument from Impermanence” (2)
1) If you had a “true self,” it would be the thing in control of the whole of you, and since none of the skandhas is in complete control of the rest—and, in fact, all seem to have some measure of control over all—none of them is your “true self.”
2) If you had a “true self,” it would be the thing about you that was permanent and unchanging, and since none of the skandhas is permanent and unchanging—and, in fact, all seem to change in relation to each other—none of them is your “true self.”

The interplay between these two arguments also combines with an even more fundamental formulation: If only the observable parts of you are valid candidates for “true selfhood,” and if the skandhas are the only things about yourself that you can observe, and if none of the skandhas is your true self, then you have no true self.

Take a look at this section of “The Questions of King Milinda,” for a kind of play-by-play of these arguments in practice. (But also remember that Milinda was Menander, a man who was raised in the aftermath of Alexandrian Greece, and so he knew the works of Socrates and Plato and Aristotle and more. So that use of the chariot metaphor isn’t an accident.)

We are an interplay of forces and names, habits and desires, and we draw a line around all of it, over and over again, and we call that thing around which we draw that line “us,” “me,” “this-not-that.” But the truth of us is far more complex than all of that. We minds in bodies and in the world in which we live and the world and relationships we create. All of which kind of puts paid to the idea that an octopus is like an alien to us because it thinks with its tentacles. We think with ours, too.

As always, my tendency is to play this forward a few years to make us a mirror via which to look back at ourselves: Combine this idea about the epistemic status of an intentionally restricted machine mind; with the StackGAN process, which does “Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks,” or, basically, you describe in basic English what you want to see and the system creates a novel output image of it; with this long read from NYT on “The Great AI Awakening.”

This last considers how Google arrived at the machine learning model it’s currently working with. The author, Gideon Lewis-Kraus, discusses the pitfalls of potentially programming biases into systems, but the whole piece displays a kind of… meta-bias? Wherein there is an underlying assumption that “philosophical questions” are, again, simply shorthand for “not practically important,” or “having no real-world applications,” even the author discusses ethics and phenomenology, and the nature of what makes a mind. In addition to that, there is a just startling lack of gender variation, within the piece.

Because asking the question, “How do the women in Silicon Valley remember that timeframe,” is likely to get you get you very different perspectives than what we’re presented with, here. What kind of ideas were had by members of marginalized groups, but were ignored or eternally back-burnered because of that marginalization? The people who lived and worked and tried to fit in and have their voices heard while not being a “natural” for the framework of that predominantly cis, straight, white, able-bodied (though the possibility of unassessed neuroatypicality is high), male culture will likely have different experience, different contextualizations, than those who do comprise the predominant culture. The experiences those marginalized persons share will not be exactly the same, but there will be a shared tone and tenor of their construction that will most certainly set itself apart from those of the perceived “norm.”

Everyone’s lived experience of identity will manifest differently, depending upon the socially constructed categories to which they belong, which means that even those of us who belong to one or more of the same socially constricted categories will not have exactly the same experience of them.

Living as a disabled woman, as a queer black man, as a trans lesbian, or any number of other identities will necessarily colour the nature of what you experience as true, because you will have access to ways of intersecting with the world that are not available to people who do not live as you live. If your experience of what is true differs, then this will have a direct impact on what you deem to be “real.”

At this point, you’re quite possibly thinking that I’ve undercut everything we discussed in the first section; that now I’m saying there isn’t anything real, and that’s it’s all subjective. But that’s not where we are. If you haven’t, yet, I suggest reading Thomas Nagel’s “What Is It Like To Be A Bat?“ for a bit on individually subjective phenomenological experience, and seeing what he thinks it does and proves. Long story short, there’s something it “is like” to exist as a bat, and even if you or I could put our minds in a bat body, we would not know what it’s like to “be” a bat. We’d know what it was like to be something that had been a human who had put its brain into a bat. The only way we’d ever know what it was like to be a bat would be to forget that we were human, and then “we” wouldn’t be the ones doing the knowing. (If you’re a fan of Terry Pratchett’s Witch books, in his Discworld series, think of the concept of Granny Weatherwax’s “Borrowing.”)

But what we’re talking about isn’t the purely relative and subjective. Look carefully at what we’ve discussed here: We’ve crafted a scenario in which identity and mind are co-created. The experience of who and what we are isn’t solely determined by our subjective valuation of it, but also by what others expect, what we learn to believe, and what we all, together, agree upon as meaningful and true and real. This is intersubjectivity. The elements of our constructions depend on each other to help determine each other, and the determinations we make for ourselves feed into the overarching pool of conceptual materials from which everyone else draws to make judgments about themselves, and the rest of our shared reality.

 

The Yellow Wallpaper

Looking at what we’ve woven, here, what we have is a process that must be undertaken before certain facts of existence can be known and understood (the experiential nature of learning and comprehension being something else that we can borrow from Buddhist thought). But it’s still the nature of such presentations to be taken up and imitated by those who want what they perceive as the benefits or credit of having done the work. Certain people will use the trappings and language by which we discuss and explore the constructed nature of identity, knowledge, and reality, without ever doing the actual exploration. They are not arguing in good faith. Their goal is not truly to further understanding, or to gain a comprehension of your perspective, but rather to make you concede the validity of theirs. They want to force you to give them a seat at the table, one which, once taken, they will use to loudly declaim to all attending that, for instance, certain types of people don’t deserve to live, by virtue of their genetics, or their socioeconomic status.

Many have learned to use the conceptual framework of social liberal post-structuralism in the same way that some viruses use the shells of their host’s cells: As armour and cover. By adopting the right words and phrases, they may attempt to say that they are “civilized” and “calm” and “rational,” but make no mistake, Nazis haven’t stopped trying to murder anyone they think of as less-than. They have only dressed their ideals up in the rhetoric of economics or social justice, so that they can claim that anyone who stands against them is the real monster. Incidentally, this tactic is also known to be used by abusers to justify their psychological or physical violence. They manipulate the presentation of experience so as to make it seem like resistance to their violence is somehow “just as bad” as their violence. When, otherwise, we’d just call it self-defense.

If someone deliberately games a system of social rules to create a win condition in which they get to do whatever the hell they want, that is not of the same epistemic, ontological, or teleological—meaning, nature, or purpose—let alone moral status as someone who is seeking to have other people in the world understand the differences of their particular lived experience so that they don’t die. The former is just a way of manipulating perceptions to create a sense that one is “playing fair” when what they’re actually doing is making other people waste so much of their time countenancing their bullshit enough to counter and disprove it that they can’t get any real work done.

In much the same way, there are also those who will pretend to believe that facts have no bearing, that there is neither intersubjective nor objective verification for everything from global temperature levels to how many people are standing around in a crowd. They’ll pretend this so that they can say what makes them feel powerful, safe, strong, in that moment, or to convince others that they are, or simply, again, because lying and bullshitting amuses them. And the longer you have to fight through their faux justification for their lies, the more likely you’re too exhausted or confused about what the original point was to do anything else.

Side-by-side comparison of President Obama’s first Inauguration (Left) and Donald Trump’s Inauguration (Right).

If we are going to maintain a sense of truth and claim that there are facts, then we must be very careful and precise about the ways in which we both define and deploy them. We have to be willing to use the interwoven tools and perspectives of facts and values, to tap into the intersubjectively created and sustained world around us. Because, while there is a case to be made that true knowledge is unattainable, and some may even try to extend that to say that any assertion is as good as any other, it’s not necessary that one understands what those words actually mean in order to use them as cover for their actions. One would just have to pretend well enough that people think it’s what they should be struggling against. And if someone can make people believe that, then they can do and say absolutely anything.


A large part of how I support myself in the endeavor to think in public is with your help, so if you like what you’ve read here, and want to see more like it, then please consider becoming either a recurring Patreon subscriber or making a one-time donation to the Tip Jar, it would be greatly appreciated.
And thank you.

 

[UPDATED 09/12/17: The transcript of this audio, provided courtesy of Open Transcripts, is now available below the Read More Cut.]

[UPDATED 03/28/16: Post has been updated with a far higher quality of audio, thanks to the work of Chris Novus. (Direct Link to the Mp3)]

So, if you follow the newsletter, then you know that I was asked to give the March lecture for my department’s 3rd Thursday Brown Bag Lecture Series. I presented my preliminary research for the paper which I’ll be giving in Vancouver, about two months from now, “On the Moral, Legal, and Social Implications of the Rearing and Development of Nascent Machine Intelligences” (EDIT: My rundown of IEEE Ethics 2016 is here and here).

It touches on thoughts about everything from algorithmic bias, to automation and a post-work(er) economy, to discussions of what it would mean to put dolphins on trial for murder.

About the dolphin thing, for instance: If we recognise Dolphins and other cetaceans as nonhuman persons, as India has done, then that would mean we would have to start reassessing how nonhuman personhood intersects with human personhood, including in regards to rights and responsibilities as protected by law. Is it meaningful to expect a dolphin to understand “wrongful death?” Our current definition of murder is predicated on a literal understanding of “homicide” as “death of a human,” but, at present, we only define other humans as capable of and culpable for homicide. What weight would the intentional and malicious deaths of nonhuman persons carry?

All of this would have to change.

Anyway, this audio is a little choppy and sketchy, for a number of reasons, and I while I tried to clean it up as much as I could, some of the questions the audience asked aren’t decipherable, except in the context of my answers. [Clearer transcript below.]

Until Next Time.

 

Continue Reading