All posts by Shen-yi Liao

Every Lab Needs a TOSTER

What feels like a very long time ago, I was trying to empirically investigate imaginative resistance with Nina Strohminger and Chandra Sripada. Our findings have been partly confirmed but also partly challenged by the recent replication effort by Mark Phelan and his student collaborators Wenjia Hu and Navin Rambharose, as part of the X-Phi Replicability Project.

In particular, these replicators challenged an inference that we made from a non-significant comparison result to there being no difference between two conditions. Specifically, we were trying to infer that genre does not make a difference to participant responses to questions about characters’ beliefs. They have shown that this inference is a mistake. As they put it, “Contra the original study, in this replication an effect of Genre on Belief was found. […] This was a very small, but significant, difference (t(97)=2.299, p=0.024, effect size r=0.052).” Thanks to Florian Cova, who noticed that the effect size should be r=0.227, which is more worrying for us.

Our inference pattern is a common but mistaken one. Daniel Lakens has argued for the need of using equivalence-testing to argue for such “no difference” conclusions. Indeed, using Lakens’s statistical package for the open-source statistical software jamovi, we can see that the inference was already not warranted by the original data.

(In this analysis, I set the upper and lower bounds to 0.2 Cohen’s dz, which corresponds to the conventional interpretation of a “small” effect. As the graph shows, our original data does not support the inference that there is no difference between the two conclusions because the top end of the 90% confidence interval exceeds the upper bound for equivalence.)

I am extremely appreciative of Phelan and colleagues for their efforts in replicating our original study. And, from this experience, I have learned that every lab needs a TOSTER!


Experiments with Coffee

We did some events on coffee. We wrote an article about them for the public audience. And, now, we also wrote an article about them for the academic audience. In addition to going into more details about the events themselves, the academic article also includes reflections on doing experimental philosophy and doing public philosophy.

Related Research Output:

Aaron Meskin and Shen-yi Liao (in press). Experimental Philosophical Aesthetics as Public Philosophy. Advances in Experimental Philosophy of Aesthetics.

100 Philosophers 100 Artworks 100 Words #46

This is my entry for Aesthetics for Birds‘s series.

Philosopher: Shen-yi Liao, University of Puget Sound

Artwork: Tehching Hsieh 1986-1999, Tehching Hsieh, 1986-1999. “Endurance performance art”: life in spacetime.

Artist’s Statement

100 words: life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is work is life is art is ?

Aesthetic Adjectives, Very Weird

Aesthetic adjectives are adjectives like “beautiful” and “elegant”. We use these terms to talk to others about what we think about about art and much more. Not surprisingly, given their centrality to aesthetic communication, aesthetic adjectives is an important component of the social dimension of aesthetics.

But how do aesthetic adjectives work in aesthetic communication? In a pair of forthcoming articles, we argue that the answer is actually quite complicated. And the answer is complicated because aesthetic adjectives do not behave similarly to other adjectives that we have a better theoretical grasp on, such as “tall” and “bent”.

Although we put forth some ideas about why they’re so weird, speaking for myself, we really just wanted to get their weirdness out there. So, we’re also very looking forward to other explanations of the puzzling data that other theorists come up with!

Related Research Outputs:

Shen-yi Liao and Aaron Meskin (in press). Aesthetic Adjectives: Experimental Semantics and Context-Sensitivity. Philosophy and Phenomenological Research. DOI:10.1111/phpr.12217. Data and material available on OSF.

Shen-yi Liao, Louise McNally, Aaron Meskin (in press). Aesthetic Adjectives Lack Uniform Behavior. Inquiry.

The Last Podcast Episode

Here is the concept for the Project’s podcast. We wanted to know what philosophers and other academics think about the central themes of our project. So, we ask our guests three questions:

  1. What does aesthetics tell us about human nature?
  2. How does empirical research bear on their research projects?
  3. What kind of experiment would you want to run on their research projects?

For various reasons, the plan for the podcast series did not take off. Still, all is not lost. We have now, finally, the last (and first) podcast episode, featuring Jim Hamilton (Kansas State University). Professor Hamilton specializes in aesthetics, especially on theater. In this episode, he tells me–amongst other things–an exciting idea that he has on empirically investigating theater audiences’ experiences.

Check it out!

Project Review 2015: Impact

The Experimental Philosophical Aesthetics and Human Nature Project is primarily funded by the European Commission’s Marie Curie grant, which explicitly includes the aim of facilitating knowledge transfer between academics and with the public.

Hence, in addition to–and, often, in connection with–our research projects, we have also embarked on a number of outreach efforts. This post focuses on our accomplishments on the impact front. (The previous year-in-review post focuses on our accomplishments on the research front.)

In Person

As mentioned in the previous post, we have collectively given eight research talks in many different places in order to share our research widely.

Even more excitingly, we gave two public talks about coffee to members of the public that included an interactive experimental component. Aaron gave a talk and conducted an informal study on gustatory testimony. (See also Aaron’s paper on taste-imony, co-authored with Jon Robson.) I gave a talk and conducted an informal study on the interaction between moral and gustatory value. (See also our paper “Morality and Aesthetics of Food”, forthcoming in Oxford Handbook of Food Ethics.) These public talks were supported by Cultural & Creative Industries Exchange and with assistance from Amanosi Ekenimoh and Nick Watts.

In Words

In connection with the public talks about coffee, we authored two public philosophy essays, published by Food&_. First, Aaron wrote an essay on latte art that weaves together philosophy, psychology, and practice. (In fact, one of Aaron’s speculations has now been independently investigated and received empirical support!) Second, Aaron and I–with the help of Nosi and Nick–wrote an essay on the interactive experimental aspects of the two public talks.

I authored two semi-academic how-to blog posts on replication and experimental philosophy and on the use of Amazon mechanical turk in experimental philosophy. Both posts are available on the widely-read Experimental Philosophy Blog.


Strangely enough, what made me the happiest about the Project in 2015 is not done by me (or Aaron).

Back in 2014, we hosted an early career workshop that aimed to bring together four senior scholars in philosophical aesthetics (Catherine Abell and Christy Mag Uidhir) and in experimental philosophy (Florian Cova and Jonathan Weinberg) along with seven junior scholars who are interested in the intersection of the two subfields. Our goal is to provide a forum for the exchange of ideas between all participants, and to help the junior scholars professionally.

So, I was most ecstatic to attend the 2015 American Society of Aesthetics Meeting, where Kris Goffin–one of the junior scholars at the workshop–presented two exciting studies that he and Florian Cova did on the phenomenon of guilty pleasures. Be on the lookout for these interesting findings to appear in print soon!

Project Review 2015: Research

Intellectual work can feel lonely, even when you are working with others. Most of the time you work on these research projects that take years to see the light of day, after countless iterations and refinements. Sometimes, the research projects never even see the light of day, for one reason or another. Where did the hours go? The loneliness comes when it feels like there’s nothing new to share.

So, it can be helpful to take a step back, to remind yourself of what you have done that can, in fact, be shared. Here is such an attempt, to remind ourselves what we, the Experimental Philosophical Aesthetics and Human Nature Project, have achieved in 2015. This post focuses on the research accomplishments. (The next year-in-review post focuses on the impact of the Project.)

Overview Phase of the Project

The first phase of the Project surveys the philosophical aesthetics literature in aesthetics for topics that lend themselves to empirical investigations. This phase has resulted in four overview articles and one bibliography.

There are two articles on the phenomenon of imaginative resistance (Miyazono and Liao 2016; Liao and Gendler 2016). There is an article on the ethical criticism of art debate, as it relates to food (Liao and Meskin forthcoming, “Morality and Aesthetics of Food”, Oxford Handbook of Food Ethics). And there is a bibliography on imagination and belief (Ichino, Miyazono, and Liao forthcoming, “Imagination and Belief”, Oxford Bibliographies in Philosophy).

Most importantly, there is now an overview article that introduces the subfield of experimental philosophical aesthetics (Cova, Garcia, and Liao 2015). As I wrote in an earlier blog post, since this subfield of study is so new, my co-authors and I have chosen a somewhat unusual approach for this overview: in addition to describing the works that have been done, we are also previewing some works in progress and envisioning what the field could become.

Experiment Phase of the Project

The second phase of the Project uses methods from cognitive science to investigate topics in philosophical aesthetics. We have conducted many studies in 2014 and 2015. And this year, we have traveled to many places to share the results with diverse audiences, giving eight research talks in total.

This year also brought the first major research publication of the Project. In “Aesthetic Adjectives” (Liao and Meskin forthcomingPhilosophy and Phenomenological Research), we report four studies that investigate the context-sensitivity of aesthetic adjectives. In short, we found that aesthetic adjectives behave weirdly, in ways that are not obviously anticipated by philosophical aestheticians or philosophers of language. This work continues the discussions initiated at the Project’s language of aesthetics workshop in 2014.

Individual Accomplishments

Aaron has articles published or in press on: imagination (Wiltshire and Meskin 2016), comics (Cook and Meskin 2015), taste-imony (Meskin and Robson 2015), the concept of art (Fokt and Meskin forthcoming, “Errors in ‘The History of an Error'”, British Journal of Aesthetics), and videogames (Meskin and Robson forthcoming, “Videogames as Self-Involving Interactive Fictions”, Journal of Aesthetics and Art Criticism).

I have an article in press on imaginative resistance (Liao forthcoming).

And James Andow, who joined us from December 2014 to January 2015 as LHRI Postdoctoral Fellow, has an impressive number of published or in press articles as well.

X-Phi Aesthetics, Present and Future

Experimental philosophy is to philosophy as behavioral economics is to economics: it uses tools such as randomized experiments and inferential statistics to challenge and improve philosophical theorizing. Experimental philosophical aesthetics is the attempt to bring this approach to the philosophical area of aesthetics.

If you are a loyal reader of our project blog, you probably already have a good sense of what experimental philosophical aesthetics is. But if you’re not, there is now an overview paper–by Florian Cova, Amanda Garcia, and myself–in Philosophy Compass that introduces this field of study.

Since this field of study is so new, we’ve chosen a somewhat unusual approach for this overview: in addition to describing the works that have been done, we are also previewing some works in progress and envisioning what the field could become. Take a look for yourself!

Related Research Output:

Florian Cova, Amanda Garcia, Shen-yi Liao (2015). Experimental Philosophy of Aesthetics. British Journal of Aesthetics 54(3): 339–355. DOI:10.1111/phc3.12271

Latte Art, Experimentally Investigated

Back in March, Aaron wrote a popular philosophy essay about latte art, which incorporates perspectives from philosophy, psychology, and practice. He concluded on a speculative note:

It seems that latte art can make a difference to us. Neither taste nor even flavour experiences are all we care about when it comes to a cup of coffee. […] Latte art surely affects the visual experience of sighted drinkers of coffee […] it seems possible—although I know of no research directly on the question—that latte art could affect how we taste the coffee beneath it. […] If this is right, then it is possible that latte art itself could have a subtle effect on perceived coffee flavour. So if latte art is an art, it could be an art that really matters!

An article in the August issue of Journal of Sensory Studies empirically investigates exactly this topic. A group of researchers, including psychologist Charles Spence and Colonna & Small‘s Maxwell Colonna-Dashwood, empirically investigated the effect of latte art on how people perceive and value coffee. What’s the practical upshot?

The results reported here suggest that the addition
of latte art influences how much people expect, and are willing to pay for milkbased coffees. As such, for the cafe owner thinking about how to increase profits, the experiments reported here suggest thapeople are willing to pay between 11–13% more for coffee with latte art than for those without it.

Very interesting finding!

The State of Reproducibility in Experimental Philosophy

Over at Feminist Philosophers, Dan Hicks casually mentions in a comment:

I haven’t had too many interactions with folks interested in experimental philosophy, but based on those interactions it seems like (1) x-phi folks are modeling their methods explicitly on these kinds of studies in psychology, while (2) not being aware of or responsive to the epistemic crisis surrounding these methods in psychology. So I would encourage x-phi folks to check out the links above.

Despite some initial defensiveness, I’ve come to think that Hicks’s remarks are actually quite reasonable for those philosophers who haven’t really kept up with the nitty-gritty happenings in experimental philosophy. Having heard about the replicability crisis in psychology, it is easy to assume that experimental philosophy is susceptible to the same problems. Having not kept up with the nitty-gritty happenings in experimental philosophy, it is easy to assume that experimental philosophers are unaware of or unresponsive to these issues.

The point of this post is to dispel those assumptions.

To be clear, my aim in writing this post isn’t really to pick on a casual comment, but to give a more general state of the art on what experimental philosophy, as a community, is doing and has been doing about methodological and epistemological issues related to reproducible science.

* * *

To start thinking about issues with reproducibility, we can go back to the context of Hicks’s comment: a “sexy” result from a low-powered social psychology study that says you can reduce implicit bias in your sleep. The replicability crisis in psychology has, in large part, to do with with replicating findings of a similar sort. As such, having learned about the replicability crisis, a philosopher is prima facie reasonable to cast a skeptical eye toward experimental philosophy findings.

The skeptical eye, I take it, has to do with a companions-in-guilt charge. Experimental philosophy often borrows methods from (social) psychology; since there’s a crisis in psychology, would it be such a surprise if experimental philosophy doesn’t suffer from the same methodological and epistemological issues?

There is something to the companions-in-guilt charge, but we should clarify who the companions really are. It’s not just psychology that suffers from a replicability crisis. It’s also cancer research, pharmaceutics, political science, behavioral economics, etc. In other words, there is a replicability crisis in science. So, philosophers should worry about experimental philosophy… insofar as and as much as they are worried about science.

* * *

 Is there reason to be more worried about experimental philosophy compared to psychology and other sciences? Not only do I think the answer to that question is ‘no’, but I think we in fact have reasons to be relatively optimistic about experimental philosophy.

The experimental philosophy community has been aware of the issues that underlie the replicability crises even before their recent publicity. For a long time (in a young research program), Joshua Knobe (and later Christian Mott) has maintained the experimental philosophy replication page, which documents experimental philosophy findings that have (and have not) replicated well. Looking at the replication page, one can see that the majority of effects in experimental philosophy have, in fact, been well-replicated. (The big exception is the cluster of demographic effects.) So, not only are experimental philosophers aware of methodological and epistemological issues with replicability, they have already taken some steps to address it.

The replication page is one exemplar of a broader pattern. The community, in general, is very welcoming to cumulative research practices, which encourage internal corrections and challenges. For example, on this very blog there was an extended discussion about the replicability of an age effect. For example, Chandra Sripada and Sara Konrath shared their data so that David Rose, Jonathan Livengood, Justin Sytsma, and Edouard Machery can re-analyze it to challenge Sripada and Konrath’s interpretation. For example, Adam Feltz and Florian Cova have systematically examined the moral responsibility and free will literature via a meta-analysis.

I’ll let you in on a little secret. In the earlier days of experimental philosophy (and to a lesser extent even today), many people ran their online studies using Knobe’s Qualtrics account. So everyone can in principle see what everyone else is working on, and even download the data if they wanted. In my view, the general culture of transparency discourages outright frauds like data manipulation or even faking data.

In addition to a general culture of transparent and cumulative research, experimental philosophers may also be in a better position than, say, psychologists because, well, they’re philosophers. And philosophy has a long tradition of thinking about the relationships between experiments, statistical inference, theory confirmation, etc. Some experimental philosophers are also philosophers of science who are attentive to such issues. For example, Edouard Machery has written about the interpretation of null results. But even the experimental philosophers who don’t also specialize in philosophy of science will have some exposure to these issues. And, based on the attendance at the Preconference Workshop on Replication in the Sciences at SPP, experimental philosophers are very keen to learn even more about these issues from both philosophers of statistics like Deborah Mayo, as well as methods experts from diverse cognate fields.

* * *

None of this is to say that current practices of experimental philosophy are anywhere near perfect. There is much more we can all do to do better: post data on repositories, run better powered studies, check with statistical and methodological consultants, keep up with latest best practices, etc. [I also have some thoughts about how to make the experimental philosophy replication page even more useful, but I’ll save that for another occasion.] But we can acknowledge the need for improvements in the future, while also acknowledge the efforts for improvements in the past. We can recognize how far experimental philosophy still has to go, with respect to the methodological and epistemological issues associated with the replicability crises, while also recognize how far it has come.

[Much of this post is based on a talk at the 2014 Experimental Philosophy UK Workshop. I thank the organizers for giving me the occasion to think more systematically about these issues.]

[x-posted at The Experimental Philosophy Blog; please comment there!]