![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Framing and Question Ambiguity
Sep. 10th, 2009 12:04 pmOne of the pleasures of big disciplinary conferences is the collision of papers that don't necessarily have much in common. (Of course, one of the downsides of big national conferences is...being in a room with people who don't have anything to say to each other. But anyway.) At APSA, I was on a panel about race, racisms, and xenophobia, on which I presented a paper on theories of recognition, the Arab-American experience of injustice, and why we need to consider discursive misrecognition as a real field for social struggle. In classic APSA form, I, an interpretivist ethnographer presenting a paper on political theory was paired with one historian and two survey research behavioralists/political psychologists. Nevertheless, the research was all fascinating, and I was glad to be on the panel.
One of the other papers was Measuring Respondent Agreement/Disagreement with Framing Experiments: Race, Religion and Voting Against Barack Obama in 2008, which Baretto and Redlawsk presented handily. You can read their abstract and paper at the link provided, but, essentially, they tested in what ways framing Obama as pro-Muslim, pro-Christian, or pro-black influenced how people later ranked him on various measures. They found that a successful framing of him as pro-Muslim hurt his numbers, pro-Christian helped a little but a failure of that frame hurt him more, and that being pro-black actually helped his numbers. They also explored the evidence that whether or not the interviewee agreed with the framing question had an independently significant effect on their support for Obama in later questions.
I found their work extremely interesting. The idea of 'framing,' is, in many ways, the way folks who don't come from my theoretical corner talk about what I call discourse; one could gloss 'discourse,' say, as the sum total of plausible frames, a sort of grammar of possibilities for understanding a political incident. I don't think this is a perfect definition of discourse, but it works, and certainly captures much of what I mean when I say it. Part of what was so interesting about this paper was that it allowed for the possibility that individuals could either accept or reject the frame presented to them: they could either agree with the way the question positioned Obama as pro-black, pro-Muslim, or pro-Christian, or disagree with it, and that choice influenced their later answers. This really is an excellent addition to the use of framing in research.
At the same time, I found myself having any number of thoughts about the structure of the questions that Baretto et al asked. I'm an interpretivist, methodologically; that means that the questions I ask about politics have to do with meaning. And these questions seem to 'mean' a number of things to me, many of which were not intentional.
Here are the three framing questions (from page 14 of the paper):
The Muslim question stopped me, every time I read it. Why? Because if you asked me that question--before the election, today, or any other day--I would answer either "very much" or "somewhat," depending on my mood. And then, if you asked me about how I felt about Obama, I would reply (again, depending on mood and recent policy decisions) fairly highly. I think it is good that an elected official be able to sympathize with the Muslim community, because I think the Muslim community has been discriminated against, and that sympathy with them might mean less systemic discrimination. Now, based on the numbers Baretto et al got, clearly I'm in the minority in this reading. But I'm not alone: on page 19, we see that 20% of those who agree with the Muslim framing rate Obama in the top quadrant in their thermometer scale, and 19% say they "often" have feelings of hope towards him or his policies.
The Christian frame also made me think. Those who agreed that Obama was sympathetic to Christians rated him reasonably highly; those who disagreed rated him very poorly on the thermometer scale at the end of the poll. My guess--and I'd have to do research to back this up, but it's got some face validity--is that those who said he was not sympathetic to 'the Christian community' had a very specific notion of who is in that Christian community. Recently in American politics, "Christians" has come to mean primarily right-leaning evangelical Protestants, and often them alone: not Catholics, not mainline Protestants, not left-leaning Christians of all denominations. Most folks who self-identify in that community are not pro-Obama, to say the least. Hearing this question, their construction of what it would take to be "pro-Christian" requires not just being familiar with the community, not just being nominally a Christian or attending a church (not a masjid or synagogue), but having a set of particular and contested religious and policy goals. Obama just doesn't. But folks who thought he was sympathetic to the Christian community probably meant another sort of Christian community. Because of these differences in the construction of the community that they are being asked about, these different answers are measuring radically different things.
In general, I'm not certain if the "sympathetic to" construction is the right one here. That element of the question is where the majority of the multiple possible readings are situated. I want to know what "sympathetic to" means, and I'm pretty sure it means different things to different interviewees. Survey researchers are not a fan of questions with so many possible readings; however, I'm of the opinion that every question is going to have multiple meanings, and that you can't escape offering complex and divergent thoughts in any of your polling information. What I do think, however, is that we need to actually explore those complexities, rather than trying (and inevitably failing) to exclude them. I think this requires research more complicated than a 20 minute phone poll. I think it requires actually asking about meaning, rather than leaving them unconstructed. That doesn't make Baretto et al's paper less interesting to me; it just opens up a new set of doors that I'd like to go down, or at least see someone else go down.
One of the other papers was Measuring Respondent Agreement/Disagreement with Framing Experiments: Race, Religion and Voting Against Barack Obama in 2008, which Baretto and Redlawsk presented handily. You can read their abstract and paper at the link provided, but, essentially, they tested in what ways framing Obama as pro-Muslim, pro-Christian, or pro-black influenced how people later ranked him on various measures. They found that a successful framing of him as pro-Muslim hurt his numbers, pro-Christian helped a little but a failure of that frame hurt him more, and that being pro-black actually helped his numbers. They also explored the evidence that whether or not the interviewee agreed with the framing question had an independently significant effect on their support for Obama in later questions.
I found their work extremely interesting. The idea of 'framing,' is, in many ways, the way folks who don't come from my theoretical corner talk about what I call discourse; one could gloss 'discourse,' say, as the sum total of plausible frames, a sort of grammar of possibilities for understanding a political incident. I don't think this is a perfect definition of discourse, but it works, and certainly captures much of what I mean when I say it. Part of what was so interesting about this paper was that it allowed for the possibility that individuals could either accept or reject the frame presented to them: they could either agree with the way the question positioned Obama as pro-black, pro-Muslim, or pro-Christian, or disagree with it, and that choice influenced their later answers. This really is an excellent addition to the use of framing in research.
At the same time, I found myself having any number of thoughts about the structure of the questions that Baretto et al asked. I'm an interpretivist, methodologically; that means that the questions I ask about politics have to do with meaning. And these questions seem to 'mean' a number of things to me, many of which were not intentional.
Here are the three framing questions (from page 14 of the paper):
Muslim Frame:
Because his uncle in Kenya is Muslim, and for a few years he was raised in Indonesia, a
Muslim country; how much do you think Barack Obama can sympathize with the Muslim
community in America? Is it very much, somewhat, only a little, not at all?
Black Frame:
Because he worked as a community organizer for a Black church in Chicago, and
represented a majority-Black district in the Illinois Senate; how much do you think
Barack Obama can sympathize with the Black community in America? Is it very much,
somewhat, only a little, not at all?
Christian Frame:
Because he was married in, and attended a Christian church, and has stressed his
Christian values; how much do you think Barack Obama can sympathize with the
Christian community in America? Is it very much, somewhat, only a little, not at all?
The Muslim question stopped me, every time I read it. Why? Because if you asked me that question--before the election, today, or any other day--I would answer either "very much" or "somewhat," depending on my mood. And then, if you asked me about how I felt about Obama, I would reply (again, depending on mood and recent policy decisions) fairly highly. I think it is good that an elected official be able to sympathize with the Muslim community, because I think the Muslim community has been discriminated against, and that sympathy with them might mean less systemic discrimination. Now, based on the numbers Baretto et al got, clearly I'm in the minority in this reading. But I'm not alone: on page 19, we see that 20% of those who agree with the Muslim framing rate Obama in the top quadrant in their thermometer scale, and 19% say they "often" have feelings of hope towards him or his policies.
The Christian frame also made me think. Those who agreed that Obama was sympathetic to Christians rated him reasonably highly; those who disagreed rated him very poorly on the thermometer scale at the end of the poll. My guess--and I'd have to do research to back this up, but it's got some face validity--is that those who said he was not sympathetic to 'the Christian community' had a very specific notion of who is in that Christian community. Recently in American politics, "Christians" has come to mean primarily right-leaning evangelical Protestants, and often them alone: not Catholics, not mainline Protestants, not left-leaning Christians of all denominations. Most folks who self-identify in that community are not pro-Obama, to say the least. Hearing this question, their construction of what it would take to be "pro-Christian" requires not just being familiar with the community, not just being nominally a Christian or attending a church (not a masjid or synagogue), but having a set of particular and contested religious and policy goals. Obama just doesn't. But folks who thought he was sympathetic to the Christian community probably meant another sort of Christian community. Because of these differences in the construction of the community that they are being asked about, these different answers are measuring radically different things.
In general, I'm not certain if the "sympathetic to" construction is the right one here. That element of the question is where the majority of the multiple possible readings are situated. I want to know what "sympathetic to" means, and I'm pretty sure it means different things to different interviewees. Survey researchers are not a fan of questions with so many possible readings; however, I'm of the opinion that every question is going to have multiple meanings, and that you can't escape offering complex and divergent thoughts in any of your polling information. What I do think, however, is that we need to actually explore those complexities, rather than trying (and inevitably failing) to exclude them. I think this requires research more complicated than a 20 minute phone poll. I think it requires actually asking about meaning, rather than leaving them unconstructed. That doesn't make Baretto et al's paper less interesting to me; it just opens up a new set of doors that I'd like to go down, or at least see someone else go down.
