1. Study questions: How users of social Q & A evaluate the quality of answers? Are there any criteria they prefer in identifying answer quality?
2.Unit of Analysis: Users of Stack Overflow (a social Q&A site for programmers: http://stackoverflow.com/)
3.Data Collection: Data collection consist of content analysis of the comments of high-quality answers (upvote score >100), statistical analysis about the relationship between upvote and different features presented in the answers, and interviews with users.
4. Data Analysis:
a) All of the interviews will be audiorecorded, transcribed, and coded using NVivo 10. Two researchers independently code all of the interviews to identify major themes for each research question. After comparing, discussing, and resolving any differences in their coding, the researchers create a codebook and use it to recode all of the data.
b) We will filter out unqualified comments and then identify specific accounts of “selection criteria” in the qualified comments using an inductive content analysis. Two researchers participate in the analysis. The initial coding scheme emerge from interview findings and previous research.
c) We will divide answer feature set into 8 groups: Structure, Length, Style, Relevance, Review History, User, User Graph, and Readability. We then conduct two series of experiments. First, we represent our question-answer pairs using only the features of each group in isolation, in order to determine the individual impact of the group. Following, we represented the Q&A pairs using all the features, leaving out one group at a time. By this way, we can verify how each group is able to contribute to the results, independently from the other groups.
5. Alternative Design Possibilities: I’m very interested in using survey to examine what selection criteria users prefer in identifying high-quality answers regrading programming, because I assume the criteria they use will be different based on individual characteristics (e.g. how long they use that site, how active they are on that site, their own expertise, etc.). Survey opens up the possibilities to approach a relatively large size of studying subjects directly (comments and statistical models can only reflect their opinions from a data perspective). However, this site does not offer in-site message function and only part of the users provide their contact information (email, etc.)on the profile pages, so sampling will be a problem if I decide to use a survey.