Understanding and evaluating the behavior of technical users. A study of developer interaction at StackOverflow
© The Author(s) 2017
Received: 27 October 2016
Accepted: 11 February 2017
Published: 27 February 2017
This paper is focused on analyzing the details of one of the most widely used Q&A technical discussion forums on the Internet: StackOverflow. StackOverflow is a huge source of information for both academics and industry practitioners and its analysis can provide useful insights. This is evident by a growing body of work dedicated to the study of this platform. There are several papers that have taken the ‘platform’ as an analyzing point to gain a few useful insights into the practices adopted by software developers. However, different from current literature, the objective of this paper is to explore the habit of users, more specifically, we want to find out how do technical users behave online? In doing so, we attempt to present a new picture that focuses more on the human aspect of this online community. We neither focus on the technical aspects of the platform nor do we focus on the technical capabilities of developers, rather, the focus of this paper is to explore the humanistic point of view of technical users. To do that, we mine StackOverflow’s repositories to find a few patterns that raise several questions. The observations presented in this paper highlight real world implications of having open call based software engineering platforms. Further, we conduct a survey of StackOverflow users to find an answer to one of the highly talked about topics on Meta (a discussion forum for StackOverflow). The subject of the survey concerns changing participation habit of users. As the paper progresses forward, we discuss factors that raise the issue of professionalism and ethics at this online venue. The findings show several malpractices happening at StackOverflow. We present a detailed analysis and discuss these issues in this paper.
KeywordsCrowdsourcing Human computer interaction Human centered computing
Over the last few years, community based Q&A forums have gained a lot of popularity. These forums present an excellent opportunity to share knowledge, and provide an ideal medium for learning. In this paper, we focus our attention to one of these community driven forums. Specifically, we consider the case of StackOverflow (SO). Though speculative, it is recognized as one of the best forums to have technical discussions online. This statement is justified by the growing interest of literature in analyzing the platform from multiple perspectives. There exists work focusing on badges , identifying expert behavior , finding emotional cues in developers , analyzing and recommending tags , gamifying participation , predicting the status of questions , and so on. But, in contrast to typical software engineering studies, our work is motivated by the eagerness to explore human behavior in open crowd based platforms. In doing so, we aim to present a different side of the coin that focuses more on the humanistic side of technical users. This motivation of ours is backed by evidence in literature where work has tried to explore the typical habits of software developers in the past [7–9]. However, and in contrast to professional office environments, one can understand Q&A forums are crowd oriented and have a strong grounding in voluntarism. In such settings, there is no command–control model, and it is the efforts/will of the volunteers that keep the platform going. It would therefore be very much worth the while to analyze the inseparable link between the volunteer and his/her attitude towards the platform. Different from current literature that maintains a balance between the human and the platform, this paper has a skew towards the human. In our attempt, we focus on typical details of human nature, and its corresponding effect that dictates the interaction of people with the online forum. We have found a few interesting patterns that present several new standpoints and raises new questions. Some of the findings are quite surprising. A few, however, blurs the boundary between ethical and non ethical practices. We discuss these instances in detail and present evidence behind each of these problems. To discover such patterns and to find answers, we use two different kind of methods. First, we conduct statistical analysis to look for patterns. This is done by querying SO’s databases online.1 Second, as participating in SO is projected as a game in literature [1, 5, 10], therefore, one of our questions is—Why did you stop playing? To find an answer, we conduct a survey with SO users. This question of ours attracted a lot of debate with the users we contacted. In our discussion, there were normal SO users, high reputation users, current SO moderators, and ex-SO moderators. We discuss the details of our attempt from “The observations” section onwards.
It has been hypothesized in literature that such open forums have the potential of turning into a huge software repository . SO being one such forum has begun to warrant substantial attention in literature. There are several studies that focus on SO from a software engineering point of view [12–14]. Further, there are different papers that tries to analyze the technical aspect of SO. For instance , tries to find patterns that makes a good code example. Barua et al.  provide details about the discussion between developers and the latest trends in SO. Linares-Vásquez et al.  studies activity in SO when the Android API changes. Literature has also analyzed SO in context of mobile development . Campos et al.  uses the criterion of text based matching to find a potential programming question for a user to answer. Moreover, work has gone deeper and has found that high reputation users are efficient in providing good answers [20, 21]. However, in contrast to these work, the study conducted in this paper makes an attempt to present a different side of SO. Literature on SO has mostly focused its attention on the technical factors, and has overlooked the issue of the human factor. We, on the other hand, will focus our attention on this issue only. Having said that, there are however a few studies focusing on technical factors with a light touch on the human factors. This is done in the context of getting an answer accepted at SO. There is work targeting the human affective state  and the presentation quality  of an answer to get it accepted. We not only draw inspiration from these work, but go deep into the human aspects and explore the behavior of technical users in detail.
As this paper tries to explore a wide array of problem at SO, therefore, there are a few subparts of the study that have similarity to existing work. In this regard, the foundational work in  tries to analyze SO and correspondingly tries to teach us a few design lessons. The study however was conducted for the initial years of SO. The initial phase witnessed several upsides, however, after almost 8 years now that SO has matured, negatives have cropped up. We not only discuss these negatives, but also present additional details. As a result, we complement the work in literature, at much the same time, we also present a few directions to build additional constructive work. Similarly, analysis conducted in  specifies that experts as well as non-expert users exhibit logarithmic growth in participation. We present contradictory evidence, and back it with a survey of users, showing that this is not the case anymore. The experts have now backed off. As specified, this paper has a skew towards human behavior at SO. One of the factors discussed in the paper is gender based participation. In this regard, work presented in  focus on women in SO. The authors found that women tend to disengage more quickly than men though their level of activity is comparable to men. Our objective with respect to this work is different. We focus on the criterion of a person, either male or female, answering questions asked by the person of the same gender. Our findings revisit the construct of sexism (for both men and women). One of the subtopics of our analysis concerns spatial dynamics of an individual. To this end, Schnek et al.  explore the information flow among continents (Asia, Europe, North America etc.). In this paper, however, we do not focus information flow, rather, we focus on the criterion of spatial characteristics making an individual help another individual. The initial pattern we observed raises a few questions about the way people participate in the real world. To thoroughly test the absurd phenomenon, we go into detail. The findings definitely show a different side of the participants at SO. It is specified in literature  that the first person to answer a question gets a high score for their post. Furthermore, the median time to answer in SO is 16 min. Taking inspiration from this report, we conducted a few tests. The results we obtained raised a few doubts. In our endeavor to clear the dilemma, we found a pattern that revisited the issue of ethics. We analyzed more, and consequently, found additional unethical practices.
In software engineering literature there is a lot of work aligned towards exploring the habits of developers. For instance, [7, 27] emphasizes on the methods and tools used by developers. LaToza et al.  points out the importance of mental models employed by software engineers. Further, there is also evidence suggesting the impact of a developer’s social circle on a project’s outcome [9, 28]. The authors of  focus on the effect of team familiarity in software development. To sum up the work in Software Engineering, there is strong evidence in literature that targets the mindset of developers. One has to understand, however, in contrast to developers working in controlled office environments, open forums are purely volunteer dependent, where every volunteer has his/her own set of standard, expectation, method of working, and motivation. It is therefore logical to analyze these attributes and put them under scrutiny. This paper has the motive to explore some of those humanistic characteristics.
StackOverflow is an online system where anyone can ask a question and get an answer from the crowd.
It has more than 4.8M users, 11M questions, 19M answers, 47M comments.
Meta is the venue where users discuss matters related to the everyday activities of SO.
Posts in SO terminology refers to questions as well as answers.
A user is awarded score points for good posts.
Reputation of users is determined by the score of the post.
A good number of reputation points results in badges.
Participation at SO is ‘gamified’ using the idea of reputation points and badges.
Nation wise responses
Ans by/ ques by
City wise responses, USA
Avg answers from same city
Avg ans from rest of the country USA
Why did you stop playing?
City wise responses, India
Avg answers from same city
Avg ans from rest of the country, India
The quantitative result for the reasons is shown in Fig. 1. The complete results are also are presented in Table 5. In the same table, the mean and the standard deviation of the reasons is presented. As users’ response can vary, therefore, Correlation Coefficient (PCC) b/w top 33% and last 33% is also shown. It is visible from the Table that the value for PCC is varying across the identified reasons. But, there is one point of interest that has high mean, low std dev, and a positive PCC value: lack of time. It is the top reason to decrease the activity. To emphasize this point, several users pointed out that they have moved in their lives, and no longer have the time to dedicate to SO. Some have moved to better positions in their work (for instance senior management), others have found jobs, some feel they need to dedicate their time someplace else, and so on. Addiction to SO was a phase in life, and they said the phase is now over. Close to lack of time, is the reason, maturity of the users. In this respect, gaining experience with time is an evolutionary human process, therefore, it is natural that users have become picky in their choice of questions. This problem has a strong relationship with yet another reason for decreasing participation, the issue of finding appropriate questions to answer. This particular category blurs the boundary between the human factors and the search algorithms developed by the system. If a search algorithm is able to match the right question for a user, then the issue of maturity and locating appropriate questions to answer will be solved. However, developing a good search algorithm for questions with code and explanation in Natural language, personalized for every user, is non-trivial.
Number of test subjects USA, city vs. rest of the country
Reason to decrease participation
I felt bored, SO is not fun anymore
Redundant/duplicate questions have cropped up
Quality of questions is not good anymore
Questions are now one liners
Difficult to find appropriate questions to answer
Lack of time. Busy with current work
There is a lot of negativity in SO
I have reached my personal goal
There are a lot of people to answer questions
New users don’t know how to ask questions or accept good answers
Problems with SO management
I have matured and have gained experience, so I am picky in my questions
Going by quantity presented in Fig. 1, duplicate questions is also among the top contenders, but the correlation is not strong. Nevertheless, emphasizing on the point of duplicacy, it is an accepted fact at SO that there are questions that are asked several times. Quantitatively speaking, this property can compel some people to decrease their activity as there is nothing new to learn and no new problems to solve. Hence, one can get ‘bored’. In a different context, it has other consequences as well. This consequence will be discussed later in “Plagiarism and taking credit” section.
All the factor discussed in this section are some of the reasons for users to decrease their activity. However, an important point of note discovered during the survey is the factor concerning Quality and duplicacy of questions. There are several posts on Meta that states SO has an increasing number of low quality questions.4 Further, users also argue about the increase in duplicate questions. There is also a discussion among SO users that quality is one of the reasons why people do not like participating. To quote from one the those discussions, ‘Yes, because the quality of “new”? questions is going in the toilet. Almost everything I see **** or is a duplicate’. However, our survey neither found the quality of questions nor duplicate questions as a possible reason to reduce participation. Though, for some users it can be a reason to stop playing, but the users we contacted did not agreed to this reason.
Quid pro quo
Gender based participation
Time to answer and number of answer in different interval of time
Does gender make a difference?
Number of answers between 0–59 s
By same user
By different user
Number of subjects in different countries
Plagiarism and taking credit
In this subsection, we highlighted the presence of plagiarism at SO, but the motive of this paper is not to provide a solution to this problem. Our objective is to bring this (and others) problems into the community so that additional investigation and constructive work can be built upon. Having said that, we will emphasize on plagiarism. Such instances happen because of three simple reasons. First, there are open ‘duplicate questions’. Second, one asking the question did not look hard enough to find the answer. Third, one answering the question wants to gain reputation points quickly without any effort.
Visit my website to find answers!!
Addiction or something else?—Long answers within 1 min!!
Before beginning the discussion in this subsection, we first show the number of questions answered based on the parameter: time to answer. The result is shown in Table 7. It is visible from the table that a good number of questions are answered within 10 minutes. An important point of note here is a lot of questions are also answered within 59 s (25,543). Though, some answers contain very little text, for instance PostIDs 80299 or 117256. However, for other questions, the observation raises a few doubts. In simple words, long answers coming in that quickly imply instances that are worth investigating. To go into the details, we designed a few tests. In the experiment, we considered answers with body length more than 500, 1000, 1500, 2000 chars resp (long answers). The motive was: If long answers are coming within a matter of seconds, then the answers are written in advance or people are trying to collude with others. The motive here is to gain reputation points. Therefore, we categorized the test in two different ways. First, we took the same person asking and answering a question. Second, we look at instances where two different users ask and answer questions. The results for the two scenarios are shown in Table 8. In the following text, for reasons of brevity, we concentrate only on 0–59 s. It is visible from the Table that there are substantial posts satisfying this criterion (answers coming within 59 s). To be specific, the number is 10,948. In this regard, lets consider the another worst case where length(ans) >2000, i.e. we have a sufficiently long answer. For this case, we present the average score of the question as well as the answer. Recall that score is responsible for increasing reputation. In cases when one who has asked a question provides an answer, the avg score (of question and answer) is 31 and 19 respectively. In other words, when one follows this behavior then one obtains 50 \(\times\) 5 reputation points (31 + 19 for the score of the post (question and answer both), reputation = reputation + score \(\times\) 5). In the other case, i.e. when two different users are participating, then the average score of questions and answers is 44 and 40 respectively, implying an increment of 220 points for the asker and 200 points for the answerer. Such high figures indicate motivation to gain reputation by using unfair means. As discussed in the beginning of this section, getting good score results in good reputation points. Further, it is also known that users of SO seek reputation. The evidence presented here indicates unfair practice to gain fake Internet points.
The evidence in the previous paragraph indicates cases of collusion and use of unfair means to gain reputation points. However, consider the case of the same user. It is possible that a few users want to share their knowledge with others, whereas others are doing this for the mere purpose of gaining reputation points. Unfortunately, to clear this doubt, we would require a manual survey of such users. In this regard, the evidence show classic symptoms of collusion, the chances that any of these users will agree to the point that they wanted to gain reputation by unfair means is unlikely. Moreover, if we think about this observation, we could also ask: Does this behavior of people (specifically, when two different users ask and answer question immediately) effect the overall “perception” and “fairness” at SO? This question remains unanswered in this paper.
Discussion and threats to validity
The paper opens up several new directions for researchers working in interdisciplinary sciences. For example, researchers in sociology, psychology can take the investigations initiated further and into the domain of specialized professionals, preferably technical users, and explore their behavior online. One has to appreciate that the behavior in an online environment can quite possibly be different from that in an office environment. Through this work, we offer some guidelines and a few starting points to put this idea under scrutiny.
Research efforts in crowdsourcing can benefit from this work in their endeavours towards targeting the appropriate set of people while assigning crowdsourcing tasks. Programmers usually comprise the niche group that contributes to crowdsourcing tasks. The assumption is that programmers are thorough professional and are neutral and unbiased in their approach. In this paper, and as is also corroborated by existed literature, we find that software programmers are quite ‘human’ in their approach . We extend the idea in  and highlight a few issues that so far eludes literature. Building upon the observations discussed in the paper, future work can explore the humanistic point of view of software programmers.
The first threat is in spatial characteristics. We found statistical evidence that indicated people preferred answering to a fellow from the same locality more. Moreover, we found precedent in literature  and tested this observation by taking in different criterion and test subjects. To unearth this fact, we needed to extract the location of users from SO’s databases. In this regard, it should be noted here that it is not mandatory that a user at SO must make his/her location public. That is, users can choose not to display their location. As a result, the analysis did not included the entire user base of SO. Though, the number of test subjects (Table 9) is acceptable, however, we could not include the entire user base of SO. This therefore is the first shortcoming with the study conducted in the paper.
The second threat is the survey of users conducted in the paper. We conducted the survey with 34 subjects. Owing to the reason pointed out section in 3.2, we could not include more users. Therefore, the reasons to decrease participation could suffer from the decline effect. Nevertheless, the survey conducted in the paper can indeed present several guidelines and can serve as a reference point for future work.
The third threat is in gender based participation. Similar to point 1, we must point out that we could not include all users in the dataset. This is because some users have display name like NPE (userid 367273). It is therefore difficult to classify such users as either males or females. Moreover, classifying users as males or females on the basis of their names is by definition biased (for example Simon can be used for both males and females). As pointed out in “Does gender make a difference?” section , we conducted the test 10 times by taking in different number of test subjects from the dataset. But, it is not claimed here that the observation is true for the entire user base of SO. Moreover, we do not claim that there is Sexism at SO. To confirm the existence of such a pattern, we need to include the entire users of SO. This, however, is easier said than done. This is because we need to classify all users into males and females accurately.
The paper reported on cases where users prefer answering to a person from the same locality more.
Through a survey, possible reasons to decrease one’s activity were discussed.
The idea of helping one and expecting help in return was nullified.
It was found that gender plays an important role at SO.
The findings further reported on the practice of plagiarism.
Through this study, instances where users employ SO as a launchpad for their personal websites were discovered.
Possible cases of collusion (to gain reputation points) were presented.
TA carried out the study in the paper, and drafted the first version of the manuscript. TA and AS designed the framework and developed the proposed method together. AS revised the first and second version of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Funding information is not applicable for this work.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Immorlica N, Stoddard G, Syrgkanis V (2015) Social status and badge design. In: Proceedings of the 24th international conference on World Wide Web. International World Wide Web Conferences Steering Committee. pp 473–483
- Pal A, Harper FM, Konstan JA (2012) Exploring question selection bias to identify experts and potential experts in community question answering. ACM Trans Inf Syst (TOIS) 30(2):10View ArticleGoogle Scholar
- Murgia A, Tourani P, Adams B, Ortu M (2014) Do developers feel emotions? An exploratory analysis of emotions in software artifacts. Proceedings of the 11th working conference on mining software repositories, ACM, pp 262–271
- Xia X, Lo D, Wang X, Zhou B (2013) Tag recommendation in software information sites. In: Proceedings of the 10th working conference on mining software repositories, IEEE Press, pp 287–296
- Cavusoglu H, Li Z, Huang KW (2015) Can gamification motivate voluntary contributions? The case of StackOverflow Q&A community. In: Proceedings of the 18th ACM conference companion on computer supported cooperative work and social computing, ACM, pp 171–174
- Correa D, Sureka A (2013) Fit or unfit: analysis and prediction of closed questions on Stack Overflow. In: Proceedings of the first ACM conference on online social networks. ACM, pp 201–212
- Cherubini M, Venolia G, DeLine R, Ko AJ (2007) Let’s go to the whiteboard: how and why software developers use drawings. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 557–566
- Espinosa JA, Slaughter SA, Kraut RE, Herbsleb JD (2007) Familiarity, complexity, and team performance in geographically distributed software development. Org Sci 18(4):613–630View ArticleGoogle Scholar
- LaToza TD, Venolia G, DeLine R (2006) Maintaining mental models: a study of developer work habits. In: Proceedings of the 28th international conference on software engineering. ACM, pp 492–501
- Vasilescu B, Capiluppi A, Serebrenik A (2013) Gender, representation and online participation: a quantitative study. Inter Comput 26(5):47Google Scholar
- Dabbish L, Stuart C, Tsay J, Herbsleb J (2012) Social coding in GitHub: transparency and collaboration in an open software repository. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, ACM, pp 1277–1286
- Morrison P, Murphy-Hill E (2013) Is programming knowledge related to age? An exploration of Stack Overflow. In: 2013 10th IEEE working conference on mining software repositories (MSR). IEEE, pp 69–72
- Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web? In: 2011 33rd international conference on Software engineering (ICSE). IEEE, pp 804–807
- Wang S, Lo D, Jiang L (2013) An empirical study on developer interactions in StackOverflow. In: Proceedings of the 28th annual ACM symposium on applied computing. ACM, pp 1019–1024
- Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example? A study of programming Q&A in StackOverflow. In: 2012 28th IEEE international conference on Software maintenance (ICSM). IEEE, pp 25–34
- Barua A, Thomas SW, Hassan AE (2014) What are developers talking about? An analysis of topics and trends in Stack Overflow. Empir Softw Eng 19(3):619–654View ArticleGoogle Scholar
- Linares-Vásquez M, Bavota G, Di Penta M, Oliveto R, Poshyvanyk D (2014) How do API changes trigger Stack Overflow discussions? A study on the android SDK. In: Proceedings of the 22nd international conference on program comprehension. ACM, pp 83–94
- Rosen C, Shihab E (2016) What are mobile developers asking about? A large scale study using StackOverflow. Empir Softw Eng 21(3):1192–1223View ArticleGoogle Scholar
- Campos EC, Souza LB, Maia MD (2016) Searching crowd knowledge to recommend solutions for API usage tasks. J Softw Evol Process 28:863–892View ArticleGoogle Scholar
- Anderson A, Huttenlocher D, Kleinberg J, Leskovec J (2012) Discovering value from community activity on focused question answering sites: a case study of Stack Overflow. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining, ACM, p 850–858
- Bazelli B, Hindle A, Stroulia E (2013) On the personality traits of StackOverflow users. In: ICSM '13 proceedings of the 2013 IEEE international conference on software maintenance, IEEE Computer Society, Washington, 22–28 Sept 2013, pp 460–463
- Asaduzzaman M, Mashiyat AS, Roy CK, Schneider KA (2013) Answering questions about unanswered questions of Stack Overflow. In: Proceedings of the 10th working conference on mining software repositories. IEEE Press, p 97–100
- Mamykina L, Manoim B, Mittal M, Hripcsak G, Hartmann B (2011) Design lessons from the fastest Q&A site in the west. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 2857–2866
- Movshovitz-Attias D, Movshovitz-Attias Y, Steenkiste P, Faloutsos C (2013) Analysis of the reputation system and user contributions on a question answering website: Stackoverflow. In: 2013 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM). IEEE, pp 886–893
- Schenk D, Lungu M (2013) Geo-locating the knowledge transfer in StackOverflow. In: Proceedings of the 2013 international workshop on social software engineering. ACM, pp 21–24
- Bhat V, Gokhale A, Jadhav R, Pudipeddi J, Akoglu L (2014) Min (e) d your tags: analysis of question response time in Stackoverflow. In: 2014 IEEE/ACM international conference on Advances in social networks analysis and mining (ASONAM). IEEE, pp 328–335
- Koenemann-Belliveau J (1991) Empirical studies of programmers: fourth workshop. Intellect Books
- Krasner H, Curtis B, Iscoe N (1987) Communication breakdowns and boundary spanning activities on large programming projects. In: Empirical studies of programmers: second workshop. Ablex Publishing Corp, pp 47–64
- Reicher S (2001) The psychology of crowd dynamics. In: Michael AH, Tindale RS (eds) Blackwell Handbook of Social Psychology: group processes, Blackwell, Oxford, pp 182–208Google Scholar
- Piliavin IM, Rodin J, Piliavin JA (1969) Good samaritanism: an underground phenomenon? J Personal Soc Psychol 13(4):289View ArticleGoogle Scholar
- Emswiller T, Deaux K, Willits JE (1971) Similarity, sex, and requests for small favors. J Appl Soc Psychol 1(3):284–291View ArticleGoogle Scholar
- Dovidio JF, Gaertner SL (1981) The effects of race, status, and ability on helping behavior. Soc Psychol Q 44:192–203View ArticleGoogle Scholar
- Bradner E, Mark G (2002) Why distance matters: effects on cooperation, persuasion and deception. In: Proceedings of the 2002 ACM conference on computer supported cooperative work. ACM, pp 226–235
- Turner RH, Killian LM (1957) Collective behavior. Prentice Hall, Englewood CliffsGoogle Scholar
- Vasilescu B (2014) Software developers are humans, too! In: Proceedings of the companion publication of the 17th ACM conference on computer supported cooperative work and social computing. ACM, pp 97–100