With more than 4,100 properties in more than 90 countries, Accor Hospitality was facing pressure from customers, as well as from shareholders, to increase customer satisfaction and quality of service during an economic downturn. It thus turned to Synthesio, a global, multilingual social-media monitoring-and-research company, to examine the more than 5,000 customer opinions posted each month on travel websites worldwide concerning Accor's various brands. Accor saw its main challenge as how to quickly identify customer dissatisfaction and then correct problems at their source. Synthesio created a tool to track the online reputations of 12,000 hotels, including those run by Accor and those run by its competitors. It quickly revealed a number of problems Accor guests were having; for example, room keys were being demagnetized unintentionally by their smartphones. Accor was then able to work with its room-key supplier to address the problem. Accor was also able to set up a rewards-and-training program that encouraged its individual hotels to connect with their guests through online conversations. Within one year of contracting Synthesio, the Novotel brand within the Accor group saw 55% growth in positive feedback in online posts, along with a significant decrease in the number of negative comments.
Social media analytics "is concerned with developing and evaluating informatics tools and frameworks to collect, monitor, analyze, summarize, and visualize social media data ... to facilit[ate] conversations and interactions ... to extract useful patterns and intelligence..."28 Accor illustrates how social media analytics can help businesses improve their reputations and resulting business performance. Ubiquitous smartphones and other mobile devices, Facebook and YouTube channels devoted to companies and products, and hashtags that make it easy to share experiences instantly combine to create a social media landscape that is quickly becoming part of the fabric of everyday business operations. As the number of users on social media sites continues to grow, so does the need for businesses to monitor and use them to their advantage.
Through the rest of this article, we explore how social media popularity necessitates use of social-media analytics, the underlying stages of the analytics process, the most common social media analytic techniques, and the ways analytics creates business value.
In the early days of social mediathe mid-2000sPR agencies would monitor customers' posts on a business's own website to try to identify and manage unhappy customers. With the explosion in the number of social media sites and volume of users on them, monitoring alone is not enough to render a complete picture of how a company is doing. Consider the pervasiveness of social mediaa:
- Social networking is the most popular online activity;
- 91% of adults online are regular users of social media; and
- Facebook, YouTube, and Twitter are the second, third, and eighth most-trafficked sites on the Internet, as of April 2014.
However, even these statistics do not fully account for the influence social media has on our lives. Users spend more than 20% of their time online on social media sites. Facebook alone has a worldwide market penetration rate over 12% of the entire online population; in North America it is 50%. These rates are growing quickly, with Facebook alone gaining 170 million new users between the first quarter of 2011 and the first quarter of 2012, an increase of 25%. Facebook mobile use is growing even more quickly, at a 67% annual clip, as of Summer 2013.
The amount of information seen by all these users on a typical day gives a clearer indication of the enormous influence of social media. As of October 2012, Facebook's nearly one billion active users collectively were spending approximately 20,000 years online each day. In the same period, YouTube reported more than one billion views, or 500 years of video (spread among 800 million unique users), and 140 million active Twitter users sent more than 340 million tweets.
These are not simply passive uses. YouTube's analysis of its videos indicates 100 million people take some sort of "social action" each week, by, say, liking, disliking, or commenting on what they see. These actions doubled from 2012 to 2013. Facebook integrates social actions in its online ads today by, for instance, allowing users to see if their friends have liked or voted on products being advertised. Likewise, hashtags on Twitter (as well as other social-media platforms) give users another quick and easy way to express their likes, dislikes, interests, and concerns, presenting further opportunities (or challenges) to businesses striving to use them.
Social media analytics involves a three-stage process: "capture," "understand," and "present" (see Figure 1), the CUP framework. Capture involves obtaining relevant social media data by monitoring, or "listening," to various social-media sources, archiving relevant data and extracting pertinent information. It can be done by a company itself or through a third-party vendor. Not all captured data is useful, however. Understand selects relevant data for modeling while removing noisy low-quality data, using various advanced data analytic methods on the data and gain insight from them. And present deals with displaying findings from the understand state in a meaningful way. We derived the CUP framework from familiar, broadly applied input-process-output models, making it consistent with Zeng et al.,28 whose monitor-and-analyze activities were subsumed by our understand stage and whose summarize-and-visualize activities fall within our present stage.
There is also some overlap among the stages; for instance, the understand stage creates models that can help in the capture stage. Likewise, visual analytics supports human judgments that complement the understand stage, as well as assist in the present stage. The stages are conducted in an ongoing, iterative manner rather than strictly linearly. If the models in the understand stage fail to reveal useful patterns, they may be improved by capturing additional data to increase their predictive power. Likewise, if presented results are not interesting or lack predictive power, it may be necessary to return to the understand or capture stages to adjust the data or tune the parameters used in the analytics. A system supporting social media analytics may go through several iterations before being truly useful. Data analysts and statisticians help develop and test systems before others use them.
Capture. For a business using social media analytics, the capture stage helps identify conversations on social media platforms related to its activities and interests. This is done by collecting enormous amounts of relevant data across hundreds or thousands of social media sources using news feeds and APIs or through crawling. The capture phase covers popular platforms (such as Facebook, Foursquare, Google+, LinkedIn, Pinterest, Twitter, Tumblr, and YouTube), as well as smaller, more specialized sources (such as Internet forums, blogs, microblogs, wikis, news sites, picture-sharing sites, podcasts, and social-bookmarking sites). An enormous amount of data is archived and available to meet businesses' needs. To prepare a dataset for the understand stage, various preprocessing steps may be performed, including data modeling, data and record linking from different sources, stemming, part-of-speech tagging, feature extraction, and other syntactic and semantic operations that support analysis. Information about businesses, users, and events, as well as user comments and feedback and other information, is also extracted for later analytical modeling and analysis.
Being tuned in to changing customer tastes and behavior, businesses can anticipate significant changes in demand and adjust accordingly by ramping production up or down.
The capture stage must balance the need to find information from all quarters (inclusivity) with a focus on sources that are most relevant and authoritative (exclusivity) to assist in more refined understanding.
Understand. When a business collects the conversations related to its products and operations, it must then assess their meaning and generate metrics useful for decision makingthe understand stage. Since the capture stage gathers data from many users and sources, a sizeable portion may be noisy and thus have to be removed prior to meaningful analysis. Simple, rule-based text classifiers or more sophisticated classifiers trained on labeled data may be used for this cleaning function. Assessing meaning from the cleaned data can involve statistical methods and other techniques derived from text and data mining, natural language processing, machine translation, and network analysis.9 The understand stage provides information about user sentimenthow customers feel about a business and its productsand their behavior, including the likelihood of, say, purchasing in response to an ad campaign. Many useful metrics and trends about users can be produced in this stage, covering their backgrounds, interests, concerns, and networks of relationships.
Note the understand stage is the core of the entire social media analytics process. Its results will have a significant effect on the information and metrics in the present stage, thus the success of future decisions or actions a business might take. Depending on techniques used and information sought, certain analyses may be preprocessed offline while others are computed on the fly using data structures optimized for anticipated ad hoc uses. Analysts and business managers may participate directly in the understand stage when visual analytics allows them to see various types and representations of data at once or create visual "slices" that make patterns more apparent.
Present. In this last stage, the results from different analytics are summarized, evaluated, and shown to users in an easy-to-understand format. Visualization techniques may be used to present useful information; one commonly used interface design is the visual dashboard, which aggregates and displays information from multiple sources. Sophisticated visual analytics go beyond the simple display of information. By supporting customized views for different users, they help make sense of large amounts of information, including patterns that are more apparent to people than to machines. Data analysts and statisticians may add extra support.
Social media analytics encompasses a variety of modeling and analytical techniques from different fields. Here, we highlight the most instrumental in understanding, analyzing, and presenting large amounts of social media data. Some techniques support several stages of social media analytics: Sentiment analysis and trend analysis primarily support the understand stage; topic modeling and social network analysis have primarily application in the understand stage but can support the capture and present stages as well; and visual analytics spans the understand and the present stages.
Opinion mining, or sentiment analysis, is the core technique behind many social media monitoring systems and trend-analysis applications.b It leverages computational linguistics, natural language processing, and other methods of text analytics to automatically extract user sentiment or opinions from text sources at any level of granularity (words or phrases, up to entire documents). Such subjective information extracted about people, products, and services supports predicting the movement of stock markets, identifying market trends, analyzing product defects, and managing crises. Relatively simple methods for sentiment analysis include word (phrase) counts (the more a product is mentioned, the more it is assumed to be liked); "polarity lexicons," or lists of positive and negative terms that can be counted when used, as in, say, text messages that mention a product by name;11 and semantic methods that may compute lexical "distances" between a product's name and each of two opposing terms (such as "poor" and "excellent") to determine sentiment.25 More complicated approaches distinguish the sentiments about more than one item referenced in the same text item (such as a sentence, paragraph, or text message).10
All told, both sophisticated and simple methods of sentiment analysis can be effective or flawed, though most research involving texts, tweets, and other short messages involves simple techniques. Though sentiment analysis is increasingly common, sampling bias in the data can skew resultseven if large data samples are confused with unbiased samplesespecially in situations where satisfied customers are silent while those with more extreme positions loudly voice their opinions.
Topic modeling is used to sift through large bodies of captured text to detect dominant themes or topics. Themes can be used to provide consistent labels to explore the text collection or build effective navigational interfaces. Themes can also be used to feed other analytical tasks (such as discovering user interests, detecting emerging topics in forums or social media postings, and summarizing parts, or all, of a text collection). Recent advances in topic modeling also allow these algorithms to be used with streaming data from Twitter and other continuous data feeds, making the technique an increasingly important analytic tool.
Topic modeling uses a variety of advanced statistics and machine-learning techniques; for instance, a number of models identify "latent" topics through the co-occurrence frequencies of words within a single communication14 or between topics and communities of users.27 Information about the position of words within messages can also be considered;26 see Blei4 for a survey of topic modeling.
Social network analysis is used on a social network graph to understand its underlying structure, connections, and theoretical properties, as well as to identify the relative importance of different nodes within the network. A social network graph consists of nodes (users) and associated relationships (edges). Relationships are typically detected through user actions connecting two people directly (such as accepting another user as a "friend"), though they may be inferred from indirect behaviors creating relationships (such as voting, tagging, and commenting).
Social network analysis is used to model social network dynamics and growth (using such features as network density and locations of new node attachments) that help monitor business activity. Social network analysis is the primary technique for identifying key influencers in viral marketing campaigns on Twitter and other social media platforms. It is also used to detect subcommunities within a larger online community (such as discussion forums), allowing greater precision in tailoring products and marketing materials. It is also useful in predictive modeling, as in marketing campaigns aimed at consumers assumed most likely to buy a particular product.5
Techniques used by social network analysis to understand the mathematical structure of graphs18 range from the simple (such as counting the number of edges a node has or computing path lengths) to the sophisticated algorithms that compute eigenvectors (as in Google's PageRank) to determine key nodes in a network. This helps determine whom, say, a business might look to on the basis of expertise and reputation. The analysis of network structure significantly predates the advent of social media, having been developed mainly for analyzing static mathematical graphs. Today's large and continually changing network structures demand new technical approaches, especially when real-time decision support is needed.
Trend analysis is used to predict future outcomes and behaviors based on historical data collected over time. Applications include forecasting the growth of customer or sales numbers, predicting the effectiveness of ad campaigns, staying ahead of shifts in consumer sentiment, and forecasting movement in, say, a stock market. Trend analysis is based on long-standing statistical methods (such as time-series analysis and regression analysis1) and other more recent modeling techniques (such as neural networks12 and support vector machines20).
Visual analytics is "the science of analytical reasoning facilitated by interactive visual interfaces."23 Spurred initially by U.S. defense needs, visualization works across different application areas to support synthesis, exploration, discovery, and confirmation of insight from data that is typically voluminous and spread among different sources. Visual analytics involves a range of activities, from data collection to data-supported decision making. Though many statistical methods underlie visual analytics (such as reducing high-dimensional data to fewer very salient dimensions), the human ability to perceive patterns and draw conclusions is a key factor as well. Indeed, when a torrent of information must be addressed quickly, combining machine and human strengths is critical, in both making decisions and being able to explain and justify them. Visual analytics shares a focus with other visualization techniques on creating economical, intuitive displays, but unlike the classical work of Tufte,24 these displays must support real-time decision making where the stakes can be high.
Visual analytic systems must be able to process data to reveal hidden structure and detail. Computational methods for data reduction, displaying correlations among disparate data sources, and letting users physically manipulate data displays all underlie visual analytics. Taking a more user-perceptual view, visual analytics can be understood as a collection of techniques that use graphical interfaces to present summarized, heterogeneous information that helps users visually inspect and understand the results of underlying computational processes. One commonly used interface design is the dashboard, where multiple metrics and key performance indicators are portrayed in a way that mimics a car's dashboard design (see Figure 2). Displays typically allow users to interactively interrogate the underlying data and perform data transformations using sliders or other types of controls. Crisis management and detecting breaking events from social media chatter can both benefit from visual analytics. The challenge for visual analytics is to remain responsive to, and create better visual representations for, increasingly massive and complex data requiring speedier interpretation and display on large numbers of devices, from handhelds to full-wall displays.
Here, we consider the techniques behind social media analytics in more detail, adopting a life-cycle analysis framework. Social media has changed our conversations about products and services but not about the business activities behind them. A life-cycle analysis considers the life of a product (or service) from design to disposal, as well as the supporting activities that take place in parallel with these activities. Although various authors describe the product life cycle with different levels of granularity, one that is quite typical suffices for our purpose, having four stages: design-development; production; utilization; and disposal.2 Social media is most relevant to the design-development and utilization stages; in addition, social media analytics helps businesses gather competitive intelligence and understand more completely their business environments, suppliers, and competitors. Our use of a life-cycle framework is consistent with other social media analyses.5
Product design-development. This first life-cycle stage covers the conceptual, preliminary, and detailed design of a product during which a variety of risks threaten success.3 Risks involving technology change may be due to misjudging the gaps in technology among different products or from time-to-market pressures. Risks involving design may be due to poor selection of product features, improper differentiation with other products, lack of modularity, or reliance on the wrong parts.
Trend analysis and other social media analytic tools help identify any changes in taste, behavior, and sentiment affecting product design and development. They let designers add and adjust features and help create sufficient lead time for creating next-generation products or even products in a completely new category. They also promote product innovation by capturing and understanding conversations involving either of two groups: loyal customers and average customers. On the one hand, a business's most loyal customers can reveal important insights, as Del Monte Foods, Inc. found in creating and launching a new dog-food product in just six weeks. On the other hand, conversations with "average" customers can also lead to product improvements; for instance, Dell Inc. created its IdeaStorm website in 2007 to solicit users' ideas about improving its computer products and services. Dell takes these suggestions seriously, soliciting comments from others as (dis)confirmation and making changes to its products as needed.
The software industry has taken the lead in social media-based product testing (leading to changes to software) by releasing various versions of its products and soliciting reactions and, in the case of open source programs, allowing user changes. Other industries have followed suit. The most advanced use of social media-based conversations is in the "co-creation" of products, where online users and businesses function as informal partners in generating new product ideas and even entirely new product categories.16
Product production. During product production, social media analytics can mitigate risk involving supply-chain responsiveness.3 Being tuned in to changing customer tastes and behavior, businesses can anticipate significant changes in demand and adjust accordingly by ramping production up or down. Visual analytics can be useful in identifying correlations, outliers, geographic patterns, and other trends that support smoother functioning. A business can also use social media analytics to learn another business with which it competes (or perhaps does not) is having trouble with a supplier, information useful in anticipating and avoiding the same problem. Close monitoring of social media can even help in technical-administrative tasks. For instance, inventory management is based on forecasts and production schedules. Social media can give advance warning when situations that might affect the acquisition of resources become less predictable, including political tensions overseas that could disrupt the flow of metals, minerals, and other vital supplies for manufacturing.
Over 50% of all online users expect a response to a complaint the same day they send it, though less than one-third receive one.
Product utilization. The most common use of social media analytics is in the product utilization life-cycle stage and involves three key social media objectives: brand awareness, brand engagement, and word of mouth.13 Brand awareness introduces customers to a brand (or product) or increases customers' familiarity with the brand. Brand engagement increases customers' connections with a brand. Word of mouth encourages users' efforts to positively influence other users' purchasing behavior.
A number of metrics have been proposed for assessing social media effectiveness in this stage;13 for example, for microblogging platforms like Twitter, simple ones include number of tweets and followers (for brand awareness), number of followers and replies (for brand engagement), and number of retweets (for word of mouth). Although they provide important information, they are no substitute for more powerful techniques in social media. For instance, influencer profiling uses social media to develop a deep understanding of users' backgrounds, tastes, and buying behavior to create better customer segmentation. Segmentation assists businesses in reaching various groups, using the differences to guide different strategies for increasing brand awareness and engagement for each group. Influencer profiling also assists in identifying social-community leaders or experts whose opinions are valuable in product development and consumer-supported customer service. Techniques for influencer profiling include social network analysis, topic modeling, and visual analytics.
Brand engagement suggests consumers feel a personal connection to a brand. Psychometric constructs suggesting brand engagement include the terms "special bond," "identify with," and "part of myself."19 Trying to create such relationships, businesses create a variety of activities, including "liking" and "commenting" on product websites. Other activities aim to generate a deeper sense of connection, often by enticing playful user actions. For instance, the German car manufacturer Audi AG was the first to use a hashtag in its 2011 Super Bowl ad, showing partying, good-looking vampires, and concluding with #SoLongVampires. This memorable hashtag could be tweeted during the most-watched sporting event of the season. More important, Audi then used social media tools to follow users who tweeted the hashtag to initiate a real-time dialogue that was one step among many in cultivating relationships with potential new customers. To the benefit of Audi and its brand, by the end of the Super Bowl, the hashtag had become a trending topic on Twitter.
More broadly, social media analytics allows businesses to judge online reactions to ad campaigns. The resulting metrics help link a campaign to subsequent sales, and thus the success of the campaign. Customers' reactions can also help alter the campaign in accordance with their likes and dislikes.
Word of mouth extends consumers' engagement beyond interactions with products to other consumers. Businesses hope these interactions, through, say, retweets, reblogs, and social tagging, are positive, though it does not always mean they are.
Customers' online complaints about products and services are common, with, for instance, nearly two-thirds of all customers worldwide using social media for this purpose. Over 50% of all online users expect a response to a complaint the same day they send it, though less than one-third receive one. Most top-50 brands worldwide never respond to customer comments on their own websites, hurting brand image and reputations.17 The viral spread of user complaints through social media can hurt business significantly.
Unlike gathering business intelligence from other sources, obtaining information from social media about suppliers or competitors (in all they do) is almost as easy for a business as monitoring its own affairs.
Real-time sentiment analysis, topic modeling, and other tools allow businesses to know how their customers feel about their products and services and respond quickly, before customer complaints become an online torrent. Trends in Twitter and other informal data generated during the cholera outbreak in Haiti following the 2010 earthquake were significantly correlated with official data used by the Haitian government and the World Health Organization to respond to the epidemic but were available two weeks earlier.8 Similarly, social media data provides early warnings that, ignored, can create impressions of a business that are difficult to overcome.
A 2010 study of 20 brand marketers worldwide showed the top 1% of a website's audience shares up to one-fifth of all links to the site and influences up to one-third of the actions taken by other users.22 Social network analysis can be used to determine who these key users are so they stay satisfied, engaged, and help a business market its products on its own website and via word of mouth through these users' social networks.
Product disposal. Nearing the end of a product's life span, consumers may have to decide how they will dispose of and replace it. For many consumers, being able to responsibly (ecologically) dispose of a product (such as a computer) may influence their overall impression of a company and its products. Creating convenient, responsible disposal and ensuring consumers are aware of it is important to the company, as well as to the environment. Social media analytics can be used to track consumer concerns, and companies themselves can engage in online conversations about disposal. Savvy companies that track these conversations can also infer that disposal may be accompanied by the purchase of a replacement item and leverage that knowledge in their marketing.
Competitive intelligence. Our discussion of the business value of social media analytics through the life-cycle framework has so far focused on products, services, and customers. But social media analytics also provides "competitive intelligence" by helping businesses understand their environments, suppliers, competitors, and overall business trends. Unlike gathering business intelligence from other sources, obtaining information from social media about suppliers or competitors (in all they do) is almost as easy for a business as monitoring its own affairs.
Social media analytics can also play a key role in helping identify and respond to crises. Ironically, businesses often cause crises through their own efforts to disseminate messages through social media. Large organizations (such as the American Red Cross, Burger King, and Chapstick) have been implicated in social media messages that were ill-received, even when, as in the case of the Red Cross in 2011, disseminated unintentionally. The Red Cross and Burger King quickly acknowledged their mistakes and took action, deftly, in the case of the Red Cross, defusing a potential crisis, first by responding with humor, then, after identifying the uncommon hashtag in the inadvertent message sent from its account, using it to generate a successful blood-donation campaign. Unlike the Red Cross and Burger King, Chapstick at first failed to respond to customers' complaints at all, then removed them from its website without responding. These actions exacerbated the bad publicity it generated from its online presence.
Even as some social media websites explode into use, quickly becoming everyday tools (Facebook launched in 2004, Twitter in 2006), new platforms are constantly emerging; for example, Pinterest launched in Summer 2011 and, as of June 2013, had approximately 50 million users worldwide. All told, a dozen sites, also as of June 2013, had at least 100,000 registered users and many more unique visitors, according to Wikipedia, including sites (such as Ozone and Sina Weibo) most of the online population has never heard of.
Even as businesses begin to recognize the business risk of ignoring social media content and conversely its inherent opportunity, their questions suggest how much remains unknown; for example, a 2012 survey of 3,800 marketers in the U.S. indicated three top concerns:21
- How to track social media return on investment;
- How to identify and engage with the most influential social media users; and
- What tactics to use to create an effective social media strategy.
Social media analytical tools are designed to address each of them. At the same time, social media transform the very nature of business. Current patterns suggest social media could produce an additional $940 billion in annual consumption, especially in electronics, hardware, software, and mobile technologies.7 Social media support "co-creation" of products, with consumers working online with companies' product designers.16
Technical challenges loom as well; for example, the extraordinary volume of big data already challenges social media analytics,6 and languages add further complications, as businesses monitor and analyze social media conversations around the world. These challenges could swell, as social media analytics begin to incorporate user-based location data facilitated by mobile technology, and consumer pressure increases to process and respond to social messages in real time.
We thank Jonathan Woody of Virginia Tech for his data-collection support related to various parts of this article. We also thank the anonymous reviewers for their valuable comments in helping us improve the article.
7. Chui, M., Manyika, J., Bughin, J., Dobbs, R., Roxburgh, C., Sarrazin, H., Sands, G., Westergren, M. The Social Economy: Unlocking Value and Productivity Through Social Technologies. Research Report. McKinsey Global Institute, 2012; http://www.mckinsey.com/insights/mgi/research/technology_and_innovation/the_social_economy
8. Chunara, R., Andrews, J.R., and Brownstein, J.S. Social and news media enable estimation of epidemiological patterns early in the 2010 Haitian cholera outbreak. The American Journal of Tropical Medicine and Hygiene 86, 1 (Jan. 2012), 3945.
14. Hofmann, T. Probabilistic latent semantic indexing. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (Berkeley, CA). ACM Press, New York, 1999, 5057.
17. Rollason, H. Why social media makes customer service better. Mashable (Sept. 29, 2012); http://mashable.com/2012/09/29/social-media-better-customer-service/
19. Sprott, D., Czellar, S., and Spangenberg, E. The importance of a general measure of brand engagement on market behavior: Development and validation of a scale. Journal of Marketing Research 46, 1 (Feb. 2009), 92104.
21. Stelzner, M.A. Social Media Marketing Industry Report. 2012; http://www.socialmediaexaminer.com/socialmedia-marketing-industry-report-2012/
22. Straley, B. How to: Target social media influencers to boost traffic and sales. Mashable (Apr. 15, 2010); http://mashable.com/2010/04/15/social-media-influencers/
25. Turney, P. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the Association for Computational Linguistics (Philadelphia). ACL, Stroudsburg, PA, 2002, 417424.
27. Yin, Z., Cao, L., Gu, Q., and Han, J. Latent community topic analysis: Integration of community discovery with topic modeling. ACM Transactions on Intelligent Systems and Technology 3, 4 (Sept. 2012), 121.
a. Throughout this article, we cite statistics from a number of websites that closely track these issues, including http://www.adweek.com, http://www.alexa.com, http://www.internetworldstats.com, and http://www.comscore.com, as well as from social media sites themselves.
b. We adopt the view of Pang and Lee15 who described the terms "opinion mining" and "sentiment analysis" as having multiple definitions, using them broadly and interchangeably to cover the subjective, textual evaluation of source materials or their features.
©2014 ACM 0001-0782/14/06
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.