How to evaluate users and contributions? CS systemsoften must manage malicious users. To do so, we can usea combination of techniques that block, detect, and deter.First, we can block many malicious users by limiting whocan make what kinds of contributions. Many e-science CSsystems, for example, allow anyone to submit data, butonly certain domain scientists to clean and merge thisdata into the central database.Second, we can detect malicious users and contributionsusing a variety of techniques. Manual techniques includemonitoring the system by the owners, distributing themonitoring workload among a set of trusted users, andenlisting ordinary users (such as flagging bad contributionson message boards). Automatic methods typicallyinvolve some tests. For example, a system can ask usersquestions for which it already knows the answers, thenuse the answers of the users to compute their reliabilityscores. 13,34 Many other schemes to compute users’ reliability/trust/fame/reputationhave been proposed. 9,26Finally, we can deter malicious users with threats of“punishment.” A common punishment is banning. Anewer, more controversial form of punishment is “publicshaming,” where a user U judged malicious is publiclybranded as a malicious or “crazy” user for the rest of thecommunity (possibly without U’s knowledge). For example,a chat room may allow users to rate other users. If the(hidden) score of a user U goes below a threshold, otherusers will only see a mechanically garbled version of U’scomments, whereas U continues to see his or her commentsexactly as written.No matter how well we manage malicious users, maliciouscontributions often still seep into the system. If so,the CS system must find a way to undo those. If the systemdoes not combine contributions (such as reviews) ordoes so only in a loose fashion (such as ratings), undoingis relatively easy. If the system combines contributionstightly, but keeps them localized, then we can still undowith relatively simple logging. For example, user edits inWikipedia can be combined extensively within a singlepage, but kept localized to that page (not propagated toother pages). Consequently, we can undo with page-levellogging, as Wikipedia does. However, if the contributionsare pushed deep into the system, then undoing can bevery difficult. For example, suppose an inference rule R iscontributed to a KB on Day 1. We then use R to infermany facts, apply other rules to these facts and other factsin the KB to infer more facts, let users edit the facts extensively,and so on. Then on Day 3, should R be found incorrect,it would be very difficult to remove R withoutreverting the KB to its state on Day 1, thereby losing allgood contributions made between Day 1 and Day 3.At the other end of the user spectrum, many CS systemsalso identify and leverage influential users, using bothmanual and automatic techniques. For example, productiveusers in Wikipedia can be recommended by otherusers, promoted, and given more responsibilities. As anotherexample, certain users of social networks highly influencebuy/sell decisions of other users. Consequently,some work has examined how to automatically identifythese users, and leverage them in viral marketing within auser community. 24CONCLUSIONWe have discussed CS systems on the World-Wide Web.Our discussion shows that crowdsourcing can be appliedto a wide variety of problems, and that it raises numerousinteresting technical and social challenges. Given the successof current CS systems, we expect that this emergingfield will grow rapidly. In the near future, we foreseethree major directions: more generic platforms, more applicationsand structure, and more users and complexcontributions.First, the various systems built in the past decade haveclearly demonstrated the value of crowdsourcing. Therace is now on to move beyond building individual systems,toward building general CS platforms that can beused to develop such systems quickly.Second, we expect that crowdsourcing will be applied toever more classes of applications. Many of these applicationswill be formal and structured in some sense, makingit easier to employ automatic techniques and to coordinatethem with human users. 37–40 In particular, a large chunkof the Web is about data and services. Consequently, weexpect crowdsourcing to build structured databases andstructured services (Web services with formalized inputand output) will receive increasing attention.Finally, we expect many techniques will be developed toengage an ever broader range of users in crowdsourcings,and to enable them, especially naïve users, to make increasinglycomplex contributions, such as creating softwareprograms and building mashups (without writingany code), and specifying complex structured data pieces(without knowing any structured query languages).References1. AAAI-08 Workshop. Wikipedia and artificial intelligence: Anevolving synergy, 2008.2. Adamic, L.A., Zhang, J., Bakshy, E. and Ackerman, M.S. Knowledgesharing and Yahoo answers: Everyone knows something. InProceedings of WWW, 2008.40
3. Chai, X., Vuong, B., Doan, A. and Naughton, J.F. Efficiently incorporatinguser feedback into information extraction and integrationprograms. In Proceedings of SIGMOD, 2009.4. The Cimple/DBLife project;http://pages.cs.wisc.edu/~anhai/projects/cimple.5. DeRose, P., Chai, X., Gao, B.J., Shen, W., Doan, A., Bohannon, P.and Zhu, X. Building community Wikipedias: A machine-humanpartnership approach. In Proceedings of ICDE, 2008.6. Fuxman, A., Tsaparas, P., Achan, K. and Agrawal, R. Using thewisdom of the crowds for keyword generation. In Proceedings ofWWW, 2008.7. Golbeck, J. Computing and applying trust in Web-based socialnetwork, 2005. Ph.D. Dissertation, University of Maryland.8. Ives, Z.G., Khandelwal, N., Kapur, A., and Cakir, M. Orchestra:Rapid, collaborative sharing of dynamic data. In Proceedings ofCIDR, 2005.9. Kasneci, G., Ramanath, M., Suchanek, M. and Weiku, G. Theyago-naga approach to knowledge discovery. SIGMOD Record 37,4, (2008), 41–47.10. Koutrika, G., Bercovitz, B., Kaliszan, F., Liou, H. and Garcia-Molina, H. Courserank: A closed-community social system throughthe magnifying glass. In The 3rd Int’l AAAI Conference on Weblogsand Social Media (ICWSM), 2009.11. Little, G., Chilton, L.B., Miller, R.C. and Goldman, M. Turkit: Toolsfor iterative tasks on mechanical turk, 2009. Technical Report.Available from glittle.org.12. McCann, R., Doan, A., Varadarajan, V., and Kramnik, A. Buildingdata integration systems: A mass collaboration approach. InWebDB, 2003.13. McCann, R., Shen, W. and Doan, A. Matching schemas in onlinecommunities: A Web 2.0 approach. In Proceedings of ICDE, 2008.14. McDowell, L., Etzioni, O., Gribble, S.D., Halevy, A.Y., Levy, H.M.,Pentney, W., Verma, D. and Vlasseva, S. Mangrove: Enticing ordinarypeople onto the semantic web via instant gratification. In Proceedingsof ISWC, 2003.15. Mihalcea, R. and Chklovski, T. Building sense tagged corporawith volunteer contributions over the Web. In Proceedings ofRANLP, 2003.16. Noy, N.F., Chugh, A. and Alani, H. The CKC challenge: Exploringtools for collaborative knowledge construction. IEEE Intelligent Systems23, 1, (2008) 64–68.17. Noy, N.F., Griffith, N. and Munsen, M.A. Collecting communitybasedmappings in an ontology repository. In Proceedings of ISWC,2008.18. Olson, M. The amateur search. SIGMOD Record 37, 2 (2008),21–24.19. Perkowitz, M. and Etzioni, O. Adaptive web sites. Comm. ACM43, 8 (Aug. 2000).20. Rahm, E. and Bernstein, P.A. A survey of approaches to automaticschema matching. VLDB J. 10, 4, (2001), 334–350.21. Ramakrishnan, R. Collaboration and data mining, 2001. Keynotetalk, KDD.22. Ramakrishnan, R., Baptist, A., Ercegovac, A., Hanselman, M.,Kabra, N., Marathe, A. and Shaft, U. Mass collaboration: A casestudy. In Proceedings of IDEAS, 2004.24. Richardson, M. and Domingos, P. Mining knowledgesharingsites for viral marketing. In Proceedings of KDD, 2002.25. Richardson, M. and Domingos, P. Building large knowledgebases by mass collaboration. In Proceedings of K-CAP, 2003.26. Sarwar, B.M., Karypis, G., Konstan, J.A. and Riedl, J. Item-basedcollaborative filtering recommendation algorithms. In Proceedingsof WWW, 2001.27. Steinmetz, R. and Wehrle, K. eds. Peer-to-peer systems and applications.Lecture Notes in Computer Science. 3485; Springer, 2005.28. Stork, D.G. Using open data collection for intelligent software.IEEE Computer 33, 10, (2000), 104–106.29. Surowiecki, J. The Wisdom of Crowds. Anchor Books, 2005.30. Tapscott, D. and Williams, A.D. Wikinomics. Portfolio, 2006.31. Time. Special Issue Person of the year: You, 2006; http://www.time.com/time/magazinearticle/0,9171,1569514,00.html.32. von Ahn, L. and Dabbish, L. Labeling images with a computergame. In Proc. of CHI, 2004.33. von Ahn, L. and Dabbish, L. Designing games with a purpose.Comm. ACM 51, 8 (Aug. 2008), 58–67.34. von Ahn, L., Maurer, B., McMillen, C., Abraham, D. and Blum, M.Recaptcha: Human-based character recognition via Web securitymeasures. Science 321, 5895, (2008), 1465–1468.35. Weld, D.S., Wu, F., Adar, E., Amershi, S., Fogarty, J., Hoffmann,R., Patel, K. and Skinner, M. Intelligence in Wikipedia. AAAI, 2008.36. Workshop on collaborative construction, management and linkingof structured knowledge (CK 2009), 2009.http://users.ecs.soton.ac.uk/gc3/iswc-workshop.37. Franklin, M, Kossman, D., Kraska, T, Ramesh, S, and Xin, R.CrowdDB: Answering queries with crowdsourcing. In Proceedingsof SIGMOD 2011.38. Marcus, A., Wu, E. and Madden, S. Crowdsourcing databases:Query processing with people. In Proceedings of CRDR 2011.39. Parameswaran, A., Sarma, A., Garcia-Molina, H., Polyzotis, N.and Widom, J. Human-assisted graph search: It’s okay to ask questions.In Proceedings of VLDB 2011.40. Parameswaran, A. and Polyzotis, N. Answering queries usinghumans, algorithms, and databases. In Proceedings of CIDR 2011.ABOUT THE AUTHORSAnHai Doan (anhai@cs.wisc.edu) is an associate professorof computer science at the University of Wisconsin-Madisonand Chief Scientist at Kosmix Corp.Raghu Ramakrishnan (ramakris@yahoo-inc.com) is ChiefScientist for Search & Cloud Computing, and a Fellow atYahoo! Research, Silicon Valley, CA, where he heads theCommunity Systems group.Alon Y. Halevy (halevy@google.com) heads the StructuredData Group at Google Research, Mountain View, CA.From: Anhai Doan, Raghu Ramarkrishnan, & Alon Y. Halevy,“Crowdsourcing: What it Means for Innovation,” Communications ofthe ACM 54, no. 4 (2011): 86-96. Used with permission.23. Rheingold, H. Smart Mobs. Perseus Publishing, 2003.41
- Page 1 and 2: VOLUME FOUR STRATEGIC PERSPECTIVESL
- Page 3 and 4: VOLUME FOUR STRATEGIC PERSPECTIVESL
- Page 5 and 6: VOLUME FOUR STRATEGIC PERSPECTIVESL
- Page 7: VOLUME FOUR STRATEGIC PERSPECTIVESL
- Page 10 and 11: 12CHAPTER 12INTRODUCTION TO STRATEG
- Page 12 and 13: 12.1 Strategic Leadership: Defining
- Page 14 and 15: mandates or resolutions that would
- Page 16 and 17: and ambiguity, aspiring strategic l
- Page 18 and 19: 12.2 National Security StrategyThe
- Page 20 and 21: within our borders has always been
- Page 22 and 23: front common challenges like violen
- Page 24 and 25: and our strategy, not sector earmar
- Page 26 and 27: thinking about organizations. She m
- Page 28 and 29: A systemic approach to failure is m
- Page 30 and 31: The late W. T. Grant Company is a r
- Page 32 and 33: the resources - setting the directi
- Page 34 and 35: focal point for describing and inte
- Page 36 and 37: Consequently, we do not restrict th
- Page 38 and 39: paragraphs, Web pages, then edit an
- Page 40 and 41: Web, can be viewed as a CS attempt
- Page 44 and 45: 13CHAPTER 13LEADING PUBLIC &VOLUNTE
- Page 46 and 47: 13.1 Leadership for Volunteers:The
- Page 48 and 49: 13.2 Take Root: Volunteer Managemen
- Page 50 and 51: QualificationsClearly list educatio
- Page 52 and 53: and effectively track their volunte
- Page 54 and 55: • Understand rules for recognitio
- Page 56 and 57: • Send a birthday card.• Submit
- Page 58 and 59: and tested more than six decades af
- Page 60 and 61: specific interests of the donors, v
- Page 62 and 63: tain) tax-exempt status from the In
- Page 64 and 65: Smucker, 1999).The Internal Revenue
- Page 66 and 67: culture is necessary to ensure the
- Page 68 and 69: 13.4 The New Look of TransparencyBy
- Page 70 and 71: ees for a couple of reasons: One, i
- Page 72 and 73: 13.5 Public and Private Management:
- Page 74 and 75: TABLE 1:FUNCTIONS OF GENERAL MANAGE
- Page 76 and 77: 3. Career System. The model corpora
- Page 78 and 79: islative charter - the Clean Air Ac
- Page 80 and 81: In controlling performance, Chapin
- Page 82 and 83: 14CHAPTER 14AIRPOWER ASSTRATEGIC LA
- Page 84 and 85: 14.1 Strategic Air Power: Fulfillme
- Page 86 and 87: carry it out. Their daylight raids
- Page 88 and 89: you did not rely on strategic bombi
- Page 90 and 91: 14.2 Warden and the Air Corps Tacti
- Page 92 and 93:
ecomes one of applying sufficient i
- Page 94 and 95:
tification, and a Jominian claim to
- Page 96 and 97:
courage the rapid and widespread ex
- Page 98 and 99:
it to influence physical players in
- Page 100 and 101:
14.4 Basic Air Force DoctrineAF Doc
- Page 102 and 103:
earthquake-stricken Haiti. The worl
- Page 104 and 105:
perspective. Airmen do not divide u
- Page 106 and 107:
Command and ControlCommand and cont
- Page 108 and 109:
14.5 Should the US Maintain the Nuc
- Page 110 and 111:
form of human government.” 20 Dem
- Page 112 and 113:
obtainable goal. See the Global Zer
- Page 114 and 115:
15CHAPTER 15ORGANIZATIONAL CULTURE
- Page 116 and 117:
15.1 Organizational CultureBy Doria
- Page 118 and 119:
Review. This action strives to unco
- Page 120 and 121:
gram will serve and then having the
- Page 122 and 123:
ticipating the changes being made b
- Page 124 and 125:
Many years of working with change p
- Page 126 and 127:
At the least, the areas of concern
- Page 128 and 129:
15.4 Developing an Innovative Cultu
- Page 130 and 131:
CONCLUDING THOUGHTSIn an ever-chang
- Page 132 and 133:
global issues. Businesses that poss
- Page 134 and 135:
— Sees the big picture—the shif
- Page 136 and 137:
16CHAPTER 16STRATEGIC COMMUNICATION
- Page 138 and 139:
16.1 Principles of Strategic Commun
- Page 140 and 141:
16.2 The Art of NegotiationBy Brend
- Page 142 and 143:
16.3 Negotiating Effectively Across
- Page 144 and 145:
hidden areas can act as cultural ho
- Page 146 and 147:
and four conflict styles. Hammer be
- Page 148 and 149:
maintaining the relationship. As th
- Page 150 and 151:
25 Mitchell R. Hammer, “Chapter 1
- Page 152 and 153:
tural, socioeconomic, and psycholog
- Page 154 and 155:
for the win-win," during which time
- Page 156 and 157:
interests and, at worst, as a gun s
- Page 158 and 159:
Public diplomacy is surely about mu
- Page 160 and 161:
But public diplomats do not have th
- Page 162 and 163:
Photo courtesy of the familyThe LEA
- Page 164:
THE CADET OATHI pledge that I will