posted by michelle — February 23, 2016
Strong case studies of digital humanities scholarship exist, and compelling resources continue to proliferate. Scholarly societies, including the American Historical Association (AHA), Modern Language Association (MLA), and more recently the College Art Association (CAA), and Society of Architectural Historians (SAH) have developed guidelines for use by institutions and individuals. Funded by a grant from the Andrew W. Mellon Foundation, CAA and SAH formed a joint task force that explored existing guidelines and literature, surveyed faculty and administrators, and developed Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History. Adopted by the boards of both societies, the guidelines are available for download on the SAH and CAA websites and we encourage their adoption and use. The task force reviewed many resources in the course of eighteen months. Some of these are described below to provide a context for current digital evaluation in addition to the recently published CAA/SAH guidelines.
Sources such as the Journal of Interactive Pedagogy and Pedagogy and Manifold Scholarship, and sites such as Mapping Gothic France, HyperCities, and ArtHistoryTeachingResources.org are places where digital scholarship is being practiced, peer reviewed, and successfully included in promotion and tenure portfolios. Benchmark portals such as the NINES (Networked Infrastructure for Nineteenth-Century Electronic Scholarship) NEH Summer Institutes (Storify of the proceedings can be found here) offer useful white papers: “Digital Humanities Scholarship: Recommendations for Chairs in Language and Literature Departments“; “Guidelines for Promotion and Tenure Committees in Judging Digital Work”; and “Statement on Authorship.” Kristine M. Bartanen’s excellent “Digital Scholarship and the Tenure and Promotion Process“ contains valuable guidance. Published to Academic Commons in July 2014, Bartenan provides a comprehensive literature review and resource portal for the evaluation of digital scholarship. The review contains both model guidelines for promotion and tenure and appendices for a range of resources, literature, and organizations pertinent to understanding and developing evaluation methods.
In the fall of 2012 the Journal of Digital Humanities published their fourth issue, which centered on the “importance of assessment and the scholarly vetting process around digital scholarship.” (The full journal issue can be accessed here.) The issue addresses the need for guidelines and dialogue for assessing digital scholarship in the humanities, offers case studies and examples, and reflects on the successes, issues, and failures of the methods employed. Though much of the content is not art or architectural history-specific, it is still relevant to CAA’s and SAH’s memberships. A broad overview of the contents of this journal issue is provided below so that readers can zero in on the material most germane to their own needs. Readers are encouraged to make use of the excellent bibliographies attached to each of the issue’s essays.
The issue is divided into four major sections: “The Problem Stated,” “Approaches,” “Institutional Guidelines,” and “Resources.” In the introductory section of “Living in a Digital World: Rethinking Peer Review, Collaboration, and Open Access,” Sheila Cavanagh highlights the importance of retraining processes and expectations of evaluation around digital scholarship, including “encouraging faculty who hire, tenure, and mentor junior scholars to acknowledge the complicated factors in the world of digital scholarship . . . for example, faculty often have difficulty identifying appropriate experts to participate in more traditional peer review processes.” In “Evaluating Collaborative Digital Scholarship (or, Where Credit is Due),” Bethany Nowviskie highlights the inability of many humanities evaluative committees to adequately understand and appropriately review and reward the type of collaborative work that digital scholarship is often predicated upon. She contends that “the T&P [tenure and promotion] process is a poor fit to good assessment (or even, really, to acknowledgement) of collaborative work, because it has evolved to focus too much on a particular fiction. That fiction is one of ‘final outputs’ in digital scholarship.”
The “Student Collaborators’ Bill of Rights“ from UCLA’s Digital Humanities program provides important guidance about how to understand, review, and reward collaborative digital humanities scholarship, especially in light of hierarchies of power experienced by student and “alt-ac” collaborators. Nowviskie hails certain aspects of the AHA Guidelines for Evaluation of Digital Scholarship, particularly their underscoring of collaborative digital work as a type of perpetual peer review. At a National Endowment for the Humanities “Off the Tracks” workshop in 2011, Nowviskie, Matt Kirschenbaum, Doug Reside, and Tom Scheinfeldt collaborated on a provocative “Bill of Rights” for collaborators that might serve as a guidepost for promotion and tenure committees to consider. In Nowviskie’s words, “We drew on our experience administering MITH [Maryland Institute for Technology in the Humanities], the Scholars’ Lab, and the Roy Rosenzweig Center for History and New Media—three centers that are sites for a great deal of collaboration among people who may have similar backgrounds as scholars and technologists, but whose formal institutional status may vary a great deal. We drafted something we called a ‘Collaborators’ Bill of Rights,’ which was later endorsed by the full workshop assembly and posted for public comment. Basically, it’s an appeal for fair, honest, legible, portable (this is important!), and prominently displayed crediting mechanisms. It also offers a dense expression of underlying requirements for healthy collaboration and adequate assessment from the point of view of practicing digital humanists, with special attention to the vulnerabilities of early-career scholars and staff or non-tenure-track faculty.” Their recommendations are as follows:
- Committees must consider not only the products of digital work but the processes by which the work was (and perhaps continues to be) co-created;
- Scholars (even while they ask to have their critical agency as individuals taken seriously in tenure and promotion cases) are obligated to make the most generous and inclusive statements possible about the contributions of others;
- Credit should be expressed richly and descriptively, but also in increasingly standardized forms, legible within a variety of disciplines and communities of practice;
- We must negotiate expressions of shared credit at the outset of projects and continually, as projects evolve;
- We must promote fair institutional policies and practices in support of shared assertion of credit, such as those which make collective and individual ownership over intellectual property meaningful and actionable;
- And, finally, we must accept that collaborators themselves, regardless of rank or status, have the ultimate authority and responsibility for expressing their contributions and the nature of their roles.
Other relevant literature includes Todd Presner’s essay “How to Evaluate Digital Scholarship,” directed at the audiences within the art and architectural history-specific community: “The document is aimed, foremost, at Academic Review Committees, Chairs, Deans, and Provosts who want to know how to assess and evaluate digital scholarship in the hiring, tenure, and promotion process. Secondarily, the document is intended to inform the development of university-wide policies for supporting and evaluating such scholarship.” Presner offers a succinct and clear checklist for promotion and tenure boards, engaging them in dialogue around initial review, crediting, intellectual rigor, impact, peer review, and experimentation, among other topics.
Geoffrey Rockwell’s “Short Guide to Evaluation of Digital Work” deepens and extends Presner’s checklist in compelling and practicable ways, and Laura Mandell’s essay “Promotion and Tenure for Digital Scholarship” offers a useful and clear coda full of concrete examples such as the HyperCities project and Voyant, a software text analysis tool. It is not overstating the importance of these three essays alone to suggest that every promotion and tenure board member might be asked to read them to educate themselves on the shifting boundaries of their discipline, and the ways in which their analysis of promotion and tenure portfolios must change and adapt. In addition, in the third section of the journal issue, the MLA’s Guidelines for Evaluating Work in Digital Humanities and Digital Media is similarly designed to “help departments and faculty members implement effective evaluation procedures for hiring, reappointment, tenure, and promotion. They apply to scholars working with digital media as their subject matter and to those who use digital methods or whose work takes digital form.”
James Smithies’s essay “Evaluating Scholarly Digital Outputs: The Six Layers Approach” offers a different style of checklist, a bird’s-eye view of the different types of digital scholarship output, but one that is most useful if the reader—part of a promotion and tenure board or otherwise—is already informed about and open to considering digital scholarship on par with analog modes. Katherine D. Harris’s “Explaining Digital Humanities in Promotion Documents” is a comprehensive and fruitful overview of her foray into learning “how best to sell Digital Humanities, scholarly editing, and Digital Pedagogy to my colleagues.” Harris, associate professor of English literature at San Jose State University, shares her statement for promotion and her approach to the practical question of formatting and outlining her digital scholarship as part of her promotion and tenure binder. With few models to consult, she crowdsourced her questions via Twitter and other channels in order to produce her promotion and tenure document, finally settling on a narrative-based approach that outlines her own collaborative and individual contributions to the field, and includes examples of her concrete research and pedagogy projects in progress and completed, publications, conferences, and supporting quotations from the Digital Humanities Community Wiki and Digital Humanities Now.
While every essay in the fall 2012 Journal of Digital Humanities issue is well worth consultation, the final resource to flag is the MLA’s “Evaluation Wiki of the Committee on Information Technology.” This report takes the form of a checklist of documents that digital scholars should amass as part of preparation for promotion and tenure, and it might be used in tandem with Harris’s model statement.
Both CAA and SAH leadership teams encourage the wide dissemination and adoption of the newly published guidelines, as well as a CAA 2017 panel or forum models that facilitate questions, feedback, and discussion around the issues raised.
Michelle Millar Fisher is a doctoral candidate in Architectural History at the CUNY Graduate Center and a Curatorial Assistant in the Department of Architecture and Design at The Museum of Modern Art.
 Daniel J. Cohen and Joan Fragaszy Troyano, “Closing the Evaluation Gap,” Journal of Digital Humanities vol. 1, no. 4 (Fall 2012): i. They continued: “As digital humanities continues to grow and as more scholars and disciplines become invested in its methods and results, institutions and scholars increasingly have been debating how to maintain academic rigor while accepting new genres and the openness that the web promotes.”
 Ibid, 9.
 Ibid, 17.
 Authored by Haley Di Pressi, Stephanie Gorman, Miriam Posner, Raphael Sasayama, and Tori Schmitt, with contributions from Roderic Crooks, Megan Driscoll, Amy Earhart, Spencer Keralis, Tiffany Naiman, and Todd Presner, this document endorses the Collaborators Bill of Rights, developed during the workshop “Off the Tracks—Laying New Lines for Digital Humanities Scholars” held at the Maryland Institute for Technology in the Humanities (MITH), January 20–21, 2011, and described below. The UCLA document adds principles to safeguard students when working on projects with scholars who are senior to them.
 Ibid, 36.
 Ibid, 91–94.
 Ibid, 71.
CAA and SAH Release Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History
posted by Christopher Howard — February 23, 2016
The College Art Association (CAA), working jointly with the Society of Architectural Historians (SAH), has released its Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History for Promotion and Tenure. The guidelines are the result of a Task Force convened by the two associations of ten members from the academic community with experience in digital research, publication, and/or scholarly communications and administration. Established by the Board of Directors in October 2014 and supported by funds from the Andrew W. Mellon Foundation, this joint task force of CAA and SAH was co-chaired by DeWitt Godfrey, president, CAA and professor of art and art history, Colgate University; and Kenneth Breisch, SAH president and assistant professor, School of Architecture, University of Southern California.
See the SAH announcement.
The project methodology was developed through a collaborative process involving the Task Force, consultant, researcher, and representatives from CAA and SAH. Research components included results from surveys of CAA and SAH members and department chairs, a similar questionnaire for graduate students, interviews with scholars and administrators, research on existing disciplinary and institutional evaluation guidelines, and a review of relevant literature. Alice Lynn McMichael, PhD candidate in Byzantine art history and digital fellow at the Graduate Center at the City University of New York, and Raym Crow, principal at Chain Bridge Group served as researcher and consultant.
In addition to a review of current guidelines and current literature on the topic, the researcher was charged with balancing broad survey data with more in-depth discussions of topics that resonate with evaluators of digital work. To this end, interviews were conducted with constituents from sixteen institutions across the United States that produce digital work regularly in art and/or architectural history. Institutions were either public and private and research-intensive or teaching-focused. Interviewees at those institutions had varying academic ranks and included department members, librarians, provosts, deans, chairs, and directors of humanities centers or similar.
Highlights from surveys from CAA and SAH members and interviews include the following:
- Most CAA respondents, across all professional ranks, have never used data gathering and imaging tools (83%), data analysis and visualization tools (80%), three-dimensional modeling (75%), digital storage and preservation tools (73%), and geospatial analysis tools (65%).
- Many characteristics of digital resources were generally valued as important, with the highest importance placed on permanent archiving, documentation of the resource, and ease of use; and relatively less importance on the availability of underlying data, financial sustainability, and permanent citation.
- Graduate students report the highest confidence in evaluating digital scholarship.
- Less than 4 percent of total respondents knew of existing evaluative criteria for digital scholarship in their departments.
- The largest barrier to digital scholarship is lack of access to training.
Read the full Guidelines here. An essay, “Case Studies and Examples for Evaluating Digital Scholarship” by Michelle Millar Fisher, a task force member, provides background information on evaluating digital resources and can be found on the College Art Association website.
Members of the Task Force included:
- Suzanne Preston Blier, Allen Whitehall Clowes Professor of Fine Arts and of African and African American Studies, Harvard University
- Kenneth Breisch, SAH President, Assistant Professor, School of Architecture, University of Southern California, and Cochair of Task Force
- Linda Downs (ex officio), Executive Director, College Art Association
- Gabrielle Esperdy, Associate Professor, School of Architecture, New Jersey Institute of Technology, and Editor of SAH Archipedia
- Michelle Miller Fisher, PhD Candidate, Art History, The Graduate Center, The City University of New York, and Curatorial Assistant, Department of Architecture and Design, Museum of Modern Art
- Pamela M. Fletcher, Professor of Art History, Chair of Art Department, and Codirector of Digital and Computational Studies Initiative, Bowdoin College
- DeWitt Godfrey, CAA President, Professor of Art and Art History, Colgate University, and Cochair of Task Force
- Anne Collins Goodyear, Codirector, Bowdoin College Museum of Art
- Paul Jaskot, Professor, History of Art and Architecture, DePaul University, and Andrew W. Mellon Professor, CASVA
- Bruce M. Mackh, Director, Arts and Cultural Management Program, Michigan State University
- Tara McPherson, Associate Professor, School of Cinematic Arts, University of Southern California
- Abby Smith Rumsey, Former Director, Scholarly Communication Institute, University of Virginia
- Pauline Saliga (ex officio), Executive Director, Society of Architectural Historians
- Ann Whiteside, Librarian and Assistant Dean of Information Services, Harvard University
The College Art Association is dedicated to providing professional services and resources for artists, art historians, and students in the visual arts. CAA serves as an advocate and a resource for individuals and institutions nationally and internationally by offering forums to discuss the latest developments in the visual arts and art history through its Annual Conference, publications, exhibitions, website, and other programs, services, and events. CAA focuses on a wide range of issues, including education in the arts, freedom of expression, intellectual-property rights, cultural heritage, preservation, workforce topics in universities and museums, and access to networked information technologies. Representing its members’ professional needs since 1911, CAA is committed to the highest professional and ethical standards of scholarship, creativity, criticism, and teaching. Learn more at collegeart.org.
Founded in 1940, the Society of Architectural Historians is a nonprofit membership organization that promotes the study, interpretation and conservation of architecture, design, landscapes and urbanism worldwide. SAH serves a network of local, national and international institutions and individuals who, by vocation or avocation, focus on the built environment and its role in shaping contemporary life. SAH promotes meaningful public engagement with the history of the built environment through advocacy efforts, print and online publications, and local, national and international programs. Learn more at sah.org.