Political Science Now

Data Access and Research Transparency Initiative (DA-RT)

Over the past few months, APSA members have participated in an extensive discussion of the values, risks, and implementation strategies involved in the desire to enhance the transparency and interpretability of empirically-based scholarship.  The immediate impetus, and shorthand, for this discussion is the Data Access and Research Transparency (DA-RT) initiative.  We want first to acknowledge and thank our many colleagues who have engaged with this initiative; the broad and respectful involvement indicates the shared concerns of APSA members and signals how seriously political scientists take our discipline, association, and professional responsibilities.

We are writing to express our views and place them in context in order to contribute to this discussion. Under long-standing association practice, editors of our journals have wide discretion in editorial standards and procedures, including a decision on whether, when, and how to implement strategies for promoting access to evidence and research interpretability.  Editors of some section journals and the APSR editors have issued guidelines for implementation (the draft APSR guidelines are here, in the section on “Data Access, Production Transparency and Analytic Transparency).” Editors of other association-wide and section journals have decided not to adopt the proposed standards at present. This variation is consistent with current practice, and it will provide useful information that will help association members evaluate innovations, address implementation problems, and assess the value of research transparency.

In the following paragraphs, we provide some background for our support for DA-RT implementation in the APSR in January 2016, and describe plans for review and action over the next year by the APSA Council on DA-RT and related issues of scholarship and publication.

An Array of Issues:  It is no surprise that so many people have worked to develop, modify, contest, and query the implementation of strategies for transparency and interpretability. These strategies raise issues ranging from epistemological differences in understanding “truth” or “fact,” to protection of human subjects, professional advancement, workload, subfield development, privacy, editorial authority, and implementation details.

To some political scientists, implementing guidelines for research transparency is essential to ensure our legitimacy, internally and externally, as scholars who seek the best evidence and interpretations and who fully engage with other researchers and the public.  To other political scientists, who share the commitment to the highest standards of research integrity and engagement, implementing these guidelines without further deliberation and revision risks making qualitative research even more difficult to pursue and publish, provides disincentives to new scholars to venture into qualitative terrains, and for some scholars, is an inappropriate model for scholarly engagement, evaluation and communication.  To yet others, the devil is in the details; some versions of research transparency and interpretability seem valuable or at least acceptable while others could harm valued research endeavors.

The first question, then, is why engage with this initiative at all?

Goals of Strategies for Transparency and Interpretability: We see three reasons to implement some version of DA-RT. The first is to protect scholarly integrity. Research is emerging across many disciplines on the difficulty of replicating or interpreting earlier research findings. Some of these findings are in canonical articles frequently taught to students; some involve drugs that can affect the course of a disease or even a life; some influence governments’ policy choices. Examples outside political science include psychology, economics (here and here) and medical research.  Data transparency can also shield students from undue influence by  powerful professors, and permit examination of claims about intentional or inadvertent plagiarism or about assertions that go beyond an evidentiary base.

Within political science, scholars have demonstrated the “.05 effect,” our version of the classic file drawer problem (here  and here).   Other political scientists have also pointed to additional values in reanalyzing  or reinterpreting the evidence that lies behind empirical results.

These concerns are not limited to research in quantitative disciplines or subfields.  Within qualitative or historical research—again looking only outside political science—disputes over books by Michael Bellesiles and  Alice Goffman (here  and here) suggest the worth of clear guidelines about how relevant portions of archives, field notes, or interviews should be made available to other scholars. Suggestions of cherry-picking a few quotations from interviews, a few sentences from archives, or a few incidents in field notes can similarly undermine confidence in research-based publications (for example, see here).  It is in the interest of all political scientists to avoid unnecessary controversies and embarrassment to ourselves and our discipline.

A second reason to implement some version of DA-RT is positive; scholars and scholarship benefit when there is shared engagement around a set of evidence and plausible interpretations.  One of us (Hochschild), for example, has learned a lot from students who have reanalyzed interview transcripts from What’s Fair?.  For other examples, see here (gated) and here.

A third reason to support provision of enough evidence to enable others to replicate one’s results or evaluate one’s interpretation is to respond to external signals.  The social sciences, like the natural sciences and medical research, are facing rising expectations for accountability. Public and private funders (such as the National Institutes of Health,  National Science Foundation, and  The Economic and Social Research Council,) increasingly require grantees to make their evidence publicly accessible, including, albeit more slowly, qualitative research (see Germany’s RatSWD).  Publishers such as Cambridge University Press, Oxford University Press, and Sage are similarly moving to support greater transparency of evidence and analytic strategies.  Our legitimacy as scholars and political scientists in speaking to the public on issues of public concern rests in part on whether we adopt and maintain common standards for evaluating evidence-based knowledge claims.

What have political scientists done so far? If one agrees that there are good reasons to engage in some fashion with the commitment to research transparency and interpretability, the next question is what have political scientists done in this arena to date?  Depending on one’s perspective, the answer is a lot, some, or nowhere nearly enough.

After appointing a drafting committee in 2010, the APSA Council approved in 2012 new language on transparency and replicability for the APSA Guide to Professional Ethics in Political Science (see pp. 8-10).  The new guidelines were made public.[1]  Since then the DA-RT committee, working on its own, has held meetings, sessions at annual conventions, and a conference (details are here). In 2014, editors from some APSA section journals and the APSR made a commitment to implement strategies for research transparency.  Since then, editors of some non-APSA political science journals have implemented guidelines and, after convening working groups and consulting with its Editorial Board to help refine their own version, the APSR editors have recently promulgated draft guidelines.  These will go into effect on a trial basis in January 2016.

Skeptics, reformers, interested bystanders, and opponents have also published, met, and discussed their understanding of DA-RT and its possible implications, especially for the many varieties of qualitative research. Publications include an Editor’s Statement in Perspectives on Politics, a newsletter of the section on Qualitative and Multi-Methods Research, and a recent petition of APSA members.  A new website offers “a place to deliberate about standards of data access and research transparency in political science.” Blog posts, tweets, and comments on all possible sides abound. Questions remain about what issues have not been fully addressed by proponents, how DA-RT might evolve over time, whether some methodological or epistemological approaches will be disproportionately advantaged or disadvantaged, and who should make crucial decisions.  Those who believe that implementation of transparency guidelines should be postponed until after further deliberation are developing a method for consultation with members of APSA sections.

For some scholars, further discussion in the absence of much concrete experience in implementing research and analytic transparency in qualitative research is unnecessary and perhaps detrimental; for others, it is essential.  We take the concerns very seriously and have spent many hours deliberating on the best way for the APSA to move forward on this issue.  Here is our view:

We Support DA-RT Implementation in the APSR in January 2016:  We have three reasons, beyond noting the general move across the social sciences toward research transparency and interpretability:

What Comes Next?  In our view, it is becoming increasingly important for the Council to establish an explicit policy with regard to Council oversight of APSA-sponsored journals.  That policy could range from 1) “each editor or editorial team has complete autonomy” to 2) “journals published under the auspices of the APSA must adhere to the following guidelines [to be filled in],” to 3) “journals published under the auspices of the APSA may [or must] adhere to the following guidelines [to be filled in], with explicit Council review in XX years.” Which option is best is not yet clear, and whatever policy the Council adopts should be the outcome of substantive engagement and deliberation among members of our profession.

We have appointed a publications policy committee comprised of current APSA Council members, who will work with the extant membership-based Publications Committee to address this growing need for Council policy on publications.  The committee has been asked to propose a set of guidelines for the Council to discuss at its meeting in April 2016.  The proposed policy guidelines will be disseminated widely and posted on APSAnet.org and www.politicalsciencenow.com. The drafting committee will solicit members’ comments. The proposed guidelines will be further discussed and voted on at the Council’s meeting in September 2016.  Review of the current implementation of DA-RT guidelines in the APSR will be part of that discussion.

In addition, over the next year, the President-Elect of APSA will propose, subject to the Council’s approval, a presidential Task Force on Professional Ethics.  It will continue and develop current discussions by APSA committees of issues in professional ethics, including but not limited to matters of research transparency and interpretability in APSA-sponsored publications. As this proposed Task Force and its agenda are developed, we will solicit additional insights and topics of concern from the membership.

We welcome responses and comments to this letter below, and encourage continued conversations about how best to achieve the goals of research transparency and interpretability in ways that are responsive to the entire APSA membership.  Disagreements will surely persist, but we hope that decisions about policies such as DA-RT will be based on deliberation, mutual trust, and shared accommodation; that is the mark of a professional association of which we can all be proud members.

Jennifer Hochschild, President

David A. Lake, President-Elect

Rodney E. Hero, Immediate Past President

[1] In the member newsletter, APSA website, and PS: Political Science and Politics:  1) Minutes of the Council Meeting,  45 (3):589;  46 (1):197; and 46 (3): 707;  2) Openness in Political Science: Data Access and Research Transparency 47 (1):19; and 3)  2014 Report of the Committee on Professional Ethics, Rights, and Freedoms 48 (1): 210-11.

Dart Discussion Resources from APSAnet.org