Selection Methodology

The studies included in the What We Know research portal are selected by the Project staff at Columbia Law School, with the input of scholarly subject matter experts from universities across the U.S. and abroad, and with the counsel of the board of advisors. We use a strict set of criteria for selecting studies based on credibility, relevance and usefulness. All studies must be peer-reviewed, published in a scholarly journal, and directly relevant to the policy question at hand. To make our selections, Project staff survey the universe of peer-reviewed, published scholarship that adds in some way to the world’s knowledge about the policy issue in question. Adding to knowledge does not necessarily mean drawing new conclusions but can include strengthening existing knowledge by corroborating what prior studies have shown.

To select included studies, we perform multiple keyword searches using scholarly databases at Columbia Law School and elsewhere, review articles and reports produced by subject experts and professional societies and organizations, and reach out to leading academic experts to ensure we have identified all relevant scholarship that meet our criteria. We assess all relevant studies published since 1985, as that constitutes roughly the most recent generation of scholarship. Studies may include primary, experimental, longitudinal, qualitative and quantitative research in a variety of social science disciplines, as well as meta-analyses and case studies.

Studies that have been criticized or rebuked for their methodologies remain eligible for inclusion, but may appear with a note if the author, the publishing journal or a critical mass of scholars has substantially challenged the methodology of the study or the peer-review process by which a particular study was accepted for publication. Only when a study has been formally retracted by its author or publisher would it be disqualified from inclusion.

Inclusion of a study does not constitute an endorsement of its conclusions by Columbia Law School or by the faculty, staff or board of the What We Know project, which does not take policy positions. Our purpose is not to pick and choose research that endorses a particular policy view but to include the broadest reasonable range of relevant scholarship so that users may both obtain an overview of the present state of scholarly knowledge on topics that are currently matters of public debate, and further examine that research directly if desired. We recognize that the peer-review process is imperfect but we operate on the principle that it represents the best method we have for holding research accountable to both good faith and sound methodologies.

%d bloggers like this: