\documentclass[12pt]{article}

\usepackage{iaw}

\usepackage{rfs}

\renewcommand{\thesection}{\arabic{section}} 
\renewcommand{\thesubsection}{\arabic{section}.\arabic{subsection}} 

\addtolength{\textheight}{0.7cm}
\addtolength{\topmargin}{-0.35cm}

\usepackage{morefloats}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% these are paper-specific definitions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand{\source}[1]{{\scriptsize\color{blue} \textbf{Source:} #1}}
\renewcommand{\source}[1]{}

\newcommand{\explain}[1]{%
  \bigskip\bigskip\par%
  \setstretch{2}
  \begin{minipage}{\textwidth}\normalsize
  \begin{doublespacing}%
    {\par\textbf{Explanations:} #1}%
  \end{doublespacing}%
  \end{minipage}
}
\newcommand{\interpret}[1]{\par\medskip\noindent\textbf{Interpretation:} #1}

\newcommand{\interbase}[2]{\explain{#1} \interpret{#2}}
\newcommand{\inter}[2][]{\interbase{See also Table~\ref{tbl:overall}.  #1}{#2}}

\def\imagetop#1{\vtop{\null\hbox{#1}}}

\usepackage{float}
\floatstyle{plaintop}
\restylefloat{table}
\restylefloat{figure}

\newenvironment{citemize}{\begin{itemize}\itemsep=-0.5ex\parsep=-0.5ex}{\end{itemize}}
\newenvironment{cenumerate}{\begin{enumerate}\itemsep=-0.5ex\parsep=-0.5ex}{\end{enumerate}}

\newcommand{\tref}[1]{\marginpar{\tiny\bf\qquad\mbox{[Insert Tbl~\ref{#1}]}}\ref{#1}}
\newcommand{\fref}[1]{\marginpar{\tiny\bf\qquad\mbox{[Insert Fig~\ref{#1}]}}\ref{#1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\title{Referee Recommendations\thanks{This research was supported in financial terms by no one other than the author.  I want to thank all editors that agreed to help me with this study (Daron Acemoglu, Franklin Allen, Harold Cole, Cam Harvey, Christian Hellwig, David Hirshleifer, James Hosek, Larry Katz, Robert Richmond, Matthew Spiegel, Joachim Voth, Fabrizio Zilibotti), the referees and authors from the SFS Cavalcade, the UCLA Office of the Human Research Protection Program (\#12-000825), multiple anonymous referees, Jaclyn Einstein, and the editor, Andrew Karolyi.  Any mistakes, errors, or misinterpretations are mine alone.  Because this paper is about heterogeneity in referees, the author's website will post some remarkably negative earlier referee reports on this very paper.}}

\author{\href{mailto:ivo.welch@gmail.com}{\textbf{Ivo Welch}}\\University of California at Los Angeles\\Anderson Graduate School of Management}

\maketitle

%\thispagestyle{empty}

\clearpage

\newcommand{\sfscav}{\textrm{SFS Cavalcade}}

\thispagestyle{empty}

\begin{abstract} \normalsize%
  This paper quantitatively analyzes referee recommendations at eight prominent economics and finance journals, and the \textrm{SFS (Society for Financial Studies) Cavalcade Conference}, where a known algorithm matched referees to submissions.  The behavior of referees was similar in all venues.  The referee-specific component in the disposition recommendation was about twice as important as the common component.  Referees differed both in their scales (some referees were intrinsically more generous than others) and in their opinions of what a good paper was (they often disagreed about the relative ordering of papers).
\end{abstract}

\textit{JEL} Code: A14

\bigskip

\bigskip

\clearpage

\setcounter{page}{1}

The editorial process determines not only the evolution of economics and finance but also the incentives and professional fates of academic economists.  Yet, its participants do not have much objective knowledge about the process.  Authors write only a few papers per year and typically receive only a few referee reports per submission.  They usually do not learn which other papers were rejected.  They rarely find out why an editor chose a particular referee, much less who the referee was.  In turn, they referee only a few papers themselves every year and rarely receive feedback about how their views lined up with those of other referees.

The heterogeneity among referee evaluations is further exacerbated by the fact that the journals themselves have also not explicitly stated their objectives and criteria other than in broad and uncontroversial terms.    For example, some referees hold the view that only the submitted paper should influence editorial decisions and that fairness to authors is a main goal.  Others hold the view that journals should select submissions to maximize their impact, allowing such factors as the identities or institutions of the authors to play a role.   However, most economists would agree that it should ideally be submission- and author-associated factors in the broadest sense and not referee-associated factors that should determine publication.\footnote{It is possible that the editorial process is a second-best solution to a moral-hazard problem: editors may have to indulge referee-idiosyncratic opinions in order to incentivize volunteer referees to participate in the editorial process.} 

My paper studies the extent to which referee recommendations reflect a shared consensus versus the extent to which they reflect  referee-specific perspectives.  If recommendations are relatively more idiosyncratic, then the evolution of knowledge is likely to be more path-dependent (and the careers of economists more random) than if recommendations reflect a general consensus.

There could be many reasons why referees share perspectives. They could agree not only with respect to the characteristics of the submissions (such as its novelty, interestingness, accuracy, rigor, and polish, as pointed out by \citealt{ellison:2002a}), but also with respect to other non-submission-related characteristics (such as the identity of the authors).  I shall refer to these aspects as the ``reliable qualities'' of submissions (not to be mistaken for the true scientific quality). Referees could also disagree for other reasons.  There could be heterogeneity in their weightings of these characteristics, or there could be referee-specific factors such as noise, skills, time investments, moods, beliefs, ideologies, personal likes,  age or cohort, professional networks, vanity, suppression of contrary evidence, or turf motives.  Of course, if referees agreed about these characteristics and placed similar weights on them (e.g., if all referees liked pro-free-market papers), these same characteristics would become repeatable commonalities among referees.  This would lead my paper to classify these components as ``reliable''---again highlighting the difference between reliability (which I can measure) and submission quality (which I cannot measure). It is by this definition that the influence of referee characteristics that are not common (reliable) across referees become idiosyncratic (unreliable).  The draw of the referee matters less when the reliable component of referee recommendations plays a more important role than the subjective component.\footnote{In my paper, I sometimes refer to the idiosyncratic referee-specific aspects as the ``subjective evaluation'' of the submission.  This is not meant to imply that the common aspects do not contain subjective but widely shared views, or that the subjective evaluation cannot be based on objective criteria that only one referee considered.}

An immediate concern in any study that seeks to determine the reliability of referees' recommendations is that editors do not choose referees randomly.  This makes it difficult to determine whether any observed consensus reflects a reliable component of the referees' views about the submissions or whether it reflects merely the editorial referee selection decision.  Therefore, my study examines referee behavior not just in the standard refereeing context (for eight journals: \textit{Econometrica} [ECMTA], the \textit{International Economic Review} [IER], the \textit{Journal of the European Economic Association} [JEEA], the \textit{Journal of Economic Theory} [JET], the \textit{Quarterly Journal of Economics} [QJE], the \textit{Rand Journal} [RAND], the \textit{Journal of Finance} [JF], and the \textit{Review of Financial Studies} [RFS]), but also in a conference venue with an unusual referee selection: In the 2012 \textrm{Society for Financial Studies (SFS) Cavalcade} conference, a known computer algorithm matched referees to submissions based only on shared expertise.

Studying the two venues represents different tradeoffs.  On the one hand, human journal editors can presumably match papers better to referee expertise, and journal referees spend more time on journal submissions than on conference submissions.  On the other hand, editors may select the number and identities of referees based on their own prior assessments of submission quality or even a desire to influence the referees' recommendation and/or the agreement among multiple referees.  My paper will show that referee behavior is very similar in both types of venues.  Without the journal data, the conference data could be viewed as too different from the journal settings.  Without the conference data, the journal data could be viewed as the result of deliberate editorial selection.   Together, Occam's razor suggests that my paper documents behavior that is typical of economics and finance referees, and not an artifact of referee selection.

\needspace{0.2\textheight}

My paper focuses on the referees' final recommendations to the editors in situations in which two referees evaluated the same paper.  The two key findings are straightforward.  First, I document that the consensus among referees was modest.  The idiosyncratic referee component is stronger than the common reliable component.  The following simple statistics put this in perspective.  The unconditional probability that a referee at the \sfscav\ would recommend accepting a paper was 28.5\%.  When one referee recommended accepting, the probability that another referee would agree increased only from 28.5\% to 38.2\%.  (At the eight journals, the equivalent figures were 31\% and 34\%, respectively.)  A decomposition model developed in my paper suggests a convenient summary statistic: similar levels of agreement among referees would have been observed if referees had placed about one-third weight on a shared signal and about two-thirds weight on their own idiosyncratic signal.  Second, I document that there was significant variation in the ``intrinsic generosities'' among referees.  For example, in the \sfscav, the probability that a referee would issue a ``must accept'' recommendation was only 3.2\%.  However, it increased to 6.9\% if this referee judged other papers, \textit{not including the current one}, to be at least of ``neutral'' average quality. \source{intro-otherrefmean.R}  Yet not all disagreement can be explained by differences in the average scorings of referees.  When two \sfscav\ referees evaluated the same two papers, they agreed which paper was better in 972 cases and disagreed in 702 cases.

Beyond these two core findings, my paper interprets some of the consequences of the observed referee behavior and documents some further empirical regularities, including the behavior of editors in at least one journal.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\section{The Data}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 

This section describes the data from the \sfscav\ and the eight economics and finance journals used in the analysis.\footnote{The gathering and analyses programs are available on the website.  The data itself is unfortunately too sensitive and confidential to be sharable.}

\subsection{The \sfscav}

I was the chair for the  2012 \sfscav\ conference.  Eighteen distinguished researchers had been chosen as program committee members by the association before my appointment.  I solicited additional referees from the set of 663 submitting authors.  The assignment of referees to papers was made by a computer program without my intervention using the following algorithm:
\begin{cenumerate}
\item %score-paperauthor.pl --- also tells how many authors had no expertise
  For each submission-referee combination, the program computed a raw proximity score, based on the number of categories that the referee and paper shared.  Categories were based on areas (four large areas like ``asset pricing,'' fifty-one subareas like ``options,'' and \textit{JEL} classification codes), on approach (such as ``theoretical''), and on level of complexity (such as ``low-tech'').  Authors and referees could choose as many designations as they liked.  For example, if a referee indicated as expertise ``Asset Pricing, International Asset Pricing, Theoretical, Structural, Mid-Tech,'' and the authors classified a submission to be ``Asset Pricing, Empirical, Structural, Mid-Tech,'' the intersection was ``Asset Pricing, Structural, Mid-Tech.''  The raw score was then the number of intersecting categories squared, divided by one plus the number of categories for the paper times one plus the number of categories for the author.  In this example, the proximity score would have been $3^2/[(5+1)\times(4+1)]=0.3$.

\item % allocatepapers
  After excluding authors of their own papers, the program iterated through the proximity-score-ranked list to assign referees to papers, making sure not to assign too many papers to each referee, and not to assign too many referees to each paper. The target number of papers per referee was 21 for the program committee members and 5 for ordinary referees.   The target number of referees per paper was 5.  
\end{cenumerate}
The median proximity score was about 0.2, with an interquartile range of about 0.1 to 0.3.  The distribution of proximity scores was similar for ordinary referees and for \sfscav\ program committee members.  Most important, because I did not intervene subjectively in the referee selection, the only paper-assignment selection bias could be one that was encoded in the computer algorithm---that is, an expertise-related one.

Table~\tref{tbl:numrefspprs} shows the final distribution of recommendations used in the analysis.   In total, 578 referees returned 3,126 recommendations on 367 papers.   The referees themselves identified 216 recommendations to be conflicted (vis-\`a-vis the submitting author), leaving 2,910 unconflicted recommendations.  Because the mean paper rating for the conflicted reports was significantly higher, the remainder of my paper focuses only on these 2,910 unconflicted recommendations.  The 18 program committee members provided between 9 and 28 paper reviews.  No ordinary referee evaluated more than 5 papers.   The most common number of referees per paper was 6--7.

\begin{table}[e]

  \bigskip

  \caption{\sfscav\ number of referees per paper and number of papers per referee \label{tbl:numrefspprs}}

  \nothing{
  \begin{ctabular}{l *{12}{c} }
    \toprule
    & \mc{11}c{\underline{Number of referees per paper}} \\[1ex]
    \#Referees & 2 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13  \\ 
    \cmidrule{2-12}
    Incidences & 1 & 8 & 31 & 85 & 61 & 35 & 34 & 31 & 27 & 13 & 8 \\
    Total Reports &   2 & 32 & 155 & 510 & 427 & 280 & 306 & 310 & 297 & 156 & 104 \\
    \midrule
    \rule{0pt}{2em}
    \#Referees & 14 & 15 & 16 & 17 & 18 & 19 & 20 & 21 & 22 & 28 & Total \\
    \cmidrule{2-12}
    Incidences & 14 & 4 & 2 & 2 & 2 & 3 & 2 & 2 & 1 & 1  & 367 \\
    Total Reports  & 196 &  60 & 32 & 34 & 36 & 57 & 40 & 42 & 22 & 28 & 3,126 \\
    \bottomrule
    \rule{0pt}{2em}
    & \mc{11}c{\underline{Number of papers per referee}} \\[1ex]
    \#Papers & 1 &  2 &  4 &  5 & 10 & 13 & 17 & 20 & 21 & 26 & 30 & Total \\
    \cmidrule{2-13}
    Incidences & 1 &  2 & 13 & 545 &  1 &  2 &  1 &  1 &  9 &  2 &  1 & 578\\
    Total Reports & 1 & 4 & 52 & 2,725 & 10 & 26 & 17 & 20 & 189 & 52 & 30 & 3,126 \\
    & \mc4{r|}{$\leftarrow$Submitter-referee} & \mc7{|l}{\sfscav\ program committee member$\rightarrow$} \\
    \bottomrule
  \end{ctabular}
}

% aa <- t(simplify2array(by( sfs, as.factor(sfs$pid), function(d) c(nrow(d),d$pid[1]))))
% bb <- summary(factor(aa[,1])); print(bb)
% cc <- as.numeric(names(bb))*bb; print(cc)
% print(cumsum(cc))

\small
  \begin{ctabular}{l *{30}{c} }
    \toprule
    & \mc{11}c{\underline{Number of referees per paper}} \\[1ex]
    \#Referees   & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 \\
    \cmidrule{2-12}
    Incidences & 1 & 1 & 3 & 22 & 49 & 73 & 59 & 36 & 39 & 24 & 15 \\
    Reports    & 1 & 2 & 9 & 88 & 245 & 438 & 413 & 288 & 351 & 240 & 165 \\
    Cumulative & 1 & 3 & 12 & 100 & 345 & 783 & 1,196 & 1,484 & 1,835 & 2,075 & 2,240 \\
    \midrule
    & \mc{11}c{\underline{Number of referees per paper}} \\[1ex]
    \#Referees & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 & 21 & 28 \\
    \cmidrule{2-12}
    Incidences & 11 & 9 & 9 & 3 & 2 & 3 & 1 & 1 & 3 & 2 & 1 \\
    Reports & 132 & 117 & 126 & 45 & 32 & 51 & 18 & 19 & 60 & 42 & 28 \\
    Cumulative & 2,372 & 2,489 & 2,615 & 2,660 & 2,692 & 2,743 & 2,761 & 2,780 & 2,840 & 2,882 & 2,910 \\
    \bottomrule
 \end{ctabular}
 
  \begin{fctabular}{l *{30}{c} }
    \toprule
    & \mc{11}c{\underline{Number of papers per referee}} \\[1ex]
    \#Papers  & 1 & 2 & 3 & 4 & 5 & 9 & 12 & 17 & 18 & 20 & 21 & 25 & 26 & 28 \\
    \cmidrule{2-15}
    Incidences & 3 & 7 & 30 & 125 & 395 & 1 & 2 & 1 & 3 & 2 & 5 & 1 & 1 & 1 \\
    Reports & 3 & 14 & 90 & 500 & 1,975 & 9 & 24 & 17 & 54 & 40 & 105 & 25 & 26 & 28 \\
    Cumulative & 3 & 17 & 107 & 607 & 2,582 & 2,591 & 2,615 & 2,632 & 2,686 & 2,726 & 2,831 & 2,856 & 2,882 & 2,910 \\
    \midrule
    & \mc5{r|}{$\leftarrow$Submitter-referee} & \mc7{|l}{\sfscav\ program committee member$\rightarrow$} \\
    \bottomrule
  \end{fctabular}

  \explain{Papers were matched to referees based only on shared expertise.  There were 578 referees evaluating 367 papers with 3,126 recommendations.   The analysis in my paper is based on the 2,910 reports in which referees rated themselves ``not conflicted.''  Only members of the program committee were asked to referee more than five papers. }

  \bigskip
\end{table}





%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Academic journals}

I also obtained access to data from six economics journals (\textit{Econometrica} [ECMTA], the \textit{International Economic Review}~[IER], the \textit{Journal of Economic Theory}~[JET], the \textit{Journal of the European Economic Association}~[JEEA], the \textit{Quarterly Journal of Economics}~[QJE], and the \textit{Rand Journal}~[RAND]) and one finance journal (the \textit{Review of Financial Studies} [RFS]) that used the Editorial Express (EE) web system.  In addition, I was given access to redacted data from the \textit{Journal of Finance}.   The editors of the EE journals ran a perl program on my behalf in-house on an EE data dump that they downloaded to their own local computers.  Thus, I never had direct access to the data itself but was still able to link referees on one paper to their decisions on other papers.  


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Frequency of multiple referees}


\begin{table}
\caption{Multiple referee situations} \label{tbl:numbers}

\begin{ctabular}{l c s c s c S  *{8}{i} cc s c s c s ccc}
\toprule
                          & Submissions with & Mean referees   & Referee   \\
                          & $>1$ referee & per submission       & pairs \\
\midrule
\textit{Econometrica} (ECMTA)      & 75\% & 2.1 & 15,826 &      & 0.64       & 0.13 & 0.11 & 0.06 & 0.04 & 0.02 \\
\textit{International Economic Review} (IER)  & 85\% & 2.1 & 9,702 & 0.05 &       0.52 & 0.07 & 0.25 & 0.04 & 0.03 & 0.03 \\
\textit{J of the European Economic Association} (JEEA)
                      & 74\% & 2.2 &  9,922 & 0.09 &       0.50 & 0.14 & 0.15 & 0.06 & 0.04 & 0.01 \\
\textit{J of Economic Theory} (JET)     & 50\% & 1.6 &  3,024 &      & 0.49       & 0.14 & 0.17 & 0.09 & 0.08 & 0.04 \\
\textit{Quarterly J of Economics} (QJE)   & 88\% & 2.6 & 16,544 & 0.09 &       0.57 & 0.12 & 0.12 & 0.05 & 0.03 & 0.01 \\
\textit{Rand Economic J} (RAND)       & 81\% & 1.9 &  2,322 & 0.11 &       0.53 & 0.14 & 0.14 & 0.05 & 0.02 & 0.01 \\
\midrule
\mc3r{6 economics journals}      &  57,340 & 0.06 & 0.56 & 0.12 & 0.15 & 0.06 & 0.04 & 0.02 \\
\\[2em]
\textit{J of Finance} (JF)         & 20\% & 1.2 &   1,856 &      & 0.60       & 0.33 &      &  0.06 & &            \\
\textit{Review of Financial Studies} (RFS)   & 31\% & 1.3 &   3,548 & 0.17 &      0.56 & 0.10 & 0.11 & 0.04 & 0.01 & 0.00 \\
\mc1l{RFS (same time)}      & 28\%  & 1.3 &   3,028 & 0.17 &      0.57 & 0.10 & 0.11 & 0.04 & 0.01 & 0.00 \\
\midrule
\mc3r{2 finance journals}       &   5,404 & 0.11 & 0.58 & 0.18 & 0.07 & 0.05 & 0.01 & 0.00 \\
\\
\mc3r{8 journals}              &   62,744 & 0.06 & 0.56 & 0.13 & 0.14 & 0.06 & 0.04 & 0.02 \\
\\
\\
\sfscav                   & All & 5.1 &  24,370 & 0.07 &      0.32 & 0.32 &       0.25 & & 0.04 &     \\
\midrule
\mc3r{All 9 venues} &   87,114 & 0.06 & 0.50 & 0.18 & 0.17 & 0.04 & 0.04 & 0.01 \\[1ex]
\bottomrule
\end{ctabular}

\explain{The first column is the venue.  The second column is the frequency of papers that had more than one referee.  The third column is the mean number of referees per paper.  The fourth column is the number of paired evaluations that are available for analysis.  Each pair is a unique combination of two referees evaluating the same paper.  (Thus, for example, one paper with 5 referees would provide 10 referee pairs.)  These statistics exclude desk rejects.}

\source{pair-stats.R}

\bigskip
\end{table}

Most of my analysis focuses on referee-pairs---that is, situations in which two or more referees evaluated the same submission.  By necessity, the referee-pair unit of analysis excluded both desk rejects and single-referee submissions. When a submitted paper had more than two referees, each possible pair evaluation was entered as one observation in much of the analysis---a paper with three [$n$] referees yielded three [$n\times(n-1)/2$] pairs.   Table~\tref{tbl:numbers} shows the fraction of submissions that were evaluated by more than one referee, the average number of referees per paper, and the number of pairs.

\nothing{About 6\% of the paired referee reports recommended accept, 33\% recommended revise, and 60\% recommended reject.  166/1411/5879}

The two finance journals tended to use fewer referees per submission than the six economics journals. The JF used more than one referee in only 20\% of their submissions. The mean number of referees was 1.2.  The RFS used more than one referee in 31\% of their submissions. The mean number of referees was 1.3.  At the economics journals, the average number of referees ranged from 1.6 referees per paper at JET to 2.1 and 2.6 referees per paper at ECMTA and the QJE.  The number of referee pairs grows quadratically with the number of referees.  Thus, while the \textit{Journal of Finance} provided only 1,856 paired referee recommendations, the QJE provided 16,544 and ECMTA provided 15,826.  With its unusually large number of referees per paper (an average of 5.1 referees per paper), the 2,910 recommendations in the \sfscav\ yielded 24,370 referee pairs.  With 87,114 referee-pair recommendations, most statistics reported in my paper have small standard errors, allowing the reader to focus on the economic meaning of the estimates.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Frequency of categorical recommendations}

\begin{table}
\caption{Frequency of referee recommendations} \label{tbl:frequency}

\begin{ctabular}{l iii S  cc s c s c s ccc}
\toprule
%                          & $>$1ref & pairs & ``strong''    & ``plain''  &  ``weak'' & &  ``strong'' &  ``w/ $\Delta$''  \\
                          & $>1$ refs & R/P & Pairs & SREJ &  REJ &  WR\&R &   R\&R &  SR\&R &  ACR &  ACC \\
%                         &  &        & 0.95 &  1.00 & 1.05 &   2  & 2.5  &   3  &   4  & 5 \\
\midrule
ECMTA                     & 75\% & 2.1 & 15,826 &      & 0.64       & 0.13 & 0.11 & 0.06 & 0.04 & 0.02 \\
IER                       & 85\% & 2.1 & 9,702 & 0.05 &       0.52 & 0.07 & 0.25 & 0.04 & 0.03 & 0.03 \\
JEEA                      & 74\% & 2.2 &  9,922 & 0.09 &       0.50 & 0.14 & 0.15 & 0.06 & 0.04 & 0.01 \\
JET                       & 50\% & 1.6 &  3,024 &      & 0.49       & 0.14 & 0.17 & 0.09 & 0.08 & 0.04 \\
QJE                       & 88\% & 2.6 & 16,544 & 0.09 &       0.57 & 0.12 & 0.12 & 0.05 & 0.03 & 0.01 \\
RAND                      & 81\% & 1.9 &  2,322 & 0.11 &       0.53 & 0.14 & 0.14 & 0.05 & 0.02 & 0.01 \\
\midrule
\mc3r{6 economics journals}      &  57,340 & 0.06 & 0.56 & 0.12 & 0.15 & 0.06 & 0.04 & 0.02 \\
\\
JF                        & 20\% & 1.2 &   1,856 &      & 0.60       & 0.33 &      &  0.06 & &            \\
RFS                       & 31\% & 1.3 &   3,548 & 0.17 &      0.56 & 0.10 & 0.11 & 0.04 & 0.01 & 0.00 \\
\mc1l{RFS (same time)}      & 28\%  & 1.3 &   3,028 & 0.17 &      0.57 & 0.10 & 0.11 & 0.04 & 0.01 & 0.00 \\
\midrule
\mc3r{2 finance journals}       &   5,404 & 0.11 & 0.58 & 0.18 &   & 0.05 & 0.01 & 0.00 \\
\midrule
\\
\mc3r{8 journals}              &   62,744 & 0.06 & 0.56 & 0.13 & 0.14 & 0.06 & 0.04 & 0.02 \\
\\
\\
\sfscav                   & All & 5.1 &  24,370 & 0.07 &      0.32 & 0.32 &       0.25 & & 0.04 &     \\
\midrule
\mc3r{All 9 venues} &   87,114 & 0.06 & 0.50 & 0.18 & 0.17 & 0.04 & 0.04 & 0.01 \\[2ex]
\mc4r{Summary categories:} & \mc2{cs}{Reject (REJ)} & Weak (WEAK) & Revise (R\&R) & \mc3c{Accept (ACC)} \\
\mc4r{} & \mc2{cs}{56\%} & 18\% & 17\% & \mc3c{9\%} \\
\bottomrule
\end{ctabular}

\explain{The first column is the venue.  The remaining columns are the unconditional frequencies based on all referee pairs.  In the Editorial Express (EE) journals, the ``definitely reject'' choice was ``SREJ,''  the ``weak revise and resubmit'' was ``WR\&R,'' the ``strong revise-resubmit'' was ``SR\&R,'' and the ``accept subject to revisions'' was ``ACR,''   The ECMTA and JET programs did not distinguish between SREJ and REJ.  Most of the analysis in my paper relies on the four summary categories at the bottom of the table.}

\source{pair-stats.R}

\bigskip
\end{table}

In my sample, referees for EE journals had seven choices: ``definitely reject,'' ``reject,'' ``weak revise and resubmit,'' ``revise and resubmit,'' ``strong revise and resubmit,'' ``accept with revisions,'' and ``accept.''  Referees for the SFS Cavalcade had five choices: ``must reject,'' ``should reject,'' ``neutral,'' ``should accept,'' and ``accept.''  Referees for the \textit{Journal of Finance} had three choices: ``reject,'' ``resubmit,'' and ``accept.''   To make the journals more comparable (and because the differences in meaning between some categories are difficult to understand), most of my analysis collapses the recommendations into four categories (except at the JF, where I only had three categories to begin with), dubbed ``Reject'' (REJ), ``Weak'' (WEAK), ``Revise'' (R\&R), and ``Accept'' (ACC).  Table~\tref{tbl:frequency} shows how this mapping was accomplished.  

More important, Table~\ref{tbl:frequency} shows that referees at the six economics journals tended to be more generous than referees at the RFS finance journal in the R\&R and better categories.  Due to the differences in the number of categories, the two other venues (JF and the \sfscav) are  difficult to compare, although the patterns seemed qualitatively similar.\nothing{Not reported, the distribution of recommendations among paired referee assessments was similar to the distribution of recommendations among all referee reports, except at the JF where multiple-referee papers had higher average recommendations.}  Overall, 56\% of the referee reports recommended REJ, 18\% were WEAK (very cautious-revise-and-resubmits), 17\% were R\&R, and 9\% were ACC.   Note that these recommendations are referee recommendations and not journal decisions.  They do not allow inferring the selectivities of the venues themselves.



\subsection{Comparison}

The journal data and the \sfscav\ data have different strengths and weaknesses.  The advantages of the journal setting are (i) journals had multi-year histories (many more refereed papers), (ii) journal editors could match papers better with referee expertise than my computer program could, and (iii) referees probably spent more time evaluating each submission.\footnote{However,  \sfscav\ referees that had better-matched expertise, that claimed to have spent more time on the paper, and that had more papers to review, did not show more or less consensus among themselves than other referees.}  The advantages of the \sfscav\ setting are (i) referee assignments are guaranteed not to be correlated with an a priori assessment of the paper's quality by the editor or with an a priori intent of editors to solicit agreement or disagreement, and (ii) each paper had an unusually large number of referees. In addition, there could be other differences.  The same referee may put different weights on different attributes in different venues.  For example, conference referees may have put relatively more weight on whether a submission was interesting than whether its proofs were correct.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Consensus}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The \sfscav\ paired recommendation matrix}

\begin{table}[e]

  \bigskip

  %% created by ee/ee-ref2.R based on 

  \caption{\sfscav\ referee recommendations conditional on one other referee's recommendations} \label{tbl:ownother}

  \begin{ctabular}{cc *{4}{r} r S r rrrrr}
    \toprule
    &           & \mc5{cS}{\underline{Other referees' recommendations}} \\
    &           &   \mc1c{MR} &   \mc1c{SR} & \mc1c{NR} & \mc1c{SA} & \mc1{cS}{MA} & \mc4c{\underline{Unconditional}} \\
    \mc2c{\underline{Own Recommendation}}
                & \mc1c{--2} & \mc1c{--1} & \mc1c{0} & \mc1c{+1} & \mc1{cS}{+2} & Pairs & Freq & Reports & Freq \\
    \midrule
    Must Reject MR & --2    & 312 &   718 &   461 &   244 &   32 & 1,767 & 0.073 & 202 & 0.069 \\
    Should Reject SR & --1    & 718 & 2,904 & 2,494 & 1,520 &  163 & 7,799 & 0.320 & 905 & 0.311 \\ 
    Neutral NR & 0      & 461 & 2,494 & 2,584 & 2,027 &  300 & 7,866 & 0.323 & 933 & 0.321 \\
    Should Accept SA & +1     & 244 & 1,520 & 2,027 & 1,844 &  368 & 6,003 & 0.246 & 743 & 0.255 \\
    Must Accept MA & +2     &  32 &   163 &   300 &   368 &   72 &   935 & 0.038 & 127 & 0.044 \\
    \midrule
    \mc7{rs}{Total:} & 24,370 & 1 & 2,910 & 1\\
    \\[1em]
    \mc8c{\underline{Translated into conditionals}} \\
    &           &   \mc1c{MR} &   \mc1c{SR} & \mc1c{NR} & \mc1c{SA} & \mc1{cS}{MA} \\
    \midrule
    Must Reject MR & --2    & 0.177 & 0.41 & 0.26 & 0.14 & 0.018   \\
    Should Reject SR & --1    & 0.092 & 0.37 & 0.32 & 0.19 & 0.021   \\
    Neutral NR & 0       & 0.059 & 0.32 & 0.33 & 0.26 & 0.038   \\
    Should Accept SA & +1     & 0.041 & 0.25 & 0.34 & 0.31 & 0.061   \\
    Must Accept MA&   +2&    0.034&   0.17& 0.32& 0.39&   0.077  \\
    \midrule
    \mc2c{Unconditional} & 0.073 & 0.320 & 0.323 & 0.246 & 0.038 \\
    \bottomrule
  \end{ctabular}

  \explain{The predicted referee recommendations are based on 2,910 pairable referee recommendations from the \sfscav.  For each referee recommendation in the first two columns, I tabulate the frequency of recommendations for the same paper by other referees.  (Thus, for example, a paper with 5 referees provided 10 referee pairs.)}

  \medskip

  \interpret{Referee recommendations on the same paper are significantly positively correlated.  They share a reliable component.  However, the correlation is modest.  The matrix does not approximate the identity matrix.  The reliable component is not large.}

  \source{ownother.R}

  \bigskip
\end{table}

To aid intuition, start with the \sfscav, the venue with the largest number of referee pairs.  Table~\tref{tbl:ownother} tabulates the observed recommendations.  The bottom two rows in the top-right subtable show that of the 2,910 unconflicted recommendations, 127 (4.4\%) advised ``Must Accept'' (MA), and 743 (25.5\%) advised ``Should Accept'' (SA).  Thus, about one in three recommendations was positive.  About one in three recommendations was neutral (although neutral is widely understood to mean rejection in highly competitive contexts).  And about one in three recommendations was negative, ``Should Reject'' or ``Must Reject.''  These probabilities do not change much when computed for individual reports instead of for paired reports.  My discussion focuses on the two highest recommendations, MA and SA.  After all, only primarily positive recommendations allow a paper to be accepted into a selective journal or conference. The unconditional probability of an MA recommendation was 3.8\% (935 out of 24,370 paired recommendations).\footnote{There was relatively more consensus for papers that received an MR.  Another referee is likely to share this view with 16.7\% probability, higher than the 7\% unconditional probability of an MR.}

The upper table shows the raw number of paired recommendations.  The lower matrix is normalized to conditional probabilities.  Inspection of the matrix reveals that there was modest consensus.  Consider a paper that received one rare MA endorsement from one referee:
\begin{itemize}\itemsep=0em\parsep=0em
\item The probability that another referee also offered an MA recommendation was $72/935\approx7.7\%$---higher than the unconditional 3.8\%, but far from 100\%.
\item The probability that another referee offered the next-best recommendation of SA was $368/935\approx39\%$---higher than the unconditional 25\%, but still not even a fair bet.
\item The probability that another referee recommended not only a reject but a strong reject (MR) was still $32/935\approx3.4\%$---lower than the unconditional 7.3\%, but not zero.
\end{itemize}
Thus, even given one MA, the chances were better that another referee would offer a negative to neutral recommendation on the submission ($495/935\approx53\%$) than that she would offer a second positive recommendation.  (A neutral evaluation essentially suggested nonselection of the submission in this competitive a venue.)

The picture is much the same when one referee reported either an SA or an MA ($6,003+935$ out of 24,370 recommendations, 28.5\%).  The probability that another referee's recommendation was SA or MA was $(72+368+1844+368)/(6,003+935)\approx38.2\%$, higher than the $28\%$ unconditional probability, but not close to~100\%.

In sum, there was more consensus than would have been observed by random chance if referees' opinions had been uncorrelated, but much less than what would have been observed if assessments had been perfectly reliable.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 

\subsection{A decomposition model of referee behavior} \label{sect:consensus}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 

%\newcommand{\Var}{\text{Var}}
%\newcommand{\Cov}{\text{Cov}}
\newcommand{\Cor}{\text{Cor}}

It is not easy to interpret pairwise recommendation matrices intuitively.  Thus, it is useful to consider a simpler model that maps recommendation matrices into summary statistics.

\subsubsection{A low-dimensional model with continuous reports}

Assume that referee $R\in(A,B)$ places weights $w_R$ on $k$ different unit-normalized and orthogonalized characteristics $c_k$ of the submission,
\[ r_R = w_R \cdot c = \sum_k w_{R,k} \cdot c_k \;.\]
Specifying the characteristics in this generic linear fashion allows the model to encompass a wide range of decision inputs, such as the submission's true scientific value, its likely  future impact, its writing quality and style, the identity of its authors, and so on.  It can also include characteristics that most academics would agree should not influence publication decisions (such as the \LaTeX\ format quality or number of vowels in the submission) and multiple noise terms.   If a referee $R$ does not observe characteristic $k$, her weight $w_{R,j}$ on this characteristic would be zero.

Assume that the editor (and my analysis) observes neither the individual characteristics nor the referees' weightings.  The editor observes only the final recommendation $r_R$.  Initially, assume that $r_R$ is not categorical but continuous, with finer gradations perhaps discernable through the reading of the full referee report.  The correlation between two referee assessments $r_A$ and $r_B$ for the same paper is
\[ \Cor(r_A,r_B) =  \frac{ w_A'\, C\, w_B }{\sqrt{ w_A'\, C\, w_A \, w_B'\, C\, w_B}} \;,\]
where $C$ is the $k$-by-$k$ matrix $c\, c'$.   For example, if referees A and B place weights $w_A=(1/6,2/6,3/6,0)$ and  $w_B=(2/6,1/6,0,3/6)$ on the $k=4$ characteristics, then the correlation among referee recommendations would be 0.286.  (In an OLS regression, one referee's opinion would explain 8.2\% of the variance in the other referee's opinion.)  The correlation is less than one, both because of differences in weighting on characteristics that both referees share (here the first two weights) and because of weights on characteristics for which referees have their own unique views (here the last two weights).\footnote{If an editor wanted to place 1/4 weight on each of the four characteristics (e.g., if the characteristics were importance of the paper's contribution to four different subfields), then she could obtain her desired estimate by averaging the two referees' recommendations.  More generally, with as many referees as characteristics, an editor who knows the weights of referees on characteristics could uncover the signals and thus determine an optimal linear combination of the signals to decide on the manuscript.} Any final correlation in referee recommendations maps into infinitely many higher-dimensional models.

The intent of my decomposition model is to characterize referee behavior with a summary statistic that can then be used in  simple thought experiments.  This model is calibrated to yield the same correlation as that observed in the data.  It maps the higher-dimensional space into a ``shared-signal'' decomposition with only three characteristics: one shared characteristic~($c_S$), one characteristic that is unique to referee~A~($c_A$), and one that is unique to referee~B~($c_B$), again with all three characteristics orthogonal and unit-normalized.  Lambda is the proportion of weight on the shared characteristic.  The referees' recommendations are
\begin{equation}
  r_A = \lambda\times c_S + (1-\lambda)\times c_A  \qquad r_B = \lambda\times c_S + (1-\lambda)\times c_B \label{eq:lambdabase}
\end{equation}
and
\[ \Cor = \frac{\lambda^2}{\lambda^2+(1-\lambda)^2} \qquad \Leftrightarrow \qquad \lambda = \frac{ \Cor - \sqrt{ \Cor - \Cor^2 }}{2 \times\Cor - 1} \;.\]
This shared-signal model gives the same correlation of 0.286 between referee assessments if both referees had placed weight $\lambda=0.387$ on the single shared characteristics $c_S$ and weights $1-\lambda=0.613$ on their unique terms, $c_A$ and $c_B$, respectively.  Any positive correlation between zero and one maps into one unique value of $\lambda$.   The extremes of this model are easy to interpret: if there is no overlap in the full high-dimensional model's weights $w_i$ or if the weights are  orthogonal ($w'_{A} w_{B}=0$), then lambda is 0.  If there is perfect overlap ($w_{A}=w_{B}$), then $\lambda=1$.   If lambda is 1/4, 1/2, or 3/4, then the correlation among referee reports $(r_A,r_B)$ is 1/10, 5/10, and 9/10, respectively.\footnote{The discussion in my paper is phrased in terms of this agreeable shared-signal summary model for its decomposition intuition, not for the presumption that referees do not make choices based on higher-dimensional evaluation functions.  My paper assumes that there is a true unknown summary parameter lambda (mapped from its true higher-dimensional function), and I observe a random draw with a sample lambda from which I infer the true lambda.}

Even if a true scientific paper quality existed upon which the paper should ideally be decided, the observed referee commonality is \textit{not} the referees' agreement about this true paper quality.  To see this, assume that this true quality is the first element in the vector $c$, and both referees observe it (perhaps with modest noise) but do not place 100\% weight on it.  The observed consensus among referees could then be higher than their agreement about the true quality---for example, if both referees based their recommendation only on a reliable but unimportant metric, such as the number of spelling errors.  The observed consensus could be lower if one referee believed that recommendations should be based on the citation counts of authors (to maximize future citation impact of the submission), while the other believed that her recommendation should be based purely on the rigor of the paper or the fact that the paper contradicts some of her own earlier research.  However, because this true paper quality is one among a number of reliable characteristics, if both referees placed positive weight on it, it would contribute to a positive correlation.  If the recommendation correlation were zero, it would seem unlikely that a true paper quality would be an important input into referee recommendations. % The exception may be paradigm-challenging papers (\citeasnoun{armstrong:1997}).

% xxx=sapply( (0:10)/10, function(l) mean( ifelse(rbinom(N,1,l), tq, rbinom(N,1,0.5)) == ifelse(rbinom(N,1,l), tq, rbinom(N,1,0.5))))
%  plot( (0:10)/10, xxx )


\subsubsection{Inferring lambda from observed discrete categories}

Although editors may be able to interpret the contents of the referee report and the letter to the editor to infer smooth gradations in recommendations, the final referee recommendations on which my own paper is based are the discrete categories reported by referees.  This makes it more difficult to infer lambda.

To fit the best lambda, I first simulate the reduced-space model based on the number of observed pairings for different lambdas.  I assume that the three characteristics, $c_S$, $c_A$, and $c_B$, are independent normals.  For example, for the SFS cavalcade, I draw 24,370 $c_S$ characteristics, 24,370 $c_A$ characteristics, and 24,370 $c_B$ characteristics.\footnote{This is to preserve the use of lambda as a summary statistic for the pairwise matrix.  For the \sfscav, I also knew the referee-paper pairings, which makes it possible to preserve the structure by simulating the 578 referees' 2,910 recommendations on 367 papers.  This provides some additional restrictions.  Not surprisingly, it barely changes the inference.  It changes the reported coefficient from 0.364 to 0.363.  The estimates in Table~\ref{tbl:yikes-ols} are based on this full-pairing estimation.}  This gives the two ``raw'' referee scores $r_A$ and $r_B$ according to Equation~(\ref{eq:lambdabase}).  Next, I need to discretize them.  As Table~\ref{tbl:frequency} showed, the frequency of categories in the recommendations varied with the venue.  For example, referees gave the highest grade in about 20\% of the JET submissions, but only in about 3.8\% of the \sfscav\ submissions.  To match each venue's recommendations, I translate the raw referee signals $r_R$ into the observed frequencies of discrete recommendations.  For example, because the \sfscav\ referees offered 935 ``must accept'' recommendations out of a total of 24,370 recommendations, the highest 935 simulated recommendations for each of the two referees is assigned into the ``must accept'' category.  My discretization procedure can be thought of as an in-sample estimator for the true unobserved frequencies of categories based on the realized frequencies of categories.

The simulated discretized referee-pair recommendation matrix is then calculated from $r_A$ and $r_B$.  If lambda is one (i.e., referees report the reliable characteristic $c_S$), the simulated matrix is diagonal.  If lambda is zero (referees report $c_A$ and $c_B$, respectively), the simulated matrix is (on average) the unconditional probability vector product.

The reported estimate of lambda, $\lambda^*$, is the one that has the least mean-squared equal-weighted probability difference (distance) between its simulated four-by-four referee-recommendation pair matrix and the empirically observed four-by-four matrix.\footnote{This is a simple simulated method of moments (SMM) estimator.  If I minimize the absolute misclassifications in Table~\ref{tbl:lambda}, the lambda inference typically changes by no more than 0.01.  The reported confidence interval is based on the critical lambdas that do not cover the observed distance to that provided by the optimal estimate of lambda $\lambda^*$ within their 90\% confidence intervals.}  The model typically provides a good fit for all venues, with cell-averaged root-mean-squared probability distances between the observed and simulated optimal probability matrix of less than 0.001\%.  This can also be translated into a number of referee recommendations that would need to be changed to perfectly match the observed matrix.  (One disadvantage of this metric is that a change from an REJ to an ACC matters as much as a change from an REJ to a WEAK.  The advantage is the intuitive nature of this metric.)  For example, for the \sfscav, there were 24,370 referee pairs.  If referees had chosen independently ($\lambda=0$), then to match the empirical distribution of recommendations in Table~\ref{tbl:ownother} would require changing 448 recommendations.  If referees had chosen identically ($\lambda=1$), it would require changing 2,458 recommendations.  At the best estimate of $\lambda$, $\lambda^*=0.364$, it would require changing only 16 referee recommendations.  The model can almost (but not quite) provide a sufficient statistic for the information in the paired data.\footnote{The results are similar for other distance statistics based on the two matrices, such as the mean-squared error in the diagonal agreement vector, the agreement only among the best recommendation, or the number of recommendations that would need to be changed to achieve perfect fit.  The normal distribution on the three $c$ characteristics fits the data modestly better than either a Cauchy or a uniform distribution.}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Estimates of consensus and lambda}

\begin{table}

\caption{Paired referee correlations and $\lambda$ parameter estimates} \label{tbl:lambda}

\begin{ctabular}{l SlllS  >{\footnotesize}cc>{\footnotesize}c s ccc iiii}
  \toprule
               & \mc3{cS}{\underline{Correlation}} &  \mc3{cs}{\underline{Agreement $\lambda$}} & \mc3c{\underline{Misclassifications}} &  \\[1ex]
  Journal      & Spear & Pear & McFd  & 5\% & Mean & 95\% & 0 & $\lambda^*$ & 1 & Agree? & Top \\
  \midrule      
  ECMTA        &  0.18 &  0.20 & 0.12   & 0.343 & 0.368 & 0.394 &   285 & 12 & 1,539  & 0.378 & 0.408 \\
  IER          &  0.26 &  0.22 & 0.16   & 0.393 & 0.437 & 0.467 &   280 & 10 &   903  & 0.442 & 0.467 \\
  JEEA         &  0.22 &  0.18 & 0.13   & 0.346 & 0.381 & 0.408 &   225 & 15 &   978  & 0.391 & 0.401 \\
  JET          &  0.25 &  0.26 & 0.16   & 0.356 & 0.405 & 0.445 &    65 & 13 &   313   & 0.395 & 0.410 \\
  QJE          &  0.19 &  0.15 & 0.12   & 0.328 & 0.352 & 0.375 &   261 & 12 & 1,585 & 0.352 & 0.362 \\
  RAND         &  0.25 &  0.24 & 0.15   & 0.368 & 0.424 & 0.465 &    56 &  5 &   214 & 0.424 & 0.439 \\
  \midrule   
  \mc1r{6 economics journals}
               &  0.25 &  0.20 &        & 0.370 & 0.388 & 0.403 & 1,233 & 23 & 5,531 & 0.393 & 0.418 \\
  \\
  RFS          &  0.14 &   0.14 & 0.10 & 0.236 & 0.309 & 0.365 & 32 & 5 & 313 & 0.314 & 0.339 \\
  JF (not EE)  &  0.12 &   0.15 & 0.09 & 0.229 & 0.319 & 0.376 & 25 & 4 & 197 & 0.319 & 0.319\\
  \midrule
  \mc1r{2 finance journals}
               & 0.15 & 0.14 &         & 0.267 & 0.324 & 0.371 & 68 & 7 & 510 & 0.329 & 0.359 \\
  \\
  \sfscav      & 0.21 & 0.20 & 0.13    & 0.344 & 0.364 & 0.378 & 448 & 16 & 2,458 & 0.359 & 0.354 \\
  \midrule
  \mc1r{All 9 venues}
               & 0.27 & 0.22 &         & 0.385 & 0.405 & 0.422 & 2,235 & 49 & 8,497 & 0.415 & 0.445 \\
  \bottomrule
\end{ctabular}

(Continued)

\end{table}
\begin{table}[e]

(Table~\ref{tbl:editor} continued.)

\bigskip

\explain{All statistics are based on referee-pair observations on the same submission.  Except for the \textit{Journal of Finance}, where referees could only recommend one of three dispositions, the four recommendation categories allowed for 16 possible pair values.  A submission with more than two referees created multiple pairs.  (This matters primarily in the \sfscav, where the average paper had about five referees.)   The correlation columns contain the Spearman (Spear) and Pearson (Pear) correlation coefficients if a distance of one is assigned between the four categories (REJ, WEAK, R\&R, ACC).  They also contain the square root of the McFadden (McF) $R^2$ from a probit explaining the recommendation of one referee with dummies spanning the recommendation of the other referee.  Lambda is a summary statistic that estimates a weight that referees place on the reliable characteristic if referees report a discretized net signal of $\lambda\times c_S + (1-\lambda)\times c_R$, where $c_S$ is the shared reliable signal, $c_R$ is the referee's own signal, and the two signals are orthogonal unit-normals.  $\lambda$ minimizes the equal-weighted probability deviation in the simulated versus the empirical squared probability matrix.  The last three columns show the number of recommendations that would have to be changed to achieve perfect fit. (A lambda $\lambda^+$ that would minimize these misclassifications is very similar to $\lambda^*$ but not identical.) }

\medskip

\textbf{Interpretation:} Referees at ECMTA, the QJE, the RFS, and the JF had lambda statistics between 0.30 and 0.37.  Referees at other economics journals had lambda statistics around 0.40.

\source{pair-stats.R and matrix2lambda }

\bigskip
\end{table}


The left columns in Table~\tref{tbl:lambda} show the pairwise referee correlations and inferred lambda parameters if a distance of one is assigned between the four categories (REJ,WEAK,R\&R,ACC).  The Spearman and Pearson correlation coefficients are around 20\% to 25\% for the six economics journals, 14\% to 15\% for the two finance journals, and 20\% for the \sfscav.  The square root of the McFadden $R^2$ from an ordered probit predicting one referee's decision with that of the other (either as a single value or a set of dummies) is about half the size of the more common correlation coefficients.

The right columns show the estimates of lambda.  They seem high relative to the correlation coefficients because even relatively strong referee agreement results only in modest extra consensus.\footnote{For intuition, if referees either observed or did not observe the true characteristic of the submission with probability $\lambda$, then consensus above the expected frequency under the null would occur only when both referees get to see the true characteristic.  One random and one correct draw do not yield more consensus.  Thus, the consensus would increase with probability $\lambda^2$.  If $\lambda=0.5$, then each referee has a 50\% probability of observing the reliable characteristic.  If only one referee observes the reliable quality and the other observes a random draw, the consensus is the same as if both referees observe a random draw.  Thus, ``excess consensus'' occurs only $50\%\times\%=25\%$ of the time.}  The lambda estimate is 0.39 for the six economics journals, 0.32 for the two finance journals,\footnote{For the RFS, the estimates are very similar when I focus only on referees that were solicited on the same day.}  and 0.36 for the \sfscav.  The 5\% to 95\% ranges that cannot be rejected are narrow, because the number of observations is large and the model has only 10 probabilities to fit with three cutoff and one lambda parameters.  The last three columns show the number of reports that would have to be changed to fit the observed empirical pairwise matrix perfectly.  Over all nine venues, the model would have fit perfectly if 49 referee recommendations (out of 87,114) were changed.  This is much better than the 2,235 recommendations that would have to be changed if there was no consensus ($\lambda=0$) or the 8,497 recommendations if there was perfect consensus ($\lambda=1$).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Counterfactuals}

The distribution of recommendations (Table~\ref{tbl:frequency}) and the estimates of lambda (Table~\ref{tbl:lambda}) make it possible to consider thought experiments (albeit assuming that the pairwise recommendations are representative of recommendations for all submitted papers, including single-refereed submissions and zero-referee desk rejects).  I estimate the referee recommendations that a submitting author and the editors can expect for submissions with a given reliable common characteristic (not true quality).  To do so, I consider two values for lambda, $\lambda\approx0.33$ (roughly representative of ECMTA, the QJE, the RFS, and the JF), and $\lambda\approx0.40$ (roughly representative of the full set of venues, including the IER, JEEA, JET, and RAND).  Similarly, I use recommendation category frequencies in line with the empirical data.  They are not intended to match any one venue perfectly.  In these hypotheticals, I presume that the editor has knowledge of the raw referee recommendations, and not just the discretized recommendations.


\subsubsection{Hypothetical referee recommendation probabilities}

\begin{figure}[e]

\bigskip

\caption{Conditional probabilities of referee recommendations based on estimated $\lambda$} \label{fig:fig1-probr}

\begin{ctabular}{cc}
  \underline{``Finance''} & \underline{``Overall''} \\
  \footnotesize $\lambda=0.33$
  &
  \footnotesize $\lambda=0.40$
  \\
  \includegraphics[width=0.48\textwidth]{fig1-1000dots-fin} 
  &
  \includegraphics[width=0.48\textwidth]{fig1-1000dots-all} 
  \\
  \footnotesize $\lambda=0.33, p=(0.69,0.18,0.07,0.06)$
  &
  \footnotesize $\lambda=0.40, p=(0.56,0.18,0.07,0.09)$
  \\
  \includegraphics[width=0.48\textwidth]{fig1-area-fin}
  &
  \includegraphics[width=0.48\textwidth]{fig1-area-all}
  \\[-1em]
   \mc2c{\footnotesize\underline{50\% Probability Threshold on Reported Decision}} \\
   \scriptsize
   \begin{tabular}{c@{ }c@{ }c}
     (Top 31\%) & (Top 13\%) & (Top 6\%) \\
     WEAK & R\&R & ACC \\
     \midrule
     87th & 99th   & 100th \\
   \end{tabular}
   &
   \scriptsize
   \begin{tabular}{c@{ }c@{ }c}
     (Top 44\%) &  (Top 26\%) &  (Top 9\%) \\
     WEAK & R\&R & ACC \\
     \midrule
     61th & 87th   & 99th \\
   \end{tabular}
\end{ctabular}

\bigskip

(Continued)

\end{figure}

\addtocounter{figure}{-1}

\begin{figure}[e]

\bigskip

\caption{(Continued) Conditional Probabilities of Referee Recommendations Based on Estimated $\lambda$}

\vspace*{0.1\textheight}

\explain{The left side (right side) assumes a lambda of 0.33 (0.40).  The top graph plots the typical relation between the true reliable characteristic $c_S$ percentile rank and the referee-reported percentile rank $r_R$ for 1,000 hypothetical submissions.  (The ``reliable'' characteristic $c_S$ is \textit{not} a measure of the true scientific quality of the submission.)  The horizontal lines in the top graphs indicate the frequencies of the four categories in the data, which are also inputs into  the bottom graphs.  These probabilities of REJ, WEAK, R\&R, and ACC are 0.69, 0.18, 0.07, and 0.06 (0.56, 0.18, 0.07, and 0.09), respectively.  The lower graphs show the probabilities of different categorical recommendations.  The numerical 50\% horizontal crossings are listed below.  For example, in the left bottom graph, a submission that was truly ranked at the 87th percentile had a 50\% probability of receiving a WEAK referee recommendation or better.  To expect an R\&R with greater than 50\% probability required a submission with a true $c_S$ characteristic rank of 99\% or better.}

\bigskip

\vspace*{0.25\textheight}

\source{fig1.R}

\end{figure}

Figure~\fref{fig:fig1-probr} plots the recommendations of a single referee $r_R$ as a function of the true reliable characteristic of the paper, $c_s$.  The top graphs show sample random draws that are intended to give a visual impression for how the reliable $c_S$ ranks of 1,000 submissions tend to map into referee reports $r_R$.  The recommendations are neither random nor strongly associated with the true reliable characteristic.  The bottom graphs show the probabilities.  The 50\% horizontal line can be used to assess the asymptotic consensus of referees.     If the probability of receiving a given report is below 50\%, then as the number of referees increases (goes to infinity), the probability that a majority of referees will recommend it decreases with the number of referees (goes to zero).  The graph and table below shows that if $\lambda=0.33$, the top one percentile paper (the 10th best submission out of 1,000) should expect the majority of its referee reports not to recommend a (strong) revise-and-resubmit or accept, even though such recommendations constitute 13\% of the referee reports.  Another striking observation is that the more referees the editor consults for the top two percentile paper, the lower is the probability that its \textit{average} review will be an R\&R or better.

%%%%%%%%%%%%%%%%
\subsubsection{Hypothetical journal decisions}

\begin{figure}[e]

\bigskip

\caption{Cumulative distribution of $c_s$ among the top-10\% $r_R$ ranked submissions} \label{fig:fig2-cdf}

\begin{ctabular}{cc}
  \underline{``Finance''} & \underline{``Overall''} \\
  \footnotesize $\lambda=0.33$ & \footnotesize $\lambda=0.40$ \\
   \includegraphics[width=0.48\textwidth]{fig2-cdf-fin} & \includegraphics[width=0.48\textwidth]{fig2-cdf-all} \\
\end{ctabular}

\explain{Assume that journals accept the submissions that gather the top-10\% highest average referee reports (mean $r_R$).  These graphs then plot the cumulative probability density of the accepted submissions.  The cross-hatch is the mean.  For example, if $\lambda=0.33$, if a journal evaluates each submission with three referees, then about 36\% of the accepted papers have a true common characteristic $c_S$ that is below the 80th percentile.  Because the common characteristic $c_S$ is normally distributed, the 80th percentile corresponds to a true common characteristic $Z=0.8416$.}

\bigskip

\source{fig2-cdf.R}

\end{figure}

Assume now, in addition, that a journal wants to continue with the 10\% of its submissions that have the highest reliable characteristics $c_S$ (not necessarily the best papers, but those that would elicit the most referee approval if repeated many times),  that the journal considers each referee to be equally informed, and that the editor has no signal of her own.   (In Section~\ref{subsect:editorial-decisions}, editors will be assumed to have their own signals, too.) In this case, the editor would continue with the 10\% of the submissions that have the highest ranked mean recommendations, $\bar{r}_R$. 

\newcommand{\longtext}[8]{With one referee,  the average true percentile rank of continuing papers on the common characteristic is #1, about #2 of the continuations are from the bottom half, and #3 of the continuations are misclassified in that their true common characteristic is less than 0.9.  With three referees, the average true percentile rank increases to #4, about #5 rank in the bottom half, and #6 are not truly top-10\%.  Even with ten referees, the average percentile rank is #7, almost no paper from the bottom half is continued, and about #8 of the accepted submissions are not top-10\%.}

Figure~\fref{fig:fig2-cdf} illustrates the journal's selection:
\begin{description}
\item[Lambda of 0.33:] \longtext{0.72}{20\%}{70\%}{0.82}{8\%}{57\%}{89\%}{40\%}
\item[Lambda of 0.4:] \longtext{0.77}{13\%}{65\%}{0.86}{3\%}{48\%}{92\%}{30\%}
\end{description}
Referee reports are informative but not very reliable.

\medskip

% p=data.frame(n=1:length(sA), f=c(rep(0,NH),rep(1,NH)), s=c(sA[1:NH],sBCD[(NH+1):N]))
% p$r= prank(p$s)
% iaw.summary(subset(p, p$r>0.9))

It is not obvious what the decision rule should be if there is variation in the number of referees that are assigned to submissions.  For example, assume that all papers are drawn from the same common characteristics distribution and that lambda is 0.33.  If one half of the submissions randomly receive one referee report, while the other half receives two referee reports, then two-thirds of the accepted (i.e., top-10\% ranked) papers will come from the first (single-referee) set.  This is because it is more common to draw a far-positive outlier with fewer referee reports.  The problem is isomorphic to the discrimination-inference problem in \citeasnoun{cornell-welch:2006}.\label{pg:cornell-welch}\footnote{See \citeasnoun{cornell-welch:2006} for more detail on the model.  The problem is worse with discrete categories.  Consider an example in which a journal consults two referees for some papers and one referee for others.  If a journal is so selective that it only publishes papers with the highest implied inference, it may only publish papers with two (good) reviews.  One-review papers cannot reach as high an inference.  If the journal publishes more papers, so that it accepts papers with one (good) review, the probability that a two-review paper is published is less than the probability that a one-review paper is published.  (The comparison is between getting two-out-of-two good reviews and one-out-of-one good reviews.)}  The decision problem does not become easier if editors follow a known deterministic rule that assigns multiple referees to some but not other submissions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Editorial decisons and referee reports}\label{subsect:editorial-decisions}

One journal editor remarked that he ignored the categorical referee ratings and focused only on the detailed reports.  It is therefore an interesting question whether the categorical referee reports are generally congruous with the editorial decisions.  This journal provided me with a second data set to consider its editorial decision statistics.\footnote{Below, I report statistics based on referee reports that were solicited at the time of the original paper submission, but the results are almost the same if I also include reports solicited later (sequentially, i.e., after the first reports came in).}

\begin{table}

\caption{Editorial congruance with referees and editorial inference weights, given $\lambda=0.33$} \label{tbl:editor}

\begin{ctabular}{c i r s r r  r r s i@{}i  r r  r r }
         \mc3l{\underline{Solo referees}} & \mc4c{\underline{Editor rejects (83\%)}} &&&\mc4c{\underline{Editor R\&R (17\%)}} \\[1ex]
         &        &        & \mc2c{Empirical}  &  \mc2c{If $w_E=$} &  & & \mc2c{Empirical}     &  \mc2c{If $w_E=$} \\[-0.6ex]
Referees & Editor & Papers & \#Obs & Freq & $0.5$ & $0.35$ & Referees & Editor & \#Obs & Freq & $0.5$ & $0.35$ \\
\midrule
         REJ &   REJ & 2,641 &2,529 &0.957 &0.928 & 0.961 &         REJ &    RR & 112 &0.042 &0.071 & 0.038 \\
        WEAK &   REJ & 236 & 117 &0.495 &0.619 & 0.577 &        WEAK &    RR & 119 &0.504 &0.380 & 0.422 \\
        R\&R &   REJ & 301 & 52 &0.172 &0.386 & 0.221 &        R\&R  &    RR & 249 &0.827 &0.613 & 0.778 \\
         ACC &   REJ & 94 &  10 &0.106 &0.120 & 0.014 &         ACC &    RR &  84 &0.893 &0.879 & 0.985 \\
\midrule
\\
         \mc3l{\underline{Dual referees}} & \mc4c{\underline{Editor rejects (80\%)}} &&&\mc4c{\underline{Editor R\&R (20\%)}} \\[1ex]
         &        &        & \mc2c{Empirical}  &  \mc2c{If $w_E=$} &  & & \mc2c{Empirical}     &  \mc2c{If $w_E=$} \\[-0.6ex]
Referees & Editor & Papers & \#Obs & Freq & $0.5$ & $0.35$ & Referees & Editor & \#Obs & Freq & $0.5$ & $0.35$ \\
\midrule
        \mc8{l}{\underline{Agreeing referees --- negative}} \\
       REJ,REJ &   REJ & 817 & 794 &0.971 &0.964 & 0.982 &       REJ,REJ &    RR &  23 &0.028 &0.035 & 0.017 \\[1ex]

       \mc7l{\underline{Agreeing referees --- positive}} \\
     R\&R,WEAK &   REJ & 28 &  4 &0.142 &0.296 & 0.187 &     R\&R,WEAK &    RR &  24 &0.857 &0.703 & 0.812 \\
   R\&R,R\&R  &   REJ & 25 &  1 &0.040 &0.130 & 0.023 &   R\&R,R\&R  &    RR &  24 &0.960 &0.869 & 0.976 \\
      ACC,WEAK &   REJ & 11 &  1 &0.090 &0.247 & 0.174 &      ACC,WEAK &    RR &  10 &0.909 &0.752 & 0.825 \\
    ACC,R\&R  &   REJ & 23 &  0 & 0.000 &0.049 & 0.011 &    ACC,R\&R  &    RR &  23 & 1.000 &0.950 & 0.988 \\
       ACC,ACC &   REJ & 3 &  0 & 0.000 & 0.000 & 0.000  &       ACC,ACC &    RR &   3 & 1.000 &1.000 & 1.000 \\[1ex]

       \mc9l{\underline{Ambivalent referees}} \\
     WEAK,WEAK &   REJ & 21 & 12 &0.571 &0.518 & 0.400 &     WEAK,WEAK &    RR &   9 &0.428 &0.481 & 0.599 \\[1ex]

      REJ,WEAK &   REJ & 195 & 146 &0.748 &0.751 & 0.715 &      REJ,WEAK &    RR &  49 &0.251 &0.248 & 0.284 \\
    REJ,R\&R  &   REJ & 197 & 121 &0.614 &0.559 & 0.507 &    REJ,R\&R  &    RR &  76 &0.385 &0.440 & 0.492 \\
       ACC,REJ &   REJ &  66 &  28 &0.424 &0.440 & 0.462 &       ACC,REJ &    RR &  38 &0.575 &0.559 & 0.537 \\

\bottomrule
\end{ctabular}

(Continued)

\end{table}
\begin{table}[e]

(Table~\thetable\ continued.)
\bigskip


\explain{This table shows the expected frequency of editorial decisions and referee recommendations.  Both referees and editors are assumed to see a final signal $r$ that has a weight of $\lambda=0.33$ on the common characteristic $c_S$ and weight 0.67 on their own idiosyncratic signals ($c_R$ or $c_E$).  The editor continued with 18.1\% of the submissions.  This means the editor would accept the 18.1\% of submissions for which her $c_S$ inference was highest.  An editor may put less weight on her own signals $w_E$ if she believes it to be less informative.  If $w_E=0.5$, then with one referee, an optimizing editor would place as much weight on the referees' signals as on her own signal.  More generally, with one referee (two referees) she would base her decision on the rank of $[w_E\times r_E +(1-w_E)\times r_A]$ ($[w_E\times r_E +(1-w_E)\times (r_A+r_B)]/[w_E+(1-w_E)\times2]$).  In the ``EW'' column, the editor places equal weight $w_E=0.5$ on her own signal as on a referee's signal.  The behavior of editors seems to match a $w_E\approx0.35$; that is, the editors placed about twice as much weight on referees' signals as on their own.  The resulting probabilities of rejection and continuation are in the $0.35$ columns.}

\interpret{The empirical evidence suggests that it was rare for the editors of this journal to override the referees' recommendations.}

\source{ee/sfs-editor/editorial-decisions.Rout}


\bigskip
\end{table}

In the following analysis, I consider only submissions in which editors consulted one or two referees. (Three-referee submissions were too rare.)  In the data, the referees were more favorably disposed toward dual-refereed papers, hinting that editors deliberately assigned two referees to papers for which they had more favorable priors:
\begin{ctabular}{l s c s *{4}{c} s c }
               & Papers & REJ & WEAK & R\&R & ACC & Editor continued \\
               \midrule
  Solo referee & 3,272 & 80.7\% & 7.2\% & 9.2\% & 2.9\%  & 17.2\% \\
  Dual referee & 1,386 & 75.5\% & 10.0\% & 10.8\% & 3.8\%  & 20.1\% \\
%   All          & 4,658 & 78.3\% & 8.5\% & 10.0\% & 3.3\%  & 18.1\% \\
\end{ctabular}
In the 1,386 dual-referee situations, if referees had had perfect agreement, there should have been $0.755\times1,386=1,046$ REJ,REJ pairs.  If referees had been uncorrelated, there should have been $0.8072^2\times1,386\approx790$ REJ,REJ pairs.  Table~\tref{tbl:editor} shows that there were 817 REJ,REJ pairs in the data, in line with my earlier findings of modest agreement among referees.

Unlike most other results in my paper, the main units of analysis in this section are not referee pairs but editor-referee pairs and editor-referee-referee triplets.  Table~\tref{tbl:editor} shows that the decisions of editors aligned closely but not perfectly with the recommendations of the referees.  Solo referees recommended 2,641 rejections, 236 weak revisions, 301 strong revisions, and 94 accepts.  When a solo referee recommended an REJ, the editors rejected the paper in 2,529 out of 2,643 cases (95.7\%) and issued an R\&R in the other 112 cases (4.3\%).  When two referees recommended rejection, the editor rejected in 794 out of 817 cases (97.2\%) and continued in 23 cases (2.8\%).  In the 3,916 cases in which at least one out of two referees recommended rejection, the editor rejected the paper in 3,618 cases (92.4\%) and continued in 298 cases (7.6\%).

Given that that only 18.1\% of the submissions were continued, it is not surprising that most negative referee reports result in rejections.  This would be the case even if editors believed that they had independent information that was as relevant as that of the referee: after one negative referee report, even very positive editors' signals would rarely be sufficient to rank a submission among the top 18\%.

However, the evidence suggests editors followed referees even more than would have been expected if they had had their own signal of equal quality and placed equal weight on it.  This estimate comes from a model analogous to the paired-referee model in which editors place some weight on their own private signal and some weight on the referees' reports.  In this case, the editorial decision would be based on
\[ \text{Final Editor's $c_S$ Inference} =   \left\{
  \begin{array}{csc}
    w_E\times r_E + (1-w_E)\times r_A & \text{with one referee} \\[1ex]
    \frac{w_E\times r_E + (1-w_E)\times (r_A+r_B)}{w_E+(1-w_E)\times2} & \text{with two referees} \\
  \end{array}\right.
\]
where $w_E$ is the weight to be estimated; $r_E$, $r_A$, and $r_B$ are generated as before (each with a lambda of 1/3, i.e., $r_R= \lambda c_S + (1-\lambda) c_R$ and $R \in (A,B,E)$); and editors continue with the top 18.1\% of submissions based on their best final inference.  When simulated with $w_E=1/2$, which would have editors weigh their own signal as strongly as those of referees, Table~\ref{tbl:editor} shows that this model predicts less consensus between editor and referee than was observed.   For example, if editors had weighed their own signal and one REJ referee recommendation equally, they should have followed the referee only 92.8\% and not the observed 95.7\% of the time.   Similarly, if they had weighted their own signal and one WEAK recommendation equally, they should have continued only 38.0\% and not 50.4\% of the time.  If they had weighted their own signal and one R\&R recommendation equally, they should have continued only 61.3\% and not 82.7\% of the time.    Similarly, with two referees, editors tended to follow referees more often than would have been observed with equal-weight signals.  However, excess following was not always the case.  For example, editors did follow ACC recommendations roughly as would have been expected with equal-quality signals.

 % This is consistent with the view that the editors handled hundreds of papers and were presumably consulting referees to learn whether they should accept or reject submissions.

%% footnoted armstrong is from [page 11]

Overall, the behavior of editors is better explained by a model in which editors have less information or place less weight on their own information.  The best fit is a model with $w_E\approx0.35$.  It predicts that only 3.8\% of submissions with one single rejection recommendation would have been continued (overriding the referee), which actually happened in 4.2\% of submissions; that editors should have continued with papers with one WEAK referee recommendation 42.2\% of the time; and that editors should have continued with papers with one R\&R recommendation 77.8\% of the time.\footnote{My findings are in line with speculations about the editorial process in other fields. \citeasnoun{armstrong:1997} describes that editors in many disciplines seem to behave similarly.  Editors concerned about fairness tend to treat referees' (categorical) recommendations as votes.  This means that the most prestigious journals often do not end up publishing papers that receive mixed reviews. \citeasnoun{kupfersmid-wonderly:1994} summarize four empirical studies that led them to conclude that papers with mixed reviews are unlikely to be published.  \citeasnoun{munley-sharkin-gelso:1988} and \citeasnoun{marsh-ball:1989} find that mixed-review papers have a low probability of being published.  \citeasnoun{bakanic-mcphail-simon:1990} find that a single negative recommendation at the \textit{American Sociological Review} (ASR) from 1977 to 1982 often resulted in rejection.  Similarly, \citeasnoun{bakanic-mcphail-simon:1987}, \citeasnoun{beyer-chanove-fox:1995}, and (\citealt[Table~8]{blank:1991}) find that editors' decisions are highly predictable from the average categorical ratings of the reviewers. Finally, \citeasnoun{simon-bakanic-mcphail:1986} find that the ASR agreed with authors on only 13\% of appeals.  \citeasnoun{bayar-chemmanur:2012} produce an academic model of the editorial decision.  However, they also consider referee-specific biases that an editor can undo.}

It is an interesting question whether the editors at this journal preferred papers that elicit strong disagreement to papers that elicit broader but lukewarm support.  There are too few observations in Table~\ref{tbl:editor} to generalize, but it hints that editors preferred the latter: they continued with 85.7\% of the papers that received two WEAK endorsements, but only with 57.5\% of the papers that received one ACC and one REJ endorsement.

The journal also permitted me to look at whether editors prefer multiple referee reports (at least for some papers) because they may reduce the probability that a rejection will be appealed.  That is, editors may well be aware of the difficult decision problems across papers with multiple referees, but deem it less important than reducing the number of appeals.  The evidence does not suggest that this was the case.  Papers with one referee were appealed 1.8\% of the time.  Papers with two referees were appealed 1.4\% of the time.  Papers with three referees were appealed 4.4\% of the time.  And papers with four or more referees were appealed 25\% of the time.  Thus, multiple referees did not serve, on average, as a ``firewall'' against appeals.\footnote{Appeals are rare, which means that further inference is difficult.  Nevertheless, it was surprising that for papers with more than one referee, there was no strong association between the decision to appeal and the average referee report.  That is, papers that were rejected by all referees were about as often appealed as papers that had at least one favorable referee.  (It was rare that single-refereed papers with only one very positive referee were rejected.)}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Identifying Referee-Specific Factors} \label{sect:reffactors}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The previous section showed that referee-specific views are more important than common views in explaining referee recommendations, but it could not identify the sources of disagreement.  The heterogeneity in recommendations could be based on paper-specific attributes.   It is interesting to ask whether the heterogeneity in recommendations can be traced back to specific referee characteristics rather than submission characteristics.  Unfortunately, the data about referees is limited---for example, I have no information on such referee attributes as their motives or views (\citealt{bayar-chemmanur:2012}).  However, I do have some limited data from the \sfscav\ on how referees evaluated themselves, how often they had opposite perspectives, and how generous they were to other submissions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Self-reported referee characteristics}

In the \sfscav, 460 referees answered questions about what kind of referee they considered themselves to be.

Of these 460 referees, 33 referees with 168 recommendations considered themselves generally to be ``more critical,'' 91 referees with 428 recommendations considered themselves to be ``less critical.''   The 168 self-declared more-critical referee reports suggested that only 3.6\% of the submissions that were assigned to them fell into the ``must-accept'' (MA) category, and that only 24\% fell into the ``must-accept'' or ``should-accept'' categories (MA/SA).  The equivalent figures were 4.0\% and 27\% for 1,636 referee reports that considered themselves typical, and 4.4\% and 40\% for 428 referee reports that considered themselves less critical.\footnote{Unfortunately, I had asked referees to rate themselves when they returned their reviews, not at the time when they specified their expertise.  Thus, it is possible that their self-assessments were contaminated by the specific papers they refereed.}

Referees who considered themselves ``more discriminating'' provided 180 referee reports.  These referees suggested that only 3.3\% of papers were MUST ACCEPT and only 24\% were MUST ACCEPT or SHOULD ACCEPT.  The same figures were 3.9\% and 29\% for referees that considered themselves typical, and 4.8\% and 32\% for referees that considered themselves less discriminating.  The evidence suggests that ``discriminating referees'' were more negative.  Not reported, separate lambda estimations in the most-discriminating referee group and regression-interaction specifications suggest that when both referees considered themselves to be more discriminating, they tended to agree less with one another.  However, the difference is not statistically or economically significant.  Thus, it is safe to say that the evidence suggests that more discriminating referees were no more in agreement than other referees, but not that they were less in agreement.

The Appendix briefly describes some additional referee-specific attributes associated with their recommendations.  For example, there is evidence that younger faculty (having receiver their PhDs more recently) and faculty from more prestigious universities were more negative.

In sum, the referees themselves proclaimed that their identities would influence their recommendations, above and beyond the submission itself, and these claims were backed up by data on their behavior.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Referee-specific fixed effects}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To measure the magnitude of the extent to which scale effects (differences in referees' generosities) can help explain referee recommendations,  it would be ideal to predict each referee's recommendation $r_R$ with two variables: the paper's true reliable characteristic $c_S$ and the subjective generosity of referee $R$, which can be viewed as a mean shift $m_R$ of her signal.  Unfortunately, neither is observed.  Therefore, my analysis explains referee recommendations with two noisy proxies: the recommendation of other referees on the same paper (which presumably reflects the reliable component of referee judgments about paper characteristics) and the referees' own mean recommendations on other unrelated papers (which presumably reflects the mean scales of the referees).

%\subsubsection{\sfscav}

\newcommand{\mcu}[3]{\mc{#1}{#2}{\underline{#3}}}

\begin{table}%[e]

  \bigskip

  \caption{Predicting \sfscav\ referee recommendations} \label{tbl:yikes-ols}

  \begin{smctabular}{cllc lc ll}
    \mc3c{ Method } & $R^2$ &  & Intercept & \shortstack{Other referees' rec on\\same refereed paper} & \shortstack{Referee's own mean\\on other papers} \\
    \midrule
    \mcu8l{24,354 pairs} \\[1ex]
    &A    & OLS& 7.1\% & Coef   & --0.0780 & 0.2196 (0.220) & 0.2745 (0.157)\\
    &\;.1   & OLS& 4.7\% & Coef   & --0.112 & 0.2163 (0.216) &\\
    &\;.2   & OLS& 2.3\% & Coef   & --0.110 & & 0.2663 (0.152) \\
    &&&&   \mc1l{\;(se)}  &  && (0.011) \\[1ex] %% se of intercept is 0.006
    &&&&  \mc4c{ $\Rightarrow r_i = 0.498\times c_A \qquad + \; 0.306\times c_S \qquad\qquad\qquad + \; 0.196\times m_i $ }\\
    &&&&  \mc4c{ \hspace*{8.9em} (0.012) \hspace*{8em} (0.014)} \\[2ex]


    \mcu8l{12,175 nonreversed pairs} \\[1ex]
    &B    & OLS & 7.9\% & Coef   & --0.0784 & 0.2253 (0.220) & 0.3089 (0.180) \\
    &&&   & se &  & (0.009) & (0.016) \\[1ex]

    &C & Probit & 3.0\% / 8.1\% & Coef & --1.6/--0.3/0.5/1.7 & 0.246 & 0.349 \\
    & &  & & se & & (0.010) & (0.016) \\[1ex]
    &D & Logit & 2.9\% / 7.8\% & Coef & --2.7/--0.6/0.9/3.1   & 0.425  & 0.591 \\
    & & & & se & & (0.017) & (0.029) \\[3ex]

    &E    & OLS & 35.1\% & Coef   & \mc2c{Full Fixed (Paper) Effects} & 0.3440 (0.201) \\
    &&&   & se  &  &  & (0.014) \\[2ex]

    \mcu8l{2,904 report means \source{eeyikes-sfscav}} \\[1ex]
     &F    & OLS & 17.4\% & Coef   & --0.0030 & 0.6646 (0.386) & 0.3114 (0.176) \\
     &&&   & se  &  & (0.028) & (0.031) \\[3ex]

    \mcu8l{1,343 Non-repeated referees (1+2,3+4,...)} \\[1ex]
    &G    & OLS & 7.7\% & Coef   & --0.0615 & 0.2190 (0.214) & 0.3072 (0.182) \\
     && &  & se  &  & (0.026) & (0.046) \\[3ex]

    \bottomrule
  \end{smctabular}

  \medskip

(Continued)

\end{table}
\begin{table}[e]

(Table~\thetable\ continued.)

\bigskip

  \explain{The predicted referee recommendations are for 2,903 referee recommendations from the \sfscav\ with data for both variables.  For the OLS regressions, the standardized coefficient is in parentheses to the right of the plain coefficient.  It multiplies the coefficient by the standard  deviation of X over the standard deviation of Y.  The OLS standard errors are White-heteroscedasticity adjusted.  The Probit and Logit $R^2$ are the McFadden / Maximum-Likelihood $R^2$s, respectively.  The equation below the A.2 model fits a model equivalent to Equation~(\ref{eq:lambdabase}) and Table~\ref{tbl:lambda}, but adds an additional term for the referee mean $m_R$ (also drawn from a unit-normal) that matches the OLS coefficients of model A.}

  % Not reported---if only the first two other refs are compared, the accuracy
  % declines to a std coef of around 0.26 from 0.39.

  \medskip

  \interpret{The mean of recommendations of unrelated papers by the same referee is an important marginal predictor of this referee's recommendation, although the $m_R$ coefficient suggests that it is weaker than the effect of other referees' opinions on the same paper $c_S$.  The statistical significance and economic meaning of the coefficients are largely unaffected by method.}

  \source{sfscavalcade/yikes-sfscav.R: 24,351 df (pairs). seems to be ee/yikes.R now}

  \bigskip
\end{table}

The analysis is most intuitive if we first estimate referee recommendations according to an admittedly arbitrary scale.  For the \sfscav, I coded recommendations as --2 (MR), --1 (SR), 0 (NR), 1 (SA), and 2 (MA).\footnote{Different numerical values for the categories, combinations of categories, and full set of X-dummies for different categories yield very similar results.   The adjusted $R^2$ in the OLS regression is lower if I factor the other referees' recommendations into intercepts to avoid scaling.  The coefficient ordering remains sensible, though.  The coefficient on the referee's own mean on other papers is not affected.}  Table~\tref{tbl:yikes-ols} shows the estimation results using different specifications.  Specification~(A) uses all referee pairs and predicts the recommendations of one referee with that of the other, using simple ordinary least squares (OLS) regression.  The standardized OLS coefficients (in parentheses) that express both X and Y in terms of standard-deviation normalized quantities make it easy to interpret economic meaning.  In~(A), the standardized coefficient for the other referee's recommendations for a given paper (the ``submission mean,'' the proxy for $c_S$) is 0.220.  The standardized coefficient on this referee's mean recommendations on other papers (not including the current paper---i.e., a measure of the ``referee generosity''---the proxy for $m_R$) is 0.157.  This suggests that the common characteristic of the paper embodied in the submission mean is more important than the referee's mean (on other papers), but both are important.  Because the correlation between the two independent variables is~--2\%, specifications A.1 and A.2 show that the two variables have almost the same coefficients by themselves.

As noted, both $c_S$ and $m_R$ are unobserved.  A simulated method of moments estimation of the coefficients on a summary model, equivalent to Equation~(\ref{eq:lambdabase}) and Table~\ref{tbl:lambda} but adding an additional term for the referee mean (also drawn from a unit-normal) that matches the OLS coefficients of model~A ($r_R = \lambda_1\times c_S + \lambda_2\times m_R + (1-\lambda_1-\lambda_2)\times c_R$), can adjust both for noise caused by the discreteness in observations, and for the fact that the proxies are noisy.  Such an estimation suggests that the observed coefficients would have obtained if referees had placed $\lambda_1\approx30.6\%$ weight on the common signal $c_S$, $\lambda_2\approx19.6\%$ weight on their own referee-mean $m_R$, and 49.8\% weight on other referee-specific signals $c_R$.

Variations in specification, (B) through (G), show that the inference is robust.  The coefficients are similar  using only non-reversed pairs (i.e., if for a given paper, referee~1 is used to predict referee~2's recommendation, I do not predict referee~2's recommendation with referee~1), using a probit, using a logit, using only unique paper-referee combinations (i.e., for a given paper, I use referee pair 1-2, then 3-4, then 5-6, etc.), or using the mean report of other referees instead of the actual report.  With thousands of observations, the coefficients are always statistically significantly different from zero or one.



%\subsubsection{All Venues}

\begin{table}

\bigskip

\caption{Consensus and referee mean effects from different venues} \label{tbl:eeyikes}

\begin{ctabular}{c S c}
  Economics journals (all EE) & Finance venues \\[1em]
\imagetop{
\begin{tabular}{l S rr }
  \toprule
                              &  Other ref(s) & Own ref \\
  Journal                     & same ppr & other ppr(s) \\
  \midrule
  ECMTA                        &    0.20 & 0.19 \\
                               & (0.01) & (0.01) \\[1ex]
  JEEA                         &    0.19 & 0.15 \\
                               & (0.01) & (0.01) \\[1ex]
  JET                          &    0.27 & 0.12 \\
                               & (0.02) & (0.02) \\[1ex]
  QJE                          &    0.15 & 0.32 \\
                               & (0.01) & (0.01) \\[1ex]
  IER                          &    0.24 & 0.18 \\
                               & (0.01) & (0.01) \\[1ex]
  RAND                         &    0.25 & 0.14 \\
                               & (0.02) & (0.03) \\[1ex]
  \midrule
                  Mean         &  0.22  & 0.19  \\[3ex]
  \bottomrule
%  EREH                         &    0.22 & 0.00 \\
%                               & (0.05) & (0.06) \\[1ex]
\end{tabular}}
&
\imagetop{
\begin{tabular}{l  rr }
  \toprule
                               & Other ref(s) & Own ref \\
  Journal                      & same ppr & other ppr(s) \\
  \midrule
%  RoF                          &    0.11 & 0.19 \\
%                               & (0.05) & (0.06) \\[1ex]
  RFS                          &    0.14 & 0.29 \\
                               & (0.02) & (0.02) \\[1ex]
  JF (not EE)                &   0.10 & 0.25 \\ % &   0.0969 & 0.2477 \\
                               & (0.02) & (0.02) \\[4ex]
  \sfscav         & 0.386 & 0.176 \\ % &  0.3859 & 0.1758 \\ %
  \tiny (in Table~\ref{tbl:yikes-ols} under ``2,904 report means'') & (0.02) & (0.02) \\
  \bottomrule
\end{tabular}
}
\end{ctabular}

(Continued)

\end{table}
\begin{table}[e]

(Table~\thetable\ continued.)

\bigskip

\explain{These are the standardized coefficients and their standard errors (in parentheses) from OLS regressions explaining referee recommendations with two independent variables: the response of another referees to the same paper, and the referee's own mean response to other papers not under review.  The latter can be largely viewed as fixed referee mean effects.  The reported coefficients are analogs of the standardized coefficients in Table~\ref{tbl:yikes-ols} under ``2,904 report means.'' }

% The \sfscav\ and JF data are not from Editorial Express.

\bigskip

\textbf{Interpretation:}  Within the six economics journals, referee mean fixed effects are about as important in explaining a referee's recommendation as is a second referee's opinion on the same paper.  Within the two finance journals, referee mean effects are even more (twice as) important.  The \sfscav\ had the most agreement among referees relative to the referee fixed effects. %, perhaps because many more referees flowed into the calculation of the paper mean.

\source{ee/eeyikes.pl\\}
\source{ee/eeyikes-sfscav.R\\}
\source{sfscav/yikes.R\\}
\source{afa-jf/step2a-analysis.Rout}

\bigskip

\end{table}

Table~\tref{tbl:eeyikes} reports the results of OLS regressions analogous to model (F) from Table~\ref{tbl:yikes-ols} for all venues.  The reported coefficients are again standardized to allow the reader to focus on the relative economic significance.  The findings are similar.  The referee's intrinsic generosity was important in all venues.  In fact, at the RFS, the JF, and the QJE (which here seems to follow the finance journal patterns), the OLS coefficient on the own-referee mean was twice as high as the coefficient on the submission mean: to get a good referee report, it was as important to draw an intrinsically generous referee with a high $m_R$ as it was to write a paper whose common reliable component $c_S$ was high.  The relative importance of the two factors mattered only mildly across journals.  The referee draw was relatively less important at the less selective journals in my sample.  In particular, the coefficient on the submission mean at the JEEA, JET, IER, and RAND was higher than the coefficient on the referee mean.  In addition, the one venue in which the coefficient on the ``other referees' mean on the same submission'' was highest relative to the coefficient on the ``referees' means on other papers'' (0.386 versus 0.176) was the \sfscav.  This suggests that the average recommendation at the \sfscav\ reflected relatively more on the paper and relatively less on the referees.  However, this was the case primarily because there were more referee recommendations for each paper.   Table~\ref{tbl:yikes-ols} suggests that under a variety of ``only-one-other-referee'' specifications, the \sfscav\ coefficients were more similar to those observed at the other venues.

In sum, it is safe to conclude that a referee report reflects about as much on this referee's intrinsic generosity as it reflects on what another referee would say about the paper.   If an editor wants to extract the reliable common component $c_S$ from the referee reports, it would seem important to adjust referees' recommendations for their average recommendations on earlier submissions.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Scale differences and preference reversals}

\begin{figure}%[e]

  \bigskip

  \caption{Relative pairwise ordering of shared referee assignments at the \sfscav} \label{fig:pairwise}

  \begin{ctabular}{cc}
    \includegraphics[width=1.0\textwidth]{fig-sharedplot} \\[3ex]
    \raisebox{1em}{\normalsize
    \begin{tabular}{lr}
      Total pairs (duplicated) & 4,416 \\
      Same distance between papers & 1,121 \\
      \qquad Identical recommendations & 456 \\
      \qquad Same difference, but offset & 756 \\
      Incl 0 diffs, same orderings & 2,093 \\
      Excl 0 diffs, same orderings & 972 \\
      Opposite orderings & 702 \\
    \end{tabular}
    }
  \end{ctabular}

  \medskip

(Continued)

\end{figure}

\begin{figure}[e]

(Figure~\thefigure\ continued.)\bigskip

\bigskip

  \explain{This is a two-dimensional representation of a three-dimensional histogram depicting the relative ordering when two referees evaluated the same two papers.  For each referee, I calculate the difference between their two paper recommendations.  On the 45-degree diagonal, referees agreed exactly on the relative difference between two papers.  In the first and third quadrant, referees shared the same ordering of the papers.  The 1,212 pairs on the diagonal shared the same numerical difference in relative ordering.  In 456 pairs, referees had the same exact recommendations on both papers; in 756 pairs, one referee had a higher recommendation, but both referees agreed on the numerical distance between the two papers. This is a pure mean effect.  In the second and fourth quadrant, referees have different preference orderings.}

  \medskip

  \interpret{Referees had modest agreement about relative pairwise ordering.  However, preference reversals were common.}

  \source{sharedplot.R}

  \bigskip


\end{figure}

Referees could have differed because they had different scales (referee-specific means or standard deviations) or because they disagreed fundamentally about which paper was better.  In the \sfscav, there were many referees who shared assignments for the same two papers. (This was because assignments were expertise-based.) I can therefore examine whether referees agreed or disagreed on the relative ordering of the same two papers.  For each of the two referees, A and B, I calculate the difference between the referees' two paper recommendations.  I then count the instances defined by the difference for A and the difference for B for each overlapping paper.

Figure~\fref{fig:pairwise} is a two-dimensional representation of this three-dimensional histogram.  (There is symmetry in this graph, because I repeat the calculation with A and B reversed.)  The graph suggests the following:
\begin{itemize}
\item On the 45-degree diagonal, there are 1,212 cases out of 4,416 pairs in which referees agreed exactly on the relative difference between two papers.  This includes 456 cases in which referees were in perfect agreement about the qualities of the two papers.  It also includes 756 cases in which referees agreed about the relative qualities but had different mean scales (thresholds). For example, one referee could have ranked the papers ``must accept'' (2) and ``neutral'' (0), while the other could have ranked them as ``neutral'' (0) and ``must reject'' (--2).

\item In the first and third quadrants, but off-diagonal, referees shared the same relative ordering of the papers, with both preferring the same paper, but they could not agree on how much better one paper was relative to the other.  One referee ranks one paper much more highly relative to its competitor paper than the other referee (e.g., the eighteen papers at coordinate +1,+3).  These cases can be viewed as referees having different scales (possibly on multiple moments) in translating their paper views into numerical scores.  Excluding cases in which one referee ranked the quality difference in both papers the same, there were 972 pairings in which referees agreed at least on the relative ordering of two papers.  Including these cases increases the agreement to 2,093 pairings.

\item In the second and fourth quadrants, referees had different preference orderings.  There were 702 cases in which one referee strictly preferred one paper, and the other referee strictly preferred the other paper.
\end{itemize}

The presence of cases in which referees disagreed about the ordering of papers rejects the hypothesis that referee recommendations are completely due to differences in scales.  Because 75\% of recommendations shared the relative ordering, while 25\% reversed it, there is, however, also more scale agreement than what would have been observed under chance.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Literature} \label{sect:literature}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The peer-review literature is voluminous and spans many disciplines.  \citeasnoun{cole:1992}, \citeasnoun{weller:2002}, and \citeasnoun{armstrong:1997} summarize it.  These existing studies suggested that reviewers are enamored with (i) statistical significance, (ii) large sample sizes, (iii) complex procedures, and (iv) obscure writing (verbiage).  \nothing{Peer reviews of medical manuscripts improve manuscripts in the eyes of other readers, although revisions do not improve readability.}  Reviews do not seem to improve readability and catch many errors, such as in citations, descriptions of the content of prior research, basic abuse of statistics, plagiarism, or previous publication. 

Armstrong's most important concern is with the clash between innovative research and existing belief systems, as well as journals' concerns about ``quality'' over ``importance.'' (\citealt{ellison:2002a,ellison:2002b} discuss how the publication process has slowed down over the years due to increased polishing and revising.)  Armstrong describes various experimental papers that show that scientists rate controversial research more highly if it corresponds to their own beliefs.  Indeed, controversial findings almost never receive unanimous support, and other evidence shows that editors rarely publish papers with conflicting reviews.  (The evidence in my own paper similarly suggests that one negative review usually results in rejection.)  Armstrong concludes that the barriers to publication of controversial or important research may be too great to make it worthwhile to work on such issues.  Similarly, \cite{gans-shepherd:1994} solicited informal but highly entertaining and troubling descriptions of the publication process for now-classic economics papers. Armstrong even notes a silver lining: the lack of reliability in reviews improves chances of publishing papers with innovative findings, but only if such papers are not evaluated by many referees.  Lack of reliability then becomes a useful randomizing device.

%%% Laband and Piette 1994 does something similar to BEP

\bigskip

The most relevant related literature to my paper are articles that study in quantitative terms the reliability (agreement) of referee accept/reject recommendations and the fixed-effect and other biases of referees.  This literature is much smaller. \citeasnoun{cole-cole-simon:1981} find that a second set of National Science Foundation (NSF) proposal reviewers for 150 proposals had only modest agreement with the first (regular) set of NSF reviewers.  \citeasnoun{peters-ceci:1982} resubmitted twelve recently published psychology papers under fictitious names and institutions to the same journal again. They summarize their findings as follows: ``Of 38 editors and reviewers, only three (8\%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89\%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as `serious methodological flaws.'\,'' \citeasnoun{fiske-fogg:1990} found that in the typical paper, two reviews of the same paper had no critical point in common. \citeasnoun{blank:1991} provides some evidence of referee consensus and similarly finds only modest association.  \citeasnoun{cherkashin-demidova-imai-krishna:2009} study the citation impact of 21 coeditors at the \textit{Journal of International Economics} (JIE) and document editor fixed effects.

\bigskip

Two current papers examine the journal review process in economics.  The first is \citeasnoun{Brogaard-Engelberg-Parson:2011}, which looks at how editor rotations associate with increased publication rates of ``nearby'' authors.  By showing that these papers are also more cited in the long run, the evidence rejects a nepotism hypothesis in favor of an informedness hypothesis.  The second is \citeasnoun{card-dellavigna:2013}, which looks at how a recent policy change in maximum submission paper lengths can be used to measure the elasticity of authors to submit papers to the AER and to the JEEA.  The AER seems to have enough market power to induce authors to change their drafts to qualify for submission, while the JEEA does not.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion} \label{sect:editorialize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The editorial process sets not only the direction of economics as a social science but also the incentives and professional fates of academic economists.  It consumes significant resources, both in terms of the time and effort by the editor and referees, and in terms of the time of authors adjusting their papers to make them more amenable to publication.  \cite{gans-shepherd:1994}  write about how the vagaries of the process have annoyed even the most distinguished economists---and on occasion editors \cite{spiegel:2012}.

The evidence in my paper quantified to what extent referee recommendations are influenced by referee-specific idiosyncratic components and by common reliable characteristics of the submissions (including author-related information).  The fact that the referee behavior is similar in the editorial-selection journal process and in the automated objective expertise-based referee-selection process suggests that the documented patterns are more likely to be due to referees themselves than to referee selection.  The evidence further suggests that the referee-specific component is partly due to the fact that some referees are intrinsically more generous than others, and partly due to the fact that referees cannot agree with one another about how good papers are relative to one another.

Suggestions to improve the refereeing process are always controversial.  In fact, it is not even clear that it is desirable to increase the consistency and reliability of reports across referees.   Reliability is not necessarily a measure of the quality or the future impact of the paper. (Gans and Shepherd note that many of the [previously] ``rejected classic articles'' were ultimately published by editors against the advice of the referees.)  Nevertheless, my paper suggests a number of changes where the primary cost amounts to little more than a change in the editorial web system:
\begin{itemize}
\item%%
  It is not clear whether the low or zero active experimentation (common at many journals) is an optimal choice.  My paper had to rely on indirect evidence from a non-journal venue to suggest that referee selection effects do not seem to have influenced greatly findings about referee behavior.  Journals could gain better and more relevant information if they themselves experimented.  If the current process is optimal, then small perturbations from the currently-optimal process would not be very costly.

  For example, it would be interesting to ask some referees for some additional quantifiable information, such as (i) whether they believe that another referee would likely agree or disagree with their assessment of the reviewed paper, and/or (ii) whether another referee would likely give the submission a higher or lower assessment. \citeasnoun{lynch:1998} points out that most referees have a distorted view of how much their views represent a general consensus of their peers.  Do they?

\item%%
  The widely used current editorial systems do not provide editors with information about referees' past recommendation behavior.  The benefit of an enhanced system with this information easily on hand is that it could make it easier for the editor to extract the referee-specific non-paper-specific signal in the referee report.

\item%%
  Journals could provide more feedback to referees.   Specifically, they could provide automatic notification of what other referees recommended on the same submission.  If it were the default to inform every referee of the final recommendation of other referees, it would not place editors in the awkward position of selectively informing referees about how different their views were.

\item%%
  Editors could ``even out'' the number of referees per paper.  The cost is that such homogeneity reduces the flexibility of editors to deal with papers and authors that are different.  The benefit is that a homogeneous number of referees reduces the very difficult problem of extracting the reliable component from (and deciding fairly across) papers with different numbers of referees.

\end{itemize}


\bigskip

\clearpage

\begin{doublespacing}
  \bibliographystyle{rfs}
  \bibliography{welch-2014-referees}
\end{doublespacing}

\clearpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\appendix
\setcounter{table}{0}
%\renewcommand{\thetable}{\Alph{table}}
\addtocontents{lot}{\bigskip}
\addtocontents{lot}{Appendix}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 
\begin{Large}{\noindent Appendix\\[0.2\baselineskip]Related Findings}\end{Large}

The \sfscav\ also included a set of additional variables that allowed some more analysis of referee-related behavior patterns.  The following inference explaining referee recommendations and consensus\footnote{The consensus inference is based either on lambda estimates in a subset of papers or on cross-variables explaining one referee's recommendation with that of another multiplied by the additional variable} among referees is based on simple correlations between variables provided by referees and authors themselves.

\begin{description}
\item[Execution quality and interestingness:] (Mean) Referees valued execution quality and degree of interesting-ness of the paper roughly equally.  (Consensus) Referees had no more agreement about whether papers were interesting than about whether they were well executed.

\nothing{For the 390 recommendations that ranked a paper both well-executed and interesting, there was less agreement on the score than would have been expected by chance.}

\item[Author age:] (Mean) There was no meaningful difference in referee recommendations for submissions by author teams that were younger or older, where age was measured either as the  average year of the PhD or as the age of the PhD of the oldest member.

\item[Author university rank:] (Mean) When there was at least one author from a highly ranked university,\footnote{This is a dummy largely based on the ranking in the list of top Arizona PhD program list \url{http://wpcarey.asu.edu/fin-rankings/rankings/results.cfm}: NYU, Chicago, Harvard, Penn, UCLA, Michigan, Duke, OSU, Columbia, Northwestern, MIT, Stanford, Cornell, Texas, USC, LBS, Illinois, BC, UNC, Berkeley, Maryland, ASU, Hong Kong, Purdue, Rochester.  I added LSE, Princeton, and Yale.} the probability of the paper receiving an MA was 10.1\%.  When there was none, it was 2.2\%.  (Consensus) The lambda statistics for agreement among referees were 0.12 and 0.18, respectively.  That is, referees agreed more about papers written by authors from lower-ranked universities.
  
This contrasts with the findings of \citeasnoun{peters-ceci:1982}, who found in a quasi-experiment that reviewers in psychology journals were biased against authors from less prestigious institutions.  \citeasnoun{perlman:1982}, however, argues that authors at higher-ranked institutions write more cited papers, which \textit{should} influence the reviewing process.  (He equates citation impact with quality.)

\item[Author U.S. university:] (Mean)  When there was at least one U.S. author on the team, the probability of an MA was 5.7\%.  When there was none, it was 1.5\%.  (Consensus) The agreement among referees about submission quality was similar.

\item[Other referee recommendations:] There is some evidence that I could have predicted even better with a more complex function than the referee's mean.  When a referee had not offered even one single other high recommendation, such a referee was marginally more likely to recommend an acceptance on the current paper.  However, the effect is not strong and could easily have been the result of specification search.

\item[Refereeing frequency:] (Mean) Referees who reported to have refereed more papers and accepted more papers in the past (elsewhere) were less likely to accept a paper for the conference.  (Consensus) There was modestly greater consensus among referees with more past refereeing.

\item[Referee age:] There was no clear pattern related to the referee's age (where age is PhD cohort).

\item[Referee PhD university rank:] Referees with PhDs from highly ranked universities were modestly more critical (mean of --0.18), but showed less consensus than their complement.

\item[Referee university rank:]  Referees currently at highly ranked universities were more critical (mean of --0.29) and showed modestly more consensus than their complement (0.28 versus 0.23).
\end{description}





\end{document}

