Skip to content

Commit

Permalink
add sampled phrases for co-review policies
Browse files Browse the repository at this point in the history
  • Loading branch information
@tklebel
tklebel committed Jul 23, 2020
1 parent cf4f71f commit 7ee40f8
Show file tree
Hide file tree
Showing 3 changed files with 275 additions and 8 deletions.
34 changes: 32 additions & 2 deletions 02-analysis-writeup.Rmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -525,7 +525,7 @@ top_terms <- stemmed_words %>%
# sample from variants
set.seed(1234)
stemmed_words %>%
top_variants <- stemmed_words %>%
count(word, word_stemmed, name = "variant_count") %>%
right_join(top_terms) %>%
arrange(word_stemmed, desc(variant_count)) %>%
Expand All @@ -536,13 +536,43 @@ stemmed_words %>%
variants = map_chr(variants, paste, collapse = "; ")) %>%
select(-word, -variant_count) %>%
distinct() %>%
arrange(desc(prop_of_texts)) %>%
arrange(desc(prop_of_texts))
top_variants %>%
select(Term = word_stemmed, Variants = variants, `Term frequency` = n,
`Proportion of policies that contain term` = prop_of_texts) %>%
knitr::kable(caption = "Propensity of terms in co-review policies")
```


```{r sample-co-rev-phrases}
co_rev_sentences <- distinct_coreview %>%
unnest_tokens(sentence, coreview_policy, token = "sentences") %>%
distinct()
# search_terms <- top_terms$word_stemmed
search_terms <- top_variants %>%
pull(variants) %>%
str_split(pattern = "; ") %>%
flatten_chr()
set.seed(98375)
sampled_sentences <- co_rev_sentences %>%
mutate(matching = map(sentence, str_detect, search_terms)) %>%
unnest(matching) %>%
mutate(term = rep(search_terms, nrow(co_rev_sentences))) %>%
filter(matching) %>%
group_by(term) %>%
slice_sample(n = 1)
sampled_sentences %>%
mutate(sentence = str_replace(sentence, term, paste0("*", term, "*"))) %>%
select(term, sample_phrase = sentence) %>%
knitr::kable(caption = "Sample phrases for prominent terms in co-review policies")
```


The most prominent themes that emerge are:

- Individuals with varying stakes regarding peer review: editor, colleague,
Expand Down
197 changes: 192 additions & 5 deletions 02-analysis-writeup.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@

<h1 class="title toc-ignore">Analysis Write-up</h1>
<h4 class="author">Thomas Klebel</h4>
<h4 class="date">Last updated 2020-07-05</h4>
<h4 class="date">Last updated 2020-07-23</h4>

</div>

Expand Down Expand Up @@ -1012,6 +1012,193 @@ <h1>Co-Review</h1>
</tr>
</tbody>
</table>
<table>
<caption><span id="tab:sample-co-rev-phrases">Table 3: </span>Sample phrases for prominent terms in co-review policies</caption>
<colgroup>
<col width="4%"></col>
<col width="95%"></col>
</colgroup>
<thead>
<tr class="header">
<th align="left">term</th>
<th align="left">sample_phrase</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="left">advice</td>
<td align="left">&quot; in some instances, reviewers may feel that it would be helpful to obtain additional <em>advice</em> from a colleague.</td>
</tr>
<tr class="even">
<td align="left">author</td>
<td align="left">since peer review is confidential, you also must not share information about the review with anyone without permission from the editors and <em>author</em>s.</td>
</tr>
<tr class="odd">
<td align="left">authorization</td>
<td align="left">this means you can’t share them with anyone without prior <em>authorization</em> from the editor.</td>
</tr>
<tr class="even">
<td align="left">authors</td>
<td align="left">“since peer review is confidential, you also must not share information about the review with anyone without permission from the editors and <em>authors</em>.”</td>
</tr>
<tr class="odd">
<td align="left">collaborate</td>
<td align="left">…reviewers can <em>collaborate</em> with trainees (graduate students and post-docs) in the evaluation of manuscripts… however, we ask that reviewers keep the number of collaborators to a minimum and include the identities of all the individuals involved in the “comments to the editors” component of their review.</td>
</tr>
<tr class="even">
<td align="left">collaborating</td>
<td align="left">we encourage referees to inform <em>collaborating</em> reviewers about appropriate guidelines and ethics for peer review, as outlined in this document.</td>
</tr>
<tr class="odd">
<td align="left">collaborators</td>
<td align="left">…reviewers can collaborate with trainees (graduate students and post-docs) in the evaluation of manuscripts… however, we ask that reviewers keep the number of <em>collaborators</em> to a minimum and include the identities of all the individuals involved in the “comments to the editors” component of their review.</td>
</tr>
<tr class="even">
<td align="left">colleague</td>
<td align="left">you should not show the paper to anyone else, including <em>colleague</em>s or students, unless you have asked them to write a review, or to help with your review.</td>
</tr>
<tr class="odd">
<td align="left">colleagues</td>
<td align="left">although referees may consult and seek advice from other researchers or <em>colleagues</em>, the referee must ensure that the confidentiality of these materials is preserved.</td>
</tr>
<tr class="even">
<td align="left">confidential</td>
<td align="left">annals of internal medicine expects reviewers to handle manuscripts in a <em>confidential</em> manner.</td>
</tr>
<tr class="odd">
<td align="left">confidentiality</td>
<td align="left">… in such cases, we ask that the reviewer contact the editor in advance to ensure that the editor has the opportunity to take additional information into account before permitting communications that have the potential to violate <em>confidentiality</em>.</td>
</tr>
<tr class="even">
<td align="left">consult</td>
<td align="left">although referees may <em>consult</em> and seek advice from other researchers or colleagues, the referee must ensure that the confidentiality of these materials is preserved.</td>
</tr>
<tr class="odd">
<td align="left">consulted</td>
<td align="left">inferred that this is not allowed from the following: “treat the manuscript as confidential: the manuscript (or its existence) should not be shown to, disclosed to, or discussed with others, except in special cases, where specific scientific advice may be sought; in that event the editor must be informed and the identities of those <em>consulted</em> disclosed.”</td>
</tr>
<tr class="even">
<td align="left">consulting</td>
<td align="left"><em>consulting</em> with experts from outside the referee’s own laboratory may be acceptable, but please check with the editors before doing so, to avoid involving anyone who may have been excluded by the authors.</td>
</tr>
<tr class="odd">
<td align="left">disclose</td>
<td align="left">if you do choose to discuss the manuscript and/or your review with a professional colleague whose input you request as part of your review process, you are responsible for ensuring that they are made fully aware of the confidential nature of the discussion and that they must not <em>disclose</em> any information about the manuscript until the article is published.</td>
</tr>
<tr class="even">
<td align="left">disclosed</td>
<td align="left">in such instances, the identities of those to be consulted should be <em>disclosed</em> to the editor in advance&quot; it is acceptable to consult with laboratory colleagues, but please identify them to the editors.</td>
</tr>
<tr class="odd">
<td align="left">discuss</td>
<td align="left">we ask referees to treat the review process as strictly confidential, and not to <em>discuss</em> the manuscript with anyone not directly involved in the review.</td>
</tr>
<tr class="even">
<td align="left">discussed</td>
<td align="left">such documents should neither be disclosed to nor <em>discussed</em> with others except, in special cases, when shared in confidence with persons from whom specific expert advice may be sought.</td>
</tr>
<tr class="odd">
<td align="left">discussion</td>
<td align="left">if you do choose to discuss the manuscript and/or your review with a professional colleague whose input you request as part of your review process, you are responsible for ensuring that they are made fully aware of the confidential nature of the <em>discussion</em> and that they must not disclose any information about the manuscript until the article is published.</td>
</tr>
<tr class="even">
<td align="left">editor</td>
<td align="left">it is acceptable to consult with laboratory colleagues, but please identify them to the <em>editor</em>s.</td>
</tr>
<tr class="odd">
<td align="left">editors</td>
<td align="left">“peer reviewers are required to maintain confidentiality about the manuscripts they review and must not divulge any information about a specific manuscript or its content to any third party without prior permission from the journal <em>editors</em>.”</td>
</tr>
<tr class="even">
<td align="left">identities</td>
<td align="left">however, we ask that reviewers keep the number of collaborators to a minimum and include the <em>identities</em> of all the individuals involved in the “comments to the editors” component of their review.</td>
</tr>
<tr class="odd">
<td align="left">identity</td>
<td align="left">the <em>identity</em> of any co-reviewer and any potential conflicting or competing interests they may have must be disclosed when submitting your review.</td>
</tr>
<tr class="even">
<td align="left">inform</td>
<td align="left">if you do choose to discuss the manuscript and/or your review with a professional colleague whose input you request as part of your review process, you are responsible for ensuring that they are made fully aware of the confidential nature of the discussion and that they must not disclose any <em>inform</em>ation about the manuscript until the article is published.</td>
</tr>
<tr class="odd">
<td align="left">information</td>
<td align="left">peer reviewers are required to maintain confidentiality about the manuscripts they review and must not divulge any <em>information</em> about a specific manuscript or its content to any third party without prior permission from the journal editors.</td>
</tr>
<tr class="even">
<td align="left">informed</td>
<td align="left">the manuscript (or its existence) should not be shown to, disclosed to, or discussed with others, except in special cases, where specific scientific advice may be sought; in that event the editor must be <em>informed</em> and the identities of those consulted disclosed.</td>
</tr>
<tr class="odd">
<td align="left">involve</td>
<td align="left">“if you wish to <em>involve</em> a specially qualified colleague in the review (or perhaps want to guide a junior colleague in learning how to review), you must contact the editorial office and ask for permission ahead of time.”</td>
</tr>
<tr class="even">
<td align="left">involved</td>
<td align="left">we ask referees to treat the review process as strictly confidential, and not to discuss the manuscript with anyone not directly <em>involved</em> in the review.</td>
</tr>
<tr class="odd">
<td align="left">involving</td>
<td align="left">consulting with experts from outside the referee’s own laboratory may be acceptable, but please check with the editors before doing so, to avoid <em>involving</em> anyone who may have been excluded by the authors.</td>
</tr>
<tr class="even">
<td align="left">journal</td>
<td align="left">if you wish to have another expert co-review the manuscript with you, you must first obtain permission from the <em>journal</em> office.&quot;</td>
</tr>
<tr class="odd">
<td align="left">manuscript</td>
<td align="left">…reviewers can collaborate with trainees (graduate students and post-docs) in the evaluation of <em>manuscript</em>s… however, we ask that reviewers keep the number of collaborators to a minimum and include the identities of all the individuals involved in the “comments to the editors” component of their review.</td>
</tr>
<tr class="even">
<td align="left">manuscripts</td>
<td align="left">for example, you could share your own <em>manuscripts</em> and the reviews you received as an author with your students to facilitate their learning.</td>
</tr>
<tr class="odd">
<td align="left">not</td>
<td align="left">if you do choose to discuss the manuscript and/or your review with a professional colleague you are responsible for ensuring that they are made fully aware of the confidential nature of the discussion and that they must <em>not</em> disclose any information about the manuscript until the article is published.</td>
</tr>
<tr class="even">
<td align="left">peer</td>
<td align="left">we encourage referees to inform collaborating reviewers about appropriate guidelines and ethics for <em>peer</em> review, as outlined in this document.</td>
</tr>
<tr class="odd">
<td align="left">permission</td>
<td align="left">if you wish to have another expert co-review the manuscript with you, you must first obtain <em>permission</em> from the journal office.&quot;</td>
</tr>
<tr class="even">
<td align="left">review</td>
<td align="left">a <em>review</em>er may request advice from another party, subject to the general principle of confidentiality and notification of the jci.</td>
</tr>
<tr class="odd">
<td align="left">reviewer</td>
<td align="left">please be sure to contact the handling editor and indicate upon submitting your review that it was completed with a co-<em>reviewer</em>.&quot;</td>
</tr>
<tr class="even">
<td align="left">reviewers</td>
<td align="left"><em>reviewers</em> are expected to keep manuscripts confidential.</td>
</tr>
<tr class="odd">
<td align="left">share</td>
<td align="left">please do not <em>share</em> the manuscript with any colleagues without the explicit permission of the editor.</td>
</tr>
<tr class="even">
<td align="left">shared</td>
<td align="left">to protect the authors’ work as well as your anonymity, communications regarding the manuscript and its parts, including the abstract, may not be <em>shared</em> for any reason.</td>
</tr>
<tr class="odd">
<td align="left">sharing</td>
<td align="left">reviewers should not keep copies of reviewed manuscripts in their personal files and are prohibited from <em>sharing</em> copies of the manuscript with others, except with the permission of the editor.</td>
</tr>
<tr class="even">
<td align="left">student</td>
<td align="left">we understand that doctoral <em>student</em>s need to learn how to write reviews, but there are other methods that can be used.</td>
</tr>
<tr class="odd">
<td align="left">students</td>
<td align="left">do not have your <em>students</em> write your reviews for amr.</td>
</tr>
</tbody>
</table>
<p>The most prominent themes that emerge are:</p>
<ul>
<li>Individuals with varying stakes regarding peer review: editor, colleague,
Expand Down Expand Up @@ -1102,7 +1289,7 @@ <h1>Preprints</h1>
<li>Results on whether preprints can be cited (n = 193)</li>
</ol>
<table>
<caption><span id="tab:preprint-tables">Table 3: </span>Posting of prepritns</caption>
<caption><span id="tab:preprint-tables">Table 4: </span>Posting of prepritns</caption>
<colgroup>
<col width="27%"></col>
<col width="53%"></col>
Expand Down Expand Up @@ -1403,7 +1590,7 @@ <h1>Preprints</h1>
</tbody>
</table>
<table>
<caption><span id="tab:preprint-tables">Table 3: </span>Citing of prepritns</caption>
<caption><span id="tab:preprint-tables">Table 4: </span>Citing of prepritns</caption>
<colgroup>
<col width="39%"></col>
<col width="32%"></col>
Expand Down Expand Up @@ -1797,7 +1984,7 @@ <h1>The Landscape of Open Science Policies</h1>
policies than the average on the right.</li>
</ol>
<!-- The following sections are outdated. -->
<p>Numerical output from the MCA is shown in table <a href="#tab:mca-table">4</a>. We can
<p>Numerical output from the MCA is shown in table <a href="#tab:mca-table">5</a>. We can
see, that the contribution to the geometric layout is highest for the types of
peer review, policies for citing prepreints and whether reviewer identities are
revealed to authors (column inertia).
Expand Down Expand Up @@ -1831,7 +2018,7 @@ <h1>The Landscape of Open Science Policies</h1>
from the SSH are below average in this regard.</p>
<!-- This goes into the appendix. -->
<table style="width:100%;">
<caption><span id="tab:mca-table">Table 4: </span>Numerical output from Multiple Correspondence analysis</caption>
<caption><span id="tab:mca-table">Table 5: </span>Numerical output from Multiple Correspondence analysis</caption>
<colgroup>
<col width="28%"></col>
<col width="3%"></col>
Expand Down
Loading

0 comments on commit 7ee40f8

Please sign in to comment.