How review works

On this page

• Overview of review process
  • What happens after review?
• Instructions for package authors
• Instructions for reviewers
• Checklist
  • Information and metadata
  • Documentation
  • Testing
  • Code
• Glossary
  • API - application programming interface
  • Version control
  • Repository
  • License

Overview of review process

1. Package author initiates a github issue under the software-review repository of the SMARTbiomed github organization
2. The software coordinator does an initial review
3. Once the package has passed initial review, they find two peer reviewers
4. Peer reviewers return comments as github comments under the issue
5. Author responds to comments and updates package, as needed
6. Decision is made, and once accepted, package is posted on the SMARTbiomed website and publicized with a “peer-reviewed” badge that looks like this
7. Author is encouraged to draft a software manuscript for submission to an appropriate scientific journal e.g., Journal of Statistical Software, the R journal, Journal of Computational and Graphical Statistics, Bioinformatics.

What happens after review?

1. Software coordinator regularly reviews individual packages (every 4 months), and links packages that work together, checks for maintenance issues such as broken dependencies or unaddressed bug reports. Also can identify opportunities for packages to work together in a bigger pipeline and develop tutorials or teaching materials
2. Software coordinator reviews processes (annually) to identify areas for improvement or modification. Provides a report of progress made and lessons learned.
3. Develop workshops for collaborators to learn about the guidelines and specific issues to enhance package development

Instructions for package authors

If you are the author of a software package that you would like to be reviewed, prepare the following things:

• A link to the package source code available in a public repository, e.g., Github.
• A cover letter indicating
  • What the package goals are
  • What the other available packages with similar goals are
  • Why this one is needed
  • A clearly scientifically motivated example in a common, complex disease setting
  • An indication of what theme/category it belongs to (causal inference, risk prediction, machine learning)
• Suggestions for reviewers, check the list of mentors
• A methods paper or manuscript, if one is available

When you are ready, create a github issue following the review template. If you are unsure, you can submit a pre-submission inquiry to get a fast response from the software coordinator.

Please read our code of conduct. Information about the status of your review and related comments will be associated with that github issue. Make sure your notification settings are such that you will recieve emails when the issue is updated.

Instructions for reviewers

You will be tagged and invited to comment on a github issue that contains information about the package to be reviewed. Follow the instructions and leave a comment on the github issue with your review. Please read our code of conduct.

The purpose of the code review is to learn from each other and improve the quality of our research software. This checklist is merely a guide you can use to check for the presence and quality of these items, and to identify deficiencies that can be improved. The list is not exhaustive, nor is it expected that every single item is checked.

Refer to this checklist during your review, and remember to be constructive and kind in your comments.

Checklist

Information and metadata

• Package title short and descriptive
• Software category (e.g., which theme or themes does it fall under) and keywords provided
• Source code publicly available on an appropriate repository
• OSI accepted license clearly stated and included with the source code.
• Version controlled (preferably with git) and using semantic versioning starting from 0.1.0:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes MINOR version when you add functionality in a backward compatible manner PATCH version when you make backward compatible bug fixes

• List of package maintainers (with email), authors and contributors, including authors of any code copied or included from other sources
• Package dependencies clearly indicated, including what systems the software is confirmed to run on (e.g., linux, windows).
• Packages that require non standard build tools, dependencies that are not easily obtained, or have other specialized requirements should consider providing a dockerfile and including examples of how it is used.
• Link to bug tracker provided

Documentation

• Readme
• Short description of the goals/main purpose of the package avoiding the use of jargon
• Installation instructions
• Brief demonstration of the usage of the package
• Description of how the package relates to other packages in the organization, and how it compares to packages with similar - [ ] goals
• Links to further documentation and tutorials
• Links to pre-built binaries (using r-universe for R packages for example)
• Tutorials/vignettes
• Extended and completely reproducible demonstration of usage with strongly motivated examples and interpretation of results in the scientific context (task oriented, i.e., how to do the analysis)
• Illustration of theory/methodology and comparison to alternatives (enhances understanding of methodology)
• Illustration/instructions about implementation and possible extensions/connections to other software packages within the organization (enhances understanding of software)
• Function reference – for each function in the package
• Description of what the function does
• Function inputs description
• Function output description
• Examples
• Changelog/news
• What has changed since the previous version
• Documentation website
• E.g., using pkgdown or sphinx
• Includes all of the above in an easily browsable setting

Testing

• Unit tests
• All functions should be tested to ensure functionality and input/output expectations are met
• Validation tests
• Statistical methods should use benchmark data or simulation studies to demonstrate validity of the method, e.g., type I error, bias, efficiency, whatever is relevant to the method. The code to make the simulated data, or the data itself should be provided with the code as an example for users so the code editor can run spot-checking.
• Integration tests
• If the package interacts with other software packages, tests should check that they function together as expected
• Tests of graphical interfaces should be present where applicable
• Tests should be automatically run during a build process

Code

• Code is organized into separate source files in a sensible manner with descriptive names
• Functions and arguments names work together to form a common, logical programming API that is easy to read and auto-complete
• Consider an object_verb() naming scheme for functions or object.verb() for object oriented paradigms, where object refers to the data and verb the primary action. This can be extended to include multiple words separated by underscores in order to be as descriptive as necessary. This scheme helps make code readable and easy to auto-complete.
• For functions that manipulate an object/data and return an object/data of the same type, make the object/data the first argument of the function so as to enhance compatibility with pipe operators (base R’s |>, magrittr’s %>%) and with other packages
• We recommend using the following style guides for R and python. Other languages may also have “official” or highly regarded style guides. While the use of a particular style is not required, it is required to use the same style consistently throughout the package.
• Avoid function name conflicts with base packages or other popular ones
• Argument naming and order should be consistent across functions that use similar inputs.
• Functions should return coherent data structures rather than making modifications to the global environment and/or output to the hard disk as side effects. Output to disk should be explicit and obvious to the user.
  
• Any obvious inefficiencies should be fixed (create separate lists for different languages)
• Input validation/checking

Glossary

API - application programming interface

This refers to the way that users and other software interact with your software package. It could be using a graphical interface, or using code. It generally refers to the manner of getting your package to do something, and how your package returns a result.

Version control

The practice of tracking and managing changes to software code. Version control systems are software tools that help software teams manage changes to source code over time. Examples of version control systems are git and svn.

Repository

A centralized, external storage location for software packages. Examples include github, CRAN, Bioconductor.

License

A software license is a document that provides legal guidelines for the use and distribution of software. These is a list of open source licenses here: https://opensource.org/licenses.