Makita
Empirical validation is the most critical form of evidence available in machine learning. Especially settings when model performance directly leads to, or removes from, adoption in research domains. In the context of systematic reviews, where we want researchers to rely on the machine learning methods we provide, the way to build trust is by verifying the work.
ASReview Makita (MAKe IT Automatic) was created to support this need. Makita is a workflow generator for simulation studies using the command line of ASReview LAB. By standardizing and simplifying how experiments are defined, configured and executed, it lowers the effort barrier towards solid, repeatable, large scale simulation studies and therefore empirical proof of functionality.
Progress
Makita is fully functional software, working stand-alone as a python package. It had been used in multiple as a basis for setting up simulation studies. The tool is published as a peer-reviewed article in Software Impacts describing the design and utility of Makita.
Using Makita templates, different study structures can be generated to fit the needs of your very own study too. If your study requires a unique template, you can create a new one and use it instead.
What Makita does:
- Setting up a workflow for running a large-scale simulation study
- Preparing a Github repository
- Automating the many lines of code needed
- Creating a batch script for running the simulation study with just one line of code
- Making your research fully reproducible
- Allowing you to add templates to accommodate your own specific research question
What Makita does not do:
- Executing jobs or tasks itself
- Being a black-box
- Writing your paper
People involved
- Jelle Teijema - PhD Candidate
- Jonathan de Bruin - Lead developer
- Rens van de Schoot - Advisor