---
title             : "Supplementary Materials: Improving measurements of similarity judgments with machine-learning algorithms"
shorttitle        : "Supplementary Materials"

author: 
  - name          : "Jeffrey R. Stevens"
    affiliation   : "1"
    corresponding : yes    # Define only one corresponding author
    address       : "B83 East Stadium, Center for Brain, Biology & Behavior, University of Nebraska-Lincoln, Lincoln, Nebraska 68588, USA"
    email         : "jeffrey.r.stevens@gmail.com"
  - name          : "Alexis Polzkill Saltzman"
    affiliation   : "1"
  - name          : "Tanner Rasmussen"
    affiliation   : "1"
  - name          : "Leen-Kiat Soh"
    affiliation   : "1"

affiliation:
  - id            : "1"
    institution   : "University of Nebraska-Lincoln"



figsintext        : yes
figurelist        : no
tablelist         : no
footnotelist      : no
lineno            : no

lang              : "english"
class             : "doc"
keep_tex          : "TRUE"
output            : papaja::apa6_pdf
---

```{r include = FALSE}
library(papaja)
```

\renewcommand{\thetable}{S\arabic{table}}
\setcounter{table}{0}
\renewcommand{\thefigure}{S\arabic{figure}}
\setcounter{figure}{0}

\renewcommand{\arraystretch}{1.35}

<!--(Table S1)-->
```{r predictors,  out.width='100%'}
predictors_df <- data.frame(pred_name = c("Small value", "Large value", "Difference", "Ratio", "Mean ratio", "Log ratio", "Relative difference", "Disparity ratio", "Salience", "Discriminability", "Logistic"), pred_func = c("$S$", "$L$", "$L-S$", "$\\frac{S}{L}$", "$\\frac{S}{\\frac{S+L}{2}}$", "log($\\frac{S}{L}$)", "$\\frac{L-S}{L}$", "$\\frac{L-S}{\\frac{S+L}{2}}$", "$\\frac{L-S}{S+L}$", "log($\\frac{L}{L-S}$)", "$\\frac{1}{1+e^{L-S}}$"))
apa_table(predictors_df, caption = "Predictors", col.names = c("Predictor name", "Value/function"), note = "Table from Stevens \\& Soh (2018).", placement = "h", escape = FALSE)
```

(ref:importance-tab-note) Drawn from \emph{caret} package documentation (Kuhn, 2020). Table used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.

<!--(Table S2)-->
```{r predictorimportance, results = "asis", out.width='100%'}
pred_imp_data <- data.frame(alg = c("C5.0", "", "CART", "", "kNN", "Naive Bayes", "Neural network", "Random forest", "", "", "Regression"), pred_imp = c("Percentage of training set samples that fall into all the terminal nodes", "after the split", "Reduction in the loss function (e.g., mean squared error) attributed to", "each variable at each split", "Area under the ROC curve", "Area under the ROC curve", "Absolute values of node weights", "Difference between prediction accuracy on the out-of-bag portion of", "the data and after permuting each predictor variable, averaged over", "all trees and normalized by the standard error", "Absolute value of the t–statistic for each model parameter"))
apa_table(pred_imp_data, caption = "Predictor importance", col.names = c("Algorithm", "Predictor importance calculation"), note = "(ref:importance-tab-note)", placement = "h", escape = FALSE)
```

\newpage

(ref:pairwise-amt-fig-cap) Pairwise correlations for *amount* similarity judgment predictors. Diagonal shows histogram, below diagonal shows correlation plots, above diagonal shows correlation coefficients. Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.

<!--(Figure S1)-->
```{r amountcorr-fig, fig.cap = "(ref:pairwise-amt-fig-cap)", out.width = "100%"}
knitr::include_graphics(path = c("figures/predictor_correlations_amount.png"))
```

(ref:pairwise-delay-fig-cap) Pairwise correlations for *delay* similarity judgment predictors. Diagonal shows histogram, below diagonal shows correlation plots, above diagonal shows correlation coefficients. Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.

<!--(Figure S2)-->
```{r delaycorr-fig, fig.cap = "(ref:pairwise-delay-fig-cap)", out.width = "100%"}
knitr::include_graphics(path = c("figures/predictor_correlations_delay.png"))
```

<!--(Figure S3)-->
```{r accprecrecall-fig, fig.cap = "Accuracy, precision, and recall rates for each algorithm, judgment type, and data set (A = Data set 1, B = Data set 2). Algorithms are ordered by overall accuracy rates. Dots represent means, error bars represent within-subjects 95$\\%$ confidence intervals, boxplot horizontal lines represent medians, boxes represent interquartile range, whiskers represent 1.5 $\\times$ interquartile range. Outliers are not shown. Note the y-axis is truncated at 0.6 to enlarge the presentation of the means and confidence intervals. Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.", fig.align = "center", out.width = "95%"}
knitr::include_graphics(path = c("figures/accuracy_precision_recall_datasets.png"))
```

<!--(Figure S4)-->
```{r importance-dataset-fig, fig.cap = "Predictor importance for each judgment type and data set (A = Data set 1, B = Data set 2). Predictor importance refers to the relative contribution of each predictor to the response. Predictors are ordered by overall mean importance. Dots represent means, error bars represent within-subjects 95$\\%$ confidence intervals, boxplot horizontal lines represent medians, boxes represent interquartile range, whiskers represent 1.5 $\\times$  interquartile range. Outliers are not shown. Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.", out.width = "100%"}
knitr::include_graphics(path = c("figures/predictor_importance_datasets.png"))
```

<!--(Figure S5)-->
```{r importance-alg-fig, fig.cap = "Predictor importance for each judgment type and algorithm. Predictor importance refers to the relative contribution of each predictor to the response. Predictors are ordered by overall mean importance. Dots represent means, error bars represent between-subjects 95$\\%$ confidence intervals (the failure of regression models to calculate importance for the difference predictor prevents calculation of within-subject confidence intervals), boxplot horizontal lines represent medians, boxes represent interquartile range, whiskers represent 1.5 $\\times$ interquartile range. Outliers are not shown. Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.", out.width = "100%"}
knitr::include_graphics(path = c("figures/predictor_importance_algorithm.png"))
```

<!--(Figure S6)-->
```{r samplesize-fig, fig.cap = "Mean predictive accuracy for each sample size, judgment type, data set, and ordering (A = Random, B = Sequential). Sample size refers to number of questions per participant used to train the algorithms. Random refers to a random sample of training questions used to predict a random sample of 10 testing questions.  Sequential refers to a sample of training questions drawn in order of presentation to each participant that was used to predict a random sample of 10 testing questions.  Dots represent means, and error bars represent between-subjects 95$\\%$ confidence intervals (within-subject confidence intervals were not used because excessive missing data for small sample sizes caused too many participants to be removed from the calculations). Figure used with permission under a CC-BY4.0 license: Stevens et al., 2020; available at https://doi.org/10.17605/OSF.IO/WYTD9.", fig.align = "center", out.width = "90%"}
knitr::include_graphics(path = c("figures/accuracy_samplesize_order.png"))
```

