Calculate confusion and inversion matrices based on WAIC values.
This function computes the confusion matrix and inversion matrix normalized
from a DataFrame containing simulation results. The confusion matrix
reflects how often each model is selected as the best model when each model
is true (i.e., P(estimated | true)). The inversion matrix normalized
represents the proportion of times each model is estimated to be the best
given each true model (i.e., P(true | estimated)).
Parameters:
-
(
DataFrame)
–
A DataFrame containing the following columns:
"simulated_model": Label of the model used for simulation."estimation_model": Label of the model used for estimation."iteration": Iteration identifier (i.e., the iteration of
the model comparison procedure, where multiple datasets are
simulated and then fitted).
A further column corresponding to the given metric should also be
present. This column should correspond to the value of the
metric argument.
-
(
str, default:
'waic'
)
–
The name of the column to be used as the metric for
Returns:
-
(ndarray, ndarray)
–
np.ndarray: The confusion matrix representing how often each model is
estimated to be the best when each model is true.
-
(ndarray, ndarray)
–
np.ndarray: The inversion matrix normalized, indicating the proportion
of times each model is the true generating model given each
estimated model.
Example
confusion_matrix, inversion_matrix =
calculate_confusion_and_inversion(results)
Here, results is expected to be a DataFrame with at least the columns
"simulated_model", "estimation_model", "iteration", and "waic".
The results can then be plotted using
model_fit_tools.plotting.plot_matrices:
model_fit_tools.plotting.plot_matrices(confusion_matrix,
inversion_matrix, models)
Source code in model_fit_tools/model_comparison.py
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97 | def calculate_confusion_and_inversion(
df: pd.DataFrame, metric: str = "waic"
) -> (np.ndarray, np.ndarray):
"""
Calculate confusion and inversion matrices based on WAIC values.
This function computes the confusion matrix and inversion matrix normalized
from a DataFrame containing simulation results. The confusion matrix
reflects how often each model is selected as the best model when each model
is true (i.e., `P(estimated | true)`). The inversion matrix normalized
represents the proportion of times each model is estimated to be the best
given each true model (i.e., `P(true | estimated)`).
Args:
df (pd.DataFrame): A DataFrame containing the following columns:\n
- `"simulated_model"`: Label of the model used for simulation.
- `"estimation_model"`: Label of the model used for estimation.
- `"iteration"`: Iteration identifier (i.e., the iteration of
the model comparison procedure, where multiple datasets are
simulated and then fitted).\n
A further column corresponding to the given metric should also be
present. This column should correspond to the value of the
`metric` argument.
metric (str): The name of the column to be used as the metric for
selecting
the winning model. Higher values should indicate better models.
Defaults to "waic".
Returns:
np.ndarray: The confusion matrix representing how often each model is
estimated to be the best when each model is true.
np.ndarray: The inversion matrix normalized, indicating the proportion
of times each model is the true generating model given each
estimated model.
Example:
`confusion_matrix, inversion_matrix =
calculate_confusion_and_inversion(results)`
Here, `results` is expected to be a DataFrame with at least the columns
"simulated_model", "estimation_model", "iteration", and "waic".
The results can then be plotted using
`model_fit_tools.plotting.plot_matrices`:
`model_fit_tools.plotting.plot_matrices(confusion_matrix,
inversion_matrix, models)`
"""
# Extract unique models and initialize matrix
models = df["simulated_model"].unique()
n_models = len(models)
confusion_matrix = np.zeros((n_models, n_models))
# Ensure the metric column exists in the dataframe
if metric not in df.columns:
raise ValueError(
f"Metric {metric} is not a column in the provided DataFrame."
)
# Calculate confusion matrix
for n, simulated_model in enumerate(models):
for i in df["iteration"].unique():
# Extract relevant run results
run_results = df[
(df["iteration"] == i)
& (df["simulated_model"] == simulated_model)
].copy()
# Convert the 'estimation_model' column to a Categorical type with
# custom order
run_results["estimation_model"] = pd.Categorical(
run_results["estimation_model"],
categories=models,
ordered=True,
)
# Sort the DataFrame by 'estimation_model'
run_results = run_results.sort_values("estimation_model")
# Update confusion matrix
confusion_matrix[np.argmax(run_results[metric].values), n] += 1
# Normalize confusion matrix
confusion_matrix = confusion_matrix / len(df["iteration"].unique())
# Calculate inversion matrix
inversion_matrix = confusion_matrix.T
return confusion_matrix, inversion_matrix
|