Skip to content

Correct the sign of context effects #616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
strengejacke merged 2 commits into from
May 5, 2026
Merged

Correct the sign of context effects #616

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2cad835
Correct the sign of context effects
strengejacke May 5, 2026
5edaad3
fix
strengejacke May 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Type: Package
Package: modelbased
Title: Estimation of Model-Based Predictions, Contrasts and Means
Version: 0.14.0.11
Version: 0.14.0.12
Authors@R:
c(person(given = "Dominique",
family = "Makowski",
Expand Down
7 changes: 5 additions & 2 deletions R/get_contexteffects.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@
# if we have stratified by another group, we need the difference between
# contrasts at each group level
if (is.null(my_args$by)) {
comparison <- stats::as.formula("~I(diff(x))")
comparison <- stats::as.formula("~revpairwise")
Comment thread
strengejacke marked this conversation as resolved.
} else if (length(my_args$by) > 2) {
# it is not possible to have more than two by-variables for now
insight::format_error(
"It is not possible to have more than two variables in `by` when calculating contrasts of slopes."
)
} else {
comparison <- stats::as.formula(paste(
"~I(diff(x)) |",
"~revpairwise |",
paste(my_args$by, collapse = "+")
))
}
Expand Down Expand Up @@ -96,6 +96,9 @@
}
}

# not needed
out$hypothesis <- NULL

# save some labels for printing
attr(out, "by") <- attr(out, "hypothesis_by") <- my_args$by
attr(out, "contrast") <- my_args$contrast
Expand Down
36 changes: 18 additions & 18 deletions tests/testthat/test-estimate_contrasts_context.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ test_that("estimate_contrast, context effects, linear", {
se <- coef(summary(m))[2:3, 2]

out <- estimate_contrasts(m, c("bill_len_between", "bill_len_within"))
expect_equal(out$Difference, b[2] - b[1], tolerance = 1e-4, ignore_attr = TRUE)
expect_equal(out$Difference, b[1] - b[2], tolerance = 1e-4, ignore_attr = TRUE)
expect_equal(out$SE, sqrt((se[1]^2 + se[2]^2)), tolerance = 1e-4, ignore_attr = TRUE)
expect_true(!is.null(out$p))

output <- capture.output(out)
expect_identical(output[3], "Difference | SE | 95% CI | z | p")
expect_identical(output[3], "Difference | SE | 95% CI | z | p")

expect_message(
estimate_contrasts(
Expand All @@ -34,16 +34,16 @@ test_that("estimate_contrast, context effects, linear", {
expect_named(out, c("year", "Difference", "SE", "CI_low", "CI_high", "z", "p"))
x1 <- estimate_slopes(m, "bill_len_within", by = "year")
x2 <- estimate_slopes(m, "bill_len_between", by = "year")
expect_equal(out$Difference, x1$Slope - x2$Slope, tolerance = 1e-4)
expect_equal(out$Difference, x2$Slope - x1$Slope, tolerance = 1e-4)

out <- estimate_contrasts(m, c("bill_len_between", "bill_len_within", "year"))
expect_named(
out,
c("Level1", "Level2", "Parameter", "Difference", "SE", "CI_low", "CI_high", "z", "p")
c("Level1", "Level2", "Difference", "SE", "CI_low", "CI_high", "z", "p")
)
expect_equal(
out$Difference,
c(-0.04317, -0.08635, -0.04317),
c(0.04317, 0.08635, 0.04317),
tolerance = 1e-4,
ignore_attr = TRUE
)
Expand All @@ -60,14 +60,14 @@ test_that("estimate_contrast, context effects, linear", {
c(
"Marginal Contrasts Analysis",
"",
"sex | year | Difference | SE | 95% CI | z | p",
"-----------------------------------------------------------------",
"female | 2007 | 0.28 | 0.08 | [ 0.12, 0.45] | 3.41 | < .001",
"female | 2008 | 0.26 | 0.06 | [ 0.15, 0.37] | 4.47 | < .001",
"female | 2009 | 0.24 | 0.10 | [ 0.05, 0.42] | 2.48 | 0.013",
"male | 2007 | 0.46 | 0.10 | [ 0.26, 0.65] | 4.60 | < .001",
"male | 2008 | 0.28 | 0.06 | [ 0.15, 0.40] | 4.41 | < .001",
"male | 2009 | 0.10 | 0.10 | [-0.11, 0.30] | 0.93 | 0.355",
"sex | year | Difference | SE | 95% CI | z | p",
"-------------------------------------------------------------------",
"female | 2007 | -0.28 | 0.08 | [-0.45, -0.12] | -3.41 | < .001",
"female | 2008 | -0.26 | 0.06 | [-0.37, -0.15] | -4.47 | < .001",
"female | 2009 | -0.24 | 0.10 | [-0.42, -0.05] | -2.48 | 0.013",
"male | 2007 | -0.46 | 0.10 | [-0.65, -0.26] | -4.60 | < .001",
"male | 2008 | -0.28 | 0.06 | [-0.40, -0.15] | -4.41 | < .001",
"male | 2009 | -0.10 | 0.10 | [-0.30, 0.11] | -0.93 | 0.355",
"",
"Variable predicted: bill_dep",
"Predictors contrasted: bill_len_between, bill_len_within",
Expand All @@ -86,9 +86,9 @@ test_that("estimate_contrast, context effects, linear", {
"",
"Level1 | Level2 | year | Difference | SE | 95% CI | z | p",
"--------------------------------------------------------------------------",
"male | female | 2007 | 0.17 | 0.13 | [-0.08, 0.43] | 1.33 | 0.183",
"male | female | 2008 | 0.02 | 0.09 | [-0.15, 0.18] | 0.18 | 0.853",
"male | female | 2009 | -0.14 | 0.14 | [-0.42, 0.14] | -0.99 | 0.320",
"male | female | 2007 | -0.17 | 0.13 | [-0.43, 0.08] | -1.33 | 0.183",
"male | female | 2008 | -0.02 | 0.09 | [-0.18, 0.15] | -0.18 | 0.853",
"male | female | 2009 | 0.14 | 0.14 | [-0.14, 0.42] | 0.99 | 0.320",
"",
"Variable predicted: bill_dep",
"Predictors contrasted: bill_len_between, bill_len_within",
Expand All @@ -107,7 +107,7 @@ test_that("estimate_contrast, context effects, glm", {
se <- coef(summary(m))[2:3, 2]

out <- estimate_contrasts(m, c("bill_len_between", "bill_len_within"))
expect_equal(out$Odds_Ratio, exp(b[2] - b[1]), tolerance = 1e-4, ignore_attr = TRUE)
expect_equal(out$Odds_Ratio, exp(b[1] - b[2]), tolerance = 1e-4, ignore_attr = TRUE)
expect_true(!is.null(out$p))

output <- capture.output(out)
Expand All @@ -118,5 +118,5 @@ test_that("estimate_contrast, context effects, glm", {
c("bill_len_between", "bill_len_within"),
predict = "response"
)
expect_equal(out$Probability, -0.01784138, tolerance = 1e-4, ignore_attr = TRUE)
expect_equal(out$Probability, 0.01784138, tolerance = 1e-4, ignore_attr = TRUE)
})
24 changes: 12 additions & 12 deletions vignettes/practical_context_effect.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ estimate_contrasts(mixed, c("phq4_within", "phq4_between")) |>
display(format = "tt")
```

The output shows a significant contrast of 2.41 between the within- and between-effects. Since the between-effect in our model (-6.13) is stronger (more negative) than the within-effect (-3.72), the context-effect (Between minus Within) is -2.41.
The output shows a significant contrast of -2.41 between the within- and between-effects. Since the between-effect in our model (-6.13) is stronger (more negative) than the within-effect (-3.72), the context-effect (Between minus Within) is -2.41.

**What does this mean practically?**

Expand All @@ -140,9 +140,9 @@ model_parameters(mixed, effects = "fixed") |> display(format = "tt")

The results table now shows us whether time has a moderating influence on our effects. To see this, we look at the two interaction terms at the bottom of the table:

- Interaction `time × phq4_within`: The coefficient (0.33) is small and not statistically significant (p = 0.592). This means that the negative effect of individual, temporary fluctuations remains largely stable over time. When a patient's psychological burden acutely increases, their quality of life decreases to a very similar extent at any given time point.
- Interaction `time × phq4_within`: The coefficient (0.33) is small and not statistically significant ($p = 0.592$). This means that the negative effect of individual, temporary fluctuations remains largely stable over time. When a patient's psychological burden acutely increases, their quality of life decreases to a very similar extent at any given time point.

- Interaction `time × phq4_between`: This coefficient (-0.66) is also not statistically significant (p = 0.077). The negative sign suggests that the gap between patients potentially widens slightly over time. The "disadvantage" in quality of life experienced by patients with a generally high psychological baseline burden might further increase over the course of the study compared to less burdened patients.
- Interaction `time × phq4_between`: This coefficient (-0.66) is also not statistically significant ($p = 0.077$). The negative sign suggests that the gap between patients potentially widens slightly over time. The "disadvantage" in quality of life experienced by patients with a generally high psychological baseline burden might further increase over the course of the study compared to less burdened patients.

These temporal dynamics can be best understood visually using Estimated Marginal Means. The generated plots confirm the statistical results from the table very clearly.

Expand Down Expand Up @@ -186,9 +186,9 @@ estimate_contrasts(

The contrast analysis reveals a clear and interesting trajectory: the context effect grows substantially stronger as time progresses.

- *At Time 1 (Baseline):* The difference between the within- and between-effect is relatively small (1.32) and not statistically significant (p = 0.188). This indicates that at the beginning of the observations, it does not matter much whether a patient's psychological burden is acute (a temporary spike) or chronic (a generally high baseline). The immediate impact on their Quality of Life is very similar.
- *At Time 1 (Baseline):* The difference between the within- and between-effect is relatively small (-1.32) and not statistically significant ($p = 0.188$). This indicates that at the beginning of the observations, it does not matter much whether a patient's psychological burden is acute (a temporary spike) or chronic (a generally high baseline). The immediate impact on their Quality of Life is very similar.
Comment thread
strengejacke marked this conversation as resolved.

- *At Time 2 and Time 3:* As the study progresses, the contrast becomes highly significant and the gap widens (increasing to 2.31, then 3.29).
- *At Time 2 and Time 3:* As the study progresses, the contrast becomes highly significant and the gap widens (an absolute increase to 2.31, then 3.29).

**What does this mean practically?**

Expand All @@ -206,7 +206,7 @@ estimate_contrasts(
display(format = "tt")
```

This pairwise comparison table adds a crucial statistical caveat to our visual and descriptive observations. The Difference column here represents the mathematical change in the size of the context effect between two specific time points (e.g., the context effect grew by 0.99 points from Time 1 to Time 2).
This pairwise comparison table adds a crucial statistical caveat to our visual and descriptive observations. The Difference column here represents the mathematical change in the size of the context effect between two specific time points (e.g., the context effect decreased by 0.99 points from Time 1 to Time 2).

However, looking at the statistics, you will notice that the estimated difference between any two adjacent time points is exactly identical, yielding the exact same standard error and p-value.

Expand All @@ -221,7 +221,7 @@ estimate_contrasts(mixed, c("phq4_within", "phq4_between")) |>

**What does this mean practically?**

The highly significant overall contrast (2.31, p < .001) confirms that a substantial context effect is at play throughout the entire observation period.
The highly significant overall contrast (-2.31, $p < .001$) confirms that a substantial context effect is at play throughout the entire observation period.

In clinical terms, while an acute, temporary spike in psychological symptoms (the within-effect) definitely harms a patient's quality of life, carrying a chronically high baseline burden (the between-effect) takes a significantly heavier toll. A patient who generally suffers from high psychological distress faces an overarching "trait penalty" that consistently depresses their quality of life more severely than mere day-to-day fluctuations would suggest.

Expand All @@ -246,9 +246,9 @@ estimate_contrasts(

The marginal contrasts analysis yields nuanced results that add an important layer to our understanding of the context effect:

- *Low Education:* The contrast (1.69) is not statistically significant (p = 0.192). For these patients, there is no meaningful difference between an acute symptom spike and a chronically high baseline. Both states depress their quality of life similarly.
- *Mid Education:* The contrast (3.92) is large and highly significant (p < .001). This group actually drives the overall context effect we observed in the previous models. For middle-educated patients, a chronically high psychological burden carries a massive additional penalty compared to a temporary acute spike.
- *High Education:* Interestingly, the contrast reverses its sign (-1.76) but is not statistically significant (p = 0.337). This indicates that for highly educated patients, the context effect disappears entirely.
- *Low Education:* The contrast (-1.69) is not statistically significant ($p = 0.192$). For these patients, there is no meaningful difference between an acute symptom spike and a chronically high baseline. Both states depress their quality of life similarly.
- *Mid Education:* The contrast (-3.92) is large and highly significant ($p < .001$). This group actually drives the overall context effect we observed in the previous models. For middle-educated patients, a chronically high psychological burden carries a massive additional penalty compared to a temporary acute spike (their quality of life is decreasing by 3.92 points).
- *High Education:* Interestingly, the contrast reverses its sign (1.76) but is not statistically significant ($p = 0.337$). This indicates that for highly educated patients, the context effect disappears entirely.

**What does this mean practically?**

Expand All @@ -274,8 +274,8 @@ estimate_contrasts(

The output now displays the mathematical difference in the size of the context effect between two specific groups.

- *Mid vs. Low (p = 0.154) & High vs. Low (p = 0.124):* The differences involving the low-education group are not statistically significant. The confidence intervals are quite wide, suggesting high variance or a smaller sample size within this specific intersection of the data.
- *High vs. Mid (Difference = -5.69, p = 0.005):* This is the crucial finding. The context effect for the highly educated group is significantly smaller (by 5.69 points) than for the middle-educated group.
- *Mid vs. Low ($p = 0.154$) & High vs. Low ($p = 0.124$):* The differences involving the low-education group are not statistically significant. The confidence intervals are quite wide, suggesting high variance or a smaller sample size within this specific intersection of the data.
- *High vs. Mid (Difference = 5.69, $p = 0.005$):* This is the crucial finding. The context effect for the highly educated group is significantly larger (by 5.69 points) than for the middle-educated group.

**What does this mean practically?**

Expand Down
Loading