Scoping / Documentation / Context issues -- How to be specific?

[J-Moravec]

Hello,

I have been trying to find a good documentation on how to avoid potential scoping issues.

For instance, consider

library("data.table")
dt = data.table(a = 1:3, b = 4:6)
dt[a %in% 1:3]

So far so good, a is interpreted within the dt. But what if we set a = 4:6?
Documentation often talks about the . and .. calls, but those are specific to j and not i part of data.table.

a = 4:6
dt[a %in% a]

obviously wouldn't have desired effect.

But the Introduction to data.table doesn't really offer a solution.
https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html

Most stackoverflow or AI answers are incorrect, often they will say to use the get() solution, but it doesn't work either.

The solution is introduced in https://cran.r-project.org/web/packages/data.table/vignettes/datatable-programming.html, use the env = list(...), such as:

dt[col %in% value, env = list(col = "a", value = a)

Yet, in some cases, wrapping in I is also required if the value is of character, because it would be interpreted as a column.

dt[] = lapply(dt, as.character)
a = "1"
dt[col %in% value, env = list(col = "a", value = a))] # fails because "1" is not a column
dt[col %in% value, env = list(col = "a", value = I(a))] # works
dt[col %in% value, env = list(col = as.name("a"), value = I(a))] # safest?

All these things are imo required when he user wants to be specific, like this variable names a column in a data.table, while this_one names a variable comming from outer environment. This way wires won't be crossed.

This should IMO be in the "introduction to data.table" as a simple case or much more complex "Programming on data.table`

Apparently, env is a new interface and get(), mget() etc. were at one point interfaces of data.table but were discontinued (likely because they were buggy).

https://stackoverflow.com/a/54800108

---

tl,dr: Add a note in "Introduction to data.table" about the env interface and how to pass column names/values as variables.

[jangorecki]

Majority of dt users do not need to parametrize columns, so I would not say it is so important in introduction, but of course mentioning would be still very useful.

"safest?" is not needed.

What I also do is to prefix env list names with a dot to reduce chance of collisions with existing columns, like column called col or value.

Using value is completely optional here, you can pass it directly in the expression in i arg, then you avoid need for using I().

[J-Moravec]

What I also do is to prefix env list names with a dot to reduce chance of collisions with existing columns, like column called col or value.

In our data scheme, column starting with dot are unfortunately allowed. Wouldn't env or wrappng with I() or as.name() prevent potential name clashes. Just trying to avoid potential future issues with name clashing.

Using value is completely optional here, you can pass it directly in the expression in i arg, then you avoid need for using I().

I don't understand this.

Are you saying that substituting using the value as in:

dt[a %in% value, env = list(value = I(a))]

is optional, and I can just use:

dt[a %in% a]

Obviously, I can't just put it there because the context will be interpreted differently.

My two aims are:

1. Prevent name clash where the same column exists as a variable I am trying to use
2. Prevent name clash, where a column is requested that does not exist in a table, but a value in a local environment is used instead.

[MichaelChirico]

In our data scheme, column starting with dot are unfortunately allowed.

Actually I think that's perfect, 0 chance of collision then, as I understood Jan meant dt[a %in% .a, env = list(.a = a)].

[J-Moravec]

Actually I think that's perfect, 0 chance of collision then, as I understood Jan meant dt[a %in% .a, env = list(.a = a)].

Isn't that specifically documented as requiring I() if the variable a is a string?

dt = data.table(a = as.character(1:3), b = as.character(4:6))
a = "1"
dt[a %in% .a, env = list(.a = a)]
Error in eval(stub[[3L]], x, enclos) : object '1' not found

# compare with:
dt[a %in% .a, env = list(.a = I(a)]
        a      b
   <char> <char>
1:      1      4

This has the disadvantage of not working with NULL because NULL can't have attributes btw. (which is an issue for me that I am trying to figure how to solve nicely with a single query for multiple subsetting)

[jangorecki]

In our data scheme, column starting with dot are unfortunately allowed.

Actually I think that's perfect, 0 chance of collision then, as I understood Jan meant dt[a %in% .a, env = list(.a = a)].

yes, exactly that.

Using value is completely optional here, you can pass it directly in the expression in i arg, then you avoid need for using I().

I don't understand this.

dt = data.table(a = as.character(1:3), b = as.character(4:6))
.a = "1"
dt[a %in% .a]

My two aims are:

1. Prevent name clash where the same column exists as a variable I am trying to use
2. Prevent name clash, where a column is requested that does not exist in a table, but a value in a local environment is used instead.

none of those are data.table specific

1. add a prefix or suffix to your variable
2. a) ensure that column you are requesting exists in data: stopifnot("a" %in% names(dt))
   b) e <- new.env(parent = baseenv()); e$dt <- dt; eval(quote(dt[a %in% .a]), env = e)

---

Purpose of env is to be able to substitute names (columns, functions) by providing them as character variables. Values do not need to be substituted as they can be passed directly.