Skip to content

Lexer/parser: Pine line-wrapping rule + dangling Dedents between call arguments#50

Open
FellowTraveler wants to merge 2 commits into
ferranbt:mainfrom
FellowTraveler:pine-v6-syntax-fixes
Open

Lexer/parser: Pine line-wrapping rule + dangling Dedents between call arguments#50
FellowTraveler wants to merge 2 commits into
ferranbt:mainfrom
FellowTraveler:pine-v6-syntax-fixes

Conversation

@FellowTraveler

Copy link
Copy Markdown

Summary

This is the lexer/parser slice of #49, split out (as offered there) so each piece can be reviewed on its own. The code is unchanged from #49 — the commits are cherry-picked as-is. The three split PRs are independent: each compiles and passes the full workspace suite standalone (cargo test, just fmt-check, just clippy all clean on this branch), and any merge order is conflict-free.

Both fixes were found while running real published Pine strategies through pinecone bar-by-bar.

1. fix(parser): consume dangling Dedents between call arguments

A named argument whose value wraps onto an indented continuation line (the common table.cell(..., bgcolor = cond ? a :\n<indent>b, ...) idiom) leaves a closing Dedent in the token stream after the comma. arguments() only skipped Newline+Indent there, so the next argument failed to parse. The fix consumes such Dedents after a comma, guarded so the argument list's own closing Dedent (immediately followed by RParen) is preserved.

Test: multiline-ternary named argument followed by another argument — fails to parse before the fix.

2. feat(lexer): Pine's line-wrapping rule

Per the Pine Script reference (Line wrapping), a statement may wrap onto following lines as long as the continuation's indentation is not a multiple of 4 spaces — multiples of 4 are reserved for delimiting local blocks. The lexer treated every deeper line start as Indent, so this canonical published-script style failed with Unexpected token: And:

can_trade = trades_this_day < 2
         and under_profit_cap
         and above_loss_limit

The fix: at line start, when the indentation is not a multiple of 4, join the line to the previous logical line (drop the Newline that ended it; leave the indent stack untouched). The existing testdata corpus contains no non-multiple-of-4 indentation, so its behavior is unchanged — verified by the full suite.

Tests: lexer token-stream tests (a wrapped continuation emits no Indent/Dedent and no Newline before the continuation's operator; a 4-space block still emits Indent/Dedent), plus parser tests for a wrapped and-chain and a wrapped ternary inside an if block.

Note: the two commits are ordered — the lexer commit's parser tests build on the Dedent fix — so they travel together in this PR.

Validation

Beyond the tests in this PR, the combined change set (this PR plus its two siblings from #49) was validated externally: we run full Pine strategies through pinecone one bar at a time and diff the emitted entries/exits against TradingView's own strategy-tester CSV export for the same symbol/timeframe. With all of them in place, two real strategies match TV 100% — 56/56 signals on a 4H strategy and 21/21 on a daily strategy, including exit-reason order comments — over their full comparison windows.

A named argument whose value wraps onto an indented continuation line
(e.g. a multiline ternary) leaves a closing Dedent in the token stream
after the comma. arguments() only skipped Newline+Indent there, so the
next argument failed to parse. Consume such Dedents after a comma,
guarded so the argument list's own closing Dedent (immediately followed
by RParen) is preserved.

Test: multiline-ternary named argument followed by another argument
(fails with IndentationError-free parse error before this fix).
… of 4)

Pine's line-wrapping spec: a statement may wrap onto following lines as
long as the continuation's indentation is NOT a multiple of 4 spaces —
4-space multiples are reserved for delimiting local blocks. The lexer
treated every deeper line start as Indent, so the canonical wrapped
style failed to parse:

    can_trade = a < 2
             and b
             and c
    -> 'Unexpected token: And'

Fix: at line start, when the indentation is not a multiple of 4, join
the line to the previous logical line — drop the Newline that ended it
and leave the indent stack untouched. Existing testdata is unaffected
(it contains no non-multiple-of-4 indentation).

Tests: lexer token-stream tests (a wrapped continuation emits no
Indent/Dedent and no Newline before the continuation's operator; a
4-space block still emits Indent/Dedent) and parser tests for a wrapped
and-chain and a wrapped ternary inside an if-block.

@ferranbt ferranbt left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Only one comment.

}

#[test]
fn test_line_wrapped_bool_expression() {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you move these two tests to the testdata folder?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants