Guard against cyclic trees in process_tag/process_element (#256) - #274
Open
chiliec wants to merge 1 commit into
Open
Guard against cyclic trees in process_tag/process_element (#256)#274chiliec wants to merge 1 commit into
chiliec wants to merge 1 commit into
Conversation
…hanm#256) The mutual recursion between process_tag and process_element has no cycle guard. A well-formed BeautifulSoup tree is acyclic, but some HTML producers (e.g. PDF-to-HTML pipelines feeding marker-pdf) can yield a graph where a descendant references an ancestor. Converting such input recurses without bound and crashes with RecursionError. Track the ids of the tags on the current descent path (an optional _visited argument, defaulted to None so existing callers and custom converters are unaffected) and return '' when a tag is re-encountered, breaking the cycle. A fresh copy is threaded per level, so a node legitimately reached via two sibling paths is still processed each time. Adds a regression test. Fixes matthewwithanm#256.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #256.
Problem
The mutual recursion introduced in 1.2.2 between
process_tagandprocess_elementhas no cycle guard. A well-formed BeautifulSoup tree is acyclic, but some HTML producers — notably PDF-to-HTML pipelines (the reporter hit this viamarker-pdf) — can yield a graph where a descendant node references an ancestor. Converting such input recurses without bound and crashes:Because it aborts with an unhandled exception, a single malformed document takes down the whole conversion.
Fix
Thread an optional
_visitedset of tagid()s through the descent. When a tag already on the current path is re-encountered, return `` to break the cycle instead of recursing into it._visiteddefaults toNone, so existing callers and any custom converters that callprocess_tag/process_elementdirectly are unaffected._visited | {node_id}), so it only detects a node repeating on the current descent path — a node legitimately reached via two separate sibling branches is still processed each time.Test
Adds
test_cyclic_tree_does_not_recurse, which builds a cyclic tree (a<p>whose contents include its own ancestor<div>), lowers the recursion limit, and asserts conversion completes withoutRecursionError.Verification (local, Python 3.11)
RecursionError; with the fix it passes.pytest tests/— full suite passes (84 tests, up from 83), no regressions.flake8 --ignore=E501,W503 markdownify tests(astoxruns it) is clean.