Add ScanMalware expansion module - #803
Open
jonaslejon wants to merge 1 commit into
Open
Conversation
Queries the public ScanMalware API for a domain, hostname, URL or IP attribute. No API key is required: the API is anonymous, and a key only raises the rate limit. For a domain or hostname it returns two different populations, subdomains seen in Certificate Transparency DNS records and hosts a browser actually resolved and requested while rendering pages on that domain. Dev and staging hosts that hold no certificate appear only in the second. For a URL it returns the most recent sandboxed scan of that host with its verdict and a link to the report, and for an IP the URLs in the public archive that resolved to that address. Results are capped at max_results attributes, 200 by default, because a busy domain can return well over a thousand subdomains. The cap and any truncation reported by the API are both surfaced as attributes rather than applied silently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New expansion + hover module: ScanMalware
Adds
misp_modules/modules/expansion/scanmalware.py, enrichingdomain,hostname,url,ip-src,ip-dstanddomain|ipattributes from the publicScanMalware API, returning
misp_standardresults built with PyMISP.No API key. The API is anonymous and a key only raises the rate limit
(600 req/min anonymous), so
moduleconfigis["api_url", "max_results"]andrequirementsis empty. The module works on a fresh install with nothing configured.Behaviour
/api/v1/ct/dns/returns subdomains seen in Certificate Transparency DNS records;/api/v1/hosts/returns hosts a browser actually resolved and requested while renderingpages on that domain. The second is what certificate-based sources structurally cannot
see: dev and staging hosts that hold no certificate. Querying only the first loses them.
link to the full report, and the IPs contacted while rendering.
max_resultsdefaults to 200. Unbounded,cloudflare.comreturns 1,207 attributes, which makes an event unusable. The cap andany truncation the API itself reports are both added as visible text attributes rather
than applied silently, so an analyst can tell a shortened list from a complete one.
scanmalware-misp/1.0User-Agent.Measured output
Tests
tests/test_scanmalware.py, 12 tests, no network access. They cover the cases that wouldotherwise degrade quietly: wildcard certificate entries (
*.example.com) being droppedrather than emitted as hosts, truncation and the result cap both being reported, a URL
being reduced to its host before lookup, an unreachable API degrading instead of raising,
and no auth header ever being sent.
Clean under the repository's own
black(line length 120, preview,string_processing),isortandflake8settings.Note on documentation: I deliberately have not committed regenerated
documentation/output. Runningdocumentation/generate.pylocally drops every modulewhose optional dependencies are missing, which produced a 4,600-line deletion across
unrelated modules. Happy to add the generated entries if you would like them in this PR,
but it seemed safer to leave that to a run with the full dependency set. The logo is
included since
moduleinforeferences it.Disclosure: I run ScanMalware, so this is a vendor-submitted module. Flagging it up
front rather than leaving it to be found in review.