Skip to content

Add ScanMalware expansion module - #803

Open
jonaslejon wants to merge 1 commit into
MISP:mainfrom
jonaslejon:feat/scanmalware-expansion
Open

Add ScanMalware expansion module#803
jonaslejon wants to merge 1 commit into
MISP:mainfrom
jonaslejon:feat/scanmalware-expansion

Conversation

@jonaslejon

Copy link
Copy Markdown

New expansion + hover module: ScanMalware

Adds misp_modules/modules/expansion/scanmalware.py, enriching domain, hostname, url,
ip-src, ip-dst and domain|ip attributes from the public
ScanMalware API, returning misp_standard results built with PyMISP.

No API key. The API is anonymous and a key only raises the rate limit
(600 req/min anonymous), so moduleconfig is ["api_url", "max_results"] and
requirements is empty. The module works on a fresh install with nothing configured.

Behaviour

  • A domain or hostname is queried against two different populations, not one.
    /api/v1/ct/dns/ returns subdomains seen in Certificate Transparency DNS records;
    /api/v1/hosts/ returns hosts a browser actually resolved and requested while rendering
    pages on that domain. The second is what certificate-based sources structurally cannot
    see: dev and staging hosts that hold no certificate. Querying only the first loses them.
  • A URL returns the most recent completed sandbox scan of its host, with the verdict, a
    link to the full report, and the IPs contacted while rendering.
  • An IP returns the URLs in the public archive that resolved to that address.
  • Results are capped, and the cap says so. max_results defaults to 200. Unbounded,
    cloudflare.com returns 1,207 attributes, which makes an event unusable. The cap and
    any truncation the API itself reports are both added as visible text attributes rather
    than applied silently, so an analyst can tell a shortened list from a complete one.
  • Only the queried indicator leaves the MISP instance. Requests carry a
    scanmalware-misp/1.0 User-Agent.

Measured output

domain  hkr.se            ->  70 attributes  (domain 70)
domain  cloudflare.com    -> 201 attributes  (domain 200 + the cap notice)
url     https://wpsec.com/->  82 attributes  (verdict text, report link, 80 contacted IPs)
ip-src  172.67.216.2      ->   3 attributes  (archive summary + resolved hosts)

Tests

tests/test_scanmalware.py, 12 tests, no network access. They cover the cases that would
otherwise degrade quietly: wildcard certificate entries (*.example.com) being dropped
rather than emitted as hosts, truncation and the result cap both being reported, a URL
being reduced to its host before lookup, an unreachable API degrading instead of raising,
and no auth header ever being sent.

Clean under the repository's own black (line length 120, preview,
string_processing), isort and flake8 settings.

Note on documentation: I deliberately have not committed regenerated
documentation/ output. Running documentation/generate.py locally drops every module
whose optional dependencies are missing, which produced a 4,600-line deletion across
unrelated modules. Happy to add the generated entries if you would like them in this PR,
but it seemed safer to leave that to a run with the full dependency set. The logo is
included since moduleinfo references it.


Disclosure: I run ScanMalware, so this is a vendor-submitted module. Flagging it up
front rather than leaving it to be found in review.

Queries the public ScanMalware API for a domain, hostname, URL or IP attribute.
No API key is required: the API is anonymous, and a key only raises the rate
limit.

For a domain or hostname it returns two different populations, subdomains seen in
Certificate Transparency DNS records and hosts a browser actually resolved and
requested while rendering pages on that domain. Dev and staging hosts that hold
no certificate appear only in the second.

For a URL it returns the most recent sandboxed scan of that host with its verdict
and a link to the report, and for an IP the URLs in the public archive that
resolved to that address.

Results are capped at max_results attributes, 200 by default, because a busy
domain can return well over a thousand subdomains. The cap and any truncation
reported by the API are both surfaced as attributes rather than applied silently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant