URL and domain functions
Split a URL or host name into its parts and score it, for DGA domains, DNS tunnels and lookalike domains.
url_host, url_domain, url_tld, url_subdomain and the other url_* functions, plus shannon for entropy and levenshtein for edit distance. They run in ClickHouse, in eval, where and detection rules, and replace Splunk URL Toolbox.Functions
url_tld gives co.uk, the end of url_domain. url_host gives www.mail.example.co.uk, the subdomain and the domain together.
| Function | Returns | Example |
|---|---|---|
url_scheme(url) | The scheme; http when the value has none | https |
url_host(url) | The host name, in lower case | www.mail.example.co.uk |
url_port(url) | The port, 0 when the URL has none | 8443 |
url_path(url) | The path | /login |
url_query(url) | The query string, without ? | next=/inbox |
url_fragment(url) | The fragment, without # | top |
url_domain(url) | The registered domain; an IP address stays the address | example.co.uk |
url_tld(url) | The public suffix | co.uk |
url_subdomain(url) | Everything in front of the registered domain | www.mail |
shannon(str) | Shannon entropy in bits per character, 0 for an empty value | shannon("xk2j9qz7vb") is about 3.3 |
levenshtein(a, b) | Edit distance, case-insensitive | levenshtein("micros0ft.com", "microsoft.com") is 1 |
The url_* functions take a full URL, a bare host or a DNS query. A value without a scheme, such as evil.com/a?b=1, is read as http://evil.com/a?b=1.
Examples
Random-looking domains (DGA)
Random names score about 3.5 bits per character or more. Short names score low anyway, so set a minimum length.
earliest=-24h index=dns
| eval domain=url_domain(query), label=replace(domain, "\..*$", "")
| where shannon(label) >= 3.5 AND len(label) >= 12
| stats count by src, domain
| sort - countLookalike domains
Domains one or two edits away from your own, such as examp1e.com:
earliest=-24h index=dns
| eval domain=url_domain(query)
| where levenshtein(domain, "example.com") <= 2 AND domain != "example.com"
| stats count by src, domainDNS tunnels
Many long, unique subdomains under one domain point to data sent out over DNS:
earliest=-1h index=dns
| eval domain=url_domain(query), sub=url_subdomain(query)
| where len(sub) > 40
| stats dc(sub) as unique_subdomains by src, domain
| where unique_subdomains > 100Web traffic by host and path
For proxy and secure web gateway logs, where url holds the full address. A web server access log only has the path there.
earliest=-24h index=web
| eval host=url_host(url), path=url_path(url)
| stats count by host, path
| sort - countThe thresholds are a starting point; tune them on your own data. See LPQL syntax for the rest of the language and field names for query, src and url.
From Splunk URL Toolbox
The Splunk migration converts the URL Toolbox macros to eval with these functions. The ut_* field names stay, so the rest of the search works as it is.
| Splunk macro | Becomes |
|---|---|
ut_parse_simple(url) | eval ut_scheme, ut_netloc, ut_path, ut_query, ut_fragment |
ut_parse(url), ut_parse_extended(url) | the same, plus ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain, ut_subdomain_count, ut_port |
ut_shannon(word) | eval ut_shannon=shannon(word) |
ut_levenshtein(a, b) | eval ut_levenshtein=levenshtein(a, b) |
ut_meaning, ut_bayesian, ut_suites, ut_countset | not yet; the converter reports them |
Good to know
url_domain uses the suffix list built into ClickHouse. It knows the common second levels such as co.uk and com.au, not every rule of the Public Suffix List. For rare suffixes the result can differ from URL Toolbox.url_port returns 0 when the URL has no port; URL Toolbox returned 80. For an IP address, url_domain returns the address itself and url_tld is empty.Frequently asked questions
- Does LogPulse have an alternative to Splunk URL Toolbox?
- Yes. LPQL has url_scheme, url_host, url_port, url_path, url_query, url_fragment, url_domain, url_tld and url_subdomain, plus shannon and levenshtein. They work in eval, where and detection rules, without a lookup or an app.
- How do I find DGA domains in LogPulse?
- Take the first label of the registered domain and score it with shannon: random names score about 3.5 bits per character or more. For example: index=dns | eval label=replace(url_domain(query), "\..*$", "") | where shannon(label) >= 3.5 AND len(label) >= 12.
- How do I find lookalike domains?
- Compare the registered domain with your own domain: where levenshtein(url_domain(query), "example.com") <= 2 AND url_domain(query) != "example.com".
- Do Splunk searches with ut_parse_extended convert?
- Yes. The Splunk migration converts ut_parse, ut_parse_simple, ut_parse_extended, ut_shannon and ut_levenshtein to eval with these functions and keeps the ut_* field names. ut_meaning, ut_bayesian, ut_suites and ut_countset are not converted yet.
- Does url_domain use the full Public Suffix List?
- No. It uses the suffix list built into ClickHouse, which knows the common second levels such as co.uk and com.au. For rare suffixes the result can differ from URL Toolbox.