Search

URL and domain functions

Split a URL or host name into its parts and score it, for DGA domains, DNS tunnels and lookalike domains.

In short
LPQL has url_host, url_domain, url_tld, url_subdomain and the other url_* functions, plus shannon for entropy and levenshtein for edit distance. They run in ClickHouse, in eval, where and detection rules, and replace Splunk URL Toolbox.

Functions

Which function returns which part
httpsurl_scheme
www.mailurl_subdomain
example.co.ukurl_domain
8443url_port
/loginurl_path
next=/inboxurl_query
topurl_fragment

url_tld gives co.uk, the end of url_domain. url_host gives www.mail.example.co.uk, the subdomain and the domain together.

FunctionReturnsExample
url_scheme(url)The scheme; http when the value has nonehttps
url_host(url)The host name, in lower casewww.mail.example.co.uk
url_port(url)The port, 0 when the URL has none8443
url_path(url)The path/login
url_query(url)The query string, without ?next=/inbox
url_fragment(url)The fragment, without #top
url_domain(url)The registered domain; an IP address stays the addressexample.co.uk
url_tld(url)The public suffixco.uk
url_subdomain(url)Everything in front of the registered domainwww.mail
shannon(str)Shannon entropy in bits per character, 0 for an empty valueshannon("xk2j9qz7vb") is about 3.3
levenshtein(a, b)Edit distance, case-insensitivelevenshtein("micros0ft.com", "microsoft.com") is 1

The url_* functions take a full URL, a bare host or a DNS query. A value without a scheme, such as evil.com/a?b=1, is read as http://evil.com/a?b=1.

Examples

Random-looking domains (DGA)

Random names score about 3.5 bits per character or more. Short names score low anyway, so set a minimum length.

earliest=-24h index=dns
| eval domain=url_domain(query), label=replace(domain, "\..*$", "")
| where shannon(label) >= 3.5 AND len(label) >= 12
| stats count by src, domain
| sort - count

Lookalike domains

Domains one or two edits away from your own, such as examp1e.com:

earliest=-24h index=dns
| eval domain=url_domain(query)
| where levenshtein(domain, "example.com") <= 2 AND domain != "example.com"
| stats count by src, domain

DNS tunnels

Many long, unique subdomains under one domain point to data sent out over DNS:

earliest=-1h index=dns
| eval domain=url_domain(query), sub=url_subdomain(query)
| where len(sub) > 40
| stats dc(sub) as unique_subdomains by src, domain
| where unique_subdomains > 100

Web traffic by host and path

For proxy and secure web gateway logs, where url holds the full address. A web server access log only has the path there.

earliest=-24h index=web
| eval host=url_host(url), path=url_path(url)
| stats count by host, path
| sort - count

The thresholds are a starting point; tune them on your own data. See LPQL syntax for the rest of the language and field names for query, src and url.

From Splunk URL Toolbox

The Splunk migration converts the URL Toolbox macros to eval with these functions. The ut_* field names stay, so the rest of the search works as it is.

Splunk macroBecomes
ut_parse_simple(url)eval ut_scheme, ut_netloc, ut_path, ut_query, ut_fragment
ut_parse(url), ut_parse_extended(url)the same, plus ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain, ut_subdomain_count, ut_port
ut_shannon(word)eval ut_shannon=shannon(word)
ut_levenshtein(a, b)eval ut_levenshtein=levenshtein(a, b)
ut_meaning, ut_bayesian, ut_suites, ut_countsetnot yet; the converter reports them

Good to know

Note: url_domain uses the suffix list built into ClickHouse. It knows the common second levels such as co.uk and com.au, not every rule of the Public Suffix List. For rare suffixes the result can differ from URL Toolbox.
Note: url_port returns 0 when the URL has no port; URL Toolbox returned 80. For an IP address, url_domain returns the address itself and url_tld is empty.

Frequently asked questions

Does LogPulse have an alternative to Splunk URL Toolbox?
Yes. LPQL has url_scheme, url_host, url_port, url_path, url_query, url_fragment, url_domain, url_tld and url_subdomain, plus shannon and levenshtein. They work in eval, where and detection rules, without a lookup or an app.
How do I find DGA domains in LogPulse?
Take the first label of the registered domain and score it with shannon: random names score about 3.5 bits per character or more. For example: index=dns | eval label=replace(url_domain(query), "\..*$", "") | where shannon(label) >= 3.5 AND len(label) >= 12.
How do I find lookalike domains?
Compare the registered domain with your own domain: where levenshtein(url_domain(query), "example.com") <= 2 AND url_domain(query) != "example.com".
Do Splunk searches with ut_parse_extended convert?
Yes. The Splunk migration converts ut_parse, ut_parse_simple, ut_parse_extended, ut_shannon and ut_levenshtein to eval with these functions and keeps the ut_* field names. ut_meaning, ut_bayesian, ut_suites and ut_countset are not converted yet.
Does url_domain use the full Public Suffix List?
No. It uses the suffix list built into ClickHouse, which knows the common second levels such as co.uk and com.au. For rare suffixes the result can differ from URL Toolbox.