Passive Subdomain Enumeration (Part 1)

What is sub-domain Enumeration?

Subdomain enumeration is a process of finding subdomains for one or more domains.

Why need sub-domain enumeration?

• Sub-domain enumeration helps to create a scope of security assessment by revealing domains/sub-domains of a target organization.
• Sub-domain enumeration increases the chance of finding vulnerabilities.
• The sub-domain enumeration helps us in finding the web applications that might be forgotten/left unattended by the organization for maintenance or other reasons and may lead to the disclosure of critical vulnerabilities.

Types of sub-domain enumeration

There are two types of enumeration techniques available which consist of other sub techniques.

1. Passive sub-domain enumeration

In passive sub-domain enumeration, an adversary or tester gathers the sub-domain information without directly connecting to the infrastructure managed by the organization. In this process, the adversary or tester gathers the information from third parties like, the information is gathered by Virustotal, DNSDumster, certificate, etc. Generally, any alerts of flags are not raised during such probing.

Passive sub-domain enumeration techniques:

• Certificate Transparency
• ASN Discovery
• Using Search Engines
• Using DNS aggregators (Github, Virustotal, etc)
• Subject alternate name (SAN)
• Using public datasets
• DNS Enum using Cloudflare

2. Active sub-domain enumeration

In an active sub-domain enumeration, the adversary or tester gathers the information by directly probing the infrastructure managed by the organization. In an active enumeration, the detection of adversary or tester may be possible by the organization. Such kind of probing may raise alerts and/or flags.

Active sub-domain enumeration techniques:

• Brute force enumeration
• Dictionary enumeration
• Zone walking
• DNS record
• HTTP headers
• DNS cache snooping

Passive sub-domain enumeration techniques

By Certificate Transparency

Before understanding the certificate transparency, we would first need to understand the basic terminology related to a digital certificate.

What is a digital certificate?

A digital certificate is a certificate that is issued by a trusted third party which is called a certificate authority to verify the identity of the certificate sender or holder.

For example, The driving license assigned by the government authority which is a third party whom we trust. The traffic police check if the ride can or cannot ride by checking the driving license. Here in this example traffic police are like users who access the website and government authority is the certificate authority whereas the rider is the website. 

Who are certificate authorities?

Certificate authorities or CA are the globally trusted third-party companies like VeriSign, GeoTrust, GoDaddy, etc that manage three major tasks:

• Issues certificates
• Confirm the identity of the certificate owner
• Provide proof that the certificate is valid

What is a digital signature?

A digital signature is a mathematical way of verifying the authenticity of digital messages or documents.

The steps followed in creating a digital signature are:

1. The website holder generates the “public key” and “private key”.
2. The website holder sends the “public key” with some other information like holder name, subject, serial number, signature algorithm, signature, etc to the certificate authority.
3. The certificate authority verifies the data provided by the Website holder then builds the certificate and finally digitally signs it for the holder. A certificate is a document that contains necessary information about the website holder like the holder public key, expiration date, name of the certificate holder, the digital signature of the certificate-issuing authority.
4. CA send the certificate to the website holder. The holder configures the certificate in the server.

How Certificate Authority Digitally Sign the Certificate?

How the certificate works when a user accesses a website?

1. When we access the https website in the browser, let say Google, google server sends the server public key and the certificate which was signed by CA to the user.
2. Now the user browser verifies the authenticity of the certificate. All the browsers come with pre-installed globally trusted CA public keys. The browser tries to decrypt the certificate that was digitally signed by the CA public key.
3. If the browser is able to decrypt the signature (which means, it is a trustworthy website) then it proceeds to the next step else it stops and shows a red cross before the URL.
4. As told in the above steps, Google sends its public key when we enter https://www.google.com in the browser. Any data encrypted with this public key can only be decrypted by Google’s private key which Google does not share with anyone.
5. After certificate validation, the browser creates a new symmetric key let us say “Session Key” and make 2 copies of it. These keys can encrypt as well as decrypt the data.
6. The browser then encrypts one copy of the session key and other request data with Google’s public key and sends it back to the Google server.
7. Then Google’s server decrypts the encrypted data using its private key and gets the “Session Key”, and other requested data.
8. When Google sends the HTTP data to the browser, it first encrypts the data with the “Session Key” and the browser decrypts the data with the other copy of the session key because this key is a symmetric key.
9. Similarly, when the browser sends the HTTP data to the Google server it encrypts it with the “Session Key” which the server decrypts on the other side.

What is Certificate Transparency?

Certificate Transparency is the open-source framework for the Certificate Authorities (CAs) under which they log the certificates to the domain name owners. In this way, anyone can see which CA has issued a certificate for which domains. It is like the inventory of all certificates, certificate authorities, and domains.

Why we need Certificate Transparency?

1. By compromising the infrastructure of the certificate authority, the adversary can maliciously issue the certificates by the certificate authority without the consent of CA.
2. The certificate authority can mistakenly issue a certificate to the wrong owner.

The problem with the previous CAs infrastructure was that there was no effective way to audit or monitor SSL certificates in real-time. So, when any missteps or malicious activities happen, the suspect certificate was not usually detected and revoked for weeks or months. These miss issues of certificates were used to spoof a legitimate website or to install malicious software etc.

Case Study: The DigiNotar was a Dutch certificate authority that was compromised, and the adversary used the CAs system to issue 500 fake SSL certificates. In the investigation, it was discovered that the adversary issued the wildcard certificate for google.com. Which gave the adversary the ability to impersonate Google. This was widely used by the adversary to attack Gmail users in Iran.

How does Certificate Transparency help in OSINT?

Now we know that certificate transparency logs all the entries of the issued certificates in an inventory. This includes domain names, sub-domain names, and email addresses. This is publicly available to everyone. By using CT (Certificate Transparency) logs an adversary can gather basic information about the organization’s infrastructure in a passive way.

CT logs search engines:

• https://crt.sh/
• https://censys.io/
• https://developers.facebook.com/tools/ct/
• https://google.com/transparencyreport/https/ct/
• https://sslmate.com/certspotter/

Note: It may be possible that the domain/sub-domain during recon may not resolve because the domain/sub-domain names may not exist. Certificate Transparency logs are only appended and there is no way to delete these domains from CT logs.

By using Search Engines:

Search engines are one of the best techniques to find subdomains in a passive way. During my research, I found that “Site:” operator which was used to search domain and subdomains was working in the below-mentioned search engines:

Example: site:example.com

• Google
• Bing
• Yahoo
• Yandex
• Duckduckgo
• Aol

By using Online DNS Tools:

During my research I found 9 sub-domain enumeration services:

https://decoder.link/

https://searchdns.netcraft.com/

https://dnsdumpster.com/

https://www.virustotal.com/gui/home/search

https://pentest-tools.com/information-gathering/find-subdomains-of-domain#

https://findsubdomains.com/

https://hackertarget.com/find-dns-host-records/

https://www.pkey.in/tools-i/search-subdomains

https://spyse.com/

Comparison of online available subdomains enumeration services:

The subdomains number mentioned in the pie-chart were not validated, so it may be possible that these subdomains are false positive, but you can validate by using “massdns” or any other tool that resolves subdomains.

By using ASN (Autonomous System Number) number

An autonomous system is a set of interconnected networks under single administrative control. In simple term, ISP (Internet Service Provider) is an autonomous system where one or more intradomain routing protocols are used to handle the networking inside the autonomous system and only one inter-domain routing protocol which is BGP (Border Gateway Protocol) is used to handle the routing between different autonomous systems.

Autonomous System Number

An autonomous system number is a unique number that is given to an Autonomous system and which is assigned by IANA (Internet Assigned Numbers Authority). Autonomous system numbers can be public or private.

Private ASN: A private ASN is a unique identifier for an AS that is used for communicating an AS to one entity.

Public ASN: A public ASN is a unique identifier that is advertised to the public internet. This is also used to communicate between ASNs.

Every ISP buys the IP address pool and ASN number from IANA. The IP address pool and ANS number are unique. It helps to differentiate the network from other networks. For BGP connection with multiple networks, ASN is needed and for an internal autonomous system (AS) IP address is needed. BGP protocol works on the ASN number.

In the above image, when BOB sends a message to Marley from Reliance Jio ISP to BSNL ISP below are the steps that are performed for sending messages.

• When BOB sends a message to Marley, it is first delivered to Reliance Jio ISP using intradomain routing.
• Reliance Jio ISP checks the destination IP address and then search for the ASN number where the source IP address exists, here in this case Reliance Jio finds the BSNL ASN number.
• The BGP protocol is used for message delivery between both autonomous systems.
• BSNL ISP received the message and then the message is delivered to Marley.

Note: Many large network infrastructure organizations also have the ASN numbers, so it’s not necessary that only ISP has the ASN number. It usually depends on the network infrastructure in use.

How is ASN number used in domain enumeration?

• Initially, find ASN number by IP address or domain name. ASN number will help you to get the IP address pool.
• Resolve the IP address to the domain using dig, host, or any other tools. Note that IP address to the domain only resolves when the PTR record is configured in DNS configuration.

Online tools to find ASN number

 http://ip-api.com/json/[IP Address/Domain name]

https://www.radb.net/query?

https://bgp.he.net/

https://mxtoolbox.com/asn.aspx

https://hackertarget.com/as-ip-lookup/

http://whois.domaintools.com/

https://who.is/

https://asn.cymru.com/cgi-bin/whois.cgi

Online tools to find IP pool from ASN number

https://bgp.he.net/

https://mxtoolbox.com/asn.aspx

https://hackertarget.com/as-ip-lookup/

We can use WHOIS queries and Nmap script to find all the IP addresses that belong to the ASN

Example:

whois -h whois.radb.net  -- '-i origin AS36459' | grep -Eo "([0-9.]+){4}/[0-9]+" | uniq

 nmap –script targets-asn –script-args targets-asn.asn=17012

By using Subject Alternative Name (SAN)

What is Subject Alternate Name (SAN)

It is an extension to the X.509 specification that allows us to secure additional domains or subdomains by a single SSL certificate. It is also called a “multi-domain SSL certificate”.

So, now the question is what is the difference between a multi-domain SSL certificate and a wildcard SSL certificate?

Wildcard Certificate

Wildcard certificate only secures the multiple subdomains under the main domain.

Example: If issued a wildcard certificate on *.example.com then this certificate secures only subdomains which ends with “example.com”.

www.example.com

test.example.com

ftp.example.com

Multi-Domain SSL Certificate

The multi-domain SSL certificate secures up to 250 unique domain names or subdomains and that domain/subdomains names mentioned in the Subject Alternative Names (SAN) field in the certificate.

Example: If issued a multi-domain SSL certificate on “example.com” then we can add multiple other domains and subdomains in SAN to protect them by SSL.

example.com

test.com

ftp.example.com

Tools to extract domain names from SAN

OpenSSL

true | openssl s_client -connect wikimedia.org:443 2>/dev/null \

| openssl x509 -noout -text \

| perl -l -0777 -ne ‘@names=/\bDNS:([^\s,]+)/g; print join(“\n”, sort @names);’

By Python Script

https://github.com/appsecco/the-art-of-subdomain-enumeration/blob/master/san_subdomain_enum.py

Command: python san_subdomain_enum.py wikimedia.org

By using Public Dataset (Rapid7)

Project Sonar is an initiative by Rapid7. Rapid7 initiates the project Sonar, where Rapid7 performs Internet scanning to collect Internet-wide scan data and then publish the results publicly for free and some data is paid. Datasets cover areas like:

• Forward DNS
• Reverse DNS
• HTTP Responses
• SSL Certificates
• Port Scans

These datasets can be useful during an offensive engagement like in this case for subdomain enumeration. Although finding data in large datasets is very time-consuming.

Rapid7 Datasets Link: https://opendata.rapid7.com/

By using Cloudflare

Cloudflare is the network of servers that are spread all over the world to create CDN (Content delivery network) like technology. This network act as a reverse proxy server between users and the server that holds the website.

Difference between reverse proxy and forward proxy

Reverse Proxy

• A reverse proxy is a server-side concept and acting on behalf of the server/service/content producer.
• A reverse proxy is a server that is in front of the webserver/service and when the client requests for websites/services, the request is initially delivered to the reverse proxy, and then the reverse proxy communicates to the webserver/service on behalf of the server.
• A reverse proxy is implemented to increase performance and security.
• A reverse proxy is used for load balancing, SSL multiplexing, caching, application firewall, authentication, etc.

Forward Proxy

• A forward proxy is a client-side concept and acting on behalf of the requestor or service consumer.
• A forward proxy is a server that is in front of client machines and when the client requests for websites/services, the request is first delivered to the forward proxy, and then the forward proxy communicates to the webserver/service on behalf of the client.
• A forward proxy is used for content filtering, Email security, Nating, etc.

Content Delivery Network (CDN):

It is a network of distributed caching servers with a web application firewall, which are used for delivering the content in the most optimal way and improve end user’s performance.

Steps for subdomain enumeration by Cloudflare:

• Login to Cloudflare https://www.cloudflare.com/login
• Click on “Add site” to your account https://www.cloudflare.com/a/add-site
• Provide the target domain as a site you want to add.
• Wait for Cloudflare to dig through DNS data and display the results.

CONCLUSION

The blog post covers different passive techniques of the subdomain enumeration. These techniques provide information regarding all the domains/subdomains of the target organization. The information contains all those subdomains that currently may or may not be in use by the organization. As these are passive reconnaissance techniques, so, it helps adversary or attacker to gather the information without directly engaging with the target organization. It also provides an understanding of the target organization’s infrastructure and web technologies which will be helpful for further attacks.

You can read part 2 (Active Subdomain Enumeration) here.