Using httpx and EyeWitness

Written By Indrajeet Bhuyan

In my previous articles, I have talked about different tools which will help you find more flaws in the target website. Today in this article we will explore not one but two tools. Both the tools will help us increase our attack surface and also do things fast. 

Often when we do bug bounty or external pentesting, we get our scope something like this: *.example.com. What this means is that all the subdomains under example.com are valid for the scope. 

Here is the bugcrowd page of tesla and as you can see in the scope they have mentioned *.tesla.com which means that as a bug hunter we can find flaws in any of their subdomain and report. Whenever we see something like this, we start looking for subdomains of the website using tools like assetfinder, sublist3r, crt.sh, etc. These tools do a great job of finding most of the subdomains. 

Here I have used a tool called assetfinder to get the subdomains of tesla.com. You can use other tools as well as online services like crt.sh 

The Problem 

The problem with working with subdomains is that as you can see in this example of Tesla, we have 517 subdomains from assetfinder. If we use more tools, we might end up getting more subdomains. But we need to understand that not all subdomains are alive. 

As these subdomain finder tools use various services to check for subdomains they don’t check if they are alive or not and it becomes a tedious task to check all the subdomains one by one and check which subdomain is alive and which is not. 

The solution  

To solve this issue we have a great tool called httpx. Httpx is a tool made by Project Discovery which provides a fast and multi-purpose HTTP toolkit that allows running multiple probers using retriable library. It is designed to maintain the result reliability with increased threads. 

Supported probes of Httpx : 

Probes	Default check	Probes	Default check
URL	true	IP	true
Title	true	CNAME	true
Status Code	true	Raw HTTP	false
Content Length	true	HTTP2	false
TLS Certificate	true	HTTP Pipeline	false
CSP Header	true	Virtual host	false
Line Count	true	Word Count	true
Location Header	true	CDN	false
Web Server	true	Paths	false
Web Socket	true	Ports	false
Response Time	true	Request Method	true
Favicon Hash	false	Probe Status	false
Body Hash	true	Header Hash	true
Redirect chain	false	URL Scheme	true
JARM Hash	false	ASN	false

Installing httpx is very simple. You will need go 1.17 to install it successfully. Run the following command to install it : 

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest 

We found 517 subdomains of Tesla in our previous example, but we don’t know how many of them are alive. Now, let us run httpx to see how many subdomains are alive: 

cat tesla.txt| httpx > test-alive.txt 

Earlier we saved all the subdomain lists in tesla.txt and not we piped tesla.txt to httpx and saved the response in another file called teslalive 

Once done let us see the number of subdomains it returns: 

As you can see out of the 517 subdomains that assetfinder found for us, only 117 are alive. This is the power of httpx. In this example, I have shown just a simple httpx use case. httpx also has a few other features like discovering vhosts, finding status codes, and many more silent features. 

Now after running httpx we get the list of alive domains. In our case, it is 117 domains. Now, this further leads to another problem that is faced by a lot of people.  

The Problem (2) 

When testing for a target like Tesla or some similar big target we get a long list of alive subdomains. In our case, we found 117 alive subdomains. It is a time-consuming task to open all the sites one by one and see what exactly the site is doing or which site we should target first. 

Even though as a penetration tester we should check all the subdomains but if somehow we can categorize the subdomains based on high-value targets and low-value targets then it can help us a lot in testing. What I mean by the high-value target is that suppose one of the sites is a login form then it can be a high-value subdomain as we can try various ways to bypass the login form and get inside.

The Solution 

For this problem, we can use a tool called as Eyewitness. Eyewitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible. 

Setup: 

• Navigate into the Python/setup directory 

• Run the setup.sh script 

Usage: 

./EyeWitness.py -f filename –timeout optionaltimeout 

Here I ran this command to get screenshot of Tesla subdomains: 

./EyeWitness -f /home/kali/teslalive 

Here is what the final report looks like : 

As you can see, we have a subdomain with a login button. So maybe we can make this subdomain our priority while testing.I hope you now understand httpx and Eyewitness and how we can use these two tools together to help us in our testing.