Ever since Google started using HTTPS as a ranking signal in the hopes of making the Internet safer, you can find more and more websites and online services starting to use HTTPS. HTTPS is safe in the sense that when you visit a website that uses the secure HTTPS protocol instead of HTTP, anyone such as your Internet Service Provider or network administrator that runs a packet sniffer on the network level cannot see what exactly you are doing.
Visiting a website that uses HTTPS on all pages is definitely good but there is a small problem whereby you won’t be able to obtain the direct download links from a web installer that downloads from a HTTPS URL. An example is 2 years ago you were able to use URLSnooper, HTTPNetworkSniffer, Free HTTP Sniffer, Socket Sniffer or PacketViewer to display the hidden remote URL for Adobe Flash Player Installer but these tools no longer work because the installer downloads from an encrypted HTTPS source, rather than HTTP.
As you can see from the screenshot above, NetLimiter shows that the Adobe Download Manager connects to port 443 which is used for the encrypted HTTPS protocol (HTTP uses port 80). In this article we’ll show you some tools that can decrypt the encrypted HTTPS traffic so that you get the same level of details as HTTP.
If you want to obtain the direct download link from a webpage, you can get it directly from the web browser without relying on third party tools. Let’s look at downloading iTunes from the official Apple’s website as an example. There is a “Download Now” button but it isn’t directly linked to the setup installer file using the standard HREF HTML tag. What you can do is start downloading the file, then press Ctrl+J to bring up the window that shows the progress of the download.
You can right click on the URL and select copy download link. Fortunately this method is quite standard for Firefox, Chrome, Opera, Internet Explorer and Microsoft Edge.
For some software that uses a small front end executable to download the real full installer, you will need a HTTP debugging tool that supports HTTPS protocol.
Fiddler is a free HTTP debugger which listens to the network adapters on your computer and automatically captures HTTP(s) traffic for analysis. There are 2 versions of Fiddler available to download but they are actually the same in terms of features. Although it is mentioned that users of Windows 8+ should choose Fiddler4, you can still run it on Windows 7 if .NET Framework version 4 is installed on the operating system.
By default Fiddler does not decrypt HTTPS traffic and it has to be enabled in the options. Run Fiddler, click on Tools at the menubar and select “Fiddler Options“. Click on the HTTPS tab and tick the “Decrypt HTTPS traffic” checkbox. You must install the Fiddler Root Certificate for it to work so you must click the Yes button on a few windows for the installation to be successful.
Once that is done, Fiddler can now capture and decrypt HTTPS traffic. Simply start downloading and look at the HTTPS column in Fiddler that points to an executable file extension. To copy the URL, right click on the row, go to Copy and choose “Just Url” or alternatively press Ctrl+U.
2. HTTP Debugger
HTTP Debugger is a shareware that cost $49.95 for a single user license but they offer a 14 days trial version without limitations. Unlike Fiddler that needs to install a root certificate to decode HTTPS traffic, HTTP Debugger works out of the box and can start capturing and decrypting HTTPS sessions by running the program.
You can easily copy the direct download link by right clicking on the row and select “Copy URL”. What we like about HTTP Debugger is it’s easy to use and has a clean and modern graphical user interface.
3. HTTP Analyzer
HTTP Analyzer is created by a company from China and has been around for quite a while. They started off with a plugin for Internet Explorer that allows you to inspect and edit the live HTML DOM at any web page and has slowly evolved into a standalone application that can sniff HTTP and HTTPS protocols from any running applications.
HTTP Analyzer is also shareware and cost a whopping $109 for a non-commercial stand alone license but a free trial version is available to use for 15 days. The usage is similar to HTTP Debugger where you only need to run the program and it will start sniffing for HTTP and HTTPS traffic from any source.
The screenshot above shows the captured full HTTPS URL from the adobe download manager. Copying the URL is as easy as right clicking on the cell and select “Copy Cell https://…”
Final Notes: Fiddler would obviously be your first choice because it is free. However we experienced some hiccups such as taking very long time to connect and sometimes doesn’t even connect at all to HTTPS URLs even after successfully installing the root certificate. Comparing the freeware Fiddler with shareware HTTP Debugger and HTTP Analyzer, the latter are much more stable and just work out of the box.