Capturing network traffic using Selenium WebDriver/Firebug

I wanted to figure out a way to verify that the tracking beacon for Google Analytics was correctly being sent from the web page I was testing.
I figured out a couple of ways exist if you want to inspect http traffic going out from your web page is

• Setup a internal proxy server that will route your http traffic whose logs you could then inspect to verify the http request you wanted to verify.
• Use the Firebug extension along with netExport to log the traffic coming from the webpage.

Since setting up an internal proxy server was not really an option, I decided to try out the Firebug extension route and wanted to document it if it turns out to useful for somebody else trying to do the same

The first thing you will need to do is to setup the Firefox profile using the Selenium WebDriver to launch Firefox with the Firebug/Net extension enabled. You will also have to setup the profile in such a way that it dumps the http traffic log to an external file. The dumped file would be of the ‘har’ format which seems similar to json. So you can look for the traffic you want then using gson or jackson libraries.

So lets get started

Getting the Firebug/Net extension
I want to enable the extension every time my test runs, either in CI or locally. So I have committed both the extensions to source control (Our internal CI does not allow for any external communication). I then add the extensions to the Firefox profile

File firebug = new File(System.getProperty("user.dir") + "\\resources\\firebug-1.10.6.xpi");
File netExport = new File(System.getProperty("user.dir") + "\\resources\\netExport-0.9b3.xpi");

FirefoxProfile profile = new FirefoxProfile();
try {
profile.addExtension(firebug);
profile.addExtension(netExport);
} catch (IOException e) {
e.printStackTrace();
}


Update Firefox Profile
Here we will be setting other preferences to ensure that the firebug extension is enabled, is capturing HTTP traffic and will dump them in a log file


//Setting Firebug preferences
profile.setPreference("extensions.firebug.currentVersion", "2.0");
profile.setPreference("extensions.firebug.addonBarOpened", true);
profile.setPreference("extensions.firebug.console.enableSites", true);
profile.setPreference("extensions.firebug.script.enableSites", true);
profile.setPreference("extensions.firebug.net.enableSites", true);
profile.setPreference("extensions.firebug.previousPlacement", 1);
profile.setPreference("extensions.firebug.allPagesActivation", "on");
profile.setPreference("extensions.firebug.onByDefault", true);
profile.setPreference("extensions.firebug.defaultPanelName", "net");

// Setting netExport preferences
profile.setPreference("extensions.firebug.netexport.alwaysEnableAutoExport", true);
profile.setPreference("extensions.firebug.netexport.autoExportToFile", true);
profile.setPreference("extensions.firebug.netexport.Automation", true);
profile.setPreference("extensions.firebug.netexport.showPreview", false);
profile.setPreference("extensions.firebug.netexport.defaultLogDir", "C:\\workspace\\CaptureNetworkTraffic");


You can probably get away with setting just a subset of these preferences but I have not gone into doing that yet.

Launch Firefox with the desired capabilites

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);

WebDriver driver = new FirefoxDriver(capabilities);


Capture traffic

try {
Thread.sleep(10000);
driver.get("http://www.google.com");
Thread.sleep(10000);
} catch(InterruptedException ie) {
ie.printStackTrace();
}
driver.quit();


This will dump traffic into a ‘.har’ extension file at the location you have specified in the ‘extensions.firebug.netexport.defaultLogDir’ firefox preference above. The file reads similar to a json file and jackson even includes harlib which can be used to read/write har files.

Complete Java Program


import java.io.File;
import java.io.IOException;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.firefox.internal.ProfilesIni;

public class Example {
public static void main(String[] args) {
System.out.println(System.getProperty("user.dir"));

File firebug = new File(System.getProperty("user.dir") + "\\resources\\firebug-1.10.6.xpi");
File netExport = new File(System.getProperty("user.dir") + "\\resources\\netExport-0.9b3.xpi");

FirefoxProfile profile = new FirefoxProfile();
try {
profile.addExtension(firebug);
profile.addExtension(netExport);
} catch (IOException e) {
e.printStackTrace();
}

profile.setPreference("app.update.enabled", false);

//Setting Firebug preferences
profile.setPreference("extensions.firebug.currentVersion", "2.0");
profile.setPreference("extensions.firebug.addonBarOpened", true);
profile.setPreference("extensions.firebug.console.enableSites", true);
profile.setPreference("extensions.firebug.script.enableSites", true);
profile.setPreference("extensions.firebug.net.enableSites", true);
profile.setPreference("extensions.firebug.previousPlacement", 1);
profile.setPreference("extensions.firebug.allPagesActivation", "on");
profile.setPreference("extensions.firebug.onByDefault", true);
profile.setPreference("extensions.firebug.defaultPanelName", "net");

// Setting netExport preferences
profile.setPreference("extensions.firebug.netexport.alwaysEnableAutoExport", true);
profile.setPreference("extensions.firebug.netexport.autoExportToFile", true);
profile.setPreference("extensions.firebug.netexport.Automation", true);
profile.setPreference("extensions.firebug.netexport.showPreview", false);
profile.setPreference("extensions.firebug.netexport.defaultLogDir", "C:\\workspace\\CaptureNetworkTraffic");

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setBrowserName("firefox");
capabilities.setPlatform(org.openqa.selenium.Platform.ANY);
capabilities.setCapability(FirefoxDriver.PROFILE, profile);

WebDriver driver = new FirefoxDriver(capabilities);
try {
Thread.sleep(10000);
driver.get("http://www.google.com");
Thread.sleep(10000);
} catch(InterruptedException ie) {
ie.printStackTrace();
}
driver.quit();
}
}