Make Your Tests Saucey - Running Selenium Tests Against Multiple Platforms and Browsers

Cucumber, Selenium and Gherkin have become popular tools for helping teams implement test automation or what is often referred to as Acceptance Test Driven Development (ATDD) or Behavior Driven Development (BDD). By running your tests locally, you can ensure your application runs as expected on your OS/Browser setup, but to truly know that the app works on other configurations, you need a way to emulate numerous environments. SauceLabs is a web service that you can sign up to which will give the ability to test on many OS/Browser setups, including numerous mobile options.

In this article, we will cover:

1. Why Use SauceLabs?
2. Connecting Your Tests to SauceLabs
3. Parameterize your Rake File to Pass in Platform and Browser
4. Using a Secure Tunnel to Access Your Internal Application

In this article series we use Ruby as the implementation language (and we recommend Ruby when there is no other existing preference). However, these examples will translate easily to other languages like Java. 

If you are not familiar with Selenium, Cucumber and Gherkin take a look at our related introductory blog "The 5 Step Guide for Selenium, Cucumber, and Gherkin" and "Rake Up and Optimize Your Tests"

1. Why Use SauceLabs?

SauceLabs is a service which offers a variety of features to improve your testing. As mentioned above, SauceLabs allows you to run your Selenium tests on nearly any OS and Browser combination. At any time, you can kick off a virtual session by specifying the OS and Browser, as well as the version of the browser, which allows you to run any manual tests you need against virtually any environment.

With just a couple of modifications to your Selenium tests, it is possible to run your automated tests against any OS/Browser combination as well. Want to see how your app holds up on various operating systems and browsers? It's as easy as a few new parameters.

Not only do you get the ability to run flexible tests on a variety of platforms, but the feedback provided is amazing, usually far more detail than you need. Each automated or manual test can be replayed as a video, or frame by frame with each call sent to the browser. Failures can be examined directly in the dashboard, making it easy to see exactly which test failed at which step.

Additionally, SauceLabs offers excellent reporting, allowing you to easily gather analytics on number of tests executed, pass fail rate, and more.

2. Connecting your Tests to SauceLabs

SauceLabs has made it easy to connect your Selenium project to their virtual environment. The first step is to register for an account with SauceLabs; they offer a free 14 day trial, but after that you can reference their pricing plans here.

Once you have created your account, click the drop down next to your username, and select "My Account". This will bring you to your account page; the thing to make note of here is "Access Key", which at first appears masked. By combining your username and access key, you get the Sauce URL you will need to plug in to your ruby code below.

https://YOUR_SAUCE_USERNAME:YOUR_SAUCE_ACCESS_KEY@ondemand.saucelabs.com:443/wd/hub

SauceLabs offer extensive documentation on how to connect your tests to Sauce on their wiki, https://wiki.saucelabs.com/display/DOCS/Ruby+Test+Setup+Example. The first thing to do is to install the Sauce Whisk gem by running the following terminal command from within your project:

gem install sauce_whisk

Then add a require statement to use the gem, either in your ruby steps file or in your setup folder.

require 'sauce-whisk'

All that is left is to define a capabilities object in which you specify the platform, browser, and browser version, then create a new remote WebDriver using the capabilities object you define and specifying your Sauce URL.

caps = {
  :platform => "Windows 7",
  :browserName => "Chrome",
  :version => "latest",
  :name => "#{scenario.feature.name } - #{scenario.name}"
}
sauceUrl = "https://SAUCE_USERNAME:SAUCE_ACCESS_KEY@ondemand.saucelabs.com:443/wd/hub".strip
$driver = Selenium::WebDriver.for(:remote, :url => sauceUrl, :desired_capabilities => caps)

After updating your ruby code, you can continue to run your tests as normal, but instead of seeing the tests run local, you will just see the results of the test coming back from SauceLabs. You can also log in to the SauceLabs dashboard to see your tests executing in real time.

 3. Parameterize your Rake File to Pass in Platform and Browser

So far, we have seen how to run your tests on a specific platform/browser combination, but this approach is not very flexible, requiring code changes whenever you want to run tests against a different environment. To make this easier, we recommend adding new parameters for platform, browser, and browser version in your Rake file. This will allow you to pass in whatever platform/browser/version combination you want on the fly. If you followed our previous blog "Rake Up and Automize Tests" [TODO: Published Link] you should easily be able to modify your Rake file to accept two new parameters.

task :run_tests_sauce, [:environment, :platform, :browser, :browser_version] do |t, args|
    cucumberStatement = 'cucumber ENVIRONMENT=' + args[:environment] + 
	cucumberStatement +=' SAUCE=true PLATFORM=\'' + args[:platform] + '\' BROWSER=' + args[:browser] + ' BROWSER_VERSION=' + args[:browser_version] 
	puts cucumberStatement
	sh(cucumberStatement) do |success, _exit_code |
		@success &= success
	end
end

Then all that is left is to declare your capabilities object with the values from the parameter in your Before hook:

task :run_tests_sauce, [:environment, :platform, :browser, :browser_version] do |t, args|
    cucumberStatement = 'cucumber ENVIRONMENT=' + args[:environment] + 
	cucumberStatement +=' SAUCE=true PLATFORM=\'' + args[:platform] + '\' BROWSER=' + args[:browser] + ' BROWSER_VERSION=' + args[:browser_version] 
	puts cucumberStatement
	sh(cucumberStatement) do |success, _exit_code |
		@success &= success
	end
end

Now you can run your tests against any platform/browser combination Sauce offers with a simple terminal command:

$ rake run_tests_sauce['dev','Windows 7','chrome','latest']
$ rake run_tests_sauce['dev','OS X 10.9','firefox','45.0']
$ rake run_tests_sauce['dev','Windows 10','microsoftedge','latest']

 4. Using Secure Tunnel to Connect to Your Internal Application

If you are working on an intranet application, you may be worried you cannot use SauceLabs. Fear not, as Sauce has an extensive solution for you called Sauce Connect Proxy. By creating a secure tunnel between SauceLabs' virtual environment, then using that tunnel when running your tests, you can run your tests even if the app is not publically accessible.

SauceLabs has documentation on how to set up your tunnel here: https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy. You will likely need to work with your networking team to get the required ports open and set up proxy server. Once it is properly set up and configured, you should see 1 Active Tunnel when you log in to the Sauce dashboard and examine the left most navigation.

To use the tunnel once it is set up, all you need to do is declare appropriate environment variables, we recommend adding them to your cucumber statement in your Rake file.

cucumberStatement = 'cucumber ENVIRONMENT=' + args[:environment] + 
cucumberStatement +=' SAUCE=true PLATFORM=\'' + args[:platform] + '\' BROWSER=' + args[:browser] + ' BROWSER_VERSION=' + args[:browser_version] 
cucumberStatement +=' SAUCE_USERNAME=' + YOUR_SAUCE_USERNAME + ' SAUCE_ACCESS_KEY=' + YOUR_SAUCE_ACCESS_KEY + ' SAUCE_PARENT_TUNNEL=' + YOUR_TUNNEL_NAME + ' http_proxy=' YOUR_HTTP_PROXY_ADDRESS + ' no_proxy= https://YOUR_SAUCE_USERNAME:YOUR_SAUCE_ACCESS_KEY@ondemand.saucelabs.com:443/wd/hub'

Then, when you create your capabilities object, add a statement to specify the tunnel:

caps = {
  :platform =>  "#{ENV['PLATFORM']}",
  :browserName => "#{ENV['BROWSER']}",
  :version => "#{ENV['BROWSER_VERSION']}",
  :name => "#{scenario.feature.name } - #{scenario.name}",
  :parentTunnel => "#{ENV['SAUCE_PARENT_TUNNEL]"
}

And now you can successfully run your tests via sauce through the tunnel to connect to your internal application

 This just scratches the surface of what SauceLabs offers. Some additional features include running tests against mobile, running your tests in Sauce in parallel, and tying in your CI/CD process. Additionally, Sauce offers numerous integrations and plugins that will make your life easy, including hooking up to your Jenkins jobs or plugging in to JIRA.

To become an ICAgile Certified Professional in Agile Testing or to learn more about automation and Agile testing check out our courses.