Investigating the challenges in Selenium usage and improving the testing efficiency of Selenium-based load tests"
Selenium automates the testing of web applications. Quality Assurance (QA) engineers usually
write Selenium scripts to test the content that is rendered in a browser and simulate user
interactions with the web content for web applications. Apart from test automation, Selenium is
also used to perform load testing and web scraping tasks. Selenium has gained a great amount
of attention from users as observed from the trend of Selenium-tagged questions on
StackOverflow (a popular online development forum). The percentage of StackOverflow
questions that are tagged with ‘Selenium’ has been constantly increasing (i.e., it increased six
folds, from 2011 to 2018), reaching around 0.6% in 2018.
In this thesis, we first explore the Selenium-related questions on StackOverflow to understand
users' questions about Selenium, programming-language specific issues, browser-specific
concerns and the difficult aspects of Selenium. Some of the notable findings from our empirical
study are: Questions about Python have become the most frequent and the fastest growing
programming language related questions in Selenium, while questions on Java have become
stable in recent years. We also find that Chrome tag is the most frequently used browser tag
among other browser related tags associated with Selenium. However, different browsers may
be used for different purposes. For example, PhantomJS widely used for web scraping while
Firefox is frequently used with Selenium-IDE.
Several users discuss performance issues in Selenium tests on StackOverflow. Prior work on
Selenium also points out that Selenium tests are resource intensive. As Selenium needs to
launch a browser for each instance (e.g., a client) of a test, it is usually very resource consuming
to run a load test using Selenium as one would need to run thousands of user instances.
Therefore, we investigate the challenges in Selenium-based load tests by performing
experiments using different type of browsers. We propose an approach to improve the testing
efficiency of Selenium-based load tests. Our approach shares browser instances between user
instances, thereby reducing the performance overhead that is introduced by launching many
browser instances. The main contribution of this thesis is that our approach can significantly
increase the number of user instances that can be tested on a test driver machine without
overloading the machine. We improve the testing efficiency of Selenium-based load tests by at
least 20%.
The insights that we share in this work can help Selenium providers understand how Selenium
is used in practice and how they can offer better support for Selenium. Further, software
practitioners can benefit from our approach to improve the efficiency of Selenium-based load
tests.