Open in app

Sign in

Write

Sign in

Selenium Support

alex_ber
Geek Culture
Published in
18 min readApr 18, 2021

--

Useful functions for Selenium umbrella project.

I’ve started to use Selenium’s related products and how found that documentation is misleading, it doesn’t promote best practice. Moreover, many components has memory/resource leaks that are not fixed for years. So I’ve created a utility project that encapsulated my “fixes” to Selenium and promotes best practice.

I will list some of the capabilities of my library:

  • create/destroy BmpDaemon(aka browsermobproxy.Server).
  • create/destroy BmpProxy (aka browsermobproxy.Client).
  • create/destroy SeleniumWebDriver (for example selenium.webdriver.Chrome.webdriver). (Can be any supported browser).
  • Taking screenshots.
  • Preparing browser’s data-dir for usage.
  • Enabling browser to download files.
  • Capturing network in har format.
  • Waiting for page to load.
  • Synchronous click (on the button).
  • Wait for Google Chrome to finish to download file (Chrome specific).
  • Wait for display.

The source code you can found here. It is available as part of my AlexBerUtils s project.

You can install SeleniumSupport from PyPi:

python -m pip install -U selenium-support

See here for more details explanation on how to install.

Selenum-4 is still in development, see What is Selenium 4? The latest in Automated Browser Testing, so this story is about Selenium 3.

Standard code to start working with Selenium looks like this:

path = ‘bin/browsermob-proxy’ #your path to browsermob-proxy
server = Server(path)
server.start()
proxy = server.create_proxy()

https://medium.com/@jiurdqe/how-to-get-json-response-body-with-selenium-amd-browsermob-proxy-71f10335c66

Another a little bit variant:
settings.py:

class Config(object):
    CHROME_PATH = '/Library/Application Support/Google/chromedriver76.0.3809.68'
    BROWSERMOB_PATH = '/usr/local/bin/browsermob-proxy-2.1.4/bin/browsermob-proxy'


class Docker(Config):
    CHROME_PATH = '/usr/local/bin/chromedriver'import Server from selenium
import contextlib
from settings import Config as config@contextlib.contextmanager 
def browser_and_proxy(): 
    server = Server(config.BROWSERMOB_PATH) 
    server.start() 
    proxy = server.create_proxy()
    #...
    # Set up Chrome
    option = webdriver.ChromeOptions()
    option.add_argument('--proxy-server=%s' % proxy.proxy)    prefs = {"profile.managed_default_content_settings.images": 2}
    option.add_experimental_option("prefs", prefs)
    option.add_argument('--headless')
    option.add_argument('--no-sandbox')
    option.add_argument('--disable-gpu')

    capabilities = DesiredCapabilities.CHROME.copy()
    capabilities['acceptSslCerts'] = True
    capabilities['acceptInsecureCerts'] = True

    path = config.CHROME_PATH
    browser = webdriver.Chrome(options=option,
                           desired_capabilities=capabilities,
                           executable_path=path)

    try:
      yield browser, proxy
    finally:
      browser.quit()
      server.stop()#...with browser_and_proxy() as (browser, proxy):
        browser.get('https://www.airbnb.com/s/Seattle--WA--United-States/homes')
#...

Based on https://medium.com/@sshevlyagin/browsermob-proxy-and-selenium-knowing-when-requests-finish-44d4b52851c8

  1. First of all, it is clear that code snippet above has sever memory/resource leaks — server and proxy are not closed.
  2. Second code snippet also has (several) leaks. Most obvious one — if browser.quite() raise exception than server.stop() will never called and will cause leaks. Another one, proxy should be also closed by calling proxy.close().
  3. None-trivial leaks has server.stop() method. I will describe it in details below.
  4. None-trivial leaks has browser.stop() method. I will describe it in details below. This leak has the same nature as in p.3

What type has proxy in the line above?

proxy = server.create_proxy()

Without looking to the implementation of create_proxy() method you will never guess that it is actually … Client.(browsermobproxy.Client to be more precise).

Quote from the documentation https://github.com/lightbody/browsermob-proxy#getting-started-standalone:

If you’re running BrowserMob Proxy within a Java application or Selenium test, get started with Embedded Mode. If you want to run BMP from the command line as a standalone proxy, start with Standalone…

To run in standalone mode from the command line, first download the latest release from the releases page, or build the latest from source.

Start the REST API:

./browsermob-proxy -port 8080

Then create a proxy server instance:

curl -X POST http://localhost:8080/proxy
 {"port":8081}

The “port” is the port of the newly-created proxy instance, so configure your HTTP client or web browser to use a proxy on the returned port. For more information on the features available in the REST API, see the REST API documentation.

Let’s follow the link. Quote from https://github.com/lightbody/browsermob-proxy#rest-api

New in 2.1: LittleProxy is the default implementation of the REST API. You may specify --use-littleproxy false to disable LittleProxy in favor of the legacy Jetty 5-based implementation.

To get started, first start the proxy by running browsermob-proxy or browsermob-proxy.bat in the bin directory:

$ sh browsermob-proxy -port 8080
INFO 05/31 03:12:48 o.b.p.Main - Starting up...
2011-05-30 20:12:49.517:INFO::jetty-7.3.0.v20110203
2011-05-30 20:12:49.689:INFO::started o.e.j.s.ServletContextHandler{/,null}
2011-05-30 20:12:49.820:INFO::Started SelectChannelConnector@0.0.0.0:8080

Once started, there won’t be an actual proxy running until you create a new proxy. You can do this by POSTing to /proxy:

[~]$ curl -X POST http://localhost:8080/proxy
{"port":8081}

or optionally specify your own port:

[~]$ curl -X POST -d 'port=8089' http://localhost:8080/proxy
{"port":8089}

or if running BrowserMob Proxy in a multi-homed environment, specify a desired bind address (default is 0.0.0.0):

[~]$ curl -X POST -d 'bindAddress=192.168.1.222' http://localhost:8080/proxy
{"port":8086}

Once that is done, a new proxy will be available on the port returned. All you have to do is point a browser to that proxy on that port and you should be able to browse the internet.

A little be under the hood

server.start()

This command actually executes this command
browsermob-proxy -port 8080

Note: On Mac it is actually sh browsermob-proxy -port 8080.

Note: On Windows, if you launch a subprocess from within a non-console process a console window (conhost.exe) appears. For the details see https://code.activestate.com/recipes/409002-launching-a-subprocess-without-a-console-window/ and https://docs.python.org/3/library/subprocess.html#subprocess.STARTUPINFO.dwFlags

Note also: Effectively the command will be browsermob-proxy.bat -port 8080 running in conhost.exe.

I will get back to this point below. So, server.start() uses subprocess.Popen(self.command...) where command is one of the variants above. There won’t be an actual proxy (you may think that browsermob-proxy starts “actual proxy”, ha-ha, it doesn’t). This command starts the BMP daemon — this is standalone Java application that acts as part of the OS.

How do you start “actual proxy”?
proxy = server.create_proxy()

Where proxy has type, as I’ve said above, Client(browsermobproxy.Client to be more precise).

Under the hood, creating proxy make the following API call:
curl -X POST http://localhost:8080/proxy

And the call returns port-number where you have dedicated “actual” proxy.

Note:

  • The best practice will be to allocate “actual” proxy for every process. The returned port-number should be passed to browser.
  • If you need to open multiple browsers in your application it is better to use different processes and not threads.

General note

In the text below I will use the following naming convention:

  • BMP Daemon — refer to the Java program that starts with some variant of sh browsermob-proxy -port 8080 command.
  • BMP Proxy — refer to browsermobproxy.Client type, this is “actual” proxy that is created for actual usage, one per Python Process.
  • Web Driver (I don’t like driver shortcut; driver has broader meaning outside Selenium world) — refer to the code that “controls” browser. This is part of Selenium. Actually, this is the main part of Selenium itself that I’m using.
  • Many of the function takes dictionary as parameters. You can create dict with hard-coded parameters, you can pass parameters as named variable (kwargs) or you can put parameters in some configuration file and parse it do dict. If you chose the later you has basically the following options:
  1. Chose your favorite Yml parser and parse to to dict. For example, ruyaml, pyyaml, etc.
  2. You can use my ymlparsers module.
  3. If you want to have different values dependent on the environment/profile you’re working on (dev, prod, for example) you can use my init_app_conf module.

My personal choise is init_app_conf module.

BMP Daemon

The code:

where dd can be described as following:

Note: See “Many of the function takes dictionary as parameters” section above on how to convert yml file to dict dd.

The explanation:

BMPDaemon is context-manager that responsible for starting BMP Daemon. In the exit from the code block inside context-manager, BMP Daemon will be stopped, see closeBmpDaemon().

Starting the BMP Daemon is basically done by following code:

server = Server(**daemon_init_d)
server.start(**daemon_start_d)

where daemon_init_d represent section below init

and daemon_start_d represent section below start.

Note:

  • You can can start BMP Daemon manually, outside your code. Alternatively, you can wrap is as actual OS Daemon that startד on OS start-up.
  • If you want to start BMP Daemon programmatically, for example, using BMPDaemon utility it is recommended to do it from the Main Process.
  • If you want to use BMP Daemon programmatically, you shouldn’t forget to close it. More on closing below.
  • You can use BMPDaemon utility just as function call, if you really want. In such scenario you should use try-finally block and make explicit call to closeBmpDaemon() function. More on closing below.
  • If you have multiple application that uses BMP Daemon, you have 2 basic choices:
  1. Start BMP Daemon outside of scope of these application (maybe as 3 application or as OS Daemon or just manually, see above) in some predefined port (the default is 8080) and write your application code that assumes that BMP Daemon is up and running.
  2. Each application will start BMP Daemon on different port.

I want to emphasize, technically it is sufficient to have only 1 running BMP Daemon for all application (You will create BMP Proxy per application).

Personally, I’ve found the second option easier to manage — namely to have multiple BMP Daemons, one per application. This is indeed waste of resources, but the application lifecycle is much easier to manage and you don’t have some external dependencies. Note, however, that in such case you should, at least in one application, explicitly provide the port number. It is better that they will far away one from another, because BMP Proxy is created as next port number.

To make an explicit example, in the code above BMP Daemon will start on port 8090. First BMP Proxy will be in port 8091, each subsequent BMP Proxy will take next port number. If you know, that you will have up to 9 Processes each of each will use BMP Proxy, BMP Proxy will occupy port range 8091–8099. So, if you want to have another BMP Daemon, it can’t be 8091, and it better be at least 8100 (or below 8090).

daemon_init_d — this dict is passed to __init__():

  • The default value of path is browsermob-proxy. (On Windows it is browsermob-proxy.bat). If this file is not available in OS environment variables, you should provide explicit file to the executable file like in the example above.
  • The default value of options['port'] is 8080. As in the documentation quoted above, this is the default port when BMP daemon will run. Usually, however, this port is used by some other application, so you may want to change it.

daemon_start_d — this dict is passed to start():

This context-manager also worries to close BMP Daemon. See closeBmpDaemon() below. Again, if you use it as regular function, this will not happen.

Closing BMP Daemon

If you’re using BMPDaemon as context-manager, it will worry to close the BMP Daemon. If you want to do it yourself, you can call closeBmpDaemon() function.

You may wonder why call to bmp_daemon.stop() is not sufficient, it looks like this method is specifically designed for this.

First of all, there is open issue Java Processes not getting killed in Mac. Quote:

Hi even after doing proxy_server.stop() and proxy.close() browsermob proxy is leaving java processes running the background in Mac.

There is unmerge partial fix Run and kill as process group.

Here you can find some guidelines on how to close BMP Dameon on Windows.

What’s going on?

The root cause is the following. When start() method of BMP Daemon is called, it uses subprocess.Popen with some variant of command sh browsermob-proxy -port 8080. As described above in “A little be under the hood” section on Windows Python process start’s cmd that executes bat-file that starts Java process.

When you’re calling stop() function on bmp_daemon, it successfully kills conhost.exe process. On Windows (and on Mac as per #76) OS doesn’t kills grand process automatically (I don’t know what’s happen on Linux, but I suspect it works their), that is stop() function kills conhost.exeprocess, but on Windows (and Mac) the Java process survives.

The proposed fix in a nutshell is the following:

self.log_file = open(os.path.join(log_path, log_file), 'w')self.process = subprocess.Popen(self.command,                                                   preexec_fn=os.setsid,                                                   stdout=self.log_file,                                                   stderr=subprocess.STDOUT)group_pid = os.getpgid(self.process.pid)
self.process.kill()                               self.process.wait()
os.killpg(group_pid, signal.SIGINT)

It is claimed that this fix works on Mac.

On Windows this fix has the following form:

self.log_file = open(os.path.join(log_path, log_file), 'w')self.process = subprocess.Popen(self.command,                                                   creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,                                                   stdout=self.log_file,                                                   stderr=subprocess.STDOUT)group_pid = self.process.pid
self.process.kill()                               self.process.wait()
os.killpg(group_pid, signal.SIGINT)

This doesn’t work on Windows.

Basically, what the code above tries to do is to create “process group” around browsermob-proxy -port 8080, so all child subprocesses will be group into one entity that can be killed using OS System call.

Here https://github.com/AutomatedTester/browsermob-proxy-py/issues/8#issuecomment-679150656 is another implementation of the same idea. My solution is another implementation.

I don’t modify existing code, but rather use same wrapping function. Basically, I’m doing the following:

  1. I’m using psutil to get all child processes that was creating as part of execution subprocess.Popen(self.command..). Internally, each such process has (PID and it’s creation time).
  2. I’m calling bmp_daemon.stop().
    This is essential because it has the following call self.process.poll() in it. This call ensure that the self.log_file will be properly closed.
  3. Now, I’m iterating over the child process from p.1. calling
    import psutil
    from contextlib import suppress
    with suppress(psutil.NoSuchProcess):
    child.send_signal(signal.SIGTERM)
    Note: This command can kill only the process we have cached earlier in p.1. If process was already finished we will get NoSuchProcessthat we’re just suppressing (send_signal() has decorator @_assert_pid_not_reused that is responsible for not killing reused id for the new process; internally it fetch process by PID and check that it has the same creation time and if it doesn’t NoSuchProcess is raised).

In summary, we’re fetching all child (grandchild and all ancestors) process ids that was open in bmp_daemon.start(). We’re calling bmp_daemon.stop() and then we’re killing all ancestor’s processes.

If the process was meanwhile terminated, we’re ignoring if.

If the process id was reused, we also do nothing.

BrowserDataDir, BMPProxy, SeleniumWebDriver, Screenshot

The code:

dd can be described as following:

Note: See “Many of the function takes dictionary as parameters” section above on how to convert yml file to dict dd.

The explanation:

BrowserDataDir — this context manager can be used for reuse of user data dir. It unzips file template (‘path/to/chrome_data_dir.zip’ in our case) to work_dir (‘logs’ in our case).

It returns directory (browser_data_dir in our case) with extracted content from template.

In the exit from the code block inside context-manager, the work_dir is removed.

We’re using browser_data_dir a bit later to populate d[‘web_driver’][‘arguments’] that is passed as kwargs to SeleniumWebDriver.

Note: In configuration yml we have following line ‘--user-data-dir={data_dir}’. Call to _arg_format() will effectively substitute {data_dir} to browser_data_dir.
If you don’t need to pass --user-data-dir to the Selenium Web Driver (such as chromedriver.exe), then
you should remove the usage of BrowserDataDir,
remove line ‘--user-data-dir={data_dir}’
and don’t poputlate d[‘web_driver’][‘arguments’] (kwargs to SeleniumWebDriver).

BMPPRoxy— this context manager is designed to create BMP Proxy on new port. It assumes that BMPDaemon is already up. It doesn’t require BMPDaemon object, but only some of it’s parameters.

By default it assumes, that BMPDaemon is running on localhost. By default it also assumes that it is running on port 8080, you may want to override this value by supplying d[‘browsermob’][‘daemon][‘init][‘options’][‘port’]. It is recommended to supply the same dict that you have supplied to BMPDaemon.

If you recall the code snippet above, it uses create_proxy() method on BMPDaemon to create BMPPRoxy and internally it reuse some of the parameters. My implementation bypasses usage of create_proxy() method on BMPDaemon by mimicking it’s behavior.

This context manager returns BMP Proxy.

In the exit from the code block inside context-manager, it closes BMP Proxy.

SeleniumWebDriver — this context manager is designed to create Web Driver. It assumes that BMPDaemon is already up. You may pass BMPPRoxy if you want to use it. It doesn’t require BMPDaemon object. It returns Web Driver. In the exit from the code block inside context-manager, it close Web Driver, see closeSeleniumWebDriver() for the details.

Some basic architecture description: Selenium is framework. One of it’s main components is Web Driver. For each mainstream browser, such as Google Chrome, Mozilla Firefox, there is dedicated Web Driver type. In this way, each Web Driver type “knows” internals of the corresponded browser. The main purpose of Web Driver is provide browser agnostic API. Internally, each Web Driver “knows” know to translate such browser agnostic API calls to specific browser API calls.

Note: Web Driver consists of Python wrapper and some executable file (chromedriver.exe, for example). The term ‘Web Driver’ (confusingly) refers to both.

Selenum-3 lucks some functionality, for example, network monitoring. There are many Selenium extensions to provide extra functionality. One of this extensions is BMPPRoxy. It’s host&port can be passed as Web Driver’s option’s argument; this way all network communication between browser and Internet will be passed through BMPPRoxy. Note: Selenum-4 will have this functional built-in.

Note: There is some complexity on how to create general-purpose context-manager that will work for any Web Driver. For example, Google Chrome and Mozilla Firefox has different option’s class. My implementation is inspired by https://github.com/clemfromspace/scrapy-selenium/blob/develop/scrapy_selenium/middlewares.py

In the code sample above I’m passing browsermobproxy (it’s host&port will be passed as Web Driver’s option’s argument), in addition I’m passing web_driver. It has following entries:

‘name’:’chrome’

it is used to determine specific (for the browser) Web Driver. In this case it will be ‘selenium.webdriver.chrome.webdriver’.

‘log_file’:’path/to/logs/chromedriver.log’

All logs from the Selenium’s Web Driver component will be redirected to this log_file.

Note: This creates some problems in the Web Driver closing. They’re handled in closeSeleniumWebDriver() below.

‘path’:’resources/chromedriver.exe’

Path to the executable file of the Web Driver. It is needed for the Python wrapper to invoke it. This is where actual browser control part sits.

‘arguments’=[‘ --headless’, ‘--window-size=1920,1080’, ‘--ignore-certificate-errors’, ‘--disable-useAutomationExtension’, ‘--user-data-dir=logs/chrome_dcm_data_dirvwg1nmbv’]

‘experimental_options’:’excludeSwitches’: [‘enable-logging’, ‘enable-automation’]

Let’s go one by one.

Headless mode.

A headless browser is a browser without a graphical user interface. Running a browser in a headless mode means that it can run without the need for dedicated display or graphics. It gets the name headless because it runs without the Graphical User Interface (GUI). The GUI of a program is referred to as the head.

Before about 2017 none of the major browser doesn’t support headless mode. So, if you want to run headless with Selenium you should use another browsers, such as HtmlUnit or PhantomJS.

Headless Chrome is shipping in Chrome 59. See also here.

Using Headless Mode in Firefox. See also here.

So, from about 2017 you can use your favorite browser (it exists also in another browser that are not mentioned here) in headless mode.

Personally, I’m using full version of the browser in dev, and headless mode in prod. I’m achieving this using different profiles for my configuration file. For more information on this, see My major init_app_conf module

window-size

1366x768 is the default resolution for headless mode in Mozilla Firefox. In headless mode, the Google Chrome browser start in 800x600. Many sites “looks ugly” and are hardly usable in such resolution — you can’t click on the buttons, for example, so you should override it’s default.

ignore-certificate-errors

Before starting the https communication, the website will present browser with a certificate to identify itself. Some sites works with self signed certificates (it can be for another reason, but this is most common one), that are not trusted by browsers, so you will get an error page with the message “Your connection is not secure”.

If you’re using Google Chrome, passing --ignore-certificate-errors in argument’s option is the easiest way to solve this problem.

For alternatives or for another browser’s solution, see http://allselenium.info/selfsigned-certificates-python-selenium/

Note: Currently desired_capabilities are not supported. See https://github.com/alex-ber/selenium-support/issues/1

disable-useAutomationExtension

This is Google Chrome specific feature.

This should be use in conjunction will

options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging', 'enable-automation'])

https://stackoverflow.com/questions/47392423/python-selenium-devtools-listening-on-ws-127-0-0-1

https://stackoverflow.com/questions/46423361/chrome-devmode-suddenly-turning-on-in-selenium

These 3 options will remove theDevTools message popping up and will prevent entering Google Chromes’ devmode.

user-data-dir

This is Google Chrome specific feature. For Mozilla Firefox alternative see https://stackoverflow.com/a/55636113/1137529 or https://firefox-source-docs.mozilla.org/testing/geckodriver/CrashReports.html

This is totally optional feature. It is better to use it together with BrowserDataDir (see above).

If you want your browser to work with some predefined user data dir (“profile” in Mozilla Firefox).

For example, on Windows you can create shortcut: “C:\Program Files (x86)\Google\Chrome\Application\chrome.exe” --user-data-dir=C:\tmp\chrome_profile. Click on it to create all internal folders and than pass ‘C:\tmp\chrome_profile’ as --user-data-dir to Google Chrome option’s argument.

Closing Selenium’s Web Driver

If you’re using SeleniumWebDriver as context-manager, it will worry to close Web Driver. If you want to do it yourself, you can call closeSeleniumWebDriver() function.

You may wonder why call to web_driver.quit() is not sufficient. Note (be aware!): you shouldn’t confuse this call with web_driver.close(). The last call will close only the tab and not browser itself.

If you have some Déjà vu while reading the lines above, you may recall, that something very similar is going on in closeBmpDaemon() function.

I will not going to repeat myself, the implementation is very close to that one.

But why web_driver.quit() is not enough?

I have noticed, that sometimes, when exception is raised, but not always, Google Chrome browser and/or chromedriver.exe remain residents in memory.

When I’ve added the same logic to close all ancestor processes that was opened in Web Driver initialization, this never happens again.

Screenshot

It is designed to be used as context-manager.

If you want API for simple function call, please use save_screenshot().

You may want to guard piece of you code with this context-manager. It is required that you instantiated web_driver first. If in the code block inside context-manager exception will be raised, than some additional action will be taken.

If logger is not None, warning logger message will be issued to it.

If we have instance of WebDriverException that has screen on it, it will be used to generate png file of screenshot.

If screen is None or we don’t have instance of WebDriverException, that we will actively make screenshot. Note, that this attempt may fail.

In the code snippet above:

web_driver is mandatory field. Action is just indication where exception occures. screenshots_dir is directory to save screenshots. logger is passed to have warning message issued.

Save screenshot

save_screenshot() is regular function API. If you want a context-manager, please use Screenshot.

Maybe, you have some try-finally block and you want when you’ve caught an exception from web_driver to get screenshot in order to understand better what went wrong. Personally, I prefer to use Screenshot, but in some complex scenario you may want to have better control.

Note: you can define dd[‘browser_download_folder’] as following

dd['files']['browser_download_folder'] = str(Path(Path.home(), 'Downloads'))

Enable download in Google Chrome

enable_chrome_download() is Google Chrome specific function.

In Google Chrome in headless mode download is disabled by default. It’s a “feature”, for security. If you want to enable download you can use this function. See https://stackoverflow.com/questions/45631715/downloading-with-chrome-headless-and-selenium for more details.

Set new HAR

set_new_har() is convenient wrapper to bmp_proxy.new_har() with capture* parameters. It is used to get network transmission. For example, if you want to get response body. See https://medium.com/@jiurdqe/how-to-get-json-response-body-with-selenium-amd-browsermob-proxy-71f10335c66 for more details.

This call is equivalent to click on “Start recording” on Network tab of Page Inspector.

In the above sample code we have call set_new_har(bmp_proxy, har_name) where bmp_proxy is object created above and har_name is str. After this call is done all network communication from the browser to the site is recorded in BMP Proxy.

If you want to retrieve it, you can use following template:

Note:

  1. In line 4 we’re waiting (5 ms) for network traffic to stop (with timeout of 70 ms). It is needed to ensure that BMP Proxy has recorded all network traffic (up to this point).
  2. In line 9 the usage of reversed. The quoted link doesn’t have this. If you have multiple calls to same URL, you should look on last result, so you should reverse the order of log entries.

Wait for some basic page’s element to be loaded

wait_page_loaded()is helper function to ensure that some basic elements of the page, such as title are loaded.

Usage example:

from alexber.seleniumsupport import wait_page_loaded
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)
wait_page_loaded(wait)

Click on WebElement

click_sync()- sometimes calling click() on WebElement raise some weird exception. The best practice will be to use wait.until(EC.element_to_be_clickable((By.XPATH, ‘xpath’))). This is “dirty” solution that make synchronous call (by using JavaScript) on WebElement.click()
(WebElement is typically button). See https://stackoverflow.com/a/58378714/1137529 for more details.

Usage example:

from alexber.seleniumsupport import click_sync
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)click_sync(web_driver,
                 wait.until(EC.element_to_be_clickable(
                    (By.XPATH, 'xpath'))))

Wait for Google Chrome to finish Downloads

wait_chrome_file_finished_downloades() is Google Chrome specific function.
It works directly with file system. You should know the file_name beforehand. It relies on Google Chrome following internal mechanism:

  • when Google Chrome downloads file, it has extension “.crdownload”. When downloads is finished it Google Chrome rename the file removing this extension.

This function doesn’t rely on Google Chrome’s downloads status. There are 2 main reason for this design choice:

Note: On Unix-based systems, scandir() uses the system’s opendir() and readdir() functions. On Windows, it uses the Win32 FindFirstFileW and FindNextFileW functions.

https://docs.python.org/3/library/os.html#os.scandir

Quote from FindFirstFileW and FindNextFileW:

To perform this operation as a transacted operation, use the FindFirstFileTransacted function.
…
If there is a transaction bound to the file enumeration handle, then the files that are returned are subject to transaction isolation rules.
…
Note: In rare cases or on a heavily loaded system, file attribute information on NTFS file systems may not be current at the time this function is called. To be assured of getting the current NTFS file system file attributes, call the GetFileInformationByHandle function.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx

Those “rare cases” are not such rare. I have encounter them regularly: Google Chrome from another process downloads file, it has “.crdownload” extension. I’m checking from Python code if the the file with “.crdownload” extension exists and get result, that it doesn’t found, while it clearly still downloaded. I think it takes some time to the change to the filesysem (more princely file attributes) to be visible. In order to mitigate this issue I have added simple sleep-retry mechanism. Before I’m consulting with file system I make a sleep to ensure that any change that was done to file system will be visible to Python Process.

Note: if the file is very bigger (more than 200MB) you may need to increase retries number.

Wait for display

wait_for_display() —Sometimes, we want to make Selenium Web driver wait until elements style attribute has changed. This is usefull for dynamically loaded material. For example, we want to wait for the display style to change to none (or to “inline-block" or some other value). See https://stackoverflow.com/questions/34915421/make-selenium-driver-wait-until-elements-style-attribute-has-changed

Usage example:

from alexber.seleniumsupport import wait_for_display
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)wait.until(wait_for_display((By.XPATH, 'xpath')))

For further reading:

Locating elements
https://www.selenium.dev/documentation/en/getting_started_with_webdriver/locating_elements/

ChromeDriver — WebDriver for Chrome
https://sites.google.com/a/chromium.org/chromedriver/

Browser Manipulation
https://www.selenium.dev/documentation/en/webdriver/browser_manipulation/#browser-navigation

Race conditton (waits)
https://www.selenium.dev/documentation/en/webdriver/waits/

Find Element From Element (Web Element)
https://www.selenium.dev/documentation/en/webdriver/web_element/

Working with select elements
https://www.selenium.dev/documentation/en/support_packages/working_with_select_elements/

WebDriver API provides the ability to set a proxy for the browser,
and there are a number of proxies that will programmatically allow you to manipulate
the contents of requests sent to and received from the web server
https://www.selenium.dev/documentation/en/worst_practices/http_response_codes/

Selenium 4 Alpha is now available 11/17/2020
https://testguild.com/selenium-4/

We are using Chrome driver version XXX, please see Chromedriver arguments
https://peter.sh/experiments/chromium-command-line-switches/

Issue 696481: Headless mode doesn’t save file downloads
https://bugs.chromium.org/p/chromium/issues/detail?id=696481
https://stackoverflow.com/questions/45631715/downloading-with-chrome-headless-and-selenium

browsermob-proxy GitHub
Setting Up a Proxy, Extracting a SAML Assertion
https://medium.com/@arkadyt/setting-up-programmatic-aws-access-for-mfa-protected-federated-identities-3be22bcecf4b
https://github.com/lightbody/browsermob-proxy (may be misslieading!!!)
https://medium.com/@jiurdqe/how-to-get-json-response-body-with-selenium-amd-browsermob-proxy-71f10335c66

Google Chrome/Mozilla Firefox recomended extensions:

XPath Helper

Inspector

Python
Python3
Selenium
Selenium Webdriver
Webdriver

--

--

alex_ber
Geek Culture

Written by alex_ber

66 Followers

Senior Software Engineer at Pursway

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams