ScraperETC¶

ScraperETC Core Module

This module provides lightweight wrappers around common web scraping operations using Selenium and the requests library. It is designed to simplify setup, reduce boilerplate, and improve stealth when interacting with websites for automated data extraction.

Features¶

Headless Chrome WebDriver setup via Selenium or undetected-chromedriver
Streamlined requests.get() wrapper with default anti-bot headers
Robust WebDriverWait abstraction with human-readable selectors
Built-in validation for PDF content and HTTP response codes

Defaults like user-agent strings and Chrome options are selected to minimize bot detection during web scraping tasks.

All functions are usable independently and aim to minimize external dependencies, making the module suitable for rapid prototyping, automation scripts, and CI pipelines.

Note

Only Chrome is supported at this time. Chrome must be installed and on your system PATH.

scraper_etc.scraper_etc.http_GET(url, header=None, timeout=30)¶

Waits for a web element (or elements) to meet a specified expected condition using Selenium’s WebDriverWait.

Parameters:

driver (selenium.webdriver.Chrome) – The WebDriver instance (standard or undetected_chromedriver).
by ({'CSS_SELECTOR', 'XPATH', 'ID', 'CLASS_NAME', 'NAME', 'TAG_NAME', 'LINK_TEXT'}) – The strategy to locate elements. Case-insensitive.
selector (str) – The actual query string to locate the element(s).
ec ({'located', 'all_located', 'clickable'}, default 'located') –
The expected condition to wait for:
- ’located’: Wait for a single element to be present in the DOM.
- ’all_located’: Wait for all matching elements to be present in the DOM.
- ’clickable’: Wait for an element to be both present and interactable.
delay (float, default 10) – Maximum number of seconds to wait before raising a TimeoutException.

Returns:

A single WebElement for ‘located’ and ‘clickable’, or a list of WebElements for ‘all_located’.

Return type:

WebElement or list of WebElement

Raises:

ValueError – If by or ec is not one of the accepted values.
selenium.common.exceptions.TimeoutException – If the expected condition is not met before the timeout period.

scraper_etc.scraper_etc.http_GET_valid_pdf(url, header=None, raise_error=False)¶

Perform an HTTP GET request and verify that the response is a valid PDF.

This function wraps http_GET() and performs two validation checks: - Ensures the response has a 200 OK status code. - Ensures the response content starts with the PDF signature (“%PDF”).

Parameters:

url (str) – The full URL to request via HTTP GET.
header (dict[str, str] | None, optional) – Optional headers to include in the request. If None, a default anti-bot header is used.
raise_error (bool, optional) – If True, raises an error for any of the following: - Non-200 status code - Response content is not a valid PDF

Returns:

The requests.Response object if the request succeeds and content is a valid PDF. Returns False if either check fails and raise_error is False.

Return type:

requests.Response or bool

Raises:

requests.RequestException – For any network-related errors during the request.
requests.HTTPError – If the response status code is not 200 and raise_error=True.
ValueError – If the response is not a valid PDF and raise_error=True.

Examples

>>> res = http_GET_valid_pdf("https://example.com/file.pdf")
>>> if res:
...     with open("file.pdf", "wb") as f:
...         f.write(res.content)

scraper_etc.scraper_etc.response_is_pdf(res, raise_error=False)¶

Check if an HTTP response has a 200 OK status code.

This function verifies whether the given requests.Response object indicates a successful HTTP request (status code 200). If raise_error is set to True, it raises a requests.HTTPError with details about the failed response.

Parameters:

res (requests.Response) – The response object returned by requests.get(), requests.post(), etc.
raise_error (bool, default False) – Whether to raise an HTTPError if the response status code is not 200. If False, the function returns False and prints a warning message.

Returns:

True if the response status code is 200; otherwise, False.

Return type:

bool

Raises:

requests.HTTPError – If raise_error=True and the response status code is not 200.

Examples

>>> response = requests.get("https://example.com")
>>> response_is_valid(response)
True

>>> response = requests.get("https://example.com/broken")
>>> response_is_valid(response)
Status code 404
False

>>> response_is_valid(response, raise_error=True)
Traceback (most recent call last):
    ...
requests.exceptions.HTTPError: ...

scraper_etc.scraper_etc.response_is_valid(res, raise_error=False)¶

Check whether a requests.Response object has a successful (HTTP 200 OK) status code.

This helper function evaluates whether the response was successful. If the status code is not 200 and raise_error is True, it raises a requests.HTTPError. Otherwise, it prints a warning and returns False.

Parameters:

res (requests.Response) – The response object to validate.
raise_error (bool, default False) – If True, raises a requests.HTTPError when the status code is not 200. If False, prints the error and returns False.

Returns:

True if the response status code is 200 (OK), False otherwise.

Return type:

bool

Raises:

requests.HTTPError – If the response code is not 200 and raise_error=True.

Examples

>>> res = requests.get("https://example.com")
>>> if response_is_valid(res):
...     print("Success!")

>>> # Raise an error instead of printing
>>> response_is_valid(res, raise_error=True)

scraper_etc.scraper_etc.setup_chrome_driver(user_agent=None, undetected_chrome=True, chrome_main_version=138, headless=False, add_default_options=True, additional_options=None, experimental_options=None)¶

Instantiates and returns a configured Chrome WebDriver instance.

This function creates a Chrome WebDriver instance using either the standard selenium.webdriver.Chrome or the stealthier undetected_chromedriver.Chrome. It supports customizing user agents, headless mode, default security workarounds, and the ability to inject additional or experimental driver options.

Parameters:

user_agent (str or None, default None) – Custom user agent string to be used by the driver. If None, a generic desktop Chrome user agent is used.
undetected_chrome (bool, default True) – If True, uses undetected_chromedriver.Chrome, which is designed to bypass basic bot-detection techniques. If False, uses the standard selenium.webdriver.Chrome.
chrome_main_version (int, default 131) – Only used when undetected_chrome=True. Specifies the main Chrome version installed on your system. This helps undetected_chromedriver locate the appropriate driver version.
headless (bool, default False) – If True, Chrome will run in headless mode (no GUI). This is useful for running in server environments or CI pipelines.
add_default_options (bool, default True) –
If True, appends a set of default options aimed at increasing stability and bypassing common certificate or security errors:

[
”–ignore-certificate-errors”, “–disable-dev-shm-usage”, “–disable-extensions”, “–disable-browser-side-navigation”, “–disable-web-security”, “–allow-insecure-localhost”

]
additional_options (list of str or None, default None) – A list of additional Chrome command-line options to pass (e.g., –disable-popup-blocking, –start-maximized). These are passed directly to options.add_argument(…).
experimental_options (dict[str, object] or None, default None) –
A dictionary of experimental options to be passed to options.add_experimental_option(key, value). For example:

{“excludeSwitches”: [“enable-automation”]}

Returns:

A Chrome driver instance with the specified configuration.

Return type:

selenium.webdriver.Chrome or undetected_chromedriver.Chrome

scraper_etc.scraper_etc.webdriver_wait(driver, by, selector, ec='located', delay=10)¶

Wait for a specific web element to be present in the DOM using Selenium’s WebDriverWait.

Parameters:

driver (selenium.webdriver.Chrome) – The webdriver instance to search with. Works with undetected_chromedriver too.
by (str) –

The type of selector to use. Must be one of:
[“CSS_SELECTOR”, “XPATH”, “ID”, “CLASS_NAME”, “NAME”, “TAG_NAME”, “LINK_TEXT”]

Case-insensitive.
selector (str) – The actual query string to locate the element.
delay (float, default 10) – Maximum number of seconds to wait before raising a TimeoutException.

Returns:

The first element matching the selector, once it appears in the DOM.

Return type:

selenium.webdriver.remote.webelement.WebElement

Raises:

ValueError – If by is not a recognized selector type.
TimeoutException – If no matching element is found before the timeout.