Long-term support concerns with network interception due to Selenium Wire deprecation

[pradeepvegesna]

We have several use cases that require request and response interception for UI testing, and we’re considering using SeleniumBase for this purpose. However, I noticed that SeleniumBase's wire mode relies on Selenium Wire, which is deprecated since January 2024.

Although network interception is still functional through Selenium Wire, I’m concerned about the impact on long-term support. Could you provide guidance on the following:

Are there plans to update or replace Selenium Wire within SeleniumBase to ensure continued support for network interception?
Is it advisable to continue using the current setup, or would you recommend alternative approaches for network interception within SeleniumBase?
Any insights on future plans for handling network intercepts would be greatly appreciated.

[mdmintz]

The Selenium-Wire replacement is already included within SeleniumBase's new CDP Mode.

Here's an example of that, (SeleniumBase/examples/cdp_mode/raw_res_sb.py):

"""Using CDP.network.ResponseReceived and CDP.network.RequestWillBeSent."""
import colorama
import mycdp
import sys
from seleniumbase import SB

c1 = colorama.Fore.BLUE + colorama.Back.LIGHTYELLOW_EX
c2 = colorama.Fore.BLUE + colorama.Back.LIGHTGREEN_EX
cr = colorama.Style.RESET_ALL
if "linux" in sys.platform:
    c1 = c2 = cr = ""

async def send_handler(event: mycdp.network.RequestWillBeSent):
    r = event.request
    s = f"{r.method} {r.url}"
    for k, v in r.headers.items():
        s += f"\n\t{k} : {v}"
    print(c1 + "*** ==> RequestWillBeSent <== ***" + cr)
    print(s)

async def receive_handler(event: mycdp.network.ResponseReceived):
    print(c2 + "*** ==> ResponseReceived <== ***" + cr)
    print(event.response)

with SB(uc=True, test=True, locale_code="en") as sb:
    sb.activate_cdp_mode("about:blank")
    sb.cdp.add_handler(mycdp.network.RequestWillBeSent, send_handler)
    sb.cdp.add_handler(mycdp.network.ResponseReceived, receive_handler)
    url = "https://seleniumbase.io/apps/calculator"
    sb.cdp.open(url)
    sb.sleep(1)

Usage is different from regular Selenium-Wire, but it can do all the same things (and more) with better flexibility/control.

The greatest benefit for some people is that CDP Mode is part of UC Mode, which provides stealth to avoid bot-detection.
The old Selenium-Wire integration is not compatible with UC Mode, but CDP Mode patches the previous issues with Selenium-Wire that prevented compatibility.

Here's another example, (SeleniumBase/examples/cdp_mode/raw_req_sb.py), where specific requests were filtered out (intercepted and blocked), to prevent images from loading:

"""Using CDP.fetch.RequestPaused to filter content in real-time."""
import mycdp
from seleniumbase import SB

async def request_paused_handler(event, tab):
    r = event.request
    is_image = ".png" in r.url or ".jpg" in r.url or ".gif" in r.url
    if not is_image:  # Let the data through
        tab.feed_cdp(mycdp.fetch.continue_request(request_id=event.request_id))
    else:  # Block the data (images)
        TIMED_OUT = mycdp.network.ErrorReason.TIMED_OUT
        s = f"BLOCKING | {r.method} | {r.url}"
        print(f" >>> ------------\n{s}")
        tab.feed_cdp(mycdp.fetch.fail_request(event.request_id, TIMED_OUT))

with SB(uc=True, test=True, locale_code="en") as sb:
    sb.activate_cdp_mode("about:blank")
    sb.cdp.add_handler(mycdp.fetch.RequestPaused, request_paused_handler)
    url = "https://gettyimages.com/photos/firefly-2003-nathan"
    sb.cdp.open(url)
    sb.sleep(5)

If people don't need the stealth features or the other improvements made to intercepting and handling network requests & responses, then they can continue using the existing Selenium-Wire integration as it currently stands. Or, if people want the upgrades, then the new integration is ready in CDP Mode, as shown above.

[mdmintz]

Closing this in favor of #3247.

[unpacify]

okay so i tried this out with url = "https://www.kohls.com/product/prd-6908521/girls-5-16-zeroxposur-2-pc-cozy-active-knit-top-jogger-set.jsp?color=Plum&storeIds=348&prdPV=1"

and there's only 1 request response i need, and it doesnt return it. that's the "https://www.kohls.com/web/productInventoryPrice/6908521?storeNum=348" request, which usually returns a 403 error.

the depreciated selenium wire works and retrieves that request response, but is causing massive amounts of memory to be used.