URL types

[brunns]

I like to use yarl for storing and manipulating URLs. The requests library used to let me use these directly, but httpx does not:

import yarl, requests, httpx
url = yarl.URL("https://brunn.ing")
r1 = requests.get(url)
r2 = httpx.get(url)
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_api.py", line 195, in get
    return request(
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_api.py", line 109, in request
    return client.request(
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_client.py", line 812, in request
    request = self.build_request(
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_client.py", line 366, in build_request
    url = self._merge_url(url)
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_client.py", line 396, in _merge_url
    merge_url = URL(url)
  File "/Users/brunns/dev/brunns/mbtest/.tox/py39/lib/python3.9/site-packages/httpx/_urls.py", line 121, in __init__
    raise TypeError(
TypeError: Invalid type for url.  Expected str or httpx.URL, got <class 'yarl.URL'>: URL('https://brunn.ing')

I'm moving a largish codebase over from requests to httpx, and having to wrap my URLs in str(url) in many places is painful, and I'm not sure it should be necessary.

(I used to use furl rather than yarl- the same issue exists with that.)

[ketpatil77]

If repeating str(url) throughout the migration is the main problem, you can normalize your known third-party URL type once at the client's public request-building boundary:

import httpx
import yarl

class YarlClient(httpx.Client):
    def build_request(self, method, url, **kwargs):
        if isinstance(url, yarl.URL):
            url = str(url)
        return super().build_request(method, url, **kwargs)

def handler(request):
    return httpx.Response(200, json={
        "method": request.method,
        "target": request.url.raw_path.decode("ascii"),
        "marker": request.headers["x-example"],
    })

with YarlClient(
    base_url="https://example.com/api/",
    headers={"X-Example": "preserved"},
    transport=httpx.MockTransport(handler),
) as client:
    absolute = yarl.URL("https://example.com/a%2Fb?x=a%2Bb", encoded=True)
    assert client.get(absolute).json()["target"] == "/a%2Fb?x=a%2Bb"
    result = client.post(yarl.URL("child"), json={"value": 1})
    assert result.json() == {"method": "POST", "target": "/api/child", "marker": "preserved"}

The inherited high-level request methods use build_request, so this handles both GET and POST without overriding every verb. It retains HTTPX's normal header/body/base-URL handling and only coerces actual yarl.URL objects; other unsupported types still go through HTTPX's validation.

A base_url supplied to the constructor is outside this override, so convert that one explicitly if it is also a yarl object. Direct httpx.Request(...) construction is likewise separate. For async code, apply the same synchronous build_request override to an httpx.AsyncClient subclass; request construction itself is not async.

This is an application adapter rather than native yarl/furl support. Converting at the boundary also makes the handoff between the libraries' URL normalization rules explicit; test encoded-path cases important to your application.

References: HTTPX client request construction, yarl URL string representation.

AI-assisted answer using OpenAI Codex. Tested with Python 3.11.9, HTTPX 0.28.1, and yarl 1.25.1: encoded slash/plus preservation, absolute and relative URLs, GET/POST, and default header preservation using MockTransport.