Author here - This tool is temporarily not working.

[harry-s-grewal]

Context:

Realtor.ca does its best to block scraping. They have a lot of money that they put behind that - for example, they totally blocked my attempts to use ChromeDriver through Selenium (even when I used the 'undetectable' ChromeDriver).

Issue:

They changed the way the search works. Check out the headers, they seem to include a cookie now, plus some encoding options. Don't have the time to dig deep into it now.

Next Steps:

If you have any suggestions, leave them as a reply.

[whydna]

any way to contact you? have some ideas we can collab on.

[rydeordie99]

@harry-s-grewal I solved the issue! Requires the session cookies to be grabbed by a browser in undetectable mode first, then those can be fed into the post request. I used patchright in chromium browser mode. I added a lot of values to the data form, and I am unsure which ones are necessary though.

[JohanHuynh0130]

@rydeordie99 hey man you mind showing me how? Was struggling at this haha. Thank you so much!!!

[rydeordie99]

@JohanHuynh0130 Here is some partial code that was working previously. You just use the headers, and data in your request and it should work. Key things were the Cookie key in the headers that had to be generated after reaching the page using Patchright browser.

import requests
from patchright.sync_api import sync_playwright

with sync_playwright() as playwright:
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://realtor.ca")
    page.wait_for_timeout(5000)
    page.goto("https://realtor.ca/map")
    cookies = page.context.cookies()
    browser.close()

cookie_string = ""
for cookie in cookies:
    cookie_string += f'{cookie["name"]}={cookie["value"]}; '

headers = {
    "Referer": "https://www.realtor.ca/",
    "Origin": "https://www.realtor.ca/",
    "Host": "api2.realtor.ca",
    "Cookie": cookie_string,
}
data = {
    "LatitudeMin": "53.39539675355482",
    "LatitudeMax": "53.51182802662509",
    "LongitudeMin": "-113.71370377219628",
    "LongitudeMax": "-113.41831081208356",
    "BuildingTypeId": "1",
    "RecordsPerPage": "200",
    "CultureId": "1",
    "CurrentPage": "1",
    "ApplicationId": "1",
    "ZoomLevel": "12",
    "Sort": "6-D",
    "PropertyTypeGroupID": "1",
    "TransactionTypeId": "2",
    "PropertySearchTypeId": "0",
    "NumberOfDays": "2",
    "Currency": "CAD",
    "IncludeHiddenListings": "false",
    "Version": "7.0",
}

r = requests.post(
    "https://api2.realtor.ca/Listing.svc/PropertySearch_Post",
    headers=headers,
    data=data,
)

[rachitt96]

I have created on Python library which is working. If anyone wants to use that:

pip install pyRealtor

PyPi link: https://pypi.org/project/pyRealtor/

And if you don't want to use Python, then I have created front-end website using pyRealtor as back-end: https://pyrealtor.web.app/

[harry-s-grewal]

@rydeordie99 I tried your code and it doesn't seem to work anymore. Doing some experiments, but I'm not sure what I'm missing.