Anti-scraping Efforts by Realtor

Realtor.com and Move.com have undergone efforts to block ‘web bots’ from scraping listings from over one million pages a day, and so far they have been quite successful in preventing the activity.

Nearly 140 million interactions are monitored by the companies each day, identifying patterns which highlight such undesired behaviour like the systematic copying of listing data, and when detected, the suspicious IPs are blocked and thus not allowed to access the property portals again.

Scraping is commonly done online, and the perpetrators are able to perform the act at a low cost, by hiring offshore third party companies.

An example of a solicitation from a website where companies or individuals post projects up for bid was posted on Realtor’s Lockbox Blog.

—–

—–

“I am looking for someone to write a program to scrape data from Realtor.com,” stated the bid request. “The program would take U.S. ZIP codes (from a ‘.txt’ file), input them into Realtor.com, and then take the output from the pages that follow and create a ‘.csv’ file for each ZIP code. Ultimately, I will be running the program for thousands of different ZIP codes. I am willing to pay up to $50 for this program.”

According to a survey conducted by the Council of Multiple Listing Services (CMLS), protecting listing data is a key issues that MLSs face today. Almost 80% of the respondents who participated in the survey, expressed concern regarding unauthorised online use of their data.

Realtor.com stated on its blog: ‘We effectively block about 1.5 MILLION pages from being scraped everyday. And our blocking is so successful, that scraping attempts have declined tremendously on REALTOR.com®.”

“We understand that agents work hard for their clients, and they work hard for their listing data – and REALTOR.com® and MOVE are committed to ensuring that they decide what happens to it.”

Leave a Comment

*


six + 3 =