Our crawler

If you run an estate agency in Spain, you may have seen BaldosaBot in your logs. It’s ours.

It reads public listing pages so that Baldosa, a free weekly email of Spanish homes, can feature houses and send readers to the agency selling them. Until October 2026 it was called SpainUniqueHomesBot, after our working title, Spain Unique Homes.

What it reads

Public pages only: your sitemap, your listing pages and the photographs on them. From a listing it keeps the title, price, size, location, description, photo links and the address of the page, so readers can go straight to it. It doesn’t log in or fill in forms, and it reads nothing behind a password.

How it behaves

• It names itself in every request, with a way to reach us: Mozilla/5.0 (compatible; BaldosaBot/1.0; +https://baldosa.casa/bot/; agencias@baldosa.casa)
• It follows your robots.txt, including wildcard rules and Crawl-delay. Rules you wrote for SpainUniqueHomesBot still apply to it, and if your robots.txt has rules for both names, it follows the stricter one.
• It waits at least 1.5 seconds between requests to your site and opens one connection at a time.
• It visits about once a day and reads pages it hasn’t seen before. Now and then it checks your whole list of listings to see which have gone.
• If your site asks browsers to run a JavaScript check, it opens the page in a real browser that still calls itself BaldosaBot. It won’t solve a CAPTCHA, hide behind a proxy or use another name.
• A 401, 403 or 429 answer, or a robots.txt rule, stops it.

What we do with it

Each week Baldosa features a handful of houses. Every pick names your agency and links to your listing. Before an issue goes out we check each pick on your site again, so a house you’ve sold drops out.

We also look at listing photos to spot the same house listed by more than one agency, so we can avoid featuring it twice.

How to opt out

Write to agencias@baldosa.casa (a one-word reply, «baja» or «no», is enough), or add these two lines to your robots.txt:

User-agent: BaldosaBot
Disallow: /

We’ll stop reading your site, take your listings out of future issues and stop writing to you, and we’ll confirm it by email.

How to let it in

If your firewall or bot check turns BaldosaBot away, you can allow its name. In robots.txt it takes these two lines:

User-agent: BaldosaBot
Allow: /

Rather send us your listings?

A Kyero XML feed works well: send the link to agencias@baldosa.casa. Once we load your feed, we read it instead of your site.