The Events Montréal crawler
Events Montréal brings what’s on in Greater Montréal into one place. To do that, a crawler reads the calendars that venues, festivals and organisers publish on their own websites.
What it reads
iCalendar feeds, RSS and Atom feeds and structured event data (schema.org) on public pages, and sometimes an event’s own page when the calendar does not give its date. It records the title, date, venue, price, description and the address of the image when there is one, with the link to the original page. The public site shows only the facts and the link: it does not republish descriptions or images.
How often
Each source is read on the schedule set for it, once a day by default. The crawler waits at least 1.5 seconds between two requests to the same site. When a site cannot send its robots.txt (a timeout or a server error), the crawler reads nothing there for 1 hour.
robots.txt
The crawler reads each site’s robots.txt before reading any page there, and follows its rules. It always identifies itself with this user agent:
SortiesMontrealBot/1.0 (+https://sortiesmontreal.ca/robot)
One address is queried without robots.txt: OpenStreetMap’s geolocation service (Nominatim), an interface meant for such requests, at most one request a second and under its usage policy.
Blocking it
To stop it reading anything on your site, add these lines to your robots.txt. It takes them into account normally within 24 hours, and after 7 days at the latest (how long it may follow a copy it kept):
User-agent: SortiesMontrealBot
Disallow: /
Contact
For a question, or to have your site or an event removed, write to us: bonjour@sortiesmontreal.ca