# All roboots will spider the domain User-agent: * Disallow: # Disallow directory /gallery/ User-agent: * Disallow: /gallery/ # Disallow directory /development/ User-agent: * Disallow: /development/ # Disallow directory /includes/ User-agent: * Disallow: /includes/ # Disallow directory /modules/ User-agent: * Disallow: /modules/