This page introduces Websites in SearchBot, explains what they are and discusses their use.


What are Websites in SearchBot?

Websites are targets for SearchBot to scrape. When you create a new website, SearchBot will run a crawler that downloads all of the content of your website. It follows all internal links on your website, so it only scrapes what is navigable by your users.

Once a page has been downloaded, it is stored and given a unique ID. That website is then ready to be consumed by a Bot.

Creating a Website

To create a website, navigate to the Websites page in SearchBot, then click Add Website.

Website fields

The following table describes the fields contained within a Website.

Last Scraped AtWhen the website was last crawled, scraped and stored.
Last Scrape StateThe state of the latest scrape (“Unknown”, “Created”, “Started”, “Finished”).
Last Scrape StatusThe status of the latest scrape (“Unknown”, “Running”, “Succeeded”, “Failed”).
Last Scrape ProgressA depiction of the progress of the latest scrape.

Deleting a Website

To delete a website, press the button on the far right of the list of Websites.

Last updated on