Websites
This page introduces Websites in SearchBot, explains what they are and discusses their use.
Introduction
What are Websites in SearchBot?
Websites are targets for SearchBot to scrape. When you create a new website, SearchBot will run a crawler that downloads all of the content of your website. It follows all internal links on your website, so it only scrapes what is navigable by your users.
Once a page has been downloaded, it is stored and given a unique ID. That website is then ready to be consumed by a Bot.
Creating a Website
To create a website, navigate to the Websites page in SearchBot, then click Add Website.
Website fields
The following table describes the fields contained within a Website.
| Name | Description |
|---|---|
| Last Scraped At | When the website was last crawled, scraped and stored. |
| Last Scrape State | The state of the latest scrape (“Unknown”, “Created”, “Started”, “Finished”). |
| Last Scrape Status | The status of the latest scrape (“Unknown”, “Running”, “Succeeded”, “Failed”). |
| Last Scrape Progress | A depiction of the progress of the latest scrape. |
Deleting a Website
To delete a website, press the button on the far right of the list of Websites.
Last updated on