DomCopWe launched a new feature in February 2016 called the “Personal expired domain crawler”. This is available only under the Guru Plan.
The crawler works differently from the way the DomCop expires lists work. Instead of finding expired domains and seeing if they have great back links, the crawler finds great broken back links and finds if the domain is available.
That is, the crawler crawls websites on your behalf and looks for any broken links from those websites. If a broken link is found, it then checks whether the domain is available. If a domain is found to be available, it then gets all the metrics for the domain that we normally have at DomCop (Majestic, Moz, SimilarWeb, SEMrush, Alexa etc). This is also called expired domains mining.
Every domain that the crawler finds for you, is only available to you. Your list of domains is not shared with any other user.
We have rolled out the guru plan under three sub plans
- Guru Plan I – $184/mo – Gives you 10 crawlers
- Guru Plan II – $372/mo – Gives you 30 crawlers
- Guru Plan III – $556/mo – Gives you 50 crawlers
Each sub plan give you a fixed number of crawlers. For example, the Guru Plan III gives you 50 crawlers – this means that at any time, 50 crawlers will be working on 50 different websites for you. There are no limitations on the number of jobs you can setup, or the number of websites you can crawl.
Crawler Dashboard
When you sign up for the Guru plan, you will see a new link at the top called “Crawler Dashboard“. When you visit the link you will see a screen like the screenshot below
Well, on your first login, this screen will be empty. But as your personal crawler crawls websites and finds domains, you will be able to see the stats on how it is doing on this page.
Creating a new crawl job
To create a new crawl job, click on the “Create new crawl job” button. This will take you to a page where you can choose the type of crawl job you would like to create. I am listing the different types of crawl jobs below. I will explain them in more detail later on in the article.
- Website Crawl – this is the default crawl job, where you can enter a bunch of websites (up to 500) and have the personal crawler crawl all of them for you.
- Keyword Search – with this type of job you can crawl all those websites that are found in the Google search engine results for specific keywords. You can enter up to 20 keywords at once.
- Reverse Website – using this job, you can crawl all those websites that link to a single or a set of websites. This is a great place to enter your competitor websites (or websites in a similar niche). You can enter up to 50 competitor websites at once.
- Niche Website – this is the easiest job to setup. Simply choose your niche from the list shown and the crawler will pick up random websites from your niche.
Choosing settings for each crawl job
Let’s go in details to understand the various settings in each crawl job.
Website Crawl
Enter Websites: As you can see, I’ve added some random websites with Alexa Rank > 50,000. You can choose websites from any such list – Top Alexa, Top Majestic or you can add any other website that you want to crawl.
Crawl Depth: This is the depth to which the website shall be crawled. A higher depth means more pages crawled, but it also means that the crawler takes a long time to complete. You should test with different depths to see what works for you. If you are unsure, go with the default Level 3.
Max Pages to Crawl: If you want the crawler to stop after it has crawled a certain number of pages, you can choose that from here. As you can see I’ve kept this to a max of 5 million pages.
Max Time to Crawl: This is similar to the above setting. It decides how long you want the crawler to run. If I have a large list of jobs setup, I can force this job to end within 16 hours after it starts.
Job Name (optional): Enter a relevant job name if you want. If you leave this empty, a default format name will be applied
Email on completion (optional): If you want to be notified via email when the job is complete, you can enable this checkbox.
Keyword Search Crawl
The keyword search crawler performs searches on Google and gets the top 10/20 results. It then crawls these results.
Enter Keywords: You can enter up to 20 keywords from your niche here.
Country: Choose the country you want the Google search to be done in. Default – US.
Language: Choose the language that you want the Google searches to be performed in. Default – English.
Results Since: If you want to include older search results, choose the year.
SERP Pages: You can choose whether you want the top 20, 50 or 100 results from Google to be crawled.
If you enter 20 keywords and select 20 results from the SERPs you can actually get 20 x 20 = 400 websites from this crawl job.
Reverse Website Crawl Job
Enter Websites: Enter high ranking competitor websites. The crawler will use the SemRush API to get the top 50 backlinks to each of these websites and will crawl those websites. You can enter a maximum of 50 websites here. If each of those has at least 50 backlinks, you will have a total of 50 x 50 = 250 websites to crawl.
Niche Website Crawl Job
Choose your Niche: You can choose the parent and the child category of your niche.
Number of websites to crawl: here you can choose how many niche sites the crawler will crawl for this job
The crawler will randomly pick up websites that fall in your specific niche and start crawling the number of websites that you setup.
Viewing all the domains found and search/sorting by metrics
Your crawled domains are perfectly integrated into the DomCop domain grid. If you have used DomCop before, you will not have any issues with this section.
Your crawled domains show up in a new tab called “Crawler” next to the “Watchlist” tab.
You can search and sort these domains in exactly the same way as you would sort and search through any of the other sections (Expiring, Expired, Archive or Watchlist). You have access to the Simple and Advanced search screens as well as all the metrics.
You can also filter domains on the type of crawl jobs that they were found through or the status of the crawl jobs. So, you can look for all domains that were found through completed keyword search crawl jobs.
You can also filter domains based on a specific crawl job or a specific website inside a crawl job.
The domains in the crawler tab are only shown to you. This is your private list of domains.
Miscellaneous Settings
Show Domains – we find domains with all extensions in our crawls. However, due to the limitation with availability check APIs, we cannot check the availability of all the domains. By default, we do not show these domains to you. However, since these domains might be available, you can choose to see them in the domain list by changing the setting “Crawled Domain Setting” on the My Account page.
Tips
- Try not to crawl famous websites like huffingtonpost.com. These have been crawled by thousands of crawlers and any domains you will find here are most likely worthless.
- Focus on websites in your niche. Since there will be less competition for niche specific websites, your chances of finding great domains go up if you focus on your niche.
- You get a fixed number of crawlers (10) when you sign up. Each crawler can only crawl a single website at one time. It is therefore helpful to have at least 10 websites in every job. (or have multiple jobs)
- Once you’ve set up your jobs, give the crawler time and space to do its work. As per my stats, you need to crawl around 2000 pages to find an available domain.
You can go ahead and purchase the Guru Plan right away!
If you have any questions or face any issues please do not hesitate in contacting me at support@domcop.com
Happy Crawling!
-Sophia
DomCop Fanatical Support