Duplicate content (DC) is nothing more, than the same content duplicated on different pages of the website.
DC on our website may arise as a result of deliberate action (unconsolidated content strategy), but it often happens that it is strictly technical (e.g. indexing the same resources under multiple URLs by the page engine) or human error.
Clusteric Auditor can help you automatically identify this kind of situation by finding pages that are almost identical, for example, very similar articles.

How do you find duplicate content on the site?

We start with on-site analysis mode.

In our example, we will try to analyze DIY guide on one of the popular construction-related sites.
Let’s provide the address where we want to start crawling the site.

We choose the analysis of internal links (without other resources, we are only interested in texts).
In addition, we necessarily set the density analysis of words in the content of the page – it is the basis for a later duplicate content scan.

In the next screen we set one more configuration option – in our example we want to analyze the website only in the section devoted to guides.

We run the analysis. When it finishes, we go to the “Generate” and “Similarity scan” tabs.

We’re interested in Google’s point of view, so we’re focusing on indexable resources and without canonical URLs. In addition, you’ll usually want to exclude page text from the analysis that is repeated on many other pages.
These are usually menu elements, constant components of headers or footers, information on privacy policy, cookies, GDPR, etc.

We can also determine what the minimum degree of similarity will cause the texts to appear in the list (scale up to 100 points).
Strong duplicates will usually have over 70 points, but by setting lower values ​​we can also look for articles that are simply related thematically and use the statement for a slightly different purpose (e.g. adding related or internal links, change content strategy or rewrite them).
We run the analysis and after a while (depending on the size of the project) we receive:

If we look at the results in our example, we will see that the list has two strong duplicates (90 and 96 points), but also texts with a very high degree of coverage with minor changes relative to each other (73 points).

The resulting report can also be exported to XLS.

We strongly advise checking duplicate/similar content on your website to avoid technical problems, cannibalisation and improve content strategy.
The free version provides a full on-site report for 1000 URLs and this option is also limited to 1000 URLs, which is almost like unlimited as you can use filters in the crawl to split analyses.

The contact data collection module has been extended with additional fields of telephone numbers and NIP / TAX field

For the harvester, we’ve added the option to limit the URLs collected below the first 100 results.

changelog:
1.79, 30/10/2019,
Application notation on macOS.
Retaining anchor classification when adding new links to the project.
Extension of contact data collection by telephone numbers and NIP numbers.
An additional filter in the URL clipping tool.
Harvester – the ability to limit the number of URLs collected below the first 100 results.

1.79.5, 11/11/2019,
Adapting the program to changes in the structure of Google results.

We have also worked on improving the speed of other analyzes, especially in conditions of reduced proxy performance.

1.77 from 27.05.2019
Keyword Difficulties – Adjustments to match the new presentation of data in search results.
Export / import settings and custom evaluation formulas to / from file.

1.77.5, 10/06/2019
Important update for the new version of macOS.
Analyze Google indexation.
Keyword difficulty mode – performance update.

1.78, 10/06/2019,
Bug fix: some analysis was slowed down.

The report will contain information on the statistical distribution of keywords linking to our domain and the share of these anchors in individual groups (money, brand, etc.). In anchor classification, you can define your own groups by analyzing, for example, the popularity of a given section or author. This type of distribution can also be used to define linking strategies and content marketing.

Changelog: 1.76:
New reports: anchor classification.
Important update for Win 10 users (v. 1903).

We described exactly how to use this feature in the article: Recognition of similarities and duplicate content in Clusteric

Changelog: 1.75:
On-site audit: DC scan / similarities (“Generate …” tab).
On-site audit: support for gzip compressed pages.
Additional Meuse API settings.

We also detect page associations at the IP level, not just at the domain.

Changelog 1.74:
New formula: detection of pages on IP addresses, not domains.
On-site audit: word density (URL details).
Amendments.

changelog:
On-site audit – sitemap generator.
Small tools – combining XLS files – preserving duplicate columns.
Improving Chinese, Korean and Japanese language recognition.
Amendments.

When analysing incoming links, SEO reports are very helpful for understanding their quantity and quality.

Automated creation of SEO reports, showing both the basic data on the number of links and their parameters, keywords and the profile evaluation of these links are now available in Clusteric.

A typical SEO report consists of

Table of contents

1. Audit/domain overview
2. Link profile overview
2.1. Links’ health
2.2. Link status
3. How the website is linked
3.1. SITE-WIDE links
3.2. Rel attribute
3.3. Anchors distribution
4. Where the links come from
4.1. Location
4.2. Linking websites
5. Authority
5.1. Indexation
5.2. Social metrics
5.3. Link strength metrics
5.4. Traffic ranks
5.5. Risk factors
6. Audit summary
6.1. General advice

Creating a report, after analysing links first, is completely automated.

In the agency version, it is additionally possible to export reports to open files (.doc) for further processing.

Reports in the agency version do not have Clusteric branding and are also a great opportunity to acquire new clients. With a small amount of time invested in creating reports on the initial situation of the audited website and the professional appearance of these reports, conversations are easier.

How to create an automated SEO report for backlink evaluation?

1. Download the backlinks you want to analyse. If you have other backlink sources, add them as well.

2. Identify the keywords being researched (up to 1000 keywords).

3. Specify how large the sample of links is to be tested.
Side-wide links have similar parameters, you should not always test them all, especially for large amounts of data.

4. Collect data using Clusteric (the amount of data depends on the chosen mode).
5. Clusteric will suggest an automatic rating and divide the links into several categories.
5. Audit the links according to your own criteria. Extract trusted domains and investigate spam profiles. Assign keywords and analyse content based on link profiles.
6. After the audit, generate a PDF report.

Automatically generated PDF reports on the surveyed link profile occupy about 35 A4 pages. They include descriptions of SEO issues, analysis of the current situation as well as charts and tag clouds.

You can download a sample SEO report from backlink analysis here:

Examples of report elements.

This report was created entirely automatically, based on data from the backlink database collected by Clusteric robots and the default data sampling.

If you need additional data in the report or you have an idea how to interpret them differently, please contact us.

This tool has innumerable applications in the process of combining and processing analytical data from various sources.

Starting from combining data downloaded from the Google Search Console with Google Analytics data at the URL level, through data from external sources combined with internal data.

A few examples of application:

E-commerce and SEO.

Sales data at the level of a given product, combined with the quality and quantity of links leading to the subpage of this product.

E-commerce and customer satisfaction (NPS)
Analytics of customer satisfaction with the product or support on the chat combined with sales from individual URLs.

Advanced SEO.

Merged report: keywords (GSC), website speed (GA), sales (GA), margin (internal data), quality of subpage (PA) and position on given keywords (external data, position monitoring) for a given language.

Analysis of expired domains for the PBN.

The combination of the export “Expired domains” with data on quantity and keywords (titles) downloaded from WebArchive (crawl onsite with individual parameters).

The possibilities of combining and analyzing data are unlimited.

How to work with the program and efficiently combine files?

1. Prepare the XLS files so that the column with the URL is first.
2. The files contain column names in the first row. Watch out for exports from Google Search Console, these usually contain a description of the export at the beginning of the file.
3. Make sure that there are no duplicates in the files.
4. If you use data with relative URLs, such as exports from Google Analytics (/product name.html), add the domain in the “Prefix” field.
5. Merge files and work on ordered data.

Use XLS files to prserv data formatting.

Change log:

New small tool – merging xls / xlsx files.
Proxy server management – optimizations.
version for MacOS – problems with auditing sites using TLS have been fixed.