SharePoint Document Issues

How to Find Duplicate SharePoint Files Using PowerShell

Maira Mehtab Sep 22, 2026 869 views

Question details

The user needs a method to identify duplicate files across a large SharePoint document library containing over 30,000 files in multiple folders and wants to output their exact file paths using PowerShell.

Product: SharePoint, PowerShell
Device & OS: not provided
Scenario: Managing and cleaning up a massive volume of files in a SharePoint document library without relying on a manual Excel export.
Observed behavior: The user wants a script to scan, identify duplicates by file hash, and export the file paths to a CSV, while overcoming potential connection errors related to modern authentication changes.

Before you start

Ensure you have PowerShell 7 or later installed on your system and possess sufficient administrative privileges to register an Entra ID application for modern SharePoint authentication.

Solution 1Recommended

Use PnP PowerShell to Identify and Export Duplicate Files

This method utilizes PnP PowerShell to connect to your SharePoint site, calculate file hashes to find true duplicates, and export the results to a CSV file.

Due to recent PnP PowerShell authentication updates in 2024, standard connection methods may return a 'Specified method is not supported' error. You must register an Entra ID application and use an interactive login with a Client ID to successfully authenticate.

Install PnP PowerShell

Open PowerShell 7 or later and run the command 'Install-Module -Name PnP.PowerShell' to ensure you have the latest module required for SharePoint management.

Register an Entra ID Application

Navigate to the Microsoft Entra admin center, register a new application, and grant it the required SharePoint permissions to obtain a valid Client ID for authentication.

Connect to SharePoint Online

Run the connection command 'Connect-PnPOnline <site URL> -Interactive -ClientId <client ID>', replacing the placeholders with your actual site URL and the generated Client ID.

Enumerate Files and Calculate Hashes

Write a script loop to iterate through the document libraries and their files. Use the file properties to calculate a unique hash for each file.

Group and Export to CSV

Group the files by their calculated hashes. Filter for groups containing more than one item (duplicates) and use the 'Export-Csv' cmdlet to output these groups along with their full paths to a local CSV file.

Authentication Updates: If you receive a 'Specified method is not supported' error during connection, it indicates you need to switch to the Entra ID interactive authentication method with a registered Client ID.

Free Microsoft Office alternative

Easily Manage Your Exported CSV Data with WPS Office

After running your PowerShell script and exporting your duplicate SharePoint file lists to a CSV, use WPS Office to effortlessly open, format, and analyze your spreadsheets. WPS Office is a lightweight, highly compatible, and free alternative to Microsoft Office.

1. Open WPS Spreadsheet: Launch the WPS Office application and open the Spreadsheet module.
2. Import the Exported CSV: Go to 'File', click 'Open', and navigate to the location where your PowerShell script saved the CSV file.
3. Analyze Duplicate Data: Use WPS Spreadsheet's built-in sorting and filtering tools to organize your file paths and effectively plan your SharePoint cleanup.

Completely free and lightweight office suite for daily productivitySeamlessly open, edit, and analyze CSV and Excel (.xlsx) filesHighly compatible with Microsoft Office formats and layoutsFamiliar and intuitive user interface requires no learning curve

Frequently Asked Questions

Why do I get a 'Specified method is not supported' error when connecting to PnP PowerShell?

This error generally occurs because of the 2024 PnP PowerShell authentication changes. To resolve it, you must register an Entra ID application and include the -ClientId parameter alongside the -Interactive flag in your connection command.

Do I need to download all 30,000 files to find duplicates?

No. By using PnP PowerShell, you can access file metadata and calculate file hashes directly on the SharePoint server side, which allows you to identify duplicates without downloading the actual file contents to your local machine.

Is PnP.PowerShell compatible with older versions of PowerShell?

It is highly recommended to install and run PnP.PowerShell in PowerShell 7 or later. Older versions like Windows PowerShell 5.1 may lack compatibility with newer authentication modules and modern security standards.

Document & File Issues