mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-14 15:52:18 +00:00
2025.11.25.0
3
Home.md
3
Home.md
@@ -204,6 +204,9 @@ This button looks like a bookmark. When you click on this button, a form for dow
|
|||||||
|
|
||||||
You may see a message like `Host [Site] is not ready` or `Host [Site] is unavailable`. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time.
|
You may see a message like `Host [Site] is not ready` or `Host [Site] is unavailable`. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time.
|
||||||
|
|
||||||
|
Hot-keys:
|
||||||
|
- `Esc` - close the form
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Download
|
### Download
|
||||||
|
|||||||
82
Settings.md
82
Settings.md
@@ -243,6 +243,7 @@ Version **2025.09.05** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/
|
|||||||
- [YouTube](#youtube)
|
- [YouTube](#youtube)
|
||||||
- [ThisVid](#thisvid)
|
- [ThisVid](#thisvid)
|
||||||
- [TikTok](#tiktok)
|
- [TikTok](#tiktok)
|
||||||
|
- [xHamster](#xhamster)
|
||||||
|
|
||||||
### OF-Scraper
|
### OF-Scraper
|
||||||
<!--
|
<!--
|
||||||
@@ -273,11 +274,7 @@ You can modify an existing query if you need to. This can be suitable when the s
|
|||||||
|
|
||||||
The Reddit parser can parse data without cookies, but you can add it if you like.
|
The Reddit parser can parse data without cookies, but you can add it if you like.
|
||||||
|
|
||||||
**You don't need to set up [credentials](#how-to-get-reddit-credentials) like `Login`, `Password`, `Client ID`, `Client Secret` and `Bearer token`. The OAuth authorization ability has been added for future purposes. You can configure them if you want, but Reddit allows you to download without them. On the other hand, if you use cookies to download timelines, you need to set up credentials.**
|
**You need to set up [credentials](#how-to-get-reddit-credentials) like `Login`, `Password`, `Client ID`, `Client Secret` and `Bearer token` (don't copy manually). When setting up OAuth, you need to check `Use the token to download the timeline` & `Use cookies to download the timeline`.**
|
||||||
|
|
||||||
**Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth!**
|
|
||||||
|
|
||||||
**If nothing downloads, use OAuth!**
|
|
||||||
|
|
||||||
- Authorization
|
- Authorization
|
||||||
- `Login` - your authorization username ([how to get](#how-to-get-credentials))
|
- `Login` - your authorization username ([how to get](#how-to-get-credentials))
|
||||||
@@ -290,6 +287,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
|
|||||||
- `Use the token to download saved posts`
|
- `Use the token to download saved posts`
|
||||||
- `Use cookies to download the timeline`
|
- `Use cookies to download the timeline`
|
||||||
- `Saved posts user` - your personal Reddit username to download your saved posts (this feature requires cookies)
|
- `Saved posts user` - your personal Reddit username to download your saved posts (this feature requires cookies)
|
||||||
|
- `Concurrent downloads` - the number of concurrent downloads
|
||||||
<!--
|
<!--
|
||||||
- Other parameters
|
- Other parameters
|
||||||
- `Use M3U8` - This setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
|
- `Use M3U8` - This setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
|
||||||
@@ -297,6 +295,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
|
|||||||
- `Check image: get original` - get the original image if it exists
|
- `Check image: get original` - get the original image if it exists
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Reddit requirements
|
### Reddit requirements
|
||||||
@@ -339,6 +339,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
|
|||||||
- `Concurrent downloads` - the number of concurrent downloads.
|
- `Concurrent downloads` - the number of concurrent downloads.
|
||||||
- `Use the new Twitter icon (X)` - *restart SCrawler to take effect*
|
- `Use the new Twitter icon (X)` - *restart SCrawler to take effect*
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Twitter requirements
|
### Twitter requirements
|
||||||
@@ -374,6 +376,8 @@ To completely download a large profile, you can use sleep timers. The sleep time
|
|||||||
- `User name`- your account username (handle)
|
- `User name`- your account username (handle)
|
||||||
- `Password` - your account password
|
- `Password` - your account password
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Bluesky requirements
|
### Bluesky requirements
|
||||||
@@ -397,7 +401,9 @@ To completely download a large profile, you can use sleep timers. The sleep time
|
|||||||
- `keydb_api` - change the keydb_api. **Change this value only if you know what you are doing.**
|
- `keydb_api` - change the keydb_api. **Change this value only if you know what you are doing.**
|
||||||
- `Private key` - path to the DRM key file `private_key.pem`
|
- `Private key` - path to the DRM key file `private_key.pem`
|
||||||
- `Client ID` - path to the DRM key file `client_id.bin`
|
- `Client ID` - path to the DRM key file `client_id.bin`
|
||||||
- `backend` - the value of `advanced_options` in the configuration. If you can't download the video, try using 'httpx'.
|
- `backend` - the value of `advanced_options` in the configuration. If you can't download the video, try using `httpx`.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
@@ -519,6 +525,8 @@ To support the downloading of this site you should add the Mastodon domain to th
|
|||||||
|
|
||||||
**If an error occurred and the log told you that you need to update your credentials, you also need to enable the disabled functions (`Download timeline`, `Download stories`) back.**
|
**If an error occurred and the log told you that you need to update your credentials, you also need to enable the disabled functions (`Download timeline`, `Download stories`) back.**
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Instagram data means
|
### Instagram data means
|
||||||
@@ -635,6 +643,8 @@ result_cancel-->[*]
|
|||||||
- Download
|
- Download
|
||||||
- `Download data` - The internal value indicates that site data should be downloaded. It becomes unchecked when the site returns an error. When you update your credentials, you need to checked this check box.
|
- `Download data` - The internal value indicates that site data should be downloaded. It becomes unchecked when the site returns an error. When you update your credentials, you need to checked this check box.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Threads requirements
|
### Threads requirements
|
||||||
@@ -658,6 +668,8 @@ The `Accept` and `UserAgent` headers must be obtained from the page `https://fac
|
|||||||
|
|
||||||
If you can't find `x-ig-app-id` or `x-asbd-id`, you can get them from Instagram.
|
If you can't find `x-ig-app-id` or `x-asbd-id`, you can get them from Instagram.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Facebook requirements
|
### Facebook requirements
|
||||||
@@ -678,6 +690,8 @@ The `User ID` and `UserAgent` values must be obtained from the page `https://jus
|
|||||||
|
|
||||||
**How to find [headers and values](#how-to-find-headers)**
|
**How to find [headers and values](#how-to-find-headers)**
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### JustForFans requirements
|
### JustForFans requirements
|
||||||
@@ -701,6 +715,8 @@ The `User ID` and `UserAgent` values must be obtained from the page `https://jus
|
|||||||
- `UserAgent` - UserAgent to use in requests ([how to find](#how-to-find-useragent))
|
- `UserAgent` - UserAgent to use in requests ([how to find](#how-to-find-useragent))
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### RedGifs requirements
|
### RedGifs requirements
|
||||||
@@ -740,12 +756,15 @@ The `User ID` and `UserAgent` values must be obtained from the page `https://jus
|
|||||||
|
|
||||||
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings).
|
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings).
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### YouTube requirements
|
### YouTube requirements
|
||||||
- [`Cookies`](#how-to-set-up-cookies) for private data
|
- [`Cookies`](#how-to-set-up-cookies) for private data
|
||||||
- [**yt-dlp**](#YT-DLP)
|
- [**yt-dlp**](#YT-DLP)
|
||||||
- [**ffmpeg**](#ffmpeg)
|
- [**ffmpeg**](#ffmpeg)
|
||||||
|
- [**Deno**](#deno)
|
||||||
|
|
||||||
### YouTube operational API
|
### YouTube operational API
|
||||||
To download communities, you need to install the `YouTube operational API`.
|
To download communities, you need to install the `YouTube operational API`.
|
||||||
@@ -781,10 +800,33 @@ cd C:\wamp64\www\YouTube-operational-API\
|
|||||||
git pull https://github.com/Benjamin-Loison/YouTube-operational-API
|
git pull https://github.com/Benjamin-Loison/YouTube-operational-API
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Deno
|
||||||
|
**With the release of yt-dlp version `2025.11.12`, external JavaScript runtime support has arrived.**
|
||||||
|
**All users who intend to use yt-dlp with YouTube are strongly encouraged to install one of the supported JS runtimes.**
|
||||||
|
|
||||||
|
The following JavaScript runtimes are currently supported (_in order of recommendation, from strongest to weakest_):
|
||||||
|
|
||||||
|
- **recommended for most users**
|
||||||
|
- https://deno.com/
|
||||||
|
- https://github.com/denoland/deno
|
||||||
|
- note: *if downloading from [Deno's GitHub releases](https://github.com/denoland/deno/releases/latest), get `deno` **not** `denort`*
|
||||||
|
- minimum Deno version supported by yt-dlp: `2.0.0`
|
||||||
|
- *the latest version of Deno is strongly recommended*
|
||||||
|
|
||||||
|
**You can find full information [here](https://github.com/yt-dlp/yt-dlp/issues/15012)**
|
||||||
|
|
||||||
|
#### Install Deno
|
||||||
|
Just launch PowerShell and run the following command:
|
||||||
|
```PowerShell
|
||||||
|
irm https://deno.land/install.ps1 | iex
|
||||||
|
```
|
||||||
|
|
||||||
## Pinterest
|
## Pinterest
|
||||||
- `Concurrent downloads` - the number of concurrent downloads.
|
- `Concurrent downloads` - the number of concurrent downloads.
|
||||||
- `Saved posts user` - personal profile username.
|
- `Saved posts user` - personal profile username.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### Pinterest requirements
|
### Pinterest requirements
|
||||||
@@ -802,6 +844,8 @@ git pull https://github.com/Benjamin-Loison/YouTube-operational-API
|
|||||||
- `Use video date as file date` - set the file date to the date the video was added (website) (if available)
|
- `Use video date as file date` - set the file date to the date the video was added (website) (if available)
|
||||||
- `Use video date as file date (standalone downloader)` - set the file date to the date the video was added (website) (if available)
|
- `Use video date as file date (standalone downloader)` - set the file date to the date the video was added (website) (if available)
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### TikTok requirements
|
### TikTok requirements
|
||||||
@@ -841,6 +885,8 @@ if exist %p% (
|
|||||||
- `Download GIFs as mp4` - download gifs in 'mp4' format instead of native 'webm'.
|
- `Download GIFs as mp4` - download gifs in 'mp4' format instead of native 'webm'.
|
||||||
- `Saved posts user` - personal profile username (to download saved posts)
|
- `Saved posts user` - personal profile username (to download saved posts)
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### PornHub requirements
|
### PornHub requirements
|
||||||
@@ -859,9 +905,12 @@ About photos. Photo download problems have the lowest priority. You can still cr
|
|||||||
- `Download UHD` - Download UHD (4K) content
|
- `Download UHD` - Download UHD (4K) content
|
||||||
- `Re-encode downloaded videos if necessary` - If enabled and the video is downloaded in a non-native format, the video will be re-encoded. **Attention! Enabling this setting results in maximum CPU usage.**
|
- `Re-encode downloaded videos if necessary` - If enabled and the video is downloaded in a non-native format, the video will be re-encoded. **Attention! Enabling this setting results in maximum CPU usage.**
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### XHamster requirements
|
### XHamster requirements
|
||||||
|
- [yt-dlp](#YT-DLP)
|
||||||
- [ffmpeg](#ffmpeg) is required anyway
|
- [ffmpeg](#ffmpeg) is required anyway
|
||||||
- [`Cookies`](#how-to-set-up-cookies) for downloading saved posts
|
- [`Cookies`](#how-to-set-up-cookies) for downloading saved posts
|
||||||
|
|
||||||
@@ -869,6 +918,8 @@ About photos. Photo download problems have the lowest priority. You can still cr
|
|||||||
- `Download UHD` - Download UHD (4K) content.
|
- `Download UHD` - Download UHD (4K) content.
|
||||||
- `Playlist of saved videos` - Your personal videos playlist to download as 'saved posts'. This playlist must be private (Visibility = `Only me`). It also required cookies. This playlist must be entered by pattern: `https://www.xvideos.com/favorite/01234567/playlistname`.
|
- `Playlist of saved videos` - Your personal videos playlist to download as 'saved posts'. This playlist must be private (Visibility = `Only me`). It also required cookies. This playlist must be entered by pattern: `https://www.xvideos.com/favorite/01234567/playlistname`.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### XVIDEOS requirements
|
### XVIDEOS requirements
|
||||||
@@ -882,6 +933,8 @@ About photos. Photo download problems have the lowest priority. You can still cr
|
|||||||
- `Favourite videos` - download favourite videos.
|
- `Favourite videos` - download favourite videos.
|
||||||
- `Different folders` - Use different folders to store video files. If `true`, then public videos will be stored in the `Public` folder, private - in the `Private` folder. If `false`, all videos will be stored in the `Video` folder.
|
- `Different folders` - Use different folders to store video files. If `true`, then public videos will be stored in the `Public` folder, private - in the `Private` folder. If `false`, all videos will be stored in the `Video` folder.
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### ThisVid requirements
|
### ThisVid requirements
|
||||||
@@ -900,6 +953,8 @@ About photos. Photo download problems have the lowest priority. You can still cr
|
|||||||
|
|
||||||
## LPSG
|
## LPSG
|
||||||
|
|
||||||
|
[See also site default fields](#site-default-fields)
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
### LPSG requirements
|
### LPSG requirements
|
||||||
@@ -942,6 +997,9 @@ It is very easy to create a new group. Just click the `Add a new download group`
|
|||||||
- `Sites` - You can select sites.
|
- `Sites` - You can select sites.
|
||||||
- `Pencil` button for selected sites. Only users from the sites you have selected will be downloaded.
|
- `Pencil` button for selected sites. Only users from the sites you have selected will be downloaded.
|
||||||
- `Red X` button for excluded sites. Users from the sites you have selected will be excluded from the download.
|
- `Red X` button for excluded sites. Users from the sites you have selected will be excluded from the download.
|
||||||
|
- `Groups` - You can select groups/
|
||||||
|
- `Pencil` edit groups. The groups you selected will **also** be downloaded
|
||||||
|
- `Left checkbox` - If checked, only the selected groups will be downloaded. All other options will be ignored. If unchecked: all the options of the new group will be used to filter the users to be downloaded; the selected groups will be also downloaded "as is", using their options.
|
||||||
|
|
||||||
You can set both selected and excluded options for each selection.
|
You can set both selected and excluded options for each selection.
|
||||||
|
|
||||||
@@ -978,9 +1036,8 @@ You can set up automatic downloads. You can find these settings in `Settings - A
|
|||||||

|

|
||||||
|
|
||||||
**Plan**:
|
**Plan**:
|
||||||
- `Disable` - Disable automatic downloads.
|
- `Disabled` - Disable automatic downloads.
|
||||||
- `Specified` - Specify custom download options (read more [here](#download-groups)).
|
- `Enabled` - Enable automatic downloads (read more [here](#download-groups)).
|
||||||
- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download-groups)).
|
|
||||||
- `Show notifications` - Show notification when there is new downloaded data.
|
- `Show notifications` - Show notification when there is new downloaded data.
|
||||||
- `Simple` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored.
|
- `Simple` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored.
|
||||||
- `Image` - Show downloaded image in notification.
|
- `Image` - Show downloaded image in notification.
|
||||||
@@ -1002,7 +1059,7 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
|
|||||||
# How to set up cookies
|
# How to set up cookies
|
||||||
|
|
||||||
## First method
|
## First method
|
||||||
1. Using a browser extension to receive site cookies, copy the cookies (or save them to a file) (I use this to get cookies: [Official site](https://www.editthiscookie.com/blog/2014/03/install-editthiscookie/), [Chrome Web Store](https://chromewebstore.google.com/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg)).
|
1. Using a browser extension to receive site cookies, copy the cookies (or save them to a file) (you can use one of the [recommended extensions](#list-of-cookie-editing-extensions) or any other).
|
||||||
1. Open the settings of the site you want to configure (for example, `Main window` - `Settings` - `Instagram`).
|
1. Open the settings of the site you want to configure (for example, `Main window` - `Settings` - `Instagram`).
|
||||||
1. Find the `Cookies` field.
|
1. Find the `Cookies` field.
|
||||||
1. Click on the `Pencil` button.
|
1. Click on the `Pencil` button.
|
||||||
@@ -1010,7 +1067,10 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
|
|||||||
1. Paste the copied cookies text into the window that opens (or press `Ctrl+O` to load cookies from a saved file (if you saved cookies to a file)) and click `OK`.
|
1. Paste the copied cookies text into the window that opens (or press `Ctrl+O` to load cookies from a saved file (if you saved cookies to a file)) and click `OK`.
|
||||||
1. Click OK to close the cookies editor and save the cookies.
|
1. Click OK to close the cookies editor and save the cookies.
|
||||||
|
|
||||||
**Another similar extension: [Official site](https://cookie-editor.com/), [Chrome Web Store](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)**
|
### List of cookie editing extensions
|
||||||
|
- EditThisCookie ❌ - [Official site](https://www.editthiscookie.com/blog/2014/03/install-editthiscookie/), [Chrome Web Store](https://chromewebstore.google.com/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg), [GitHub](https://github.com/fcapano/Edit-This-Cookie)
|
||||||
|
- **EditThisCookie v3** ✔️ - *(EditThisCookie fork)* [Official site](https://editcookie.com/), [GitHub](https://github.com/mrfanii/Edit-This-Cookie-V3), [Chrome Web Store](https://chromewebstore.google.com/detail/editthiscookie-v3/ojfebgpkimhlhcblbalbfjblapadhbol)
|
||||||
|
- **Cookie-Editor** ✔️ - [Official site](https://cookie-editor.com/), [GitHub](https://github.com/moustachauve/cookie-editor), [Chrome Web Store](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
|
||||||
|
|
||||||
## Second method
|
## Second method
|
||||||
1. Open Google Chrome, Microsoft Edge or FireFox.
|
1. Open Google Chrome, Microsoft Edge or FireFox.
|
||||||
|
|||||||
Reference in New Issue
Block a user