geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix links

Andy
2023-05-09 05:12:44 +03:00
parent 820822b1ac
commit d25e2b1319
3 changed files with 66 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Home.md

@@ -200,7 +200,7 @@ Double-click the user to open the content folder.
## Download separate video
Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader)
Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
![Separate video downloader window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTube.png)
@@ -298,7 +298,7 @@ You can pick up one or more labels in this form.
### Deleting a user
SCrawler has advanced deletion options. It can delete a user/[collection](#Deleting%20a%20collection) with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options.
SCrawler has advanced deletion options. It can delete a user/[collection](#Deleting-a-collection) with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options.
```mermaid
graph TB
@@ -344,7 +344,7 @@ When you add a user to the collection, a new folder will be created in the colle
- `Split` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
- `Cancel` - Cancel operation.
Go to a [diagram](#Deleting%20a%20user) that illustrates all the available options.
Go to a [diagram](#Deleting-a-user) that illustrates all the available options.
### Collections example

5
Plugins.md

@@ -268,6 +268,8 @@ interface IPluginContentProvider : IDisposable
event ProgressChangedEventHandler ProgressChanged;
delegate void ProgressMaximumChangedEventHandler(int Value, bool Add);
event ProgressMaximumChangedEventHandler ProgressMaximumChanged;
event ProgressChangedEventHandler ProgressPreChanged;
event ProgressMaximumChangedEventHandler ProgressPreMaximumChanged;
IThrower Thrower {get; set;};
ILogProvider LogProvider {get; set;};
ISiteSettings Settings {get; set;};
@@ -298,6 +300,9 @@ interface IPluginContentProvider : IDisposable
Raise the `ProgressChanged` event when the file is downloaded.
Raise the `ProgressMaximumChanged` event before downloading files. The `Value` argument specifies the number of media files to download.
Raise the `ProgressPreChanged` event when parsing content.
Raise the `ProgressPreMaximumChanged` event when the amount of parse content has changed. The `Value` argument specifies the amount of content to be parsed.
See the [Thrower](#IThrower) , [LogProvider](#ILogProvider) and [Settings](#ISiteSettings) property environments in their respective chapters. **Don't set these properties manually.** They will be set by SCrawler.
`Name` is the username returned from the settings class when a new user is created.

116
Settings.md

@@ -7,7 +7,7 @@
- `Maximum download tasks of users` - the number of simultaneously downloading profiles.
- `Maximum download tasks of channels` - the number of simultaneously downloading channels.
- `Check new version at start`
- `UserAgent` - default UserAgent to use in requests ([how to find](#how%20to%20find%20useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect.
- `UserAgent` - default UserAgent to use in requests ([how to find](#how-to-find-useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect.
- `Imgur Client ID` - Imgur client ID to Bypass NSFW protection and download galleries
- `Userlist image` - customize the image that will be displayed at the background of the main window
- `Userlist color` - set the background color and font color of the main window (`F` button to select font color; `C` button to select background color)
@@ -33,8 +33,8 @@
- `Open the Info form when the download start`
- `Open the Progress form when the download start`
- `Don't open again` - Do not automatically open the corresponding form if it was once closed
- `Folder cmd` - the [command](#folder%20command) to open a folder
- `Close cmd` - this [command](#scrawler%20script%20text%20examples) will be executed when SCrawler is closed
- `Folder cmd` - the [command](#folder-command) to open a folder
- `Close cmd` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed
![Behavior settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBehavior.png)
@@ -67,7 +67,7 @@
- `Create video thumbnail` - create video thumbnail for downloaded video.
- `Update the YouTube output path when you change the output path` - update YouTube output path every time you change file destination.
- `Load downloaded YouTube videos to the form` - If checked, downloaded YouTube videos will be loaded to the form. Otherwise, all downloaded data will be loaded to the form except YouTube data.
- `Clear YouTube videos when clearing the list` - If checked, YouTube videos will also be removed from the list. This action will also affect the standalone [`YouTubeDownloader`](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader) app.
- `Clear YouTube videos when clearing the list` - If checked, YouTube videos will also be removed from the list. This action will also affect the standalone [`YouTubeDownloader`](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader) app.
![Downloader settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalDownloader.png)
@@ -82,9 +82,9 @@
- `Date` - append date to file name
- `Time` - append time to file name
- Date positions `Start/End` - date and/or time will be appended to the end or beginning of the file name
- `Script` - [script](#how%20to%20use%20the%20script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the `Use script` option.
- `After download cmd` - this [command](#scrawler%20script%20text%20examples) will be executed after all downloads are completed
- `Add missing information to log` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing%20posts) exist.
- `Script` - [script](#how-to-use-the-script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the `Use script` option.
- `After download cmd` - this [command](#scrawler-script-text-examples) will be executed after all downloads are completed
- `Add missing information to log` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing-posts) exist.
- `Add missing errors to log` - Each error that prevents SCrawler from downloading a file will be added to the log.
- `Trying to download missing posts using regular download` - If missing posts exist, the missing posts will attempt to be downloaded via user download.
@@ -115,24 +115,24 @@ If you see something like `Jobs <number>` in the progress bar, it means that SCr
## Sites requirements
- [Reddit](#reddit%20requirements)
- [Twitter](#twitter%20requirements)
- [Mastodon](#Mastodon%20requirements)
- [Instagram](#instagram%20requirements)
- [RedGifs](#reddit%20requirements)
- [YouTube](#YouTube%20requirements)
- [Pinterest](#Pinterest%20requirements)
- [TikTok](#tiktok%20requirements)
- [PornHub](#pornhub%20requirements)
- [XHamster](#xhamster%20requirements)
- [XVIDEOS](#xvideos%20requirements)
- [ThisVid](#ThisVid%20requirements)
- [LPSG](#lpsg%20requirements)
- [Reddit](#reddit-requirements)
- [Twitter](#twitter-requirements)
- [Mastodon](#Mastodon-requirements)
- [Instagram](#instagram-requirements)
- [RedGifs](#reddit-requirements)
- [YouTube](#YouTube-requirements)
- [Pinterest](#Pinterest-requirements)
- [TikTok](#tiktok-requirements)
- [PornHub](#pornhub-requirements)
- [XHamster](#xhamster-requirements)
- [XVIDEOS](#xvideos-requirements)
- [ThisVid](#ThisVid-requirements)
- [LPSG](#lpsg-requirements)
## Site default fields
- `Path` - where users' data will be placed.
- `Saved posts path` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. *Saved Twitter posts are posts that you have bookmarked.*
- `Cookies` add [cookies](#how%20to%20set%20up%20cookies) from your browser (click the pencil button to open the cookies editor).
- `Cookies` add [cookies](#how-to-set-up-cookies) from your browser (click the pencil button to open the cookies editor).
- `Download site data` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- `Get user media only` - the default for creating new users
- `Temporary`, `Download images/videos` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the `Defaults` tab. If the checkbox is in an intermediate state, then the same name setting of the `Defaults` tab will be used instead. Otherwise, this setting will be applied.
@@ -167,10 +167,10 @@ The Reddit parser can parse data without cookies, but you can add it if you like
![Reddit settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteReddit.png)
### Reddit requirements
- [Cookies](#how%20to%20set%20up%20cookies) for downloading saved posts
- [Cookies](#how-to-set-up-cookies) for downloading saved posts
- [ffmpeg](#ffmpeg) for downloading videos hosted on reddit
- [Imgur client ID](#how%20to%20find%20imgur%20client%20id) to download Imgur content posted on reddit
- [RedGifs credentials](#redgifs%20requirements) to download RedGifs content posted on reddit
- [Imgur client ID](#how-to-find-imgur-client-id) to download Imgur content posted on reddit
- [RedGifs credentials](#redgifs-requirements) to download RedGifs content posted on reddit
### Reddit user settings
- `View` - works the same as Reddit view modes (`new`, `hot`, `top`)
@@ -179,8 +179,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
![Reddit user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsRedditView.png)
## Twitter
- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`.
- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`.
- `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
- `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
@@ -189,9 +189,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
![Twitter settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTwitter.png)
### Twitter requirements
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [`Authorization`](#how%20to%20find%20twitter%20tokens)
- [`Token`](#how%20to%20find%20twitter%20tokens)
- [`Cookies`](#how-to-set-up-cookies)
- [`Authorization`](#how-to-find-twitter-tokens)
- [`Token`](#how-to-find-twitter-tokens)
- [`yt-dlp`](#YT-DLP) for download saved posts (bookmarked)
### Twitter user settings
@@ -217,8 +217,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
## Mastodon
- `My Domain` - your account domain without `https://` (for example, `mastodon.social`)
- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`.
- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`.
- `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
- `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
@@ -229,9 +229,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
### Mastodon requirements
- `My Domain`
- [`Authorization`](#how%20to%20find%20twitter%20tokens)
- [`Token`](#how%20to%20find%20twitter%20tokens)
- [`Authorization`](#how%20to%20find%20twitter%20tokens) and [`Token`](#how%20to%20find%20twitter%20tokens) for each domain you want to download from (see [additional settings](#Mastodon%20additional%20setting))
- [`Authorization`](#how-to-find-twitter-tokens)
- [`Token`](#how-to-find-twitter-tokens)
- [`Authorization`](#how-to-find-twitter-tokens) and [`Token`](#how-to-find-twitter-tokens) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting))
### Mastodon additional setting
- `Domains` - select or add a new domain for which you want to add authorization
@@ -242,9 +242,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
## Instagram
- Authorization
- `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how%20to%20find%20instagram%20hash)).
- `x-csrftoken` - [how to find](#how%20to%20find%20instagram%20authorization%20headers)
- `x-ig-app-id`, `ix-ig-www-claim` - [how to find](#how%20to%20find%20instagram%20authorization%20headers)
- `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)).
- `x-csrftoken` - [how to find](#how-to-find-instagram-authorization-headers)
- `x-ig-app-id`, `ix-ig-www-claim` - [how to find](#how-to-find-instagram-authorization-headers)
- `Saved posts user` - your personal Instagram username to download your saved posts
- Other parameters
- `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
@@ -256,18 +256,18 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- `Get timeline` - default value for new users
- `Get stories` - default value for new users
- `Get tagged photos` - default value for new users
- `Tagged notify limit` - Limit of new tagged posts when you receive a notification (read more [here](#instagram%20tagged%20posts%20limit))
- `Tagged notify limit` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit))
![Instagram settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteInstagram.png)
### Instagram requirements
- **[COOKIES](#how%20to%20set%20up%20cookies) ARE REQUIRED ANYWAY**
- [`Hash`](#how%20to%20find%20instagram%20hash) *(only for downloading tagged posts)*
- [`x-csrftoken`](#how%20to%20find%20instagram%20authorization%20headers)
- [`x-ig-app-id`](#how%20to%20find%20instagram%20authorization%20headers)
- [`ix-ig-www-claim`](#how%20to%20find%20instagram%20authorization%20headers)
- **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
- [`Hash`](#how-to-find-instagram-hash) *(only for downloading tagged posts)*
- [`x-csrftoken`](#how-to-find-instagram-authorization-headers)
- [`x-ig-app-id`](#how-to-find-instagram-authorization-headers)
- [`ix-ig-www-claim`](#how-to-find-instagram-authorization-headers)
To download tagged posts you also need [`Hash`](#how%20to%20find%20instagram%20hash)
To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash)
### How to find Instagram Hash
1. Open browser developer tools
@@ -373,13 +373,13 @@ result_cancel-->[*]
## RedGifs
- `Token refresh interval` - Interval (in minutes) to refresh the token
- `Token` - "Bearer" token ([how to find](#how%20to%20find%20redgifs%20token))
- `UserAgent` - UserAgent to use in requests ([how to find](#how%20to%20find%20useragent))
- `Token` - "Bearer" token ([how to find](#how-to-find-redgifs-token))
- `UserAgent` - UserAgent to use in requests ([how to find](#how-to-find-useragent))
![RedGifs settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteRedGifs.png)
### RedGifs requirements
- [`Token`](#How%20to%20find%20RedGifs%20token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.*
- [`Token`](#How-to-find-RedGifs-token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.*
### How to find RedGifs token
1. Open browser developer tools
@@ -395,12 +395,12 @@ result_cancel-->[*]
- `Download user playlists`
- `Use cookies` - default value for new users. Use cookies to download the user.
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader#Settings).
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings).
![YouTube settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteYouTube.png)
### YouTube requirements
- [`Cookies`](#how%20to%20set%20up%20cookies) for private data
- [`Cookies`](#how-to-set-up-cookies) for private data
- [**yt-dlp**](#YT-DLP)
- [**ffmpeg**](#ffmpeg)
@@ -411,7 +411,7 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You
![Pinterest settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSitePinterest.png)
### Pinterest requirements
- [`Cookies`](#how%20to%20set%20up%20cookies) for private data
- [`Cookies`](#how-to-set-up-cookies) for private data
- [**gallery-dl**](#Gallery-dl)
## TikTok
@@ -419,7 +419,7 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You
![TikTok settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTikTok.png)
### TikTok requirements
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [`Cookies`](#how-to-set-up-cookies)
### TikTok limits
@@ -435,7 +435,7 @@ Unfortunately, I didn't understand how to use the 'Cursor' option to get the JSO
### PornHub requirements
- [ffmpeg](#ffmpeg) is required anyway
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts
- [`Cookies`](#how-to-set-up-cookies) for downloading private videos and saved posts
- `Saved posts user` for downloading saved posts
### PornHub additional information
@@ -451,7 +451,7 @@ About photos. Photo download problems have the lowest priority. You can still cr
### XHamster requirements
- [ffmpeg](#ffmpeg) is required anyway
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading saved posts
- [`Cookies`](#how-to-set-up-cookies) for downloading saved posts
## XVIDEOS
- `Download UHD` - Download UHD (4K) content.
@@ -461,7 +461,7 @@ About photos. Photo download problems have the lowest priority. You can still cr
### XVIDEOS requirements
- [ffmpeg](#ffmpeg) is required anyway
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts
- [`Cookies`](#how-to-set-up-cookies) for downloading private videos and saved posts
- `Playlist of saved videos` for downloading saved posts
## ThisVid
@@ -472,7 +472,7 @@ About photos. Photo download problems have the lowest priority. You can still cr
![ThisVid settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteThisVid.png)
### ThisVid requirements
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [`Cookies`](#how-to-set-up-cookies)
- [**yt-dlp**](#YT-DLP)
## LPSG
@@ -480,7 +480,7 @@ About photos. Photo download problems have the lowest priority. You can still cr
![LPSG settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteLPSG.png)
### LPSG requirements
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [`Cookies`](#how-to-set-up-cookies)
# Download groups
@@ -532,8 +532,8 @@ You can set up automatic downloads. You can find these settings in `Settings-Aut
- `Disable` - Disable automatic downloads.
- `All` - Download all existing user profiles (`Ready for download` is not respected).
- `Default` - Download all existing user profiles marked `Ready for download` (equivalent to `Download all`).
- `Specified` - Specify custom download options (read more [here](#download%20groups)).
- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download%20groups)).
- `Specified` - Specify custom download options (read more [here](#download-groups)).
- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download-groups)).
- `Show notifications` - Show notification when there is new downloaded data.
- `Simple` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored.
- `Image` - Show downloaded image in notification.