From d25e2b1319f01a56cdc70a93255a7d2ba22ed3c6 Mon Sep 17 00:00:00 2001 From: Andy <88590076+AAndyProgram@users.noreply.github.com> Date: Tue, 9 May 2023 05:12:44 +0300 Subject: [PATCH] Fix links --- Home.md | 6 +-- Plugins.md | 5 +++ Settings.md | 116 ++++++++++++++++++++++++++-------------------------- 3 files changed, 66 insertions(+), 61 deletions(-) diff --git a/Home.md b/Home.md index 8ac6681..acc37d3 100644 --- a/Home.md +++ b/Home.md @@ -200,7 +200,7 @@ Double-click the user to open the content folder. ## Download separate video -Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader) +Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader) ![Separate video downloader window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTube.png) @@ -298,7 +298,7 @@ You can pick up one or more labels in this form. ### Deleting a user -SCrawler has advanced deletion options. It can delete a user/[collection](#Deleting%20a%20collection) with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options. +SCrawler has advanced deletion options. It can delete a user/[collection](#Deleting-a-collection) with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options. ```mermaid graph TB @@ -344,7 +344,7 @@ When you add a user to the collection, a new folder will be created in the colle - `Split` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted. - `Cancel` - Cancel operation. -Go to a [diagram](#Deleting%20a%20user) that illustrates all the available options. +Go to a [diagram](#Deleting-a-user) that illustrates all the available options. ### Collections example diff --git a/Plugins.md b/Plugins.md index 3afd913..7e23f83 100644 --- a/Plugins.md +++ b/Plugins.md @@ -268,6 +268,8 @@ interface IPluginContentProvider : IDisposable event ProgressChangedEventHandler ProgressChanged; delegate void ProgressMaximumChangedEventHandler(int Value, bool Add); event ProgressMaximumChangedEventHandler ProgressMaximumChanged; + event ProgressChangedEventHandler ProgressPreChanged; + event ProgressMaximumChangedEventHandler ProgressPreMaximumChanged; IThrower Thrower {get; set;}; ILogProvider LogProvider {get; set;}; ISiteSettings Settings {get; set;}; @@ -298,6 +300,9 @@ interface IPluginContentProvider : IDisposable Raise the `ProgressChanged` event when the file is downloaded. Raise the `ProgressMaximumChanged` event before downloading files. The `Value` argument specifies the number of media files to download. +Raise the `ProgressPreChanged` event when parsing content. +Raise the `ProgressPreMaximumChanged` event when the amount of parse content has changed. The `Value` argument specifies the amount of content to be parsed. + See the [Thrower](#IThrower) , [LogProvider](#ILogProvider) and [Settings](#ISiteSettings) property environments in their respective chapters. **Don't set these properties manually.** They will be set by SCrawler. `Name` is the username returned from the settings class when a new user is created. diff --git a/Settings.md b/Settings.md index 40342dc..5ca6b58 100644 --- a/Settings.md +++ b/Settings.md @@ -7,7 +7,7 @@ - `Maximum download tasks of users` - the number of simultaneously downloading profiles. - `Maximum download tasks of channels` - the number of simultaneously downloading channels. - `Check new version at start` -- `UserAgent` - default UserAgent to use in requests ([how to find](#how%20to%20find%20useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect. +- `UserAgent` - default UserAgent to use in requests ([how to find](#how-to-find-useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect. - `Imgur Client ID` - Imgur client ID to Bypass NSFW protection and download galleries - `Userlist image` - customize the image that will be displayed at the background of the main window - `Userlist color` - set the background color and font color of the main window (`F` button to select font color; `C` button to select background color) @@ -33,8 +33,8 @@ - `Open the Info form when the download start` - `Open the Progress form when the download start` - `Don't open again` - Do not automatically open the corresponding form if it was once closed -- `Folder cmd` - the [command](#folder%20command) to open a folder -- `Close cmd` - this [command](#scrawler%20script%20text%20examples) will be executed when SCrawler is closed +- `Folder cmd` - the [command](#folder-command) to open a folder +- `Close cmd` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed ![Behavior settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBehavior.png) @@ -67,7 +67,7 @@ - `Create video thumbnail` - create video thumbnail for downloaded video. - `Update the YouTube output path when you change the output path` - update YouTube output path every time you change file destination. - `Load downloaded YouTube videos to the form` - If checked, downloaded YouTube videos will be loaded to the form. Otherwise, all downloaded data will be loaded to the form except YouTube data. -- `Clear YouTube videos when clearing the list` - If checked, YouTube videos will also be removed from the list. This action will also affect the standalone [`YouTubeDownloader`](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader) app. +- `Clear YouTube videos when clearing the list` - If checked, YouTube videos will also be removed from the list. This action will also affect the standalone [`YouTubeDownloader`](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader) app. ![Downloader settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalDownloader.png) @@ -82,9 +82,9 @@ - `Date` - append date to file name - `Time` - append time to file name - Date positions `Start/End` - date and/or time will be appended to the end or beginning of the file name -- `Script` - [script](#how%20to%20use%20the%20script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the `Use script` option. -- `After download cmd` - this [command](#scrawler%20script%20text%20examples) will be executed after all downloads are completed -- `Add missing information to log` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing%20posts) exist. +- `Script` - [script](#how-to-use-the-script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the `Use script` option. +- `After download cmd` - this [command](#scrawler-script-text-examples) will be executed after all downloads are completed +- `Add missing information to log` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing-posts) exist. - `Add missing errors to log` - Each error that prevents SCrawler from downloading a file will be added to the log. - `Trying to download missing posts using regular download` - If missing posts exist, the missing posts will attempt to be downloaded via user download. @@ -115,24 +115,24 @@ If you see something like `Jobs ` in the progress bar, it means that SCr ## Sites requirements -- [Reddit](#reddit%20requirements) -- [Twitter](#twitter%20requirements) -- [Mastodon](#Mastodon%20requirements) -- [Instagram](#instagram%20requirements) -- [RedGifs](#reddit%20requirements) -- [YouTube](#YouTube%20requirements) -- [Pinterest](#Pinterest%20requirements) -- [TikTok](#tiktok%20requirements) -- [PornHub](#pornhub%20requirements) -- [XHamster](#xhamster%20requirements) -- [XVIDEOS](#xvideos%20requirements) -- [ThisVid](#ThisVid%20requirements) -- [LPSG](#lpsg%20requirements) +- [Reddit](#reddit-requirements) +- [Twitter](#twitter-requirements) +- [Mastodon](#Mastodon-requirements) +- [Instagram](#instagram-requirements) +- [RedGifs](#reddit-requirements) +- [YouTube](#YouTube-requirements) +- [Pinterest](#Pinterest-requirements) +- [TikTok](#tiktok-requirements) +- [PornHub](#pornhub-requirements) +- [XHamster](#xhamster-requirements) +- [XVIDEOS](#xvideos-requirements) +- [ThisVid](#ThisVid-requirements) +- [LPSG](#lpsg-requirements) ## Site default fields - `Path` - where users' data will be placed. - `Saved posts path` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. *Saved Twitter posts are posts that you have bookmarked.* -- `Cookies` add [cookies](#how%20to%20set%20up%20cookies) from your browser (click the pencil button to open the cookies editor). +- `Cookies` add [cookies](#how-to-set-up-cookies) from your browser (click the pencil button to open the cookies editor). - `Download site data` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded. - `Get user media only` - the default for creating new users - `Temporary`, `Download images/videos` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the `Defaults` tab. If the checkbox is in an intermediate state, then the same name setting of the `Defaults` tab will be used instead. Otherwise, this setting will be applied. @@ -167,10 +167,10 @@ The Reddit parser can parse data without cookies, but you can add it if you like ![Reddit settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteReddit.png) ### Reddit requirements -- [Cookies](#how%20to%20set%20up%20cookies) for downloading saved posts +- [Cookies](#how-to-set-up-cookies) for downloading saved posts - [ffmpeg](#ffmpeg) for downloading videos hosted on reddit -- [Imgur client ID](#how%20to%20find%20imgur%20client%20id) to download Imgur content posted on reddit -- [RedGifs credentials](#redgifs%20requirements) to download RedGifs content posted on reddit +- [Imgur client ID](#how-to-find-imgur-client-id) to download Imgur content posted on reddit +- [RedGifs credentials](#redgifs-requirements) to download RedGifs content posted on reddit ### Reddit user settings - `View` - works the same as Reddit view modes (`new`, `hot`, `top`) @@ -179,8 +179,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like ![Reddit user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsRedditView.png) ## Twitter -- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space. -- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`. +- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space. +- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`. - `Download GIFs` - *(default for new users) this can also be configured for a specific user.* - `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.* - `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.* @@ -189,9 +189,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like ![Twitter settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTwitter.png) ### Twitter requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) -- [`Authorization`](#how%20to%20find%20twitter%20tokens) -- [`Token`](#how%20to%20find%20twitter%20tokens) +- [`Cookies`](#how-to-set-up-cookies) +- [`Authorization`](#how-to-find-twitter-tokens) +- [`Token`](#how-to-find-twitter-tokens) - [`yt-dlp`](#YT-DLP) for download saved posts (bookmarked) ### Twitter user settings @@ -217,8 +217,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like ## Mastodon - `My Domain` - your account domain without `https://` (for example, `mastodon.social`) -- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space. -- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`. +- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space. +- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`. - `Download GIFs` - *(default for new users) this can also be configured for a specific user.* - `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.* - `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.* @@ -229,9 +229,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like ### Mastodon requirements - `My Domain` -- [`Authorization`](#how%20to%20find%20twitter%20tokens) -- [`Token`](#how%20to%20find%20twitter%20tokens) -- [`Authorization`](#how%20to%20find%20twitter%20tokens) and [`Token`](#how%20to%20find%20twitter%20tokens) for each domain you want to download from (see [additional settings](#Mastodon%20additional%20setting)) +- [`Authorization`](#how-to-find-twitter-tokens) +- [`Token`](#how-to-find-twitter-tokens) +- [`Authorization`](#how-to-find-twitter-tokens) and [`Token`](#how-to-find-twitter-tokens) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting)) ### Mastodon additional setting - `Domains` - select or add a new domain for which you want to add authorization @@ -242,9 +242,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like ## Instagram - Authorization - - `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how%20to%20find%20instagram%20hash)). - - `x-csrftoken` - [how to find](#how%20to%20find%20instagram%20authorization%20headers) - - `x-ig-app-id`, `ix-ig-www-claim` - [how to find](#how%20to%20find%20instagram%20authorization%20headers) + - `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)). + - `x-csrftoken` - [how to find](#how-to-find-instagram-authorization-headers) + - `x-ig-app-id`, `ix-ig-www-claim` - [how to find](#how-to-find-instagram-authorization-headers) - `Saved posts user` - your personal Instagram username to download your saved posts - Other parameters - `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)* @@ -256,18 +256,18 @@ The Reddit parser can parse data without cookies, but you can add it if you like - `Get timeline` - default value for new users - `Get stories` - default value for new users - `Get tagged photos` - default value for new users -- `Tagged notify limit` - Limit of new tagged posts when you receive a notification (read more [here](#instagram%20tagged%20posts%20limit)) +- `Tagged notify limit` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit)) ![Instagram settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteInstagram.png) ### Instagram requirements -- **[COOKIES](#how%20to%20set%20up%20cookies) ARE REQUIRED ANYWAY** -- [`Hash`](#how%20to%20find%20instagram%20hash) *(only for downloading tagged posts)* -- [`x-csrftoken`](#how%20to%20find%20instagram%20authorization%20headers) -- [`x-ig-app-id`](#how%20to%20find%20instagram%20authorization%20headers) -- [`ix-ig-www-claim`](#how%20to%20find%20instagram%20authorization%20headers) +- **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY** +- [`Hash`](#how-to-find-instagram-hash) *(only for downloading tagged posts)* +- [`x-csrftoken`](#how-to-find-instagram-authorization-headers) +- [`x-ig-app-id`](#how-to-find-instagram-authorization-headers) +- [`ix-ig-www-claim`](#how-to-find-instagram-authorization-headers) -To download tagged posts you also need [`Hash`](#how%20to%20find%20instagram%20hash) +To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash) ### How to find Instagram Hash 1. Open browser developer tools @@ -373,13 +373,13 @@ result_cancel-->[*] ## RedGifs - `Token refresh interval` - Interval (in minutes) to refresh the token -- `Token` - "Bearer" token ([how to find](#how%20to%20find%20redgifs%20token)) -- `UserAgent` - UserAgent to use in requests ([how to find](#how%20to%20find%20useragent)) +- `Token` - "Bearer" token ([how to find](#how-to-find-redgifs-token)) +- `UserAgent` - UserAgent to use in requests ([how to find](#how-to-find-useragent)) ![RedGifs settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteRedGifs.png) ### RedGifs requirements -- [`Token`](#How%20to%20find%20RedGifs%20token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.* +- [`Token`](#How-to-find-RedGifs-token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.* ### How to find RedGifs token 1. Open browser developer tools @@ -395,12 +395,12 @@ result_cancel-->[*] - `Download user playlists` - `Use cookies` - default value for new users. Use cookies to download the user. -See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader#Settings). +See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings). ![YouTube settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteYouTube.png) ### YouTube requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) for private data +- [`Cookies`](#how-to-set-up-cookies) for private data - [**yt-dlp**](#YT-DLP) - [**ffmpeg**](#ffmpeg) @@ -411,7 +411,7 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You ![Pinterest settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSitePinterest.png) ### Pinterest requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) for private data +- [`Cookies`](#how-to-set-up-cookies) for private data - [**gallery-dl**](#Gallery-dl) ## TikTok @@ -419,7 +419,7 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You ![TikTok settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTikTok.png) ### TikTok requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) +- [`Cookies`](#how-to-set-up-cookies) ### TikTok limits @@ -435,7 +435,7 @@ Unfortunately, I didn't understand how to use the 'Cursor' option to get the JSO ### PornHub requirements - [ffmpeg](#ffmpeg) is required anyway -- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts +- [`Cookies`](#how-to-set-up-cookies) for downloading private videos and saved posts - `Saved posts user` for downloading saved posts ### PornHub additional information @@ -451,7 +451,7 @@ About photos. Photo download problems have the lowest priority. You can still cr ### XHamster requirements - [ffmpeg](#ffmpeg) is required anyway -- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading saved posts +- [`Cookies`](#how-to-set-up-cookies) for downloading saved posts ## XVIDEOS - `Download UHD` - Download UHD (4K) content. @@ -461,7 +461,7 @@ About photos. Photo download problems have the lowest priority. You can still cr ### XVIDEOS requirements - [ffmpeg](#ffmpeg) is required anyway -- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts +- [`Cookies`](#how-to-set-up-cookies) for downloading private videos and saved posts - `Playlist of saved videos` for downloading saved posts ## ThisVid @@ -472,7 +472,7 @@ About photos. Photo download problems have the lowest priority. You can still cr ![ThisVid settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteThisVid.png) ### ThisVid requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) +- [`Cookies`](#how-to-set-up-cookies) - [**yt-dlp**](#YT-DLP) ## LPSG @@ -480,7 +480,7 @@ About photos. Photo download problems have the lowest priority. You can still cr ![LPSG settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteLPSG.png) ### LPSG requirements -- [`Cookies`](#how%20to%20set%20up%20cookies) +- [`Cookies`](#how-to-set-up-cookies) # Download groups @@ -532,8 +532,8 @@ You can set up automatic downloads. You can find these settings in `Settings-Aut - `Disable` - Disable automatic downloads. - `All` - Download all existing user profiles (`Ready for download` is not respected). - `Default` - Download all existing user profiles marked `Ready for download` (equivalent to `Download all`). -- `Specified` - Specify custom download options (read more [here](#download%20groups)). -- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download%20groups)). +- `Specified` - Specify custom download options (read more [here](#download-groups)). +- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download-groups)). - `Show notifications` - Show notification when there is new downloaded data. - `Simple` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored. - `Image` - Show downloaded image in notification.