geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2024.4.10.0

Andy
2024-04-10 06:27:31 +03:00
parent 5a53d76540
commit 472952f64f
4 changed files with 82 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
Home.md

@@ -142,6 +142,8 @@ Load sessions:
- `Clear session` - clear list of recent downloads
- `Merge special feeds` - merge multiple special feeds into one
- `Select all/none`
- `Save current view`
- `Load view (from saved)` - load one of your previously saved views
Change view:
- `Downloads` - show downloaded regular users
@@ -160,14 +162,13 @@ Change view:
Hot-keys:
- `F5` - refresh
- `Ctrl+G` - go to a specific page
- `Home` - go to first page
- `End` - go to last page
- `F3`, `Up`, `Page Up` - go to previous page
- `F4`, `Down`, `Page Down` - go to next page
![Feed session context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedWindowSessionContext.png)
Hot keys
- `F5` - `Refresh`
- `F3` - Previous page
- `F4` - Next page
You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
### Channels
@@ -194,11 +195,8 @@ You may see a message like `Host [Site] is not ready` or `Host [Site] is unavail
### Download
- `Download selected (F5)` - start downloading one or more selected users.
- `Download all`: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!**
- `Download all (F6)` - Download all users marked `Ready for download` from all sites.
- `Download all site users` - Download all users marked `Ready for download` from specific sites.
- `Download all FULL` - Download all users from all sites. **The `Ready for download` option will be ignored.**
- `Download all site users FULL` - Download all users from specific sites. **The `Ready for download` option will be ignored.**
- `Download all (F6)` - download all users marked `Ready for download` from all sites.
- `Download all [FULL]` - download all users from all sites. **The `Ready for download` option will be ignored.**
- `Download (advanced)` - Filter the users you want to download and download them.
- `Standalone downloader` - Just a [downloader](#download-separate-video) for single videos.
- `Add a new download group` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)).
@@ -234,28 +232,15 @@ Users view modes.
- The first block specifies **how** users will be displayed in the form.
- `Large/Small images` - with icons.
- `List` / `Details` - simple, without icons.
- The second block specifies what **types of user** will be displayed in the form.
- `Show users` - show regular users in user list.
- `Show subscriptions` - show [subscriptions](#subscriptions) in user list.
- The third block specifies the user **sites** that will be displayed in the form.
- `All` - default
- `Specific sites` - select sites to be displayed
- The fourth block specifies **which** users will be displayed in the form.
- `All` - default.
- `Regular` - only show users who are not marked as `Favorite` or `Temporary`.
- `Temporary` - only show users marked as temporary.
- `Favorite` - only show users marked as favorite.
- `Deleted` - only show users whose profiles have been deleted
- `Suspended` - only show users whose profiles have been suspended
- `Labels` - show users labeled with one or more selected label
- `No Labels` - only show users without a label
- `Excluded labels` - exclude from displaying users with the selected labels (this is suitable for situations where you want to hide a user but don't want to delete)
- `Ignore excluded labels` - Ignore excluding labels. This allows you to show users that should be excluded from display without resetting the current selection of excluded labels.
- The second block specifies **which** users will be displayed in the form.
- `All` - sgow all users
- `Show groups instead of labels` - show sites groups instead of labels (when filtering users by labels)
- `Advanced filter` - advanced filter of users you want to display
- The fifth block specifies the **date range** of the last download, whose users will be displayed in the form.
- `Limit dates (not in range)` - filter users whose last download date is not in the selected date range
- `Limit dates (in range)` - filter users whose last download date is in the selected date range
- `Group users` - group users by site or labels
- `Filter` - filter the users you want to display
- View third (read [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups) about filter settings)
- `Save current view`
- `Save current view as download group`
- `Load view (from saved)` - load one of your previously saved views (filters, download groups, scheduler plans)
**You can also change the view by (un)grouping by site using the `Show groups` and `Use user grouping` [options in the settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#basis).**

14
Plugins.md

@@ -154,6 +154,9 @@ interface ISiteSettings: IDisposable
Icon Icon {get;};
Image Image {get;};
string Site {get;};
string CMDEncoding {get; set;};
IEnumerable<String> EnvironmentPrograms {get; set;};
void EnvironmentProgramsUpdated()
string AccountName {get; set;};
bool Temporary {get; set;};
ISiteSettings DefaultInstance {get; set;};
@@ -163,7 +166,6 @@ interface ISiteSettings: IDisposable
ExchangeOptions IsMyUser(string UserURL);
ExchangeOptions IsMyImageVideo(string URL);
IPluginContentProvider GetInstance(Download What);
IEnumerable<KeyValuePair<string, string>> XMLFields();
void BeginInit();
void EndInit();
string AvailableText {get; set;};
@@ -190,6 +192,12 @@ interface ISiteSettings: IDisposable
`Site`, `Icon` and `Image` are properties that provide the site name and site icon (in `Icon` and `Image` formats).
`CMDEncoding` - user-selected the command-line encoding.
`EnvironmentPrograms` - user-selected paths to programs such as yt-dlp, gallery-dl, ffmpeg, curl, etc.
`EnvironmentProgramsUpdated` - this function will be called when `EnvironmentPrograms` or `CMDEncoding` changes
`AccountName` property set before calling the `BeginInit` function.
`Temporary` property means that this instance is temporary. Use when cloning an existing instance or create a new one (before saving data).
@@ -401,6 +409,8 @@ There are three initialization constructors:
- By `value` and `type`.
- By `value`, `type` and `function`. If you want, you can delegate a function to be called when the value changes. You can see an example in the Instagram `SiteSettings` class
The `Checked` parameter is intended to define the inheritance value and other parameters that you can use with the `OnCheckboxCheckedChange` function.
If your initial value is null, you **MUST** set the type.
Use `BeginInit` and `EndInit` to pause handlers. Use `Clone` to properly clone a property using the `ISiteSettings` `Clone` and `Update` functions.
@@ -452,6 +462,8 @@ Options:
- `AllowNull` - If `false`, then the settings form will display an error message when trying to save the property value if the property value is `null`. Default: `true`.
- `LeftOffset` - Just a design option. This is just a control offset from the left border of the form (just for beauty).
- `IsAuth` - Default: `false`. If at least one property has a `PropertyOption` with this parameter, then the controls in the settings form will be divided into two blocks: `Authorization` and `Other`. Just a design option.
- `InheritanceName` - the name of a constant from global settings for value inheritance and automatic updating.
- `Category` - category name.
- `IsInformationLabel` - Just a design option. Specifies that this property is just information.
- `LabelTextAlign` - The alignment of the control (label) text, if `IsInformationLabel`.

68
Settings.md

@@ -9,8 +9,6 @@
- `Check new version at start`
- `UserAgent` - default UserAgent to use in requests ([how to find](#how-to-find-useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect.
- `Imgur Client ID` - Imgur client ID to Bypass NSFW protection and download galleries
- `Show groups` - show user site groups
- `Use user grouping` - Group users by groups and/or labels. If disabled, users will not be grouped by site or label.
![Basis settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBasis.png)
@@ -33,6 +31,12 @@
![Environment settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalEnvironment.png)
## Headers
**Here you can configure default browser headers, which can be used in the site settings. In the site settings, where available, you will see a checkbox to the left of the header text field. If checked, the value will be inherited from the global settings default values.**
![Default headers settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalHeaders.png)
## Behavior
- `Exit confirm` - ask for confirmation before closing the program
- `Close to tray` - close program to system tray
@@ -43,6 +47,9 @@
- `Don't open again` - Do not automatically open the corresponding form if it was once closed
- `Folder cmd` - the [command](#folder-command) to open a folder
- `Close cmd` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed
- `Use 'F6' to download all users`
- `'F6' confirmation` - request confirmation to download all users when pressing `F6`
- `Confirmation of downloading all` - request confirmation to download all users anyway
![Behavior settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBehavior.png)
@@ -112,6 +119,7 @@
- `Download limit for channel user` - the amount of media will be downloaded if the user added from the channel.
- `Copy channel user image` - the image posted by user in the channel will be copied to the user folder when user will be created.
- `Create temporary users` - users will be created marked 'Temporary' when created from channel.
- `Set the user's mark 'Ready for download' when add from channels`
![Channels settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalChannels.png)
@@ -192,10 +200,10 @@ Version **1.26.4** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1
### YT-DLP
Version **2023.11.16** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2023.11.16))
Version **2024.04.09** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2024.04.09))
- x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.11.16/yt-dlp.exe)
- x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.11.16/yt-dlp_x86.exe)
- x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.04.09/yt-dlp.exe)
- x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.04.09/yt-dlp_x86.exe)
**The following sites use yt-dlp:**
- [YouTube](#youtube)
@@ -245,6 +253,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- `Saved posts user` - your personal Reddit username to download your saved posts (this feature requires cookies)
- Other parameters
- `Use M3U8` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
- `Check image` - check the image if it exists before downloading (**it makes downloading very slow**)
- `Check image: get original` - get the original image if it exists
![Reddit settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteReddit.png)
@@ -309,6 +319,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
**[Partial support](#paid-sites-partial-support)**
- `Download timeline` - download user timeline
- `Download stories` - download profile stories if they exists
- `Download highlights` - download profile highlights if they exists
- `Download chat` - download unlocked chat media
- `user-id`, `x-bc`, `app-token`, `sec-sc-ua`, `UserAgent` - required headers ([how to find](#how-to-find-headers) (the request you need must start with `posts?limit=.....`))
@@ -382,13 +394,13 @@ To support the downloading of this site you should add the Mastodon domain to th
- Authorization
- `x-csrftoken` - [how to find](#how-to-find-headers) *(can be automatically extracted from cookies)*
- `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
- `Saved posts user` - your personal Instagram username to download your saved posts
- Other parameters
- `Use GraphQL to download` - Use GraphQL to download data instead of the current query algorithm.
- `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download reels` - Download reels *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download stories` - Download stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download stories: user` - Download active (non-pinned) user stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download tagged` - Download tagged posts *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Request timer (any)` - the timer (in milliseconds) that SCrawler should wait before executing the next request. The default value is 1'000. **It is highly recommended not to change the default value.**
- `Request timer` - this is the time value (in milliseconds) the program will wait before processing the next `Request time counter` request (**it is highly recommended not to change this default value**)
- `Request time counter` - how many requests will be sent to Instagram before the program waits `Request timer` milliseconds (**it is highly recommended not to change this default value**)
- `Post limit timer` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
@@ -499,6 +511,8 @@ result_cancel-->[*]
## Threads
- `x-csrftoken` - [how to find](#how-to-find-headers)
- `x-ig-app-id`, `x-asbd-id`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
- `Request timer (any)` - the timer (in milliseconds) that SCrawler should wait before executing the next request. The default value is 1'000. **It is highly recommended not to change the default value.**
- `Download data` - The internal value indicates that site data should be downloaded. It becomes unchecked when the site returns an error. When you update your credentials, you need to checked this check box.
![Threads settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteThreads.png)
@@ -508,6 +522,8 @@ result_cancel-->[*]
## Facebook
- `x-ig-app-id`, `x-asbd-id`, `Accept`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
- `Request timer (any)` - the timer (in milliseconds) that SCrawler should wait before executing the next request. The default value is 1'000. **It is highly recommended not to change the default value.**
- `Download data` - The internal value indicates that site data should be downloaded. It becomes unchecked when the site returns an error. When you update your credentials, you need to checked this check box.
The `Accept` and `UserAgent` headers must be obtained from the page `https://facebook.com/<USERNAME>`.
@@ -598,11 +614,13 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You
## TikTok
- `Remove tags from title` - if the title contains tags, they will be removed (only works with `Use native title`)
- `Use native title` - use a user-created video title for the filename instead of the video ID
- `Use native title in standalone downloader` - use a user-created video title for the filename instead of the video ID
- `Use native title (standalone downloader)` - use a user-created video title for the filename instead of the video ID
- `Add video ID to video title` - the video ID will be added to the file name
- `Add video ID to video title (standalone downloader)` - the video ID will be added to the file name
- `Use regex to clean video title`
- `Title regex` - regex to clean video title (only works with checked `Use regex to clean video title`)
- `Use video date as file date` - set the file date to the date the video was added (website) (if available)
- `Use video date as file date (standalone downloader)` - set the file date to the date the video was added (website) (if available)
![TikTok settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTikTok.png)
@@ -695,18 +713,35 @@ About photos. Photo download problems have the lowest priority. You can still cr
# Download groups
**Press `Alt+F1` to display users** *(works in scheduler, scheduler plans, download groups)*. Click on a user and press `Ctrl+F` to find that user in the main window.
In many cases, you may need to download some users. You can group these users and make it easier to download them without having to select them every time.
It is very easy to create a new group. Just click the `Add a new download group` button (main window - `Download all` menu) to create a new one.
It is very easy to create a new group. Just click the `Add a new download group` button (main window - `Download` menu) to create a new one.
- `Name` - group name
- `Temporary` - users marked as temporary (indeterminate state to not use this option) will be downloaded
- `Favorite` - users marked as favorite (indeterminate state to not use this option) will be downloaded
- `Ready for download` - users marked as `Ready for download` will be downloaded
- `Regular`- users not marked as temporary or favorite
- `Temporary` - users marked as temporary
- `Favorite` - users marked as favorite
- `User exists` - include users not marked as 'Suspended' or 'Deleted'
- `User suspended` - include users marked as 'Suspended' *(read [here](https://github.com/AAndyProgram/SCrawler/wiki#user-colors) about user internal marks)*
- `User deleted` - include users marked as 'Deleted' *(read [here](https://github.com/AAndyProgram/SCrawler/wiki#user-colors) about user internal marks)*
- `Ready for download` - users marked as `Ready for download`
- `Ignore ready for download` - this option tells the program to ignore the `Ready for download` user option and download the user anyway
- `Down` - filter users who have been (not)downloaded in the last `x` days. `-1` or `0` to disable. `Checked` = downloaded in the last `x` days. `Unchecked` = NOT downloaded in the last `x` days"
- `Users`
- `Subscriptions` - include subscriptions in the download queue (read about subscriptions [here](https://github.com/AAndyProgram/SCrawler/wiki#subscriptions)).
- `Subscriptions only` - include **only** subscriptions in the download queue (only works with `Subscriptions`).
- `In range` - Last download date range.
- If checked, filter users whose last download date is within the selected date range.
- If unchecked, filter users whose last download date is outside the selected date range.
- If no dates are checked, this option will be ignored.
- Dates:
- If only one date is checked, then that date is the limit.
- Example:
- You have the 'In range' checkbox and the 'From' date checked. This way, only those users whose last download date is greater than the 'From' date will be downloaded.
- `Users` - the number of users that to be downloaded. The number is 0 = all users. Number greater than 0 = number of users from the beginning to the end of the list. Number less than 0 = number of users from end to the beginning of the list.
- No labels
- Ignore excluded labels
- `Labels` - You can select labels.
- `Pencil` button for selected labels. Only users who have one or more of these labels will be downloaded.
- `Red X` button for excluded labels. Users who have one or more of these labels will be excluded from the download.
@@ -723,8 +758,9 @@ You can set both selected and excluded options for each selection.
For each group, SCrawler creates a new menu, which is placed in the `Download all` menu of the main window. If a group has a number on the left (1-9), that group can be downloaded using `Ctrl`+`Number`. Each group also has several options:
- `Edit` - edit group
- `Delete` - delete group
- `Download` - download with the options you have set
- `Download FULL` - Download with the options you have set. `Ready for download` option will be ignored.
- `Clone and add` - clone the group, change parameters and add this group as a new one
- `Clone and download` - clone the group, change parameters and download filtered users (this group will not be added as a new one)
- `Download` - download with the options you have set (`Ctrl+Click` to download, exclude from feed)
![Groups context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowGroups.png)
@@ -749,8 +785,6 @@ You can set up automatic downloads. You can find these settings in `Settings-Aut
**Plan**:
- `Disable` - Disable automatic downloads.
- `All` - Download all existing user profiles (`Ready for download` is not respected).
- `Default` - Download all existing user profiles marked `Ready for download` (equivalent to `Download all`).
- `Specified` - Specify custom download options (read more [here](#download-groups)).
- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download-groups)).
- `Show notifications` - Show notification when there is new downloaded data.

3
YouTube downloader.md

@@ -31,7 +31,8 @@ You can change the setting of the video you want to download.
5. You can also change the subtitles output format in the `Format` combo box.
6. You can also select additional subtitles/audio formats in the corresponding fields. This means that all subtitles/audio will be extracted from the video, converted to the format you choose, and saved as a separate file.
7. You can also choose the destination of the output file.
8. Click `Download` to confirm your choices, or `Cancel` to cancel the download.
8. You can also add downloaded item(s) to a playlist, change video FPS, audio bitrate.
9. Click `Download` to confirm your choices, or `Cancel` to cancel the download.
*Right-click the `Browse` button to select a new location and add the new location to the list.*