geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2023.4.28.0

Andy
2023-04-28 10:17:34 +03:00
parent 0f1a64d46f
commit 820822b1ac
6 changed files with 964 additions and 827 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
Channels.md

@@ -1,69 +1,69 @@
![Channels form](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/Channels.png)
![Channels form](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/Channels.png)
This form allows you to receive channels' posts. Each post includes a username, post ID and an image (or thumbnail for a video). You can download a certain number of posts, download all posts up to a specific date, or download posts up to a specific post ID.
The downloaded post data will be saved in the program cache folder (```Data path```). This cache will be deleted after the program is closed.
The downloaded post data will be saved in the program cache folder (`Data path`). This cache will be deleted after the program is closed.
You can download channels with users that you have in the program together.
Each user you selected in the channel form will be added to the temporary list and their number will be shown in the caption of the ```Add``` button. This is a temporary list. To add users from this list to the program, you need to click the ```Add``` button.
Each user you selected in the channel form will be added to the temporary list and their number will be shown in the caption of the `Add` button. This is a temporary list. To add users from this list to the program, you need to click the `Add` button.
To add a user to the temporary list, you just need to mark the post in the posts list. The user ID will be added to this list after you change the page (using the page change buttons) or forcibly click the ```Add``` button (in this case, users will be added to the program).
To add a user to the temporary list, you just need to mark the post in the posts list. The user ID will be added to this list after you change the page (using the page change buttons) or forcibly click the `Add` button (in this case, users will be added to the program).
## Buttons
- ```Download/Stop``` - start/stop download.
- ```Add``` - add selected users to the program (from the temporary list) for future download.
- ```Next/Back``` - change pages.
- ```All Channels``` - download all channels.
- ```Curved arrows``` - update channels list.
- ```+``` - add a new channel.
- ```X``` - delete channel.
- ```Edit``` - change channel options.
- `Download/Stop` - start/stop download.
- `Add` - add selected users to the program (from the temporary list) for future download.
- `Next/Back` - change pages.
- `All Channels` - download all channels.
- `Curved arrows` - update channels list.
- `+` - add a new channel.
- `X` - delete channel.
- `Edit` - change channel options.
![Channel options](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsRedditView.png)
- ```i``` - info button. The statistics of the selected channel will be shown.
![Channel options](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsRedditView.png)
- `i` - info button. The statistics of the selected channel will be shown.
![One channel info](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/ChannelsStats2.png)
- ```Hide exists users``` - users who are already in the program will be skipped.
- ```Info``` - The statistics of all channels will be shown. You can also delete channels in this form. The information will be shown according to the template: **[```ChannelName```]: Users: ```n1``` (avg: ```n2```; s: ```n3```); Posts: ```n4``` (s: ```n5```); My users: ```n6```**
- ```n1``` - total number of added users
- ```n2``` - average number of added users
- ```n3``` - number of added users in the current session
- ```n4``` - average number of downloaded posts
- ```n5``` - number of downloaded posts in the current session
- ```n6``` - users from this channel who are already in the program
![One channel info](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/ChannelsStats2.png)
- `Hide exists users` - users who are already in the program will be skipped.
- `Info` - The statistics of all channels will be shown. You can also delete channels in this form. The information will be shown according to the template: **[`ChannelName`]: Users: `n1` (avg: `n2`; s: `n3`); Posts: `n4` (s: `n5`); My users: `n6`**
- `n1` - total number of added users
- `n2` - average number of added users
- `n3` - number of added users in the current session
- `n4` - average number of downloaded posts
- `n5` - number of downloaded posts in the current session
- `n6` - users from this channel who are already in the program
![Channels stats](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/ChannelsStats.png)
![Channels stats](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/ChannelsStats.png)
## Limits
- ```Count``` - download a certain number of posts.
- ```Post``` - download posts up to a specific post ID.
- ```Date``` - download all posts up to a specific date.
- ```Default``` - the program will automatically determine the downloading mode:
- `Count` - download a certain number of posts.
- `Post` - download posts up to a specific post ID.
- `Date` - download all posts up to a specific date.
- `Default` - the program will automatically determine the downloading mode:
- if it is a new channel, the first 1000 posts will be downloaded;
- if the channel has already been downloaded in the past, the program will download each post until the date and until the post ID is found in the saved channel information.
## List context menu
- ```Open user/post``` - open user/post in default browser.
- ```Open picture``` - open the original cached image in your default image viewer.
- ```Open folder``` - open channel cache folder.
- ```Remove user from selected``` - if you added a user to the temporary list by mistake, you can click this button to remove him from.
- ```Add/Remove this user to/from the BlackList```
- `Open user/post` - open user/post in default browser.
- `Open picture` - open the original cached image in your default image viewer.
- `Open folder` - open channel cache folder.
- `Remove user from selected` - if you added a user to the temporary list by mistake, you can click this button to remove him from.
- `Add/Remove this user to/from the BlackList`
You can read more about the BlackList [here](https://github.com/AAndyProgram/SCrawler/wiki#black-list).
## Hot keys
- ```Insert``` - add a new channel.
- ```F5``` - start downloading.
- ```F2``` - go to previous images **page**.
- ```F3``` - go to next images **page**.
- ```F1``` - go to previous **channel**.
- ```F4``` - go to next **channel**.
- ```F8``` - add selected users to the program (from the temporary list) for future download.
- `Insert` - add a new channel.
- `F5` - start downloading.
- `F2` - go to previous images **page**.
- `F3` - go to next images **page**.
- `F1` - go to previous **channel**.
- `F4` - go to next **channel**.
- `F8` - add selected users to the program (from the temporary list) for future download.
Double-click the post thumbnail to open its full-size image.

389
Home.md

@@ -1,19 +1,19 @@
# Main Window
![Main window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindow.png)
![Main window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindow.png)
This window contains all data management tools.
- ```Add/Edit/Delete``` - actions with users.
- ```Refresh``` - refresh the user list. You do not need to press this button. Added just in case...
- ```Info``` - [show](#info) downloaded users and what exactly and how many were downloaded.
- ```Feed``` - [show](#feed) recently downloaded data
- ```Channels``` - open [Channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels) form
- ```Saved posts``` - download [saved posts](#saved-posts)
- ```Download``` - [download](#download) operations
- ```Stop``` - stop downloading
- ```View``` - [change](#view) view mode
- ```LOG``` - program error log and information
- `Add/Edit/Delete` - actions with users.
- `Refresh` - refresh the user list. You do not need to press this button. Added just in case...
- `Info` - [show](#info) downloaded users and what exactly and how many were downloaded.
- `Feed` - [show](#feed) recently downloaded data
- `Channels` - open [Channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels) form
- `Saved posts` - download [saved posts](#saved-posts)
- `Download` - [download](#download) operations
- `Stop` - stop downloading
- `View` - [change](#view) view mode
- `LOG` - program error log and information
## Buttons
@@ -25,46 +25,46 @@ This form will show downloaded items (sorted by latest download date). Each line
Double-clicking on the data line will open the content folder.
- ```View``` - view modes:
- ```Session``` - show downloaded results for this session only;
- ```All``` - show downloaded results for all users, sorted by last downloaded date.
- ```Up / Down``` - list navigation buttons
- ```Find``` - find the selected user in the main window
- ```Refresh``` - update information list. You don't need to press this button. Added just in case...
- ```Clear``` - clear the information list of the current download session.
- `View` - view modes:
- `Session` - show downloaded results for this session only;
- `All` - show downloaded results for all users, sorted by last downloaded date.
- `Up / Down` - list navigation buttons
- `Find` - find the selected user in the main window
- `Refresh` - update information list. You don't need to press this button. Added just in case...
- `Clear` - clear the information list of the current download session.
In the ```Session``` view, the list is updated automatically.
In the `Session` view, the list is updated automatically.
Hot keys:
- ```F2``` - Down
- ```F3``` - Up
- ```F4``` - Find
- ```F5``` - Refresh
- `F2` - Down
- `F3` - Up
- `F4` - Find
- `F5` - Refresh
![Info window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/DownloadInfo.png)
![Info window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/DownloadInfo.png)
### Missing posts
This form shows posts that have not been downloaded. This form is opened by right-clicking on the '[Info](#info)' button in the main window
Buttons:
- ```Delete``` - delete selected posts
- ```Update``` - check the missing posts again and refill the list
- ```Download ALL``` - download all missing posts (not working, in development)
- ```Info``` - show post information
- `Delete` - delete selected posts
- `Update` - check the missing posts again and refill the list
- `Download ALL` - download all missing posts (not working, in development)
- `Info` - show post information
Context menu:
- ```Download``` - download selected post
- ```Open post``` - open post in browser
- ```Open user folder``` - open user folder
- ```Show post information``` - show post information
- ```Find user``` - find user in the main window
- ```Delete post``` - delete selected post
- `Download` - download selected post
- `Open post` - open post in browser
- `Open user folder` - open user folder
- `Show post information` - show post information
- `Find user` - find user in the main window
- `Delete post` - delete selected post
Hot keys:
- ```F1``` - ```Info```
- `F1` - `Info`
![Missing posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MissingPosts.png)
![Missing posts window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MissingPosts.png)
### Feed
@@ -72,36 +72,36 @@ This form will show the recently downloaded data (images and videos). Each image
Double-clicking on an image will open the original image in an external viewer.
![Feed window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindow.png)
![Feed window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedWindow.png)
Buttons:
- ```Refresh``` - the feed will be updated to receive new downloaded items and/or remove missing items
- ```Clear``` - clear list of recent downloads
- ```Delete selected``` - delete marked items from the disk
- `Refresh` - the feed will be updated to receive new downloaded items and/or remove missing items
- `Clear` - clear list of recent downloads
- `Delete selected` - delete marked items from the disk
Context menu:
- ```Open picture/video``` - open picture/video in external viewer
- ```Open user``` (with folder image) - open user folder
- ```Open user``` (with globe) - open user site
- ```Open post``` - open post in browser
- ```Find user``` - find user in the main window
- ```Information``` - show post information
- ```Delete picture/video``` - delete picture/video
- `Open picture/video` - open picture/video in external viewer
- `Open user` (with folder image) - open user folder
- `Open user` (with globe) - open user site
- `Open post` - open post in browser
- `Find user` - find user in the main window
- `Information` - show post information
- `Delete picture/video` - delete picture/video
![Feed context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowItemContext.png)
![Feed context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedWindowItemContext.png)
Load sessions:
- ```Load last session``` - load last saved session data
- ```Select loading session``` - Load the data of the session you have chosen (the session selection form will open)
- `Load last session` - load last saved session data
- `Select loading session` - Load the data of the session you have chosen (the session selection form will open)
*The buttons are only displayed if the ```Store session data``` option is [enabled](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed)*
*The buttons are only displayed if the `Store session data` option is [enabled](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed)*
![Feed session context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowSessionContext.png)
![Feed session context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedWindowSessionContext.png)
Hot keys
- ```F5``` - ```Refresh```
- ```F3``` - Previous page
- ```F4``` - Next page
- `F5` - `Refresh`
- `F3` - Previous page
- `F4` - Next page
You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
@@ -115,120 +115,160 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the ```Saved posts user``` textbox. Click OK. Go to the main window.
Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window.
This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the ```!Saved posts``` folder.
This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the `!Saved posts` folder.
![Saved posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SavedPosts.png)
![Saved posts window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SavedPosts.png)
### Download
- ```Download selected (F5)``` - start downloading one or more selected users.
- ```Download all```: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!**
- ```Download all (F6)``` - Download all users marked ```Ready for download``` from all sites.
- ```Download all site users``` - Download all users marked ```Ready for download``` from specific sites.
- ```Download all FULL``` - Download all users from all sites. **The ```Ready for download``` option will be ignored.**
- ```Download all site users FULL``` - Download all users from specific sites. **The ```Ready for download``` option will be ignored.**
- ```Standalone downloader``` - Just a [downloader](#download-separate-video) for single videos.
- ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)).
- ```Silent mode``` - Temporarily disabling notifications.
- ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)).
- ```Pause automation``` - [Pause the automation process](#pause-automation).
- ```Stop``` - stop all download operations.
- `Download selected (F5)` - start downloading one or more selected users.
- `Download all`: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!**
- `Download all (F6)` - Download all users marked `Ready for download` from all sites.
- `Download all site users` - Download all users marked `Ready for download` from specific sites.
- `Download all FULL` - Download all users from all sites. **The `Ready for download` option will be ignored.**
- `Download all site users FULL` - Download all users from specific sites. **The `Ready for download` option will be ignored.**
- `Standalone downloader` - Just a [downloader](#download-separate-video) for single videos.
- `Add a new download group` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)).
- `Silent mode` - Temporarily disabling notifications.
- `Automation` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)).
- `Pause automation` - [Pause the automation process](#pause-automation).
- `Stop` - stop all download operations.
[HotKeys](#hot-keys)
![Download context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
![Download context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowGroups.png)
#### Pause automation
This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.
- ```1 hour``` - pause automation for 1 hour
- ```2 hour``` - pause automation for 2 hours
- ```3 hour``` - pause automation for 3 hours
- ```4 hour``` - pause automation for 4 hours
- ```6 hour``` - pause automation for 6 hours
- ```12 hour``` - pause automation for 12 hours
- ```Until``` - you will be prompted to enter the time you want to pause the task(s)
- ```Unlimited``` - pause the task until you turn it off or close the program
- ```Disable``` - disable pause
- `1 hour` - pause automation for 1 hour
- `2 hour` - pause automation for 2 hours
- `3 hour` - pause automation for 3 hours
- `4 hour` - pause automation for 4 hours
- `6 hour` - pause automation for 6 hours
- `12 hour` - pause automation for 12 hours
- `Until` - you will be prompted to enter the time you want to pause the task(s)
- `Unlimited` - pause the task until you turn it off or close the program
- `Disable` - disable pause
![Pause context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowPause.png)
![Pause context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowPause.png)
### View
- ```View``` - users view modes.
- `View` - users view modes.
- The first block specifies **how** users will be displayed in the form.
- ```Large/Small images``` - with icons.
- ```List``` / ```Details``` - simple, without icons.
- `Large/Small images` - with icons.
- `List` / `Details` - simple, without icons.
- The second block specifies the user **sites** that will be displayed in the form.
- ```All``` - default
- ```Specific sites``` - select sites to be displayed
- `All` - default
- `Specific sites` - select sites to be displayed
- The third block specifies **which** users will be displayed in the form.
- ```All``` - default.
- ```Regular``` - only show users who are not marked as ```Favorite``` or ```Temporary```.
- ```Temporary``` - only show users marked as temporary.
- ```Favorite``` - only show users marked as favorite.
- ```Deleted``` - only show users whose profiles have been deleted
- ```Suspended``` - only show users whose profiles have been suspended
- ```Labels``` - show users labeled with one or more selected label
- ```No Labels``` - only show users without a label
- ```Excluded labels``` - exclude from displaying users with the selected labels (this is suitable for situations where you want to hide a user but don't want to delete)
- ```Ignore excluded labels``` - Ignore excluding labels. This allows you to show users that should be excluded from display without resetting the current selection of excluded labels.
- ```Show groups instead of labels``` - show sites groups instead of labels (when filtering users by labels)
- `All` - default.
- `Regular` - only show users who are not marked as `Favorite` or `Temporary`.
- `Temporary` - only show users marked as temporary.
- `Favorite` - only show users marked as favorite.
- `Deleted` - only show users whose profiles have been deleted
- `Suspended` - only show users whose profiles have been suspended
- `Labels` - show users labeled with one or more selected label
- `No Labels` - only show users without a label
- `Excluded labels` - exclude from displaying users with the selected labels (this is suitable for situations where you want to hide a user but don't want to delete)
- `Ignore excluded labels` - Ignore excluding labels. This allows you to show users that should be excluded from display without resetting the current selection of excluded labels.
- `Show groups instead of labels` - show sites groups instead of labels (when filtering users by labels)
- The fourth block specifies the **date range** of the last download, whose users will be displayed in the form.
- ```Limit dates (not in range)``` - filter users whose last download date is not in the selected date range
- ```Limit dates (in range)``` - filter users whose last download date is in the selected date range
- `Limit dates (not in range)` - filter users whose last download date is not in the selected date range
- `Limit dates (in range)` - filter users whose last download date is in the selected date range
![Main window view context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowView.png)
![Main window view context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowView.png)
### LOG
The error log will be opened by clicking on this button. The log contains program and download errors. The log is saved automatically when you close the program, but you still can save, copy, or clear it (using the special buttons on the log form) if you want.
## Hot keys
- ```F1``` - program information and checking for a new version
- ```Insert``` - add user
- ```Del``` - delete one or more selected users
- ```Enter``` - open user folder
- ```F3``` - edit user
- ```F5```, ```Ctrl+F5``` - start downloading one or more selected users (pressing with ```Ctrl``` excludes this data from the feed)
- ```F6```, ```Ctrl+F6``` - start downloading all your users marked as ```Ready for download``` (pressing with ```Ctrl``` excludes this data from the feed)
- ```Ctrl+F``` - open search form
- ```Ctrl+NumericKey``` - download group with this number
- `F1` - program information and checking for a new version
- `Insert` - add user
- `Del` - delete one or more selected users
- `Enter` - open user folder
- `F3` - edit user
- `F5`, `Ctrl+F5` - start downloading one or more selected users (pressing with `Ctrl` excludes this data from the feed)
- `F6`, `Ctrl+F6` - start downloading all your users marked as `Ready for download` (pressing with `Ctrl` excludes this data from the feed)
- `Ctrl+F` - open search form
- `Ctrl+NumericKey` - download group with this number
Double-click the user to open the content folder.
## Download separate video
*The download does not start automatically in this form*
Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader)
- ```Add``` - add one video or Instagram post to download.
- ```Add list``` - add one or more video URLs to download.
- ```Delete``` - remove URL.
- ```Download``` - download all videos you added.
- ```Open saving path``` - a folder will open where the videos are downloaded to.
![Separate video downloader window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SeparateVideoDownloader.png)
![Separate video downloader window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTube.png)
## Tray
- ```Silent mode``` - Temporarily disabling notifications.
- ```Pause automation``` - [Pause the automation process](#pause-automation).
- [```Channels```](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- [```Feed```](#feed)
- ```Show/Hide``` - show/hide SCrawler
- ```Close``` - close the program
- ```Close (no script)``` - close the program ignore the script (if you configured the script in the settings)
- `Silent mode` - Temporarily disabling notifications.
- `Pause automation` - [Pause the automation process](#pause-automation).
- [`Channels`](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- [`Feed`](#feed)
- `Show/Hide` - show/hide SCrawler
- `Close` - close the program
- `Close (no script)` - close the program ignore the script (if you configured the script in the settings)
![Tray context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/TrayContextMenu.png)
![Tray context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/TrayContextMenu.png)
## Search users
```Ctrl+F``` to start searching
`Ctrl+F` to start searching
![Search form](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SearchUsers.png)
![Search form](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SearchUsers.png)
## Users list
### Add user
In the main window, click the `Add` button to open the user creation form.
Paste the copied URL into the `User name` field. **Ignore if the username is set to something other than the actual `User name`.** For some sites, this is necessary for internal purposes. If the site is identified, don't change the `User name` manually.
If you want to add a folder as a user, you need to use the following pattern: `D:\Data\`.
![Add user](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/CreateUserClear.png)
- `User name` - **Paste the user profile URL here.**
- `Friendly name` - if configured, this name will be displayed in the user list instead of the username (if this field is green, it means that the user site name is shown). `F` button to set the friendly name you configured; `S` button to set site name.
- `Special path` - specify a path other than the default path for user data. If you set an absolute path *(for example, `D:\Data\`)*, that path will be used as the user's path. If you set a relative path *(for example, `SomeFolder`, `SomeFolder\SomeFolder2`)*, then this path will be created relative to the root folder for this site (configured in the site settings).
- `Options` - additional user options.
- `Sites combo box` - site pointers.
- `Temporary` - mark user as `Temporary`.
- `Favorite` - mark user as `Favorite`.
- `Download images/videos` - for the created user, only the selected media types will be downloaded.
- `Ready for download` - download this user in the operation `Download all`.
- `Get user media only` - this is an option for twitter only. Download all posts posted by this author.
- `Add by list` - add users' profiles by list (new line as separator (`Shift+Enter` to insert new line)).
- If you add users by the URL, you must check the `Auto detect site` checkbox;
- If you are adding users by username, you must select the site in the combo box.
- `Auto detect site` - add users by the URLs.
- `Description` - some information you want to keep. If the `Add by list` checkbox is checked, you must enter a list of users in this textbox (`Shift+Enter` to insert new line);
- `Labels` - labels associated with this user (used to filter users in the main window).
- `Script` - If the checkbox is checked, script to be executed after the user download is complete. If the text is empty, the script from the global settings will be executed. [How to use the script](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-use-the-script).
If neither `Temporary` nor `Favorite` is checked, the user will be displayed in the regular site group.
**In collection editing mode checkboxes have three states. When loaded, an indeterminate state means that profiles have different values for this option. When you save, checkboxes that are in indeterminate state will not be applied.**
#### Labels
In this form, you can create labels and select them for users.
Click the `+` button to add a new one.
You can pick up one or more labels in this form.
![Labels Form](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/Labels.png)
#### Hot keys
- `F2` - open advanced options
- `F4` - open labels form for pick up a label
### User colors
- Users whose profile has been **deleted** are marked in **red**.
- Users whose profile has been **suspended** are marked in **yellow**.
@@ -236,32 +276,58 @@ Double-click the user to open the content folder.
- Collections that contain active profiles and deleted profiles are marked in **blue**.
### Context menu
- ```Download data``` - download this user/collection data.
- ```Download data limited``` - download this user/collection data only the specified number of posts.
- ```Download data to the date``` - download this user/collection data up to selected date.
- ```Edit user information``` - open a form to change user parameters.
- ```Delete user/collection``` - delete one or more selected users/collections.
- ```Copy data to another folder``` - allows you to copy user data to another folder.
- ```Change favorite``` - if the user was marked as favorite, he becomes a regular; otherwise, he becomes the favorite.
- ```Change temporary``` - if the user was marked as temporary, he becomes a regular; otherwise, he becomes the temporary.
- ```Change ready for download``` - download this user in the operation ```Download all```.
- ```Change labels``` - change user labels.
- ```Change script usage``` - change script usage for selected users.
- ```Add to collection``` - add user to collection.
- ```Merge collection files``` - move files to collection root folder.
- ```Change folder``` - change user data folder
- ```Change image``` - change user icon.
- ```Open contains folder``` - open selected user/collection folder.
- ```Open site``` - open user profile.
- ```Information``` - some user information.
- `Download data` - download this user/collection data.
- `Download data limited` - download this user/collection data only the specified number of posts.
- `Download data to the date` - download this user/collection data up to selected date.
- `Edit user information` - open a form to change user parameters.
- `Delete user/collection` - delete one or more selected users/collections.
- `Copy data to another folder` - allows you to copy user data to another folder.
- `Change favorite` - if the user was marked as favorite, he becomes a regular; otherwise, he becomes the favorite.
- `Change temporary` - if the user was marked as temporary, he becomes a regular; otherwise, he becomes the temporary.
- `Change ready for download` - download this user in the operation `Download all`.
- `Change labels` - change user labels.
- `Change script usage` - change script usage for selected users.
- `Add to collection` - add user to collection.
- `Merge collection files` - move files to collection root folder.
- `Change folder` - change user data folder
- `Change image` - change user icon.
- `Open contains folder` - open selected user/collection folder.
- `Open site` - open user profile.
- `Information` - some user information.
![Context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainContext.png)
![Context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainContext.png)
### Deleting a user
SCrawler has advanced deletion options. It can delete a user/[collection](#Deleting%20a%20collection) with or without a ban, delete a user with his files, delete a user from SCrawler without deleting his files. The following diagram shows all available options.
```mermaid
graph TB
style C fill:#f00,color:white,font-weight:bold
subgraph Delete user
A[Delete]
A -->|With ban|A1[Enter ban reason]
A -->|Without ban|B[Process]
A -->|User only|Q[Delete user/collection from SCrawler]
A1 --> B
B -->|Is user| C(Delete files)
C--> Q
Q --> Z[Exit]
end
subgraph Delete collection
B-->|Is collection| CC[Collection deletion confirmation message]
CC-->|Delete| C
CC-->|Split| E[Split collection]
CC-->|Cancel| Z
E-->EE[Extract users and add them to SCrawler as regular users]
EE-->Q
end
```
## Collections
I've developed collections for users with multiple profiles, but you can use them however you like.
To add a user to the collection, click ```Add to collection``` in the context menu. Add a new one by clicking the ```+``` button, or select an existing one.
To add a user to the collection, click `Add to collection` in the context menu. Add a new one by clicking the `+` button, or select an existing one.
When you add a user to a collection, the user's data will be moved to the collection folder with the name you specified.
@@ -269,26 +335,29 @@ You can select multiple users and a collection to add users to this collection.
### Merging collection files
When you add a user to the collection, a new folder will be created in the collection folder named by pattern ```Site```_```UserName```. That is, each profile in the collection is stored separately. When you merge, all files in the profiles' folders are moved to the root of the collection.
When you add a user to the collection, a new folder will be created in the collection folder named by pattern `Site`_`UserName`. That is, each profile in the collection is stored separately. When you merge, all files in the profiles' folders are moved to the root of the collection.
### Deleting a collection
1. Select the collection you want to delete.
1. Click the ```Delete``` button or right-click on the collection and select ```Delete user / collection```.
- ```Delete``` - The collection and all its files will be permanently deleted.
- ```Split``` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
- ```Cancel``` - Cancel operation.
1. Click the `Delete` button or right-click on the collection and select `Delete user / collection`.
- `Delete` - The collection and all its files will be permanently deleted.
- `Split` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
- `Cancel` - Cancel operation.
Go to a [diagram](#Deleting%20a%20user) that illustrates all the available options.
### Collections example
You have two users: ```SomeUserName1``` (from Reddit) and ```SomeUserName1``` (from Twitter).
1. Creation of the ```First collection``` collection.
1. The folder ```First collection``` will be created in the collections folder (you specify it in the program settings).
1. The ```SomeUserName1``` (Reddit) folder will be moved to the ```First collection``` folder and renamed to ```Reddit_SomeUserName1```.
1. The ```SomeUserName1``` (Twitter) folder will be moved to the ```First collection``` folder and renamed to ```Twitter_SomeUserName1```.
You have two users: `SomeUserName1` (from Reddit) and `SomeUserName1` (from Twitter).
1. Creation of the `First collection` collection.
1. The folder `First collection` will be created in the collections folder (you specify it in the program settings).
1. The `SomeUserName1` (Reddit) folder will be moved to the `First collection` folder and renamed to `Reddit_SomeUserName1`.
1. The `SomeUserName1` (Twitter) folder will be moved to the `First collection` folder and renamed to `Twitter_SomeUserName1`.
If you want to merge the collection.
Files from the ```Reddit_SomeUserName1``` and ```Twitter_SomeUserName1``` folders will be moved to the ```First collection``` folder.
Files from the `Reddit_SomeUserName1` and `Twitter_SomeUserName1` folders will be moved to the `First collection` folder.
### Virtual mode
@@ -300,7 +369,7 @@ Adding a user to a virtual collection is possible only in virtual mode.
# Import users
```Settings-Import users```
`Settings-Import users`
You can import users that are not in SCrawler from the specific folder. You can also scan existing SCrawler directories to find missing users.
@@ -308,7 +377,7 @@ You can import users that are not in SCrawler from the specific folder. You can
# Black list
The program has the ```Black list```, which included users you have added to. It is only a blacklist of the program and no impact on the site!
The program has the `Black list`, which included users you have added to. It is only a blacklist of the program and no impact on the site!
Users in this list will be skipped when downloading channels. If you add a new user to the program and this user is already blacklisted, you will receive a notification about. The notification includes the username and the reason (if specified) for deletion.

538
Plugins.md

@@ -1,10 +1,10 @@
You can develop your plugin for any site you want. I created a plugin environment with full SCrawler integration. You can display any class properties in SCrawler's internal edit forms.
The class that contains all the available objects is the Instagram settings:
- [```SiteSettings```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/SiteSettings.vb)
- [```UserData```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/UserData.vb)
- [```SavedPosts```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/ProfileSaved.vb)
- [```SiteSettingsBase```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Base/SiteSettingsBase.vb)
- [`SiteSettings`](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/SiteSettings.vb)
- [`UserData`](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/UserData.vb)
- [`SavedPosts`](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/ProfileSaved.vb)
- [`SiteSettingsBase`](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Base/SiteSettingsBase.vb)
You can look at them as an example.
@@ -14,121 +14,150 @@ You can also use the following plugins as an example:
**See how easy it is to create a plugin and make it for your site.**
The ```SCrawler.PluginProvider.dll``` is fully compatible with the most popular language - CSharp.
The `SCrawler.PluginProvider.dll` is fully compatible with the most popular language - CSharp.
Net.Framework version is 4.6.1.
# Plugins environment
The plugin concept is based on two main interfaces. The first interface - [```ISiteSettings```](#isitesettings) - is the major interface of the plugin. This class **must** provide all instances, validators, and other objects required by SCrawler. The second interface - [```IPluginContentProvider```](#iplugincontentprovider) - is an instance of UserData.
The plugin concept is based on two main interfaces. The first interface - [`ISiteSettings`](#isitesettings) - is the major interface of the plugin. This class **must** provide all instances, validators, and other objects required by SCrawler. The second interface - [`IPluginContentProvider`](#iplugincontentprovider) - is an instance of UserData.
# How to make a plugin
1.
1. Create a new Net.Framework library project.
1. Add a reference to ```SCrawler.PluginProvider.dll```
1. Add a reference to `SCrawler.PluginProvider.dll`
1.
1. Create a new settings class (e.g. ```SiteSettings```).
1. Implement the ```ISiteSettings``` interface.
1. Add the required ```Manifest``` [class attribute](#manifest).
1. Add optional attributes if you need: [```SeparatedTasks```](#separatedtasks), [```SavedPosts```](#savedposts), [```SpecialForm```](#specialform)
1. Create a new settings class (e.g. `SiteSettings`).
1. Implement the `ISiteSettings` interface.
1. Add the required [`Manifest`](#manifest) class attribute.
1. Add optional attributes if you need: [`SeparatedTasks`](#separatedtasks), [`SavedPosts`](#savedposts), [`SpecialForm`](#specialform)
1.
- Set values for [Site](#site), [Icon](#icon), [Image](#image).
- ```Site``` is a required property (cannot be null)
- If your plugin doesn't have an ```Icon``` or ```Image```, these properties must return ```Nothing``` (```null``` in C#).
- Set values for `Site`, `Icon`, `Image`.
- `Site` is a required property (cannot be null)
- If your plugin doesn't have an `Icon` or `Image`, these properties must return `Nothing` (`null` in C#).
- **Attention! Don't add any attributes to these properties!**
1. Define your own properties if you need to. To allow users to change the property values of your class and interact with SCrawler, this property must be an AutoProperty and must be declared as ```PropertyValue``` [object](#propertyvalue).
1. Define your own properties if you need to. To allow users to change the property values of your class and interact with SCrawler, this property must be an AutoProperty and must be declared as [`PropertyValue`](#propertyvalue) oblect.
1. Write the code for the interface functions. **If a function has a return type, then that function MUST return a value!**
1.
- Develop a user instance of the ```IPluginContentProvider``` interface
- This instance must be returned by the [```GetInstance```](#getinstance) function
- Develop a user instance of the `IPluginContentProvider` interface
- This instance must be returned by the [`GetInstance`](#getinstance) function
1. If you plugin has special options, develop an options exchange class and options editor form.
1. It is done. Your plugin is ready to work. :blush:
# How it works
1. When launched, SCrawler loads plugins from the ```Plugins``` folder.
1. A new [```SettingsHost```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/PluginsEnvironment/Hosts/SettingsHost.vb) is created for each plugin.
1. Get class attributes
1. Invoke [```BeginInit```](#begininit) function
1. Sending XML data to your plugin class using the [```Load```](#load) function.
1. Get class members
1. Create hosts for [```PropertyValue```](#propertyvalue)
1. Adding [property updaters](#propertyupdater) to the property host
1. Adding [property providers](#provider) to the property host
1. Adding [property data checkers](#propertiesdatachecker) to the property host
1. Invoke [```EndInit```](#begininit) function
1. Get property values stored in XML (and create new ones if they don't exist) and replace default values with XML values.
1. Creating a list of users using the [```GetInstance```](#getinstance) function
- For each user functions [```XmlFieldsSet```](#xmlfieldsset) and [```XmlFieldsGet```](#xmlfieldsget) will be called
1. When downloading
1. If [```Available(Main)```](#available) = ```false```, users of your plugin will not be downloaded.
1. Invoke [```DownloadStarted(Main)```](#downloadstarted)
1. Per downloading user
1. If [```ReadyToDownload(Main)```](#readytodownload) = ```true```
1. Invoke [```BeforeStartDownload(User, Main)```](#beforestartdownload)
1. Downloading data
1. Sending information to your instance:
- ```Settings``` = your settings instance
- ```Thrower```
- ```LogProvider```
- ```Name```
- ```ID```
- ```ParseUserMediaOnly```
- ```UserDescription```
- ```UserExists```
- ```UserSuspended```
- ```IsSavedPosts```
- ```SeparateVideoFolder```
- ```DataPath```
- ```PostsNumberLimit```
- ```ExistingContentList``` = list of existing content stored in user files
- ```TempPostsList``` = list of post IDs stored in user files
1. Invoke [```GetMedia```](#getmedia)
1. Get the ```TempPostsList``` list back
1. Get other parameters back
- ```Name```
- ```ID```
- ```UserDescription```
- ```UserExists```
- ```UserSuspended```
1. If the instance does not have [```UseInternalDownloader```](#useinternaldownloader) attribute, invoke [```Download```](#download) function
1. Invoke [```AfterDownload(User, Main)```](#afterdownload)
1. Invoke [```DownloadDone(Main)```](#downloaddone)
1. When downloading saved posts
1. If [```ReadyToDownload(SavedPosts)```](#readytodownload) = ```true```
1. If [```Available(SavedPosts)```](#available) = ```true```
1. Invoke [```DownloadStarted(SavedPosts)```](#downloadstarted)
1. Get an instance with [```GetInstance(SavedPosts)```](#getinstance). The Instance must contain the UserName (```Name```)
1. Set ```IsSavedPosts``` = ```true```
1. Invoke [```BeforeStartDownload(User, SavedPosts)```](#beforestartdownload)
1. The same steps as in: ```When downloading``` - ```Per downloading user``` - ```Downloading data```
1. Invoke [```AfterDownload(User, SavedPosts)```](#afterdownload)
1. Invoke [```DownloadDone(SavedPosts)```](#downloaddone)
1. When user creating / updating
1. If your class has special options ([```SpecialForm(False)```](#specialform)), the ```Options``` button will be enabled
1. [```UserOptions(OptionsArg, False)```](#useroptions) will be invoked. If ```OptionsArg``` is ```Nothing``` (```null``` in C#), you **MUST** create a new exchange object.
1. By clicking on the ```Options``` button, [```UserOptions(OptionsArg, True)```](#useroptions) will be invoked. If ```OptionsArg``` is ```Nothing``` (```null``` in C#), you **MUST** create a new exchange object. Then open the form to change the settings.
1. When updating a user, [```ExchangeOptionsGet```](#exchangeoptionsget) will be invoked. You **MUST** return a new exchange object with user-saved options. At the end of editing the user, [```ExchangeOptionsSet```](#exchangeoptionsset) will be invoked with your exchange object. You **MUST** apply the options of your exchange object to the user.
## Download
```mermaid
stateDiagram
classDef OK fill:#063,color:white,font-weight:bold
classDef pluginInt fill:#ff9
class Down OK
class DownSet, GetMedia, ReconfList, GrabBack, Down2 pluginInt
Start: User added to download
HostAvailable: ISiteSettings.Available
HostReady: ISiteSettings.ReadyToDownload
HostBefore: ISiteSettings.BeforeStartDownload
HostAfter: ISiteSettings.AfterDownload
Down: Download
Exit: Exit
Skip: Skip downloading
DownSet: Set class (IPluginContentProvider) parameters (1)
GetMedia: Call IPluginContentProvider.GetMedia function (2)
ReconfList: Reconfigure media list to send to downloader (4)
GrabBack: Get back parameters(3)
Down2: Call IPluginContentProvider.Download function
[*]-->Start
Start-->HostAvailable
HostAvailable-->HostReady: True
HostAvailable-->Exit: False
HostReady-->Skip: False
HostReady-->HostBefore: True
Skip-->Exit
HostBefore-->Down
Down-->DownSet
DownSet-->GetMedia
GetMedia-->GrabBack
GrabBack-->ReconfList
ReconfList-->Down2
Down2-->HostAfter
HostAfter-->Exit
Exit-->[*]
```
**Notes**:
1. SCrawler sets all `IPluginContentProvider` properties.
2. This is where you need to get the posts, their download URLs, and information about the posts (such as date, id, user, etc.).
3. SCrawler gets back parameters `Name`, `ID`, `UserDescription`, `UserExists` and `UserSuspended`.
4. All posts that are not requested by the user will be removed from the `TempMediaList` (eg images will be removed if the user only wants to download videos).
## Create user
```mermaid
stateDiagram
Start: User creating
HostMyUser: ISiteSettings.IsMyUser
HostOptions: ISiteSettings.UserOptions
Exit
note right of HostOptions
Open form if OpenForm is true
end note
[*]-->Start
Start-->HostMyUser
HostMyUser-->Exit: False
HostMyUser-->HostOptions: True
HostOptions-->Exit
Exit-->[*]
```
## Edit site settings
```mermaid
stateDiagram
Start: Call ISiteSettings.BeginEdit
SaveBefore: Call ISiteSettings.BeginUpdate
Save: Call ISiteSettings.Update
SaveAfter: Call ISiteSettings.EndUpdate
Exit: Call ISiteSettings.EndEdit
[*]-->Start
Start-->SaveBefore: Ok
Start-->Exit: Cancel
SaveBefore-->Save
Save-->SaveAfter
SaveAfter-->Exit
Exit-->[*]
```
# Interfaces
## ISiteSettings
**Don't duplicate the ```Temporary```, ```Favorite```, ```Path``` and ```SavedPostsPath``` properties in the settings class. These properties are basic host data.**
C# explanation
``` CSharp
interface ISiteSettings
{
Icon Icon {get;};
Image Image {get;};
string Site {get;};
string GetUserUrl(IPluginContentProvider User, bool Channel);
ILogProvider Logger {get; set;};
string GetUserUrl(IPluginContentProvider User);
ExchangeOptions IsMyUser(string UserURL);
ExchangeOptions IsMyImageVideo(string URL);
IEnumerable GetSpecialData(string URL);
IPluginContentProvider GetInstance(Download What);
void Load(IEnumerable<KeyValuePair<string, string>> XMLValues);
IEnumerable<KeyValuePair<string, string>> XMLFields();
@@ -142,6 +171,8 @@ interface ISiteSettings
void DownloadDone(Download What);
void BeginEdit();
void EndEdit();
void BeginUpdate();
void EndUpdate();
void Update();
void Reset();
void OpenSettingsForm();
@@ -150,185 +181,83 @@ interface ISiteSettings
}
```
### Site
`Site`, `Icon` and `Image` are properties that provide the site name and site icon (in `Icon` and `Image` formats).
Site name. This ReadOnly property must only provide the name of the site. For example: for instagram.com you should set the site name ```Instagram```. The first letter must be uppercase.
`IsMyUser` and `IsMyImageVideo` are URLs validators. These functions should return a value (`ExchangeOptions` with `Exists` set to `true`) indicating whether the user or media belongs to your plugin.
### Icon
`GetInstance` must return an instance for downloading data. The `What` parameter specifies what type of instance is being requested.
Site icon for forms and other objects (can be null).
`BeginInit` and `EndInit` are functions that are called when the plugin is initializing and completed initialization.
### Image
`BeginEdit` and `EndEdit` are functions that are called when the plugin settings are being edited by the SCrawler site editor form.
Site image for controls (such as buttons) and other objects.
`BeginUpdate`, `EndUpdate` and `Update` are function that are called by the site editor form (when the plugin settings have changed and need to be saved) and by the settings class.
### Logger
`Available` function must return a value indicating that the site is available. If false is returned, users will not be downloaded. **Called when jobs the pool is created.**
This is the SCrawler log provider. It is set automatically by SCrawler on initialization.
`ReadyToDownload` function must return the same value as the `Available` function, but it is **called before particular user actually downloads**.
### GetUserUrl
`BeforeStartDownload` and `AfterDownload` are functions that are called before and after a particular user downloads.
```Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String```
The `DownloadDone` function is called after all jobs have completed.
This function should return the user's URL based on the ```UserName``` argument. This function is called when the user clicks the ```Open site``` button in SCrawler.
If your plugin has an additional settings form, then this form must be opened in `ShowDialog` mode when calling the `OpenSettingsForm` function.
The ```Channel``` argument specifies that the user is the a channel.
`UserOptions` is the function for exchanging additional user options when creating a user. If `OpenForm` argument is true, you must open a form (in `ShowDialog` mode) where the user can manage additional user options.
### IsMyUser
`GetUserPostUrl` must return the post url provided by the `Media` argument.
```Function IsMyUser(ByVal UserURL As String) As ExchangeOptions```
## IDownloadableMedia
This function should validate the user URL (```UserURL``` argument) and return an [ExchangeOptions](#exchangeoptions) object if it is your plugin user; otherwise, return a new empty ```ExchangeOptions```
``` CSharp
interface IDownloadableMedia : IUserMedia, IDisposable
{
event EventHandler CheckedChange;
event EventHandler ThumbnailChanged;
event EventHandler StateChanged;
Icon SiteIcon {get;};
string Site {get;};
string SiteKey {get;};
string ThumbnailUrl {get; set;};
string ThumbnailFile {get; set;};
string Title {get; set;};
int Size {get; set;};
TimeSpan Duration {get; set;};
object Progress {get; set;};
bool HasError {get;};
bool Exists {get;};
bool Checked {get; set;};
IPluginContentProvider Instance {get; set;};
void Download(bool UseCookies,Threading.CancellationToken Token);
void Delete(bool RemoveFiles);
void Load(string File);
void Save();
string ToString();
string ToString(bool ForMediaItem);
}
```
If this is the user of your plugin, you must extract the UserName from the URL and return a new ```ExchangeOptions``` instance with the ```UserName``` and ```Site``` name.
Raise the `CheckedChange` event when value `Checked` has changed.
Raise the `ThumbnailChanged` event when value `ThumbnailUrl` or `ThumbnailFile` has changed.
Raise the `StateChanged` event when value `DownloadState` has changed.
If it is a channel, you must also set the ```IsChannel``` field (in ```ExchangeOptions```) to ```True```
`Site` is the name of your plugin's site. `SiteKey` is the name of the plugin. `ThumbnailUrl`, `ThumbnailFile`, `Title`, `Size` and `Duration` are information about the media being downloaded.
For your convenience, this structure provides two default initializers. Use them so you don't forget the fields.
`Progress` is a class that manages a progress bar. **Don't set this property manually.** This property is necessary for the internal needs of SCrawler.
`Exists` is a property indicating that media is parsed or loaded from a file. Should be true if the file downloaded and exists or not downloaded and the URL is OK.
### IsMyImageVideo
`Checked` is a value indicating the checked state on the form.
```Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions```
`Instance` is an instance of `IPluginContentProvider`. **Don't set this property manually.** It will be set later during SCrawler's internal algorithms.
Same as [```IsMyUser```](#ismyuser), but this function checks the ```URL``` argument to see if the ```URL``` is your site's media data. It is used for standalone video downloader.
`Download` is a function that will be called when the user downloads a file.
Must return the same as the [```IsMyUser```](#ismyuser), but ```UserName``` can be **any non-empty value**.
`Delete` is the function that will be called when the user requests to remove a file from the list. `RemoveFiles` indicating that the user also wants to delete the downloaded file.
### GetSpecialData
`Load` is the function that will be called when the user opens the downloader form. `Save` is the function that will be called when the user downloads the file.
```Function GetSpecialData(ByVal URL As String) As IEnumerable```
After ```IsMyImageVideo```, if this is your plugin's media file, this function will be called.
This function must return an IEnumerable ```PluginUserMedia``` object. This object must contain the data URL and the file name (with extension). Any other fields are optional.
- ```URL``` - media URL
- ```Path``` - last destination path
- ```AskForPath``` - if ```true```, you need to ask the user to select a path
**Only if the data is downloaded by your class:**
- If the returned data does not exists and ```AskForPath``` = ```true```, you need to return a ```PluginUserMedia``` object with ```SpecialPath``` = selected path.
- If the returned data exists, you need to replace ```PluginUserMedia.SpecialPath``` with the selected path.
### GetInstance
```Function GetInstance(ByVal What As Download) As IPluginContentProvider```
**This is the required main function that returns a UserData instance!**
The ```What``` argument indicates who called this function.
- ```Main``` - user instance
- ```SavedPosts``` - instance of saved posts
- ```Channel``` - an instance of a channel (main window).
You must return an initialized instance of the ```IPluginContentProvider``` object.
### GetUserPostUrl
```Function GetUserPostUrl(ByVal UserID As IPluginContentProvider, ByVal Media As IUserMedia) As String```
Return post URL based on user and media.
### Load
```Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))```
This function sends the XML fields of your site node to your settings class when the host is initialized.
### BeginInit
Called at the start of host initialization.
### EndInit
Called at the end of host initialization.
### BeginEdit
Called when opening the settings form in SCawler
### EndEdit
Called when closing the settings form in SCawler
### BeginUpdate
Called from the settings root when the settings ```BeginUpdate``` function is called.
### EndUpdate
Called from the settings root when the settings ```EndUpdate``` function is called.
### Available
```Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean```
If your plugin needs to be checked for availability (you can see an example in the [```API.Reddit.SiteSettings```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Reddit/SiteSettings.vb)), place the check code here. Otherwise, this function **MUST** return ```true```.
If this function returns ```false```, users of your plugin site will not be downloaded.
```Silent``` stands for show notification or not.
### ReadyToDownload
```Function ReadyToDownload(ByVal What As Download) As Boolean```
**It is not the same as ```Available```.**
This function is an additional method for checking your class parameters. If your parameters don't need to be validated, just return true.
With this function, you can check the main settings parameters that authorization depends on (for example) or anything else your parser requires.
### DownloadStarted
```Sub DownloadStarted(ByVal What As Download)```
This function will be called when the parser **thread** is initialized. **Not for user parsing!**
### BeforeStartDownload
```Sub BeforeStartDownload(ByVal User As Object, ByVal What As Download)```
This function will be called before the user starts downloading. Here you can set some of user options you need.
### AfterDownload
```Sub AfterDownload(ByVal User As Object, ByVal What As Download)```
This function will be called after the user download is complete.
### Update
```Sub Update()```
This function will be called when saving site settings on the site settings form. You don't need to update the ```PropertyValue``` properties (these properties are updated automatically).
### Reset
```Sub Reset()```
Not used. Added just in case.
### DownloadDone
```Sub DownloadDone(ByVal What As Download)```
This function will be called when the parser **thread** is closed.
### OpenSettingsForm
If your site settings have a special settings form, this function will be called by clicking on the button in the site settings form.
**Your form must be opened with the ```ShowDialog``` method!**
### UserOptions
```Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)```
**If your site settings have special user options**, this function will be called by clicking on the ```Options``` button in the ```UserCreatorForm```.
```Options``` is an object parameter (**reference**). **If it is null, you must create a new instance of your option exchange class.** SCrawler does not know what class you are using to exchange. So SCrawler will exchange this class (as an object) between your settings class and user instance! So **this class object cannot be null**!
If the ```OpenForm``` argument = ```True```, your options form **must** be opened (using the ShowDialog method). This form must provide controls for change parameters obtained using the ```Options```.
`ToString` is the function that should returns title of the media. `ForMediaItem` is an argument indicating that the title requesting a list media item.
## IPluginContentProvider
@@ -337,8 +266,8 @@ interface IPluginContentProvider : IDisposable
{
delegate void ProgressChangedEventHandler(int Count);
event ProgressChangedEventHandler ProgressChanged;
delegate void TotalCountChangedEventHandler(int Count);
event TotalCountChangedEventHandler TotalCountChanged;
delegate void ProgressMaximumChangedEventHandler(int Value, bool Add);
event ProgressMaximumChangedEventHandler ProgressMaximumChanged;
IThrower Thrower {get; set;};
ILogProvider LogProvider {get; set;};
ISiteSettings Settings {get; set;};
@@ -361,103 +290,35 @@ interface IPluginContentProvider : IDisposable
void ExchangeOptionsSet(object Obj);
void XmlFieldsSet(List<KeyValuePair<string, string>> Fields);
List<KeyValuePair<string, string>> XmlFieldsGet();
void GetMedia();
void Download();
void GetMedia(Threading.CancellationToken Token);
void Download(Threading.CancellationToken Token);
}
```
### ProgressChanged
Raise the `ProgressChanged` event when the file is downloaded.
Raise the `ProgressMaximumChanged` event before downloading files. The `Value` argument specifies the number of media files to download.
```Event ProgressChanged(ByVal Count As Integer)```
See the [Thrower](#IThrower) , [LogProvider](#ILogProvider) and [Settings](#ISiteSettings) property environments in their respective chapters. **Don't set these properties manually.** They will be set by SCrawler.
Event for performing progress. When this event is raised, SCrawler downloading progress will be performed for this number (```Count``` (default = 1)).
`Name` is the username returned from the settings class when a new user is created.
`ID` is the user ID on the site.
`UserDescription` is the user description.
`UserExists` and `UserSuspended` are properties indicating that the user is exists/suspended on the site
### TotalCountChanged
`ExchangeOptionsGet` and `ExchangeOptionsSet` are the functions for exchanging additional user options when the user is being edited by the user.
```Event TotalCountChanged(ByVal Count As Integer)```
`XmlFieldsGet` and `XmlFieldsSet` are the functions for exchanging additional user options when the user is being loaded by the SCrawler (from an XML file).
When you are starts downloading (before start), raise this event with the ```Count``` = total amount of media to download.
`ParseUserMediaOnly`, `IsSavedPosts`, `PostsNumberLimit`, `DownloadDateFrom`, `DownloadDateTo` are properties that define what the user want to download.
### Thrower
`SeparateVideoFolder` is a property that specifies that the video content should be separated from other content (such as images).
This is an exception thrower.
`TempPostsList` is a list of all post IDs downloaded by the plugin.
`ExistingContentList` is a list of posts downloaded by the plugin.
`TempMediaList` is a temporary list of posts which should be downloaded by the plugin.
While debugging, you can use any class you want, but **SCrawler will replace that instance with my thrower**.
To work, you don't need to develop a special object of this interface.
Read more [here](#ithrower)
### LogProvider
This is an interface for sending exceptions and messages to the program log.
While debugging, you can use any class you want, but **SCrawler will replace that instance with my log provider**.
To work, you don't need to develop a special object of this interface.
Read more [here](#ilogprovider)
### Settings
This object will be replaced by SCrawler with the active settings instance. **ATTENTION! Don't instantiate settings!**
### Other properties
**All of these properties will be delivered using SCrawler!**
- ```Name``` - user name
- ```ID```
- ```ParseUserMediaOnly```
- ```UserDescription```
- ```ExistingContentList``` - collection of the existing downloaded data
- ```TempPostsList``` - collection of the existing downloaded post id
- ```TempMediaList``` - this is a list of new media to download. You **must** add data to this list!
- ```UserExists``` - indicates that the user exists on the site
- ```UserSuspended``` - indicates that the user's profile suspended (or blocked) on the site
- ```IsSavedPosts``` - this is the instance for downloading saved posts
- ```SeparateVideoFolder``` - if true, the videos **must** be saved separately from the images folder
- ```DataPath``` - download path
- ```PostsNumberLimit``` - the **maximum** number of posts a user has requested to download! This is a **nullable** argument. So, if it's ```Nothing``` (or ```null``` in C#), then no limited download is requested!
- ```DownloadDateFrom```, ```DownloadDateTo``` - the **lowest** and **highest** date of posts a user has requested to download! This is a **nullable** argument. So, if it's ```Nothing``` (or ```null``` in C#), then no limited download is requested!
### ExchangeOptionsGet
```Function ExchangeOptionsGet() As Object```
This function is called by the UserCreatorForm when **editing** an existing user (not when creating a new one).
This function must return the ```ExchangeOptions``` (same as [```UserOptions```](#useroptions) function) with the values stored in this user instance.
### ExchangeOptionsSet
```Sub ExchangeOptionsSet(ByVal Obj As Object)```
Set options in this instance with ```ExchangeOptions```. ```Obj``` is the same object as in [```UserOptions```](#useroptions) function.
### XmlFieldsSet
```Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))```
Delivery of XML fields stored in the user settings file, to the current instance.
- Key - XML field name
- Value - XML field value
### XmlFieldsGet
```Function XmlFieldsGet() As List(Of KeyValuePair(Of String, String))```
You must return the XML fields that you want to store in the user settings file.
- Key - XML field name
- Value - XML field value
### GetMedia
Parse site user data
### Download
Download data using parsed information
`GetMedia`: collect posts from the site and add then to the `TempMediaList` list.
`Download`: download posts provided by the `TempMediaList` list.
## IPropertyProvider
@@ -468,17 +329,15 @@ interface IPropertyProvider : IFormatProvider
}
```
### PropertyName
The name of the property you are working with
If your `IFormatProvider` class implements this interface, the `PropertyName` property will be set to the currently editable property name by SCrawler's internal algorithms.
## IThrower
This is an exception thrower. If the user requested to cancel operation or delete user instance, an exception will be thrown. Use the ```IThrower.ThrowAny``` method in your code to stop executing when it's requested!
This is an exception thrower. If the user has requested to cancel the operation or delete the user instance, an exception will be thrown. Use the `IThrower.ThrowAny` method in your code to stop execution when requested!
Exceptions:
- ```OperationCanceledException``` - when user requested to cancel operation
- ```ObjectDisposedException``` - when user deletes user instance
- ```OperationCanceledException``` - when user requested to cancel the operation
- ```ObjectDisposedException``` - when user deletes the user instance
## ILogProvider
@@ -527,8 +386,8 @@ Currently used in [```IsMyUser```](#ismyuser) and [```IsMyImageVideo```](#ismyim
- ```UserName``` - user name or any other data (if specified in the method description)
- ```SiteName``` - site name
- ```HostKey``` - aet automatically via ```SettingsHost```
- ```IsChannel``` - this user is a channel
- ```HostKey``` - set automatically via ```SettingsHost```
- `Exists`
# Attributes
@@ -571,7 +430,7 @@ I have developed a fully integrated plugin environment. This means that any of y
**```SpecialForm(True)```**, for class settings. This means that an additional button will be added to the site settings form to opening your settings form. In this case, it is necessary to develop an additional form that must be opened (using the ```ShowDialog``` method) by the ```OpenSettingsForm``` [class function](#opensettingsform).
**```SpecialForm(False)```**, for class settings. This means that the ```Options``` button will be enabled on the user creator form. In this case, you must develop an additional exchange class and form. This works with the ```UserOptions``` class function. This [function](#useroptions) has two arguments: ```Options``` (reference) and ```OpenForm```.
**```SpecialForm(False)```**, for user settings. This means that the ```Options``` button will be enabled on the user creator form. In this case, you must develop an additional exchange class and form. This works with the ```UserOptions``` class function. This [function](#useroptions) has two arguments: ```Options``` (reference) and ```OpenForm```.
## Provider
@@ -644,5 +503,4 @@ Assembly attribute. Not used yet, but in future versions will allow SCrawler to
| Decimal | decimal | System.Decimal |
| String | string | System.String |
| Date | DateTime | System.DateTime |
| TimeSpan | TimeSpan | System.TimeSpan |
| TimeSpan | TimeSpan | System.TimeSpan |

627
Settings.md

@@ -1,218 +1,284 @@
# Program settings
## Basis
- ```Data path``` - this is the root path where the data will be placed.
- ```Large/Small image size``` - the size of the user icons.
- ```Collections folder``` - just the name of the collections folder.
- ```Maximum download tasks of users``` - the number of simultaneously downloading profiles.
- ```Maximum download tasks of channels``` - the number of simultaneously downloading channels.
- ```Check new version at start```
- ```UserAgent``` - default UserAgent to use in requests ([how to find](how-to-find-useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect.
- ```Imgur Client ID``` - Imgur client ID to Bypass NSFW protection and download galleries
- ```Userlist image``` - customize the image that will be displayed at the background of the main window
- ```Userlist color``` - set the background color and font color of the main window (`F` button to select font color; `C` button to select background color)
- ```Show groups``` - show user site groups
- ```Use user grouping``` - Group users by groups and/or labels. If disabled, users will not be grouped by site or label.
- `Data path` - this is the root path where the data will be placed.
- `Large/Small image size` - the size of the user icons.
- `Collections folder` - just the name of the collections folder.
- `Maximum download tasks of users` - the number of simultaneously downloading profiles.
- `Maximum download tasks of channels` - the number of simultaneously downloading channels.
- `Check new version at start`
- `UserAgent` - default UserAgent to use in requests ([how to find](#how%20to%20find%20useragent)). This is not a required field. Use it if you need to replace the UserAgent for all sites. A restart of SCrawler is required to take effect.
- `Imgur Client ID` - Imgur client ID to Bypass NSFW protection and download galleries
- `Userlist image` - customize the image that will be displayed at the background of the main window
- `Userlist color` - set the background color and font color of the main window (`F` button to select font color; `C` button to select background color)
- `Show groups` - show user site groups
- `Use user grouping` - Group users by groups and/or labels. If disabled, users will not be grouped by site or label.
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBasis.png)
![Basis settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBasis.png)
## Environment
- `ffmpeg` - path to ffmpeg.exe file
- `cURL` - path to curl.exe file
- `yt-dlp` - path to yt-dlp.exe file
- `gallery-dl` - path to gallery-dl.exe file
- `CMD Encoding` - command line encoding. It is highly recommended to change this value to an encoding that fully supports your language. The default value is 65001 (Unicode).
![Environment settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalEnvironment.png)
## Behavior
- ```Exit confirm``` - ask for confirmation before closing the program
- ```Close to tray``` - close program to system tray
- ```Fast profiles loading``` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.**
- ```Delete data to recycle bin``` - delete data to recycle bin or permanent
- ```Open the Info form when the download start```
- ```Open the Progress form when the download start```
- ```Don't open again``` - Do not automatically open the corresponding form if it was once closed
- ```Folder cmd``` - the [command](#folder-command) to open a folder
- ```Close cmd``` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed
- `Exit confirm` - ask for confirmation before closing the program
- `Close to tray` - close program to system tray
- `Fast profiles loading` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.**
- `Delete data to recycle bin` - delete data to recycle bin or permanent
- `Open the Info form when the download start`
- `Open the Progress form when the download start`
- `Don't open again` - Do not automatically open the corresponding form if it was once closed
- `Folder cmd` - the [command](#folder%20command) to open a folder
- `Close cmd` - this [command](#scrawler%20script%20text%20examples) will be executed when SCrawler is closed
![Behavior settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
![Behavior settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalBehavior.png)
## Notifications
- ```Silent mode``` - Temporarily disabling notifications. This setting is not stored in the settings file. It is valid until you turn it off or close the program.
- ```Show notification``` - This is the base value of notifications. If you disable it, notifications will not appear at all.
- ```Profiles``` - Show notifications when profiles download is complete.
- ```AutoDownloader``` - Show AutoDownloader notifications.
- ```Channels``` - Show notifications when channels download is complete.
- ```Saved posts``` - Show notifications when saved posts download is complete.
- `Silent mode` - Temporarily disabling notifications. This setting is not stored in the settings file. It is valid until you turn it off or close the program.
- `Show notification` - This is the base value of notifications. If you disable it, notifications will not appear at all.
- `Profiles` - Show notifications when profiles download is complete.
- `AutoDownloader` - Show AutoDownloader notifications.
- `Channels` - Show notifications when channels download is complete.
- `Saved posts` - Show notifications when saved posts download is complete.
- `Standalone downloader` - Show notifications when all downloads in standalone downloader are completed.
- `Standalone downloader (every download)` - Show notifications when download in standalone downloader is complete.
![Notifications settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalNotifications.png)
![Notifications settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalNotifications.png)
## Defaults
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
- ```Temporary``` - this parameter specifies how users will be created by default in the user creation form.
- ```Download images/videos``` - defaults for creating new users
- ```Download jpg instead of webp``` - save ```webp``` images as ```jpg```
- ```Use the site name as a friendly name``` - use the user's site name as a friendly name
- `Separate video folder` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
- `Temporary` - this parameter specifies how users will be created by default in the user creation form.
- `Download images/videos` - defaults for creating new users
- `Download jpg instead of webp` - save `webp` images as `jpg`
- `Use the site name as a friendly name` - use the user's site name as a friendly name
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDefaults.png)
![Default settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalDefaults.png)
## Downloader
- `Max. jobs count` - maximum number of concurrent jobs.
- `Download automatically` - start downloading automatically when new URL is added.
- `Remove downloaded automatically` - remove downloaded item from the list when item downloading is complete.
- `DoubleClick opens` - what do you want to open when you double click on an item.
- `Create video thumbnail` - create video thumbnail for downloaded video.
- `Update the YouTube output path when you change the output path` - update YouTube output path every time you change file destination.
- `Load downloaded YouTube videos to the form` - If checked, downloaded YouTube videos will be loaded to the form. Otherwise, all downloaded data will be loaded to the form except YouTube data.
- `Clear YouTube videos when clearing the list` - If checked, YouTube videos will also be removed from the list. This action will also affect the standalone [`YouTubeDownloader`](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader) app.
![Downloader settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalDownloader.png)
## Downloading
- ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line.
- ```Update user site name every time``` - this means that the user site name (if implemented) will be checked when the user is parsed.
- ```Change file names``` - name files by name (not by original names)
- `Update user description every time` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line.
- `Update user site name every time` - this means that the user site name (if implemented) will be checked when the user is parsed.
- `Change file names` - name files by name (not by original names)
- Options:
- ```Replace file name by date``` - the file name will be replaced with the date of the file was posted
- ```Add date/time to file name``` - the date will be appended to the file name
- `Replace file name by date` - the file name will be replaced with the date of the file was posted
- `Add date/time to file name` - the date will be appended to the file name
- Parameters:
- ```Date``` - append date to file name
- ```Time``` - append time to file name
- Date positions ```Start/End``` - date and/or time will be appended to the end or beginning of the file name
- ```Script``` - [script](#how-to-use-the-script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the ```Use script``` option.
- ```After download cmd``` - this [command](#scrawler-script-text-examples) will be executed after all downloads are completed
- ```Add missing information to log``` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing-posts) exist.
- ```Add missing errors to log``` - Each error that prevents SCrawler from downloading a file will be added to the log.
- ```Trying to download missing posts using regular download``` - If missing posts exist, the missing posts will attempt to be downloaded via user download.
- `Date` - append date to file name
- `Time` - append time to file name
- Date positions `Start/End` - date and/or time will be appended to the end or beginning of the file name
- `Script` - [script](#how%20to%20use%20the%20script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the `Use script` option.
- `After download cmd` - this [command](#scrawler%20script%20text%20examples) will be executed after all downloads are completed
- `Add missing information to log` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing%20posts) exist.
- `Add missing errors to log` - Each error that prevents SCrawler from downloading a file will be added to the log.
- `Trying to download missing posts using regular download` - If missing posts exist, the missing posts will attempt to be downloaded via user download.
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
![Downloading settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalDownloading.png)
## Channels
- ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
- ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
- ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
- `Channels rows/columns` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- `Download limit for channel user` - the amount of media will be downloaded if the user added from the channel.
- `Copy channel user image` - the image posted by user in the channel will be copied to the user folder when user will be created.
- `Create temporary users` - users will be created marked 'Temporary' when created from channel.
![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalChannels.png)
![Channels settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalChannels.png)
## Feed
- ```Feed rows/columns``` - how many rows and columns will be displayed in the feed form
- ```Center images in grid (number of visible images)``` - don't fit images to the grid, but center them and set the number of visible images (only works when the number of columns is 1)
- ```Feed colors``` - set the background color and font color of the feed window (`F` button to select font color; `C` button to select background color)
- ```Endless feed``` - go to the next page when you reach the end
- ```Add the session number to the post title``` - The session number will be displayed to the left of the post title. A session is every time when you (or [AutoDownloader](#automation)) download data.
- ```Add the date to the post title``` - The download date and time will be displayed to the right of the post title.
- ```Store session data``` - If checked, session data will be stored in an xml file.
- `Feed rows/columns` - how many rows and columns will be displayed in the feed form
- `Center images in grid (number of visible images)` - don't fit images to the grid, but center them and set the number of visible images (only works when the number of columns is 1)
- `Feed colors` - set the background color and font color of the feed window (`F` button to select font color; `C` button to select background color)
- `Endless feed` - go to the next page when you reach the end
- `Add the session number to the post title` - The session number will be displayed to the left of the post title. A session is every time when you (or [AutoDownloader](#automation)) download data.
- `Add the date to the post title` - The download date and time will be displayed to the right of the post title.
- `Store session data` - If checked, session data will be stored in an xml file.
![Feed settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalFeed.png)
![Feed settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalFeed.png)
# Site settings
If you see something like `Jobs <number>` in the progress bar, it means that SCrawler is still collecting information. In some cases, SCrawler takes some time to collect information for downloading files.
## Sites requirements
- [Reddit](#reddit-requirements)
- [Twitter](#twitter-requirements)
- [Instagram](#instagram-requirements)
- [RedGifs](#reddit-requirements)
- [TikTok](#tiktok-requirements)
- [PornHub](#pornhub-requirements)
- [XHamster](#xhamster-requirements)
- [XVIDEOS](#xvideos-requirements)
- [LPSG](#lpsg-requirements)
- [Reddit](#reddit%20requirements)
- [Twitter](#twitter%20requirements)
- [Mastodon](#Mastodon%20requirements)
- [Instagram](#instagram%20requirements)
- [RedGifs](#reddit%20requirements)
- [YouTube](#YouTube%20requirements)
- [Pinterest](#Pinterest%20requirements)
- [TikTok](#tiktok%20requirements)
- [PornHub](#pornhub%20requirements)
- [XHamster](#xhamster%20requirements)
- [XVIDEOS](#xvideos%20requirements)
- [ThisVid](#ThisVid%20requirements)
- [LPSG](#lpsg%20requirements)
## Site default fields
- ```Path``` - where users' data will be placed.
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. *Saved Twitter posts are posts that you have bookmarked.*
- ```Cookies``` add [cookies](#how-to-set-up-cookies) from your browser (click the pencil button to open the cookies editor).
- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- ```Get user media only``` - the default for creating new users
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- `Path` - where users' data will be placed.
- `Saved posts path` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. *Saved Twitter posts are posts that you have bookmarked.*
- `Cookies` add [cookies](#how%20to%20set%20up%20cookies) from your browser (click the pencil button to open the cookies editor).
- `Download site data` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- `Get user media only` - the default for creating new users
- `Temporary`, `Download images/videos` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the `Defaults` tab. If the checkbox is in an intermediate state, then the same name setting of the `Defaults` tab will be used instead. Otherwise, this setting will be applied.
## ffmpeg
Version **5.1.2-full_build-www.gyan.dev**
ffmpeg is required for several sites like Reddit, PornHub, XHamster, XVIDEOS, etc. **It is included in the SCrawler release, but you can download ffmpeg from:**
- x64 version - [release](https://github.com/GyanD/codexffmpeg/releases/tag/5.1.2); [zip](https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build.zip)
- x86 version - [release](https://github.com/yt-dlp/FFmpeg-Builds/releases/tag/autobuild-2022-11-30-12-57); [zip](https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2022-11-30-12-57/ffmpeg-N-109274-gd7a5f068c2-win32-gpl.zip)
## Gallery-dl
Version **1.25.2** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1.25.2); [exe](https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build.zip))
## YT-DLP
Version **2023.03.04** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2023.03.04))
- x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.03.04/yt-dlp.exe)
- x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.03.04/yt-dlp_x86.exe)
## Reddit
The Reddit parser can parse data without cookies, but you can add it if you like.
- ```Use M3U8``` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
- `Use M3U8` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
- `Saved posts user` - your personal Reddit username to download your saved posts (this feature requires cookies)
![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
![Reddit settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteReddit.png)
### Reddit requirements
- [Cookies](#how-to-set-up-cookies) for downloading saved posts
- [Cookies](#how%20to%20set%20up%20cookies) for downloading saved posts
- [ffmpeg](#ffmpeg) for downloading videos hosted on reddit
- [Imgur client ID](#how-to-find-imgur-client-id) to download Imgur content posted on reddit
- [RedGifs credentials](#redgifs-requirements) to download RedGifs content posted on reddit
- [Imgur client ID](#how%20to%20find%20imgur%20client%20id) to download Imgur content posted on reddit
- [RedGifs credentials](#redgifs%20requirements) to download RedGifs content posted on reddit
### Reddit user settings
- ```View``` - works the same as Reddit view modes (```new```, ```hot```, ```top```)
- ```Period``` - only works with ```Top``` view mode and the same as Reddit periods
- `View` - works the same as Reddit view modes (`new`, `hot`, `top`)
- `Period` - only works with `Top` view mode and the same as Reddit periods
![Reddit user settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsRedditView.png)
![Reddit user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsRedditView.png)
## Twitter
- [```Authorization```](#how-to-find-twitter-tokens) - Bearer token. Must start with ```Bearer ``` word followed by a space.
- [```Token```](#how-to-find-twitter-tokens) - in this field you need to put ```x-csrf-token```.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (bookmarked).
- ```Download GIFs``` - *(default for new users) this can also be configured for a specific user.*
- ```GIFs special folder``` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: ```SomeFolderName```, ```SomeFolderName\SomeFolderName2```). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- ```GIF prefix``` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
- ```Use MD5 comparison``` - *(default for new users)* each image will be checked for existence using MD5 (this may be suitable for the users who post the same image many times). *This can also be configured for a specific user.*
- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`.
- `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
- `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
- `Use MD5 comparison` - *(default for new users)* each image will be checked for existence using MD5 (this may be suitable for the users who post the same image many times). *This can also be configured for a specific user.*
![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTwitter.png)
![Twitter settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTwitter.png)
### Twitter requirements
- [```Cookies```](#how-to-set-up-cookies)
- [```Authorization```](#how-to-find-twitter-tokens)
- [```Token```](#how-to-find-twitter-tokens)
- ```Saved posts user``` for download saved posts (bookmarked)
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [`Authorization`](#how%20to%20find%20twitter%20tokens)
- [`Token`](#how%20to%20find%20twitter%20tokens)
- [`yt-dlp`](#YT-DLP) for download saved posts (bookmarked)
### Twitter user settings
- ```Download GIFs``` - *same as default twitter settings, but for this user*
- ```GIFs special folder``` - *same as default twitter settings, but for this user*
- ```GIF prefix``` - *same as default twitter settings, but for this user*
- ```Use MD5 comparison``` - *same as default twitter settings, but for this user*
- ```Remove existing duplicates``` - Existing files will be checked for duplicates and duplicates removed. **Works only on the first activation 'Use MD5 comparison'.**
- `Download GIFs` - *same as default twitter settings, but for this user*
- `GIFs special folder` - *same as default twitter settings, but for this user*
- `GIF prefix` - *same as default twitter settings, but for this user*
- `Use MD5 comparison` - *same as default twitter settings, but for this user*
- `Remove existing duplicates` - Existing files will be checked for duplicates and duplicates removed. **Works only on the first activation 'Use MD5 comparison'.**
![Twitter user settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsTwitterUser.png)
![Twitter user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsTwitterUser.png)
### How to find Twitter tokens
1. Open Google Chrome, Edge or FireFox.
1. Go to https://twitter.com
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl+Shift+I```).
1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
1. Go to the network tab
1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**)
1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with ```TweetDetail?```... and click on it
1. Scroll down to ```Request Headers``` then find the fields below:
- ```authorization: Bearer .......``` - this is the authorization token. You need to copy the text from starting with the word ```Bearer``` to the end of the line and paste it into the ```Authorization``` field on the Twitter settings form.
- ```x-csrf-token: ...............``` - this is csrf token. You need to copy all the text after the words ```x-csrf-token:``` and paste it into the ```Token``` field on the Twitter settings form
1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with `TweetDetail?`... and click on it
1. Scroll down to `Request Headers` then find the fields below:
- `authorization: Bearer .......` - this is the authorization token. You need to copy the text from starting with the word `Bearer` to the end of the line and paste it into the `Authorization` field on the Twitter settings form.
- `x-csrf-token: ...............` - this is csrf token. You need to copy all the text after the words `x-csrf-token:` and paste it into the `Token` field on the Twitter settings form
## Mastodon
- `My Domain` - your account domain without `https://` (for example, `mastodon.social`)
- [`Authorization`](#how%20to%20find%20twitter%20tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how%20to%20find%20twitter%20tokens) - in this field you need to put `x-csrf-token`.
- `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
- `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
- `Use MD5 comparison` - *(default for new users)* each image will be checked for existence using MD5 (this may be suitable for the users who post the same image many times). *This can also be configured for a specific user.*
- `User related to my domain` - open user profiles and user posts through my domain.
![Mastodon settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteMastodon.png)
### Mastodon requirements
- `My Domain`
- [`Authorization`](#how%20to%20find%20twitter%20tokens)
- [`Token`](#how%20to%20find%20twitter%20tokens)
- [`Authorization`](#how%20to%20find%20twitter%20tokens) and [`Token`](#how%20to%20find%20twitter%20tokens) for each domain you want to download from (see [additional settings](#Mastodon%20additional%20setting))
### Mastodon additional setting
- `Domains` - select or add a new domain for which you want to add authorization
- `Auth` - Bearer token. Must start with `Bearer ` word followed by a space.
- `Token` - in this field you need to put `x-csrf-token`.
![Mastodon additional settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteMastodonAdditional.png)
## Instagram
- Authorization
- ```Hash``` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)).
- ```x-csrftoken``` - [how to find](#how-to-find-instagram-authorization-headers)
- ```x-ig-app-id```, ```ix-ig-www-claim``` - [how to find](#how-to-find-instagram-authorization-headers)
- ```Saved posts user``` - your personal Instagram username to download your saved posts
- `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how%20to%20find%20instagram%20hash)).
- `x-csrftoken` - [how to find](#how%20to%20find%20instagram%20authorization%20headers)
- `x-ig-app-id`, `ix-ig-www-claim` - [how to find](#how%20to%20find%20instagram%20authorization%20headers)
- `Saved posts user` - your personal Instagram username to download your saved posts
- Other parameters
- ```Download timeline``` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- ```Download stories``` - Download stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- ```Download tagged``` - Download tagged posts *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
- ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
- ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- ```Get timeline``` - default value for new users
- ```Get stories``` - default value for new users
- ```Get tagged photos``` - default value for new users
- ```Tagged notify limit``` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit))
- `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download stories` - Download stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download tagged` - Download tagged posts *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Request timer` - this is the time value (in milliseconds) the program will wait before processing the next `Request time counter` request (**it is highly recommended not to change this default value**)
- `Request time counter` - how many requests will be sent to Instagram before the program waits `Request timer` milliseconds (**it is highly recommended not to change this default value**)
- `Post limit timer` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- `Get timeline` - default value for new users
- `Get stories` - default value for new users
- `Get tagged photos` - default value for new users
- `Tagged notify limit` - Limit of new tagged posts when you receive a notification (read more [here](#instagram%20tagged%20posts%20limit))
![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
![Instagram settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteInstagram.png)
### Instagram requirements
- **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
- [```Hash```](#how-to-find-instagram-hash)
- [```x-csrftoken```](#how-to-find-instagram-authorization-headers)
- [```x-ig-app-id```](#how-to-find-instagram-authorization-headers)
- [```ix-ig-www-claim```](#how-to-find-instagram-authorization-headers)
- **[COOKIES](#how%20to%20set%20up%20cookies) ARE REQUIRED ANYWAY**
- [`Hash`](#how%20to%20find%20instagram%20hash) *(only for downloading tagged posts)*
- [`x-csrftoken`](#how%20to%20find%20instagram%20authorization%20headers)
- [`x-ig-app-id`](#how%20to%20find%20instagram%20authorization%20headers)
- [`ix-ig-www-claim`](#how%20to%20find%20instagram%20authorization%20headers)
To download tagged posts you also need [```Hash```](#how-to-find-instagram-hash)
To download tagged posts you also need [`Hash`](#how%20to%20find%20instagram%20hash)
### How to find Instagram Hash
1. Open browser developer tools
1. Go to ```Network```
1. Go to `Network`
1. If this tab contain any data, clear it.
1. Open any profile and go to the `Tagged` section for ```Hash```.
1. Click ```Ctrl+F``` and find the text ```query_hash=```. The alphanumeric value after the equal sign is the required hash.
- You can find two different hashes. The first is good, the second is wrong.
1. Open any profile and go to the `Tagged` section for `Hash`.
1. Click `Ctrl+F` and find the text `query_hash=`. The alphanumeric value after the equal sign is the required hash.
- **The hash is the value after the equal sign, 32 characters long.** You can find two different hashes. The first (`abcde12345abcde12345abcde12345ab`) is good, the second (`54321edcba54321edcba54321edcba54`) is wrong.
- https://www.instagram.com/graphql/query/?query_hash=abcde12345abcde12345abcde12345ab&variables={"id":"123456789","first":12}
- https://www.instagram.com/graphql/query/?query_hash=54321edcba54321edcba54321edcba54&variables={"user_id":"123456789","include_chaining":true,"include_reel":false,"include_suggested_users":false,"include_logged_out_extras":false,"include_highlight_reels":true,"include_live_status":true}
1. Copy this and paste it into the corresponding (```Hash```) field on the Instagram settings form.
1. Copy this and paste it into the corresponding (`Hash`) field on the Instagram settings form.
**If you don't see the required hash, refresh the page several times or go to the main page and return to the profile until you see the hash you need.**
@@ -220,72 +286,157 @@ To download tagged posts you also need [```Hash```](#how-to-find-instagram-hash)
### How to find Instagram authorization headers
1. Open browser developer tools
1. Go to ```Network```
1. Go to `Network`
1. If this tab contain any data, clear it.
1. Open profile tagged photos.
1. Click ```Ctrl+F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
1. Click `Ctrl+F` and find the text `x-ig-app-id` and `x-ig-www-claim`.
1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
### Instagram limits
Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. This was added to bypass error 429 and prevent account ban.
**After 195 requests Instagram downloader becomes much slower.** Slowdown can be configured in the Instagram [settings](#Instagram) (`Request timer`, `Request time counter`, `Post limit timer`). Change these parameters at your own risk.
### Instagram tagged posts limit
When the number of tagged posts exceeds the tagged posts limit, you will be asked what to do about it. This message box protects you from account ban and long wait times.
**Instagram tagged posts no longer provide the total amount of tagged posts. I've corrected the tagged posts notification, but now I can't tell how many requests will be spent on downloading tagged posts. And from now on, one request will be spent on downloading each tagged post, because Instagram doesn't provide complete information about the tagged post with the site's response. In this case, if the number of tagged posts is 1000, 1000 requests will be spent. Be careful when downloading them. I highly recommend that you forcefully disable the downloading of tagged posts for a while (`Settings-Instagram-Download tagged`).**
**ATTENTION. Change the default parameters at your own risk. If you do this, I don't guarantee that your account will not be banned.**
- ```Continue``` - Continue downloading
- ```Continue unnotified``` - Continue downloading and cancel further notifications in the current downloading session
- ```Limit``` - Enter the limit of posts you want to download
- ```Confirm``` - Confirm the number you entered
- ```Try again``` - You will be asked again about the limit
- ```Other options``` - The main message with options will be displayed again
- ```Cancel``` - Cancel tagged posts download operation
- ```Disable and cancel``` - Disable downloading tagged data and cancel downloading tagged data
- ```Cancel``` - Cancel tagged posts download operation
- `Continue` - Continue downloading
- `Continue unnotified` - Continue downloading and cancel further notifications in the current downloading session
- `Limit` - Enter the limit of posts you want to download
- `Confirm` - Confirm the number you entered
- `Try again` - You will be asked again about the limit
- `Other options` - The main message with options will be displayed again
- `Cancel` - Cancel tagged posts download operation
- `Disable and cancel` - Disable downloading tagged data and cancel downloading tagged data
- `Cancel` - Cancel tagged posts download operation
```mermaid
stateDiagram
state "Message" as Start
state "Continue" as btt1_1
state "Continue unnotified" as btt1_2
state "Limit" as btt1_3
state "Disable and cancel" as btt1_4
state "Cancel" as btt1_5
state "Confirm" as btt2_1
state "Try again" as btt2_2
state "Other options" as btt2_3
state "Cancel" as btt2_4
state "Continue download" as result_continue
state "Cancel download" as result_cancel
state "Enter the number of posts from user that you want to download" as opt1
[*]-->Start
Start-->btt1_1
Start-->btt1_2
Start-->btt1_3
Start-->btt1_4
Start-->btt1_5
btt1_1-->result_continue
btt1_2-->result_continue
note left of btt1_2
Continue downloading and cancel further
notifications in the current downloading session.
end note
btt1_3-->opt1
btt1_4-->result_cancel
note right of btt1_4
Disable downloading tagged data for this user
and cancel downloading tagged data.
end note
btt1_5-->result_cancel
opt1-->btt2_1
opt1-->btt2_2
opt1-->btt2_3
opt1-->btt2_4
btt2_1-->result_continue
btt2_2-->opt1
btt2_3-->Start
btt2_4-->result_cancel
result_continue-->[*]
result_cancel-->[*]
```
## RedGifs
- ```Token refresh interval``` - Interval (in minutes) to refresh the token
- ```Token``` - "Bearer" token ([how to find](#how-to-find-redgifs-token))
- `Token refresh interval` - Interval (in minutes) to refresh the token
- `Token` - "Bearer" token ([how to find](#how%20to%20find%20redgifs%20token))
- `UserAgent` - UserAgent to use in requests ([how to find](#how%20to%20find%20useragent))
![RedGifs settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteRedGifs.png)
![RedGifs settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteRedGifs.png)
### RedGifs requirements
- [```Token```](#how-to-find-redgifs-token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.*
- [`Token`](#How%20to%20find%20RedGifs%20token) - click on the curved arrows to refresh the token. Updated automatically. *You don't need to set it manually.*
### How to find RedGifs token
1. Open browser developer tools
1. Go to ```Network```
1. Go to `Network`
1. If this tab contain any data, clear it.
1. Open any RedGifs profile.
1. Click ```Ctrl+F``` and find the text ```Bearer```.
1. Copy the value and paste it into the ```Token``` field on the RedGifs settings form.
1. Click `Ctrl+F` and find the text `Bearer`.
1. Copy the value and paste it into the `Token` field on the RedGifs settings form.
## YouTube
- `Download user videos`
- `Download user shorts`
- `Download user playlists`
- `Use cookies` - default value for new users. Use cookies to download the user.
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader#Settings).
![YouTube settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteYouTube.png)
### YouTube requirements
- [`Cookies`](#how%20to%20set%20up%20cookies) for private data
- [**yt-dlp**](#YT-DLP)
- [**ffmpeg**](#ffmpeg)
## Pinterest
- `Concurrent downloads` - the number of concurrent downloads.
- `Saved posts user` - personal profile username.
![Pinterest settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSitePinterest.png)
### Pinterest requirements
- [`Cookies`](#how%20to%20set%20up%20cookies) for private data
- [**gallery-dl**](#Gallery-dl)
## TikTok
![TikTok settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTikTok.png)
![TikTok settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTikTok.png)
### TikTok requirements
- [```Cookies```](#how-to-set-up-cookies)
- [`Cookies`](#how%20to%20set%20up%20cookies)
### TikTok limits
Unfortunately, I didn't understand how to use the 'Cursor' option to get the JSON response. In this case, I can only parse an HTML page that only provides the latest 30 videos. **If you know how to use the cursor option to get a JSON response, you can post with your suggestion [here](https://github.com/AAndyProgram/SCrawler/issues/75).**
## PornHub
- ```Download GIF``` - default for new users.
- ```Download GIFs as mp4``` - download gifs in 'mp4' format instead of native 'webm'.
- ```Photo ModelHub only``` - download photo only from ModelHub. Prornstar photos hosted on PornHub itself will not be downloaded. **Attention! Downloading photos hosted on PornHub is a very heavy job.**
- ```Saved posts user``` - personal profile username (to download saved posts)
- `Download GIF` - default for new users.
- `Download GIFs as mp4` - download gifs in 'mp4' format instead of native 'webm'.
- `Photo ModelHub only` - download photo only from ModelHub. Prornstar photos hosted on PornHub itself will not be downloaded. **Attention! Downloading photos hosted on PornHub is a very heavy job.**
- `Saved posts user` - personal profile username (to download saved posts)
![PornHub settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSitePornHub.png)
![PornHub settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSitePornHub.png)
### PornHub requirements
- [ffmpeg](#ffmpeg) is required anyway
- [```Cookies```](#how-to-set-up-cookies) for downloading private videos and saved posts
- ```Saved posts user``` for downloading saved posts
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts
- `Saved posts user` for downloading saved posts
### PornHub additional information
@@ -294,123 +445,145 @@ About videos. PornHub has at least three page views (as I've seen). If you find
About photos. Photo download problems have the lowest priority. You can still create an issue including the account address where photos are not downloading, but I can't tell you when I will have time to fix it.
## XHamster
- ```Download UHD``` - Download UHD (4K) content
- `Download UHD` - Download UHD (4K) content
![XHamster settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteXHamster.png)
![XHamster settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteXHamster.png)
### XHamster requirements
- [ffmpeg](#ffmpeg) is required anyway
- [```Cookies```](#how-to-set-up-cookies) for downloading saved posts
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading saved posts
## XVIDEOS
- ```Download UHD``` - Download UHD (4K) content.
- ```Playlist of saved videos``` - Your personal videos playlist to download as 'saved posts'. This playlist must be private (Visibility = ```Only me```). It also required cookies. This playlist must be entered by pattern: ```https://www.xvideos.com/favorite/01234567/playlistname```.
- `Download UHD` - Download UHD (4K) content.
- `Playlist of saved videos` - Your personal videos playlist to download as 'saved posts'. This playlist must be private (Visibility = `Only me`). It also required cookies. This playlist must be entered by pattern: `https://www.xvideos.com/favorite/01234567/playlistname`.
![XVIDEOS settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteXvideos.png)
![XVIDEOS settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteXvideos.png)
### XVIDEOS requirements
- [ffmpeg](#ffmpeg) is required anyway
- [```Cookies```](#how-to-set-up-cookies) for downloading private videos and saved posts
- ```Playlist of saved videos``` for downloading saved posts
- [`Cookies`](#how%20to%20set%20up%20cookies) for downloading private videos and saved posts
- `Playlist of saved videos` for downloading saved posts
## ThisVid
- `Public videos` - download public videos.
- `Private videos` - download private videos.
- `Different folders` - Use different folders to store video files. If `true`, then public videos will be stored in the `Public` folder, private - in the `Private` folder. If `false`, all videos will be stored in the `Video` folder.
![ThisVid settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteThisVid.png)
### ThisVid requirements
- [`Cookies`](#how%20to%20set%20up%20cookies)
- [**yt-dlp**](#YT-DLP)
## LPSG
![LPSG settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteLPSG.png)
![LPSG settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteLPSG.png)
### LPSG requirements
- [```Cookies```](#how-to-set-up-cookies)
- [`Cookies`](#how%20to%20set%20up%20cookies)
# Download groups
In many cases, you may need to download some users. You can group these users and make it easier to download them without having to select them every time.
It is very easy to create a new group. Just click the ```Add a new download group``` button (main window - ```Download all``` menu) to create a new one.
It is very easy to create a new group. Just click the `Add a new download group` button (main window - `Download all` menu) to create a new one.
- ```Name``` - group name
- ```Temporary``` - users marked as temporary (indeterminate state to not use this option) will be downloaded
- ```Favorite``` - users marked as favorite (indeterminate state to not use this option) will be downloaded
- ```Ready for download``` - users marked as ```Ready for download``` will be downloaded
- ```Ignore ready for download``` - this option tells the program to ignore the ```Ready for download``` user option and download the user anyway
- ```Labels``` - You can select labels.
- ```Pencil``` button for selected labels. Only users who have one or more of these labels will be downloaded.
- ```Red X``` button for excluded labels. Users who have one or more of these labels will be excluded from the download.
- ```Sites``` - You can select sites.
- ```Pencil``` button for selected sites. Only users from the sites you have selected will be downloaded.
- ```Red X``` button for excluded sites. Users from the sites you have selected will be excluded from the download.
- `Name` - group name
- `Temporary` - users marked as temporary (indeterminate state to not use this option) will be downloaded
- `Favorite` - users marked as favorite (indeterminate state to not use this option) will be downloaded
- `Ready for download` - users marked as `Ready for download` will be downloaded
- `Ignore ready for download` - this option tells the program to ignore the `Ready for download` user option and download the user anyway
- `Labels` - You can select labels.
- `Pencil` button for selected labels. Only users who have one or more of these labels will be downloaded.
- `Red X` button for excluded labels. Users who have one or more of these labels will be excluded from the download.
- `Sites` - You can select sites.
- `Pencil` button for selected sites. Only users from the sites you have selected will be downloaded.
- `Red X` button for excluded sites. Users from the sites you have selected will be excluded from the download.
You can set both selected and excluded options for each selection.
**Only those users who match all of these parameters (logical operator ```AND```) will be downloaded.**
**Only those users who match all of these parameters (logical operator `AND`) will be downloaded.**
![Group creator](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/GroupCreating.png)
![Group creator](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/GroupCreating.png)
For each group, SCrawler creates a new menu, which is placed in the ```Download all``` menu of the main window. If a group has a number on the left (1-9), that group can be downloaded using ```Ctrl```+```Number```. Each group also has several options:
- ```Edit``` - edit group
- ```Delete``` - delete group
- ```Download``` - download with the options you have set
- ```Download FULL``` - Download with the options you have set. ```Ready for download``` option will be ignored.
For each group, SCrawler creates a new menu, which is placed in the `Download all` menu of the main window. If a group has a number on the left (1-9), that group can be downloaded using `Ctrl`+`Number`. Each group also has several options:
- `Edit` - edit group
- `Delete` - delete group
- `Download` - download with the options you have set
- `Download FULL` - Download with the options you have set. `Ready for download` option will be ignored.
![Groups context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
![Groups context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowGroups.png)
# Automation
You can set up automatic downloads. You can find these settings in ```Settings-Automation``` or by clicking the ```Automation``` button in the ```Download all``` drop-down list in the main window.
You can set up automatic downloads. You can find these settings in `Settings-Automation` or by clicking the `Automation` button in the `Download all` drop-down list in the main window.
![Groups context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
![Groups context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowGroups.png)
**Scheduler**:
- ```Add```, ```Edit```, ```Delete``` - operations with the selected plan;
- ```Update``` - refresh list;
- ```Start``` - run the created (stopped) plan;
- ```Skip``` - skip next run.
- `Add`, `Edit`, `Delete` - operations with the selected plan;
- `Update` - refresh list;
- `Start` - run the created (stopped) plan;
- `Skip` - skip next run.
![Scheduler](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsScheduler.png)
![Scheduler](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsScheduler.png)
**Plan**:
- ```Disable``` - Disable automatic downloads.
- ```All``` - Download all existing user profiles (```Ready for download``` is not respected).
- ```Default``` - Download all existing user profiles marked ```Ready for download``` (equivalent to ```Download all```).
- ```Specified``` - Specify custom download options (read more [here](#download-groups)).
- ```Groups``` - Automatic downloading of one or more existing groups (read more [here](#download-groups)).
- ```Show notifications``` - Show notification when there is new downloaded data.
- ```Simple``` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored.
- ```Image``` - Show downloaded image in notification.
- ```User icon``` - Show user icon in notification.
- ```Timer``` - Download timer (in minutes).
- ```Delay``` - Plan launch delay when SCrawler starts.
- `Disable` - Disable automatic downloads.
- `All` - Download all existing user profiles (`Ready for download` is not respected).
- `Default` - Download all existing user profiles marked `Ready for download` (equivalent to `Download all`).
- `Specified` - Specify custom download options (read more [here](#download%20groups)).
- `Groups` - Automatic downloading of one or more existing groups (read more [here](#download%20groups)).
- `Show notifications` - Show notification when there is new downloaded data.
- `Simple` - Show a simple notification instead of a user notification. This means that if any user data has been downloaded with the plan, a simple notification will be shown with the number of users downloaded. The 'Image' and 'User icon' parameters will be ignored.
- `Image` - Show downloaded image in notification.
- `User icon` - Show user icon in notification.
- `Timer` - Download timer (in minutes).
- `Delay` - Plan launch delay when SCrawler starts.
For ```All```, ```Default```, and ```Specified``` modes, labels and sites can be configured to exclude them.
For `All`, `Default`, and `Specified` modes, labels and sites can be configured to exclude them.
![Automation settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsAutoDownloader.png)
![Automation settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsAutoDownloader.png)
# Folder command
You can specify a command to open a folder in a special program. Pattern: ```Command "{0}"```. ```"{0}"``` is the path argument. I like to use total commander and I want to open folders in this program. Example: ```D:\TotalCMD\TOTALCMD64.EXE /o "{0}"```.
You can specify a command to open a folder in a special program. Pattern: `Command "{0}"`. `"{0}"` is the path argument. I like to use total commander and I want to open folders in this program. Example: `D:\TotalCMD\TOTALCMD64.EXE /o "{0}"`.
```"{0}"``` - required argument!
`"{0}"` - required argument!
This command can be a batch command or any script file (bat, ps1 or whatever you want) with the incoming argument as a folder path.
# How to set up cookies
## First method
1. Using a browser extension to receive site cookies, copy the cookies (or save them to a file).
1. Open the settings of the site you want to configure (for example, `Main window` - `Settings` - `Instagram`).
1. Find the `Cookies` field.
1. Click on the `Pencil` button.
1. In the cookies editor that opens, click the `Import cookies from Netscape file` button or the `Import cookies from JSON file` button if your cookies are in JSON format.
1. Paste the copied cookies text into the window that opens (or press `Ctrl+O` to load cookies from a saved file (if you saved cookies to a file)) and click `OK`.
1. Click OK to close the cookies editor and save the cookies.
## Second method
1. Open Google Chrome, Microsoft Edge or FireFox.
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl+Shift+I```).
1. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
1. Copy all text **with mouse**.
1. Open the settings of the site you want to configure (for example, ```Main window``` - ```Settings``` - ```Instagram```).
1. Find the ```Cookies``` field.
1. Click on the ```Pencil``` button.
1. In the cookies editor that opens, click the ```From Google Chrome``` button.
1. Paste the copied cookies text into the opened window and click the ```OK``` button.
1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
1. In the opened window, go to `Application` - `Storage` - `Cookies`.
1. Copy all text **using mouse**.
1. Open the settings of the site you want to configure (for example, `Main window` - `Settings` - `Instagram`).
1. Find the `Cookies` field.
1. Click on the `Pencil` button.
1. In the cookies editor that opens, click the `From Google Chrome` button.
1. Paste the copied cookies text into the window that opens and click `OK`.
1. Click OK to close the cookies editor and save the cookies.
# How to find Imgur client ID
1. Open browser developer tools
1. Go to ```Network```
1. Go to `Network`
1. Go to https://imgur.com/
1. In the list on the left, find a query that contains ```client_id``` and click on it
1. Copy the client id from the ```Request URL```
1. In the list on the left, find a query that contains `client_id` and click on it
1. Copy the client id from the `Request URL`
1. Open settings
1. Paste the copied value into the ```Imgur Clien ID``` field
1. Paste the copied value into the `Imgur Clien ID` field
# How to find UserAgent

60
Users.md

@@ -1,60 +0,0 @@
# Creating user
In the main window, click the ```Add``` button to open the user creation form.
![Add user](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/CreateUserClear.png)
- ```User name``` - you can specify only a username or a link by pattern:
- Twitter:
- https://twitter.com/SomeUserName
- Reddit:
- https://reddit.com/user/SomeUserName
- https://reddit.com/u/SomeUserName
- https://reddit.com/r/SomeSubredditName
- Instagram
- https://www.instagram.com/SomeUserName
- RedGifs
- https://www.redgifs.com/users/SomeUserName
- XVIDEOS
- https://www.xvideos.com/Account-Type/SomeUserName
- Account-Type/SomeUserName
- PornHub
- https://www.pornhub.com/users/UserName
- XHamster
- https://xhamster.com/users/UserName
- Path
- `D:\Data\`
- ```Friendly name``` - if configured, this name will be displayed in the user list instead of the username (if this field is green, it means that the user site name is shown). `F` button to set the friendly name you configured; `S` button to set site name.
- ```Special path``` - specify a path other than the default path for user data.
- ```Sites combo box``` - site pointers.
- ```Channel``` - profile is subreddit.
- ```Options``` - additional user options
- ```Temporary``` - mark user as ```Temporary```.
- ```Favorite``` - mark user as ```Favorite```.
- ```Download images/videos``` - for the created user, only the selected media types will be downloaded.
- ```Ready for download``` - download this user in the operation ```Download all```.
- ```Get user media only``` - this is an option for twitter only. Download all posts posted by this author.
- ```Add by list``` - add users' profiles by list (new line as separator (```Shift+Enter``` to insert new line)).
- If you add users by the URL, you must check the ```Auto detect site``` checkbox;
- If you are adding users by username, you must select the site in the combo box.
- ```Auto detect site``` - add users by the URLs.
- ```Description``` - some information you want to keep. If the ```Add by list``` checkbox is checked, you must enter a list of users in this textbox (```Shift+Enter``` to insert new line);
- ```Labels``` - labels associated with this user (used to filter users in the main window).
- ```Script``` - If the checkbox is checked, script to be executed after the user download is complete. If the text is empty, the script from the global settings will be executed. [How to use the script](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-use-the-script).
If neither ```Temporary``` nor ```Favorite``` is checked, the user will be displayed in the regular site group.
**In collection editing mode checkboxes have three states. When loaded, an indeterminate state means that profiles have different values for this option. When you save, checkboxes that are in indeterminate state will not be applied.**
# Labels
In this form, you can create labels and select them for users.
Click the ```+``` button to add a new one.
You can pick up one or more labels in this form.
![Labels Form](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/Labels.png)
# Hot keys
- ```F2``` - open advanced options
- ```F4``` - open labels form for pick up a label

97
YouTube downloader.md Normal file

@@ -0,0 +1,97 @@
This is a standalone application. It works separately from SCrawler.
- `Settings`
- `Add`
- `Add`
- `Add playlist array`
- `Add (without shorts)` - download all videos except `Shorts`
- `Add (shorts only)` - download only `Shorts` videos.
- `Download` - start download
- `Stop` - stop download
- `Delete` - delete selected items (with files)
- `Clear` - remove all downloaded items (without files)
- `Clear all` - remove all items (without files)
- `LOG` - error log
![YouTube app main window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTube.png)
# Hot keys
- `Insert` - add to download
- `Ctrl+Insert` - add to download (using cookies)
# Video
The program automatically detects what you adding to. When you add a video, the following form will open. The download parameters are set automatically according to the [settings](#Settings) you have specified.
You can change the setting of the video you want to download.
1. Choose what you want to download: video or extract audio.
2. Select output audio/video format
3. You can choose a different audio codec if you like.
4. You can choose which subtitles you want to download from the list of available subtitles for a particular video. If the list is empty or the `Subtitles` text box is inactive, then there are no subtitles available for the video.
5. You can also change the subtitles output format in the `Format` combo box.
6. You can also select additional subtitles/audio formats in the corresponding fields. This means that all subtitles/audio will be extracted from the video, converted to the format you choose, and saved as a separate file.
7. You can also choose the destination of the output file.
8. Click `Download` to confirm your choices, or `Cancel` to cancel the download.
![YouTube video](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTubeVideo.png)
# Playlist
When you add a playlist, the following form will open. This form is similar to the video download form, but with a few differences.
- Instead of a list of resolution, you see a list of videos.
- All parameters will apply to all videos. The resolution value is the limit. If you set the resolution to `1080` and the video is only available in lower resolutions, the maximum resolution will be selected.
- Audio codec, video format and additional subtitles/audio will be applied to all videos.
- The list of available subtitles will be populated from the collected subtitles from all videos in the playlist. They will be downloaded if they are actually available for video.
![YouTube playlists](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTubePlaylist.png)
## Playlists parser
If you want to correctly download all albums of an artist, you can use a playlist parser.
**Example**.
I want to download this artist: https://music.youtube.com/channel/UC25tCnonOu_M3ojPEi57nWA
1. Open Chrome dev tools.
2. Go to the following URL: https://music.youtube.com/channel/UC25tCnonOu_M3ojPEi57nWA
3. To the right of the albums, click on the `More` button.
4. Scroll down to the bottom of the page.
5. In the dev tools, look for the last page that starts with `https://music.youtube.com/youtubei/v1/browse?key=..........`.
6. Open the `Response` tab.
7. Copy all text.
8. Open the YouTube app.
9. Click `Add playlist array`.
10. Press `Ctrl+O`.
11. Paste all the copied text into the `In` text box.
12. Use the `Remove` text box to remove all incorrect playlist names. In the current example, incorrect names start with `RDAMP`. You can get the correct name pattern by clicking on the playlist. In the `https://music.youtube.com/playlist?list=OLAK5uy_l9d-qJGn4l0z4t5aQXSYeJFsg7hJiQfGA` URL, playlist ID is `OLAK5uy_l9d-qJGn4l0z4t5aQXSYeJFsg7hJiQfGA`.
13. Click the `OK` button on both forms.
![YouTube playlist parser](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTubePlaylistParser.png)
# Music
This is a special form for music. When adding an artist or music album, this form opens. Similar to the video download form, you can choose audio options, download (or not) lyrics, audio formats, and additional audio/lyrics formats. You can also uncheck any albums and/or single tracks that you don't want to download.
![YouTube music](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTubeMusic.png)
# Settings
Here you can configure the default settings. I won't describe all the settings, because each setting is described in the bottom toolbar. Click on a setting and look at the bottom for what that setting means.
![YouTube settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTubeSettings.png)
# Dependencies
If you want to install the YouTube downloader in a separate directory, you can create a new folder and copy the following files there:
- Microsoft.Toolkit.Uwp.Notifications.dll
- PersonalUtilities.dll
- PersonalUtilities.Notifications.dll
- SCrawler.PluginProvider.dll
- SCrawler.YouTube.dll
- System.ValueTuple.dll
- ffmpeg.exe
- ffplay.exe
- ffprobe.exe
- yt-dlp.exe