2022.10.18.0

Home.md

@@ -6,8 +6,14 @@ This window contains all data management tools.
 
 - ```Add/Edit/Delete``` - actions with users.
 - ```Refresh``` - refresh the user list. You do not need to press this button. Added just in case...
-- ```Info``` - show downloaded users and what exactly and how many were downloaded.
+- ```Info``` - [show](#info) downloaded users and what exactly and how many were downloaded.
-- ```Feed``` - show recently downloaded data
+- ```Feed``` - [show](#feed) recently downloaded data
+- ```Channels``` - open [Channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels) form
+- ```Saved posts``` - download [saved posts](#saved-posts)
+- ```Download``` - [download](#download) operations
+- ```Stop``` - stop downloading
+- ```View``` - [change](#view) view mode
+- ```LOG``` - program error log and information
 
 ## Buttons
 
@@ -15,7 +21,7 @@ This window contains all data management tools.
 
 This form will show downloaded items (sorted by latest download date). Each line of data contains information about how many images and videos have been downloaded for the user. If no media is downloaded, the user will not appear in this form.
 
-*Right-clicking on this button opens the 'Missing posts' window.*
+*Right-clicking on this button opens the '[Missing posts](#missing-posts)' window.*
 
 Double-clicking on the data line will open the content folder.
 
@@ -37,39 +43,8 @@ Hot keys:
 
 ![Info window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/DownloadInfo.png)
 
-### Feed
-
-This form will show the recently downloaded data (images and videos). Each image/video contains download information (tooltip), file address, user information, etc.
-
-Double-clicking on an image will open the original image in an external viewer.
-
-Buttons:
-- ```Refresh``` - the feed will be updated to receive new downloaded items and/or remove missing items
-- ```Clear``` - clear list of recent downloads
-- ```Delete selected``` - delete marked items from the disk
-
-Context menu:
-- ```Open picture/video``` - open picture/video in external viewer
-- ```Open user``` (with folder image) - open user folder
-- ```Open user``` (with globe) - open user site
-- ```Open post``` - open post in browser
-- ```Find user``` - find user in the main window
-- ```Information``` - show post information
-- ```Delete picture/video``` - delete picture/video
-
-Hot keys
-- ```F5``` - ```Refresh```
-- ```F3``` - Previous page
-- ```F4``` - Next page
-
-You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
-
-![Feed window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindow.png)
-
 ### Missing posts
 
-**Still under development**
-
 This form shows posts that have not been downloaded. This form is opened by right-clicking on the '[Info](#info)' button in the main window
 
 Buttons:
@@ -87,10 +62,49 @@ Context menu:
 - ```Delete post``` - delete selected post
 
 Hot keys:
-- ```F1``` - ```Update```
+- ```F1``` - ```Info```
 
 ![Missing posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MissingPosts.png)
 
+### Feed
+
+This form will show the recently downloaded data (images and videos). Each image/video contains download information (tooltip), file address, user information, etc.
+
+Double-clicking on an image will open the original image in an external viewer.
+
+![Feed window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindow.png)
+
+Buttons:
+- ```Refresh``` - the feed will be updated to receive new downloaded items and/or remove missing items
+- ```Clear``` - clear list of recent downloads
+- ```Delete selected``` - delete marked items from the disk
+
+Context menu:
+- ```Open picture/video``` - open picture/video in external viewer
+- ```Open user``` (with folder image) - open user folder
+- ```Open user``` (with globe) - open user site
+- ```Open post``` - open post in browser
+- ```Find user``` - find user in the main window
+- ```Information``` - show post information
+- ```Delete picture/video``` - delete picture/video
+
+![Feed context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowItemContext.png)
+
+Load sessions:
+- ```Load last session``` - load last saved session data
+- ```Select loading session``` - Load the data of the session you have chosen (the session selection form will open)
+
+*The buttons are only displayed if the ```Store session data``` option is [enabled](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed)*
+
+![Feed session context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowSessionContext.png)
+
+Hot keys
+- ```F5``` - ```Refresh```
+- ```F3``` - Previous page
+- ```F4``` - Next page
+
+You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
+
 ### Channels
 
 The channels form will open.
@@ -99,34 +113,49 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel
 
 ### Saved posts
 
-**This function requires [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
-**[Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash) and cookies required to download saved Instagram posts**
+Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the ```Saved posts user``` textbox. Click OK. Go to the main window.
-
-Go to Settings - Settings - Reddit/Instagram/Twitter. Enter your username in the "Saved posts user" textbox. Click OK. Go to the main window.
 
 This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the ```!Saved posts``` folder.
 
 ![Saved posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SavedPosts.png)
 
 ### Download
-
+- ```Download selected (F5)``` - start downloading one or more selected users.
-- ```Download selected``` - start downloading one or more selected users.
+- ```Download all```: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!**
-- ```Download all```: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER OPTIONS.!**
+  - ```Download all (F6)``` - Download all users marked ```Ready for download``` from all sites.
-  - ```Download all``` - Download all users marked ```Ready for download``` from all sites.
   - ```Download all site users``` - Download all users marked ```Ready for download``` from specific sites.
   - ```Download all FULL``` - Download all users from all sites. **The ```Ready for download``` option will be ignored.**
   - ```Download all site users FULL``` - Download all users from specific sites. **The ```Ready for download``` option will be ignored.**
-  - ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups))
+- ```Standalone downloader``` - Just a [downloader](#download-separate-video) for single videos.
-  - ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
+- ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)).
-  - ```Pause automation``` - Pause the automation process. This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.
+- ```Silent mode``` - Temporarily disabling notifications.
-- ```Download video``` - download a separate video (Reddit and Twitter videos supported) or Instagram post (photo and video).
+- ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)).
+- ```Pause automation``` - [Pause the automation process](#pause-automation).
 - ```Stop``` - stop all download operations.
 
-![Saved posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
+[HotKeys](#hot-keys)
+
+![Download context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
+
+#### Pause automation
+
+This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.
+
+- ```1 hour``` - pause automation for 1 hour
+- ```2 hour``` - pause automation for 2 hours
+- ```3 hour``` - pause automation for 3 hours
+- ```4 hour``` - pause automation for 4 hours
+- ```6 hour``` - pause automation for 6 hours
+- ```12 hour``` - pause automation for 12 hours
+- ```Until``` - you will be prompted to enter the time you want to pause the task(s)
+- ```Unlimited``` - pause the task until you turn it off or close the program
+- ```Disable``` - disable pause
+
+![Pause context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowPause.png)
 
 ### View
-
 - ```View``` - users view modes.
   - The first block specifies **how** users will be displayed in the form.
     - ```Large/Small images``` - with icons.
@@ -150,20 +179,22 @@ This button looks like a bookmark. When you click on this button, a form for dow
     - ```Limit dates (not in range)``` - filter users whose last download date is not in the selected date range
     - ```Limit dates (in range)``` - filter users whose last download date is in the selected date range
 
-  ![Main window context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindow2.png)
+![Main window view context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowView.png)
 
 ### LOG
 
 The error log will be opened by clicking on this button. The log contains program and download errors. The log is saved automatically when you close the program, but you still can save, copy, or clear it (using the special buttons on the log form) if you want...
 
 ## Hot keys
-
+- ```F1``` - program information and checking for a new version
 - ```Insert``` - add user
-- ```F3``` - edit user
-- ```F5``` - start downloading one or more selected users
-- ```F6``` - start downloading all your users marked as ```Ready for download```.
-- ```F2``` - download a separate video or Instagram post
 - ```Del``` - delete one or more selected users
+- ```Enter``` - open user folder
+- ```F3``` - edit user
+- ```F5```, ```Ctrl+F5``` - start downloading one or more selected users (pressing with ```Ctrl``` excludes this data from the feed)
+- ```F6```, ```Ctrl+F6``` - start downloading all your users marked as ```Ready for download``` (pressing with ```Ctrl``` excludes this data from the feed)
+- ```Ctrl+F``` - open search form
+- ```Ctrl+NumericKey``` - download group with this number
 
 Double-click the user to open the content folder.
 
@@ -177,7 +208,23 @@ Double-click the user to open the content folder.
 - ```Download``` - download all videos you added.
 - ```Open saving path``` - a folder will open where the videos are downloaded to.
 
-  ![Separate video downloader window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SeparateVideoDownloader.png)
+![Separate video downloader window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SeparateVideoDownloader.png)
+
+## Tray
+- ```Silent mode``` - Temporarily disabling notifications.
+- ```Pause automation``` - [Pause the automation process](#pause-automation).
+- [```Feed```](#feed)
+- ```Show/Hide``` - show/hide SCrawler
+- ```Close``` - close the program
+- ```Close (no script)``` - close the program ignore the script (if you configured the script in the settings)
+
+![Tray context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/TrayContextMenu.png)
+
+## Search users
+
+```Ctrl+F``` to start searching
+
+![Search form](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SearchUsers.png)
 
 ## Users list
 
@@ -217,15 +264,11 @@ When you add a user to a collection, the user's data will be moved to the collec
 When you add a user to the collection, a new folder will be created in the collection folder named by pattern ```Site```_```UserName```. That is, each profile in the collection is stored separately. When you merge, all files in the profiles' folders are moved to the root of the collection.
 
 ### Deleting a collection
-
 1. Select the collection you want to delete.
 1. Click the ```Delete``` button or right-click on the collection and select ```Delete user / collection```.
-1. In the dialog box, click ```Delete``` button.
+    - ```Delete``` - The collection and all its files will be permanently deleted.
-    - You will be prompted to delete the collection and all of its files:
+    - ```Split``` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
-      - ```Yes``` - The collection and all its files will be permanently deleted.
+    - ```Cancel``` - Cancel operation.
-	  - ```No``` - You will be prompted to split the collection.
-	    - ```Yes``` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
-	    - ```No``` - Cancel operation.
 
 ### Collections example
 

Plugins.md

@@ -128,7 +128,7 @@ interface ISiteSettings
     string GetUserUrl(string UserName,bool Channel);
     ExchangeOptions IsMyUser(string UserURL);
     ExchangeOptions IsMyImageVideo(string URL);
-    IEnumerable<PluginUserMedia> GetSpecialData(string URL);
+    IEnumerable GetSpecialData(string URL);
     IPluginContentProvider GetInstance(Download What);
     void Load(IEnumerable<KeyValuePair<string, string>> XMLValues);
     IEnumerable<KeyValuePair<string, string>> XMLFields();
@@ -140,6 +140,8 @@ interface ISiteSettings
     void BeforeStartDownload(object User, Download What);
     void AfterDownload(object User, Download What);
     void DownloadDone(Download What);
+    void BeginEdit();
+    void EndEdit();
     void Update();
     void Reset();
     void OpenSettingsForm();
@@ -229,7 +231,7 @@ Must return the same as the [```IsMyUser```](#ismyuser), but ```UserName``` can
 
 ### GetSpecialData
 
-```Function GetSpecialData(ByVal URL As String) As IEnumerable(Of PluginUserMedia)```
+```Function GetSpecialData(ByVal URL As String) As IEnumerable```
 
 After ```IsMyImageVideo```, if this is your plugin's media file, this function will be called.
 
@@ -275,6 +277,13 @@ Called at the start of host initialization.
 ### EndInit
 
 Called at the end of host initialization.
+### BeginEdit
+
+Called when opening the settings form in SCawler
+
+### EndEdit
+
+Called when closing the settings form in SCawler
 
 ### BeginUpdate
 

Settings.md

@@ -11,10 +11,9 @@
 - ```Show groups``` - Show user site groups
 - ```Use user grouping``` - Group users by groups and/or labels. If disabled, users will not be grouped by site or label.
 
-  ![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBasis.png)
+![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBasis.png)
 
 ## Behavior
-
 - ```Exit confirm``` - ask for confirmation before closing the program
 - ```Close to tray``` - close program to system tray
 - ```Show notifications``` - show notifications when download is complete
@@ -26,7 +25,17 @@
 - ```Folder cmd``` - the [command](#folder-command) to open a folder
 - ```Close cmd``` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed
 
-  ![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
+![Behavior settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
+
+## Notifications
+- ```Silent mode``` - Temporarily disabling notifications. This setting is not stored in the settings file. It is valid until you turn it off or close the program.
+- ```Show notification``` - This is the base value of notifications. If you disable it, notifications will not appear at all.
+- ```Profiles``` - Show notifications when profiles download is complete.
+- ```AutoDownloader``` - Show AutoDownloader notifications.
+- ```Channels``` - Show notifications when channels download is complete.
+- ```Saved posts``` - Show notifications when saved posts download is complete.
+
+![Notifications settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalNotifications.png)
 
 ## Defaults
 - ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
@@ -34,7 +43,7 @@
 - ```Download images/videos``` - defaults for creating new users
 - ```Download jpg instead of webp``` - save ```webp``` images as ```jpg```
 
-  ![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDefaults.png)
+![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDefaults.png)
 
 ## Downloading
 - ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line.
@@ -50,8 +59,9 @@
 - ```After download cmd``` - this [command](#scrawler-script-text-examples) will be executed after all downloads are completed
 - ```Add missing information to log``` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing-posts) exist.
 - ```Add missing errors to log``` - Each error that prevents SCrawler from downloading a file will be added to the log.
+- ```Trying to download missing posts using regular download``` - If missing posts exist, the missing posts will attempt to be downloaded via user download.
 
-  ![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
+![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
 
 ## Channels
 - ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
@@ -59,59 +69,81 @@
 - ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
 - ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
 
-  ![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalChannels.png)
+![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalChannels.png)
 
 ## Feed
-
 - ```Feed rows/columns``` - how many rows and columns will be displayed in the feed form
 - ```Endless feed``` - go to the next page when you reach the end
 - ```Add the session number to the post title``` - The session number will be displayed to the left of the post title. A session is every time when you (or [AutoDownloader](#automation)) download data.
 - ```Add the date to the post title``` - The download date and time will be displayed to the right of the post title.
+- ```Store session data``` - If checked, session data will be stored in an xml file.
 
-  ![Feed settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalFeed.png)
+![Feed settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalFeed.png)
 
 # Site settings
 
+## Sites requirements
+
+- [Reddit](#reddit-requirements)
+- [Twitter](#twitter-requirements)
+- [Instagram](#instagram-requirements)
+- [RedGifs](#reddit-requirements)
+- [TikTok](#tiktok-requirements)
+- [XVIDEOS](#xvideos-requirements)
+- [LPSG](#lpsg-requirements)
+
+## Site default fields
+- ```Path``` - where Reddit users' data will be placed.
+- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. Saved Twitter posts are posts that you have bookmarked.
+- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
+- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
+- ```Get user media only``` - the default for creating new users
+- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
+
+## ffmpeg
+
+To download videos hosted on Reddit you need to download the ffmpeg.exe and place it in the SCrawler folder. It is also required for xvideos downloader. **You can download ffmpeg from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)**
+
+**ffmpeg only works with the x64 version of the program.**
+
 ## Reddit
 
 The Reddit parser can parse data without cookies, but you can add it if you like.
 
-  - ```Path``` - where Reddit users' data will be placed.
+- ```Use M3U8``` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
-  - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. Saved Twitter posts are posts that you have bookmarked.
+- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
-  - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
-  - ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
-  - ```Get user media only``` - the default for creating new users
-  - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
-  - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
-  - ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
-  - ```Use M3U8``` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
 
-    ![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
+![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
+
+### Reddit requirements
+- [Cookies](#how-to-set-up-cookies) for downloading saved posts
+- [ffmpeg](#ffmpeg) for downloading videos hosted on reddit (only for x64 program)
+- [Imgur clien ID](#how-to-find-imgur-client-id) to download Imgur content posted on reddit
+- [RedGifs credentials](#redgifs-requirements) to download RedGifs content posted on reddit
 
 ### Reddit user settings
+- ```View``` - works the same as Reddit view modes (```new```, ```hot```, ```top```)
+- ```Period``` - only works with ```Top``` view mode and the same as Reddit periods
 
-  - ```View``` - works the same as Reddit view modes (```new```, ```hot```, ```top```)
+![Reddit user settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsRedditView.png)
-  - ```Period``` - only works with ```Top``` view mode and the same as Reddit periods
-
-    ![Reddit user settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsRedditView.png)
 
 ## Twitter
-  - ```Path``` - where Twitter users' data will be placed.
+- [```Authorization```](#how-to-find-twitter-tokens) - Bearer token. Must start with ```Bearer ``` word followed by a space.
-  - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. Saved Twitter posts are posts that you have bookmarked.
+- [```Token```](#how-to-find-twitter-tokens) - in this field you need to put ```x-csrf-token```.
-  - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
+- ```Saved posts user``` - your personal Reddit username to download your saved posts (bookmarked).
-  - ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
-  - ```Get user media only``` - the default for creating new users
-  - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
-  - ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
-  - ```Token``` - in this field you need to put ```x-csrf-token```.
 
-    ![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTwitter.png)
+![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTwitter.png)
+
+### Twitter requirements
+- [```Cookies```](#how-to-set-up-cookies)
+- [```Authorization```](#how-to-find-twitter-tokens)
+- [```Token```](#how-to-find-twitter-tokens)
+- ```Saved posts user``` for download saved posts (bookmarked)
 
 ### How to find Twitter tokens
-
 1. Open Google Chrome, Edge or FireFox.
 1. Go to https://twitter.com
-1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
+1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl+Shift+I```).
 1. Go to the network tab
 1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**)
 1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with ```TweetDetail?```... and click on it
@@ -121,48 +153,71 @@ The Reddit parser can parse data without cookies, but you can add it if you like
 
 
 ## Instagram
+- Authorization
+  - ```Hash``` - in this field you need to put the hash of the Instagram session ([how to find](#how-to-find-instagram-hash)).
+  - ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash).
+  - ```x-csrftoken``` - [how to find](#how-to-find-instagram-authorization-headers)
+  - ```x-ig-app-id```, ```ix-ig-www-claim``` - [how to find](#how-to-find-instagram-authorization-headers)
+  - ```Saved posts user``` - your personal Instagram username to download your saved posts
+- Other parameters
+  - ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
+  - ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
+  - ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
+- ```Get stories```
+- ```Get tagged photos```
+- ```Tagged notify limit``` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit))
 
-  - ```Path``` - where Instagram users' data will be placed.
+![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
-  - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
+
-  - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
+- Other settings
-  - ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
+  - Download Timeline
-  - ```Get user media only``` - the default for creating new users
+  - Download Stories and Tagged data
-  - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
+  - Download saved posts
-  - Authorization
+
-    - ```Hash``` - **\[Required\]** in this field you need to put the hash of the Instagram session ([how to find](#how-to-find-instagram-hash)).
+**ATTENTION!**
-    - ```Hash 2``` - **\[For saved Instagram posts only\]** in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash).
+
-    - ```x-csrftoken``` - **\[Required\]**  read [here](#how-to-find-instagram-authorization-headers) how to find them
+**These settings are not for configuration! Sometimes SCrawler determines some of the error codes as expired credentials. In order to protect your Instagram account from being banned, SCrawler temporarily disables the download module where an error occurred. In this case, a notification will be added to the log. If you have verified your credentials and there are no changes, you can simply re-enable the module manually.**
-    - ```x-ig-app-id```, ```ix-ig-www-claim``` - **\[Required\]**  read [here](#how-to-find-instagram-authorization-headers) how to find them
+
-    - ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**)
+**When you change credentials, SCrawler changes these values depending on the required fields.**
-  - Other parameters
+
-    - ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
+![Instagram additional settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagramAdditional.png)
-    - ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
+
-    - ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
+### Instagram requirements
-    - ```Get stories```
+
-    - ```Get tagged photos```
+**[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
-    - ```Tagged notify limit``` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit))
+
+- To download timeline, stories and tagged posts
+  - [```Cookies```](#how-to-set-up-cookies)
+  - [```Hash```](#how-to-find-instagram-hash)
+  - [```x-csrftoken```](#how-to-find-instagram-authorization-headers)
+  - [```x-ig-app-id```](#how-to-find-instagram-authorization-headers)
+  - [```ix-ig-www-claim```](#how-to-find-instagram-authorization-headers)
+- To download saved posts
+  - [```Cookies```](#how-to-set-up-cookies)
+  - [```Hash 2```](#how-to-find-instagram-hash)
+  - [```x-csrftoken```](#how-to-find-instagram-authorization-headers)
+  - [```x-ig-app-id```](#how-to-find-instagram-authorization-headers)
+  - [```ix-ig-www-claim```](#how-to-find-instagram-authorization-headers)
+  - ```Saved posts user```
 
-    ![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
 
 ### How to find Instagram Hash
-
 1. Open browser developer tools
 1. Go to ```Network```
 1. If this tab contain any data, clear it.
 1. Open any profile for ```Hash``` or your saved Instagram posts for ```Hash 2```.
 1. Click ```Ctrl+F``` and find the text ```query_hash=```. The  alphanumeric value after the equal sign is the required hash.
     - You can find two different hashes. The first is good, the second is wrong.
-	  - https://www.instagram.com/graphql/query/?query_hash=abcde12345abcde12345abcde12345ab&variables={"id":"123456789","first":12}
+      - https://www.instagram.com/graphql/query/?query_hash=abcde12345abcde12345abcde12345ab&variables={"id":"123456789","first":12}
-	  - https://www.instagram.com/graphql/query/?query_hash=54321edcba54321edcba54321edcba54&variables={"user_id":"123456789","include_chaining":true,"include_reel":false,"include_suggested_users":false,"include_logged_out_extras":false,"include_highlight_reels":true,"include_live_status":true}
+      - https://www.instagram.com/graphql/query/?query_hash=54321edcba54321edcba54321edcba54&variables={"user_id":"123456789","include_chaining":true,"include_reel":false,"include_suggested_users":false,"include_logged_out_extras":false,"include_highlight_reels":true,"include_live_status":true}
 1. Copy this and paste it into the  corresponding (```Hash```, ```Hash 2```) field on the Instagram settings form.
 
 ### How to find Instagram authorization headers
-
 1. Open browser developer tools
 1. Go to ```Network```
 1. If this tab contain any data, clear it.
 1. Open profile tagged photos.
-1. Click ```Ctrl``` + ```F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
+1. Click ```Ctrl+F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
 1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
 
 ### Instagram limits
@@ -186,15 +241,48 @@ When the number of tagged posts exceeds the tagged posts limit, you will be aske
 - ```Cancel``` - Cancel tagged posts download operation
 
 ## RedGifs
+- ```Token``` - "Bearer" token ([how to find](#how-to-find-redgifs-token))
 
-  - ```Path``` - where RedGifs users' data will be placed.
+![RedGifs settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteRedGifs.png)
-  - ```Saved posts path``` - *unused*
-  - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
-  - ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
-  - ```Get user media only``` - the default for creating new users
-  - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
 
-    ![RedGifs settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteRedGifs.png)
+### RedGifs requirements
+- [```Cookies```](#how-to-set-up-cookies)
+- [```Token```](#how-to-find-redgifs-token)
+
+### How to find RedGifs token
+1. Open browser developer tools
+1. Go to ```Network```
+1. If this tab contain any data, clear it.
+1. Open any RedGifs profile.
+1. Click ```Ctrl+F``` and find the text ```Bearer```.
+1. Copy the value and paste it into the ```Token``` field on the RedGifs settings form.
+
+## TikTok
+
+![TikTok settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTikTok.png)
+
+### TikTok requirements
+- [```Cookies```](#how-to-set-up-cookies)
+  
+### TikTok limits
+
+Unfortunately, I didn't understand how to use the 'Cursor' option to get the JSON response. In this case, I can only parse an HTML page that only provides the latest 30 videos. **If you know how to use the cursor option to get a JSON response, you might create an issue with your suggestion.**
+
+## XVIDEOS
+- ```Download UHD``` - Download UHD (4K) content
+
+![XVIDEOS settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteXvideos.png)
+
+### XVIDEOS requirements
+
+Nothing is needed to download xvideos content, not even cookies.
+
+## LPSG
+
+![LPSG settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteLPSG.png)
+
+### LPSG requirements
+- [```Cookies```](#how-to-set-up-cookies)
 
 # Download groups
 
@@ -258,7 +346,6 @@ You can specify a command to open a folder in a special program. Pattern: ```Com
 This command can be a batch command or any script file (bat, ps1 or whatever you want) with the incoming argument as a folder path.
 
 # How to set up cookies
-
 1. Open Google Chrome, Microsoft Edge or FireFox.
 1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
 1. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
@@ -274,11 +361,13 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
 **Lines containing a cookie value in curly braces should simply be removed (for example, `{"i_p":0000000000000,"i_l":1}`).**
 
 # How to find Imgur client ID
-
 1. Open browser developer tools
 1. Go to ```Network```
+1. Go to https://imgur.com/
 1. In the list on the left, find a query that contains ```client_id``` and click on it
 1. Copy the client id from the ```Request URL```
+1. Open settings
+1. Paste the copied value into the ```Imgur Clien ID``` field
 
 # How to use the script
 

Users.md

@@ -3,26 +3,21 @@
 In the main window, click the ```Add``` button to open the user creation form.
 
 ![Add user](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/CreateUserClear.png)
-
 - ```User name``` - you can specify only a username or a link by pattern:
   - Twitter:
     - https://twitter.com/SomeUserName
-	- SomeUserName
   - Reddit:
     - https://reddit.com/user/SomeUserName
-	- https://reddit.com/r/SomeSubredditName
+    - https://reddit.com/r/SomeSubredditName
-	- u/SomeUserName
+    - u/SomeUserName
-	- r/SomeSubredditName
+    - r/SomeSubredditName
-	- SomeUserName
   - Instagram
     - https://www.instagram.com/SomeUserName
-	- SomeUserName
   - RedGifs
     - https://www.redgifs.com/users/SomeUserName
-	- SomeUserName
   - XVIDEOS
     - https://www.xvideos.com/Account-Type/SomeUserName
-	- Account-Type/SomeUserName
+    - Account-Type/SomeUserName
 - ```Friendly name``` - if configured, this name will be displayed in the user list instead of the username.
 - ```Special path``` - specify a path other than the default path for user data.
 - ```Sites combo box``` - site pointers.
@@ -33,14 +28,13 @@ In the main window, click the ```Add``` button to open the user creation form.
 - ```Download images/videos``` - for the created user, only the selected media types will be downloaded.
 - ```Ready for download``` - download this user in the operation ```Download all```.
 - ```Get user media only``` - this is an option for twitter only. Download all posts posted by this author.
-- ```Add by list``` - add users' profiles by list (new line as separator (```Shift``` + ```Enter``` to insert new line)).
+- ```Add by list``` - add users' profiles by list (new line as separator (```Shift+Enter``` to insert new line)).
-  - If you add users by the URLs, you must check the ```Auto detect site``` checkbox;
+  - If you add users by the URL, you must check the ```Auto detect site``` checkbox;
-  - If you are adding users by usernames, you must select the ```Reddit/Twitter/Instagram``` site.
+  - If you are adding users by username, you must select the site in the combo box.
 - ```Auto detect site``` - add users by the URLs.
-- ```Description``` - some information you want to keep. If the ```Add by list``` checkbox is checked, you must enter a list of users in this textbox (```Shift``` + ```Enter``` to insert new line);
+- ```Description``` - some information you want to keep. If the ```Add by list``` checkbox is checked, you must enter a list of users in this textbox (```Shift+Enter``` to insert new line);
 - ```Labels``` - labels associated with this user (used to filter users in the main window).
 - ```Script``` - If the checkbox is checked, script to be executed after the user download is complete. If the text is empty, the script from the global settings will be executed. [How to use the script](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-use-the-script).
-
 If neither ```Temporary``` nor ```Favorite``` is checked, the user will be displayed in the regular site group.
 
 # Labels