2022.10.18.0

Moved UserMedia xml initialization to the structure itself
Added download with feed skip
Added silent mode (temporary disabling notifications)
Added additional Instagram protection
Excluding users whose profiles do not exist from downloading with groups and AutoDownloader
Feed: delete file bugs; reorder data after file deletion; video playback bugs
SiteSettingsForm: enable 'OK' button when editing cookies
Fixed collection users ban
Settings: disabling ffmpeg missing notification; advanced notification management
Added 'ToolStripKeyMenuItem' control
Plugins: deprecated XVIDEOS and LPSG plugin libraries; moved them to SCrawler.
Updated license
PluginProvider: added 'BeginEdit' and 'EndEdit' function to ISiteSettings; changed GetSpecialData (ISiteSettings) return type to IEnumerable
PluginsEnvironment: removed 'IsMyClass' attribute
MainFrame: grouped all download buttons into one menu; reorganized code; removed 'F2' hotkey
AutoDownloader: added advanced pause options; added buttons to tray icon and AutoDownloader form
MissingPosts: finished; activated functions that were disabled; added download functions to UserData classes
UserDataBase: ability to use responser; ability to download m3u8; extended 'DownloadingException' with optional argument 'EObj'; user index in collection (button tag) changed to user instance; extended information with user labels; updated 'ProcessException' function
Replaced download buttons with 'KeyClick' control
Replaced FDatePickerForm with my library's form
Collections: Deleting multiple collections - disabled confirmation; ban each user in collection

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,7 +12,7 @@ A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:
-1. Profile URL: 
+1. **Profile URL**: 
 2. Do something
 3. See error
 
@@ -26,8 +26,10 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Release information (please complete the following information):**
+ - OS [e.g. Windows 10, Windows 11]
  - Architecture [e.g. x86, x64]
  - Version [e.g. 2.0.0.0]
+ - NET.Framework version
 
 **Additional context**
 Add any other context about the problem here.

.github/ISSUE_TEMPLATE/plugin_add.md

@@ -1,6 +1,6 @@
 ---
-name: Add plugin
+name: I developed a plugin for SCrawler
-about: Add plugin to plugin list
+about: I developed a plugin for SCrawler. Add plugin to plugin list.
 title: "[NEW PLUGIN]"
 labels: 'New Plugin'
 assignees: ''

.gitignore

@@ -35,6 +35,7 @@ bld/
 [Ll]ogs/
 ffmpeg/
 Info/
+Hidden/
 
 # Visual Studio 2015/2017 cache/options directory
 .vs/

CONTRIBUTING.md

@@ -3,39 +3,38 @@
 I welcome requests! Follow these steps to contribute:
 
 1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
-2. Let me know you are working on it by posting a comment on the issue.
+1. Let me know you are working on it by posting a comment on the issue.
-3. If you find an error in the code, please provide a link to the file and the line number.
+1. If you find an error in the code, please provide a link to the file and the line number.
-4. If you have a suggestion to change the code, you can post a block of code to replace. I don't currently have time to learn pull requests, so it might work this way.
+1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
 
 # How to build from source
-
 1. Delete the "PersonalUtilities" project from the solution.
-2. Add the latest version of the "PersonalUtilities.dll" library (from the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest)).
+1. Delete the "PersonalUtilities.Notifications" project from the solution.
-3. Import PersonalUtilities.Functions for the whole project.
+1. The following libraries must be added to project references with the '**Copy to output folder**' option:
+  - ```PersonalUtilities.dll```
+  - ```PersonalUtilities.Notifications.dll```
+  - ```Microsoft.Toolkit.Uwp.Notifications.dll```
+  - ```System.ValueTuple.dll```
+1. Import PersonalUtilities.Functions for the whole project.
 
-**Always use the correct "PersonalUtilities.dll" library. You must download this library from the release of the code you downloaded.**
+**Always use the correct libraries. You must download libraries from the same release date as the code commit date.**
 
 # How to request a new site
-
 1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
-2. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
+1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
-    - If I'm interested in a site you want to add, it may be added in future releases.
-      - If the site has an API that does not require authorization, it may be added in the coming releases.
-      - You can make it faster by posting a link to the API. **I don't use OAuth authentication** in my application, so if it's not too hard to make a new parsing algorithm **without OAuth** authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
-      - If the site does not have an API that does not require authorization, this may take some time.
-	- If you will be posting request urls **without OAuth** authentication, I might consider adding your site if I have time.
-    - If I'm **not** interested in the site you want to add, you can pay to have it added by making a donation of approximately $10. **But before that, you still need to create an issue. If I'm not interested, you can offer me a deal to develop it for money. I'll check the site you want to add, check the availability of the API and tell you how much time I need to develop it and the price. If you agree, I will do it.** [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
 
+# Requirements for new site requests
+
+**Attention! I'll add a new site only if I'm interested. I also have a life, and any development takes time.**
+
+- Post a link to the site's API
+- Post request URLs **without OAuth** authentication
+- Post a **complete cURL** request which provides the required information (JSON is better)
+
+**I don't use OAuth authentication** in my application, so if it's not too hard to make a new parsing algorithm **without OAuth** authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
+
+If I'm interested in a site you want to add, it may be added in future releases.
 
 # Sites I will never develop
-
 - Facebook
-
+- Tumblr
-# Sites requested by users
-
-- TikTok
-  - API for receiving data without authorization was not found. Therefore, I don't have time to start developing this site parsing algorithm. If anyone knows of requests that may collect data without OAuth authentication, please let me know.
-  
-# Contact me
-
-[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)

Changelog.md

@@ -1,5 +1,293 @@
+# 2022.10.18.0
+
+*2022-10-18*
+
+- Added
+  - **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
+  - **Search form** (```Ctrl+F```)
+  - Feed improvements
+  - Ability to save the download session for viewing later
+  - Ability to download user, excluding from the feed (use the ```Ctrl``` key with a button click of with a hot key press)
+  - Ability to disable the notification about the absence of the ffmpeg.exe file
+  - Extended user information with labels  
+  - Advanced AutoDownloader pause options
+  - Added pause buttons to tray icon and AutoDownloader form
+  - Additional Instagram protection
+  - Advanced notification management
+  - Silent mode (temporarily disable notification)
+  - Excluding users whose profiles do not exist from downloading with groups and AutoDownloader
+  - Minor improvements
+- Updated
+  - Grouped all download buttons into one menu
+  - **Finished missing posts**. You can now download missing posts if they exist.
+  - PluginProvider: added ```BeginEdit``` and ```EndEdit``` methods
+  - PluginProvider: ```GetSpecialData``` return type changed from ```IEnumerable(Of PluginUserMedia)``` to ```IEnumerable```
+  - XVIDEOS and LPSG plugins are moved from libraries to SCrawler
+- Fixed
+  - (Issue #69) **RedGifs data is not downloading**. Requires cookies and token.
+  - Some minor bugs when deleting a collection
+  - Feed: start video playing may cause the program to freeze (strange behavior of the vlc library)
+  - Feed: videos hosted on Reddit not showing up in feed
+  - Feed: minor bugs
+  - Collection users were not banned when deleted with the ban option
+  - When trying to delete multiple collections, each collection asked for confirmation to delete
+  - Minor bugs
+
+# 2022.9.24.0
+
+*2022-09-24*
+
+- Added
+  - Ability to copy user data to another destination
+  - Ability to add 'Session' and 'Date' values to the post title in the feed
+  - Minor feed improvements
+  - The newly created collection will now appear at the top of the list (after reopening the form)
+  - Ability to add multiple users at a time to the collection.
+- Fixed
+  - Autodownloader opens a compressed image instead of a full one
+  - Incorrect resizing of the feed grid after deleting a media file
+  - Incorrect behavior when deleting/removing a user from a collection.
+  - An incorrect function that displayed the number of spent Instagram requests.
+  - Bug in the XVIDEOS downloader
+  - Minor bugs
+
+# 2022.9.17.0
+
+*2022-09-17*
+
+- Added
+  - Added two date filters to filter users (in range, not in range)
+  - (Request #71) Download data for a specific date range
+  - The ability to disable site downloading (in the site settings form)
+- Updated
+  - Plugins
+- Fixed
+  - (Issue #71) ```Download data to the date``` doesn't work for Twitter
+  - Download data for a specific date range doesn't work for multiple users
+  - Incorrect feed sorting algorithm
+  - Minor bugs
+
+# 2022.9.16.0
+
+*2022-09-16*
+
+- Fixed
+  - Failed to get video thumbnail for channel video post
+  - Incorrect rendering of the 'Feed' table when the number of columns is more than one
+  - Minor design bugs
+
+# 2022.9.13.0
+
+*2022-09-13*
+
+- Added
+  - Video duration to the feed
+- Fixed
+  - (Issue #70) Instagram posts not downloading if there are pinned posts that have already been downloaded
+  - Minor bugs
+
+# 2022.9.10.0
+
+*2022-09-10*
+
+- Fixed
+  - The memory is still leaking. This time because of the video. *Using WMP was not the best choice.*
+
+# 2022.9.8.1
+
+*2022-09-08*
+
+- Fixed
+  - Unexpected memory leak when using the 'Feed' form
+
+# 2022.9.8.0
+
+*2022-09-08*
+
+- Added
+  - **Feed** (feed of downloaded media files)
+  - Missing posts tracking and management
+  - Simple scheduler notifications
+- Fixed
+  - (Issue #67) Saved Instagram posts not downloading
+
+# 2022.8.28.0
+
+*2022-08-28*
+
+- Added
+  - RedGifs icon
+- Fixed
+  - Incorrect number of posts displayed in the Reddit channels downloader.
+
+# 2022.8.22.0
+
+*2022-08-22*
+
+- Added
+  - Ability to enable/disable the display of the downloaded image in toast notifications (AutoDownloader)
+  - Ability to enable/disable the display of the user icon in toast notifications (AutoDownloader)
+  - Downloading with standalone video downloader has been moved to a separate thread
+- Fixed
+  - (Issue #35) The file name does not change only by date
+  - (Issue #62) Internal library error
+  - AutoDownloader option ```Show notifications``` not saved
+  - Minor bugs
+
+# 2022.7.7.0
+
+*2022-07-07*
+
+- Added
+  - **Scheduler** (creating multiple automation tasks)
+  - Automation startup delay
+  - Download ```webp``` in ```jpg``` format
+  - Development: the ability to create a label control, that provides some information
+- Removed
+  - Instagram auto-fill hash from cookies
+- Updated
+  - Plugins
+- Fixed
+  - ```Stop``` option not working properly
+  - In some cases, Twitter image is not downloading
+  - Minor bugs
+
+# 2022.6.10.0
+
+*2022-06-10*
+
+**Attention! From now on, Instagram requires Cookies, Hash and authorization headers!**
+
+- Fixed
+  - Can't get Instagram user ID
+
+# 2022.6.6.0
+
+*2022-06-06*
+
+- Added
+  - Ability to pause automation
+- Fixed
+  - GIFs from Twitter not downloading
+  - Not quite correct algorithm for stopping automation
+
+# 2022.6.3.0
+
+*2022-06-03*
+
+Changed version numbering method. From now on, new versions will be numbered by release date (YYYY.M.D)
+
+**Attention! Starting with this release, SCrawler may not work on windows 7 and 8 or may not work correctly. All future releases will only be guaranteed to work on windows 10 and 11.**
+
+- Added
+  - **Automation** (downloading data automatically every ```X``` minutes)
+  - Expanded settings for Instagram tagged posts that are downloaded for the first time.
+- Fixed
+  - Videos hosted on Reddit that are downloaded via m3u8 playlists are missing an audio track.
+  - Instagram hash not able to be auto-filled from cookies
+
+# 3.0.0.10
+
+*2022-05-23*
+
+- Added
+  - **Downloading groups**
+  - **Download saved Twitter posts** (bookmarks)
+  - Ability to enable/disable progress form opening at the start of downloading
+  - Ability to enable/disable Info form opening at the start of downloading
+  - The ability to disable the opening of forms Info and Progress at the start of downloads if it was once closed
+  - Focusing the main window when opening Info or Progress forms
+  - Ability to execute a script/command when closing SCrawler
+  - Ability to execute a script/command after all downloads are completed
+  - Minor improvements
+- Fixed
+  - Instagram tagged data not downloading (now requires one more parameter **x-csrftoken** to download tagged data)
+  - In some cases, Instagram Stories cannot be downloaded due to forbidden Windows characters
+  - Separate Instagram posts were not downloading via the Video Downloader form.
+  - In some cases, an Imgur video hosted on Reddit won't download
+  - Gfycat data not downloading from saved Reddit posts
+  - In some cases, the date and time are not added to the filename
+  - Unable to download photos from Twitter in full resolution (4K)
+
+# 3.0.0.9
+
+*2022-04-24*
+
+- Added
+  - Excluded labels
+  - Ability to disable user grouping
+  - Ability to show groups of user sites when filtering by labels
+- Fixed
+  - Removed adding "No Parsed" internal label when not needed
+  - Redownloading Instagram Stories
+
+# 3.0.0.8
+
+*2022-04-19*
+
+- Added
+  - Script mode ```command```
+  - Disabled Instagram error 403 (Forbidden) logging for downloading tagged data
+- Fixed
+  - The script does not run after the user download is complete
+
+# 3.0.0.7
+
+*2022-04-14*
+
+- Added
+  - Ability to run a script after the user download is complete
+  - Hotkey ```F2``` for additional options in the user creation form
+- Fixed
+  - (Issue #32) In some cases, Date and Time are still not added for Stories and Tagged Photos
+  - (Issue #33) Instagram Stories downloading error
+  - LPSG downloader does not download all content
+
+# 3.0.0.6
+
+*2022-04-04*
+
+- Added
+  - ```GoTo Start``` channels button
+  - ```GoTo End``` channels button
+- Fixed
+  - In some cases, saved Reddit posts didn't fully download
+  - Incorrect Reddit accessibility check algorithm
+  - Incorrect behavior of the main progress bar when downloading saved posts
+  - (Issue #25) Date and Time not added for Stories and Tagged Photos
+
+# 3.0.0.5
+
+*2022-04-02*
+
+- Added
+  - ```New```, ```Hot```, ```Top``` Reddit channel and user download modes
+
+# 3.0.0.4
+
+*2022-03-26*
+
+- Fixed
+  - External plugins do not save information about downloaded files
+  - The user cannot be added to the collection if a special path has been specified.
+
+# 3.0.0.3
+
+*2022-03-24*
+
+- Added
+  - Download all by specific sites
+  - Download all, ignoring the ```Ready for download``` option
+  - Download all by specific sites, ignoring the ```Ready for download``` option
+- Fixed
+  - (Issue #19) Typo in default Instagram settings (Post limit timer)
+  - Typo when applying "Download UHD" in XVIDEOS plugin
+  - The sites filter does not work unless the "Fast profiles loading" option is enabled.
+
 # 3.0.0.2
 
+*2022-03-22*
+
 - Added
   - **LPSG** site plugin
   - **XVIDEOS** site plugin
@@ -10,6 +298,8 @@
 
 # 3.0.0.1
 
+*2022-03-20*
+
 - Added
   - Download data up to a specific date
   - Update and Reset functions in the plugin (ISiteSettings)
@@ -23,6 +313,8 @@
 
 # 3.0.0.0
 
+*2022-03-17*
+
 **Attention! This version of the program makes changes user data file (Users.xml). Once you start using this version, you will not be able to use previous versions of the program. Therefore, it is highly recommended to archive the program settings folder and archive the users' data files (you can use the [```ArchiveSCrawlerUsersDataFiles.bat```](Tools/ArchiveSCrawlerUsersDataFiles.bat) tool to archive the data files of all users).**
 
 - Added
@@ -63,6 +355,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 2.0.0.4
 
+*2022-02-07*
+
 **Removed compatibility of program settings with version 1.0.0.4 and lower.**
 
 **If your program version is 1.0.0.4 and lower, it is strongly recommended that you upgrade to release 2.0.0.1 to update the program settings (and run the program). Then update to this release. Otherwise, you will have to configure the program settings again**
@@ -77,6 +371,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 2.0.0.3
 
+*2022-02-02*
+
 **Removed compatibility of program settings with version 1.0.0.4 and lower.**
 
 **If your program version is 1.0.0.4 and lower, it is strongly recommended that you upgrade to release 2.0.0.1 to update the program settings (and run the program). Then update to this release. Otherwise, you will have to configure the program settings again**
@@ -92,6 +388,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 2.0.0.2
 
+*2022-01-23*
+
 **This is the last release that supports program settings of version 1.0.0.4 and lower. Compatibility of program settings with version 1.0.0.4 and lower will be removed in future releases. It is strongly recommended that you upgrade to this release before future releases. Otherwise, you will have to configure the program settings again. If your program version is 1.0.1.0 or higher, you should not pay attention to this message.**
 
 - Added
@@ -114,6 +412,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 2.0.0.1
 
+*2021-12-29*
+
 - Added
   - Download individual Imgur media files (use the "Download video" form).
 - Fixed
@@ -122,6 +422,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 2.0.0.0
 
+*2021-12-27*
+
 - Added
   - **Instagram**
   - Filter by site
@@ -139,6 +441,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 1.0.1.0
 
+*2021-12-20*
+
 - Added
   - Extended site settings
   - Non-existend users will be marked in red
@@ -160,6 +464,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 1.0.0.4
 
+*2021-12-12*
+
 - Added
   - Full channels support (you can now add channel (subreddit) for standard download)
   - ```Ready for download``` now available for collections and can be changed for multiple user
@@ -168,12 +474,16 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 1.0.0.3
 
+*2021-12-11*
+
 - Fixed
   - Custom "Download videos" option is not saved
   - The "Download all" button is not activated after changing modes
 
 # 1.0.0.2
 
+*2021-12-10*
+
 - Added
   - Ability to choose what types of media you want to download (images only, videos only, both)
   - Ability to name files by date
@@ -182,6 +492,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 1.0.0.1
 
+*2021-12-09*
+
 - Added
   - Limited download if user added from the channel
   - Forced limited download for any user
@@ -204,4 +516,6 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
 
 # 1.0.0.0
 
+*2021-12-07*
+
 Initial release

FAQ.md

@@ -2,6 +2,8 @@
 
 **Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
 
+**Also read [here](README.md) for basic information.**
+
 Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
 
 Any other questions I will keep in this file.
@@ -16,7 +18,7 @@ A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
 
 #### Q: **I can't copy cookies.**
 
-A: Use the mouse. Don't use ```Ctrl``` + ```A```!
+A: Use the mouse. Don't use ```Ctrl+A```!
 
 ----
 
@@ -42,6 +44,32 @@ A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-
 
 A: Check your credentials. Both of these sites require cookies. Check your [Twitter tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) and [Instagram settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-settings). If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
 
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
+
+----
+
+#### Q: **I have set credentials but still nothing is downloading**
+
+A: Click the ```Start downloading``` button
+
+----
+
+#### Q: **Where can I find the release?**
+
+A: https://github.com/AAndyProgram/SCrawler/releases/latest
+
+----
+
+#### Q: **How to run the program?**
+
+A: Double-click ```SCrawler.exe```
+
+----
+
+#### Q: **Where to find binaries?**
+
+A: https://github.com/AAndyProgram/SCrawler/releases/latest
+
 ----
 
 #### Q: **Does the program remember the last download and check for new posts, downloading only new posts? Or does the program download the entire profile every time?**
@@ -64,4 +92,28 @@ A: There is no functionality to remove an individual label. You can open the ```
 
 #### Q: **How to remove a user from the blacklist**
 
-A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist```` button.
+A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist``` button.
+
+----
+
+#### Q: **Why don't you answer how it works**
+
+A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
+
+----
+
+#### Q: **You lost me. Your program is too complicated.**
+
+A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
+
+----
+
+#### Q: **I can't configure something**
+
+A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
+
+----
+
+#### Q: **Can you add a step-by-step guide or video on how to use the program?**
+
+A: **NO**! I will not do it. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.

HowToSupport.md

@@ -2,7 +2,7 @@ Your support is very valuable to me. Any support is greatly appreciated. Your su
 
 You can support the program by:
   - **Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
-  - :heavy_dollar_sign: making donaion making donations on this site: https://ko-fi.com/andyprogram
+  - :heavy_dollar_sign: make a donation on this site: https://ko-fi.com/andyprogram
   - :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
   - :speech_balloon: tell your friends about the program
   - :heart: like the program on this site: https://alternativeto.net/software/scrawler/about/

Plugins.md

@@ -1,3 +1,4 @@
 List of available plugins:
-- LPSG
+
-- XVIDEOS
+Tools:
+- [image2post](https://github.com/unknown81311/SCrawler-image2post) by @unknown81311: **get reddit post URL from file.**

ProgramsComparison.md

@@ -5,6 +5,8 @@ https://www.4kdownload.com/products/product-stogram
 | Option | SCrawler | 4K Stogram |
 | ---- | ---- | ---- |
 | User managament | **Advanced** | Primitive |
+| Automatic downloads | **Yes** | No |
+| Downloading groups | **Yes** | No |
 | Labeling users | **Yes** | No |
 | Filtering | **Yes** | No |
 | Collections | **Yes** | No |
@@ -16,10 +18,10 @@ https://www.4kdownload.com/products/product-stogram
 | Download posts by location | No | **Yes** |
 | Save Private Instagram Content with Permission| Yes | Yes |
 | Download Instagram Stories and Highlights | Yes | Yes |
-| See Others Instagram Feed As Your Own | No | **Yes** |
+| See Others Instagram Feed As Your Own | Yes | Yes |
 | Download Instagram Video Posts | Yes | Yes |
 | Backup Your Instagram Account | Yes | Yes |
-| Save Instagram Posts by Date | No (only limited download) | **Yes** |
+| Save Instagram Posts by Date | Yes | Yes |
 | Download Instagram Saved Posts | Yes | Yes |
 | Download Instagram Tagged Posts | Yes | Yes |
 | Export and import subscriptions | No | **Yes** |
@@ -29,7 +31,7 @@ https://www.4kdownload.com/products/product-stogram
 | Automatic Subscriptions Update | **Free** | Paid (43.56 EUR) |
 | Posts and Captions Export | No | Paid (43.56 EUR) |
 | Advertisements free | **No ADs at all for free** | Paid (14.52) |
-| Operating Systems | Windows 7+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
+| Operating Systems | Windows 10+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
 | Select want content type to download | **Yes** | No |
 | Instagram support | Yes | Yes |
 | Twitter support | **Yes** | No |
@@ -44,6 +46,8 @@ https://github.com/RipMeApp/ripme
 | Option | SCrawler | RipMeApp |
 | ---- | ---- | ---- |
 | User managament | **Advanced** | No |
+| Automatic downloads | **Yes** | No |
+| Downloading groups | **Yes** | No |
 | Labeling users | **Yes** | No |
 | Filtering | **Yes** | No |
 | Collections | **Yes** | No |
@@ -62,9 +66,9 @@ https://github.com/RipMeApp/ripme
 | Export and import subscriptions | No | No |
 | **Paid** | **No** | **No** |
 | **Free options** | The program is completely free | The program is completely free, but site limits are not declared |
-| Operating Systems | Windows 7+ | Windows, MacOS, Linux |
+| Operating Systems | Windows 10+ | Windows, MacOS, Linux |
 | Select want content type to download | Yes | Yes |
-| Suported sites | 3 internal and any site using plugins | 86+ sites (declared) |
+| Suported sites | 6 internal and any site using plugins | 86+ sites (declared) |
 | Other sites support | **Yes** | No |
 | Still supported | **Yes** | **No (last release date May 4, 2021)** |
 
@@ -75,4 +79,4 @@ https://github.com/mikf/gallery-dl
 
 **CLI tool**! Configured with JSON files only. Users need to learn complex configuration options, JSON, commands to use that tool. Very difficult to configure.
 
-SCrawler has advanced user management, collections, labels, groups, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
+SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.

README.md

@@ -1,12 +1,14 @@
-# Social networks crawler
+# :rainbow_flag: Social networks crawler :rainbow_flag:
 
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest)
-[![GitHub](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/)
+[![GitHub license](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
 [![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
 [![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
 [![How to support](https://img.shields.io/badge/HowToSupport-green)](HowToSupport.md)
 
-A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram).
+A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram, TikTok, RedGifs, XVIDEOS, LPSG).
+
+**If you like SCrawler, please like the program on [this site]( https://alternativeto.net/software/scrawler/about/)**
 
 Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
 
@@ -19,22 +21,24 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 
 # What can program do:
 - Download pictures and videos from users' profiles and subreddits:
-  - Reddit images;
+  - Reddit images, galleries of images, videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
-  - Reddit galleries of images;
+  - Redgifs videos (https://www.redgifs.com/);
-  - Redgifs hosted videos (https://www.redgifs.com/);
-  - Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
   - Twitter images and videos;
-  - Instagram images and videos.
+  - Instagram images and videos, tagged posts, stories;
-  - Imgur images, galleries and videos
+  - TikTok videos ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
-  - Gfycat videos
+  - Imgur images, galleries and videos;
+  - Gfycat videos;
   - [Other](#supported-sites) supported sites
-- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels).
+- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
-- Download [saved Reddit and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
+- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
-- Add users from parsed channel.
+- Add users from parsed channel
-- **Advanced user management.**
+- **Advanced user management**
-- Labeling users.
+- **Automation** (downloading data automatically every ```X``` minutes)
-- Adding users to favorites and temporary.
+- **Feed** (feed of downloaded media files)
-- Filter exists users by label or group.
+- Labeling users
+- Create download groups
+- Adding users to favorites and temporary
+- Filter exists users by label or group
 - Selection of media types you want to download (images only, videos only, both)
 - Download a special video, image or gallery
 - Making collections (grouping users into collections)
@@ -48,6 +52,7 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 - **Reddit**
 - **Twitter**
 - **Instagram**
+- **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
 - RedGifs
 - Imgur
 - Gfycat
@@ -55,7 +60,9 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 - XVIDEOS
 - [Other sites](Plugins.md)
 
-# How does it works:
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
+
+# How it works
 
 First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
 
@@ -67,26 +74,45 @@ The program parses all user posts, obtain MD5  images hash and compares them wit
 
 The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
 
-You can read about Instagram restrictions [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-limits)
-
 ## How to request a new site
 
 Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
 
 # Requirements
 
-- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
+- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for Twitter (if you want to download data from Twitter)
+- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for Instagram (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for saved Instagram posts, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for Stories and Tagged data
-- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)). **ffmpeg only works with the x64 version of the program.**
 
 # Guide
 
+- [Main window](https://github.com/AAndyProgram/SCrawler/wiki)
+  - [Users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
+    - [Add/Edit/Delete users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
+    - [Collections](https://github.com/AAndyProgram/SCrawler/wiki#collections)
+    - [User operations](https://github.com/AAndyProgram/SCrawler/wiki#context-menu)
+    - [User labels](https://github.com/AAndyProgram/SCrawler/wiki/Users#labels)
+  - **[DOWNLOAD](https://github.com/AAndyProgram/SCrawler/wiki#download)**
+    - [Automation](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
+    - [Download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
+  - [Downloading information](https://github.com/AAndyProgram/SCrawler/wiki#info)
+  - [Reddit channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
+  - [Saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)
+  - [View modes, filters](https://github.com/AAndyProgram/SCrawler/wiki#view)
+- **[SETTINGS](https://github.com/AAndyProgram/SCrawler/wiki/Settings)**
+  - **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
+  - [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
+  - [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
+  - [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
+  - [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
+  - [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
+  - [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
+  - [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
+
 **Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
 
 # Installation
 
-**Just unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
+**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
 
 **Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
 
@@ -96,9 +122,7 @@ Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest)
 
 # How to build from source
 
-1. Delete the "PersonalUtilities" project from the solution.
+Read about how to build from source [here](CONTRIBUTING.md#how-to-build-from-source)
-1. Add the latest version of the "PersonalUtilities.dll" library (from the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest)).
-1. Import PersonalUtilities.Functions for the whole project.
 
 # How to make a plugin
 
@@ -112,22 +136,9 @@ Read more about how to support the program [here](HowToSupport.md).
 
 The program has an intuitive interface.
 
-You need to set up authorization for Twitter and Instagram:
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for **Twitter** (if you want to download data from Twitter)
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for **Instagram** (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for **saved Instagram posts**, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for **Stories** and **Tagged data**
 
-Just add a user profile and click the ```Start downloading``` button.
+Just add a user profile and **click the ```Download``` button**.
-
-You can add users by patterns:
-- https://www.instagram.com/SomeUserName
-- https://twitter.com/SomeUserName
-- https://reddit.com/user/SomeUserName
-- https://reddit.com/r/SomeSubredditName
-- https://www.redgifs.com/users/SomeUserName
-- u/SomeUserName
-- r/SomeSubredditName
-- SomeUserName (in this case, you need to select the user's site)
-- SomeSubredditName
 
 Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
 
@@ -140,7 +151,3 @@ Create a shortcut for the program. Open shortcut properties. In the ```Shortcut`
 Example: ```D:\Programs\SCrawler\SCrawler.exe v```
 
 ![Separate video downloader](ProgramScreenshots/SeparateVideoDownloader.png)
-
-# Contact me
-
-[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)

SCrawler.Plugin.LPSG/Declarations.vb

@@ -9,10 +9,15 @@
 Imports PersonalUtilities.Functions.RegularExpressions
 Friend Module Declarations
     Friend ReadOnly Property PhotoRegEx As RParams = RParams.DM("(https://www.lpsg.com/attachments)(.+?)(?="")", 0, RegexReturn.List)
+    Friend ReadOnly Property PhotoRegExExt As New RParams("img.data.src=""(/proxy[^""]+?)""", Nothing, 1, RegexReturn.List) With {
+        .Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
     Friend ReadOnly Property NextPageRegex As RParams = RParams.DMS("<link rel=""next"" href=""(.+?/page-(\d+))""", 2)
-    Private Const FileUrlRegexDefault As String = "(.+[^/]+?)(jpg|jpeg|gif|png)"
+    Private Const FileUrlRegexDefault As String = "([^/]+?)(jpg|jpeg|gif|png|webm)"
+    Private ReadOnly InputFReplacer As New ErrorsDescriber(EDP.ReturnValue)
+    Private ReadOnly InputForbidRemover As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, Input, Input.StringRemoveWinForbiddenSymbols(, InputFReplacer))
     Friend ReadOnly Property FileRegEx As New RParams(FileUrlRegexDefault, Nothing, 0) With {
         .Converter = Function(ByVal Input As String) As String
+                         Input = InputForbidRemover.Invoke(Input)
                          If Not Input.IsEmptyString Then
                              Dim lv$ = Input.Split("-").LastOrDefault
                              If Not lv.IsEmptyString Then
@@ -22,5 +27,8 @@ Friend Module Declarations
                          End If
                          Return Input
                      End Function}
+    Friend ReadOnly Property FileRegExExt As New RParams(FileUrlRegexDefault, 0, Nothing, InputForbidRemover)
+    Friend ReadOnly Property FileRegExExt2 As New RParams("([^/]+?)(?=(\Z|&))", 0, Nothing, InputForbidRemover)
     Friend ReadOnly Property FileExistsRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 2)
+    Friend ReadOnly TempListAddParams As New ListAddParams(LAP.NotContainsOnly) With {.Comparer = New FComparer(Of PluginUserMedia)(Function(x, y) x.URL = y.URL)}
 End Module

SCrawler.Plugin.LPSG/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("1.0.0.0")>
+<Assembly: AssemblyVersion("2022.10.12.0")>
-<Assembly: AssemblyFileVersion("1.0.0.0")>
+<Assembly: AssemblyFileVersion("2022.10.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.Plugin.LPSG/SiteSettings.vb

@@ -30,7 +30,7 @@ Public Class SiteSettings : Implements ISiteSettings
                 .LoadSettings()
             Else
                 .CookiesDomain = "www.lpsg.com"
-                .Cookies = New CookieKeeper("www.lpsg.com")
+                .Cookies = New CookieKeeper(.CookiesDomain)
             End If
         End With
     End Sub
@@ -54,6 +54,10 @@ Public Class SiteSettings : Implements ISiteSettings
     End Sub
 #End Region
 #Region "Update"
+    Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
+    End Sub
+    Public Sub EndEdit() Implements ISiteSettings.EndEdit
+    End Sub
     Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
     End Sub
     Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -81,7 +85,7 @@ Public Class SiteSettings : Implements ISiteSettings
             Else
                 Return Nothing
             End If
-        Catch ex As Exception
+        Catch
             Return Nothing
         End Try
     End Function
@@ -91,10 +95,13 @@ Public Class SiteSettings : Implements ISiteSettings
     Public Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia) Implements ISiteSettings.GetSpecialData
         Return Nothing
     End Function
-    Public Function Available(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.Available
+    Public Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
-        Return True
+        Return If(Responser.Cookies?.Count, 0) > 0
     End Function
     Public Function ReadyToDownload(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.ReadyToDownload
         Return True
     End Function
+    Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+        Return String.Empty
+    End Function
 End Class

SCrawler.Plugin.LPSG/UserData.vb

@@ -7,6 +7,9 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports UStates = SCrawler.Plugin.PluginUserMedia.States
+Imports UTypes = SCrawler.Plugin.PluginUserMedia.Types
+Imports Converters = PersonalUtilities.Functions.SymbolsConverter.Converters
 Public Class UserData : Implements IPluginContentProvider
 #Region "XML names"
     Private Const Name_LatestPage As String = "LatestPage"
@@ -35,7 +38,8 @@ Public Class UserData : Implements IPluginContentProvider
     Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
     Public Property DataPath As String Implements IPluginContentProvider.DataPath
     Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
-    Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
+    Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
+    Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
 #End Region
 #Region "Interface exchange options"
     Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -58,14 +62,14 @@ Public Class UserData : Implements IPluginContentProvider
 #End Region
     Private Property LatestPage As String = String.Empty
     Private Property Responser As Response = Nothing
+    Private Enum Mode : Internal : External : End Enum
     Public Sub GetMedia() Implements IPluginContentProvider.GetMedia
         Try
             If Not Responser Is Nothing Then Responser.Dispose()
             Responser = New Response
             With Responser : .Copy(Settings.Responser) : .Error = EDP.ThrowException : End With
 
-            Dim l As List(Of String) = Nothing
+            Dim NextPage$
-            Dim NextPage$ = String.Empty
             Dim r$
             Dim _LPage As Func(Of String) = Function() If(LatestPage.IsEmptyString, String.Empty, $"page-{LatestPage}")
 
@@ -76,35 +80,60 @@ Public Class UserData : Implements IPluginContentProvider
                 Thrower.ThrowAny()
                 If Not r.IsEmptyString Then
                     NextPage = RegexReplace(r, NextPageRegex)
-                    l.ListAddList(RegexReplace(r, PhotoRegEx), LAP.NotContainsOnly)
+                    UpdateMediaList(RegexReplace(r, PhotoRegEx), Mode.Internal)
+                    UpdateMediaList(RegexReplace(r, PhotoRegExExt), Mode.External)
                     If NextPage = LatestPage Or NextPage.IsEmptyString Then Exit Do Else LatestPage = NextPage
                 Else
                     Exit Do
                 End If
             Loop
 
-            If l.ListExists Then
-                Dim f As SFile
-                For Each u$ In l
-                    If Not IsEmptyString(RegexReplace(u, FileExistsRegEx)) Then
-                        f = CStr(RegexReplace(u, FileRegEx))
-                        f.Path = DataPath.CSFilePSN
-                        f.Separator = "\"
-                        TempMediaList.Add(New PluginUserMedia With {.ContentType = PluginUserMedia.Types.Picture, .URL = u, .File = f})
-                    End If
-                Next
             If TempMediaList.ListExists And ExistingContentList.ListExists Then _
                TempMediaList.RemoveAll(Function(m) ExistingContentList.Exists(Function(mm) mm.URL = m.URL))
-            End If
         Catch oex As OperationCanceledException
         Catch dex As ObjectDisposedException
         Catch ex As Exception
-            LogProvider.Add(ex, "[LPSG.UserData.GetMedia]")
+            If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                LogProvider.Add("LPSG not available")
+            Else
+                LogProvider.Add(ex, $"[LPSG.UserData.GetMedia({Name})]")
+            End If
         End Try
     End Sub
+    Private Sub UpdateMediaList(ByVal l As List(Of String), ByVal m As Mode)
+        If l.ListExists Then
+            Dim f As SFile
+            Dim u$
+            Dim exists As Boolean
+            Dim r As RParams
+            Dim ude As New ErrorsDescriber(EDP.ReturnValue)
+            For Each url$ In l
+                If Not url.IsEmptyString Then u = SymbolsConverter.Decode(url, {Converters.HTML, Converters.ASCII}, ude) Else u = String.Empty
+                If Not u.IsEmptyString Then
+                    exists = Not IsEmptyString(RegexReplace(u, FileExistsRegEx))
+                    If m = Mode.Internal Then
+                        r = FileRegEx
+                    Else
+                        r = FileRegExExt
+                        If Not exists Then
+                            r = FileRegExExt2
+                            exists = Not IsEmptyString(RegexReplace(u, FileRegExExt2))
+                        End If
+                    End If
+                    If exists Then
+                        f = CStr(RegexReplace(u, r))
+                        f.Path = DataPath.CSFilePSN
+                        f.Separator = "\"
+                        If f.Extension.IsEmptyString Then f.Extension = "jpg"
+                        TempMediaList.ListAddValue(New PluginUserMedia With {.ContentType = UTypes.Picture, .URL = url, .File = f}, TempListAddParams)
+                    End If
+                End If
+            Next
+        End If
+    End Sub
     Public Sub Download() Implements IPluginContentProvider.Download
         Try
-            With Responser : .UseWebClient = True : .UseWebClientCookies = True : End With
+            With Responser : .UseWebClient = True : .UseWebClientCookies = True : .ResetError() : End With
             If TempMediaList.ListExists Then
                 Dim m As PluginUserMedia
                 Dim eweb As ErrorsDescriber = EDP.ThrowException
@@ -112,15 +141,21 @@ Public Class UserData : Implements IPluginContentProvider
                 For i% = 0 To TempMediaList.Count - 1
                     Thrower.ThrowAny()
                     m = TempMediaList(i)
-                    m.DownloadState = PluginUserMedia.States.Tried
+                    m.DownloadState = UStates.Tried
                     Try
                         If Not m.URL.IsEmptyString And Not m.File.IsEmptyString Then
                             Responser.DownloadFile(m.URL, m.File, eweb)
-                            m.DownloadState = PluginUserMedia.States.Downloaded
+                            m.DownloadState = UStates.Downloaded
                         Else
-                            m.DownloadState = PluginUserMedia.States.Skipped
+                            m.DownloadState = UStates.Skipped
+                        End If
+                    Catch wex As Exception
+                        If Responser.Client.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                            LogProvider.Add("LPSG not available")
+                        Else
+                            m.DownloadState = UStates.Missing
+                            m.Attempts += 1
                         End If
-                    Catch ex As Exception
                     End Try
                     RaiseEvent ProgressChanged(1)
                     TempMediaList(i) = m

SCrawler.Plugin.XVIDEOS/M3U8.vb

@@ -41,13 +41,13 @@ Friend NotInheritable Class M3U8
             Return Nothing
         Catch ex As Exception
             Logger.Add(ex, "[M3U8.Save]")
-            Return Nothing
+            ex.HelpLink = 1
+            Throw ex
         Finally
             CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
         End Try
     End Function
-    Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal ffmpegFile As SFile, ByVal f As SFile,
+    Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal ffmpegFile As SFile, ByVal f As SFile, ByRef Logger As ILogProvider) As SFile
-                                    ByRef Logger As ILogProvider) As SFile
         Try
             If Not URL.IsEmptyString Then
                 Using w As New WebClient
@@ -61,8 +61,8 @@ Friend NotInheritable Class M3U8
             End If
             Return Nothing
         Catch ex As Exception
-            Logger.Add(ex, "[M3U8.Download]")
+            If Not ex.HelpLink = 1 Then Logger.Add(ex, $"[M3U8.Download({URL}, {Appender}, {ffmpegFile}, {f})]")
-            Return Nothing
+            Throw ex
         End Try
     End Function
 End Class

SCrawler.Plugin.XVIDEOS/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("1.0.0.0")>
+<Assembly: AssemblyVersion("2022.10.12.0")>
-<Assembly: AssemblyFileVersion("1.0.0.0")>
+<Assembly: AssemblyFileVersion("2022.10.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.Plugin.XVIDEOS/SettingsForm.Designer.vb

@@ -25,14 +25,9 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
         Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
         Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(SettingsForm))
         Me.LIST_DOMAINS = New System.Windows.Forms.ListBox()
-        Me.ToolbarTOP = New System.Windows.Forms.ToolStrip()
-        Me.BTT_ADD = New System.Windows.Forms.ToolStripButton()
-        Me.BTT_DELETE = New System.Windows.Forms.ToolStripButton()
         CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
         CONTAINER_MAIN.ContentPanel.SuspendLayout()
-        CONTAINER_MAIN.TopToolStripPanel.SuspendLayout()
         CONTAINER_MAIN.SuspendLayout()
-        Me.ToolbarTOP.SuspendLayout()
         Me.SuspendLayout()
         '
         'CONTAINER_MAIN
@@ -41,7 +36,7 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
         'CONTAINER_MAIN.ContentPanel
         '
         CONTAINER_MAIN.ContentPanel.Controls.Add(Me.LIST_DOMAINS)
-        CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 266)
+        CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 291)
         CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
         CONTAINER_MAIN.LeftToolStripPanelVisible = False
         CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
@@ -50,52 +45,15 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
         CONTAINER_MAIN.Size = New System.Drawing.Size(384, 291)
         CONTAINER_MAIN.TabIndex = 0
         '
-        'CONTAINER_MAIN.TopToolStripPanel
-        '
-        CONTAINER_MAIN.TopToolStripPanel.Controls.Add(Me.ToolbarTOP)
-        '
         'LIST_DOMAINS
         '
         Me.LIST_DOMAINS.Dock = System.Windows.Forms.DockStyle.Fill
         Me.LIST_DOMAINS.FormattingEnabled = True
         Me.LIST_DOMAINS.Location = New System.Drawing.Point(0, 0)
         Me.LIST_DOMAINS.Name = "LIST_DOMAINS"
-        Me.LIST_DOMAINS.Size = New System.Drawing.Size(384, 266)
+        Me.LIST_DOMAINS.Size = New System.Drawing.Size(384, 291)
         Me.LIST_DOMAINS.TabIndex = 0
         '
-        'ToolbarTOP
-        '
-        Me.ToolbarTOP.Dock = System.Windows.Forms.DockStyle.None
-        Me.ToolbarTOP.GripStyle = System.Windows.Forms.ToolStripGripStyle.Hidden
-        Me.ToolbarTOP.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_ADD, Me.BTT_DELETE})
-        Me.ToolbarTOP.Location = New System.Drawing.Point(0, 0)
-        Me.ToolbarTOP.Name = "ToolbarTOP"
-        Me.ToolbarTOP.Size = New System.Drawing.Size(384, 25)
-        Me.ToolbarTOP.Stretch = True
-        Me.ToolbarTOP.TabIndex = 0
-        '
-        'BTT_ADD
-        '
-        Me.BTT_ADD.AutoToolTip = False
-        Me.BTT_ADD.BackColor = System.Drawing.Color.FromArgb(CType(CType(192, Byte), Integer), CType(CType(255, Byte), Integer), CType(CType(192, Byte), Integer))
-        Me.BTT_ADD.ForeColor = System.Drawing.Color.DarkGreen
-        Me.BTT_ADD.Image = CType(resources.GetObject("BTT_ADD.Image"), System.Drawing.Image)
-        Me.BTT_ADD.ImageTransparentColor = System.Drawing.Color.Magenta
-        Me.BTT_ADD.Name = "BTT_ADD"
-        Me.BTT_ADD.Size = New System.Drawing.Size(49, 22)
-        Me.BTT_ADD.Text = "Add"
-        '
-        'BTT_DELETE
-        '
-        Me.BTT_DELETE.AutoToolTip = False
-        Me.BTT_DELETE.BackColor = System.Drawing.Color.FromArgb(CType(CType(255, Byte), Integer), CType(CType(192, Byte), Integer), CType(CType(192, Byte), Integer))
-        Me.BTT_DELETE.ForeColor = System.Drawing.Color.Maroon
-        Me.BTT_DELETE.Image = CType(resources.GetObject("BTT_DELETE.Image"), System.Drawing.Image)
-        Me.BTT_DELETE.ImageTransparentColor = System.Drawing.Color.Magenta
-        Me.BTT_DELETE.Name = "BTT_DELETE"
-        Me.BTT_DELETE.Size = New System.Drawing.Size(60, 22)
-        Me.BTT_DELETE.Text = "Delete"
-        '
         'SettingsForm
         '
         Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -112,18 +70,11 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
         Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
         Me.Text = "Settings"
         CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-        CONTAINER_MAIN.TopToolStripPanel.ResumeLayout(False)
-        CONTAINER_MAIN.TopToolStripPanel.PerformLayout()
         CONTAINER_MAIN.ResumeLayout(False)
         CONTAINER_MAIN.PerformLayout()
-        Me.ToolbarTOP.ResumeLayout(False)
-        Me.ToolbarTOP.PerformLayout()
         Me.ResumeLayout(False)
 
     End Sub
 
     Private WithEvents LIST_DOMAINS As Windows.Forms.ListBox
-    Private WithEvents ToolbarTOP As Windows.Forms.ToolStrip
-    Private WithEvents BTT_ADD As Windows.Forms.ToolStripButton
-    Private WithEvents BTT_DELETE As Windows.Forms.ToolStripButton
 End Class

SCrawler.Plugin.XVIDEOS/SettingsForm.resx

@@ -120,60 +120,7 @@
   <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
-  <metadata name="ToolbarTOP.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
-    <value>17, 17</value>
-  </metadata>
   <assembly alias="System.Drawing" name="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
-  <data name="BTT_ADD.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
-    <value>
-        iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
-        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAN+SURBVEhLrZVZSFRRGMdvKa6oE5Y1bpTtRY2pJYk3ayyd
-        xgrvaDXlQotLKVqSo7QR2QIREUX0EgUSET1V9NTyYBHt6ZTOqNPY7DomWNbriX/fuYzkg2jY/eAPM5zD
-        //+d7/7OvcJkFXIlqDX0ahDGE18LbJt6cSOtWWTaDhGbPonQdZEsIrY7cuWQwLapFzfZ9FlkES8ERL8W
-        EPteQMLHYFR69coFUNcs5o2AWe+nYW5HKFK7otHkL1IuoKBbZLzzFHMo0qwxWN8zG2eHSpULkGxalmIO
-        w2qrCvm2eJr/AlwZrlIuYN9XHcuwqKC3JWC3czGqXKm4OVKvXMARTyHTfUlEmWspatxp9D8Ld0ZM/xYw
-        EeejMnkMv0tdS1DnzkBzv4jT/jy0fm/EzLvhE0p9P7JV7jB3lHHiu6BHhMGmRbkjH3WebTjq245z/nL6
-        nYFjvnVo8efjzKAOrcONuD1skoNuDNfj8lAFWgZL0OQrphEWoMa1E+oHkZADuHn0K2L8rYDEjhBoCENO
-        itSXgr2u5TjkycTx/g1koMP5Qb0ccHogj8xEVLlTsaNvIXJ750BjiUZSewhU5LPTIv0N4J1zc76osUTR
-        ZjWKAuaHvZk44dPijF9HAXk4OaCFyZdNzyIde5zLUGifh5yeOKzoikJ8RzBU7wSE06Us+jQmoMAqIpnM
-        U6mDXJtaxrDSrUGDdw2O9a/HCTI96suByZuNw57VqKa1MtcybLXPxTo6qWzeHgx+Gbn5tCcCpHdjAnbZ
-        NmKtNZYwTESJYwnq6WE2EinNZMhNj3izZDXQqKppJJymLfZkZHfHYWlnJNQfghBFIw5rEyA8Jj2ggJdj
-        Ag769cw0ZGAtP8rYhZH97NLPanbpVxW7/PMAu/atlnSQzNeg0qWB0bkIensSmj3lqHUaf1d4ilipU2LG
-        LxIzWCUmfSZ9JL2VmBzAMR0PsVHxTbeGmlmleyWMjoXYTKfMpNPWuI3y2oTimE5WfOP1bw1sBz2X/N4E
-        pNONnm8Oxx6HQTYJbJt6cZOLAzUsrzdeftHxt2kskbLbFpjx/xY3OeWtYKssMUgkUmYQzvzbUDzK+f8W
-        N6l3lbCk9lCoCMOI5wKCnhIlZgUD9tkNjJuHUefTOYaPKOCNggEl3RILp84FukDCQ9I9CnihYEBxpySP
-        hNiWL5DURnqmVACxzI3G1aScC8If1IGynvf45pQAAAAASUVORK5CYII=
-</value>
-  </data>
-  <data name="BTT_DELETE.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
-    <value>
-        iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
-        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAVFSURBVEhLjZVrTJNXGMcLQmdHO6Sdt6ygAyYgN+1bS+uE
-        4ASlxVfSUkFkRMyouILinShGYzTRKCZL9mH7sH3YMj+YzUSXZXNeQOcEFkFhysbVQWlL6dsLvMwtWaBn
-        /1OKl1A2nuSX8p7znP/z5znnPa+ARntubkRjXNxnd5Ys6b2xbNmn+XL5WxgO80/+T/SoVOo/YmLu9S9d
-        +uhqfLwRQxEgxD9J44nZLH6Qm/tNT20tGbh4kTw9dMh3Q6NpyJBI3sF0+FRW8OhimHX2kpJB/vx5Ml5f
-        TwaKitwfJyZWxURGRmF6qkhDXNwn3QcOkKenTpGBkyeJFcmWI0fIdbX6XppEsgIpQYv0aTTvQdzCX7hA
-        eKzl6+rI+IkTpJdlPd/K5R8hRQrCBHdkMisVh3MyeOwYsSJp+OxZYq2t9d1QqX5OlUgSkPhKEbRlo724
-        2EqdU/FxiPOHD5NxGPOWlZHGxYs7kaYGEsGt6OhLvRUVZAAFLMB69CixY8HImTPEjnbdVqubU8TiRCT7
-        i/QoFBuHi4qsY+fOER5mxmGKx7rx/fvJ6O7dpEGl4splsq+RqgNSgWH58mXXlcp7/Tt3+oaQZAPDcDOC
-        PXGhZc6DB8kdFFktFic9ZJhcm9FoHzt92t8SHo55tJevqSFjEL+rUrkTRKIvIWwCaYBuuCCckckSUaTZ
-        smMHse/ZQxzV1WRk3z7CoZiH/jfV1b4WjaZjsKDAyh8/7nfMY45HLl9VNSWuVLqTRKJL0KsAyQHxUOAP
-        4WqpNOlHhaJpqLTU56isJE7gMpuJGwJeFPPu3UvG4HaMCuOZx7NfHO1t0mg86WIxFafOV4L54MVRDYRQ
-        JZWuvJWe3mzfupU4y8uJC3hMJuKFw1EUHMUGjhYUEO/69cSr0RA3aMrI8K6WSC5j/S5AnYvADPHpEK6J
-        ikpuSEtrtrHshHvbNuLSaomLYQgnlxNOIiFceDjh5s0jI2Kx73ZsrEcdFUU3tBKkgNfBrOLT4W/Xg/T0
-        9mEIOqgYcAIq7CcsjPzOMH8bExO/R/6HIBXMSdwf9rIys33dOotDJJopDuiYNT5+opFl+5OiowuwJBLM
-        TdxrMh3kNmzgHHA/m7gNDAJLcjJ5VFjYuX3t2lVY+p/Xij9cJlMdt2mTyxERMUPcid7TdlHxAdAbGkqe
-        gO5Vq3zthYW/GdVq2qbZi3h27TrG5eW5gzl3ymRkKC/vH3tCgo+K9wTE20ATeMwwk+1GY+f22YrgKNZx
-        Wq3HIRb7hV9py6JFpEun+/N0VlZL2+bNw5akJPIYoq3gPvgJNIL2NWsmW43Gjg1pafSCfHHVuysqajmd
-        zoNjN1N84ULSk5//rIphbiG1Oi8l5f1Ovb6/C72fFm8AN8F10KpSTbYUFrZ9wbJ044UgROA2GKwjCxYE
-        dU7FKxWKm0g0A7qR0n3Z2e92GAx9Hampz8V/AN+Ba6A9K2ui1WDoRu5yIBQ4jcYud0xMUOdmhrmNpKqA
-        +PSXSrg/J0eNje1rS031vSx+JSSEtGZmTtxlWRvyckGUoKW0tGSYZV0uiL7k/K89wcWnQ1iTnZ2Bje3F
-        C0muBsRxbUxe1mqHdCtWfI6cqQKI+TeLio5bt2zxONVqX49O96xGqWzA+Gzi0yE0ZWYqaDtase4XjWby
-        ilZre0MkqsecHky1CBHCxMZGPjQYvnqq1zsv5+Tg9IXWYJwBYjpPk2YJ4bXi4g9+1ett91mW0yYk0G+B
-        AcgBPa7P19I/FgAlyA/8zvX1fw3Qo0lPzmbwNgj6ws0D1DH9WNNf+jyXoCbo/f9mAFowYEwg+Bc5Ntw7
-        FHW1qQAAAABJRU5ErkJggg==
-</value>
-  </data>
   <data name="$this.Icon" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         AAABAAUAEBAAAAEAIABoBAAAVgAAABgYAAABACAAiAkAAL4EAAAgIAAAAQAIAKgIAABGDgAAMDAAAAEA

SCrawler.Plugin.XVIDEOS/SettingsForm.vb

@@ -8,28 +8,24 @@
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Forms
-Public Class SettingsForm : Implements IOkCancelToolbar
+Public Class SettingsForm
-    Private ReadOnly MyDefs As DefaultFormProps
+    Private WithEvents MyDefs As DefaultFormOptions
     Private ReadOnly Property Settings As SiteSettings
-    Friend Sub New(ByRef s As SiteSettings)
+    Friend Sub New(ByRef s As SiteSettings, ByRef Design As XML.XmlFile)
         InitializeComponent()
-        MyDefs = New DefaultFormProps
         Settings = s
+        MyDefs = New DefaultFormOptions(Me, Design)
     End Sub
     Private Sub SettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
-        Try
         With MyDefs
-                .MyViewInitialize(Me, Settings.Design, True)
+            .MyViewInitialize(True)
+            .AddEditToolbar({EditToolbar.ControlItem.Add, EditToolbar.ControlItem.Delete})
             .AddOkCancelToolbar()
-                .DelegateClosingChecker()
             If Settings.Domains.Count > 0 Then Settings.Domains.ForEach(Sub(d) LIST_DOMAINS.Items.Add(d))
             .EndLoaderOperations()
         End With
-        Catch ex As Exception
-            MyDefs.InvokeLoaderError(ex)
-        End Try
     End Sub
-    Private Sub ToolbarBttOK() Implements IOkCancelToolbar.ToolbarBttOK
+    Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
         Settings.Domains.Clear()
         With LIST_DOMAINS
             If .Items.Count > 0 Then
@@ -39,10 +35,7 @@ Public Class SettingsForm : Implements IOkCancelToolbar
         Settings.UpdateDomains()
         MyDefs.CloseForm()
     End Sub
-    Private Sub ToolbarBttCancel() Implements IOkCancelToolbar.ToolbarBttCancel
+    Private Sub MyDefs_ButtonAddClick(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonAddClick
-        MyDefs.CloseForm(Windows.Forms.DialogResult.Cancel)
-    End Sub
-    Private Sub BTT_ADD_Click(sender As Object, e As EventArgs) Handles BTT_ADD.Click
         Dim nd$ = InputBoxE("Enter a new domain using the pattern [xvideos.com]:", "New domain")
         If Not nd.IsEmptyString Then
             If Not LIST_DOMAINS.Items.Contains(nd) Then
@@ -52,11 +45,10 @@ Public Class SettingsForm : Implements IOkCancelToolbar
             End If
         End If
     End Sub
-    Private Sub BTT_DELETE_Click(sender As Object, e As EventArgs) Handles BTT_DELETE.Click
+    Private Sub MyDefs_ButtonDeleteClickE(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonDeleteClickE
         If _LatestSelected.ValueBetween(0, LIST_DOMAINS.Items.Count - 1) Then
             Dim n$ = LIST_DOMAINS.Items(_LatestSelected)
-            If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
+            If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?", "Removing domains"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
-                        "Removing domains"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
                 LIST_DOMAINS.Items.RemoveAt(_LatestSelected)
                 MsgBoxE($"Domain [{n}] removed")
             Else

SCrawler.Plugin.XVIDEOS/SiteSettings.vb

@@ -26,7 +26,7 @@ Public Class SiteSettings : Implements ISiteSettings
     Public Property Logger As ILogProvider Implements ISiteSettings.Logger
 #Region "M3U8"
     Private ReadOnly OS64 As Boolean
-    Private ReadOnly FfmpegExists As Boolean
+    Friend ReadOnly FfmpegExists As Boolean
     Friend ReadOnly FfmpegFile As SFile
     Friend ReadOnly Property UseM3U8 As Boolean
         Get
@@ -41,7 +41,6 @@ Public Class SiteSettings : Implements ISiteSettings
     Public ReadOnly Property Responser As Response
     Private Const DomainsDefault As String = "xvideos.com|xnxx.com"
     Private _Initialized As Boolean = False
-    Friend Design As XmlFile
     Public Sub New()
         Responser = New Response($"Settings\Responser_{Site}.xml")
         With Responser
@@ -78,12 +77,12 @@ Public Class SiteSettings : Implements ISiteSettings
             If Not ACheck(SiteDomains.Value) Then SiteDomains.Value = DomainsDefault
             Domains.ListAddList(CStr(SiteDomains.Value).Split("|"), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
             Domains.ListAddList(DomainsDefault.Split("|"), LAP.NotContainsOnly)
-            SiteDomains.Value = Domains.ListToString(, "|")
+            SiteDomains.Value = Domains.ListToString("|")
             _DomainsUpdateInProgress = False
         End If
     End Sub
 #Region "Downloading"
-    Public Function Available(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.Available
+    Public Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
         Return UseM3U8
     End Function
     Public Function ReadyToDownload(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.ReadyToDownload
@@ -98,6 +97,10 @@ Public Class SiteSettings : Implements ISiteSettings
     Public Sub DownloadDone(ByVal What As ISiteSettings.Download) Implements ISiteSettings.DownloadDone
     End Sub
 #End Region
+    Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
+    End Sub
+    Public Sub EndEdit() Implements ISiteSettings.EndEdit
+    End Sub
     Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
     End Sub
     Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -109,10 +112,8 @@ Public Class SiteSettings : Implements ISiteSettings
     Public Sub Reset() Implements ISiteSettings.Reset
     End Sub
     Public Sub OpenSettingsForm() Implements ISiteSettings.OpenSettingsForm
-        Using f As New SettingsForm(Me)
+        Using Design As New XmlFile("Settings\Design_XVIDEOS.xml")
-            Design = New XmlFile("Settings\Design_XVIDEOS.xml")
+            Using f As New SettingsForm(Me, Design) : f.ShowDialog() : End Using
-            f.ShowDialog()
-            Design.Dispose()
         End Using
     End Sub
     Public Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean) Implements ISiteSettings.UserOptions
@@ -182,4 +183,7 @@ Public Class SiteSettings : Implements ISiteSettings
         End If
         Return Nothing
     End Function
+    Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+        Return String.Empty
+    End Function
 End Class

SCrawler.Plugin.XVIDEOS/UserData.vb

@@ -10,10 +10,12 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Tools.WEB
 Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports UStates = SCrawler.Plugin.PluginUserMedia.States
+Imports UTypes = SCrawler.Plugin.PluginUserMedia.Types
 Public Class UserData : Implements IPluginContentProvider
 #Region "Interface declarations"
-    Public Event ProgressChanged(Count As Integer) Implements IPluginContentProvider.ProgressChanged
+    Public Event ProgressChanged(ByVal Count As Integer) Implements IPluginContentProvider.ProgressChanged
-    Public Event TotalCountChanged(Count As Integer) Implements IPluginContentProvider.TotalCountChanged
+    Public Event TotalCountChanged(ByVal Count As Integer) Implements IPluginContentProvider.TotalCountChanged
     Public Property Thrower As IThrower Implements IPluginContentProvider.Thrower
     Public Property LogProvider As ILogProvider Implements IPluginContentProvider.LogProvider
     Public Property ESettings As ISiteSettings Implements IPluginContentProvider.Settings
@@ -35,7 +37,8 @@ Public Class UserData : Implements IPluginContentProvider
     Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
     Public Property DataPath As String Implements IPluginContentProvider.DataPath
     Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
-    Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
+    Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
+    Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
 #End Region
 #Region "Interface exchange options"
     Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -54,14 +57,21 @@ Public Class UserData : Implements IPluginContentProvider
     Private Property Responser As Response
     Public Sub GetMedia() Implements IPluginContentProvider.GetMedia
         Try
-            If Not Settings.UseM3U8 Then LogProvider.Add("File [ffmpeg.exe] not found") : Exit Sub
+            If Not Settings.UseM3U8 Then
+                If Settings.FfmpegExists Then
+                    LogProvider.Add($"XVIDEOS [{Name}]: The plugin only works with x64 OS.")
+                Else
+                    LogProvider.Add($"XVIDEOS [{Name}]: File [ffmpeg.exe] not found")
+                End If
+                Exit Sub
+            End If
             If Not Responser Is Nothing Then Responser.Dispose()
             Responser = New Response
             Responser.Copy(Settings.Responser)
 
             Dim NextPage% = 0
             Dim r$
-            Dim j As EContainer, jj As EContainer
+            Dim jj As EContainer
             Dim e As ErrorsDescriber = EDP.ThrowException
             Dim user$ = Settings.GetUserUrl(Name, False)
             Dim p As PluginUserMedia
@@ -71,9 +81,8 @@ Public Class UserData : Implements IPluginContentProvider
                 Thrower.ThrowAny()
                 r = Responser.GetResponse($"https://www.xvideos.com/{user}/videos/new/{If(NextPage = 0, String.Empty, NextPage)}",, e)
                 If Not r.IsEmptyString Then
-                    If Not EnvirSet Then UserExists = True : UserSuspended = False
+                    If Not EnvirSet Then UserExists = True : UserSuspended = False : EnvirSet = True
-                    j = JsonDocument.Parse(r).XmlIfNothing
+                    With JsonDocument.Parse(r).XmlIfNothing
-                    With j
                         If .Contains("videos") Then
                             With .Item("videos")
                                 If .Count > 0 Then
@@ -84,12 +93,16 @@ Public Class UserData : Implements IPluginContentProvider
                                             .URL = $"https://www.xvideos.com{jj.Value("u")}"
                                         }
                                         If Not p.PostID.IsEmptyString And Not jj.Value("u").IsEmptyString Then
-                                            If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else Exit Do
+                                            If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else .Dispose() : Exit Do
                                         End If
                                     Next
+                                Else
+                                    .Dispose()
+                                    Exit Do
                                 End If
                             End With
                         Else
+                            .Dispose()
                             Exit Do
                         End If
                         .Dispose()
@@ -102,9 +115,7 @@ Public Class UserData : Implements IPluginContentProvider
             If TempMediaList.Count > 0 Then
                 For i% = 0 To TempMediaList.Count - 1
                     Thrower.ThrowAny()
-                    With TempMediaList(i)
+                    With TempMediaList(i) : TempMediaList(i) = GetVideoData(.URL, Responser, Settings.DownloadUHD.Value, .PostID, LogProvider) : End With
-                        TempMediaList(i) = GetVideoData(.URL, Responser, Settings.DownloadUHD.Value, .PostID, LogProvider)
-                    End With
                 Next
                 TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
             End If
@@ -146,8 +157,8 @@ Public Class UserData : Implements IPluginContentProvider
                         Dim t$ = RegexReplace(r, VideoTitleRegex)
                         r = resp.GetResponse(m,, EDP.ThrowException)
                         If Not r.IsEmptyString Then
-                            Dim ls As List(Of VSize) = FNF.RegexFields(Of VSize)(r, {M3U8Reparse}, {1, 2})
+                            Dim ls As List(Of VSize) = RegexFields(Of VSize)(r, {M3U8Reparse}, {1, 2})
-                            If ls.ListExists And DownloadUHD Then ls.RemoveAll(Function(v) v.Size > 1080)
+                            If ls.ListExists And Not DownloadUHD Then ls.RemoveAll(Function(v) v.Size > 1080)
                             If ls.ListExists Then
                                 ls.Sort()
                                 m = $"{appender}/{ls(0).Value}"
@@ -164,7 +175,7 @@ Public Class UserData : Implements IPluginContentProvider
                                 End If
                                 If Not m.IsEmptyString Then
                                     Return New PluginUserMedia With {
-                                        .ContentType = PluginUserMedia.Types.m3u8,
+                                        .ContentType = UTypes.m3u8,
                                         .PostID = pID,
                                         .URL = m,
                                         .File = $"{t}.mp4",
@@ -194,9 +205,15 @@ Public Class UserData : Implements IPluginContentProvider
                     m = TempMediaList(i)
                     f = m.File
                     f.Path = DefPath
+                    m.DownloadState = UStates.Tried
+                    Try
                         f = M3U8.Download(m.URL, m.SpecialFolder, Settings.FfmpegFile, f, LogProvider)
                         m.File = f
-                    If Not f.IsEmptyString Then m.DownloadState = PluginUserMedia.States.Downloaded
+                        m.DownloadState = UStates.Downloaded
+                    Catch ex As Exception
+                        m.DownloadState = UStates.Missing
+                        m.Attempts += 1
+                    End Try
                     TempMediaList(i) = m
                     RaiseEvent ProgressChanged(1)
                 Next

SCrawler.PluginProvider/.editorconfig

@@ -1,3 +1,3 @@
 [*.vb]
 # Modifier preferences
-file_header_template = Copyright (C) 2022  Andy\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>
+file_header_template = Copyright (C) 2023  Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>

SCrawler.PluginProvider/Attributes/Attributes.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -32,6 +32,10 @@ Namespace Plugin.Attributes
         Public Property AllowNull As Boolean = True
         ''' <summary>Offset the control from the left border of the form.<br/>Default: 100</summary>
         Public Property LeftOffset As Integer = 100
+        ''' <summary>This control is an information label.<br/>Default: <see langword="False"/></summary>
+        Public Property IsInformationLabel As Boolean = False
+        ''' <summary>Label text alignment.<br/>Default: <see cref="Drawing.ContentAlignment.TopCenter"/></summary>
+        Public Property LabelTextAlign As Drawing.ContentAlignment = Drawing.ContentAlignment.TopCenter
         ''' <summary>This is an authorization property</summary>
         Public Property IsAuth As Boolean = False
         ''' <summary>Initialize a new property option attribute</summary>

SCrawler.PluginProvider/Interfaces/IPluginContentProvider.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -26,7 +26,8 @@ Namespace Plugin
         Property SeparateVideoFolder As Boolean
         Property DataPath As String
         Property PostsNumberLimit As Integer?
-        Property PostsDateLimit As Date?
+        Property DownloadDateFrom As Date?
+        Property DownloadDateTo As Date?
         Function ExchangeOptionsGet() As Object
         Sub ExchangeOptionsSet(ByVal Obj As Object)
         Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))

SCrawler.PluginProvider/Interfaces/ISiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -21,8 +21,9 @@ Namespace Plugin
         Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String
         Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
         Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
-        Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia)
+        Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
         Function GetInstance(ByVal What As Download) As IPluginContentProvider
+        Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
 #Region "XML Support"
         Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))
 #End Region
@@ -31,9 +32,11 @@ Namespace Plugin
         Sub EndInit()
         Sub BeginUpdate()
         Sub EndUpdate()
+        Sub BeginEdit()
+        Sub EndEdit()
 #End Region
 #Region "Site availability"
-        Function Available(ByVal What As Download) As Boolean
+        Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
         Function ReadyToDownload(ByVal What As Download) As Boolean
 #End Region
 #Region "Downloading"

SCrawler.PluginProvider/My Project/AssemblyInfo.vb

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
 <Assembly: AssemblyDescription("Plugin provider for SCrawler")>
 <Assembly: AssemblyCompany("AndyProgram")>
 <Assembly: AssemblyProduct("SCrawler.PluginProvider")>
-<Assembly: AssemblyCopyright("Copyright ©  2022")>
+<Assembly: AssemblyCopyright("Copyright © 2023")>
 <Assembly: AssemblyTrademark("AndyProgram")>
 
 <Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("3.0.0.2")>
+<Assembly: AssemblyVersion("2022.10.18.0")>
-<Assembly: AssemblyFileVersion("3.0.0.2")>
+<Assembly: AssemblyFileVersion("2022.10.18.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.PluginProvider/ObjectInterfaces/ILogProvider.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/ObjectInterfaces/IThrower.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/Objects/ExchangeOptions.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -13,13 +13,13 @@ Namespace Plugin
         Public HostKey As String
         Public IsChannel As Boolean
         Public Exists As Boolean
-        Public Sub New(ByVal Site As String, ByVal _Name As String)
+        Public Sub New(ByVal Site As String, ByVal Name As String)
-            UserName = _Name
+            UserName = Name
             SiteName = Site
         End Sub
-        Public Sub New(ByVal Site As String, ByVal _Name As String, ByVal _IsChannel As Boolean)
+        Public Sub New(ByVal Site As String, ByVal Name As String, ByVal IsChannel As Boolean)
-            Me.New(Site, _Name)
+            Me.New(Site, Name)
-            IsChannel = _IsChannel
+            Me.IsChannel = IsChannel
         End Sub
     End Structure
 End Namespace

SCrawler.PluginProvider/Objects/PluginUserMedia.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -17,7 +17,7 @@ Namespace Plugin
             GIF = 50
             m3u8 = 100
         End Enum
-        Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : End Enum
+        Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : Missing = 4 : End Enum
         Public ContentType As Integer
         Public URL As String
         Public MD5 As String
@@ -26,5 +26,6 @@ Namespace Plugin
         Public PostID As String
         Public PostDate As Date?
         Public SpecialFolder As String
+        Public Attempts As Integer
     End Structure
 End Namespace

SCrawler.PluginProvider/Objects/PropertyData.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -10,9 +10,9 @@ Namespace Plugin
     Public Structure PropertyData
         Public ReadOnly Name As String
         Public ReadOnly Value As Object
-        Public Sub New(ByVal _Name As String, ByVal _Value As Object)
+        Public Sub New(ByVal Name As String, ByVal Value As Object)
-            Name = _Name
+            Me.Name = Name
-            Value = _Value
+            Me.Value = Value
         End Sub
     End Structure
 End Namespace

SCrawler.PluginProvider/Objects/PropertyValue.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.sln

@@ -17,9 +17,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 EndProject
 Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.PluginProvider", "SCrawler.PluginProvider\SCrawler.PluginProvider.vbproj", "{D4650F6B-5A54-44B6-999B-6C675B7116B1}"
 EndProject
-Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.Plugin.LPSG", "SCrawler.Plugin.LPSG\SCrawler.Plugin.LPSG.vbproj", "{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}"
+Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "PersonalUtilities.Notifications", "..\..\MyUtilities\PersonalUtilities.Notifications\PersonalUtilities.Notifications.vbproj", "{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}"
-EndProject
-Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.Plugin.XVIDEOS", "SCrawler.Plugin.XVIDEOS\SCrawler.Plugin.XVIDEOS.vbproj", "{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
@@ -67,30 +65,18 @@ Global
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x64.Build.0 = Release|x64
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x86.ActiveCfg = Release|x86
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x86.Build.0 = Release|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x64.ActiveCfg = Debug|x64
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|x64.ActiveCfg = Debug|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x64.Build.0 = Debug|x64
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|x64.Build.0 = Debug|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x86.ActiveCfg = Debug|x86
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|x86.ActiveCfg = Debug|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x86.Build.0 = Debug|x86
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|x86.Build.0 = Debug|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|Any CPU.ActiveCfg = Release|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|Any CPU.Build.0 = Release|Any CPU
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|Any CPU.Build.0 = Release|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x64.ActiveCfg = Release|x64
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|x64.ActiveCfg = Release|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x64.Build.0 = Release|x64
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|x64.Build.0 = Release|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x86.ActiveCfg = Release|x86
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|x86.ActiveCfg = Release|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x86.Build.0 = Release|x86
+		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Release|x86.Build.0 = Release|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x64.ActiveCfg = Debug|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x64.Build.0 = Debug|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x86.ActiveCfg = Debug|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x86.Build.0 = Debug|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|Any CPU.ActiveCfg = Release|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|Any CPU.Build.0 = Release|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x64.ActiveCfg = Release|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x64.Build.0 = Release|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x86.ActiveCfg = Release|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x86.Build.0 = Release|x86
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE

SCrawler/.editorconfig

@@ -123,4 +123,4 @@ insert_final_newline=false
 [*.vb]
 # Modifier preferences
 visual_basic_preferred_modifier_order = Partial,Default,Private,Protected,Public,Friend,NotOverridable,Overridable,MustOverride,Overloads,Overrides,MustInherit,NotInheritable,Static,Shared,Shadows,ReadOnly,WriteOnly,Dim,Const,WithEvents,Widening,Narrowing,Custom,Async:suggestion
-file_header_template = Copyright (C) 2022  Andy\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>
+file_header_template = Copyright (C) 2023  Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>

SCrawler/API/Base/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler/API/Base/DownDetector.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -16,7 +16,7 @@ Namespace API.Base
         Friend Structure Data : Implements IRegExCreator, IComparable(Of Data)
             Friend [Date] As Date
             Friend Value As Integer
-            Friend Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
                 If ParamsArray.ListExists Then
                     Try : [Date] = Date.Parse(ParamsArray(0)) : Catch : End Try
                     If ParamsArray.Length > 1 Then Value = AConvert(Of Integer)(ParamsArray(1), 0)
@@ -26,32 +26,39 @@ Namespace API.Base
             Public Overrides Function ToString() As String
                 Return $"{AConvert(Of String)([Date], ADateTime.Formats.BaseDateTime, String.Empty)} [{Value}]"
             End Function
-            Friend Function CompareTo(ByVal Other As Data) As Integer Implements IComparable(Of Data).CompareTo
+            Private Function CompareTo(ByVal Other As Data) As Integer Implements IComparable(Of Data).CompareTo
                 Return [Date].CompareTo(Other.Date) * -1
             End Function
         End Structure
         Friend Shared Function GetData(ByVal Site As String) As List(Of Data)
             Try
                 Dim l As List(Of Data) = Nothing
+                Dim l2 As List(Of Data) = Nothing
                 Using w As New WebClient
                     Dim r$ = w.DownloadString($"https://downdetector.co.uk/status/{Site}/")
                     If Not r.IsEmptyString Then
-                        l = FNF.RegexFields(Of Data)(r, {Params}, {1, 2})
+                        l = RegexFields(Of Data)(r, {Params}, {1, 2})
                         If l.ListExists(2) Then
-                            Dim lDate As Date = l(0).Date
-                            Dim i%
-                            Dim indx% = -1
-                            For i = 1 To l.Count - 1
-                                If l(i).Date < lDate Then indx = i : Exit For Else lDate = l(i).Date
-                            Next
-                            If indx >= 0 Then
-                                For i = indx To 0 Step -1 : l.RemoveAt(i) : Next
-                            End If
                             l.Sort()
+                            l2 = New List(Of Data)
+                            Dim d As Data
+                            Dim eDates As New List(Of Date)
+                            Dim MaxValue As Func(Of Date, Integer) = Function(dd) (From ddd In l Where ddd.Date = dd Select ddd.Value).DefaultIfEmpty(0).Max
+                            For i% = 0 To l.Count - 1
+                                If Not eDates.Contains(l(i).Date) Then
+                                    d = l(i)
+                                    d.Value = MaxValue(d.Date)
+                                    l2.Add(d)
+                                    eDates.Add(d.Date)
+                                End If
+                            Next
+                            eDates.Clear()
+                            l.Clear()
+                            l2.Sort()
                         End If
                     End If
                 End Using
-                Return l
+                Return l2
             Catch ex As Exception
                 Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[DownDetector.GetData({Site})]")
             End Try

SCrawler/API/Base/ProfileSaved.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports SCrawler.Plugin.Hosts
 Imports System.Threading
 Imports PersonalUtilities.Forms.Toolbars
+Imports SCrawler.Plugin.Hosts
 Imports PDownload = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
     Friend NotInheritable Class ProfileSaved
@@ -21,17 +21,14 @@ Namespace API.Base
         Friend Sub Download(ByVal Token As CancellationToken)
             Try
                 If HOST.Source.ReadyToDownload(PDownload.SavedPosts) Then
-                    If HOST.Available(PDownload.SavedPosts) Then
+                    If HOST.Available(PDownload.SavedPosts, False) Then
                         HOST.DownloadStarted(PDownload.SavedPosts)
                         Dim u As New UserInfo With {.Plugin = HOST.Key, .Site = HOST.Name, .SpecialPath = HOST.SavedPostsPath}
                         Using user As IUserData = HOST.GetInstance(PDownload.SavedPosts, Nothing, False, False)
-                            If Not user Is Nothing AndAlso (Not user.Name.IsEmptyString Or Not HOST.IsMyClass) Then
+                            If Not user Is Nothing AndAlso Not user.Name.IsEmptyString Then
                                 u.Name = user.Name
-                                With DirectCast(user, UserDataBase).User
-                                    u.IsChannel = .IsChannel
-                                    u.UpdateUserFile()
-                                End With
                                 With DirectCast(user, UserDataBase)
+                                    With .User : u.IsChannel = .IsChannel : u.UpdateUserFile() : End With
                                     .User = u
                                     .LoadUserInformation()
                                     .IsSavedPosts = True
@@ -40,7 +37,7 @@ Namespace API.Base
                                 End With
                                 HOST.BeforeStartDownload(user, PDownload.SavedPosts)
                                 user.DownloadData(Token)
-                                Progress.InformationTemporary = $"Images: {user.DownloadedPictures(False)}; Videos: {user.DownloadedVideos(False)}"
+                                Progress.InformationTemporary = $"{HOST.Name} Images: {user.DownloadedPictures(False)}; Videos: {user.DownloadedVideos(False)}"
                                 HOST.AfterDownload(user, PDownload.SavedPosts)
                             End If
                         End Using
@@ -48,7 +45,7 @@ Namespace API.Base
                         Progress.InformationTemporary = $"Host [{HOST.Name}] is unavailable"
                     End If
                 Else
-                    Progress.InformationTemporary = $"Host [{HOST.Name}] is nor ready"
+                    Progress.InformationTemporary = $"Host [{HOST.Name}] is not ready"
                 End If
             Catch ex As Exception
                 Progress.InformationTemporary = $"{HOST.Name} downloading error"

SCrawler/API/Base/SiteSettingsBase.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,17 +6,23 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports SCrawler.Plugin
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.WEB
+Imports SCrawler.Plugin
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
-    Friend MustInherit Class SiteSettingsBase : Implements ISiteSettings
+    Friend MustInherit Class SiteSettingsBase : Implements ISiteSettings, IResponserContainer
         Friend ReadOnly Property Site As String Implements ISiteSettings.Site
-        Friend Overridable ReadOnly Property Icon As Icon = Nothing Implements ISiteSettings.Icon
+        Friend Overridable ReadOnly Property Icon As Icon Implements ISiteSettings.Icon
-        Friend Overridable ReadOnly Property Image As Image = Nothing Implements ISiteSettings.Image
+        Friend Overridable ReadOnly Property Image As Image Implements ISiteSettings.Image
         Private Property Logger As ILogProvider = LogConnector Implements ISiteSettings.Logger
         Friend Overridable ReadOnly Property Responser As Response
+        Private Property IResponserContainer_Responser As Response Implements IResponserContainer.Responser
+            Get
+                Return Responser
+            End Get
+            Set : End Set
+        End Property
         Friend MustOverride Function GetInstance(ByVal What As Download) As IPluginContentProvider Implements ISiteSettings.GetInstance
         Friend Sub New(ByVal SiteName As String)
             Site = SiteName
@@ -25,7 +31,15 @@ Namespace API.Base
             Site = SiteName
             Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
             With Responser
-                If .File.Exists Then .LoadSettings() Else .CookiesDomain = CookiesDomain : .SaveSettings()
+                If .File.Exists Then
+                    If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .LoadSettings()
+                Else
+                    .CookiesDomain = CookiesDomain
+                    .Cookies = New CookieKeeper(.CookiesDomain) With {.EncryptKey = SettingsCLS.CookieEncryptKey}
+                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .SaveSettings()
+                End If
             End With
         End Sub
 #Region "XML"
@@ -36,11 +50,16 @@ Namespace API.Base
         Friend Overridable Sub BeginInit() Implements ISiteSettings.BeginInit
         End Sub
         Friend Overridable Sub EndInit() Implements ISiteSettings.EndInit
+            EncryptCookies.ValidateCookiesEncrypt(Responser)
         End Sub
         Friend Overridable Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
         End Sub
         Friend Overridable Sub EndUpdate() Implements ISiteSettings.EndUpdate
         End Sub
+        Friend Overridable Sub BeginEdit() Implements ISiteSettings.BeginEdit
+        End Sub
+        Friend Overridable Sub EndEdit() Implements ISiteSettings.EndEdit
+        End Sub
 #End Region
 #Region "Before and After Download"
         Friend Overridable Sub DownloadStarted(ByVal What As Download) Implements ISiteSettings.DownloadStarted
@@ -63,6 +82,9 @@ Namespace API.Base
             End If
             Return String.Empty
         End Function
+        Friend Overridable Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+            Return String.Empty
+        End Function
         Protected UserRegex As RParams = Nothing
         Friend Overridable Function IsMyUser(ByVal UserURL As String) As ExchangeOptions Implements ISiteSettings.IsMyUser
             Try
@@ -72,7 +94,7 @@ Namespace API.Base
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Base.SiteSettingsBase.IsMyUser]")
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]")
             End Try
         End Function
         Protected ImageVideoContains As String = String.Empty
@@ -83,17 +105,25 @@ Namespace API.Base
                 Return Nothing
             End If
         End Function
-        Friend Overridable Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia) Implements ISiteSettings.GetSpecialData
+        Friend Overridable Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable Implements ISiteSettings.GetSpecialData
             Return Nothing
         End Function
-        Friend Overridable Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetSpecialDataFile(ByVal Path As String, ByVal AskForPath As Boolean, ByRef SpecFolderObj As String) As SFile
-            Return Nothing
+            Dim f As SFile = Path.CSFileP
+            If f.Name.IsEmptyString Then f.Name = "OutputFile"
+#Disable Warning BC40000
+            If Path.CSFileP.IsEmptyString Or AskForPath Then f = SFile.SaveAs(f, "File destination",,,, EDP.ReturnValue) : SpecFolderObj = f.Path
+#Enable Warning
+            Return f
         End Function
 #End Region
 #Region "Ready, Available"
-        Friend Overridable Function Available(ByVal What As Download) As Boolean Implements ISiteSettings.Available
+        Friend Overridable Function BaseAuthExists() As Boolean
             Return True
         End Function
+        Friend Overridable Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
+            Return BaseAuthExists()
+        End Function
         Friend Overridable Function ReadyToDownload(ByVal What As Download) As Boolean Implements ISiteSettings.ReadyToDownload
             Return True
         End Function

SCrawler/API/Base/Structures.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,24 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Base
     Friend Module Structures
-        Friend Structure UserMedia : Implements IEquatable(Of UserMedia)
+        Friend Structure UserMedia : Implements IEquatable(Of UserMedia), IEContainerProvider
+#Region "XML Names"
+            Friend Const Name_MediaNode As String = "MediaData"
+            Private Const Name_MediaType As String = "Type"
+            Private Const Name_MediaState As String = "State"
+            Private Const Name_MediaAttempts As String = "Attempts"
+            Private Const Name_MediaURL As String = "URL"
+            Private Const Name_MediaHash As String = "Hash"
+            Private Const Name_MediaFile As String = "File"
+            Private Const Name_MediaPostID As String = "ID"
+            Private Const Name_MediaPostDate As String = "Date"
+            Private Const Name_SpecialFolder As String = "SpecialFolder"
+#End Region
             Friend Enum Types As Integer
                 Undefined = 0
                 [Picture] = 1
@@ -18,7 +33,7 @@ Namespace API.Base
                 GIF = 50
                 m3u8 = 100
             End Enum
-            Friend Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : End Enum
+            Friend Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : Missing = 4 : End Enum
             Friend [Type] As Types
             Friend URL_BASE As String
             Friend URL As String
@@ -27,31 +42,60 @@ Namespace API.Base
             Friend Post As UserPost
             Friend PictureOption As String
             Friend State As States
+            Friend Attempts As Integer
             ''' <summary>
             ''' SomeFolder<br/>
             ''' SomeFolder\SomeFolder2
             ''' </summary>
             Friend SpecialFolder As String
-            Friend Sub New(ByVal _URL As String)
+            Friend Sub New(ByVal URL As String)
-                URL = _URL
+                Me.URL = URL
-                URL_BASE = _URL
+                URL_BASE = URL
                 File = URL
                 Type = Types.Undefined
             End Sub
-            Friend Sub New(ByVal _URL As String, ByVal _Type As Types)
+            Friend Sub New(ByVal URL As String, ByVal Type As Types)
-                Me.New(_URL)
+                Me.New(URL)
-                [Type] = _Type
+                Me.Type = Type
             End Sub
             Friend Sub New(ByVal m As Plugin.PluginUserMedia)
-                If Not IsNothing(m) Then
                 [Type] = m.ContentType
                 URL = m.URL
+                URL_BASE = URL
                 MD5 = m.MD5
                 File = m.File
                 Post = New UserPost With {.ID = m.PostID, .[Date] = m.PostDate}
                 State = m.DownloadState
                 SpecialFolder = m.SpecialFolder
+                Attempts = m.Attempts
+            End Sub
+            Friend Sub New(ByVal e As EContainer, ByVal UserInstance As IUserData)
+                Type = e.Attribute(Name_MediaType).Value.FromXML(Of Integer)(CInt(Types.Undefined))
+                State = e.Attribute(Name_MediaState).Value.FromXML(Of Integer)(CInt(States.Downloaded))
+                Attempts = e.Attribute(Name_MediaAttempts).Value.FromXML(Of Integer)(0)
+                URL = e.Attribute(Name_MediaURL).Value
+                URL_BASE = e.Value
+                MD5 = e.Attribute(Name_MediaHash).Value
+                File = e.Attribute(Name_MediaFile).Value
+
+                Dim vp As Boolean? = Nothing
+                Dim upath$ = String.Empty
+                If Not UserInstance Is Nothing Then
+                    With DirectCast(UserInstance, UserDataBase)
+                        vp = .SeparateVideoFolder
+                        upath = .MyFile.CutPath.PathWithSeparator
+                    End With
                 End If
+
+                SpecialFolder = e.Attribute(Name_SpecialFolder).Value
+                If Not SpecialFolder.IsEmptyString Then upath &= $"{SpecialFolder}\"
+                If vp.HasValue AndAlso vp.Value Then upath &= $"Video\"
+                If Not upath.IsEmptyString Then File = $"{upath.CSFilePS}{File.File}"
+
+                Post = New UserPost With {
+                    .ID = e.Attribute(Name_MediaPostID).Value,
+                    .[Date] = AConvert(Of Date)(e.Attribute(Name_MediaPostDate).Value, ParsersDataDateProvider, Nothing)
+                }
             End Sub
             Public Shared Widening Operator CType(ByVal _URL As String) As UserMedia
                 Return New UserMedia(_URL)
@@ -59,6 +103,17 @@ Namespace API.Base
             Public Shared Widening Operator CType(ByVal m As UserMedia) As String
                 Return m.URL
             End Operator
+            Public Overrides Function GetHashCode() As Integer
+                If Not File.IsEmptyString Then
+                    Return File.GetHashCode
+                ElseIf Not URL_BASE.IsEmptyString Then
+                    Return URL_BASE.GetHashCode
+                ElseIf Not URL.IsEmptyString Then
+                    Return URL.GetHashCode
+                Else
+                    Return 0
+                End If
+            End Function
             Public Overrides Function ToString() As String
                 Return URL
             End Function
@@ -71,7 +126,8 @@ Namespace API.Base
                     .URL = URL,
                     .SpecialFolder = SpecialFolder,
                     .PostID = Post.ID,
-                    .PostDate = Post.Date
+                    .PostDate = Post.Date,
+                    .Attempts = Attempts
                 }
             End Function
             Friend Overloads Function Equals(ByVal Other As UserMedia) As Boolean Implements IEquatable(Of UserMedia).Equals
@@ -80,6 +136,19 @@ Namespace API.Base
             Public Overrides Function Equals(ByVal Obj As Object) As Boolean
                 Return Equals(CType(Obj, UserMedia))
             End Function
+            Friend Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer
+                Return New EContainer(Name_MediaNode, URL_BASE, {New EAttribute(Name_MediaType, CInt(Type)),
+                                                                 New EAttribute(Name_MediaState, CInt(State)),
+                                                                 New EAttribute(Name_MediaAttempts, Attempts),
+                                                                 New EAttribute(Name_MediaURL, URL),
+                                                                 New EAttribute(Name_MediaHash, MD5),
+                                                                 New EAttribute(Name_MediaFile, File.File),
+                                                                 New EAttribute(Name_SpecialFolder, SpecialFolder),
+                                                                 New EAttribute(Name_MediaPostID, Post.ID),
+                                                                 New EAttribute(Name_MediaPostDate, AConvert(Of String)(Post.Date, ParsersDataDateProvider, String.Empty))
+                                                                }
+                                     )
+            End Function
         End Structure
         Friend Structure UserPost : Implements IEquatable(Of UserPost), IComparable(Of UserPost)
             ''' <summary>Post ID</summary>
@@ -106,14 +175,14 @@ Namespace API.Base
             Friend Function CompareTo(ByVal Other As UserPost) As Integer Implements IComparable(Of UserPost).CompareTo
                 Return GetCompareValue(Me).CompareTo(GetCompareValue(Other))
             End Function
-#End Region
             Private Function GetCompareValue(ByVal Post As UserPost) As Long
                 Dim v& = 0
                 If Post.Date.HasValue Then v = Post.Date.Value.Ticks * -1
                 Return v
             End Function
+#End Region
         End Structure
-        Friend Structure Sizes : Implements IComparable(Of Sizes)
+        Friend Structure Sizes : Implements IRegExCreator, IComparable(Of Sizes)
             Friend Value As Integer
             Friend Data As String
             Friend ReadOnly HasError As Boolean
@@ -125,6 +194,16 @@ Namespace API.Base
                     HasError = True
                 End Try
             End Sub
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    Return New Sizes With {
+                        .Value = AConvert(Of Integer)(ParamsArray(0), 0),
+                        .Data = ParamsArray(1)
+                    }
+                Else
+                    Return New Sizes
+                End If
+            End Function
             Friend Function CompareTo(ByVal Other As Sizes) As Integer Implements IComparable(Of Sizes).CompareTo
                 Return Value.CompareTo(Other.Value) * -1
             End Function

SCrawler/API/Gfycat/Envir.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.RegularExpressions
 Imports System.Net
 Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Gfycat
     Friend NotInheritable Class Envir
         Private Sub New()

SCrawler/API/Imgur/Envir.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,12 +6,12 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Net
+Imports SCrawler.API.Base
+Imports SCrawler.API.Imgur.Declarations
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.WebDocuments.JSON
-Imports System.Net
-Imports SCrawler.API.Imgur.Declarations
-Imports SCrawler.API.Base
 Namespace API.Imgur
     Namespace Declarations
         Friend Module Imgur_Declarations
@@ -67,7 +67,7 @@ Namespace API.Imgur
                 Return DownloadingException(ex, $"[API.Imgur.Envir.GetImage({URL})]", String.Empty, e)
             End Try
         End Function
-        Friend Shared Function GetVideoInfo(ByVal URL As String) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString AndAlso URL.ToLower.Contains("imgur") AndAlso Not Settings.ImgurClientID.IsEmptyString Then
                     Dim img$ = GetImage(URL, EDP.ReturnValue)
@@ -79,7 +79,8 @@ Namespace API.Imgur
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, "Imgur standalone downloader: fetch media error")
+                If Not e.Exists Then e = EDP.LogMessageValue
+                Return ErrorsDescriber.Execute(e, ex, "Imgur standalone downloader: fetch media error")
             End Try
         End Function
         Private Shared Function DownloadingException(ByVal ex As Exception, ByVal Message As String,

SCrawler/API/Instagram/AdditionalSettingsForm.Designer.vb

@@ -0,0 +1,135 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Instagram
+    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
+    Partial Friend Class AdditionalSettingsForm : Inherits System.Windows.Forms.Form
+        <System.Diagnostics.DebuggerNonUserCode()>
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            Try
+                If disposing AndAlso components IsNot Nothing Then
+                    components.Dispose()
+                End If
+            Finally
+                MyBase.Dispose(disposing)
+            End Try
+        End Sub
+        Private components As System.ComponentModel.IContainer
+        <System.Diagnostics.DebuggerStepThrough()>
+        Private Sub InitializeComponent()
+            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
+            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
+            Me.CH_DOWN_TIME = New System.Windows.Forms.CheckBox()
+            Me.CH_DOWN_TAG = New System.Windows.Forms.CheckBox()
+            Me.CH_DOWN_SAVED = New System.Windows.Forms.CheckBox()
+            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
+            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
+            CONTAINER_MAIN.ContentPanel.SuspendLayout()
+            CONTAINER_MAIN.SuspendLayout()
+            TP_MAIN.SuspendLayout()
+            Me.SuspendLayout()
+            '
+            'CONTAINER_MAIN
+            '
+            '
+            'CONTAINER_MAIN.ContentPanel
+            '
+            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(234, 78)
+            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            CONTAINER_MAIN.LeftToolStripPanelVisible = False
+            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
+            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
+            CONTAINER_MAIN.RightToolStripPanelVisible = False
+            CONTAINER_MAIN.Size = New System.Drawing.Size(234, 103)
+            CONTAINER_MAIN.TabIndex = 0
+            CONTAINER_MAIN.TopToolStripPanelVisible = False
+            '
+            'TP_MAIN
+            '
+            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            TP_MAIN.ColumnCount = 1
+            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 20.0!))
+            TP_MAIN.Controls.Add(Me.CH_DOWN_TIME, 0, 0)
+            TP_MAIN.Controls.Add(Me.CH_DOWN_TAG, 0, 1)
+            TP_MAIN.Controls.Add(Me.CH_DOWN_SAVED, 0, 2)
+            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            TP_MAIN.Location = New System.Drawing.Point(0, 0)
+            TP_MAIN.Name = "TP_MAIN"
+            TP_MAIN.RowCount = 4
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.Size = New System.Drawing.Size(234, 78)
+            TP_MAIN.TabIndex = 0
+            '
+            'CH_DOWN_TIME
+            '
+            Me.CH_DOWN_TIME.AutoSize = True
+            Me.CH_DOWN_TIME.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_TIME.Location = New System.Drawing.Point(4, 4)
+            Me.CH_DOWN_TIME.Name = "CH_DOWN_TIME"
+            Me.CH_DOWN_TIME.Size = New System.Drawing.Size(226, 19)
+            Me.CH_DOWN_TIME.TabIndex = 0
+            Me.CH_DOWN_TIME.Text = "Download Timeline"
+            Me.CH_DOWN_TIME.UseVisualStyleBackColor = True
+            '
+            'CH_DOWN_TAG
+            '
+            Me.CH_DOWN_TAG.AutoSize = True
+            Me.CH_DOWN_TAG.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_TAG.Location = New System.Drawing.Point(4, 30)
+            Me.CH_DOWN_TAG.Name = "CH_DOWN_TAG"
+            Me.CH_DOWN_TAG.Size = New System.Drawing.Size(226, 19)
+            Me.CH_DOWN_TAG.TabIndex = 1
+            Me.CH_DOWN_TAG.Text = "Download Stories and Tagged data"
+            Me.CH_DOWN_TAG.UseVisualStyleBackColor = True
+            '
+            'CH_DOWN_SAVED
+            '
+            Me.CH_DOWN_SAVED.AutoSize = True
+            Me.CH_DOWN_SAVED.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_SAVED.Location = New System.Drawing.Point(4, 56)
+            Me.CH_DOWN_SAVED.Name = "CH_DOWN_SAVED"
+            Me.CH_DOWN_SAVED.Size = New System.Drawing.Size(226, 19)
+            Me.CH_DOWN_SAVED.TabIndex = 2
+            Me.CH_DOWN_SAVED.Text = "Download saved posts"
+            Me.CH_DOWN_SAVED.UseVisualStyleBackColor = True
+            '
+            'AdditionalSettingsForm
+            '
+            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
+            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
+            Me.ClientSize = New System.Drawing.Size(234, 103)
+            Me.Controls.Add(CONTAINER_MAIN)
+            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
+            Me.KeyPreview = True
+            Me.MaximizeBox = False
+            Me.MaximumSize = New System.Drawing.Size(250, 142)
+            Me.MinimizeBox = False
+            Me.MinimumSize = New System.Drawing.Size(250, 142)
+            Me.Name = "AdditionalSettingsForm"
+            Me.ShowInTaskbar = False
+            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
+            Me.Text = "Additional settings"
+            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
+            CONTAINER_MAIN.ResumeLayout(False)
+            CONTAINER_MAIN.PerformLayout()
+            TP_MAIN.ResumeLayout(False)
+            TP_MAIN.PerformLayout()
+            Me.ResumeLayout(False)
+
+        End Sub
+        Private WithEvents CH_DOWN_TIME As CheckBox
+        Private WithEvents CH_DOWN_TAG As CheckBox
+        Private WithEvents CH_DOWN_SAVED As CheckBox
+    End Class
+End Namespace

SCrawler/API/Instagram/AdditionalSettingsForm.resx

@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="utf-8"?>
+<root>
+  <!-- 
+    Microsoft ResX Schema 
+    
+    Version 2.0
+    
+    The primary goals of this format is to allow a simple XML format 
+    that is mostly human readable. The generation and parsing of the 
+    various data types are done through the TypeConverter classes 
+    associated with the data types.
+    
+    Example:
+    
+    ... ado.net/XML headers & schema ...
+    <resheader name="resmimetype">text/microsoft-resx</resheader>
+    <resheader name="version">2.0</resheader>
+    <resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
+    <resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
+    <data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
+    <data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
+    <data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
+        <value>[base64 mime encoded serialized .NET Framework object]</value>
+    </data>
+    <data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+        <value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
+        <comment>This is a comment</comment>
+    </data>
+                
+    There are any number of "resheader" rows that contain simple 
+    name/value pairs.
+    
+    Each data row contains a name, and value. The row also contains a 
+    type or mimetype. Type corresponds to a .NET class that support 
+    text/value conversion through the TypeConverter architecture. 
+    Classes that don't support this are serialized and stored with the 
+    mimetype set.
+    
+    The mimetype is used for serialized objects, and tells the 
+    ResXResourceReader how to depersist the object. This is currently not 
+    extensible. For a given mimetype the value must be set accordingly:
+    
+    Note - application/x-microsoft.net.object.binary.base64 is the format 
+    that the ResXResourceWriter will generate, however the reader can 
+    read any of the formats listed below.
+    
+    mimetype: application/x-microsoft.net.object.binary.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
+            : and then encoded with base64 encoding.
+    
+    mimetype: application/x-microsoft.net.object.soap.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Soap.SoapFormatter
+            : and then encoded with base64 encoding.
+
+    mimetype: application/x-microsoft.net.object.bytearray.base64
+    value   : The object must be serialized into a byte array 
+            : using a System.ComponentModel.TypeConverter
+            : and then encoded with base64 encoding.
+    -->
+  <xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
+    <xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
+    <xsd:element name="root" msdata:IsDataSet="true">
+      <xsd:complexType>
+        <xsd:choice maxOccurs="unbounded">
+          <xsd:element name="metadata">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" />
+              </xsd:sequence>
+              <xsd:attribute name="name" use="required" type="xsd:string" />
+              <xsd:attribute name="type" type="xsd:string" />
+              <xsd:attribute name="mimetype" type="xsd:string" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="assembly">
+            <xsd:complexType>
+              <xsd:attribute name="alias" type="xsd:string" />
+              <xsd:attribute name="name" type="xsd:string" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="data">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+                <xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
+              <xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
+              <xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="resheader">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" />
+            </xsd:complexType>
+          </xsd:element>
+        </xsd:choice>
+      </xsd:complexType>
+    </xsd:element>
+  </xsd:schema>
+  <resheader name="resmimetype">
+    <value>text/microsoft-resx</value>
+  </resheader>
+  <resheader name="version">
+    <value>2.0</value>
+  </resheader>
+  <resheader name="reader">
+    <value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <resheader name="writer">
+    <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
+  <metadata name="TP_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
+</root>

SCrawler/API/Instagram/AdditionalSettingsForm.vb

@@ -0,0 +1,41 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Forms
+Namespace API.Instagram
+    Friend Class AdditionalSettingsForm
+        Private WithEvents MyDefs As DefaultFormOptions
+        Friend Property MyParameters As SettingsExchangeOptions
+        Friend Sub New(ByVal Parameters As SettingsExchangeOptions)
+            InitializeComponent()
+            MyParameters = Parameters
+            MyDefs = New DefaultFormOptions(Me, Settings.Design)
+        End Sub
+        Private Sub MyForm_Load(sender As Object, e As EventArgs) Handles Me.Load
+            With MyDefs
+                .MyViewInitialize(True)
+                .AddOkCancelToolbar()
+                With MyParameters
+                    CH_DOWN_TIME.Checked = .DownloadTimeline
+                    CH_DOWN_TAG.Checked = .DownloadStoriesTagged
+                    CH_DOWN_SAVED.Checked = .DownloadSaved
+                End With
+                .EndLoaderOperations()
+            End With
+        End Sub
+        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
+            MyParameters = New SettingsExchangeOptions With {
+                .DownloadTimeline = CH_DOWN_TIME.Checked,
+                .DownloadStoriesTagged = CH_DOWN_TAG.Checked,
+                .DownloadSaved = CH_DOWN_SAVED.Checked,
+                .Changed = True
+            }
+            MyDefs.CloseForm()
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Instagram/AuthNullException.vb

@@ -1,36 +0,0 @@
-' Copyright (C) 2022  Andy
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports Sections = SCrawler.API.Instagram.UserData.Sections
-Namespace API.Instagram
-    Friend Class AuthNullException : Inherits ArgumentNullException
-        Public Overrides ReadOnly Property ParamName As String
-        Public Overrides ReadOnly Property Message As String
-        Friend Sub New(ByVal s As Sections, ByVal IsSavedPosts As Boolean)
-            If IsSavedPosts Then
-                ParamName = "HashSavedPosts"
-            ElseIf s = Sections.Timeline Then
-                ParamName = "Hash"
-            Else
-                ParamName = "IG_APP_ID, IG_WWW_CLAIM"
-            End If
-            Message = $"Instagram auth for [{s}] is not set"
-        End Sub
-        Friend Shared Sub ThrowIfNull(ByVal s As Sections, ByVal IsSavedPosts As Boolean, ByVal Host As SiteSettings)
-            Dim b As Boolean = False
-            If IsSavedPosts Then
-                If Not ACheck(Host.HashSavedPosts.Value) Then b = True
-            ElseIf s = Sections.Timeline Then
-                If Not ACheck(Host.Hash.Value) Then Host.HashUpdateRequired.Value = True : b = True
-            Else
-                If Not Host.StoriesAndTaggedReady Then b = True
-            End If
-            If b Then Throw New AuthNullException(s, IsSavedPosts)
-        End Sub
-    End Class
-End Namespace

SCrawler/API/Instagram/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -10,16 +10,8 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Instagram
     Friend Module Declarations
         Friend Const InstagramSite As String = "Instagram"
+        Friend Const InstagramSiteKey As String = "AndyProgram_Instagram"
         Friend ReadOnly FilesPattern As RParams = RParams.DMS(".+?([^/\?]+?\.[\w\d]{3,4})(?=(\?|\Z))", 1, EDP.ReturnValue)
-        Friend ReadOnly Property DateProvider As New JsonDate
+        Friend ReadOnly Property DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
-        Friend Class JsonDate : Implements ICustomProvider
-            Friend Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
-                                    Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
-                Return ADateTime.ParseUnicode(Value)
-            End Function
-            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("GetFormat is not available in this context")
-            End Function
-        End Class
     End Module
 End Namespace

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -11,10 +11,8 @@ Namespace API.Instagram
     Friend Class EditorExchangeOptions
         Friend Property GetStories As Boolean
         Friend Property GetTagged As Boolean
-        Private ReadOnly Property MySiteSettings As SiteSettings
         Friend Sub New(ByVal h As ISiteSettings)
-            MySiteSettings = DirectCast(h, SiteSettings)
+            With DirectCast(h, SiteSettings)
-            With MySiteSettings
                 GetStories = CBool(.GetStories.Value)
                 GetTagged = CBool(.GetTagged.Value)
             End With

SCrawler/API/Instagram/ExitException.vb

@@ -1,15 +0,0 @@
-' Copyright (C) 2022  Andy
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.Instagram
-    Friend Class ExitException : Inherits Exception
-        Friend Sub New(ByRef CompleteArg As Boolean)
-            CompleteArg = True
-        End Sub
-    End Class
-End Namespace

SCrawler/API/Instagram/OptionsForm.Designer.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -95,13 +95,13 @@ Namespace API.Instagram
             Me.ClientSize = New System.Drawing.Size(260, 78)
             Me.Controls.Add(CONTAINER_MAIN)
             Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
             Me.KeyPreview = True
             Me.MaximizeBox = False
             Me.MaximumSize = New System.Drawing.Size(276, 117)
             Me.MinimizeBox = False
             Me.MinimumSize = New System.Drawing.Size(276, 117)
             Me.Name = "OptionsForm"
-            Me.ShowIcon = False
             Me.ShowInTaskbar = False
             Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
             Me.Text = "Options"

SCrawler/API/Instagram/OptionsForm.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -7,22 +7,19 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Forms
-Imports PersonalUtilities.Forms.Toolbars
 Namespace API.Instagram
-    Friend Class OptionsForm : Implements IOkCancelToolbar
+    Friend Class OptionsForm
-        Private ReadOnly MyDefs As DefaultFormProps
+        Private WithEvents MyDefs As DefaultFormOptions
         Private ReadOnly Property MyExchangeOptions As EditorExchangeOptions
         Friend Sub New(ByRef ExchangeOptions As EditorExchangeOptions)
             InitializeComponent()
             MyExchangeOptions = ExchangeOptions
-            MyDefs = New DefaultFormProps
+            MyDefs = New DefaultFormOptions(Me, Settings.Design)
         End Sub
         Private Sub OptionsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
             With MyDefs
-                .MyViewInitialize(Me, Settings.Design, True)
+                .MyViewInitialize(True)
                 .AddOkCancelToolbar()
-                .DelegateClosingChecker()
-                .AppendDetectors()
                 With MyExchangeOptions
                     CH_GET_STORIES.Checked = .GetStories
                     CH_GET_TAGGED.Checked = .GetTagged
@@ -30,15 +27,12 @@ Namespace API.Instagram
                 .EndLoaderOperations()
             End With
         End Sub
-        Private Sub ToolbarBttOK() Implements IOkCancelToolbar.ToolbarBttOK
+        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
             With MyExchangeOptions
                 .GetStories = CH_GET_STORIES.Checked
                 .GetTagged = CH_GET_TAGGED.Checked
             End With
             MyDefs.CloseForm()
         End Sub
-        Private Sub ToolbarBttCancel() Implements IOkCancelToolbar.ToolbarBttCancel
-            MyDefs.CloseForm(DialogResult.Cancel)
-        End Sub
     End Class
 End Namespace

SCrawler/API/Instagram/SettingsExchangeOptions.vb

@@ -0,0 +1,23 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Instagram
+    Friend Structure SettingsExchangeOptions
+        Friend DownloadTimeline As Boolean
+        Friend DownloadStoriesTagged As Boolean
+        Friend DownloadSaved As Boolean
+        Friend Changed As Boolean
+        Friend Sub New(ByVal Source As SiteSettings)
+            With Source
+                DownloadTimeline = .DownloadTimeline
+                DownloadStoriesTagged = .DownloadStoriesTagged
+                DownloadSaved = .DownloadSaved
+            End With
+        End Sub
+    End Structure
+End Namespace

SCrawler/API/Instagram/SiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -9,86 +9,145 @@
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Tools
+Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Instagram
-    <Manifest("AndyProgram_Instagram"), UseClassAsIs, SeparatedTasks(1), SavedPosts, SpecialForm(False)>
+    <Manifest(InstagramSiteKey), SeparatedTasks(1), SavedPosts, SpecialForm(False), SpecialForm(True)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
-#Region "Interface Declarations"
+#Region "Declarations"
+#Region "Images"
         Friend Overrides ReadOnly Property Icon As Icon
             Get
-                Return My.Resources.InstagramIcon
+                Return My.Resources.SiteResources.InstagramIcon_32
             End Get
         End Property
         Friend Overrides ReadOnly Property Image As Image
             Get
-                Return My.Resources.InstagramPic76
+                Return My.Resources.SiteResources.InstagramPic_76
             End Get
         End Property
 #End Region
 #Region "Providers"
-        Private Class TimersChecker : Implements ICustomProvider
+        Private Class TimersChecker : Implements IFieldsCheckerProvider
+            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
+            Private Property Name As String Implements IFieldsCheckerProvider.Name
+            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
+            Private ReadOnly LVProvider As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
             Private ReadOnly _LowestValue As Integer
             Friend Sub New(ByVal LowestValue As Integer)
                 _LowestValue = LowestValue
             End Sub
             Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
                                      Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
-                If ACheck(Of Integer)(Value) AndAlso CInt(Value) >= _LowestValue Then
+                TypeError = False
+                ErrorMessage = String.Empty
+                If Not ACheck(Of Integer)(Value) Then
+                    TypeError = True
+                ElseIf CInt(Value) < _LowestValue Then
+                    ErrorMessage = $"The value of [{Name}] field must be greater than or equal to {_LowestValue.NumToString(LVProvider)}"
+                Else
+                    Return Value
+                End If
+                Return Nothing
+            End Function
+            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
+                Throw New NotImplementedException("[GetFormat] is not available in the context of [TimersChecker]")
+            End Function
+        End Class
+        Private Class TaggedNotifyLimitChecker : Implements IFieldsCheckerProvider
+            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
+            Private Property Name As String Implements IFieldsCheckerProvider.Name
+            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
+            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+                Dim v% = AConvert(Of Integer)(Value, -10)
+                If v > 0 Or v = -1 Then
                     Return Value
                 Else
+                    ErrorMessage = $"The value of [{Name}] field must be greater than 0 or equal to -1"
                     Return Nothing
                 End If
             End Function
             Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException()
+                Throw New NotImplementedException("[GetFormat] is not available in the context of [TaggedNotifyLimitChecker]")
             End Function
         End Class
 #End Region
 #Region "Authorization properties"
-        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash", IsAuth:=True), PXML("InstaHash"), ControlNumber(0)>
+        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash", IsAuth:=True, AllowNull:=False), PXML("InstaHash"), ControlNumber(0)>
         Friend ReadOnly Property Hash As PropertyValue
-        <PropertyOption(ControlText:="Hash 2", ControlToolTip:="Instagram session hash for saved posts", IsAuth:=True), PXML("InstaHashSavedPosts"), ControlNumber(1)>
+        Private Const HashSavedPosts_Text As String = "Hash 2"
+        <PropertyOption(ControlText:=HashSavedPosts_Text, ControlToolTip:="Instagram session hash for saved posts", IsAuth:=True), PXML("InstaHashSavedPosts"), ControlNumber(1)>
         Friend ReadOnly Property HashSavedPosts As PropertyValue
-        <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True), ControlNumber(2)>
+        <PropertyOption(ControlText:="x-csrftoken", ControlToolTip:="Instagram token for tagged data", IsAuth:=True, AllowNull:=False), ControlNumber(2)>
+        Friend ReadOnly Property CSRF_TOKEN As PropertyValue
+        <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True, AllowNull:=False), ControlNumber(3)>
         Friend Property IG_APP_ID As PropertyValue
-        <PropertyOption(ControlText:="x-ig-www-claim", IsAuth:=True), ControlNumber(3)>
+        <PropertyOption(ControlText:="x-ig-www-claim", IsAuth:=True, AllowNull:=False), ControlNumber(4)>
         Friend Property IG_WWW_CLAIM As PropertyValue
-        <PropertyOption(ControlText:="Saved posts user", IsAuth:=True), PXML("SavedPostsUserName"), ControlNumber(4)>
+        Private Const SavedPostsUserName_Text As String = "Saved posts user"
+        <PropertyOption(ControlText:=SavedPostsUserName_Text, IsAuth:=True), PXML("SavedPostsUserName"), ControlNumber(5)>
         Friend ReadOnly Property SavedPostsUserName As PropertyValue
-        Friend ReadOnly Property StoriesAndTaggedReady As Boolean
+        Friend Overrides Function BaseAuthExists() As Boolean
-            Get
+            Return If(Responser.Cookies?.Count, 0) > 0 And ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value) And ACheck(CSRF_TOKEN.Value)
-                Return ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value)
+        End Function
-            End Get
+        Private Const Header_IG_APP_ID As String = "x-ig-app-id"
-        End Property
+        Private Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
+        Private Const Header_CSRF_TOKEN As String = "x-csrftoken"
+        Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
+            If Not PropName.IsEmptyString Then
+                Dim f$ = String.Empty
+                Select Case PropName
+                    Case NameOf(IG_APP_ID) : f = Header_IG_APP_ID
+                    Case NameOf(IG_WWW_CLAIM) : f = Header_IG_WWW_CLAIM
+                    Case NameOf(CSRF_TOKEN) : f = Header_CSRF_TOKEN
+                End Select
+                If Not f.IsEmptyString Then
+                    If Responser.Headers.Count > 0 AndAlso Responser.Headers.ContainsKey(f) Then Responser.Headers.Remove(f)
+                    If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
+                    Responser.SaveSettings()
+                End If
+            End If
+        End Sub
 #End Region
 #Region "Download properties"
-        Friend ReadOnly Property HashUpdateRequired As XMLValue(Of Boolean)
+        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(6)>
-        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(5)>
         Friend ReadOnly Property RequestsWaitTimer As PropertyValue
         <Provider(NameOf(RequestsWaitTimer), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerProvider As IFormatProvider
-        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(6)>
+        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(7)>
         Friend ReadOnly Property RequestsWaitTimerTaskCount As PropertyValue
         <Provider(NameOf(RequestsWaitTimerTaskCount), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerTaskCountProvider As IFormatProvider
-        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(7)>
+        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(8)>
         Friend ReadOnly Property SleepTimerOnPostsLimit As PropertyValue
         <Provider(NameOf(SleepTimerOnPostsLimit), FieldsChecker:=True)>
         Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
-        <PropertyOption(ControlText:="Get stories"), PXML, ControlNumber(8)>
+        <PropertyOption(ControlText:="Get stories"), PXML, ControlNumber(9)>
         Friend ReadOnly Property GetStories As PropertyValue
-        <PropertyOption(ControlText:="Get tagged photos"), PXML, ControlNumber(9)>
+        <PropertyOption(ControlText:="Get tagged photos"), PXML, ControlNumber(10)>
         Friend ReadOnly Property GetTagged As PropertyValue
+        <PropertyOption(ControlText:="Tagged notify limit",
+                        ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
+                        "-1 to disable"), PXML, ControlNumber(11)>
+        Friend ReadOnly Property TaggedNotifyLimit As PropertyValue
+        <Provider(NameOf(TaggedNotifyLimit), FieldsChecker:=True)>
+        Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
+#End Region
+#Region "Download ready"
+        Friend ReadOnly Property DownloadTimeline As XMLValue(Of Boolean)
+        Friend ReadOnly Property DownloadStoriesTagged As XMLValue(Of Boolean)
+        Friend ReadOnly Property DownloadSaved As XMLValue(Of Boolean)
 #End Region
 #Region "429 bypass"
-        Friend ReadOnly Property DownloadingErrorDate As XMLValue(Of Date)
+        Private ReadOnly Property DownloadingErrorDate As XMLValue(Of Date)
         Friend Property LastApplyingValue As Integer? = Nothing
         Friend ReadOnly Property ReadyForDownload As Boolean
             Get
+                If SkipUntilNextSession Then Return False
                 With DownloadingErrorDate
                     If .ValueF.Exists Then
                         Return .ValueF.Value.AddMinutes(If(LastApplyingValue, 10)) < Now
@@ -98,8 +157,11 @@ Namespace API.Instagram
                 End With
             End Get
         End Property
-        Friend ReadOnly Property LastDownloadDate As XMLValue(Of Date)
+        Private ReadOnly Property LastDownloadDate As XMLValue(Of Date)
-        Friend ReadOnly Property LastRequestsCount As XMLValue(Of Integer)
+        Private ReadOnly Property LastRequestsCount As XMLValue(Of Integer)
+        <PropertyOption(IsInformationLabel:=True), ControlNumber(100)>
+        Private Property LastRequestsCountLabel As PropertyValue
+        Private ReadOnly LastRequestsCountLabelStr As Func(Of Integer, String) = Function(r) $"Number of spent requests: {r.NumToGroupIntegral}"
         Private TooManyRequestsReadyForCatch As Boolean = True
         Friend Function GetWaitDate() As Date
             With DownloadingErrorDate
@@ -129,24 +191,26 @@ Namespace API.Instagram
             End With
         End Sub
 #End Region
-        Friend Overrides ReadOnly Property Responser As WEB.Response
+#End Region
+#Region "Initializer"
         Friend Sub New(ByRef _XML As XmlFile, ByVal GlobalPath As SFile)
-            MyBase.New(InstagramSite)
+            MyBase.New(InstagramSite, "instagram.com")
-            Responser = New WEB.Response($"{SettingsFolderName}\Responser_{Site}.xml")
 
             Dim app_id$ = String.Empty
             Dim www_claim$ = String.Empty
+            Dim token$ = String.Empty
 
             With Responser
-                If .File.Exists Then
+                If .Headers.Count > 0 Then
-                    .LoadSettings()
                     With .Headers
+                        If .ContainsKey(Header_CSRF_TOKEN) Then token = .Item(Header_CSRF_TOKEN)
                         If .ContainsKey(Header_IG_APP_ID) Then app_id = .Item(Header_IG_APP_ID)
                         If .ContainsKey(Header_IG_WWW_CLAIM) Then www_claim = .Item(Header_IG_WWW_CLAIM)
                     End With
-                Else
+                End If
-                    .CookiesDomain = "instagram.com"
+                If Not .Cookies Is Nothing Then
-                    .SaveSettings()
+                    .Cookies.ChangedAllowInternalDrop = False
+                    .Cookies.Changed = False
                 End If
             End With
 
@@ -154,58 +218,98 @@ Namespace API.Instagram
 
             SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
 
-            HashUpdateRequired = New XMLValue(Of Boolean)("InstaHashUpdateRequired", True, _XML, n)
             Hash = New PropertyValue(String.Empty, GetType(String))
             HashSavedPosts = New PropertyValue(String.Empty, GetType(String))
+            CSRF_TOKEN = New PropertyValue(token, GetType(String), Sub(v) ChangeResponserFields(NameOf(CSRF_TOKEN), v))
             IG_APP_ID = New PropertyValue(app_id, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_APP_ID), v))
             IG_WWW_CLAIM = New PropertyValue(www_claim, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_WWW_CLAIM), v))
 
+            DownloadTimeline = New XMLValue(Of Boolean)("DownloadTimeline", True, _XML, n)
+            DownloadStoriesTagged = New XMLValue(Of Boolean)("DownloadStoriesTagged", True, _XML, n)
+            DownloadSaved = New XMLValue(Of Boolean)("DownloadSaved", True, _XML, n)
+
             RequestsWaitTimer = New PropertyValue(1000)
             RequestsWaitTimerProvider = New TimersChecker(100)
             RequestsWaitTimerTaskCount = New PropertyValue(1)
             RequestsWaitTimerTaskCountProvider = New TimersChecker(1)
-            SleepTimerOnPostsLimit = New PropertyValue(6000)
+            SleepTimerOnPostsLimit = New PropertyValue(60000)
             SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
 
             GetStories = New PropertyValue(False)
             GetTagged = New PropertyValue(False)
+            TaggedNotifyLimit = New PropertyValue(200)
+            TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker
 
-            DownloadingErrorDate = New XMLValue(Of Date) With {
+            DownloadingErrorDate = New XMLValue(Of Date) With {.Provider = New XMLValueConversionProvider(Function(ss, vv) AConvert(Of String)(vv, AModes.Var, Nothing))}
-                .Provider = New XMLValueConversionProvider(Function(ss, vv) AConvert(Of String)(vv, AModes.Var, Nothing))}
             DownloadingErrorDate.SetExtended("InstagramDownloadingErrorDate", Now.AddYears(-10), _XML, n)
             LastDownloadDate = New XMLValue(Of Date)("LastDownloadDate", Now.AddDays(-1), _XML, n)
             LastRequestsCount = New XMLValue(Of Integer)("LastRequestsCount", 0, _XML, n)
+            LastRequestsCountLabel = New PropertyValue(LastRequestsCountLabelStr.Invoke(LastRequestsCount.Value))
+            AddHandler LastRequestsCount.OnValueChanged, Sub(sender, __name, __value) LastRequestsCountLabel.Value = LastRequestsCountLabelStr.Invoke(DirectCast(__value, Existable(Of Integer)).Value)
 
             UrlPatternUser = "https://www.instagram.com/{0}/"
             UserRegex = RParams.DMS("[htps:/]{7,8}.*?instagram.com/([^/]+)", 1)
             ImageVideoContains = "instagram.com"
         End Sub
-        Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
+        Private Structure LatestValues
-            Select Case What
+            Friend Hash As String
-                Case Download.Main : Return New UserData
+            Friend Hash2 As String
-                Case Download.SavedPosts
+            Friend Token As String
-                    Dim u As New UserData
+            Friend AppID As String
-                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}
+            Friend WwwClaim As String
-                    Return u
+            Friend Exists As Boolean
-            End Select
+            Friend Sub New(ByVal Source As SiteSettings)
-            Return Nothing
+                Exists = True
-        End Function
+                With Source
-        Private Const Header_IG_APP_ID As String = "x-ig-app-id"
+                    Hash = AConvert(Of String)(.Hash.Value, String.Empty)
-        Private Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
+                    Hash2 = AConvert(Of String)(.HashSavedPosts.Value, String.Empty)
-        Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
+                    With .Responser.Headers
-            If Not PropName.IsEmptyString Then
+                        If .ContainsKey(Header_CSRF_TOKEN) Then Token = .Item(Header_CSRF_TOKEN)
-                Dim f$ = String.Empty
+                        If .ContainsKey(Header_IG_APP_ID) Then AppID = .Item(Header_IG_APP_ID)
-                Select Case PropName
+                        If .ContainsKey(Header_IG_WWW_CLAIM) Then WwwClaim = .Item(Header_IG_WWW_CLAIM)
-                    Case NameOf(IG_APP_ID) : f = Header_IG_APP_ID
+                    End With
-                    Case NameOf(IG_WWW_CLAIM) : f = Header_IG_WWW_CLAIM
+                End With
-                End Select
-                If Not f.IsEmptyString Then
-                    If Responser.Headers.Count > 0 AndAlso Responser.Headers.ContainsKey(f) Then Responser.Headers.Remove(f)
-                    If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
-                    Responser.SaveSettings()
-                End If
-            End If
             End Sub
+        End Structure
+        Private LV As LatestValues = Nothing
+        Private ASO As SettingsExchangeOptions = Nothing
+        Friend Overrides Sub BeginEdit()
+            LV = New LatestValues(Me)
+            ASO = Nothing
+            MyBase.BeginEdit()
+        End Sub
+        Friend Overrides Sub EndEdit()
+            LV = Nothing
+            ASO = Nothing
+            MyBase.EndEdit()
+        End Sub
+        Friend Overrides Sub Update()
+            If LV.Exists Then
+                Dim __lv As New LatestValues(Me)
+                If If(Responser.Cookies?.Count, 0) > 0 Then
+                    Dim _cookiesChanged As Boolean = If(Responser.Cookies?.Changed, False)
+                    If Not DownloadTimeline AndAlso (_cookiesChanged Or
+                       (Not LV.Hash = __lv.Hash And Not __lv.Hash.IsEmptyString)) Then DownloadTimeline.Value = True
+                    If Not DownloadSaved AndAlso (_cookiesChanged Or (Not LV.Hash2 = __lv.Hash2 And Not __lv.Hash2.IsEmptyString)) Then DownloadSaved.Value = True
+                    If Not DownloadStoriesTagged AndAlso (
+                           _cookiesChanged Or (
+                                (Not LV.Hash = __lv.Hash Or Not LV.Token = __lv.Token Or Not LV.AppID = __lv.AppID Or Not LV.WwwClaim = __lv.WwwClaim) And
+                                (Not __lv.Hash.IsEmptyString And Not __lv.Token.IsEmptyString And Not __lv.AppID.IsEmptyString And Not __lv.WwwClaim.IsEmptyString)
+                            )) Then DownloadStoriesTagged.Value = True
+                End If
+            End If
+            If ASO.Changed Then
+                DownloadTimeline.Value = ASO.DownloadTimeline
+                DownloadStoriesTagged.Value = ASO.DownloadStoriesTagged
+                DownloadSaved.Value = ASO.DownloadSaved
+            End If
+            LV = Nothing
+            ASO = Nothing
+            If Not Responser.Cookies Is Nothing Then Responser.Cookies.Changed = False
+            MyBase.Update()
+        End Sub
+#End Region
+#Region "PropertiesDataChecker"
         <PropertiesDataChecker({NameOf(Hash), NameOf(HashSavedPosts)})>
         Private Function CheckHashControls(ByVal p As IEnumerable(Of PropertyData)) As Boolean
             If p.ListExists(2) Then
@@ -231,24 +335,90 @@ Namespace API.Instagram
                 Return False
             End If
         End Function
-        Friend Overrides Sub BeginInit()
+        <PropertiesDataChecker({NameOf(TaggedNotifyLimit)})>
-        End Sub
+        Private Function CheckNotifyLimit(ByVal p As IEnumerable(Of PropertyData)) As Boolean
-        Friend Overrides Sub EndInit()
+            If p.ListExists Then
-            If (CStr(Hash.Value).IsEmptyString Or HashUpdateRequired) AndAlso Responser.Cookies.ListExists Then GatherInstaHash()
+                Dim pi% = p.ListIndexOf(Function(pp) pp.Name = NameOf(TaggedNotifyLimit))
-        End Sub
+                If pi >= 0 Then
-        Friend Overrides Function ReadyToDownload(ByVal What As Download) As Boolean
+                    Dim v% = AConvert(Of Integer)(p(pi).Value, -10)
-            Return ActiveJobs < 2 AndAlso ReadyForDownload
+                    If v > 0 Then
+                        Return True
+                    ElseIf v = -1 Then
+                        Return MsgBoxE({"You turn off notifications for tagged posts. This is highly undesirable. Do you still want to do it?",
+                                        "Disabling tagged notification limits"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes
+                    Else
+                        Return False
+                    End If
+                End If
+            End If
+            Return False
+        End Function
+        <PropertiesDataChecker({NameOf(HashSavedPosts), NameOf(SavedPostsUserName)})>
+        Private Function CheckSavedOptions(ByVal p As IEnumerable(Of PropertyData)) As Boolean
+            If p.ListExists Then
+                Const MsgTitle$ = "Saved posts credentials"
+                Dim __hash$ = String.Empty
+                Dim __name$ = String.Empty
+                Dim _OptionlErrorText$ = $"For download saved posts, you must to set both [{HashSavedPosts_Text}] and [{SavedPostsUserName_Text}]."
+                For i% = 0 To p.Count - 1
+                    Select Case p(i).Name
+                        Case NameOf(HashSavedPosts) : __hash = p(i).Value
+                        Case NameOf(SavedPostsUserName) : __name = p(i).Value
+                    End Select
+                Next
+                If __hash = __name Then
+                    If __hash.IsEmptyString Then
+                        Return True
+                    Else
+                        MsgBoxE({$"[{HashSavedPosts_Text}] and [{SavedPostsUserName_Text}] for saved posts cannot be equal!", MsgTitle}, vbCritical)
+                    End If
+                Else
+                    If __hash.IsEmptyString Then
+                        MsgBoxE({$"[{HashSavedPosts_Text}] not set.{vbCr}{_OptionlErrorText}", MsgTitle}, vbCritical)
+                    ElseIf __name.IsEmptyString Then
+                        MsgBoxE({$"[{SavedPostsUserName_Text}] not set.{vbCr}{_OptionlErrorText}", MsgTitle}, vbCritical)
+                    Else
+                        Return True
+                    End If
+                End If
+            End If
+            Return False
+        End Function
+#End Region
+#Region "Plugin functions"
+        Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
+            Select Case What
+                Case Download.Main : Return New UserData
+                Case Download.SavedPosts
+                    Dim u As New UserData
+                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}
+                    Return u
+            End Select
+            Return Nothing
         End Function
 #Region "Downloading"
+        Friend Property SkipUntilNextSession As Boolean = False
+        Friend Overrides Function ReadyToDownload(ByVal What As Download) As Boolean
+            If ActiveJobs < 2 AndAlso Not SkipUntilNextSession AndAlso ReadyForDownload AndAlso BaseAuthExists() Then
+                Select Case What
+                    Case Download.Main : Return ACheck(Hash.Value) And DownloadTimeline
+                    Case Download.SavedPosts : Return ACheck(HashSavedPosts.Value) And DownloadSaved
+                End Select
+            End If
+            Return False
+        End Function
         Private ActiveJobs As Integer = 0
         Private _NextWNM As UserData.WNM = UserData.WNM.Notify
+        Private _NextTagged As Boolean = True
         Friend Overrides Sub DownloadStarted(ByVal What As Download)
-            If CStr(Hash.Value).IsEmptyString Or HashUpdateRequired Then GatherInstaHash()
             ActiveJobs += 1
         End Sub
         Friend Overrides Sub BeforeStartDownload(ByVal User As Object, ByVal What As Download)
             With DirectCast(User, UserData)
-                If What = Download.Main Then .WaitNotificationMode = _NextWNM
+                If What = Download.Main Then
+                    .WaitNotificationMode = _NextWNM
+                    .TaggedCheckSession = _NextTagged
+                End If
                 If LastDownloadDate.Value.AddMinutes(60) > Now Then
                     .RequestsCount = LastRequestsCount
                 Else
@@ -261,48 +431,20 @@ Namespace API.Instagram
             With DirectCast(User, UserData)
                 _NextWNM = .WaitNotificationMode
                 If _NextWNM = UserData.WNM.SkipTemp Or _NextWNM = UserData.WNM.SkipCurrent Then _NextWNM = UserData.WNM.Notify
+                _NextTagged = .TaggedCheckSession
                 LastDownloadDate.Value = Now
                 LastRequestsCount.Value = .RequestsCount
             End With
         End Sub
         Friend Overrides Sub DownloadDone(ByVal What As Download)
             _NextWNM = UserData.WNM.Notify
+            _NextTagged = True
             LastDownloadDate.Value = Now
             ActiveJobs -= 1
-            If HashUpdateRequired Then MyMainLOG = "Check your Instagram credentials"
+            SkipUntilNextSession = False
         End Sub
 #End Region
-        <PropertyUpdater(NameOf(Hash))>
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-        Friend Function GatherInstaHash() As Boolean
-            Try
-                If Not Responser.Cookies.ListExists Then Throw New Exception("Instagram cookies does not set")
-                Dim rs As New RParams("=""([^""]+?ConsumerLibCommons[^""]+?.js)""", Nothing, 1) With {.MatchTimeOut = 10}
-                Dim r$ = Responser.GetResponse("https://instagram.com",, EDP.ThrowException)
-                If Not r.IsEmptyString Then
-                    Dim hStr$ = RegexReplace(r, rs)
-                    If Not hStr.IsEmptyString Then
-                        Do While Left(hStr, 1) = "/" : hStr = Right(hStr, hStr.Length - 1) : Loop
-                        hStr = $"https://instagram.com/{hStr}"
-                        r = Responser.GetResponse(hStr,, EDP.ThrowException)
-                        If Not r.IsEmptyString Then
-                            rs = New RParams("generatePaginationActionCreators.+?.profilePosts.byUserId.get.+?queryId:.([\d\w\S]+?)""", Nothing, 1) With {.MatchTimeOut = 10}
-                            Dim h$ = RegexReplace(r, rs)
-                            If Not h.IsEmptyString Then
-                                Hash.Value = h
-                                HashUpdateRequired.Value = False
-                                Return True
-                            End If
-                        End If
-                    End If
-                End If
-                Return False
-            Catch ex As Exception
-                HashUpdateRequired.Value = True
-                Hash.Value = String.Empty
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[SiteSettings.GaterInstaHash]", False)
-            End Try
-        End Function
-        Friend Overrides Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
             Return UserData.GetVideoInfo(URL, Responser)
         End Function
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
@@ -311,5 +453,12 @@ Namespace API.Instagram
                 Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
+        Friend Overrides Sub OpenSettingsForm()
+            Using f As New AdditionalSettingsForm(If(ASO.Changed, ASO, New SettingsExchangeOptions(Me)))
+                f.ShowDialog()
+                If f.DialogResult = DialogResult.OK Then ASO = f.MyParameters
+            End Using
+        End Sub
+#End Region
     End Class
 End Namespace

SCrawler/API/Instagram/UserData.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,23 +6,25 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Net
+Imports System.Threading
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.Messaging
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.WEB
 Imports PersonalUtilities.Tools.WebDocuments.JSON
 Imports SCrawler.API.Base
-Imports System.Threading
-Imports System.Net
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Instagram
     Friend Class UserData : Inherits UserDataBase
-        Private Const MaxPostsCount As Integer = 200
+#Region "XML Names"
         Private Const Name_LastCursor As String = "LastCursor"
         Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
         Private Const Name_GetStories As String = "GetStories"
         Private Const Name_GetTagged As String = "GetTaggedData"
         Private Const Name_TaggedChecked As String = "TaggedChecked"
+#End Region
+#Region "Declarations"
         Private ReadOnly Property MySiteSettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
@@ -33,6 +35,8 @@ Namespace API.Instagram
         Private FirstLoadingDone As Boolean = False
         Friend Property GetStories As Boolean
         Friend Property GetTaggedData As Boolean
+#End Region
+#Region "Exchange options"
         Friend Overrides Function ExchangeOptionsGet() As Object
             Return New EditorExchangeOptions(HOST.Source) With {.GetStories = GetStories, .GetTagged = GetTaggedData}
         End Function
@@ -44,6 +48,8 @@ Namespace API.Instagram
                 End With
             End If
         End Sub
+#End Region
+#Region "Initializer, loader"
         Friend Sub New()
         End Sub
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
@@ -61,41 +67,64 @@ Namespace API.Instagram
                 Container.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
             End If
         End Sub
+#End Region
 #Region "Download data"
+        Private E560Thrown As Boolean = False
+        Private Class ExitException : Inherits Exception
+            Friend Shared Sub Throw560(ByRef Source As UserData)
+                If Not Source.E560Thrown Then
+                    MyMainLOG = $"{Source.ToStringForLog}: (560) Download skipped until next session"
+                    Source.E560Thrown = True
+                End If
+                Throw New ExitException
+            End Sub
+            Friend Sub New()
+            End Sub
+            Friend Sub New(ByRef CompleteArg As Boolean)
+                CompleteArg = True
+            End Sub
+        End Class
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Dim s As Sections = Sections.Timeline
             Try
+                ThrowAny(Token)
                 _InstaHash = String.Empty
                 HasError = False
-                If Not LastCursor.IsEmptyString Then
+                Dim fc As Boolean = IIf(IsSavedPosts, MySiteSettings.DownloadSaved.Value, MySiteSettings.DownloadTimeline.Value)
-                    DownloadData(LastCursor, Sections.Timeline, Token)
+                If fc And Not LastCursor.IsEmptyString Then
+                    s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
+                    DownloadData(LastCursor, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
-                If Not HasError Then
+                If fc And Not HasError Then
-                    DownloadData(String.Empty, Sections.Timeline, Token)
+                    s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
+                    DownloadData(String.Empty, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
                 If FirstLoadingDone Then LastCursor = String.Empty
                 If IsSavedPosts Then
-                    DownloadPosts(Token)
+                    If MySiteSettings.DownloadSaved Then s = Sections.SavedPosts : DownloadPosts(Token)
-                ElseIf MySiteSettings.StoriesAndTaggedReady Then
+                ElseIf MySiteSettings.BaseAuthExists() Then
-                    If GetStories Then DownloadData(String.Empty, Sections.Stories, Token)
+                    DownloadedTags = 0
-                    If GetTaggedData Then DownloadData(String.Empty, Sections.Tagged, Token)
+                    If MySiteSettings.DownloadStoriesTagged And GetStories Then s = Sections.Stories : DownloadData(String.Empty, s, Token)
+                    If MySiteSettings.DownloadStoriesTagged And GetTaggedData Then s = Sections.Tagged : DownloadData(String.Empty, s, Token)
                 End If
                 If WaitNotificationMode = WNM.SkipTemp Or WaitNotificationMode = WNM.SkipCurrent Then WaitNotificationMode = WNM.Notify
             Catch eex As ExitException
             Catch ex As Exception
-                ProcessException(ex, Token, "[API.Instagram.UserData.DownloadDataF", False)
+                ProcessException(ex, Token, "[API.Instagram.UserData.DownloadDataF]", False, s)
+            Finally
+                E560Thrown = False
             End Try
         End Sub
         Private _InstaHash As String = String.Empty
-        Friend Enum Sections
+        Private Enum Sections : Timeline : Tagged : Stories : SavedPosts : End Enum
-            Timeline
+        Private Const StoriesFolder As String = "Stories"
-            Tagged
+        Private Const TaggedFolder As String = "Tagged"
-            Stories
-        End Enum
 #Region "429 bypass"
+        Private Const MaxPostsCount As Integer = 200
         Friend Property RequestsCount As Integer = 0
         Friend Enum WNM As Integer
             Notify = 0
@@ -138,7 +167,7 @@ Namespace API.Instagram
         End Function
         Private Sub ReconfigureAwaiter()
             If WaitNotificationMode = WNM.SkipTemp Then WaitNotificationMode = WNM.Notify
-            If Caught429 Then Caught429 = False ': RequestsCount = 0
+            If Caught429 Then Caught429 = False
             ProgressTempSet = False
         End Sub
         Private Sub NextRequest(ByVal StartWait As Boolean)
@@ -148,9 +177,77 @@ Namespace API.Instagram
             End With
         End Sub
 #End Region
-        Private Const StoriesFolder As String = "Stories"
+#Region "Tags"
-        Private Const TaggedFolder As String = "Tagged"
         Private TaggedChecked As Boolean = False
+        Friend TaggedCheckSession As Boolean = True
+        Private DownloadedTags As Integer = 0
+        Private DownloadTagsLimit As Integer? = Nothing
+        Private ReadOnly Property TaggedLimitsNotifications(Optional ByVal v As Integer? = Nothing) As Boolean
+            Get
+                Return Not TaggedChecked AndAlso TaggedCheckSession AndAlso
+                       CInt(MySiteSettings.TaggedNotifyLimit.Value) > 0 AndAlso
+                       (Not v.HasValue OrElse v.Value > CInt(MySiteSettings.TaggedNotifyLimit.Value))
+            End Get
+        End Property
+        Private Function SetTagsLimit(ByVal Max As Integer, ByVal p As ANumbers) As DialogResult
+            Dim v%?
+            Dim aStr$ = $"Enter the number of posts from user {ToString()} that you want to download{vbCr}" &
+                        $"(Max: {Max.NumToString(p)}; Requests: {(Max / 12).RoundUp.NumToString(p)})"
+            Dim tryBtt As New MsgBoxButton("Try again") With {.ToolTip = "You will be asked again about the limit"}
+            Dim cancelBtt As New MsgBoxButton("Cancel") With {.ToolTip = "Cancel tagged posts download operation"}
+            Dim selectBtt As New MsgBoxButton("Other options") With {.ToolTip = "The main message with options will be displayed again"}
+            Dim m As New MMessage("You have not entered a valid posts limit", "Tagged posts download limit", {tryBtt, selectBtt, cancelBtt})
+            Dim mh As New MMessage("", "Tagged posts download limit", {"Confirm", tryBtt, selectBtt, cancelBtt}) With {.ButtonsPerRow = 2}
+            Do
+                v = AConvert(Of Integer)(InputBoxE(aStr, "Tagged posts download limit", CInt(MySiteSettings.TaggedNotifyLimit.Value)), AModes.Var, Nothing)
+                If v.HasValue Then
+                    mh.Text = $"You have entered a limit of {v.Value.NumToString(p)} posts"
+                    Select Case MsgBoxE(mh).Index
+                        Case 0 : DownloadTagsLimit = v : Return DialogResult.OK
+                        Case 1 : v = Nothing
+                        Case 2 : Return DialogResult.Retry
+                        Case 3 : Return DialogResult.Cancel
+                    End Select
+                Else
+                    Select Case MsgBoxE(m).Index
+                        Case 1 : Return DialogResult.Retry
+                        Case 2 : Return DialogResult.Cancel
+                    End Select
+                End If
+            Loop While Not v.HasValue
+            Return DialogResult.Retry
+        End Function
+        Private Function TaggedContinue(ByVal TaggedCount As Integer) As DialogResult
+            Dim agi As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
+            Dim msg As New MMessage($"The number of tagged posts by user [{ToString()}] is {TaggedCount.NumToString(agi)}" & vbCr &
+                                    $"This is about {(TaggedCount / 12).RoundUp.NumToString(agi)} requests." & vbCr &
+                                    "The tagged data download operation can take a long time.",
+                                    "Too much tagged data",
+                                    {
+                                        "Continue",
+                                        New MsgBoxButton("Continue unnotified") With {
+                                            .ToolTip = "Continue downloading and cancel further notifications in the current downloading session."},
+                                        New MsgBoxButton("Limit") With {
+                                            .ToolTip = "Enter the limit of posts you want to download."},
+                                        New MsgBoxButton("Disable and cancel") With {
+                                            .ToolTip = "Disable downloading tagged data and cancel downloading tagged data."},
+                                        "Cancel"
+                                    }, MsgBoxStyle.Exclamation) With {.DefaultButton = 0, .CancelButton = 4, .ButtonsPerRow = 2}
+            Do
+                Select Case MsgBoxE(msg).Index
+                    Case 0 : Return DialogResult.OK
+                    Case 1 : TaggedCheckSession = False : Return DialogResult.OK
+                    Case 2
+                        Select Case SetTagsLimit(TaggedCount, agi)
+                            Case DialogResult.OK : Return DialogResult.OK
+                            Case DialogResult.Cancel : Return DialogResult.Cancel
+                        End Select
+                    Case 3 : GetTaggedData = False : Return DialogResult.Cancel
+                    Case 4 : Return DialogResult.Cancel
+                End Select
+            Loop
+        End Function
+#End Region
         Private Overloads Sub DownloadData(ByVal Cursor As String, ByVal Section As Sections, ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
             Dim StoriesList As List(Of String) = Nothing
@@ -166,6 +263,7 @@ Namespace API.Instagram
                     Try
                         Dim n As EContainer, nn As EContainer, node As EContainer
                         Dim HasNextPage As Boolean = False
+                        Dim Pinned As Boolean
                         Dim EndCursor$ = String.Empty
                         Dim PostID$ = String.Empty, PostDate$ = String.Empty, SpecFolder$ = String.Empty
                         Dim TaggedCount%
@@ -175,13 +273,12 @@ Namespace API.Instagram
                         'Check environment
                         If Cursor.IsEmptyString And _InstaHash.IsEmptyString Then _
                            _InstaHash = CStr(If(IsSavedPosts, MySiteSettings.HashSavedPosts, MySiteSettings.Hash).Value)
-                        AuthNullException.ThrowIfNull(Section, IsSavedPosts, MySiteSettings)
                         If ID.IsEmptyString Then GetUserId()
                         If ID.IsEmptyString Then Throw New ArgumentException("User ID is not detected", "ID")
 
                         'Create query
                         Select Case Section
-                            Case Sections.Timeline
+                            Case Sections.Timeline, Sections.SavedPosts
                                 Dim vars$ = "{""id"":" & ID & ",""first"":50,""after"":""" & Cursor & """}"
                                 vars = SymbolsConverter.ASCII.EncodeSymbolsOnly(vars)
                                 URL = $"https://www.instagram.com/graphql/query/?query_hash={_InstaHash}&variables={vars}"
@@ -193,6 +290,7 @@ Namespace API.Instagram
                             Case Sections.Stories
                                 If Not StoriesRequested Then
                                     StoriesList = GetStoriesList()
+                                    StoriesRequested = True
                                     MySiteSettings.TooManyRequests(False)
                                     RequestsCount += 1
                                     ThrowAny(Token)
@@ -215,13 +313,13 @@ Namespace API.Instagram
                         RequestsCount += 1
                         ThrowAny(Token)
 
-                        'Data
+                        'Parsing
                         If Not r.IsEmptyString Then
                             Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
                                 n = j.ItemF(ENode).XmlIfNothing
                                 If n.Count > 0 Then
                                     Select Case Section
-                                        Case Sections.Timeline
+                                        Case Sections.Timeline, Sections.SavedPosts
                                             If n.Contains("page_info") Then
                                                 With n("page_info")
                                                     HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
@@ -235,15 +333,20 @@ Namespace API.Instagram
                                                     node = nn(0).XmlIfNothing
                                                     If IsSavedPosts Then
                                                         PostID = node.Value("shortcode")
-                                                        If Not PostID.IsEmptyString Then
+                                                        If Not PostID.IsEmptyString AndAlso _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
-                                                            If _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete) Else _SavedPostsIDs.Add(PostID)
                                                     End If
-                                                    Else
                                                     PostID = node.Value("id")
-                                                        If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
+                                                    Pinned = CBool(If(node("pinned_for_users")?.Count, 0))
+                                                    If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) And Not Pinned Then Throw New ExitException(_DownloadComplete)
                                                     _TempPostsList.Add(PostID)
                                                     PostDate = node.Value("taken_at_timestamp")
-                                                        If Not CheckDatesLimit(PostDate, DateProvider) Then Throw New ExitException(_DownloadComplete)
+                                                    If IsSavedPosts Then
+                                                        _SavedPostsIDs.Add(PostID)
+                                                    Else
+                                                        Select Case CheckDatesLimit(PostDate, DateProvider)
+                                                            Case DateResult.Skip : Continue For
+                                                            Case DateResult.Exit : If Not Pinned Then Throw New ExitException(_DownloadComplete)
+                                                        End Select
                                                         ObtainMedia(node, PostID, PostDate, SpecFolder)
                                                     End If
                                                 Next
@@ -256,31 +359,20 @@ Namespace API.Instagram
                                                 If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
                                                 _TempPostsList.Add(PostID)
                                                 ObtainMedia2(nn, PostID, SpecFolder)
+                                                DownloadedTags += 1
+                                                If DownloadTagsLimit.HasValue AndAlso DownloadedTags >= DownloadTagsLimit.Value Then Throw New ExitException(_DownloadComplete)
                                             Next
-                                            If Not TaggedChecked Then
+                                            If TaggedLimitsNotifications Then
                                                 TaggedCount = j.Value("total_count").FromXML(Of Integer)(0)
                                                 TaggedChecked = True
-                                                If TaggedCount > 200 Then
+                                                If TaggedLimitsNotifications(TaggedCount) AndAlso
-                                                    Dim a% = MsgBoxE({$"The number of tagged posts is {TaggedCount.NumToString(New ANumbers With {
+                                                   TaggedContinue(TaggedCount) = DialogResult.Cancel Then Throw New ExitException(_DownloadComplete)
-                                                                                                       .FormatOptions = ANumbers.Options.GroupIntegral})}" & vbCr &
-                                                                      "The tagged data download operation can take a long time.", "Too much tagged data"}, vbExclamation,,,
-                                                                      {"Continue",
-                                                                       New MsgBoxButton("Disable and cancel") With {
-                                                                        .ToolTip = "Disable downloading tagged data and cancel downloading tagged data."},
-                                                                       "Cancel"})
-                                                    If a > 0 Then
-                                                        If a = 1 Then GetTaggedData = False
-                                                        Throw New ExitException(_DownloadComplete)
-                                                    End If
-                                                End If
                                             End If
                                     End Select
                                 Else
-                                    If j.Value("status") = "ok" AndAlso j({"data", "user"}).XmlIfNothing.Count = 0 AndAlso _TempMediaList.Count = 0 Then
+                                    If j.Value("status") = "ok" AndAlso j({"data", "user"}).XmlIfNothing.Count = 0 AndAlso
-                                        MySiteSettings.HashUpdateRequired.Value = True
+                                       _TempMediaList.Count = 0 AndAlso Section = Sections.Timeline Then _
-                                        UserExists = False
+                                       UserExists = False : Throw New ExitException(_DownloadComplete)
-                                        Throw New ExitException(_DownloadComplete)
-                                    End If
                                 End If
                             End Using
                         Else
@@ -288,9 +380,6 @@ Namespace API.Instagram
                         End If
                         _DownloadComplete = True
                         If HasNextPage And Not EndCursor.IsEmptyString Then DownloadData(EndCursor, Section, Token)
-                    Catch iane As AuthNullException
-                        ErrorsDescriber.Execute(EDP.SendInLog, iane)
-                        Throw New ExitException(_DownloadComplete)
                     Catch eex As ExitException
                         Throw eex
                     Catch oex As OperationCanceledException When Token.IsCancellationRequested
@@ -298,7 +387,7 @@ Namespace API.Instagram
                     Catch dex As ObjectDisposedException When Disposed
                         Exit Do
                     Catch ex As Exception
-                        If DownloadingException(ex, $"data downloading error [{URL}]") = 1 Then Continue Do Else Exit Do
+                        If DownloadingException(ex, $"data downloading error [{URL}]", False, Section) = 1 Then Continue Do Else Exit Do
                     End Try
                 Loop
             Catch eex2 As ExitException
@@ -306,7 +395,7 @@ Namespace API.Instagram
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
-                ProcessException(DoEx, Token, $"data downloading error [{URL}]")
+                ProcessException(DoEx, Token, $"data downloading error [{URL}]",, Section)
             End Try
         End Sub
         Private Sub DownloadPosts(ByVal Token As CancellationToken)
@@ -327,7 +416,8 @@ Namespace API.Instagram
                             Dim e As New ErrorsDescriber(EDP.ThrowException)
                             For i% = _Index To _SavedPostsIDs.Count - 1
                                 _Index = i
-                                URL = $"https://instagram.com/p/{_SavedPostsIDs(i)}/?__a=1"
+                                'URL = $"https://instagram.com/p/{_SavedPostsIDs(i)}/?__a=1"
+                                URL = $"https://i.instagram.com/api/v1/media/{_SavedPostsIDs(i)}/info/"
                                 ThrowAny(Token)
                                 NextRequest(((i + 1) Mod 5) = 0)
                                 ThrowAny(Token)
@@ -356,21 +446,44 @@ Namespace API.Instagram
                             Next
                         End If
                         _DownloadComplete = True
+                    Catch eex As ExitException
+                        Throw eex
                     Catch oex As OperationCanceledException When Token.IsCancellationRequested
                         Exit Do
                     Catch dex As ObjectDisposedException When Disposed
                         Exit Do
                     Catch ex As Exception
-                        If DownloadingException(ex, $"downloading saved posts error [{URL}]") = 1 Then Continue Do Else Exit Do
+                        If DownloadingException(ex, $"downloading saved posts error [{URL}]", False, Sections.SavedPosts) = 1 Then Continue Do Else Exit Do
                     End Try
                 Loop
+            Catch eex2 As ExitException
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
-                ProcessException(DoEx, Token, $"downloading saved posts error [{URL}]")
+                ProcessException(DoEx, Token, $"downloading saved posts error [{URL}]",, Sections.SavedPosts)
             End Try
         End Sub
 #End Region
+#Region "Code ID converters"
+        Private Shared Function CodeToID(ByVal Code As String) As String
+            Const CodeSymbols$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
+            Try
+                If Not Code.IsEmptyString Then
+                    Dim c As Char
+                    Dim id& = 0
+                    For i% = 0 To Code.Length - 1
+                        c = Code(i)
+                        id = (id * 64) + CodeSymbols.IndexOf(c)
+                    Next
+                    Return id
+                Else
+                    Return String.Empty
+                End If
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Instagram.UserData.CodeToID({Code})", String.Empty)
+            End Try
+        End Function
+#End Region
 #Region "Obtain Media"
         Private Sub ObtainMedia(ByVal node As EContainer, ByVal PostID As String, ByVal PostDate As String, ByVal SpecFolder As String)
             Dim CreateMedia As Action(Of EContainer) =
@@ -390,11 +503,30 @@ Namespace API.Instagram
                 CreateMedia(node)
             End If
         End Sub
-        Private Sub ObtainMedia2(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing)
+        Private Sub ObtainMedia2(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
+                                 Optional ByVal DateObj As String = Nothing)
             Try
                 Dim img As Predicate(Of EContainer) = Function(_img) Not _img.Name.IsEmptyString AndAlso _img.Name.StartsWith("image_versions") AndAlso _img.Count > 0
                 Dim vid As Predicate(Of EContainer) = Function(_vid) Not _vid.Name.IsEmptyString AndAlso _vid.Name.StartsWith("video_versions") AndAlso _vid.Count > 0
                 Dim ss As Func(Of EContainer, Sizes) = Function(_ss) New Sizes(_ss.Value("width"), _ss.Value("url"))
+                Dim mDate As Func(Of EContainer, String) = Function(ByVal elem As EContainer) As String
+                                                               If elem.Contains("taken_at") Then
+                                                                   Return elem.Value("taken_at")
+                                                               ElseIf elem.Contains("imported_taken_at") Then
+                                                                   Return elem.Value("imported_taken_at")
+                                                               Else
+                                                                   Dim ev$ = elem.Value("device_timestamp")
+                                                                   If Not ev.IsEmptyString Then
+                                                                       If ev.Length > 10 Then
+                                                                           Return elem.Value("device_timestamp").Substring(0, 10)
+                                                                       Else
+                                                                           Return ev
+                                                                       End If
+                                                                   Else
+                                                                       Return String.Empty
+                                                                   End If
+                                                               End If
+                                                           End Function
                 If n.Count > 0 Then
                     Dim l As New List(Of Sizes)
                     Dim d As EContainer
@@ -408,6 +540,7 @@ Namespace API.Instagram
                             Case 1
                                 If n.Contains(img) Then
                                     t = n.Value("media_type").FromXML(Of Integer)(-1)
+                                    DateObj = mDate(n)
                                     If t >= 0 Then
                                         With n.ItemF({img, "candidates"}).XmlIfNothing
                                             If .Count > 0 Then
@@ -415,7 +548,7 @@ Namespace API.Instagram
                                                 l.ListAddList(.Select(ss), LNC)
                                                 If l.Count > 0 Then
                                                     l.Sort()
-                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, l.First.Data, PostID, Nothing, SpecialFolder), LNC)
+                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, l.First.Data, PostID, DateObj, SpecialFolder), LNC)
                                                     l.Clear()
                                                 End If
                                             End If
@@ -424,22 +557,24 @@ Namespace API.Instagram
                                 End If
                             Case 2
                                 If n.Contains(vid) Then
+                                    DateObj = mDate(n)
                                     With n.ItemF({vid}).XmlIfNothing
                                         If .Count > 0 Then
                                             l.Clear()
                                             l.ListAddList(.Select(ss), LNC)
                                             If l.Count > 0 Then
                                                 l.Sort()
-                                                _TempMediaList.ListAddValue(MediaFromData(UTypes.Video, l.First.Data, PostID, Nothing, SpecialFolder), LNC)
+                                                _TempMediaList.ListAddValue(MediaFromData(UTypes.Video, l.First.Data, PostID, DateObj, SpecialFolder), LNC)
                                                 l.Clear()
                                             End If
                                         End If
                                     End With
                                 End If
                             Case 8
+                                DateObj = mDate(n)
                                 With n("carousel_media").XmlIfNothing
                                     If .Count > 0 Then
-                                        For Each d In .Self : ObtainMedia2(d, PostID, SpecialFolder) : Next
+                                        For Each d In .Self : ObtainMedia2(d, PostID, SpecialFolder, DateObj) : Next
                                     End If
                                 End With
                         End Select
@@ -451,7 +586,8 @@ Namespace API.Instagram
             End Try
         End Sub
 #End Region
-        Private Sub GetUserId()
+#Region "GetUserId"
+        <Obsolete> Private Sub GetUserId_Old()
             Try
                 Dim r$ = Responser.GetResponse($"https://www.instagram.com/{Name}/?__a=1",, EDP.ThrowException)
                 If Not r.IsEmptyString Then
@@ -467,18 +603,35 @@ Namespace API.Instagram
                 End If
             End Try
         End Sub
+        Private Sub GetUserId()
+            Try
+                Dim r$ = Responser.GetResponse($"https://i.instagram.com/api/v1/users/web_profile_info/?username={Name}",, EDP.ThrowException)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
+                        ID = j({"data", "user"}, "id").XmlIfNothingValue
+                    End Using
+                End If
+            Catch ex As Exception
+                If Responser.StatusCode = HttpStatusCode.NotFound Or Responser.StatusCode = HttpStatusCode.BadRequest Then
+                    Throw ex
+                Else
+                    LogError(ex, "get Instagram user id")
+                End If
+            End Try
+        End Sub
+#End Region
 #Region "Pinned stories"
         Private Sub GetStoriesData(ByRef StoriesList As List(Of String), ByVal Token As CancellationToken)
             Const ReqUrl$ = "https://i.instagram.com/api/v1/feed/reels_media/?{0}"
             Dim tmpList As IEnumerable(Of String)
-            Dim qStr$, r$, sFolder$, storyID$
+            Dim qStr$, r$, sFolder$, storyID$, pid$
             Dim i% = -1
             Dim jj As EContainer, s As EContainer
             ThrowAny(Token)
             If StoriesList.ListExists Then
                 tmpList = StoriesList.Take(5)
                 If tmpList.ListExists Then
-                    qStr = String.Format(ReqUrl, tmpList.Select(Function(q) $"reel_ids=highlight:{q}").ListToString(, "&"))
+                    qStr = String.Format(ReqUrl, tmpList.Select(Function(q) $"reel_ids=highlight:{q}").ListToString("&"))
                     r = Responser.GetResponse(qStr,, EDP.ThrowException)
                     ThrowAny(Token)
                     If Not r.IsEmptyString Then
@@ -486,7 +639,7 @@ Namespace API.Instagram
                             If j.Contains("reels") Then
                                 For Each jj In j("reels")
                                     i += 1
-                                    sFolder = jj.Value("title")
+                                    sFolder = jj.Value("title").StringRemoveWinForbiddenSymbols
                                     storyID = jj.Value("id").Replace("highlight:", String.Empty)
                                     If sFolder.IsEmptyString Then sFolder = $"Story_{storyID}"
                                     If sFolder.IsEmptyString Then sFolder = $"Story_{i}"
@@ -494,7 +647,14 @@ Namespace API.Instagram
                                     If Not storyID.IsEmptyString Then storyID &= ":"
                                     With jj("items").XmlIfNothing
                                         If .Count > 0 Then
-                                            For Each s In .Self : ThrowAny(Token) : ObtainMedia2(s, storyID & s.Value("id"), sFolder) : Next
+                                            For Each s In .Self
+                                                pid = storyID & s.Value("id")
+                                                If Not _TempPostsList.Contains(pid) Then
+                                                    ThrowAny(Token)
+                                                    ObtainMedia2(s, pid, sFolder)
+                                                    _TempPostsList.Add(pid)
+                                                End If
+                                            Next
                                         End If
                                     End With
                                 Next
@@ -515,27 +675,37 @@ Namespace API.Instagram
                 End If
                 Return Nothing
             Catch ex As Exception
-                DownloadingException(ex, "API.Instagram.GetStoriesList")
+                DownloadingException(ex, "API.Instagram.GetStoriesList", False, Sections.Stories)
                 Return Nothing
             End Try
         End Function
 #End Region
-        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+#Region "Download content"
-        End Sub
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
+#End Region
+#Region "Exceptions"
+        ''' <exception cref="ExitException"></exception>
+        ''' <inheritdoc cref="UserDataBase.ThrowAny(CancellationToken)"/>
+        Friend Overrides Sub ThrowAny(ByVal Token As CancellationToken)
+            If MySiteSettings.SkipUntilNextSession Then ExitException.Throw560(Me)
+            MyBase.ThrowAny(Token)
+        End Sub
         ''' <summary>
-        ''' <inheritdoc cref="UserDataBase.DownloadingException(Exception, String)"/><br/>
+        ''' <inheritdoc cref="UserDataBase.DownloadingException(Exception, String, Boolean, Object)"/><br/>
         ''' 1 - continue
         ''' </summary>
-        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False) As Integer
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal s As Object = Nothing) As Integer
             If Responser.StatusCode = HttpStatusCode.NotFound Then
                 UserExists = False
             ElseIf Responser.StatusCode = HttpStatusCode.BadRequest Then
                 HasError = True
-                MyMainLOG = "Instagram credentials have expired"
+                MyMainLOG = $"Instagram credentials have expired [{CInt(Responser.StatusCode)}]: {ToString()} [{s}]"
-                MySiteSettings.HashUpdateRequired.Value = True
+                DisableSection(s)
+            ElseIf Responser.StatusCode = HttpStatusCode.Forbidden And s = Sections.Tagged Then
+                Return 3
             ElseIf Responser.StatusCode = 429 Then
                 With MySiteSettings
                     Dim WaiterExists As Boolean = .LastApplyingValue.HasValue
@@ -545,13 +715,29 @@ Namespace API.Instagram
                 Caught429 = True
                 MyMainLOG = $"Number of requests before error 429: {RequestsCount}"
                 Return 1
+            ElseIf Responser.StatusCode = 560 Then
+                MySiteSettings.SkipUntilNextSession = True
             Else
-                MySiteSettings.HashUpdateRequired.Value = True
+                MyMainLOG = $"Instagram hash requested [{CInt(Responser.StatusCode)}]: {ToString()} [{s}]"
+                DisableSection(s)
                 If Not FromPE Then LogError(ex, Message) : HasError = True
                 Return 0
             End If
             Return 2
         End Function
+        Private Sub DisableSection(ByVal Section As Object)
+            If Not IsNothing(Section) AndAlso TypeOf Section Is Sections Then
+                Dim s As Sections = DirectCast(Section, Sections)
+                Select Case s
+                    Case Sections.Timeline : MySiteSettings.DownloadTimeline.Value = False
+                    Case Sections.SavedPosts : MySiteSettings.DownloadSaved.Value = False
+                    Case Else : MySiteSettings.DownloadStoriesTagged.Value = False
+                End Select
+                MyMainLOG = $"[{s}] downloading is disabled until you update your credentials".ToUpper
+            End If
+        End Sub
+#End Region
+#Region "Create media"
         Private Shared Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
                                               Optional ByVal SpecialFolder As String = Nothing) As UserMedia
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
@@ -561,12 +747,16 @@ Namespace API.Instagram
             m.SpecialFolder = SpecialFolder
             Return m
         End Function
+#End Region
+#Region "Standalone downloader"
         Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Response) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString AndAlso URL.Contains("instagram.com") Then
                     Dim PID$ = RegexReplace(URL, RParams.DMS(".*?instagram.com/p/([_\w\d]+)", 1))
+                    If Not PID.IsEmptyString AndAlso Not ACheck(Of Long)(PID) Then PID = CodeToID(PID)
                     If Not PID.IsEmptyString Then
                         Using t As New UserData
+                            t.SetEnvironment(Settings(InstagramSiteKey), Nothing, False, False)
                             t.Responser = New Response
                             t.Responser.Copy(r)
                             t._SavedPostsIDs.Add(PID)
@@ -577,12 +767,15 @@ Namespace API.Instagram
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, "Instagram standalone downloader: fetch media error")
+                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, $"Instagram standalone downloader: fetch media error ({URL})")
             End Try
         End Function
+#End Region
+#Region "IDisposable Support"
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)
             If Not disposedValue And disposing Then _SavedPostsIDs.Clear()
             MyBase.Dispose(disposing)
         End Sub
+#End Region
     End Class
 End Namespace

SCrawler/API/LPSG/Declarations.vb

@@ -0,0 +1,37 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.LPSG
+    Friend Module Declarations
+        Friend ReadOnly Property PhotoRegEx As RParams = RParams.DM("(https://www.lpsg.com/attachments)(.+?)(?="")", 0, RegexReturn.List)
+        Friend ReadOnly Property PhotoRegExExt As New RParams("img.data.src=""(/proxy[^""]+?)""", Nothing, 1, RegexReturn.List) With {
+            .Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
+        Friend ReadOnly Property NextPageRegex As RParams = RParams.DMS("<link rel=""next"" href=""(.+?/page-(\d+))""", 2)
+        Private Const FileUrlRegexDefault As String = "([^/]+?)(jpg|jpeg|gif|png|webm)"
+        Private ReadOnly InputFReplacer As New ErrorsDescriber(EDP.ReturnValue)
+        Private ReadOnly InputForbidRemover As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, Input, Input.StringRemoveWinForbiddenSymbols(, InputFReplacer))
+        Friend ReadOnly Property FileRegEx As New RParams(FileUrlRegexDefault, Nothing, 0) With {
+            .Converter = Function(ByVal Input As String) As String
+                             Input = InputForbidRemover.Invoke(Input)
+                             If Not Input.IsEmptyString Then
+                                 Dim lv$ = Input.Split("-").LastOrDefault
+                                 If Not lv.IsEmptyString Then
+                                     Input = Input.Replace($"-{lv}", String.Empty)
+                                     Input &= $".{lv}"
+                                 End If
+                             End If
+                             Return Input
+                         End Function}
+        Friend ReadOnly Property FileRegExExt As New RParams(FileUrlRegexDefault, 0, Nothing, InputForbidRemover)
+        Friend ReadOnly Property FileRegExExt2 As New RParams("([^/]+?)(?=(\Z|&))", 0, Nothing, InputForbidRemover)
+        Friend ReadOnly Property FileExistsRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 2)
+        Friend ReadOnly Property TempListAddParams As New ListAddParams(LAP.NotContainsOnly) With {.Comparer = New FComparer(Of UserMedia)(Function(x, y) x.URL = y.URL)}
+    End Module
+End Namespace

SCrawler/API/LPSG/SiteSettings.vb

@@ -0,0 +1,37 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.LPSG
+    <Manifest("AndyProgram_LPSG")>
+    Friend Class SiteSettings : Inherits Base.SiteSettingsBase
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.LPSGIcon_48
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.LPSGPic_32
+            End Get
+        End Property
+        Friend Sub New()
+            MyBase.New("LPSG", "www.lpsg.com")
+            UrlPatternUser = "https://www.lpsg.com/threads/{0}/"
+            UserRegex = RParams.DMS(".+?lpsg.com/threads/([^/]+)", 1)
+        End Sub
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return If(Responser.Cookies?.Count, 0) > 0
+        End Function
+    End Class
+End Namespace

SCrawler/API/LPSG/UserData.vb

@@ -0,0 +1,104 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Imports Converters = PersonalUtilities.Functions.SymbolsConverter.Converters
+Namespace API.LPSG
+    Friend Class UserData : Inherits UserDataBase
+        Private Const Name_LatestPage As String = "LatestPage"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            If Loading Then
+                LatestPage = Container.Value(Name_LatestPage)
+            Else
+                Container.Add(Name_LatestPage, LatestPage)
+            End If
+        End Sub
+        Private Property LatestPage As String = String.Empty
+        Private Enum Mode : Internal : External : End Enum
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Responser.Error = EDP.ThrowException
+
+                Dim NextPage$
+                Dim r$
+                Dim _LPage As Func(Of String) = Function() If(LatestPage.IsEmptyString, String.Empty, $"page-{LatestPage}")
+
+                Do
+                    URL = $"https://www.lpsg.com/threads/{Name}/{_LPage.Invoke}"
+                    r = Responser.GetResponse(URL)
+                    UserExists = True
+                    UserSuspended = False
+                    ThrowAny(Token)
+                    If Not r.IsEmptyString Then
+                        NextPage = RegexReplace(r, NextPageRegex)
+                        UpdateMediaList(RegexReplace(r, PhotoRegEx), Mode.Internal)
+                        UpdateMediaList(RegexReplace(r, PhotoRegExExt), Mode.External)
+                        If NextPage = LatestPage Or NextPage.IsEmptyString Then Exit Do Else LatestPage = NextPage
+                    Else
+                        Exit Do
+                    End If
+                Loop
+
+                If _TempMediaList.ListExists And _ContentList.ListExists Then _
+                   _TempMediaList.RemoveAll(Function(m) _ContentList.Exists(Function(mm) mm.URL = m.URL))
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+        Private Sub UpdateMediaList(ByVal l As List(Of String), ByVal m As Mode)
+            If l.ListExists Then
+                Dim f As SFile
+                Dim u$
+                Dim exists As Boolean
+                Dim r As RParams
+                Dim ude As New ErrorsDescriber(EDP.ReturnValue)
+                For Each url$ In l
+                    If Not url.IsEmptyString Then u = SymbolsConverter.Decode(url, {Converters.HTML, Converters.ASCII}, ude) Else u = String.Empty
+                    If Not u.IsEmptyString Then
+                        exists = Not IsEmptyString(RegexReplace(u, FileExistsRegEx))
+                        If m = Mode.Internal Then
+                            r = FileRegEx
+                        Else
+                            r = FileRegExExt
+                            If Not exists Then
+                                r = FileRegExExt2
+                                exists = Not IsEmptyString(RegexReplace(u, FileRegExExt2))
+                            End If
+                        End If
+                        If exists Then
+                            f = CStr(RegexReplace(u, r))
+                            f.Path = MyFile.CutPath.PathNoSeparator
+                            f.Separator = "\"
+                            If f.Extension.IsEmptyString Then f.Extension = "jpg"
+                            _TempMediaList.ListAddValue(New UserMedia With {.Type = UTypes.Picture, .URL = url, .File = f}, TempListAddParams)
+                        End If
+                    End If
+                Next
+            End If
+        End Sub
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            With Responser : .UseWebClient = True : .UseWebClientCookies = True : .ResetError() : End With
+            UseResponserClient = True
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                MyMainLOG = $"{ToStringForLog()}: LPSG not available"
+                Return 1
+            Else
+                Return 0
+            End If
+        End Function
+    End Class
+End Namespace