2023.1.24.1

Some Imgur albums won't download
Added icon for standalone downloader

.gitignore

@@ -34,7 +34,9 @@ bld/
 [Ll]og/
 [Ll]ogs/
 ffmpeg/
+cURL/
 Info/
+Hidden/
 
 # Visual Studio 2015/2017 cache/options directory
 .vs/

CONTRIBUTING.md

@@ -8,20 +8,20 @@ I welcome requests! Follow these steps to contribute:
 1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
 
 # How to build from source
-
-1. Delete the "PersonalUtilities" project from the solution.
-1. Delete the "PersonalUtilities.Notifications" project from the solution.
+1. Delete the ```PersonalUtilities``` project from the solution.
+1. Delete the ```PersonalUtilities.Notifications``` project from the solution.
+1. Delete the ```cURL``` folder from the solution.
+1. Delete the ```ffmpeg.exe``` from the solution.
 1. The following libraries must be added to project references with the '**Copy to output folder**' option:
-  - ```PersonalUtilities.dll```
-  - ```PersonalUtilities.Notifications.dll```
-  - ```Microsoft.Toolkit.Uwp.Notifications.dll```
-  - ```System.ValueTuple.dll```
-1. Import PersonalUtilities.Functions for the whole project.
+    - ```PersonalUtilities.dll```
+    - ```PersonalUtilities.Notifications.dll```
+    - ```Microsoft.Toolkit.Uwp.Notifications.dll```
+    - ```System.ValueTuple.dll```
+1. Import ```PersonalUtilities.Functions``` for the whole project.
 
 **Always use the correct libraries. You must download libraries from the same release date as the code commit date.**
 
 # How to request a new site
-
 1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
 1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
 
@@ -38,14 +38,5 @@ I welcome requests! Follow these steps to contribute:
 If I'm interested in a site you want to add, it may be added in future releases.
 
 # Sites I will never develop
-
 - Facebook
-
-# Sites requested by users
-
-- TikTok
-  - API for receiving data without authorization was not found. Therefore, I don't have time to start developing this site parsing algorithm. If anyone knows of requests that may collect data without OAuth authentication, please let me know.
-  
-# Contact me
-
-[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)
+- Tumblr

Changelog.md

@@ -1,3 +1,243 @@
+# 2023.1.24.1
+
+*2023-01-24*
+
+- Added
+  - Icon for standalone downloader
+- Fixed
+  - (Issue #100) some Imgur albums won't download
+
+# 2023.1.24.0
+
+*2023-01-24*
+
+- Fixed
+  - (Issue #100) Imgur albums not downloading
+  - When deleting a collection with the 'ban' option, users in the collection are not banned
+
+# 2023.1.2.0
+
+*2023-01-02*
+
+- Added
+  - RedGifs: an ability to customize token refresh interval
+  - RedGifs: token refresh interval changed from 24 hours to 12 hours
+  - Updated labels collection
+- Fixed
+  - PornHub: bug in the downloader
+  - PornHub: download additional non-user videos
+  - Reddit: bug in standalone downloader
+  - Fixed a bug in the user list loading algorithm
+  - Notifications: pressing any button opens SCrawler
+
+# 2022.12.27.0
+
+*2022-12-27*
+
+- Added
+  - XVideos: added downloading 'Quickies'
+  - Instagram: added more enable/disable options
+- Fixed
+  - XVideos not downloading (sorry, I broke it in a previous release)
+
+# 2022.12.26.0
+
+*2022-12-26*
+
+**ATTENTION!**
+
+**Instagram requirements changed. Headers and cookies are now required to download Timeline, Stories and Saved posts; hash to download tagged posts. Please update your credentials.**
+
+**Instagram tagged posts no longer provide the total amount of tagged posts. I've corrected the tagged posts notification, but now I can't tell how many requests will be spent on downloading tagged posts. And from now on, one request will be spent on downloading each tagged post, because Instagram doesn't provide complete information about the tagged post with the site's response. In this case, if the number of tagged posts is 1000, 1000 requests will be spent. Be careful when downloading them. I highly recommend that you forcefully disable the downloading of tagged posts for a while.**
+
+- Added
+  - Updated user loading algorithm
+  - Channels button to tray context menu
+  - (Request #96) Add FFmpeg to x86 version
+- Fixed
+  - PornHub wrong behavior when downloading images
+  - Unable open XVideos user profile
+  - Cannot delete multiple collections at once
+  - Can't focus user from the download info form
+  - Instagram downloader not working
+  - (Issue #69) **RedGifs data is not downloading**. Again.
+  - Minor bugs
+
+# 2022.11.16.0
+
+*2022-11-16*
+
+**ATTENTION! This version makes changes to the base SCrawler user configuration file. Since you started using this version, you still can downgrade. BUT! Once you add a virtual collection or a virtual user to a collection, you won't be able to downgrade without losing data.**
+
+- Added
+  - **PornHub**
+  - **XHamster**
+  - An ability to download saved XVIDEOS posts
+  - Download indicator. While downloading, the rainbow tray icon changed to a blue arrow.
+  - Collections: the ability to edit a collection using a form
+  - Collections: the ability to create a **`virtual collection`** and add a **`virtual user`** to a real collection
+  - Collections: an easier way to added users to a collection
+  - Collections: an easier way to create collections
+  - Added icons for channels form context menu buttons
+  - More convenient change of user labels from the context menu of the user list
+  - Notifications: complete transition from default notifications to ToastNotifications
+  - Notifications: when you click on the notification that some of the channels are downloaded, the channels form opens
+  - Notifications: when you click on the notification that all users are downloaded, the main window form opens
+  - Notifications: when you click on the notification that the saved posts are downloaded, the saved posts form opens
+  - Import users
+  - Minor improvements
+- Plugins
+  - Added
+    - `TaskGroup` attribute
+    - `IUserMedia` interface
+  - Changed
+    - `GetUserUrl` and `GetUserPostUrl` functions: `String UserName` and `String UserID` changed to ` IPluginContentProvider User`
+- Fixed
+  - Collections editor: new added collections are still not added to the top of the collections list
+  - Users search form doesn't remember last size
+  - Minor bugs
+
+# 2022.10.23.0
+
+*2022-10-23*
+
+- Added
+  - RedGifs token Auto-Renewal
+  - Download groups: ability to select sites
+  - Download groups: ability to exclude labels and sites
+  - AutoDownloader: ability to exclude labels and sites in ```All```, ```Default``` and ```Specified``` modes
+  - The ```Download All``` button turns blue when pause is enabled
+  - Updated Twitter status codes
+  - Minor improvements
+- Fixed
+  - Updated Twitter status codes
+  - AutoDownloader: incorrect next run date in scheduler task information
+  - AutoDownloader: minor bugs
+  - (Issue #69) **RedGifs data is not downloading**. Requires token.
+  - Minor bugs
+
+# 2022.10.18.0
+
+*2022-10-18*
+
+- Added
+  - **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
+  - **Search form** (```Ctrl+F```)
+  - Feed improvements
+  - Ability to save the download session for viewing later
+  - Ability to download user, excluding from the feed (use the ```Ctrl``` key with a button click of with a hot key press)
+  - Ability to disable the notification about the absence of the ffmpeg.exe file
+  - Extended user information with labels  
+  - Advanced AutoDownloader pause options
+  - Added pause buttons to tray icon and AutoDownloader form
+  - Additional Instagram protection
+  - Advanced notification management
+  - Silent mode (temporarily disable notification)
+  - Excluding users whose profiles do not exist from downloading with groups and AutoDownloader
+  - Minor improvements
+- Updated
+  - Grouped all download buttons into one menu
+  - **Finished missing posts**. You can now download missing posts if they exist.
+  - PluginProvider: added ```BeginEdit``` and ```EndEdit``` methods
+  - PluginProvider: ```GetSpecialData``` return type changed from ```IEnumerable(Of PluginUserMedia)``` to ```IEnumerable```
+  - XVIDEOS and LPSG plugins are moved from libraries to SCrawler
+- Fixed
+  - (Issue #69) **RedGifs data is not downloading**. Requires cookies and token.
+  - Some minor bugs when deleting a collection
+  - Feed: start video playing may cause the program to freeze (strange behavior of the vlc library)
+  - Feed: videos hosted on Reddit not showing up in feed
+  - Feed: minor bugs
+  - Collection users were not banned when deleted with the ban option
+  - When trying to delete multiple collections, each collection asked for confirmation to delete
+  - Minor bugs
+
+# 2022.9.24.0
+
+*2022-09-24*
+
+- Added
+  - Ability to copy user data to another destination
+  - Ability to add 'Session' and 'Date' values to the post title in the feed
+  - Minor feed improvements
+  - The newly created collection will now appear at the top of the list (after reopening the form)
+  - Ability to add multiple users at a time to the collection.
+- Fixed
+  - Autodownloader opens a compressed image instead of a full one
+  - Incorrect resizing of the feed grid after deleting a media file
+  - Incorrect behavior when deleting/removing a user from a collection.
+  - An incorrect function that displayed the number of spent Instagram requests.
+  - Bug in the XVIDEOS downloader
+  - Minor bugs
+
+# 2022.9.17.0
+
+*2022-09-17*
+
+- Added
+  - Added two date filters to filter users (in range, not in range)
+  - (Request #71) Download data for a specific date range
+  - The ability to disable site downloading (in the site settings form)
+- Updated
+  - Plugins
+- Fixed
+  - (Issue #71) ```Download data to the date``` doesn't work for Twitter
+  - Download data for a specific date range doesn't work for multiple users
+  - Incorrect feed sorting algorithm
+  - Minor bugs
+
+# 2022.9.16.0
+
+*2022-09-16*
+
+- Fixed
+  - Failed to get video thumbnail for channel video post
+  - Incorrect rendering of the 'Feed' table when the number of columns is more than one
+  - Minor design bugs
+
+# 2022.9.13.0
+
+*2022-09-13*
+
+- Added
+  - Video duration to the feed
+- Fixed
+  - (Issue #70) Instagram posts not downloading if there are pinned posts that have already been downloaded
+  - Minor bugs
+
+# 2022.9.10.0
+
+*2022-09-10*
+
+- Fixed
+  - The memory is still leaking. This time because of the video. *Using WMP was not the best choice.*
+
+# 2022.9.8.1
+
+*2022-09-08*
+
+- Fixed
+  - Unexpected memory leak when using the 'Feed' form
+
+# 2022.9.8.0
+
+*2022-09-08*
+
+- Added
+  - **Feed** (feed of downloaded media files)
+  - Missing posts tracking and management
+  - Simple scheduler notifications
+- Fixed
+  - (Issue #67) Saved Instagram posts not downloading
+
+# 2022.8.28.0
+
+*2022-08-28*
+
+- Added
+  - RedGifs icon
+- Fixed
+  - Incorrect number of posts displayed in the Reddit channels downloader.
+
 # 2022.8.22.0
 
 *2022-08-22*

FAQ.md

@@ -18,7 +18,7 @@ A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
 
 #### Q: **I can't copy cookies.**
 
-A: Use the mouse. Don't use ```Ctrl``` + ```A```!
+A: Use the mouse. Don't use ```Ctrl+A```!
 
 ----
 
@@ -42,7 +42,9 @@ A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-
 
 #### Q: **Twitter/Instagram download failed.**
 
-A: Check your credentials. Both of these sites require cookies. Check your [Twitter tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) and [Instagram settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-settings). If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
+A: Check your credentials. Both of these sites require cookies. Check your [Twitter tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) and [Instagram settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram). If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
+
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
 ----
 
@@ -94,6 +96,24 @@ A: Just add that user back to the program. In the dialog box that opens, click o
 
 ----
 
+#### Q: **Why don't you answer how it works**
+
+A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
+
+----
+
+#### Q: **You lost me. Your program is too complicated.**
+
+A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
+
+----
+
+#### Q: **I can't configure something**
+
+A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
+
+----
+
 #### Q: **Can you add a step-by-step guide or video on how to use the program?**
 
 A: **NO**! I will not do it. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.

HowToSupport.md

@@ -1,11 +1,12 @@
 Your support is very valuable to me. Any support is greatly appreciated. Your support encourages me to make new features, update the program, add new sites, etc.
 
 You can support the program by:
-  - **Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
+  - **Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
   - :heavy_dollar_sign: make a donation on this site: https://ko-fi.com/andyprogram
   - :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
   - :speech_balloon: tell your friends about the program
   - :heart: like the program on this site: https://alternativeto.net/software/scrawler/about/
+  - :heart: like the program on this site: https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml
   - suggest my program as an alternative ([on this site](https://alternativeto.net/software/scrawler/about/)) to any program you have used before
 
 I would be very grateful for any support! :blush:

Plugins.md

@@ -1,6 +1,4 @@
 List of available plugins:
-- LPSG
-- XVIDEOS
 
 Tools:
 - [image2post](https://github.com/unknown81311/SCrawler-image2post) by @unknown81311: **get reddit post URL from file.**

ProgramsComparison.md

@@ -18,7 +18,7 @@ https://www.4kdownload.com/products/product-stogram
 | Download posts by location | No | **Yes** |
 | Save Private Instagram Content with Permission| Yes | Yes |
 | Download Instagram Stories and Highlights | Yes | Yes |
-| See Others Instagram Feed As Your Own | No | **Yes** |
+| See Others Instagram Feed As Your Own | Yes | Yes |
 | Download Instagram Video Posts | Yes | Yes |
 | Backup Your Instagram Account | Yes | Yes |
 | Save Instagram Posts by Date | Yes | Yes |
@@ -31,7 +31,7 @@ https://www.4kdownload.com/products/product-stogram
 | Automatic Subscriptions Update | **Free** | Paid (43.56 EUR) |
 | Posts and Captions Export | No | Paid (43.56 EUR) |
 | Advertisements free | **No ADs at all for free** | Paid (14.52) |
-| Operating Systems | Windows 7+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
+| Operating Systems | Windows 10+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
 | Select want content type to download | **Yes** | No |
 | Instagram support | Yes | Yes |
 | Twitter support | **Yes** | No |
@@ -66,9 +66,9 @@ https://github.com/RipMeApp/ripme
 | Export and import subscriptions | No | No |
 | **Paid** | **No** | **No** |
 | **Free options** | The program is completely free | The program is completely free, but site limits are not declared |
-| Operating Systems | Windows 7+ | Windows, MacOS, Linux |
+| Operating Systems | Windows 10+ | Windows, MacOS, Linux |
 | Select want content type to download | Yes | Yes |
-| Suported sites | 3 internal and any site using plugins | 86+ sites (declared) |
+| Suported sites | 9 internal and any site using plugins | 86+ sites (declared) |
 | Other sites support | **Yes** | No |
 | Still supported | **Yes** | **No (last release date May 4, 2021)** |
 

README.md

@@ -2,49 +2,53 @@
 
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest)
 [![GitHub license](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
+[![GitHub all releases](https://img.shields.io/github/downloads/aandyprogram/scrawler/total?label=Total%20downloads)](https://github.com/AAndyProgram/SCrawler/releases)
 [![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
 [![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
 [![How to support](https://img.shields.io/badge/HowToSupport-green)](HowToSupport.md)
 
-A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram).
+A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, LPSG).
+
+**If you like SCrawler, please like the program on [this site]( https://alternativeto.net/software/scrawler/about/)**
 
 Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
 
 [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
 
-**Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
+**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
 
 ![Main window](ProgramScreenshots/MainWindow.png)
 ![Channels window](ProgramScreenshots/Channels.png)
 
 # What can program do:
 - Download pictures and videos from users' profiles and subreddits:
-  - Reddit images;
-  - Reddit galleries of images;
-  - Reddit videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
+  - Reddit images, galleries of images, videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**)), saved posts;
   - Redgifs videos (https://www.redgifs.com/);
-  - Twitter images and videos;
-  - Instagram images and videos;
-  - Instagram tagged posts;
-  - Instagram stories;
+  - Twitter images and videos, saved (bookmarked) posts;
+  - Instagram images and videos, tagged posts, stories, saved posts;
+  - TikTok videos ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
   - Imgur images, galleries and videos;
   - Gfycat videos;
+  - PornHub images, videos, save (liked) posts;
+  - XHamster images, videos, saved posts;
+  - XVIDEOS videos;
   - [Other](#supported-sites) supported sites
 - Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
 - Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
 - Add users from parsed channel
 - **Advanced user management**
-- **Automation** (downloading data automatically every ```X``` minutes)
+- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
+- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files)
 - Labeling users
-- Create download groups
+- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
 - Adding users to favorites and temporary
-- Filter exists users by label or group
+- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
 - Selection of media types you want to download (images only, videos only, both)
-- Download a special video, image or gallery
-- Making collections (grouping users into collections)
+- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
+- Making [collections](https://github.com/AAndyProgram/SCrawler/wiki#collections) (grouping users into collections)
 - Specifying a user folder (for downloading data to another location)
 - Changing user icons
-- Changing view modes
+- Changing [view modes](https://github.com/AAndyProgram/SCrawler/wiki#view)
 - ...and many others...
 
 # Supported sites
@@ -52,26 +56,29 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 - **Reddit**
 - **Twitter**
 - **Instagram**
+- **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
 - RedGifs
 - Imgur
 - Gfycat
 - LPSG
-- XVIDEOS
+- **PornHub**
+- **XHamster**
+- **XVIDEOS**
 - [Other sites](Plugins.md)
 
-# How does it works:
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
+
+# How it works
 
 First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
 
 ## Reddit
 
-The program parses all user posts, obtain MD5  images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
+The program parses user posts, obtain MD5  images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
 
 ## Other sites
 
-The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
-
-You can read about Instagram restrictions [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-limits)
+The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
 
 ## How to request a new site
 
@@ -80,17 +87,40 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
 # Requirements
 
 - Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for Twitter (if you want to download data from Twitter)
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for Instagram (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for saved Instagram posts, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for Stories and Tagged data
-- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)). **ffmpeg only works with the x64 version of the program.**
+- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
 # Guide
 
+- [Main window](https://github.com/AAndyProgram/SCrawler/wiki)
+  - [Users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
+    - [Add/Edit/Delete users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
+    - [Collections](https://github.com/AAndyProgram/SCrawler/wiki#collections)
+    - [User operations](https://github.com/AAndyProgram/SCrawler/wiki#context-menu)
+    - [User labels](https://github.com/AAndyProgram/SCrawler/wiki/Users#labels)
+  - **[DOWNLOAD](https://github.com/AAndyProgram/SCrawler/wiki#download)**
+    - [Automation](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
+    - [Download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
+  - [Downloading information](https://github.com/AAndyProgram/SCrawler/wiki#info)
+  - [Reddit channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
+  - [Saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)
+  - [View modes, filters](https://github.com/AAndyProgram/SCrawler/wiki#view)
+- **[SETTINGS](https://github.com/AAndyProgram/SCrawler/wiki/Settings)**
+  - **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
+  - [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
+  - [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
+  - [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
+  - [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
+  - [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
+  - [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
+  - [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
+  - [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
+  - [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
+
 **Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
 
 # Installation
 
-**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
+**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
 
 **Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
 
@@ -108,28 +138,15 @@ Read about how to make plugin [here](https://github.com/AAndyProgram/SCrawler/wi
 
 # How to support
 
-Read more about how to support the program [here](HowToSupport.md).
+Read about how to support the program [here](HowToSupport.md).
 
 # Settings and usage
 
 The program has an intuitive interface.
 
-You need to set up authorization for Twitter and Instagram:
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for **Twitter** (if you want to download data from Twitter)
-- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies), [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) and [authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-authorization-headers) for **Instagram** (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for **saved Instagram posts**
+**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
-Just add a user profile and **click the ```Start downloading``` button**.
-
-You can add users by patterns:
-- https://www.instagram.com/SomeUserName
-- https://twitter.com/SomeUserName
-- https://reddit.com/user/SomeUserName
-- https://reddit.com/r/SomeSubredditName
-- https://www.redgifs.com/users/SomeUserName
-- u/SomeUserName
-- r/SomeSubredditName
-- SomeUserName (in this case, you need to select the user's site)
-- SomeSubredditName
+Just add a user profile and **click the ```Download``` button**.
 
 Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
 
@@ -141,8 +158,4 @@ Create a shortcut for the program. Open shortcut properties. In the ```Shortcut`
 
 Example: ```D:\Programs\SCrawler\SCrawler.exe v```
 
-![Separate video downloader](ProgramScreenshots/SeparateVideoDownloader.png)
-
-# Contact me
-
-[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)
+![Separate video downloader](ProgramScreenshots/SeparateVideoDownloader.png)

SCrawler.Plugin.LPSG/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2022.8.22.0")>
-<Assembly: AssemblyFileVersion("2022.8.22.0")>
+<Assembly: AssemblyVersion("2022.10.12.0")>
+<Assembly: AssemblyFileVersion("2022.10.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.Plugin.LPSG/SiteSettings.vb

@@ -54,6 +54,10 @@ Public Class SiteSettings : Implements ISiteSettings
     End Sub
 #End Region
 #Region "Update"
+    Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
+    End Sub
+    Public Sub EndEdit() Implements ISiteSettings.EndEdit
+    End Sub
     Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
     End Sub
     Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -92,9 +96,12 @@ Public Class SiteSettings : Implements ISiteSettings
         Return Nothing
     End Function
     Public Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
-        Return True
+        Return If(Responser.Cookies?.Count, 0) > 0
     End Function
     Public Function ReadyToDownload(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.ReadyToDownload
         Return True
     End Function
+    Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+        Return String.Empty
+    End Function
 End Class

SCrawler.Plugin.LPSG/UserData.vb

@@ -38,7 +38,8 @@ Public Class UserData : Implements IPluginContentProvider
     Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
     Public Property DataPath As String Implements IPluginContentProvider.DataPath
     Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
-    Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
+    Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
+    Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
 #End Region
 #Region "Interface exchange options"
     Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -152,7 +153,8 @@ Public Class UserData : Implements IPluginContentProvider
                         If Responser.Client.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
                             LogProvider.Add("LPSG not available")
                         Else
-                            m.DownloadState = UStates.Skipped
+                            m.DownloadState = UStates.Missing
+                            m.Attempts += 1
                         End If
                     End Try
                     RaiseEvent ProgressChanged(1)

SCrawler.Plugin.XVIDEOS/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2022.8.22.0")>
-<Assembly: AssemblyFileVersion("2022.8.22.0")>
+<Assembly: AssemblyVersion("2022.10.12.0")>
+<Assembly: AssemblyFileVersion("2022.10.12.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.Plugin.XVIDEOS/SettingsForm.vb

@@ -35,7 +35,7 @@ Public Class SettingsForm
         Settings.UpdateDomains()
         MyDefs.CloseForm()
     End Sub
-    Private Sub MyDefs_ButtonAddClick(ByVal Sender As Object, ByVal e As EditToolbar.EditToolbarEventArgs) Handles MyDefs.ButtonAddClick
+    Private Sub MyDefs_ButtonAddClick(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonAddClick
         Dim nd$ = InputBoxE("Enter a new domain using the pattern [xvideos.com]:", "New domain")
         If Not nd.IsEmptyString Then
             If Not LIST_DOMAINS.Items.Contains(nd) Then
@@ -45,7 +45,7 @@ Public Class SettingsForm
             End If
         End If
     End Sub
-    Private Sub MyDefs_ButtonDeleteClickE(ByVal Sender As Object, ByVal e As EditToolbar.EditToolbarEventArgs) Handles MyDefs.ButtonDeleteClickE
+    Private Sub MyDefs_ButtonDeleteClickE(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonDeleteClickE
         If _LatestSelected.ValueBetween(0, LIST_DOMAINS.Items.Count - 1) Then
             Dim n$ = LIST_DOMAINS.Items(_LatestSelected)
             If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?", "Removing domains"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then

SCrawler.Plugin.XVIDEOS/SiteSettings.vb

@@ -97,6 +97,10 @@ Public Class SiteSettings : Implements ISiteSettings
     Public Sub DownloadDone(ByVal What As ISiteSettings.Download) Implements ISiteSettings.DownloadDone
     End Sub
 #End Region
+    Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
+    End Sub
+    Public Sub EndEdit() Implements ISiteSettings.EndEdit
+    End Sub
     Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
     End Sub
     Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -179,4 +183,7 @@ Public Class SiteSettings : Implements ISiteSettings
         End If
         Return Nothing
     End Function
+    Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+        Return String.Empty
+    End Function
 End Class

SCrawler.Plugin.XVIDEOS/UserData.vb

@@ -37,7 +37,8 @@ Public Class UserData : Implements IPluginContentProvider
     Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
     Public Property DataPath As String Implements IPluginContentProvider.DataPath
     Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
-    Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
+    Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
+    Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
 #End Region
 #Region "Interface exchange options"
     Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -70,7 +71,7 @@ Public Class UserData : Implements IPluginContentProvider
 
             Dim NextPage% = 0
             Dim r$
-            Dim j As EContainer, jj As EContainer
+            Dim jj As EContainer
             Dim e As ErrorsDescriber = EDP.ThrowException
             Dim user$ = Settings.GetUserUrl(Name, False)
             Dim p As PluginUserMedia
@@ -81,8 +82,7 @@ Public Class UserData : Implements IPluginContentProvider
                 r = Responser.GetResponse($"https://www.xvideos.com/{user}/videos/new/{If(NextPage = 0, String.Empty, NextPage)}",, e)
                 If Not r.IsEmptyString Then
                     If Not EnvirSet Then UserExists = True : UserSuspended = False : EnvirSet = True
-                    j = JsonDocument.Parse(r).XmlIfNothing
-                    With j
+                    With JsonDocument.Parse(r).XmlIfNothing
                         If .Contains("videos") Then
                             With .Item("videos")
                                 If .Count > 0 Then
@@ -93,9 +93,12 @@ Public Class UserData : Implements IPluginContentProvider
                                             .URL = $"https://www.xvideos.com{jj.Value("u")}"
                                         }
                                         If Not p.PostID.IsEmptyString And Not jj.Value("u").IsEmptyString Then
-                                            If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else Exit Do
+                                            If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else .Dispose() : Exit Do
                                         End If
                                     Next
+                                Else
+                                    .Dispose()
+                                    Exit Do
                                 End If
                             End With
                         Else
@@ -208,7 +211,8 @@ Public Class UserData : Implements IPluginContentProvider
                         m.File = f
                         m.DownloadState = UStates.Downloaded
                     Catch ex As Exception
-                        m.DownloadState = UStates.Skipped
+                        m.DownloadState = UStates.Missing
+                        m.Attempts += 1
                     End Try
                     TempMediaList(i) = m
                     RaiseEvent ProgressChanged(1)

SCrawler.PluginProvider/.editorconfig

@@ -1,3 +1,3 @@
 [*.vb]
 # Modifier preferences
-file_header_template = Copyright (C) 2022  Andy\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>
+file_header_template = Copyright (C) 2023  Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>

SCrawler.PluginProvider/Attributes/Attributes.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -53,6 +53,9 @@ Namespace Plugin.Attributes
             ElementName = XMLElementName
         End Sub
     End Class
+    ''' <summary>Attribute to disable some properties for host use</summary>
+    <AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class DoNotUse : Inherits Attribute
+    End Class
     ''' <summary>Special property updater</summary>
     <AttributeUsage(AttributeTargets.Method, AllowMultiple:=True, Inherited:=False)> Public NotInheritable Class PropertyUpdater : Inherits Attribute
         Public ReadOnly Name As String
@@ -129,13 +132,26 @@ Namespace Plugin.Attributes
         ''' Predefined task counter.<br/>
         ''' <see cref="TaskCounter"/> will take precedence if it is defined.
         ''' </param>
-        Public Sub New(Optional ByVal JobsCount As Integer = -1)
-            TasksCount = JobsCount
+        Public Sub New(Optional ByVal TasksCount As Integer = -1)
+            Me.TasksCount = TasksCount
         End Sub
     End Class
     ''' <summary>A property attribute that specifies how many users should be downloaded at the same time in one thread</summary>
     <AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskCounter : Inherits Attribute
     End Class
+    ''' <remarks>
+    ''' This attribute cannot be combined with <see cref="SeparatedTasks"/>.
+    ''' If set to <see cref="SeparatedTasks"/>, this attribute will be ignored
+    ''' </remarks>
+    ''' <inheritdoc cref="SeparatedTasks"/>
+    <AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskGroup : Inherits Attribute
+        Public ReadOnly Name As String
+        ''' <summary>Initialize a new TaskGroup attribute.</summary>
+        ''' <param name="Name">Group name</param>
+        Public Sub New(ByVal Name As String)
+            Me.Name = Name
+        End Sub
+    End Class
     ''' <summary>This attribute indicates that the plugin has a SavedPosts environment</summary>
     <AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class SavedPosts : Inherits Attribute
     End Class

SCrawler.PluginProvider/Interfaces/IPluginContentProvider.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -17,16 +17,17 @@ Namespace Plugin
         Property ID As String
         Property ParseUserMediaOnly As Boolean
         Property UserDescription As String
-        Property ExistingContentList As List(Of PluginUserMedia)
+        Property ExistingContentList As List(Of IUserMedia)
         Property TempPostsList As List(Of String)
-        Property TempMediaList As List(Of PluginUserMedia)
+        Property TempMediaList As List(Of IUserMedia)
         Property UserExists As Boolean
         Property UserSuspended As Boolean
         Property IsSavedPosts As Boolean
         Property SeparateVideoFolder As Boolean
         Property DataPath As String
         Property PostsNumberLimit As Integer?
-        Property PostsDateLimit As Date?
+        Property DownloadDateFrom As Date?
+        Property DownloadDateTo As Date?
         Function ExchangeOptionsGet() As Object
         Sub ExchangeOptionsSet(ByVal Obj As Object)
         Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))

SCrawler.PluginProvider/Interfaces/ISiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -18,11 +18,12 @@ Namespace Plugin
         ReadOnly Property Image As Image
         ReadOnly Property Site As String
         Property Logger As ILogProvider
-        Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String
+        Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
         Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
         Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
-        Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia)
+        Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
         Function GetInstance(ByVal What As Download) As IPluginContentProvider
+        Function GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String
 #Region "XML Support"
         Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))
 #End Region
@@ -31,6 +32,8 @@ Namespace Plugin
         Sub EndInit()
         Sub BeginUpdate()
         Sub EndUpdate()
+        Sub BeginEdit()
+        Sub EndEdit()
 #End Region
 #Region "Site availability"
         Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean

SCrawler.PluginProvider/My Project/AssemblyInfo.vb

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
 <Assembly: AssemblyDescription("Plugin provider for SCrawler")>
 <Assembly: AssemblyCompany("AndyProgram")>
 <Assembly: AssemblyProduct("SCrawler.PluginProvider")>
-<Assembly: AssemblyCopyright("Copyright © 2022")>
+<Assembly: AssemblyCopyright("Copyright © 2023")>
 <Assembly: AssemblyTrademark("AndyProgram")>
 
 <Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2022.8.22.0")>
-<Assembly: AssemblyFileVersion("2022.8.22.0")>
+<Assembly: AssemblyVersion("2022.12.26.0")>
+<Assembly: AssemblyFileVersion("2022.12.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.PluginProvider/ObjectInterfaces/ILogProvider.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/ObjectInterfaces/IThrower.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/Objects/ExchangeOptions.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/Objects/PluginUserMedia.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -7,24 +7,46 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Namespace Plugin
-    Public Structure PluginUserMedia
-        Enum Types As Integer
-            Undefined = 0
-            [Picture] = 1
-            [Video] = 2
-            [Text] = 3
-            VideoPre = 10
-            GIF = 50
-            m3u8 = 100
-        End Enum
-        Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : End Enum
-        Public ContentType As Integer
-        Public URL As String
-        Public MD5 As String
-        Public File As String
-        Public DownloadState As Integer
-        Public PostID As String
-        Public PostDate As Date?
-        Public SpecialFolder As String
+    Public Enum UserMediaTypes As Integer
+        Undefined = 0
+        [Picture] = 1
+        [Video] = 2
+        [Text] = 3
+        VideoPre = 10
+        GIF = 50
+        m3u8 = 100
+    End Enum
+    Public Enum UserMediaStates As Integer
+        Unknown = 0
+        Tried = 1
+        Downloaded = 2
+        Skipped = 3
+        Missing = 4
+    End Enum
+    Public Structure PluginUserMedia : Implements IUserMedia
+        Public Property ContentType As Integer Implements IUserMedia.ContentType
+        Public Property URL As String Implements IUserMedia.URL
+        Public Property URL_BASE As String Implements IUserMedia.URL_BASE
+        Public Property MD5 As String Implements IUserMedia.MD5
+        Public Property File As String Implements IUserMedia.File
+        Public Property DownloadState As Integer Implements IUserMedia.DownloadState
+        Public Property PostID As String Implements IUserMedia.PostID
+        Public Property PostDate As Date? Implements IUserMedia.PostDate
+        Public Property SpecialFolder As String Implements IUserMedia.SpecialFolder
+        Public Property Attempts As Integer Implements IUserMedia.Attempts
+        Public Property [Object] As Object Implements IUserMedia.Object
     End Structure
+    Public Interface IUserMedia
+        Property ContentType As Integer
+        Property URL As String
+        Property URL_BASE As String
+        Property MD5 As String
+        Property File As String
+        Property DownloadState As Integer
+        Property PostID As String
+        Property PostDate As Date?
+        Property SpecialFolder As String
+        Property Attempts As Integer
+        Property [Object] As Object
+    End Interface
 End Namespace

SCrawler.PluginProvider/Objects/PropertyData.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.PluginProvider/Objects/PropertyValue.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler.sln

@@ -17,10 +17,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 EndProject
 Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.PluginProvider", "SCrawler.PluginProvider\SCrawler.PluginProvider.vbproj", "{D4650F6B-5A54-44B6-999B-6C675B7116B1}"
 EndProject
-Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.Plugin.LPSG", "SCrawler.Plugin.LPSG\SCrawler.Plugin.LPSG.vbproj", "{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}"
-EndProject
-Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "SCrawler.Plugin.XVIDEOS", "SCrawler.Plugin.XVIDEOS\SCrawler.Plugin.XVIDEOS.vbproj", "{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}"
-EndProject
 Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "PersonalUtilities.Notifications", "..\..\MyUtilities\PersonalUtilities.Notifications\PersonalUtilities.Notifications.vbproj", "{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}"
 EndProject
 Global
@@ -69,30 +65,6 @@ Global
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x64.Build.0 = Release|x64
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x86.ActiveCfg = Release|x86
 		{D4650F6B-5A54-44B6-999B-6C675B7116B1}.Release|x86.Build.0 = Release|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x64.ActiveCfg = Debug|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x64.Build.0 = Debug|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x86.ActiveCfg = Debug|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Debug|x86.Build.0 = Debug|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|Any CPU.ActiveCfg = Release|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|Any CPU.Build.0 = Release|Any CPU
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x64.ActiveCfg = Release|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x64.Build.0 = Release|x64
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x86.ActiveCfg = Release|x86
-		{22A130B2-DDF4-4FB5-BA38-E5DB4CF1B8A2}.Release|x86.Build.0 = Release|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x64.ActiveCfg = Debug|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x64.Build.0 = Debug|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x86.ActiveCfg = Debug|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Debug|x86.Build.0 = Debug|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|Any CPU.ActiveCfg = Release|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|Any CPU.Build.0 = Release|Any CPU
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x64.ActiveCfg = Release|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x64.Build.0 = Release|x64
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x86.ActiveCfg = Release|x86
-		{CCCF47F4-C97C-4193-AC4B-C56DF2F9AA8A}.Release|x86.Build.0 = Release|x86
 		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{FC532253-1AB3-4DEF-A28A-DFDD9A481EB2}.Debug|x64.ActiveCfg = Debug|x64

SCrawler/.editorconfig

@@ -123,4 +123,4 @@ insert_final_newline=false
 [*.vb]
 # Modifier preferences
 visual_basic_preferred_modifier_order = Partial,Default,Private,Protected,Public,Friend,NotOverridable,Overridable,MustOverride,Overloads,Overrides,MustInherit,NotInheritable,Static,Shared,Shadows,ReadOnly,WriteOnly,Dim,Const,WithEvents,Widening,Narrowing,Custom,Async:suggestion
-file_header_template = Copyright (C) 2022  Andy\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>
+file_header_template = Copyright (C) 2023  Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see <https://www.gnu.org/licenses/>

SCrawler/API/Base/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler/API/Base/DownDetector.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler/API/Base/M3U8Base.vb

@@ -0,0 +1,64 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.Base
+    Namespace M3U8Declarations
+        Friend Module M3U8Defaults
+            Friend ReadOnly TsFilesRegEx As RParams = RParams.DM(".+?\.ts[^\r\n]*", 0, RegexReturn.List)
+        End Module
+    End Namespace
+    Friend NotInheritable Class M3U8Base
+        Private Sub New()
+        End Sub
+        Friend Shared Function CreateUrl(ByVal Appender As String, ByVal File As String) As String
+            File = File.StringTrimStart("/")
+            If File.StartsWith("http") Then
+                Return File
+            Else
+                If File.StartsWith("hls/") And Appender.Contains("hls/") Then _
+                   Appender = LinkFormatterSecure(Appender.Replace("https://", String.Empty).Split("/").First)
+                Return $"{Appender.StringTrimEnd("/")}/{File}"
+            End If
+        End Function
+        Friend Shared Function Download(ByVal URLs As List(Of String), ByVal DestinationFile As SFile, Optional ByVal Responser As Responser = Nothing) As SFile
+            Dim CachePath As SFile = Nothing
+            Try
+                If URLs.ListExists Then
+                    Dim ConcatFile As SFile = DestinationFile
+                    If ConcatFile.Name.IsEmptyString Then ConcatFile.Name = "PlayListFile"
+                    ConcatFile.Extension = "mp4"
+                    CachePath = $"{DestinationFile.PathWithSeparator}_Cache\{SFile.GetDirectories($"{DestinationFile.PathWithSeparator}_Cache\",,, EDP.ReturnValue).ListIfNothing.Count + 1}\"
+                    If CachePath.Exists(SFO.Path) Then
+                        Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
+                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ReturnValue)
+                        Dim i%
+                        Dim eFiles As New List(Of SFile)
+                        Dim dFile As SFile = CachePath
+                        dFile.Extension = "ts"
+                        Using w As New DownloadObjects.WebClient2(Responser)
+                            For i = 0 To URLs.Count - 1
+                                dFile.Name = $"ConPart_{i}"
+                                w.DownloadFile(URLs(i), dFile)
+                                eFiles.Add(dFile)
+                            Next
+                        End Using
+                        DestinationFile = FFMPEG.ConcatenateFiles(eFiles, Settings.FfmpegFile, ConcatFile, p, EDP.ThrowException)
+                        eFiles.Clear()
+                        Return DestinationFile
+                    End If
+                End If
+                Return Nothing
+            Finally
+                CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
+            End Try
+        End Function
+    End Class
+End Namespace

SCrawler/API/Base/ProfileSaved.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -7,8 +7,8 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports PersonalUtilities.Forms.Toolbars
 Imports SCrawler.Plugin.Hosts
+Imports PersonalUtilities.Forms.Toolbars
 Imports PDownload = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
     Friend NotInheritable Class ProfileSaved
@@ -25,7 +25,7 @@ Namespace API.Base
                         HOST.DownloadStarted(PDownload.SavedPosts)
                         Dim u As New UserInfo With {.Plugin = HOST.Key, .Site = HOST.Name, .SpecialPath = HOST.SavedPostsPath}
                         Using user As IUserData = HOST.GetInstance(PDownload.SavedPosts, Nothing, False, False)
-                            If Not user Is Nothing AndAlso (Not user.Name.IsEmptyString Or Not HOST.IsMyClass) Then
+                            If Not user Is Nothing AndAlso Not user.Name.IsEmptyString Then
                                 u.Name = user.Name
                                 With DirectCast(user, UserDataBase)
                                     With .User : u.IsChannel = .IsChannel : u.UpdateUserFile() : End With
@@ -45,13 +45,14 @@ Namespace API.Base
                         Progress.InformationTemporary = $"Host [{HOST.Name}] is unavailable"
                     End If
                 Else
-                    Progress.InformationTemporary = $"Host [{HOST.Name}] is nor ready"
+                    Progress.InformationTemporary = $"Host [{HOST.Name}] is not ready"
                 End If
             Catch ex As Exception
                 Progress.InformationTemporary = $"{HOST.Name} downloading error"
                 ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Base.ProfileSaved.Download({HOST.Key})]")
             Finally
                 HOST.DownloadDone(PDownload.SavedPosts)
+                MainFrameObj.UpdateLogButton()
             End Try
         End Sub
     End Class

SCrawler/API/Base/SiteSettingsBase.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -7,25 +7,41 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Cookies
 Imports SCrawler.Plugin
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
-    Friend MustInherit Class SiteSettingsBase : Implements ISiteSettings
+    Friend MustInherit Class SiteSettingsBase : Implements ISiteSettings, IResponserContainer
         Friend ReadOnly Property Site As String Implements ISiteSettings.Site
         Friend Overridable ReadOnly Property Icon As Icon Implements ISiteSettings.Icon
         Friend Overridable ReadOnly Property Image As Image Implements ISiteSettings.Image
         Private Property Logger As ILogProvider = LogConnector Implements ISiteSettings.Logger
-        Friend Overridable ReadOnly Property Responser As Response
+        Friend Overridable ReadOnly Property Responser As Responser
+        Private Property IResponserContainer_Responser As Responser Implements IResponserContainer.Responser
+            Get
+                Return Responser
+            End Get
+            Set : End Set
+        End Property
         Friend MustOverride Function GetInstance(ByVal What As Download) As IPluginContentProvider Implements ISiteSettings.GetInstance
         Friend Sub New(ByVal SiteName As String)
             Site = SiteName
         End Sub
         Friend Sub New(ByVal SiteName As String, ByVal CookiesDomain As String)
             Site = SiteName
-            Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
+            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml")
             With Responser
-                If .File.Exists Then .LoadSettings() Else .CookiesDomain = CookiesDomain : .Cookies = New CookieKeeper(.CookiesDomain) : .SaveSettings()
+                If .File.Exists Then
+                    If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .LoadSettings()
+                Else
+                    .CookiesDomain = CookiesDomain
+                    .Cookies = New CookieKeeper(.CookiesDomain) With {.EncryptKey = SettingsCLS.CookieEncryptKey}
+                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .SaveSettings()
+                End If
+                If .CookiesDomain.IsEmptyString Then .CookiesDomain = CookiesDomain
             End With
         End Sub
 #Region "XML"
@@ -36,11 +52,16 @@ Namespace API.Base
         Friend Overridable Sub BeginInit() Implements ISiteSettings.BeginInit
         End Sub
         Friend Overridable Sub EndInit() Implements ISiteSettings.EndInit
+            EncryptCookies.ValidateCookiesEncrypt(Responser)
         End Sub
         Friend Overridable Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
         End Sub
         Friend Overridable Sub EndUpdate() Implements ISiteSettings.EndUpdate
         End Sub
+        Friend Overridable Sub BeginEdit() Implements ISiteSettings.BeginEdit
+        End Sub
+        Friend Overridable Sub EndEdit() Implements ISiteSettings.EndEdit
+        End Sub
 #End Region
 #Region "Before and After Download"
         Friend Overridable Sub DownloadStarted(ByVal What As Download) Implements ISiteSettings.DownloadStarted
@@ -55,14 +76,20 @@ Namespace API.Base
 #Region "User info"
         Protected UrlPatternUser As String = String.Empty
         Protected UrlPatternChannel As String = String.Empty
-        Friend Overridable Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String Implements ISiteSettings.GetUserUrl
+        Friend Overridable Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String Implements ISiteSettings.GetUserUrl
             If Channel Then
-                If Not UrlPatternChannel.IsEmptyString Then Return String.Format(UrlPatternChannel, UserName)
+                If Not UrlPatternChannel.IsEmptyString Then Return String.Format(UrlPatternChannel, User.Name)
             Else
-                If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, UserName)
+                If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, User.Name)
             End If
             Return String.Empty
         End Function
+        Private Function ISiteSettings_GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String Implements ISiteSettings.GetUserPostUrl
+            Return GetUserPostUrl(User, Media)
+        End Function
+        Friend Overridable Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return String.Empty
+        End Function
         Protected UserRegex As RParams = Nothing
         Friend Overridable Function IsMyUser(ByVal UserURL As String) As ExchangeOptions Implements ISiteSettings.IsMyUser
             Try
@@ -72,7 +99,7 @@ Namespace API.Base
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Base.SiteSettingsBase.IsMyUser]")
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]", New ExchangeOptions)
             End Try
         End Function
         Protected ImageVideoContains As String = String.Empty
@@ -83,17 +110,25 @@ Namespace API.Base
                 Return Nothing
             End If
         End Function
-        Friend Overridable Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia) Implements ISiteSettings.GetSpecialData
+        Friend Overridable Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable Implements ISiteSettings.GetSpecialData
             Return Nothing
         End Function
-        Friend Overridable Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
-            Return Nothing
+        Friend Shared Function GetSpecialDataFile(ByVal Path As String, ByVal AskForPath As Boolean, ByRef SpecFolderObj As String) As SFile
+            Dim f As SFile = Path.CSFileP
+            If f.Name.IsEmptyString Then f.Name = "OutputFile"
+#Disable Warning BC40000
+            If Path.CSFileP.IsEmptyString Or AskForPath Then f = SFile.SaveAs(f, "File destination",,,, EDP.ReturnValue) : SpecFolderObj = f.Path
+#Enable Warning
+            Return f
         End Function
 #End Region
 #Region "Ready, Available"
-        Friend Overridable Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
+        Friend Overridable Function BaseAuthExists() As Boolean
             Return True
         End Function
+        Friend Overridable Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
+            Return BaseAuthExists()
+        End Function
         Friend Overridable Function ReadyToDownload(ByVal What As Download) As Boolean Implements ISiteSettings.ReadyToDownload
             Return True
         End Function

SCrawler/API/Base/Structures.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,25 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Base
     Friend Module Structures
-        Friend Structure UserMedia : Implements IEquatable(Of UserMedia)
+        Friend Structure UserMedia : Implements IUserMedia, IEquatable(Of UserMedia), IEContainerProvider
+#Region "XML Names"
+            Friend Const Name_MediaNode As String = "MediaData"
+            Private Const Name_MediaType As String = "Type"
+            Private Const Name_MediaState As String = "State"
+            Private Const Name_MediaAttempts As String = "Attempts"
+            Private Const Name_MediaURL As String = "URL"
+            Private Const Name_MediaHash As String = "Hash"
+            Private Const Name_MediaFile As String = "File"
+            Private Const Name_MediaPostID As String = "ID"
+            Private Const Name_MediaPostDate As String = "Date"
+            Private Const Name_SpecialFolder As String = "SpecialFolder"
+#End Region
             Friend Enum Types As Integer
                 Undefined = 0
                 [Picture] = 1
@@ -18,7 +34,7 @@ Namespace API.Base
                 GIF = 50
                 m3u8 = 100
             End Enum
-            Friend Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : End Enum
+            Friend Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : Missing = 4 : End Enum
             Friend [Type] As Types
             Friend URL_BASE As String
             Friend URL As String
@@ -27,31 +43,152 @@ Namespace API.Base
             Friend Post As UserPost
             Friend PictureOption As String
             Friend State As States
+            Friend Attempts As Integer
             ''' <summary>
             ''' SomeFolder<br/>
             ''' SomeFolder\SomeFolder2
             ''' </summary>
             Friend SpecialFolder As String
-            Friend Sub New(ByVal _URL As String)
-                URL = _URL
-                URL_BASE = _URL
+            Friend [Object] As Object
+#Region "Interface Support"
+            Private Property IUserMedia_Type As Integer Implements IUserMedia.ContentType
+                Get
+                    Return Type
+                End Get
+                Set(ByVal Type As Integer)
+                    Me.Type = Type
+                End Set
+            End Property
+            Private Property IUserMedia_URL_BASE As String Implements IUserMedia.URL_BASE
+                Get
+                    Return URL_BASE
+                End Get
+                Set(ByVal URL_BASE As String)
+                    Me.URL_BASE = URL_BASE
+                End Set
+            End Property
+            Private Property IUserMedia_URL As String Implements IUserMedia.URL
+                Get
+                    Return URL
+                End Get
+                Set(ByVal URL As String)
+                    Me.URL = URL
+                End Set
+            End Property
+            Private Property IUserMedia_MD5 As String Implements IUserMedia.MD5
+                Get
+                    Return MD5
+                End Get
+                Set(ByVal MD5 As String)
+                    Me.MD5 = MD5
+                End Set
+            End Property
+            Private Property IUserMedia_File As String Implements IUserMedia.File
+                Get
+                    Return File
+                End Get
+                Set(ByVal File As String)
+                    Me.File = File
+                End Set
+            End Property
+            Private Property IUserMedia_State As Integer Implements IUserMedia.DownloadState
+                Get
+                    Return State
+                End Get
+                Set(ByVal State As Integer)
+                    Me.State = State
+                End Set
+            End Property
+            Private Property IUserMedia_PostID As String Implements IUserMedia.PostID
+                Get
+                    Return Post.ID
+                End Get
+                Set(ByVal PostID As String)
+                    Post = New UserPost(PostID, Post.Date)
+                End Set
+            End Property
+            Private Property IUserMedia_PostDate As Date? Implements IUserMedia.PostDate
+                Get
+                    Return Post.Date
+                End Get
+                Set(ByVal PostDate As Date?)
+                    Post = New UserPost(Post.ID, PostDate)
+                End Set
+            End Property
+            Private Property IUserMedia_SpecialFolder As String Implements IUserMedia.SpecialFolder
+                Get
+                    Return SpecialFolder
+                End Get
+                Set(ByVal SpecialFolder As String)
+                    Me.SpecialFolder = SpecialFolder
+                End Set
+            End Property
+            Private Property IUserMedia_Attempts As Integer Implements IUserMedia.Attempts
+                Get
+                    Return Attempts
+                End Get
+                Set(ByVal Attempts As Integer)
+                    Me.Attempts = Attempts
+                End Set
+            End Property
+            Private Property IUserMedia_Object As Object Implements IUserMedia.Object
+                Get
+                    Return Me.Object
+                End Get
+                Set(ByVal Obj As Object)
+                    Me.Object = Obj
+                End Set
+            End Property
+#End Region
+            Friend Sub New(ByVal URL As String)
+                Me.URL = URL
+                URL_BASE = URL
                 File = URL
                 Type = Types.Undefined
             End Sub
-            Friend Sub New(ByVal _URL As String, ByVal _Type As Types)
-                Me.New(_URL)
-                [Type] = _Type
+            Friend Sub New(ByVal URL As String, ByVal Type As Types)
+                Me.New(URL)
+                Me.Type = Type
             End Sub
-            Friend Sub New(ByVal m As Plugin.PluginUserMedia)
-                If Not IsNothing(m) Then
-                    [Type] = m.ContentType
-                    URL = m.URL
-                    MD5 = m.MD5
-                    File = m.File
-                    Post = New UserPost With {.ID = m.PostID, .[Date] = m.PostDate}
-                    State = m.DownloadState
-                    SpecialFolder = m.SpecialFolder
+            Friend Sub New(ByVal m As IUserMedia)
+                [Type] = m.ContentType
+                URL = m.URL
+                URL_BASE = m.URL_BASE
+                MD5 = m.MD5
+                File = m.File
+                Post = New UserPost With {.ID = m.PostID, .[Date] = m.PostDate}
+                State = m.DownloadState
+                SpecialFolder = m.SpecialFolder
+                Attempts = m.Attempts
+                Me.Object = m.Object
+            End Sub
+            Friend Sub New(ByVal e As EContainer, ByVal UserInstance As IUserData)
+                Type = e.Attribute(Name_MediaType).Value.FromXML(Of Integer)(CInt(Types.Undefined))
+                State = e.Attribute(Name_MediaState).Value.FromXML(Of Integer)(CInt(States.Downloaded))
+                Attempts = e.Attribute(Name_MediaAttempts).Value.FromXML(Of Integer)(0)
+                URL = e.Attribute(Name_MediaURL).Value
+                URL_BASE = e.Value
+                MD5 = e.Attribute(Name_MediaHash).Value
+                File = e.Attribute(Name_MediaFile).Value
+
+                Dim vp As Boolean? = Nothing
+                Dim upath$ = String.Empty
+                If Not UserInstance Is Nothing Then
+                    With DirectCast(UserInstance, UserDataBase)
+                        vp = .SeparateVideoFolder
+                        upath = .MyFile.CutPath.PathWithSeparator
+                    End With
                 End If
+
+                SpecialFolder = e.Attribute(Name_SpecialFolder).Value
+                If Not SpecialFolder.IsEmptyString Then upath &= $"{SpecialFolder}\"
+                If vp.HasValue AndAlso vp.Value Then upath &= $"Video\"
+                If Not upath.IsEmptyString Then File = $"{upath.CSFilePS}{File.File}"
+
+                Post = New UserPost With {
+                    .ID = e.Attribute(Name_MediaPostID).Value,
+                    .[Date] = AConvert(Of Date)(e.Attribute(Name_MediaPostDate).Value, ParsersDataDateProvider, Nothing)
+                }
             End Sub
             Public Shared Widening Operator CType(ByVal _URL As String) As UserMedia
                 Return New UserMedia(_URL)
@@ -59,43 +196,69 @@ Namespace API.Base
             Public Shared Widening Operator CType(ByVal m As UserMedia) As String
                 Return m.URL
             End Operator
+            Public Overrides Function GetHashCode() As Integer
+                If Not File.IsEmptyString Then
+                    Return File.GetHashCode
+                ElseIf Not URL_BASE.IsEmptyString Then
+                    Return URL_BASE.GetHashCode
+                ElseIf Not URL.IsEmptyString Then
+                    Return URL.GetHashCode
+                Else
+                    Return 0
+                End If
+            End Function
             Public Overrides Function ToString() As String
                 Return URL
             End Function
-            Friend Function PluginUserMedia() As Plugin.PluginUserMedia
-                Return New Plugin.PluginUserMedia With {
-                    .ContentType = Type,
-                    .DownloadState = State,
-                    .File = File,
-                    .MD5 = MD5,
-                    .URL = URL,
-                    .SpecialFolder = SpecialFolder,
-                    .PostID = Post.ID,
-                    .PostDate = Post.Date
-                }
-            End Function
             Friend Overloads Function Equals(ByVal Other As UserMedia) As Boolean Implements IEquatable(Of UserMedia).Equals
                 Return URL = Other.URL
             End Function
             Public Overrides Function Equals(ByVal Obj As Object) As Boolean
                 Return Equals(CType(Obj, UserMedia))
             End Function
+            Friend Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer
+                Return New EContainer(Name_MediaNode, URL_BASE, {New EAttribute(Name_MediaType, CInt(Type)),
+                                                                 New EAttribute(Name_MediaState, CInt(State)),
+                                                                 New EAttribute(Name_MediaAttempts, Attempts),
+                                                                 New EAttribute(Name_MediaURL, URL),
+                                                                 New EAttribute(Name_MediaHash, MD5),
+                                                                 New EAttribute(Name_MediaFile, File.File),
+                                                                 New EAttribute(Name_SpecialFolder, SpecialFolder),
+                                                                 New EAttribute(Name_MediaPostID, Post.ID),
+                                                                 New EAttribute(Name_MediaPostDate, AConvert(Of String)(Post.Date, ParsersDataDateProvider, String.Empty))
+                                                                }
+                                     )
+            End Function
         End Structure
         Friend Structure UserPost : Implements IEquatable(Of UserPost), IComparable(Of UserPost)
             ''' <summary>Post ID</summary>
             Friend ID As String
             Friend [Date] As Date?
-#Region "Channel compatible fields"
             Friend UserID As String
             Friend CachedFile As SFile
+#Region "Initializers"
+            Public Sub New(ByVal ID As String)
+                Me.ID = ID
+            End Sub
+            Public Sub New(ByVal [Date] As Date?)
+                Me.Date = [Date]
+            End Sub
+            Public Sub New(ByVal ID As String, ByVal [Date] As Date?)
+                Me.ID = ID
+                Me.Date = [Date]
+            End Sub
+            Public Shared Widening Operator CType(ByVal ID As String) As UserPost
+                Return New UserPost(ID)
+            End Operator
+            Public Shared Widening Operator CType(ByVal Post As UserPost) As String
+                Return Post.ID
+            End Operator
 #End Region
-            Friend Function GetImage(ByVal s As Size, ByVal e As ErrorsDescriber, ByVal NullArg As Image) As Image
-                If Not CachedFile.IsEmptyString Then
-                    Return If(PersonalUtilities.Tools.ImageRenderer.GetImage(SFile.GetBytes(CachedFile), s, e), NullArg.Clone)
-                Else
-                    Return NullArg.Clone
-                End If
+#Region "ToString"
+            Public Overrides Function ToString() As String
+                Return ID
             End Function
+#End Region
 #Region "IEquatable, IComparable Support"
             Friend Overloads Function Equals(ByVal Other As UserPost) As Boolean Implements IEquatable(Of UserPost).Equals
                 Return ID = Other.ID
@@ -113,7 +276,7 @@ Namespace API.Base
             End Function
 #End Region
         End Structure
-        Friend Structure Sizes : Implements IComparable(Of Sizes)
+        Friend Structure Sizes : Implements IRegExCreator, IComparable(Of Sizes)
             Friend Value As Integer
             Friend Data As String
             Friend ReadOnly HasError As Boolean
@@ -125,6 +288,16 @@ Namespace API.Base
                     HasError = True
                 End Try
             End Sub
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    Return New Sizes With {
+                        .Value = AConvert(Of Integer)(ParamsArray(0), 0),
+                        .Data = ParamsArray(1)
+                    }
+                Else
+                    Return New Sizes
+                End If
+            End Function
             Friend Function CompareTo(ByVal Other As Sizes) As Integer Implements IComparable(Of Sizes).CompareTo
                 Return Value.CompareTo(Other.Value) * -1
             End Function

SCrawler/API/Base/UserDataBase.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -10,7 +10,7 @@ Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Imports System.IO
 Imports System.Net
 Imports System.Threading
@@ -51,14 +51,12 @@ Namespace API.Base
 #Region "Collection buttons"
         Private _CollectionButtonsExists As Boolean = False
         Private _CollectionButtonsColorsSet As Boolean = False
-        Friend InternalCollectionIndex As Integer = -1
-        Friend WithEvents BTT_CONTEXT_DOWN As ToolStripMenuItem
+        Friend WithEvents BTT_CONTEXT_DOWN As ToolStripKeyMenuItem
         Friend WithEvents BTT_CONTEXT_EDIT As ToolStripMenuItem
         Friend WithEvents BTT_CONTEXT_DELETE As ToolStripMenuItem
         Friend WithEvents BTT_CONTEXT_OPEN_PATH As ToolStripMenuItem
         Friend WithEvents BTT_CONTEXT_OPEN_SITE As ToolStripMenuItem
-        Friend Sub CreateButtons(ByVal CollectionIndex As Integer)
-            InternalCollectionIndex = CollectionIndex
+        Friend Sub CreateButtons()
             Dim tn$ = $"[{Site}] - {Name}"
             Dim _tn$ = $"{Site}{Name}"
             Dim tnn As Func(Of String, String) = Function(Input) $"{Input}{_tn}"
@@ -70,11 +68,11 @@ Namespace API.Base
                     i = .Image
                 End If
             End With
-            BTT_CONTEXT_DOWN = New ToolStripMenuItem(tn, i) With {.Name = tnn("DOWN"), .Tag = CollectionIndex}
-            BTT_CONTEXT_EDIT = New ToolStripMenuItem(tn, i) With {.Name = tnn("EDIT"), .Tag = CollectionIndex}
-            BTT_CONTEXT_DELETE = New ToolStripMenuItem(tn, i) With {.Name = tnn("DELETE"), .Tag = CollectionIndex}
-            BTT_CONTEXT_OPEN_PATH = New ToolStripMenuItem(tn, i) With {.Name = tnn("PATH"), .Tag = CollectionIndex}
-            BTT_CONTEXT_OPEN_SITE = New ToolStripMenuItem(tn, i) With {.Name = tnn("SITE"), .Tag = CollectionIndex}
+            BTT_CONTEXT_DOWN = New ToolStripKeyMenuItem(tn, i) With {.Name = tnn("DOWN"), .Tag = Me}
+            BTT_CONTEXT_EDIT = New ToolStripMenuItem(tn, i) With {.Name = tnn("EDIT"), .Tag = Me}
+            BTT_CONTEXT_DELETE = New ToolStripMenuItem(tn, i) With {.Name = tnn("DELETE"), .Tag = Me}
+            BTT_CONTEXT_OPEN_PATH = New ToolStripMenuItem(tn, i) With {.Name = tnn("PATH"), .Tag = Me}
+            BTT_CONTEXT_OPEN_SITE = New ToolStripMenuItem(tn, i) With {.Name = tnn("SITE"), .Tag = Me}
             UpdateButtonsColor()
             _CollectionButtonsExists = True
             If _UserInformationLoaded Then _CollectionButtonsColorsSet = True
@@ -96,9 +94,16 @@ Namespace API.Base
         End Sub
 #End Region
 #Region "XML Declarations"
-        Private Const Name_Site As String = "Site"
-        Private Const Name_IsChannel As String = "IsChannel"
-        Private Const Name_UserName As String = "UserName"
+        Private Const Name_Site As String = UserInfo.Name_Site
+        Private Const Name_Plugin As String = UserInfo.Name_Plugin
+        Private Const Name_IsChannel As String = UserInfo.Name_IsChannel
+        Friend Const Name_UserName As String = "UserName"
+        Private Const Name_Model_User As String = UserInfo.Name_Model_User
+        Private Const Name_Model_Collection As String = UserInfo.Name_Model_Collection
+        Private Const Name_Merged As String = UserInfo.Name_Merged
+        Private Const Name_SpecialPath As String = UserInfo.Name_SpecialPath
+        Private Const Name_SpecialCollectionPath As String = UserInfo.Name_SpecialCollectionPath
+
         Private Const Name_UserExists As String = "UserExists"
         Private Const Name_UserSuspended As String = "UserSuspended"
         Private Const Name_FriendlyName As String = "FriendlyName"
@@ -110,8 +115,8 @@ Namespace API.Base
         Private Const Name_CreatedByChannel As String = "CreatedByChannel"
 
         Private Const Name_SeparateVideoFolder As String = "SeparateVideoFolder"
-        Private Const Name_CollectionName As String = "Collection"
-        Private Const Name_LabelsName As String = "Labels"
+        Private Const Name_CollectionName As String = UserInfo.Name_Collection
+        Friend Const Name_LabelsName As String = "Labels"
 
         Private Const Name_ReadyForDownload As String = "ReadyForDownload"
         Private Const Name_DownloadImages As String = "DownloadImages"
@@ -124,15 +129,7 @@ Namespace API.Base
         Private Const Name_ScriptUse As String = "ScriptUse"
         Private Const Name_ScriptData As String = "ScriptData"
 
-        Private Const Name_DataMerging As String = "DataMerging"
-#Region "Downloaded data"
-        Private Const Name_MediaType As String = "Type"
-        Private Const Name_MediaURL As String = "URL"
-        Private Const Name_MediaHash As String = "Hash"
-        Private Const Name_MediaFile As String = "File"
-        Private Const Name_MediaPostID As String = "ID"
-        Private Const Name_MediaPostDate As String = "Date"
-#End Region
+        <Obsolete("Use 'Name_Merged'", False)> Friend Const Name_DataMerging As String = "DataMerging"
 #End Region
 #Region "Declarations"
 #Region "Host, Site, Progress, Self"
@@ -166,6 +163,21 @@ Namespace API.Base
         End Property
         Friend Overridable Property ID As String = String.Empty Implements IContentProvider.ID, IPluginContentProvider.ID
         Friend Overridable Property FriendlyName As String = String.Empty Implements IContentProvider.FriendlyName
+        Friend ReadOnly Property UserModel As UsageModel Implements IUserData.UserModel
+            Get
+                Return User.UserModel
+            End Get
+        End Property
+        Friend Overridable ReadOnly Property CollectionModel As UsageModel Implements IUserData.CollectionModel
+            Get
+                Return User.CollectionModel
+            End Get
+        End Property
+        Friend Overridable ReadOnly Property IsVirtual As Boolean Implements IUserData.IsVirtual
+            Get
+                Return UserModel = UsageModel.Virtual
+            End Get
+        End Property
 #End Region
 #Region "Description"
         Friend Property UserDescription As String = String.Empty Implements IContentProvider.Description, IPluginContentProvider.UserDescription
@@ -241,7 +253,7 @@ Namespace API.Base
         Protected Function GetNullPicture(ByVal MaxHeigh As XML.Base.XMLValue(Of Integer)) As Bitmap
             Return New Bitmap(CInt(DivideWithZeroChecking(MaxHeigh.Value, 100) * 75), MaxHeigh.Value)
         End Function
-        Protected Function GetPicture(Of T)(Optional ByVal ReturnNullImageOnNothing As Boolean = True, Optional ByVal GetToast As Boolean = False) As T
+        Friend Function GetPicture(Of T)(Optional ByVal ReturnNullImageOnNothing As Boolean = True, Optional ByVal GetToast As Boolean = False) As T
             Dim rsfile As Boolean = GetType(T) Is GetType(SFile)
             Dim f As SFile = Nothing
             Dim p As UserImage = Nothing
@@ -318,7 +330,7 @@ BlockNullPicture:
         Friend Property SeparateVideoFolder As Boolean?
         Protected ReadOnly Property SeparateVideoFolderF As Boolean
             Get
-                Return (SeparateVideoFolder.HasValue AndAlso SeparateVideoFolder.Value) OrElse Settings.SeparateVideoFolder.Value
+                Return If(SeparateVideoFolder, Settings.SeparateVideoFolder.Value)
             End Get
         End Property
 #End Region
@@ -345,7 +357,7 @@ BlockNullPicture:
         Friend Overridable Sub ChangeCollectionName(ByVal NewName As String, ByVal UpdateSettings As Boolean)
             Dim u As UserInfo = User
             u.CollectionName = NewName
-            u.IncludedInCollection = Not NewName.IsEmptyString
+            u.UpdateUserFile()
             User = u
             If UpdateSettings Then Settings.UpdateUsersList(User)
         End Sub
@@ -358,10 +370,31 @@ BlockNullPicture:
         Friend Overridable Property ReadyForDownload As Boolean = True Implements IUserData.ReadyForDownload
         Friend Property DownloadImages As Boolean = True Implements IUserData.DownloadImages
         Friend Property DownloadVideos As Boolean = True Implements IUserData.DownloadVideos
+        Friend Property DownloadMissingOnly As Boolean = False Implements IUserData.DownloadMissingOnly
 #End Region
 #Region "Content"
         Protected ReadOnly _ContentList As List(Of UserMedia)
         Protected ReadOnly _ContentNew As List(Of UserMedia)
+        Friend ReadOnly Property LatestData As List(Of UserMedia)
+        Protected ReadOnly MissingFinder As Predicate(Of UserMedia) = Function(c) c.State = UStates.Missing
+        Friend ReadOnly Property ContentMissing As List(Of UserMedia)
+            Get
+                If _ContentList.Count > 0 Then
+                    Return _ContentList.Where(Function(c) MissingFinder(c)).ListIfNothing
+                Else
+                    Return New List(Of UserMedia)
+                End If
+            End Get
+        End Property
+        Friend Overridable ReadOnly Property ContentMissingExists As Boolean
+            Get
+                Return _ContentList.Exists(MissingFinder)
+            End Get
+        End Property
+        Friend Sub RemoveMedia(ByVal m As UserMedia, ByVal State As UStates?)
+            Dim i% = If(State.HasValue, _ContentList.FindIndex(Function(mm) mm.State = State.Value And mm.Equals(m)), _ContentList.IndexOf(m))
+            If i >= 0 Then _ContentList.RemoveAt(i)
+        End Sub
         Protected ReadOnly _TempMediaList As List(Of UserMedia)
         Protected ReadOnly _TempPostsList As List(Of String)
         Friend Function GetLastImageAddress() As SFile
@@ -443,7 +476,13 @@ BlockNullPicture:
             End Get
         End Property
         Friend Overridable Function GetUserInformation() As String
-            Dim OutStr$ = $"User: {Name}"
+            Dim OutStr$ = $"User: {Name} (site: {Site}"
+            If IncludedInCollection Then
+                OutStr &= $"; collection: {CollectionName}"
+                If CollectionModel = UsageModel.Default And UserModel = UsageModel.Virtual Then OutStr &= "; virtual"
+            End If
+            OutStr &= ")"
+            OutStr.StringAppendLine($"Labels: {Labels.ListToString}")
             OutStr.StringAppendLine($"Path: {MyFile.CutPath.Path}")
             OutStr.StringAppendLine($"Total downloaded ({DownloadedTotal(True).NumToString(ANumbers.Formats.Number, 3)}):")
             OutStr.StringAppendLine($"Pictures: {DownloadedPictures(True).NumToString(ANumbers.Formats.Number, 3)}")
@@ -480,9 +519,9 @@ BlockNullPicture:
         Private Property IPluginContentProvider_Thrower As IThrower Implements IPluginContentProvider.Thrower
         Private Property IPluginContentProvider_LogProvider As ILogProvider Implements IPluginContentProvider.LogProvider
         Friend Property ExternalPlugin As IPluginContentProvider
-        Private Property IPluginContentProvider_ExistingContentList As List(Of PluginUserMedia) Implements IPluginContentProvider.ExistingContentList
+        Private Property IPluginContentProvider_ExistingContentList As List(Of IUserMedia) Implements IPluginContentProvider.ExistingContentList
         Private Property IPluginContentProvider_TempPostsList As List(Of String) Implements IPluginContentProvider.TempPostsList
-        Private Property IPluginContentProvider_TempMediaList As List(Of PluginUserMedia) Implements IPluginContentProvider.TempMediaList
+        Private Property IPluginContentProvider_TempMediaList As List(Of IUserMedia) Implements IPluginContentProvider.TempMediaList
         Private Property IPluginContentProvider_SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
         Private Property IPluginContentProvider_DataPath As String Implements IPluginContentProvider.DataPath
         Private Sub IPluginContentProvider_XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String))) Implements IPluginContentProvider.XmlFieldsSet
@@ -527,23 +566,31 @@ BlockNullPicture:
         End Function
         Friend Overridable ReadOnly Property FitToAddParams As Boolean Implements IUserData.FitToAddParams
             Get
-                If Settings.LastUpdatedDate.HasValue AndAlso LastUpdated.HasValue AndAlso
-                   LastUpdated.Value.Date > Settings.LastUpdatedDate.Value.Date Then Return False
-                If Not Settings.Labels.ExcludedIgnore AndAlso Settings.Labels.Excluded.ValuesList.ListContains(Labels) Then Return False
-                If Settings.SelectedSites.Count = 0 OrElse Settings.SelectedSites.Contains(Site) Then
-                    Select Case Settings.ShowingMode.Value
-                        Case ShowingModes.Regular : Return Not Temporary And Not Favorite
-                        Case ShowingModes.Temporary : Return Temporary
-                        Case ShowingModes.Favorite : Return Favorite
-                        Case ShowingModes.Deleted : Return Not UserExists
-                        Case ShowingModes.Suspended : Return UserSuspended
-                        Case ShowingModes.Labels : Return Settings.Labels.Current.ValuesList.ListContains(Labels)
-                        Case ShowingModes.NoLabels : Return Labels.Count = 0
-                        Case Else : Return True
-                    End Select
-                Else
-                    Return False
-                End If
+                With Settings
+                    If LastUpdated.HasValue And Not .ViewDateMode.Value = ShowingDates.Off Then
+                        Dim f As Date = If(.ViewDateFrom.HasValue, .ViewDateFrom.Value.Date, Date.MinValue.Date)
+                        Dim t As Date = If(.ViewDateTo.HasValue, .ViewDateTo.Value.Date, Date.MaxValue.Date)
+                        Select Case DirectCast(.ViewDateMode.Value, ShowingDates)
+                            Case ShowingDates.In : If Not LastUpdated.Value.ValueBetween(f, t) Then Return False
+                            Case ShowingDates.Not : If LastUpdated.Value.ValueBetween(f, t) Then Return False
+                        End Select
+                    End If
+                    If Not .Labels.ExcludedIgnore AndAlso .Labels.Excluded.ValuesList.ListContains(Labels) Then Return False
+                    If .SelectedSites.Count = 0 OrElse .SelectedSites.Contains(Site) Then
+                        Select Case .ShowingMode.Value
+                            Case ShowingModes.Regular : Return Not Temporary And Not Favorite
+                            Case ShowingModes.Temporary : Return Temporary
+                            Case ShowingModes.Favorite : Return Favorite
+                            Case ShowingModes.Deleted : Return Not UserExists
+                            Case ShowingModes.Suspended : Return UserSuspended
+                            Case ShowingModes.Labels : Return Settings.Labels.Current.ValuesList.ListContains(Labels)
+                            Case ShowingModes.NoLabels : Return Labels.Count = 0
+                            Case Else : Return True
+                        End Select
+                    Else
+                        Return False
+                    End If
+                End With
             End Get
         End Property
         Friend Function GetLVIGroup(ByVal Destination As ListView) As ListViewGroup Implements IUserData.GetLVIGroup
@@ -568,11 +615,12 @@ BlockNullPicture:
         Friend Sub New(Optional ByVal InvokeImageHandler As Boolean = True)
             _ContentList = New List(Of UserMedia)
             _ContentNew = New List(Of UserMedia)
+            LatestData = New List(Of UserMedia)
             _TempMediaList = New List(Of UserMedia)
             _TempPostsList = New List(Of String)
             Labels = New List(Of String)
             UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
-            If InvokeImageHandler Then ImageHandler(Me)
+            If InvokeImageHandler Then MainFrameObj.ImageHandler(Me)
         End Sub
         Friend Sub SetEnvironment(ByRef h As SettingsHost, ByVal u As UserInfo, ByVal _LoadUserInformation As Boolean,
                                   Optional ByVal AttachUserInfo As Boolean = True) Implements IUserData.SetEnvironment
@@ -583,13 +631,29 @@ BlockNullPicture:
             End If
         End Sub
         ''' <exception cref="ArgumentOutOfRangeException"></exception>
-        Friend Overloads Shared Function GetInstance(ByVal u As UserInfo, Optional ByVal _LoadUserInformation As Boolean = True) As IUserData
+        Friend Shared Function GetInstance(ByVal u As UserInfo, Optional ByVal _LoadUserInformation As Boolean = True) As IUserData
             If Not u.Plugin.IsEmptyString Then
                 Return Settings(u.Plugin).GetInstance(u.DownloadOption, u, _LoadUserInformation)
             Else
                 Throw New ArgumentOutOfRangeException("Plugin", $"Plugin [{u.Plugin}] information does not recognized by loader")
             End If
         End Function
+        Friend Shared Function GetPostUrl(ByVal u As IUserData, ByVal PostData As UserMedia) As String
+            Dim uName$ = String.Empty
+            Try
+                If Not u Is Nothing AndAlso Not u.IsCollection Then
+                    With DirectCast(u, UserDataBase)
+                        If Not .User.Plugin.IsEmptyString Then
+                            uName = .User.Name
+                            Return Settings(.User.Plugin).GetUserPostUrl(.Self, PostData)
+                        End If
+                    End With
+                End If
+                Return String.Empty
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"GetPostUrl({uName}, {PostData.Post.ID})", String.Empty)
+            End Try
+        End Function
 #End Region
 #Region "Information & Content data files loader and saver"
 #Region "User information"
@@ -618,7 +682,14 @@ BlockNullPicture:
                         LastUpdated = AConvert(Of Date)(x.Value(Name_LastUpdated), ADateTime.Formats.BaseDateTime, Nothing)
                         ScriptUse = x.Value(Name_ScriptUse).FromXML(Of Boolean)(False)
                         ScriptData = x.Value(Name_ScriptData)
-                        DataMerging = x.Value(Name_DataMerging).FromXML(Of Boolean)(False)
+                        'TODELETE: UserDataBase remove old 'merge' constant
+#Disable Warning BC40000
+                        If x.Contains(Name_DataMerging) Then
+                            DataMerging = x.Value(Name_DataMerging).FromXML(Of Boolean)(False)
+                        Else
+                            DataMerging = x.Value(Name_Merged).FromXML(Of Boolean)(False)
+                        End If
+#Enable Warning
                         ChangeCollectionName(x.Value(Name_CollectionName), False)
                         Labels.ListAddList(x.Value(Name_LabelsName).StringToList(Of String, List(Of String))("|", EDP.ReturnValue), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
                         LoadUserInformation_OptionalFields(x, True)
@@ -636,7 +707,13 @@ BlockNullPicture:
                 MyFile.Exists(SFO.Path)
                 Using x As New XmlFile With {.Name = "User"}
                     x.Add(Name_Site, Site)
+                    x.Add(Name_Plugin, HOST.Key)
                     x.Add(Name_UserName, User.Name)
+                    x.Add(Name_IsChannel, IsChannel.BoolToInteger)
+                    x.Add(Name_Model_User, CInt(UserModel))
+                    x.Add(Name_Model_Collection, CInt(CollectionModel))
+                    x.Add(Name_SpecialPath, User.SpecialPath)
+                    x.Add(Name_SpecialCollectionPath, User.SpecialCollectionPath)
                     x.Add(Name_UserExists, UserExists.BoolToInteger)
                     x.Add(Name_UserSuspended, UserSuspended.BoolToInteger)
                     x.Add(Name_UserID, ID)
@@ -661,7 +738,7 @@ BlockNullPicture:
                     x.Add(Name_ScriptData, ScriptData)
                     x.Add(Name_CollectionName, CollectionName)
                     x.Add(Name_LabelsName, Labels.ListToString("|", EDP.ReturnValue))
-                    x.Add(Name_DataMerging, DataMerging.BoolToInteger)
+                    x.Add(Name_Merged, DataMerging.BoolToInteger)
 
                     LoadUserInformation_OptionalFields(x, False)
 
@@ -676,29 +753,14 @@ BlockNullPicture:
         Protected MustOverride Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
 #End Region
 #Region "User data"
-        Friend Overridable Overloads Sub LoadContentInformation()
+        Friend Overridable Overloads Sub LoadContentInformation(Optional ByVal Force As Boolean = False)
             Try
                 UpdateDataFiles()
-                If Not MyFileData.Exists Then Exit Sub
+                If Not MyFileData.Exists Or (_DataLoaded And Not Force) Then Exit Sub
                 Using x As New XmlFile(MyFileData, Protector.Modes.All, False) With {.XmlReadOnly = True, .AllowSameNames = True}
                     x.LoadData()
                     If x.Count > 0 Then
-                        Dim fs$ = MyFile.CutPath.PathWithSeparator
-                        Dim gfn As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, String.Empty,
-                                                                                If(Input.Contains("\"), Input.CSFile.File, Input))
-                        For Each v As EContainer In x
-                            _ContentList.Add(New UserMedia With {
-                                                .Type = AConvert(Of Integer)(v.Attribute(Name_MediaType).Value, 0),
-                                                .URL = v.Attribute(Name_MediaURL).Value,
-                                                .URL_BASE = v.Value,
-                                                .MD5 = v.Attribute(Name_MediaHash).Value,
-                                                .File = fs & gfn.Invoke(v.Attribute(Name_MediaFile).Value),
-                                                .Post = New UserPost With {
-                                                    .ID = v.Attribute(Name_MediaPostID).Value,
-                                                    .[Date] = AConvert(Of Date)(v.Attribute(Name_MediaPostDate).Value, ParsersDataDateProvider, Nothing)
-                                                }
-                                             })
-                        Next
+                        For Each v As EContainer In x : _ContentList.Add(New UserMedia(v, Me)) : Next
                     End If
                     _DataLoaded = True
                 End Using
@@ -712,19 +774,7 @@ BlockNullPicture:
                 If MyFileData.IsEmptyString Then Exit Sub
                 MyFileData.Exists(SFO.Path)
                 Using x As New XmlFile With {.AllowSameNames = True, .Name = "Data"}
-                    If _ContentList.Count > 0 Then
-                        For Each i As UserMedia In _ContentList
-                            x.Add(New EContainer("MediaData", i.URL_BASE, {New EAttribute(Name_MediaType, CInt(i.Type)),
-                                                                           New EAttribute(Name_MediaURL, i.URL),
-                                                                           New EAttribute(Name_MediaHash, i.MD5),
-                                                                           New EAttribute(Name_MediaFile, i.File.File),
-                                                                           New EAttribute(Name_MediaPostID, i.Post.ID),
-                                                                           New EAttribute(Name_MediaPostDate, AConvert(Of String)(i.Post.Date, ParsersDataDateProvider, String.Empty))
-                                                                          }
-                                                )
-                                 )
-                        Next
-                    End If
+                    If _ContentList.Count > 0 Then x.AddRange(_ContentList)
                     x.Save(MyFileData)
                 End Using
             Catch ex As Exception
@@ -736,7 +786,7 @@ BlockNullPicture:
 #Region "Open site, folder"
         Friend Overridable Sub OpenSite(Optional ByVal e As ErrorsDescriber = Nothing) Implements IContentProvider.OpenSite
             Try
-                Dim URL$ = HOST.Source.GetUserUrl(Name, IsChannel)
+                Dim URL$ = HOST.Source.GetUserUrl(Me, IsChannel)
                 If Not URL.IsEmptyString Then Process.Start(URL)
             Catch ex As Exception
                 If Not e.Exists Then e = New ErrorsDescriber(EDP.ShowAllMsg)
@@ -747,10 +797,55 @@ BlockNullPicture:
             GlobalOpenPath(MyFile.CutPath)
         End Sub
 #End Region
-#Region "Download functions and options"
+#Region "Download limits"
+        Protected Enum DateResult : [Continue] : [Skip] : [Exit] : End Enum
         Friend Overridable Property DownloadTopCount As Integer? = Nothing Implements IUserData.DownloadTopCount, IPluginContentProvider.PostsNumberLimit
-        Friend Overridable Property DownloadToDate As Date? = Nothing Implements IUserData.DownloadToDate, IPluginContentProvider.PostsDateLimit
-        Protected Responser As Response
+        Friend Overridable Property IncludeInTheFeed As Boolean = True
+        Private _DownloadDateFrom As Date? = Nothing
+        Private _DownloadDateFromF As Date
+        Friend Overridable Property DownloadDateFrom As Date? Implements IUserData.DownloadDateFrom, IPluginContentProvider.DownloadDateFrom
+            Get
+                Return _DownloadDateFrom
+            End Get
+            Set(ByVal d As Date?)
+                _DownloadDateFrom = d
+                If _DownloadDateFrom.HasValue Then _DownloadDateFromF = _DownloadDateFrom.Value.Date Else _DownloadDateFromF = Date.MinValue.Date
+            End Set
+        End Property
+        Private _DownloadDateTo As Date? = Nothing
+        Private _DownloadDateToF As Date
+        Friend Overridable Property DownloadDateTo As Date? Implements IUserData.DownloadDateTo, IPluginContentProvider.DownloadDateTo
+            Get
+                Return _DownloadDateTo
+            End Get
+            Set(ByVal d As Date?)
+                _DownloadDateTo = d
+                If _DownloadDateTo.HasValue Then _DownloadDateToF = _DownloadDateTo.Value Else _DownloadDateToF = Date.MaxValue.Date
+            End Set
+        End Property
+        Protected Function CheckDatesLimit(ByVal DateObj As Object, ByVal DateProvider As IFormatProvider) As DateResult
+            Try
+                If (DownloadDateFrom.HasValue Or DownloadDateTo.HasValue) AndAlso ACheck(DateObj) Then
+                    Dim td As Date? = AConvert(Of Date)(DateObj, DateProvider, Nothing)
+                    If td.HasValue Then
+                        If td.Value.ValueBetween(_DownloadDateFromF, _DownloadDateToF) Then
+                            Return DateResult.Continue
+                        ElseIf td.Value > _DownloadDateToF Then
+                            Return DateResult.Skip
+                        Else
+                            Return DateResult.Exit
+                        End If
+                    End If
+                End If
+                Return DateResult.Continue
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[UserDataBase.CheckDatesLimit({If(TypeOf DateObj Is String, CStr(DateObj), "?")})]", DateResult.Continue)
+            End Try
+        End Function
+#End Region
+#Region "Download functions and options"
+        Protected Responser As Responser
+        Protected UseResponserClient As Boolean = False
         Friend Overridable Sub DownloadData(ByVal Token As CancellationToken) Implements IContentProvider.DownloadData
             Dim Canceled As Boolean = False
             _ExternalCompatibilityToken = Token
@@ -758,43 +853,59 @@ BlockNullPicture:
                 UpdateDataFiles()
                 UserDescriptionReset()
                 If Not Responser Is Nothing Then Responser.Dispose()
-                Responser = New Response
+                Responser = New Responser
                 If Not HOST.Responser Is Nothing Then Responser.Copy(HOST.Responser)
+                'TODO: UserDataBase remove [Responser.DecodersError]
+                Responser.DecodersError = New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue) With {
+                    .DeclaredMessage = New MMessage($"SymbolsConverter error: [{ToStringForLog()}]", ToStringForLog())}
 
                 Dim UpPic As Boolean = Settings.ViewModeIsPicture AndAlso GetPicture(Of Image)(False) Is Nothing
                 Dim sEnvir() As Boolean = {UserExists, UserSuspended}
                 Dim EnvirChanged As Func(Of Boolean) = Function() Not sEnvir(0) = UserExists Or Not sEnvir(1) = UserSuspended
+                Dim _downContent As Func(Of UserMedia, Boolean) = Function(c) c.State = UStates.Downloaded
                 UserExists = True
                 UserSuspended = False
                 DownloadedPictures(False) = 0
                 DownloadedVideos(False) = 0
                 _TempMediaList.Clear()
                 _TempPostsList.Clear()
+                LatestData.Clear()
                 Dim __SaveData As Boolean = Not CreatedByChannel Or Not Settings.FromChannelDownloadTopUse
 
-                If Not _DataLoaded Then LoadContentInformation()
+                LoadContentInformation()
 
                 If MyFilePosts.Exists Then _TempPostsList.ListAddList(File.ReadAllLines(MyFilePosts))
                 If _ContentList.Count > 0 Then _TempPostsList.ListAddList(_ContentList.Select(Function(u) u.Post.ID), LNC)
 
-                ThrowAny(Token)
-                DownloadDataF(Token)
-                ThrowAny(Token)
+                If Not DownloadMissingOnly Then
+                    ThrowAny(Token)
+                    DownloadDataF(Token)
+                    ThrowAny(Token)
+                    If Settings.ReparseMissingInTheRoutine Then ReparseMissing(Token) : ThrowAny(Token)
+                Else
+                    ReparseMissing(Token)
+                End If
 
                 If _TempMediaList.Count > 0 Then
                     If Not DownloadImages Then _TempMediaList.RemoveAll(Function(m) m.Type = UTypes.GIF Or m.Type = UTypes.Picture)
                     If Not DownloadVideos Then _TempMediaList.RemoveAll(Function(m) m.Type = UTypes.Video Or
                                                                                     m.Type = UTypes.VideoPre Or m.Type = UTypes.m3u8)
+                    If DownloadMissingOnly Then _TempMediaList.RemoveAll(Function(m) Not m.State = UStates.Missing)
                 End If
 
                 ReparseVideo(Token)
                 ThrowAny(Token)
-                If _TempPostsList.Count > 0 And __SaveData Then TextSaver.SaveTextToFile(_TempPostsList.ListToString(Environment.NewLine), MyFilePosts, True,, EDP.None)
+                If _TempPostsList.Count > 0 And Not DownloadMissingOnly And __SaveData Then _
+                   TextSaver.SaveTextToFile(_TempPostsList.ListToString(Environment.NewLine), MyFilePosts, True,, EDP.None)
                 _ContentNew.ListAddList(_TempMediaList, LAP.ClearBeforeAdd)
                 DownloadContent(Token)
                 ThrowIfDisposed()
-                _ContentList.ListAddList(_ContentNew.Where(Function(c) c.State = UStates.Downloaded), LNC)
-                If DownloadedTotal(False) > 0 Or EnvirChanged.Invoke Then
+
+                If IncludeInTheFeed Then LatestData.ListAddList(_ContentNew.Where(_downContent), LNC)
+                Dim mcb& = If(ContentMissingExists, _ContentList.LongCount(Function(c) MissingFinder(c)), 0)
+                _ContentList.ListAddList(_ContentNew.Where(Function(c) _downContent(c) Or MissingFinder(c)), LNC)
+                Dim mca& = If(ContentMissingExists, _ContentList.LongCount(Function(c) MissingFinder(c)), 0)
+                If DownloadedTotal(False) > 0 Or EnvirChanged.Invoke Or Not mcb = mca Then
                     If __SaveData Then
                         LastUpdated = Now
                         RunScript()
@@ -815,7 +926,7 @@ BlockNullPicture:
                 ThrowIfDisposed()
                 If UpPic Or EnvirChanged.Invoke Then OnUserUpdated()
             Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                MyMainLOG = $"{Site} - {Name}: downloading canceled"
+                MyMainLOG = $"{ToStringForLog()}: downloading canceled"
                 Canceled = True
             Catch dex As ObjectDisposedException When Disposed
                 Canceled = True
@@ -827,20 +938,11 @@ BlockNullPicture:
                 If Not Canceled Then _DataParsed = True
                 _ContentNew.Clear()
                 DownloadTopCount = Nothing
-                DownloadToDate = Nothing
+                DownloadDateFrom = Nothing
+                DownloadDateTo = Nothing
+                DownloadMissingOnly = False
             End Try
         End Sub
-        Protected Function CheckDatesLimit(ByVal DateString As String, ByVal DateProvider As IFormatProvider) As Boolean
-            Try
-                If DownloadToDate.HasValue And Not DateString.IsEmptyString Then
-                    Dim td As Date? = AConvert(Of Date)(DateString, DateProvider, Nothing)
-                    If td.HasValue Then Return td.Value < DownloadToDate.Value
-                End If
-                Return True
-            Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[UserDataBase.CheckDatesLimit({DateString})]", True)
-            End Try
-        End Function
         Protected Sub UpdateDataFiles()
             If Not User.File.IsEmptyString Then
                 MyFileData = User.File
@@ -853,8 +955,26 @@ BlockNullPicture:
             End If
         End Sub
         Protected MustOverride Sub DownloadDataF(ByVal Token As CancellationToken)
-        Protected MustOverride Sub ReparseVideo(ByVal Token As CancellationToken)
+        Protected Overridable Sub ReparseVideo(ByVal Token As CancellationToken)
+        End Sub
+        ''' <summary>
+        ''' Missing posts must be collected from [<see cref="_ContentList"/>].<br/>
+        ''' Reparsed post must be added to [<see cref="_TempMediaList"/>].<br/>
+        ''' At the end of the function, reparsed posts must be removed from [<see cref="_ContentList"/>].
+        ''' </summary>
+        Protected Overridable Sub ReparseMissing(ByVal Token As CancellationToken)
+        End Sub
         Protected MustOverride Sub DownloadContent(ByVal Token As CancellationToken)
+        Private NotInheritable Class OptionalWebClient : Inherits DownloadObjects.WebClient2
+            Friend Sub New(ByRef Source As UserDataBase)
+                UseResponserClient = Source.UseResponserClient
+                If UseResponserClient Then
+                    RC = Source.Responser
+                Else
+                    WC = New WebClient
+                End If
+            End Sub
+        End Class
         Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
             Try
                 Dim i%
@@ -864,14 +984,16 @@ BlockNullPicture:
                     _ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
                     If _ContentNew.Count > 0 Then
                         MyFile.Exists(SFO.Path)
+                        Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
                         Dim MyDir$ = MyFile.CutPath.PathNoSeparator
                         Dim vsf As Boolean = SeparateVideoFolderF
                         Dim __isVideo As Boolean
                         Dim f As SFile
                         Dim v As UserMedia
-                        Using w As New WebClient
-                            If vsf Then SFileShares.SFileExists($"{MyDir}\Video\", SFO.Path)
-                            Progress.TotalCount += _ContentNew.Count
+
+                        Using w As New OptionalWebClient(Me)
+                            If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
+                            Progress.Maximum += _ContentNew.Count
                             For i = 0 To _ContentNew.Count - 1
                                 ThrowAny(Token)
                                 v = _ContentNew(i)
@@ -886,7 +1008,7 @@ BlockNullPicture:
 
                                 If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
 
-                                If Not v.File.IsEmptyString And Not v.URL_BASE.IsEmptyString Then
+                                If Not v.File.IsEmptyString And Not v.URL.IsEmptyString Then
                                     Try
                                         __isVideo = v.Type = UTypes.Video Or f.Extension = "mp4"
 
@@ -908,7 +1030,13 @@ BlockNullPicture:
                                             f.Path = $"{f.PathWithSeparator}Video"
                                             If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
                                         End If
-                                        w.DownloadFile(v.URL_BASE, f.ToString)
+
+                                        If v.Type = UTypes.m3u8 And UseInternalM3U8Function Then
+                                            f = DownloadM3U8(v.URL, v, f)
+                                            If f.IsEmptyString Then Throw New Exception("M3U8 download failed")
+                                        Else
+                                            w.DownloadFile(v.URL, f.ToString)
+                                        End If
 
                                         If __isVideo Then
                                             v.Type = UTypes.Video
@@ -922,7 +1050,9 @@ BlockNullPicture:
                                         v.State = UStates.Downloaded
                                         dCount += 1
                                     Catch wex As Exception
-                                        ErrorDownloading(f, v.URL_BASE)
+                                        v.Attempts += 1
+                                        v.State = UStates.Missing
+                                        If MissingErrorsAdd Then ErrorDownloading(f, v.URL)
                                     End Try
                                 Else
                                     v.State = UStates.Skipped
@@ -947,15 +1077,31 @@ BlockNullPicture:
                 HasError = True
             End Try
         End Sub
+        Protected UseInternalM3U8Function As Boolean = False
+        Protected Overridable Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+            Return Nothing
+        End Function
+        Protected Const EXCEPTION_OPERATION_CANCELED As Integer = -1
         ''' <param name="RDE">Request DownloadingException</param>
-        Protected Sub ProcessException(ByVal ex As Exception, ByVal Token As CancellationToken, ByVal Message As String, Optional ByVal RDE As Boolean = True)
+        ''' <returns>0 - exit</returns>
+        Protected Function ProcessException(ByVal ex As Exception, ByVal Token As CancellationToken, ByVal Message As String,
+                                            Optional ByVal RDE As Boolean = True, Optional ByVal EObj As Object = Nothing,
+                                            Optional ByVal ThrowEx As Boolean = True) As Integer
             If Not ((TypeOf ex Is OperationCanceledException And Token.IsCancellationRequested) Or
                     (TypeOf ex Is ObjectDisposedException And Disposed)) Then
-                If RDE AndAlso DownloadingException(ex, Message, True) = 0 Then LogError(ex, Message) : HasError = True
+                If RDE Then
+                    Dim v% = DownloadingException(ex, Message, True, EObj)
+                    If v = 0 Then LogError(ex, Message) : HasError = True
+                    Return v
+                End If
+            Else
+                'URGENT: UserDataBase.ProcessException [Throw ex]
+                If ThrowEx Then Throw ex Else Return EXCEPTION_OPERATION_CANCELED
             End If
-        End Sub
+            Return 0
+        End Function
         ''' <summary>0 - Execute LogError and set HasError</summary>
-        Protected MustOverride Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False) As Integer
+        Protected MustOverride Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False, Optional ByVal EObj As Object = Nothing) As Integer
         Protected Function ChangeFileNameByProvider(ByVal f As SFile, ByVal m As UserMedia) As SFile
             Dim ff As SFile = Nothing
             Try
@@ -996,14 +1142,14 @@ BlockNullPicture:
             End Try
         End Sub
 #End Region
-#Region "Delete, Move, Merge"
-        Friend Overridable Function Delete() As Integer Implements IUserData.Delete
+#Region "Delete, Move, Merge, Copy"
+        Friend Overridable Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer Implements IUserData.Delete
             Dim f As SFile = SFile.GetPath(MyFile.CutPath.Path)
             If f.Exists(SFO.Path, False) AndAlso (User.Merged OrElse f.Delete(SFO.Path, Settings.DeleteMode)) Then
-                ImageHandler(Me, False)
+                If Not IncludedInCollection Then MainFrameObj.ImageHandler(Me, False)
                 Settings.UsersList.Remove(User)
                 Settings.UpdateUsersList()
-                Settings.Users.Remove(Me)
+                If Not IncludedInCollection Then Settings.Users.Remove(Me)
                 Downloader.UserRemove(Me)
                 Dispose(True)
                 Return 1
@@ -1011,44 +1157,51 @@ BlockNullPicture:
                 Return 0
             End If
         End Function
-        Friend Overridable Function MoveFiles(ByVal __CollectionName As String) As Boolean Implements IUserData.MoveFiles
+        Friend Overridable Function MoveFiles(ByVal __CollectionName As String, ByVal __SpecialCollectionPath As SFile) As Boolean Implements IUserData.MoveFiles
             Dim UserBefore As UserInfo = User
             Dim Removed As Boolean = True
             Dim _TurnBack As Boolean = False
             Try
                 Dim f As SFile
+                Dim v As Boolean = IsVirtual
                 If IncludedInCollection Then
                     Settings.Users.Add(Me)
                     Removed = False
                     User.CollectionName = String.Empty
-                    User.IncludedInCollection = False
+                    User.SpecialCollectionPath = String.Empty
+                    User.UserModel = UsageModel.Default
+                    User.CollectionModel = UsageModel.Default
                 Else
                     Settings.Users.Remove(Me)
                     Removed = True
                     User.CollectionName = __CollectionName
-                    User.IncludedInCollection = True
-                    User.SpecialPath = Nothing
+                    User.SpecialCollectionPath = __SpecialCollectionPath
+                    If Not IsVirtual Then User.SpecialPath = Nothing
                 End If
                 _TurnBack = True
                 User.UpdateUserFile()
-                f = User.File.CutPath(, EDP.ThrowException)
-                If f.Exists(SFO.Path, False) Then
-                    If If(SFile.GetFiles(f,, SearchOption.AllDirectories), New List(Of SFile)).Count > 0 AndAlso
-                       MsgBoxE({$"Destination directory [{f.Path}] already exists and contains files!" & vbCr &
-                                "By continuing, this directory and all files will be deleted",
-                                "Destination directory is not empty!"}, MsgBoxStyle.Exclamation,,, {"Delete", "Cancel"}) = 1 Then
-                        MsgBoxE("Operation canceled", MsgBoxStyle.Exclamation)
-                        User = UserBefore
-                        If Removed Then Settings.Users.Add(Me) Else Settings.Users.Remove(Me)
-                        _TurnBack = False
-                        Return False
+
+                If Not v Then
+                    f = User.File.CutPath(, EDP.ThrowException)
+                    If f.Exists(SFO.Path, False) Then
+                        If If(SFile.GetFiles(f,, SearchOption.AllDirectories), New List(Of SFile)).Count > 0 AndAlso
+                           MsgBoxE({$"Destination directory [{f.Path}] already exists and contains files!" & vbCr &
+                                    "By continuing, this directory and all files will be deleted",
+                                    "Destination directory is not empty!"}, MsgBoxStyle.Exclamation,,, {"Delete", "Cancel"}) = 1 Then
+                            MsgBoxE("Operation canceled", MsgBoxStyle.Exclamation)
+                            User = UserBefore
+                            If Removed Then Settings.Users.Add(Me) Else Settings.Users.Remove(Me)
+                            _TurnBack = False
+                            Return False
+                        End If
+                        f.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException)
                     End If
-                    f.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException)
+                    f.CutPath.Exists(SFO.Path)
+                    Directory.Move(UserBefore.File.CutPath(, EDP.ThrowException).Path, f.Path)
+                    If Not ScriptData.IsEmptyString AndAlso ScriptData.Contains(UserBefore.File.PathNoSeparator) Then _
+                       ScriptData = ScriptData.Replace(UserBefore.File.PathNoSeparator, MyFile.PathNoSeparator)
                 End If
-                f.CutPath.Exists(SFO.Path)
-                Directory.Move(UserBefore.File.CutPath(, EDP.ThrowException).Path, f.Path)
-                If Not ScriptData.IsEmptyString AndAlso ScriptData.Contains(UserBefore.File.PathNoSeparator) Then _
-                   ScriptData = ScriptData.Replace(UserBefore.File.PathNoSeparator, MyFile.PathNoSeparator)
+
                 Settings.UsersList.Remove(UserBefore)
                 Settings.UpdateUsersList(User)
                 UpdateUserInformation()
@@ -1120,10 +1273,52 @@ BlockNullPicture:
             End If
             Return f
         End Function
+        Private Class FilesCopyingException : Inherits ErrorsDescriberException
+            Friend Sub New(ByVal User As IUserData, ByVal Msg As String, ByVal Path As SFile)
+                SendInLogOnlyMessage = True
+                If User.IncludedInCollection Then _MainMessage = $"[{User.CollectionName}] - "
+                _MainMessage &= $"[{User.Site}] - [{User.Name}]. {Msg}: {Path.Path}."
+            End Sub
+        End Class
+        Friend Overridable Function CopyFiles(ByVal DestinationPath As SFile, Optional ByVal e As ErrorsDescriber = Nothing) As Boolean Implements IUserData.CopyFiles
+            Dim fSource As SFile = Nothing
+            Dim fDest As SFile = Nothing
+            Try
+                Dim pOffset%
+                If IncludedInCollection Then
+                    If DataMerging Then pOffset = 1 Else pOffset = 2
+                Else
+                    pOffset = 1
+                End If
+                fSource = User.File.CutPath(pOffset).Path.CSFileP
+
+                Dim OptPath$ = String.Empty
+                If IncludedInCollection Then
+                    OptPath = $"Collections\{CollectionName}" 'Copying a collection based on the first file
+                Else
+                    OptPath = $"{Site}\{Name}"
+                End If
+                fDest = $"{DestinationPath.PathWithSeparator}{OptPath}".CSFileP
+                If fDest.Exists(SFO.Path, False) AndAlso MsgBoxE({$"The following path already exists:{vbCr}{fDest.Path}" & vbCr &
+                                                                  "Do you want to copy files here?", "Copying files"}, vbExclamation + vbYesNo) = vbNo Then _
+                   Throw New FilesCopyingException(Me, "The following path already exists", fDest)
+
+                If DestinationPath.Exists(SFO.Path, True) Then
+                    My.Computer.FileSystem.CopyDirectory(fSource, fDest, FileIO.UIOption.OnlyErrorDialogs, FileIO.UICancelOption.ThrowException)
+                Else
+                    Throw New FilesCopyingException(Me, "Cannot create the following path", fDest)
+                End If
+                Return True
+            Catch cex As OperationCanceledException
+                Return ErrorsDescriber.Execute(e, New FilesCopyingException(Me, "Copy canceled", fDest),, False)
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(e, ex,, False)
+            End Try
+        End Function
 #End Region
 #Region "Errors functions"
         Protected Sub LogError(ByVal ex As Exception, ByVal Message As String)
-            ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{IIf(IncludedInCollection, $"{CollectionName}-", String.Empty)}{Site} - {Name}: {Message}")
+            ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{ToStringForLog()}: {Message}")
         End Sub
         Protected Sub ErrorDownloading(ByVal f As SFile, ByVal URL As String)
             If Not f.Exists Then MyMainLOG = $"Error downloading from [{URL}] to [{f}]"
@@ -1138,11 +1333,14 @@ BlockNullPicture:
         End Sub
         ''' <exception cref="OperationCanceledException"></exception>
         ''' <exception cref="ObjectDisposedException"></exception>
-        Friend Overloads Sub ThrowAny(ByVal Token As CancellationToken)
+        Friend Overridable Overloads Sub ThrowAny(ByVal Token As CancellationToken)
             Token.ThrowIfCancellationRequested()
             ThrowIfDisposed()
         End Sub
 #End Region
+        Protected Function ToStringForLog() As String
+            Return $"{IIf(IncludedInCollection, $"[{CollectionName}] - ", String.Empty)}[{Site}] - {Name}"
+        End Function
         Public Overrides Function ToString() As String
             If IsCollection Then
                 Return CollectionName
@@ -1150,9 +1348,19 @@ BlockNullPicture:
                 Return IIf(FriendlyName.IsEmptyString, Name, FriendlyName)
             End If
         End Function
+        Public Overrides Function GetHashCode() As Integer
+            Dim hcStr$
+            If Not CollectionName.IsEmptyString Then
+                hcStr = CollectionName
+            Else
+                hcStr = IIf(FriendlyName.IsEmptyString, Name, FriendlyName)
+            End If
+            If hcStr.IsEmptyString Then hcStr = LVIKey
+            Return hcStr.GetHashCode
+        End Function
 #Region "Buttons actions"
-        Private Sub BTT_CONTEXT_DOWN_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_DOWN.Click
-            Downloader.Add(Me)
+        Private Sub BTT_CONTEXT_DOWN_KeyClick(sender As Object, e As MyKeyEventArgs) Handles BTT_CONTEXT_DOWN.KeyClick
+            Downloader.Add(Me, e.IncludeInTheFeed)
         End Sub
         Private Sub BTT_CONTEXT_EDIT_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_EDIT.Click
             Using f As New Editors.UserCreatorForm(Me)
@@ -1205,6 +1413,7 @@ BlockNullPicture:
                 If disposing Then
                     _ContentList.Clear()
                     _ContentNew.Clear()
+                    LatestData.Clear()
                     _TempMediaList.Clear()
                     _TempPostsList.Clear()
                     If Not Responser Is Nothing Then Responser.Dispose()
@@ -1228,6 +1437,7 @@ BlockNullPicture:
         End Sub
 #End Region
     End Class
+#Region "Base interfaces"
     Friend Interface IContentProvider
         ReadOnly Property Site As String
         Property Name As String
@@ -1250,6 +1460,9 @@ BlockNullPicture:
         ReadOnly Property IsCollection As Boolean
         Property CollectionName As String
         ReadOnly Property IncludedInCollection As Boolean
+        ReadOnly Property UserModel As UsageModel
+        ReadOnly Property CollectionModel As UsageModel
+        ReadOnly Property IsVirtual As Boolean
         ReadOnly Property Labels As List(Of String)
 #End Region
         ReadOnly Property IsChannel As Boolean
@@ -1268,6 +1481,7 @@ BlockNullPicture:
         ReadOnly Property Key As String
         Property DownloadImages As Boolean
         Property DownloadVideos As Boolean
+        Property DownloadMissingOnly As Boolean
         Property ScriptUse As Boolean
         Property ScriptData As String
         Function GetLVI(ByVal Destination As ListView) As ListViewItem
@@ -1280,12 +1494,14 @@ BlockNullPicture:
         ''' 2 - Collection removed<br/>
         ''' 3 - Collection split
         ''' </summary>
-        Function Delete() As Integer
-        Function MoveFiles(ByVal CollectionName As String) As Boolean
+        Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer
+        Function MoveFiles(ByVal CollectionName As String, ByVal SpecialCollectionPath As SFile) As Boolean
+        Function CopyFiles(ByVal DestinationPath As SFile, Optional ByVal e As ErrorsDescriber = Nothing) As Boolean
         Sub OpenFolder()
         ReadOnly Property Self As IUserData
         Property DownloadTopCount As Integer?
-        Property DownloadToDate As Date?
+        Property DownloadDateFrom As Date?
+        Property DownloadDateTo As Date?
         Sub SetEnvironment(ByRef h As SettingsHost, ByVal u As UserInfo, ByVal _LoadUserInformation As Boolean,
                            Optional ByVal AttachUserInfo As Boolean = True)
         ReadOnly Property Disposed As Boolean
@@ -1302,4 +1518,5 @@ BlockNullPicture:
         Property SkipExistsUsers As Boolean
         Property SaveToCache As Boolean
     End Interface
+#End Region
 End Namespace

SCrawler/API/BaseObjects/DomainEnvir.vb

@@ -0,0 +1,86 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Forms
+Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
+Namespace API.BaseObjects
+    Friend Interface IDomainContainer
+        ReadOnly Property Icon As Icon
+        ReadOnly Property Site As String
+        ReadOnly Property Domains As List(Of String)
+        ReadOnly Property DomainsTemp As List(Of String)
+        ReadOnly Property DomainsDefault As String
+        ReadOnly Property DomainsSettingProp As Plugin.PropertyValue
+        Property DomainsChanged As Boolean
+        Property Initialized As Boolean
+        Property DomainsUpdateInProgress As Boolean
+        Property DomainsUpdatedBySite As Boolean
+        Sub UpdateDomains()
+    End Interface
+    Friend NotInheritable Class DomainContainer
+        Private Sub New()
+        End Sub
+        Friend Shared Sub EndInit(ByVal s As IDomainContainer)
+            If ACheck(s.DomainsSettingProp.Value) Then s.Domains.ListAddList(CStr(s.DomainsSettingProp.Value).Split("|"), LAP.NotContainsOnly)
+        End Sub
+        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer)
+            UpdateDomains(s, Nothing, True)
+        End Sub
+        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer, ByVal NewDomains As IEnumerable(Of String), ByVal Internal As Boolean)
+            With s
+                If Not .Initialized Or (.DomainsUpdatedBySite And Not Internal) Then Exit Sub
+                If Not .DomainsUpdateInProgress Then
+                    .DomainsUpdateInProgress = True
+                    .Domains.ListAddList(.DomainsDefault.Split("|"), LAP.NotContainsOnly)
+                    .Domains.ListAddList(NewDomains, LAP.NotContainsOnly)
+                    .DomainsSettingProp.Value = .Domains.ListToString("|")
+                    If Not Internal Then .DomainsUpdatedBySite = True
+                    .DomainsUpdateInProgress = False
+                End If
+            End With
+        End Sub
+        Friend Shared Sub Update(ByVal s As IDomainContainer)
+            With s
+                If .DomainsChanged Then
+                    .Domains.Clear()
+                    .Domains.ListAddList(.DomainsTemp, LAP.NotContainsOnly)
+                    .UpdateDomains()
+                End If
+            End With
+        End Sub
+        Friend Shared Sub EndEdit(ByVal s As IDomainContainer)
+            s.DomainsTemp.ListAddList(s.Domains, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
+            s.DomainsChanged = False
+        End Sub
+        Friend Shared Sub OpenSettingsForm(ByVal s As IDomainContainer)
+            Dim __add As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e) e.ValueNew = InputBoxE($"Enter a new domain using the pattern [{s.Site}.com]:", "New domain").IfNullOrEmptyE(Nothing)
+            Dim __delete As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e)
+                                                                           Dim n$ = AConvert(Of String)(e.ValueCurrent, AModes.Var, String.Empty)
+                                                                           e.Result = MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
+                                                                                              "Removing domains"}, vbYesNo) = vbYes
+                                                                       End Sub
+            Using f As New SimpleListForm(Of String)(If(s.DomainsChanged, s.DomainsTemp, s.Domains), Settings.Design) With {
+                .Buttons = {ADB.Add, ADB.Delete},
+                .Mode = SimpleListFormModes.Remaining,
+                .FormText = s.Site,
+                .Icon = s.Icon,
+                .LocationOnly = True,
+                .Size = New Size(400, 330),
+                .DesignXMLNode = s.Site
+            }
+                AddHandler f.AddClick, __add
+                AddHandler f.DeleteClick, __delete
+                f.ShowDialog()
+                If f.DialogResult = DialogResult.OK Then
+                    s.DomainsChanged = True
+                    s.DomainsTemp.ListAddList(f.DataResult, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
+                End If
+            End Using
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Gfycat/Envir.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.RegularExpressions
 Imports System.Net
 Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Gfycat
     Friend NotInheritable Class Envir
         Private Sub New()

SCrawler/API/Imgur/Envir.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,16 +6,16 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.XML
-Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WebDocuments.JSON
 Imports System.Net
 Imports SCrawler.API.Base
 Imports SCrawler.API.Imgur.Declarations
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Namespace API.Imgur
     Namespace Declarations
         Friend Module Imgur_Declarations
-            Friend ReadOnly PostRegex As RParams = RParams.DMS("/([\w\d]+?)(|\.[\w]{0,4})\Z", 1)
+            Friend ReadOnly PostRegex As RParams = RParams.DMS("/([^/]+?)(|#.*?|\.[\w]{0,4})(|\?.*?)\Z", 1)
         End Module
     End Namespace
     Friend NotInheritable Class Envir
@@ -67,19 +67,21 @@ Namespace API.Imgur
                 Return DownloadingException(ex, $"[API.Imgur.Envir.GetImage({URL})]", String.Empty, e)
             End Try
         End Function
-        Friend Shared Function GetVideoInfo(ByVal URL As String) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString AndAlso URL.ToLower.Contains("imgur") AndAlso Not Settings.ImgurClientID.IsEmptyString Then
-                    Dim img$ = GetImage(URL, EDP.ReturnValue)
-                    If Not img.IsEmptyString Then
-                        Return {New UserMedia(img)}
+                    Dim imgList As List(Of String) = GetGallery(URL, EDP.ReturnValue)
+                    If imgList.ListExists Then
+                        Return imgList.Select(Function(u) New UserMedia(u))
                     Else
-                        Return GetGallery(URL, EDP.ReturnValue).ListIfNothing.Select(Function(u) New UserMedia(u))
+                        Dim img$ = GetImage(URL, EDP.ReturnValue)
+                        If Not img.IsEmptyString Then Return {New UserMedia(img)}
                     End If
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog + EDP.ReturnValue, ex, "Imgur standalone downloader: fetch media error")
+                If Not e.Exists Then e = EDP.LogMessageValue
+                Return ErrorsDescriber.Execute(e, ex, "Imgur standalone downloader: fetch media error")
             End Try
         End Function
         Private Shared Function DownloadingException(ByVal ex As Exception, ByVal Message As String,

SCrawler/API/Instagram/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -7,10 +7,49 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Clients.EventArguments
+Imports PersonalUtilities.Tools.Web.Cookies
 Namespace API.Instagram
     Friend Module Declarations
         Friend Const InstagramSite As String = "Instagram"
+        Friend Const InstagramSiteKey As String = "AndyProgram_Instagram"
         Friend ReadOnly FilesPattern As RParams = RParams.DMS(".+?([^/\?]+?\.[\w\d]{3,4})(?=(\?|\Z))", 1, EDP.ReturnValue)
         Friend ReadOnly Property DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
+        Friend Sub UpdateResponser(ByVal Source As IResponse, ByRef Destination As Responser)
+            Const r_wwwClaimName$ = "x-ig-set-www-claim"
+            Const r_tokenName$ = "csrftoken"
+            If Not Source Is Nothing Then
+                Dim isInternal As Boolean = TypeOf Source Is WebDataResponse
+                Dim wwwClaimName$, tokenName$
+                If isInternal Then
+                    wwwClaimName = r_wwwClaimName
+                    tokenName = r_tokenName
+                Else
+                    wwwClaimName = SiteSettings.Header_IG_WWW_CLAIM
+                    tokenName = SiteSettings.Header_CSRF_TOKEN
+                End If
+                Dim wwwClaim$ = String.Empty
+                Dim token$ = String.Empty
+                With Source
+                    If isInternal Then
+                        If .HeadersExists Then wwwClaim = .Headers.Value(wwwClaimName)
+                        If .CookiesExists Then token = If(.Cookies.FirstOrDefault(Function(c) c.Name = tokenName)?.Value, String.Empty)
+                    Else
+                        If .HeadersExists Then
+                            wwwClaim = .Headers.Value(wwwClaimName)
+                            token = .Headers.Value(tokenName)
+                        End If
+                    End If
+                End With
+
+                If Not wwwClaim.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_IG_WWW_CLAIM, wwwClaim)
+                If Not token.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_CSRF_TOKEN, token)
+                If Not isInternal Then
+                    Destination.Cookies.Update(Source.Cookies, CookieKeeper.UpdateModes.ReplaceByNameAll, False, EDP.SendInLog)
+                    Destination.SaveSettings()
+                End If
+            End If
+        End Sub
     End Module
 End Namespace

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -9,10 +9,12 @@
 Imports SCrawler.Plugin
 Namespace API.Instagram
     Friend Class EditorExchangeOptions
+        Friend Property GetTimeline As Boolean
         Friend Property GetStories As Boolean
         Friend Property GetTagged As Boolean
         Friend Sub New(ByVal h As ISiteSettings)
             With DirectCast(h, SiteSettings)
+                GetTimeline = CBool(.GetTimeline.Value)
                 GetStories = CBool(.GetStories.Value)
                 GetTagged = CBool(.GetTagged.Value)
             End With

SCrawler/API/Instagram/OptionsForm.Designer.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -26,6 +26,7 @@ Namespace API.Instagram
             Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
             Me.CH_GET_STORIES = New System.Windows.Forms.CheckBox()
             Me.CH_GET_TAGGED = New System.Windows.Forms.CheckBox()
+            Me.CH_GET_TIMELINE = New System.Windows.Forms.CheckBox()
             CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
             TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
             CONTAINER_MAIN.ContentPanel.SuspendLayout()
@@ -39,13 +40,13 @@ Namespace API.Instagram
             'CONTAINER_MAIN.ContentPanel
             '
             CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(260, 53)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(260, 79)
             CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
             CONTAINER_MAIN.LeftToolStripPanelVisible = False
             CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
             CONTAINER_MAIN.Name = "CONTAINER_MAIN"
             CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(260, 78)
+            CONTAINER_MAIN.Size = New System.Drawing.Size(260, 104)
             CONTAINER_MAIN.TabIndex = 0
             CONTAINER_MAIN.TopToolStripPanelVisible = False
             '
@@ -54,26 +55,28 @@ Namespace API.Instagram
             TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
             TP_MAIN.ColumnCount = 1
             TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Controls.Add(Me.CH_GET_STORIES, 0, 0)
-            TP_MAIN.Controls.Add(Me.CH_GET_TAGGED, 0, 1)
+            TP_MAIN.Controls.Add(Me.CH_GET_STORIES, 0, 1)
+            TP_MAIN.Controls.Add(Me.CH_GET_TAGGED, 0, 2)
+            TP_MAIN.Controls.Add(Me.CH_GET_TIMELINE, 0, 0)
             TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
             TP_MAIN.Location = New System.Drawing.Point(0, 0)
             TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 3
+            TP_MAIN.RowCount = 4
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(260, 53)
+            TP_MAIN.Size = New System.Drawing.Size(260, 79)
             TP_MAIN.TabIndex = 0
             '
             'CH_GET_STORIES
             '
             Me.CH_GET_STORIES.AutoSize = True
             Me.CH_GET_STORIES.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_GET_STORIES.Location = New System.Drawing.Point(4, 4)
+            Me.CH_GET_STORIES.Location = New System.Drawing.Point(4, 30)
             Me.CH_GET_STORIES.Name = "CH_GET_STORIES"
             Me.CH_GET_STORIES.Size = New System.Drawing.Size(252, 19)
-            Me.CH_GET_STORIES.TabIndex = 0
+            Me.CH_GET_STORIES.TabIndex = 1
             Me.CH_GET_STORIES.Text = "Get stories"
             Me.CH_GET_STORIES.UseVisualStyleBackColor = True
             '
@@ -81,27 +84,38 @@ Namespace API.Instagram
             '
             Me.CH_GET_TAGGED.AutoSize = True
             Me.CH_GET_TAGGED.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_GET_TAGGED.Location = New System.Drawing.Point(4, 30)
+            Me.CH_GET_TAGGED.Location = New System.Drawing.Point(4, 56)
             Me.CH_GET_TAGGED.Name = "CH_GET_TAGGED"
             Me.CH_GET_TAGGED.Size = New System.Drawing.Size(252, 19)
-            Me.CH_GET_TAGGED.TabIndex = 1
+            Me.CH_GET_TAGGED.TabIndex = 2
             Me.CH_GET_TAGGED.Text = "Get tagged data"
             Me.CH_GET_TAGGED.UseVisualStyleBackColor = True
             '
+            'CH_GET_TIMELINE
+            '
+            Me.CH_GET_TIMELINE.AutoSize = True
+            Me.CH_GET_TIMELINE.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_GET_TIMELINE.Location = New System.Drawing.Point(4, 4)
+            Me.CH_GET_TIMELINE.Name = "CH_GET_TIMELINE"
+            Me.CH_GET_TIMELINE.Size = New System.Drawing.Size(252, 19)
+            Me.CH_GET_TIMELINE.TabIndex = 0
+            Me.CH_GET_TIMELINE.Text = "Get Timeline"
+            Me.CH_GET_TIMELINE.UseVisualStyleBackColor = True
+            '
             'OptionsForm
             '
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
             Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(260, 78)
+            Me.ClientSize = New System.Drawing.Size(260, 104)
             Me.Controls.Add(CONTAINER_MAIN)
             Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
             Me.KeyPreview = True
             Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(276, 117)
+            Me.MaximumSize = New System.Drawing.Size(276, 143)
             Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(276, 117)
+            Me.MinimumSize = New System.Drawing.Size(276, 143)
             Me.Name = "OptionsForm"
-            Me.ShowIcon = False
             Me.ShowInTaskbar = False
             Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
             Me.Text = "Options"
@@ -116,5 +130,6 @@ Namespace API.Instagram
 
         Private WithEvents CH_GET_STORIES As CheckBox
         Private WithEvents CH_GET_TAGGED As CheckBox
+        Private WithEvents CH_GET_TIMELINE As CheckBox
     End Class
 End Namespace

SCrawler/API/Instagram/OptionsForm.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -21,6 +21,7 @@ Namespace API.Instagram
                 .MyViewInitialize(True)
                 .AddOkCancelToolbar()
                 With MyExchangeOptions
+                    CH_GET_TIMELINE.Checked = .GetTimeline
                     CH_GET_STORIES.Checked = .GetStories
                     CH_GET_TAGGED.Checked = .GetTagged
                 End With
@@ -29,6 +30,7 @@ Namespace API.Instagram
         End Sub
         Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
             With MyExchangeOptions
+                .GetTimeline = CH_GET_TIMELINE.Checked
                 .GetStories = CH_GET_STORIES.Checked
                 .GetTagged = CH_GET_TAGGED.Checked
             End With

SCrawler/API/Instagram/SiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -10,24 +10,25 @@ Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Forms
-Imports PersonalUtilities.Tools.WEB
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Cookies
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Instagram
-    <Manifest("AndyProgram_Instagram"), UseClassAsIs, SeparatedTasks(1), SavedPosts, SpecialForm(False)>
+    <Manifest(InstagramSiteKey), SeparatedTasks(1), SavedPosts, SpecialForm(False)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
 #Region "Declarations"
 #Region "Images"
         Friend Overrides ReadOnly Property Icon As Icon
             Get
-                Return My.Resources.InstagramIcon
+                Return My.Resources.SiteResources.InstagramIcon_32
             End Get
         End Property
         Friend Overrides ReadOnly Property Image As Image
             Get
-                Return My.Resources.InstagramPic76
+                Return My.Resources.SiteResources.InstagramPic_76
             End Get
         End Property
 #End Region
@@ -78,28 +79,23 @@ Namespace API.Instagram
         End Class
 #End Region
 #Region "Authorization properties"
-        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash", IsAuth:=True), PXML("InstaHash"), ControlNumber(0)>
-        Friend ReadOnly Property Hash As PropertyValue
-        <PropertyOption(ControlText:="Hash 2", ControlToolTip:="Instagram session hash for saved posts", IsAuth:=True), PXML("InstaHashSavedPosts"), ControlNumber(1)>
-        Friend ReadOnly Property HashSavedPosts As PropertyValue
-        <PropertyOption(ControlText:="x-csrftoken", ControlToolTip:="Instagram token for tagged data", IsAuth:=True), ControlNumber(2)>
+        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash for tagged posts", IsAuth:=True), PXML("InstaHash"), ControlNumber(0)>
+        Friend ReadOnly Property HashTagged As PropertyValue
+        <PropertyOption(ControlText:="x-csrftoken", IsAuth:=True, AllowNull:=False), ControlNumber(2)>
         Friend ReadOnly Property CSRF_TOKEN As PropertyValue
-        <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True), ControlNumber(3)>
+        <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True, AllowNull:=False), ControlNumber(3)>
         Friend Property IG_APP_ID As PropertyValue
-        <PropertyOption(ControlText:="x-ig-www-claim", IsAuth:=True), ControlNumber(4)>
+        <PropertyOption(ControlText:="x-ig-www-claim", IsAuth:=True, AllowNull:=False), ControlNumber(4)>
         Friend Property IG_WWW_CLAIM As PropertyValue
-        <PropertyOption(ControlText:="Saved posts user", IsAuth:=True), PXML("SavedPostsUserName"), ControlNumber(5)>
-        Friend ReadOnly Property SavedPostsUserName As PropertyValue
-        Friend ReadOnly Property BaseAuthExists As Boolean
-            Get
-                Return Responser.Cookies.Count > 0 And ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value) And ACheck(CSRF_TOKEN.Value)
-            End Get
-        End Property
+        Friend Overrides Function BaseAuthExists() As Boolean
+            Return Responser.CookiesExists And ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value) And ACheck(CSRF_TOKEN.Value)
+        End Function
         Private Const Header_IG_APP_ID As String = "x-ig-app-id"
-        Private Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
-        Private Const Header_CSRF_TOKEN As String = "x-csrftoken"
+        Friend Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
+        Friend Const Header_CSRF_TOKEN As String = "x-csrftoken"
+        Private _FieldsChangerSuspended As Boolean = False
         Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
-            If Not PropName.IsEmptyString Then
+            If Not _FieldsChangerSuspended And Not PropName.IsEmptyString Then
                 Dim f$ = String.Empty
                 Select Case PropName
                     Case NameOf(IG_APP_ID) : f = Header_IG_APP_ID
@@ -107,7 +103,7 @@ Namespace API.Instagram
                     Case NameOf(CSRF_TOKEN) : f = Header_CSRF_TOKEN
                 End Select
                 If Not f.IsEmptyString Then
-                    If Responser.Headers.Count > 0 AndAlso Responser.Headers.ContainsKey(f) Then Responser.Headers.Remove(f)
+                    Responser.Headers.Remove(f)
                     If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
                     Responser.SaveSettings()
                 End If
@@ -115,34 +111,45 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "Download properties"
-        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(6)>
+        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(20)>
         Friend ReadOnly Property RequestsWaitTimer As PropertyValue
         <Provider(NameOf(RequestsWaitTimer), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerProvider As IFormatProvider
-        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(7)>
+        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(21)>
         Friend ReadOnly Property RequestsWaitTimerTaskCount As PropertyValue
         <Provider(NameOf(RequestsWaitTimerTaskCount), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerTaskCountProvider As IFormatProvider
-        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(8)>
+        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(22)>
         Friend ReadOnly Property SleepTimerOnPostsLimit As PropertyValue
         <Provider(NameOf(SleepTimerOnPostsLimit), FieldsChecker:=True)>
         Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
-        <PropertyOption(ControlText:="Get stories"), PXML, ControlNumber(9)>
+        <PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users"), PXML, ControlNumber(23)>
+        Friend ReadOnly Property GetTimeline As PropertyValue
+        <PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users"), PXML, ControlNumber(24)>
         Friend ReadOnly Property GetStories As PropertyValue
-        <PropertyOption(ControlText:="Get tagged photos"), PXML, ControlNumber(10)>
+        <PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users"), PXML, ControlNumber(25)>
         Friend ReadOnly Property GetTagged As PropertyValue
         <PropertyOption(ControlText:="Tagged notify limit",
                         ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
-                        "-1 to disable"), PXML, ControlNumber(11)>
+                        "-1 to disable"), PXML, ControlNumber(26)>
         Friend ReadOnly Property TaggedNotifyLimit As PropertyValue
         <Provider(NameOf(TaggedNotifyLimit), FieldsChecker:=True)>
         Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
 #End Region
+#Region "Download ready"
+        <PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline"), PXML, ControlNumber(10)>
+        Friend ReadOnly Property DownloadTimeline As PropertyValue
+        <PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories"), PXML, ControlNumber(11)>
+        Friend ReadOnly Property DownloadStories As PropertyValue
+        <PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts"), PXML, ControlNumber(12)>
+        Friend ReadOnly Property DownloadTagged As PropertyValue
+#End Region
 #Region "429 bypass"
         Private ReadOnly Property DownloadingErrorDate As XMLValue(Of Date)
         Friend Property LastApplyingValue As Integer? = Nothing
         Friend ReadOnly Property ReadyForDownload As Boolean
             Get
+                If SkipUntilNextSession Then Return False
                 With DownloadingErrorDate
                     If .ValueF.Exists Then
                         Return .ValueF.Value.AddMinutes(If(LastApplyingValue, 10)) < Now
@@ -186,43 +193,37 @@ Namespace API.Instagram
             End With
         End Sub
 #End Region
-        Friend Overrides ReadOnly Property Responser As Response
-        Private Initialized As Boolean = False
 #End Region
 #Region "Initializer"
         Friend Sub New(ByRef _XML As XmlFile, ByVal GlobalPath As SFile)
-            MyBase.New(InstagramSite)
-            Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
+            MyBase.New(InstagramSite, "instagram.com")
 
             Dim app_id$ = String.Empty
             Dim www_claim$ = String.Empty
             Dim token$ = String.Empty
 
             With Responser
-                If .File.Exists Then
-                    .LoadSettings()
-                    With .Headers
-                        If .ContainsKey(Header_CSRF_TOKEN) Then token = .Item(Header_CSRF_TOKEN)
-                        If .ContainsKey(Header_IG_APP_ID) Then app_id = .Item(Header_IG_APP_ID)
-                        If .ContainsKey(Header_IG_WWW_CLAIM) Then www_claim = .Item(Header_IG_WWW_CLAIM)
-                    End With
-                Else
-                    .CookiesDomain = "instagram.com"
-                    .Cookies = New CookieKeeper(.CookiesDomain)
-                    .SaveSettings()
+                If .Headers.Count > 0 Then
+                    token = .Headers.Value(Header_CSRF_TOKEN)
+                    app_id = .Headers.Value(Header_IG_APP_ID)
+                    www_claim = .Headers.Value(Header_IG_WWW_CLAIM)
                 End If
+                .CookiesExtractMode = Responser.CookiesExtractModes.Response
+                .CookiesUpdateMode = CookieKeeper.UpdateModes.ReplaceByNameAll
+                .CookiesExtractedAutoSave = False
             End With
 
             Dim n() As String = {SettingsCLS.Name_Node_Sites, Site.ToString}
 
-            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
-
-            Hash = New PropertyValue(String.Empty, GetType(String))
-            HashSavedPosts = New PropertyValue(String.Empty, GetType(String))
+            HashTagged = New PropertyValue(String.Empty, GetType(String))
             CSRF_TOKEN = New PropertyValue(token, GetType(String), Sub(v) ChangeResponserFields(NameOf(CSRF_TOKEN), v))
             IG_APP_ID = New PropertyValue(app_id, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_APP_ID), v))
             IG_WWW_CLAIM = New PropertyValue(www_claim, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_WWW_CLAIM), v))
 
+            DownloadTimeline = New PropertyValue(True)
+            DownloadStories = New PropertyValue(True)
+            DownloadTagged = New PropertyValue(False)
+
             RequestsWaitTimer = New PropertyValue(1000)
             RequestsWaitTimerProvider = New TimersChecker(100)
             RequestsWaitTimerTaskCount = New PropertyValue(1)
@@ -230,6 +231,7 @@ Namespace API.Instagram
             SleepTimerOnPostsLimit = New PropertyValue(60000)
             SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
 
+            GetTimeline = New PropertyValue(True)
             GetStories = New PropertyValue(False)
             GetTagged = New PropertyValue(False)
             TaggedNotifyLimit = New PropertyValue(200)
@@ -240,44 +242,14 @@ Namespace API.Instagram
             LastDownloadDate = New XMLValue(Of Date)("LastDownloadDate", Now.AddDays(-1), _XML, n)
             LastRequestsCount = New XMLValue(Of Integer)("LastRequestsCount", 0, _XML, n)
             LastRequestsCountLabel = New PropertyValue(LastRequestsCountLabelStr.Invoke(LastRequestsCount.Value))
-            AddHandler LastRequestsCount.OnValueChanged, Sub(sender, __name, __value) LastRequestsCountLabel.Value = LastRequestsCountLabelStr.Invoke(__value)
+            AddHandler LastRequestsCount.OnValueChanged, Sub(sender, __name, __value) LastRequestsCountLabel.Value = LastRequestsCountLabelStr.Invoke(DirectCast(__value, Existable(Of Integer)).Value)
 
             UrlPatternUser = "https://www.instagram.com/{0}/"
             UserRegex = RParams.DMS("[htps:/]{7,8}.*?instagram.com/([^/]+)", 1)
             ImageVideoContains = "instagram.com"
         End Sub
-        Friend Overrides Sub BeginInit()
-        End Sub
-        Friend Overrides Sub EndInit()
-            Initialized = True
-        End Sub
 #End Region
 #Region "PropertiesDataChecker"
-        <PropertiesDataChecker({NameOf(Hash), NameOf(HashSavedPosts)})>
-        Private Function CheckHashControls(ByVal p As IEnumerable(Of PropertyData)) As Boolean
-            If p.ListExists(2) Then
-                Dim h$ = String.Empty
-                Dim hsp$ = String.Empty
-                For Each pp As PropertyData In p
-                    Select Case pp.Name
-                        Case NameOf(Hash) : h = AConvert(Of String)(pp.Value, String.Empty)
-                        Case NameOf(HashSavedPosts) : hsp = AConvert(Of String)(pp.Value, String.Empty)
-                    End Select
-                Next
-                If h.IsEmptyString And hsp.IsEmptyString Then
-                    Return True
-                Else
-                    If h = hsp Then
-                        MsgBoxE({"InstaHash for saved posts must be different from InstaHash!", "InstaHash are equal"}, vbCritical)
-                        Return False
-                    Else
-                        Return True
-                    End If
-                End If
-            Else
-                Return False
-            End If
-        End Function
         <PropertiesDataChecker({NameOf(TaggedNotifyLimit)})>
         Private Function CheckNotifyLimit(ByVal p As IEnumerable(Of PropertyData)) As Boolean
             If p.ListExists Then
@@ -303,20 +275,15 @@ Namespace API.Instagram
                 Case Download.Main : Return New UserData
                 Case Download.SavedPosts
                     Dim u As New UserData
-                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}
+                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = Site}
                     Return u
             End Select
             Return Nothing
         End Function
 #Region "Downloading"
+        Friend Property SkipUntilNextSession As Boolean = False
         Friend Overrides Function ReadyToDownload(ByVal What As Download) As Boolean
-            If ActiveJobs < 2 AndAlso ReadyForDownload AndAlso BaseAuthExists Then
-                Select Case What
-                    Case Download.Main : Return ACheck(Hash.Value)
-                    Case Download.SavedPosts : Return ACheck(HashSavedPosts.Value)
-                End Select
-            End If
-            Return False
+            Return ActiveJobs < 2 AndAlso Not SkipUntilNextSession AndAlso ReadyForDownload AndAlso BaseAuthExists() AndAlso DownloadTimeline.Value
         End Function
         Private ActiveJobs As Integer = 0
         Private _NextWNM As UserData.WNM = UserData.WNM.Notify
@@ -345,6 +312,10 @@ Namespace API.Instagram
                 _NextTagged = .TaggedCheckSession
                 LastDownloadDate.Value = Now
                 LastRequestsCount.Value = .RequestsCount
+                _FieldsChangerSuspended = True
+                IG_WWW_CLAIM.Value = Responser.Headers.Value(Header_IG_WWW_CLAIM)
+                CSRF_TOKEN.Value = Responser.Headers.Value(Header_CSRF_TOKEN)
+                _FieldsChangerSuspended = False
             End With
         End Sub
         Friend Overrides Sub DownloadDone(ByVal What As Download)
@@ -352,10 +323,11 @@ Namespace API.Instagram
             _NextTagged = True
             LastDownloadDate.Value = Now
             ActiveJobs -= 1
+            SkipUntilNextSession = False
         End Sub
 #End Region
-        Friend Overrides Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
-            Return UserData.GetVideoInfo(URL, Responser, Me)
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            Return UserData.GetVideoInfo(URL, Responser)
         End Function
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
             If Options Is Nothing OrElse Not TypeOf Options Is EditorExchangeOptions Then Options = New EditorExchangeOptions(Me)
@@ -363,6 +335,14 @@ Namespace API.Instagram
                 Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Try
+                Dim code$ = DirectCast(User, UserData).GetPostCodeById(Media.Post.ID)
+                If Not code.IsEmptyString Then Return $"https://instagram.com/p/{code}/" Else Return String.Empty
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "Can't open user's post", String.Empty)
+            End Try
+        End Function
 #End Region
     End Class
 End Namespace

SCrawler/API/Instagram/UserData.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,44 +6,83 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.XML
-Imports PersonalUtilities.Functions.Messaging
-Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
-Imports PersonalUtilities.Tools.WebDocuments.JSON
-Imports SCrawler.API.Base
 Imports System.Net
 Imports System.Threading
-Imports System.Reflection
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
+Imports PersonalUtilities.Functions.Messaging
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports SCrawler.API.Base
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Instagram
     Friend Class UserData : Inherits UserDataBase
 #Region "XML Names"
         Private Const Name_LastCursor As String = "LastCursor"
         Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
+        Private Const Name_GetTimeline As String = "GetTimeline"
         Private Const Name_GetStories As String = "GetStories"
         Private Const Name_GetTagged As String = "GetTaggedData"
         Private Const Name_TaggedChecked As String = "TaggedChecked"
 #End Region
 #Region "Declarations"
+        Private Structure PostKV : Implements IEContainerProvider
+            Private Const Name_Code As String = "Code"
+            Private Const Name_Section As String = "Section"
+            Friend Code As String
+            Friend ID As String
+            Friend Section As Sections
+            Friend Sub New(ByVal _Section As Sections)
+                Section = _Section
+            End Sub
+            Friend Sub New(ByVal _Code As String, ByVal _ID As String, ByVal _Section As Sections)
+                Code = _Code
+                ID = _ID
+                Section = _Section
+            End Sub
+            Private Sub New(ByVal e As EContainer)
+                Code = e.Attribute(Name_Code)
+                Section = e.Attribute(Name_Section)
+                ID = e.Value
+            End Sub
+            Public Shared Widening Operator CType(ByVal e As EContainer) As PostKV
+                Return New PostKV(e)
+            End Operator
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                If Not IsNothing(Obj) AndAlso TypeOf Obj Is PostKV Then
+                    With DirectCast(Obj, PostKV)
+                        Return Code = .Code And ID = .ID And Section = .Section
+                    End With
+                Else
+                    Return False
+                End If
+            End Function
+            Private Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer
+                Return New EContainer("Post", ID, {New EAttribute(Name_Section, CInt(Section)), New EAttribute(Name_Code, Code)})
+            End Function
+        End Structure
         Private ReadOnly Property MySiteSettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
             End Get
         End Property
-        Private ReadOnly _SavedPostsIDs As New List(Of String)
+        Private ReadOnly PostsKVIDs As List(Of PostKV)
+        Private ReadOnly PostsToReparse As List(Of PostKV)
         Private LastCursor As String = String.Empty
         Private FirstLoadingDone As Boolean = False
+        Friend Property GetTimeline As Boolean = True
         Friend Property GetStories As Boolean
         Friend Property GetTaggedData As Boolean
 #End Region
 #Region "Exchange options"
         Friend Overrides Function ExchangeOptionsGet() As Object
-            Return New EditorExchangeOptions(HOST.Source) With {.GetStories = GetStories, .GetTagged = GetTaggedData}
+            Return New EditorExchangeOptions(HOST.Source) With {.GetTimeline = GetTimeline, .GetStories = GetStories, .GetTagged = GetTaggedData}
         End Function
         Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
             If Not Obj Is Nothing AndAlso TypeOf Obj Is EditorExchangeOptions Then
                 With DirectCast(Obj, EditorExchangeOptions)
+                    GetTimeline = .GetTimeline
                     GetStories = .GetStories
                     GetTaggedData = .GetTagged
                 End With
@@ -52,17 +91,21 @@ Namespace API.Instagram
 #End Region
 #Region "Initializer, loader"
         Friend Sub New()
+            PostsKVIDs = New List(Of PostKV)
+            PostsToReparse = New List(Of PostKV)
         End Sub
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
             If Loading Then
                 LastCursor = Container.Value(Name_LastCursor)
                 FirstLoadingDone = Container.Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
+                GetTimeline = Container.Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
                 GetStories = Container.Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
                 GetTaggedData = Container.Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
                 TaggedChecked = Container.Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
             Else
                 Container.Add(Name_LastCursor, LastCursor)
                 Container.Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
+                Container.Add(Name_GetTimeline, GetTimeline.BoolToInteger)
                 Container.Add(Name_GetStories, GetStories.BoolToInteger)
                 Container.Add(Name_GetTagged, GetTaggedData.BoolToInteger)
                 Container.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
@@ -70,41 +113,138 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "Download data"
+        Private E560Thrown As Boolean = False
         Private Class ExitException : Inherits Exception
-            Friend Sub New(ByRef CompleteArg As Boolean)
-                CompleteArg = True
+            Friend Shared Sub Throw560(ByRef Source As UserData)
+                If Not Source.E560Thrown Then
+                    MyMainLOG = $"{Source.ToStringForLog}: (560) Download skipped until next session"
+                    Source.E560Thrown = True
+                End If
+                Throw New ExitException
             End Sub
         End Class
-        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+        Private Sub LoadSavePostsKV(ByVal Load As Boolean)
+            Dim x As XmlFile
+            Dim f As SFile = MyFilePosts
+            If Not f.IsEmptyString Then
+                f.Name &= "_KV"
+                f.Extension = "xml"
+                If Load Then
+                    PostsKVIDs.Clear()
+                    x = New XmlFile(f, Protector.Modes.All, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                    x.LoadData()
+                    If x.Count > 0 Then PostsKVIDs.ListAddList(x, LAP.IgnoreICopier)
+                    x.Dispose()
+                Else
+                    x = New XmlFile With {.AllowSameNames = True}
+                    x.AddRange(PostsKVIDs)
+                    x.Name = "Posts"
+                    x.Save(f, EDP.SendInLog)
+                    x.Dispose()
+                End If
+            End If
+        End Sub
+        Private Overloads Function PostKvExists(ByVal pkv As PostKV) As Boolean
+            Return PostKvExists(pkv.ID, False, pkv.Section) OrElse PostKvExists(pkv.Code, True, pkv.Section)
+        End Function
+        Private Overloads Function PostKvExists(ByVal PostCodeId As String, ByVal IsCode As Boolean, ByVal Section As Sections) As Boolean
+            If Not PostCodeId.IsEmptyString And PostsKVIDs.Count > 0 Then
+                If PostsKVIDs.FindIndex(Function(p) p.Section = Section AndAlso If(IsCode, p.Code = PostCodeId, p.ID = PostCodeId)) >= 0 Then
+                    Return True
+                ElseIf Not IsCode Then
+                    Return _TempPostsList.Contains(GetPostIdBySection(PostCodeId, Section)) Or
+                           _TempPostsList.Contains(PostCodeId.Replace($"_{ID}", String.Empty)) Or
+                           _TempPostsList.Contains(GetPostIdBySection(PostCodeId.Replace($"_{ID}", String.Empty), Section))
+                End If
+            End If
+            Return False
+        End Function
+        Friend Function GetPostCodeById(ByVal PostID As String) As String
             Try
-                _InstaHash = String.Empty
+                If Not PostID.IsEmptyString Then
+                    Dim f As SFile = MyFilePosts
+                    If Not f.IsEmptyString Then
+                        f.Name &= "_KV"
+                        f.Extension = "xml"
+                        Dim l As List(Of PostKV) = Nothing
+                        Using x As New XmlFile(f, Protector.Modes.All, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                            x.LoadData()
+                            l.ListAddList(x, LAP.IgnoreICopier)
+                        End Using
+                        Dim code$ = String.Empty
+                        If l.ListExists Then
+                            Dim i% = l.FindIndex(Function(p) p.ID = PostID)
+                            If i >= 0 Then code = l(i).Code
+                            l.Clear()
+                        End If
+                        Return code
+                    End If
+                End If
+                Return String.Empty
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{ToStringForLog()}: Cannot find post code by ID ({PostID})", String.Empty)
+            End Try
+        End Function
+        Private Function GetPostIdBySection(ByVal ID As String, ByVal Section As Sections) As String
+            If Section = Sections.Timeline Then
+                Return ID
+            Else
+                Return $"{Section}_{ID}"
+            End If
+        End Function
+        Private _DownloadingInProgress As Boolean = False
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Dim s As Sections = Sections.Timeline
+            Dim errorFound As Boolean = False
+            Try
+                LoadSavePostsKV(True)
+                _DownloadingInProgress = True
+                AddHandler Responser.ResponseReceived, AddressOf Responser_ResponseReceived
+                ThrowAny(Token)
                 HasError = False
-                If Not LastCursor.IsEmptyString Then
-                    DownloadData(LastCursor, Sections.Timeline, Token)
+                Dim dt As Func(Of Boolean) = Function() (CBool(MySiteSettings.DownloadTimeline.Value) And GetTimeline) Or IsSavedPosts
+                If dt.Invoke And Not LastCursor.IsEmptyString Then
+                    s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
+                    DownloadData(LastCursor, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
-                If Not HasError Then
-                    DownloadData(String.Empty, Sections.Timeline, Token)
+                If dt.Invoke And Not HasError Then
+                    s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
+                    DownloadData(String.Empty, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
                 If FirstLoadingDone Then LastCursor = String.Empty
-                If IsSavedPosts Then
-                    DownloadPosts(Token)
-                ElseIf MySiteSettings.BaseAuthExists Then
-                    DownloadedTags = 0
-                    If GetStories Then DownloadData(String.Empty, Sections.Stories, Token)
-                    If GetTaggedData Then DownloadData(String.Empty, Sections.Tagged, Token)
+                If Not IsSavedPosts AndAlso MySiteSettings.BaseAuthExists() Then
+                    If CBool(MySiteSettings.DownloadStories.Value) And GetStories Then s = Sections.Stories : DownloadData(String.Empty, s, Token)
+                    If CBool(MySiteSettings.DownloadTagged.Value) And ACheck(MySiteSettings.HashTagged.Value) And GetTaggedData Then s = Sections.Tagged : DownloadData(String.Empty, s, Token)
                 End If
                 If WaitNotificationMode = WNM.SkipTemp Or WaitNotificationMode = WNM.SkipCurrent Then WaitNotificationMode = WNM.Notify
             Catch eex As ExitException
             Catch ex As Exception
-                ProcessException(ex, Token, "[API.Instagram.UserData.DownloadDataF", False)
+                errorFound = True
+                Throw ex
+            Finally
+                E560Thrown = False
+                UpdateResponser()
+                If Not errorFound Then LoadSavePostsKV(False)
             End Try
         End Sub
-        Private _InstaHash As String = String.Empty
-        Private Enum Sections : Timeline : Tagged : Stories : End Enum
+        Private Sub UpdateResponser()
+            Try
+                If _DownloadingInProgress AndAlso Not Responser Is Nothing AndAlso Not Responser.Disposed Then
+                    _DownloadingInProgress = False
+                    RemoveHandler Responser.ResponseReceived, AddressOf Responser_ResponseReceived
+                    Declarations.UpdateResponser(Responser, MySiteSettings.Responser)
+                End If
+            Catch
+            End Try
+        End Sub
+        Private Sub Responser_ResponseReceived(ByVal Sender As Object, ByVal e As EventArguments.WebDataResponse)
+            Declarations.UpdateResponser(e, Responser)
+        End Sub
+        Private Enum Sections : Timeline : Tagged : Stories : SavedPosts : End Enum
         Private Const StoriesFolder As String = "Stories"
         Private Const TaggedFolder As String = "Tagged"
 #Region "429 bypass"
@@ -129,7 +269,7 @@ Namespace API.Instagram
                                               "What do you want to do?", "Waiting for Instagram download...",
                                               {
                                                New MsgBoxButton("Wait") With {.ToolTip = "Wait and ask again when the error is found."},
-                                               New MsgBoxButton("Wait (disable current") With {.ToolTip = "Wait and skip future prompts while downloading the current profile."},
+                                               New MsgBoxButton("Wait (disable current)") With {.ToolTip = "Wait and skip future prompts while downloading the current profile."},
                                                New MsgBoxButton("Abort") With {.ToolTip = "Abort operation"},
                                                New MsgBoxButton("Wait (disable all)") With {.ToolTip = "Wait and skip future prompts while downloading the current session."}
                                               },
@@ -164,13 +304,11 @@ Namespace API.Instagram
 #Region "Tags"
         Private TaggedChecked As Boolean = False
         Friend TaggedCheckSession As Boolean = True
-        Private DownloadedTags As Integer = 0
         Private DownloadTagsLimit As Integer? = Nothing
-        Private ReadOnly Property TaggedLimitsNotifications(Optional ByVal v As Integer? = Nothing) As Boolean
+        Private ReadOnly Property TaggedLimitsNotifications(ByVal v As Integer) As Boolean
             Get
                 Return Not TaggedChecked AndAlso TaggedCheckSession AndAlso
-                       CInt(MySiteSettings.TaggedNotifyLimit.Value) > 0 AndAlso
-                       (Not v.HasValue OrElse v.Value > CInt(MySiteSettings.TaggedNotifyLimit.Value))
+                       CInt(MySiteSettings.TaggedNotifyLimit.Value) > 0 AndAlso v > CInt(MySiteSettings.TaggedNotifyLimit.Value)
             End Get
         End Property
         Private Function SetTagsLimit(ByVal Max As Integer, ByVal p As ANumbers) As DialogResult
@@ -203,8 +341,9 @@ Namespace API.Instagram
         End Function
         Private Function TaggedContinue(ByVal TaggedCount As Integer) As DialogResult
             Dim agi As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
-            Dim msg As New MMessage($"The number of tagged posts by user [{ToString()}] is {TaggedCount.NumToString(agi)}" & vbCr &
-                                    $"This is about {(TaggedCount / 12).RoundUp.NumToString(agi)} requests." & vbCr &
+            Dim msg As New MMessage($"The number of already downloaded tagged posts by user [{ToString()}] is {TaggedCount.NumToString(agi)}" & vbCr &
+                                    "There is currently no way to know how many posts exist." & vbCr &
+                                    "One request will be spent per post." & vbCr &
                                     "The tagged data download operation can take a long time.",
                                     "Too much tagged data",
                                     {
@@ -236,39 +375,45 @@ Namespace API.Instagram
             Dim URL$ = String.Empty
             Dim StoriesList As List(Of String) = Nothing
             Dim StoriesRequested As Boolean = False
-            Dim _DownloadComplete As Boolean = False
+            Dim dValue% = 1
             LastCursor = Cursor
             Try
-                Do While Not _DownloadComplete
+                Do While dValue = 1
                     ThrowAny(Token)
                     If Not Ready() Then Thread.Sleep(10000) : ThrowAny(Token) : Continue Do
                     ReconfigureAwaiter()
 
                     Try
-                        Dim n As EContainer, nn As EContainer, node As EContainer
+                        Dim n As EContainer, nn As EContainer
                         Dim HasNextPage As Boolean = False
                         Dim EndCursor$ = String.Empty
                         Dim PostID$ = String.Empty, PostDate$ = String.Empty, SpecFolder$ = String.Empty
-                        Dim TaggedCount%
+                        Dim PostIDKV As PostKV
                         Dim ENode() As Object = Nothing
                         NextRequest(True)
 
                         'Check environment
-                        If Cursor.IsEmptyString And _InstaHash.IsEmptyString Then _
-                            _InstaHash = CStr(If(IsSavedPosts, MySiteSettings.HashSavedPosts, MySiteSettings.Hash).Value)
-                        If ID.IsEmptyString Then GetUserId()
-                        If ID.IsEmptyString Then Throw New ArgumentException("User ID is not detected", "ID")
+                        If Not IsSavedPosts Then
+                            If ID.IsEmptyString Then GetUserId()
+                            If ID.IsEmptyString Then Throw New ArgumentException("User ID is not detected", "ID")
+                        End If
 
                         'Create query
                         Select Case Section
                             Case Sections.Timeline
+                                URL = $"https://www.instagram.com/api/v1/feed/user/{Name}/username/?count=50" &
+                                        If(Cursor.IsEmptyString, String.Empty, $"&max_id={Cursor}")
+                                ENode = Nothing
+                            Case Sections.SavedPosts
+                                SavedPostsDownload(String.Empty, Token)
+                                Exit Sub
+                            Case Sections.Tagged
+                                Dim h$ = AConvert(Of String)(MySiteSettings.HashTagged.Value, String.Empty)
+                                If h.IsEmptyString Then Throw New ExitException
                                 Dim vars$ = "{""id"":" & ID & ",""first"":50,""after"":""" & Cursor & """}"
                                 vars = SymbolsConverter.ASCII.EncodeSymbolsOnly(vars)
-                                URL = $"https://www.instagram.com/graphql/query/?query_hash={_InstaHash}&variables={vars}"
+                                URL = $"https://www.instagram.com/graphql/query/?query_hash={h}&variables={vars}"
                                 ENode = {"data", "user", 0}
-                            Case Sections.Tagged
-                                URL = $"https://i.instagram.com/api/v1/usertags/{ID}/feed/?count=50&max_id={Cursor}"
-                                ENode = {"items"}
                                 SpecFolder = TaggedFolder
                             Case Sections.Stories
                                 If Not StoriesRequested Then
@@ -286,7 +431,7 @@ Namespace API.Instagram
                                 If StoriesList.ListExists Then
                                     Continue Do
                                 Else
-                                    Throw New ExitException(_DownloadComplete)
+                                    Throw New ExitException
                                 End If
                         End Select
 
@@ -296,76 +441,76 @@ Namespace API.Instagram
                         RequestsCount += 1
                         ThrowAny(Token)
 
-                        'Data
+                        'Parsing
                         If Not r.IsEmptyString Then
                             Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
-                                n = j.ItemF(ENode).XmlIfNothing
+                                n = If(ENode Is Nothing, j, j.ItemF(ENode)).XmlIfNothing
                                 If n.Count > 0 Then
                                     Select Case Section
                                         Case Sections.Timeline
-                                            If n.Contains("page_info") Then
-                                                With n("page_info")
-                                                    HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
-                                                    EndCursor = .Value("end_cursor")
-                                                End With
-                                            End If
-                                            n = n("edges").XmlIfNothing
-                                            If n.Count > 0 Then
-                                                For Each nn In n
-                                                    ThrowAny(Token)
-                                                    node = nn(0).XmlIfNothing
-                                                    If IsSavedPosts Then
-                                                        PostID = node.Value("shortcode")
-                                                        If Not PostID.IsEmptyString Then
-                                                            If _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete) Else _SavedPostsIDs.Add(PostID)
-                                                        End If
-                                                    Else
-                                                        PostID = node.Value("id")
-                                                        If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
-                                                        _TempPostsList.Add(PostID)
-                                                        PostDate = node.Value("taken_at_timestamp")
-                                                        If Not CheckDatesLimit(PostDate, DateProvider) Then Throw New ExitException(_DownloadComplete)
-                                                        ObtainMedia(node, PostID, PostDate, SpecFolder)
-                                                    End If
-                                                Next
-                                            End If
+                                            With n
+                                                HasNextPage = .Value("more_available").FromXML(Of Boolean)(False)
+                                                EndCursor = .Value("next_max_id")
+                                                If If(.Item("items")?.Count, 0) > 0 Then
+                                                    If Not DefaultParser(.Item("items"), Section, Token) Then Throw New ExitException
+                                                Else
+                                                    HasNextPage = False
+                                                End If
+                                            End With
                                         Case Sections.Tagged
-                                            HasNextPage = j.Value("more_available").FromXML(Of Boolean)(False)
-                                            EndCursor = j.Value("next_max_id")
-                                            For Each nn In n
-                                                PostID = $"Tagged_{nn.Value("id")}"
-                                                If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
-                                                _TempPostsList.Add(PostID)
-                                                ObtainMedia2(nn, PostID, SpecFolder)
-                                                DownloadedTags += 1
-                                                If DownloadTagsLimit.HasValue AndAlso DownloadedTags >= DownloadTagsLimit.Value Then Throw New ExitException(_DownloadComplete)
-                                            Next
-                                            If TaggedLimitsNotifications Then
-                                                TaggedCount = j.Value("total_count").FromXML(Of Integer)(0)
+                                            With n
+                                                If .Contains("page_info") Then
+                                                    With .Item("page_info")
+                                                        HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
+                                                        EndCursor = .Value("end_cursor")
+                                                    End With
+                                                Else
+                                                    HasNextPage = False
+                                                End If
+                                                If If(.Item("edges")?.Count, 0) > 0 Then
+                                                    For Each nn In .Item("edges")
+                                                        PostIDKV = New PostKV(Section)
+                                                        If nn.Count > 0 AndAlso nn(0).Count > 0 Then
+                                                            With nn(0)
+                                                                PostIDKV = New PostKV(.Value("shortcode"), .Value("id"), Section)
+                                                                If PostKvExists(PostIDKV) Then
+                                                                    Throw New ExitException
+                                                                Else
+                                                                    If Not DownloadTagsLimit.HasValue OrElse PostsToReparse.Count + 1 < DownloadTagsLimit.Value Then
+                                                                        _TempPostsList.Add(GetPostIdBySection(PostIDKV.ID, Section))
+                                                                        PostsKVIDs.ListAddValue(PostIDKV, LAP.NotContainsOnly)
+                                                                        PostsToReparse.ListAddValue(PostIDKV, LNC)
+                                                                    ElseIf DownloadTagsLimit.HasValue OrElse PostsToReparse.Count + 1 >= DownloadTagsLimit.Value Then
+                                                                        Throw New ExitException
+                                                                    End If
+                                                                End If
+                                                            End With
+                                                        End If
+                                                    Next
+                                                Else
+                                                    HasNextPage = False
+                                                End If
+                                            End With
+                                            If TaggedLimitsNotifications(PostsToReparse.Count) Then
                                                 TaggedChecked = True
-                                                If TaggedLimitsNotifications(TaggedCount) AndAlso
-                                                   TaggedContinue(TaggedCount) = DialogResult.Cancel Then Throw New ExitException(_DownloadComplete)
+                                                If TaggedContinue(PostsToReparse.Count) = DialogResult.Cancel Then Throw New ExitException
                                             End If
                                     End Select
                                 Else
-                                    If j.Value("status") = "ok" AndAlso j({"data", "user"}).XmlIfNothing.Count = 0 AndAlso
+                                    If j.Value("status") = "ok" AndAlso If(j("items")?.Count, 0) = 0 AndAlso
                                        _TempMediaList.Count = 0 AndAlso Section = Sections.Timeline Then _
-                                       UserExists = False : Throw New ExitException(_DownloadComplete)
+                                       UserExists = False : Throw New ExitException
                                 End If
                             End Using
                         Else
-                            Throw New ExitException(_DownloadComplete)
+                            Throw New ExitException
                         End If
-                        _DownloadComplete = True
+                        dValue = 0
                         If HasNextPage And Not EndCursor.IsEmptyString Then DownloadData(EndCursor, Section, Token)
                     Catch eex As ExitException
                         Throw eex
-                    Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                        Exit Do
-                    Catch dex As ObjectDisposedException When Disposed
-                        Exit Do
                     Catch ex As Exception
-                        If DownloadingException(ex, $"data downloading error [{URL}]", Section, False) = 1 Then Continue Do Else Exit Do
+                        dValue = ProcessException(ex, Token, $"data downloading error [{URL}]",, Section, False)
                     End Try
                 Loop
             Catch eex2 As ExitException
@@ -373,15 +518,15 @@ Namespace API.Instagram
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
-                ProcessException(DoEx, Token, $"data downloading error [{URL}]")
+                ProcessException(DoEx, Token, $"data downloading error [{URL}]",, Section)
             End Try
         End Sub
         Private Sub DownloadPosts(ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
-            Dim _DownloadComplete As Boolean = False
+            Dim dValue% = 1
             Dim _Index% = 0
             Try
-                Do While Not _DownloadComplete
+                Do While dValue = 1
                     ThrowAny(Token)
                     If Not Ready() Then Thread.Sleep(10000) : ThrowAny(Token) : Continue Do
                     ReconfigureAwaiter()
@@ -389,12 +534,11 @@ Namespace API.Instagram
                     Try
                         Dim r$
                         Dim j As EContainer, jj As EContainer
-                        Dim _MediaObtained As Boolean
-                        If _SavedPostsIDs.Count > 0 And _Index <= _SavedPostsIDs.Count - 1 Then
+                        If PostsToReparse.Count > 0 And _Index <= PostsToReparse.Count - 1 Then
                             Dim e As New ErrorsDescriber(EDP.ThrowException)
-                            For i% = _Index To _SavedPostsIDs.Count - 1
+                            For i% = _Index To PostsToReparse.Count - 1
                                 _Index = i
-                                URL = $"https://instagram.com/p/{_SavedPostsIDs(i)}/?__a=1"
+                                URL = $"https://www.instagram.com/api/v1/media/{PostsToReparse(i).ID}/info/"
                                 ThrowAny(Token)
                                 NextRequest(((i + 1) Mod 5) = 0)
                                 ThrowAny(Token)
@@ -404,17 +548,9 @@ Namespace API.Instagram
                                 If Not r.IsEmptyString Then
                                     j = JsonDocument.Parse(r)
                                     If Not j Is Nothing Then
-                                        _MediaObtained = False
-                                        If j.Contains({"graphql", "shortcode_media"}) Then
-                                            With j({"graphql", "shortcode_media"}).XmlIfNothing
-                                                If .Count > 0 Then ObtainMedia(.Self, _SavedPostsIDs(i), String.Empty, String.Empty) : _MediaObtained = True
-                                            End With
-                                        End If
-                                        If Not _MediaObtained AndAlso j.Contains("items") Then
+                                        If If(j("items")?.Count, 0) > 0 Then
                                             With j("items")
-                                                If .Count > 0 Then
-                                                    For Each jj In .Self : ObtainMedia2(jj, _SavedPostsIDs(i)) : Next
-                                                End If
+                                                For Each jj In .Self : ObtainMedia(jj, PostsToReparse(i).ID) : Next
                                             End With
                                         End If
                                         j.Dispose()
@@ -422,50 +558,109 @@ Namespace API.Instagram
                                 End If
                             Next
                         End If
-                        _DownloadComplete = True
-                    Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                        Exit Do
-                    Catch dex As ObjectDisposedException When Disposed
-                        Exit Do
+                        dValue = 0
+                    Catch eex As ExitException
+                        Throw eex
                     Catch ex As Exception
-                        If DownloadingException(ex, $"downloading saved posts error [{URL}]") = 1 Then Continue Do Else Exit Do
+                        dValue = ProcessException(ex, Token, $"downloading posts error [{URL}]",, Sections.Tagged, False)
                     End Try
                 Loop
+            Catch eex2 As ExitException
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
-                ProcessException(DoEx, Token, $"downloading saved posts error [{URL}]")
+                ProcessException(DoEx, Token, $"downloading posts error [{URL}]",, Sections.Tagged)
             End Try
         End Sub
-        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
-        End Sub
-#End Region
-#Region "Obtain Media"
-        Private Sub ObtainMedia(ByVal node As EContainer, ByVal PostID As String, ByVal PostDate As String, ByVal SpecFolder As String)
-            Dim CreateMedia As Action(Of EContainer) =
-                Sub(ByVal e As EContainer)
-                    Dim t As UTypes = If(e.Value("is_video").FromXML(Of Boolean)(False), UTypes.Video, UTypes.Picture)
-                    Dim tmpValue$
-                    If t = UTypes.Picture Then
-                        tmpValue = e.Value("display_url")
-                    Else
-                        tmpValue = e.Value("video_url")
+        Private Sub SavedPostsDownload(ByVal Cursor As String, ByVal Token As CancellationToken)
+            Dim URL$ = $"https://www.instagram.com/api/v1/feed/saved/posts/?max_id={Cursor}"
+            Dim HasNextPage As Boolean = False
+            Dim NextCursor$ = String.Empty
+            ThrowAny(Token)
+            Dim r$ = Responser.GetResponse(URL)
+            Dim nodes As IEnumerable(Of EContainer) = Nothing
+            If Not r.IsEmptyString Then
+                Using e As EContainer = JsonDocument.Parse(r)
+                    If If(e?.Count, 0) > 0 Then
+                        With e
+                            HasNextPage = .Value("more_available").FromXML(Of Boolean)(False)
+                            NextCursor = .Value("next_max_id")
+                            If .Contains("items") Then nodes = (From ee As EContainer In .Item("items") Where ee.Count > 0 Select ee(0))
+                        End With
+                        If nodes.ListExists Then
+                            DefaultParser(nodes, Sections.SavedPosts, Token)
+                            If HasNextPage And Not NextCursor.IsEmptyString Then SavedPostsDownload(NextCursor, Token)
+                        End If
                     End If
-                    If Not tmpValue.IsEmptyString Then _TempMediaList.ListAddValue(MediaFromData(t, tmpValue, PostID, PostDate, SpecFolder), LNC)
-                End Sub
-            If node.Contains({"edge_sidecar_to_children", "edges"}) Then
-                For Each edge As EContainer In node({"edge_sidecar_to_children", "edges"}) : CreateMedia(edge("node").XmlIfNothing) : Next
-            Else
-                CreateMedia(node)
+                End Using
             End If
         End Sub
-        Private Sub ObtainMedia2(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
+        Private Function DefaultParser(ByVal Items As IEnumerable(Of EContainer), ByVal Section As Sections, ByVal Token As CancellationToken,
+                                       Optional ByVal SpecFolder As String = Nothing) As Boolean
+            ThrowAny(Token)
+            If Items.Count > 0 Then
+                Dim PostIDKV As PostKV
+                Dim Pinned As Boolean
+                Dim PostDate$
+                If SpecFolder.IsEmptyString Then
+                    Select Case Section
+                        Case Sections.Tagged : SpecFolder = TaggedFolder
+                        Case Sections.Stories : SpecFolder = StoriesFolder
+                        Case Else : SpecFolder = String.Empty
+                    End Select
+                End If
+                For Each nn In Items
+                    With nn
+                        PostIDKV = New PostKV(.Value("code"), .Value("id"), Section)
+                        Pinned = .Contains("timeline_pinned_user_ids")
+                        If PostKvExists(PostIDKV) And Not Pinned Then Return False
+                        _TempPostsList.Add(PostIDKV.ID)
+                        PostsKVIDs.ListAddValue(PostIDKV, LAP.NotContainsOnly)
+                        PostDate = .Value("taken_at")
+                        If Not IsSavedPosts Then
+                            Select Case CheckDatesLimit(PostDate, DateProvider)
+                                Case DateResult.Skip : Continue For
+                                Case DateResult.Exit : If Not Pinned Then Return False
+                            End Select
+                        End If
+                        ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate)
+                    End With
+                Next
+                Return True
+            Else
+                Return False
+            End If
+        End Function
+#End Region
+#Region "Code ID converters"
+        Private Shared Function CodeToID(ByVal Code As String) As String
+            Const CodeSymbols$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
+            Try
+                If Not Code.IsEmptyString Then
+                    Dim c As Char
+                    Dim id& = 0
+                    For i% = 0 To Code.Length - 1
+                        c = Code(i)
+                        id = (id * 64) + CodeSymbols.IndexOf(c)
+                    Next
+                    Return id
+                Else
+                    Return String.Empty
+                End If
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Instagram.UserData.CodeToID({Code})", String.Empty)
+            End Try
+        End Function
+#End Region
+#Region "Obtain Media"
+        Private Sub ObtainMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
                                  Optional ByVal DateObj As String = Nothing)
             Try
                 Dim img As Predicate(Of EContainer) = Function(_img) Not _img.Name.IsEmptyString AndAlso _img.Name.StartsWith("image_versions") AndAlso _img.Count > 0
                 Dim vid As Predicate(Of EContainer) = Function(_vid) Not _vid.Name.IsEmptyString AndAlso _vid.Name.StartsWith("video_versions") AndAlso _vid.Count > 0
                 Dim ss As Func(Of EContainer, Sizes) = Function(_ss) New Sizes(_ss.Value("width"), _ss.Value("url"))
                 Dim mDate As Func(Of EContainer, String) = Function(ByVal elem As EContainer) As String
+                                                               If Not DateObj.IsEmptyString Then Return DateObj
                                                                If elem.Contains("taken_at") Then
                                                                    Return elem.Value("taken_at")
                                                                ElseIf elem.Contains("imported_taken_at") Then
@@ -474,7 +669,7 @@ Namespace API.Instagram
                                                                    Dim ev$ = elem.Value("device_timestamp")
                                                                    If Not ev.IsEmptyString Then
                                                                        If ev.Length > 10 Then
-                                                                           Return elem.Value("device_timestamp").Substring(0, 10)
+                                                                           Return ev.Substring(0, 10)
                                                                        Else
                                                                            Return ev
                                                                        End If
@@ -530,7 +725,7 @@ Namespace API.Instagram
                                 DateObj = mDate(n)
                                 With n("carousel_media").XmlIfNothing
                                     If .Count > 0 Then
-                                        For Each d In .Self : ObtainMedia2(d, PostID, SpecialFolder, DateObj) : Next
+                                        For Each d In .Self : ObtainMedia(d, PostID, SpecialFolder, DateObj) : Next
                                     End If
                                 End With
                         End Select
@@ -543,22 +738,6 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "GetUserId"
-        <Obsolete> Private Sub GetUserId_Old()
-            Try
-                Dim r$ = Responser.GetResponse($"https://www.instagram.com/{Name}/?__a=1",, EDP.ThrowException)
-                If Not r.IsEmptyString Then
-                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
-                        ID = j({"graphql", "user"}, "id").XmlIfNothingValue
-                    End Using
-                End If
-            Catch ex As Exception
-                If Responser.StatusCode = HttpStatusCode.NotFound Or Responser.StatusCode = HttpStatusCode.BadRequest Then
-                    Throw ex
-                Else
-                    LogError(ex, "get Instagram user id")
-                End If
-            End Try
-        End Sub
         Private Sub GetUserId()
             Try
                 Dim r$ = Responser.GetResponse($"https://i.instagram.com/api/v1/users/web_profile_info/?username={Name}",, EDP.ThrowException)
@@ -607,7 +786,7 @@ Namespace API.Instagram
                                                 pid = storyID & s.Value("id")
                                                 If Not _TempPostsList.Contains(pid) Then
                                                     ThrowAny(Token)
-                                                    ObtainMedia2(s, pid, sFolder)
+                                                    ObtainMedia(s, pid, sFolder)
                                                     _TempPostsList.Add(pid)
                                                 End If
                                             Next
@@ -631,7 +810,7 @@ Namespace API.Instagram
                 End If
                 Return Nothing
             Catch ex As Exception
-                DownloadingException(ex, "API.Instagram.GetStoriesList", Sections.Stories, False)
+                DownloadingException(ex, "API.Instagram.GetStoriesList", False, Sections.Stories)
                 Return Nothing
             End Try
         End Function
@@ -642,19 +821,24 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "Exceptions"
+        ''' <exception cref="ExitException"></exception>
+        ''' <inheritdoc cref="UserDataBase.ThrowAny(CancellationToken)"/>
+        Friend Overrides Sub ThrowAny(ByVal Token As CancellationToken)
+            If MySiteSettings.SkipUntilNextSession Then ExitException.Throw560(Me)
+            MyBase.ThrowAny(Token)
+        End Sub
         ''' <summary>
-        ''' <inheritdoc cref="UserDataBase.DownloadingException(Exception, String)"/><br/>
+        ''' <inheritdoc cref="UserDataBase.DownloadingException(Exception, String, Boolean, Object)"/><br/>
         ''' 1 - continue
         ''' </summary>
-        Protected Overloads Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False) As Integer
-            Return DownloadingException(ex, Message, Sections.Timeline, FromPE)
-        End Function
-        Private Overloads Function DownloadingException(ByVal ex As Exception, ByVal Message As String, ByVal s As Sections, ByVal FromPE As Boolean) As Integer
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal s As Object = Nothing) As Integer
             If Responser.StatusCode = HttpStatusCode.NotFound Then
                 UserExists = False
             ElseIf Responser.StatusCode = HttpStatusCode.BadRequest Then
                 HasError = True
-                MyMainLOG = $"Instagram credentials have expired [{CInt(Responser.StatusCode)}]: {ToString()} [{s}]"
+                MyMainLOG = $"Instagram credentials have expired [{CInt(Responser.StatusCode)}]: {ToStringForLog()} [{s}]"
+                DisableSection(s)
             ElseIf Responser.StatusCode = HttpStatusCode.Forbidden And s = Sections.Tagged Then
                 Return 3
             ElseIf Responser.StatusCode = 429 Then
@@ -666,13 +850,26 @@ Namespace API.Instagram
                 Caught429 = True
                 MyMainLOG = $"Number of requests before error 429: {RequestsCount}"
                 Return 1
+            ElseIf Responser.StatusCode = 560 Then
+                MySiteSettings.SkipUntilNextSession = True
             Else
                 MyMainLOG = $"Instagram hash requested [{CInt(Responser.StatusCode)}]: {ToString()} [{s}]"
+                DisableSection(s)
                 If Not FromPE Then LogError(ex, Message) : HasError = True
                 Return 0
             End If
             Return 2
         End Function
+        Private Sub DisableSection(ByVal Section As Object)
+            If Not IsNothing(Section) AndAlso TypeOf Section Is Sections Then
+                Dim s As Sections = DirectCast(Section, Sections)
+                Select Case s
+                    Case Sections.Timeline : MySiteSettings.DownloadTimeline.Value = False
+                    Case Else : MySiteSettings.DownloadTagged.Value = False
+                End Select
+                MyMainLOG = $"[{s}] downloading is disabled until you update your credentials".ToUpper
+            End If
+        End Sub
 #End Region
 #Region "Create media"
         Private Shared Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
@@ -686,16 +883,17 @@ Namespace API.Instagram
         End Function
 #End Region
 #Region "Standalone downloader"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Response, ByVal _Settings As SiteSettings) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Responser) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString AndAlso URL.Contains("instagram.com") Then
                     Dim PID$ = RegexReplace(URL, RParams.DMS(".*?instagram.com/p/([_\w\d]+)", 1))
+                    If Not PID.IsEmptyString AndAlso Not ACheck(Of Long)(PID) Then PID = CodeToID(PID)
                     If Not PID.IsEmptyString Then
                         Using t As New UserData
-                            t.SetEnvironment(Settings(_Settings.GetType.GetCustomAttribute(Of Plugin.Attributes.Manifest)().GUID), Nothing, False, False)
-                            t.Responser = New Response
+                            t.SetEnvironment(Settings(InstagramSiteKey), Nothing, False, False)
+                            t.Responser = New Responser
                             t.Responser.Copy(r)
-                            t._SavedPostsIDs.Add(PID)
+                            t.PostsToReparse.Add(New PostKV With {.ID = PID})
                             t.DownloadPosts(Nothing)
                             Return ListAddList(Nothing, t._TempMediaList)
                         End Using
@@ -703,13 +901,19 @@ Namespace API.Instagram
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, "Instagram standalone downloader: fetch media error")
+                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, $"Instagram standalone downloader: fetch media error ({URL})")
             End Try
         End Function
 #End Region
 #Region "IDisposable Support"
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            If Not disposedValue And disposing Then _SavedPostsIDs.Clear()
+            If Not disposedValue Then
+                UpdateResponser()
+                If disposing Then
+                    PostsKVIDs.Clear()
+                    PostsToReparse.Clear()
+                End If
+            End If
             MyBase.Dispose(disposing)
         End Sub
 #End Region

SCrawler/API/LPSG/Declarations.vb

@@ -0,0 +1,37 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.LPSG
+    Friend Module Declarations
+        Friend ReadOnly Property PhotoRegEx As RParams = RParams.DM("(https://www.lpsg.com/attachments)(.+?)(?="")", 0, RegexReturn.List)
+        Friend ReadOnly Property PhotoRegExExt As New RParams("img.data.src=""(/proxy[^""]+?)""", Nothing, 1, RegexReturn.List) With {
+            .Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
+        Friend ReadOnly Property NextPageRegex As RParams = RParams.DMS("<link rel=""next"" href=""(.+?/page-(\d+))""", 2)
+        Private Const FileUrlRegexDefault As String = "([^/]+?)(jpg|jpeg|gif|png|webm)"
+        Private ReadOnly InputFReplacer As New ErrorsDescriber(EDP.ReturnValue)
+        Private ReadOnly InputForbidRemover As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, Input, Input.StringRemoveWinForbiddenSymbols(, InputFReplacer))
+        Friend ReadOnly Property FileRegEx As New RParams(FileUrlRegexDefault, Nothing, 0) With {
+            .Converter = Function(ByVal Input As String) As String
+                             Input = InputForbidRemover.Invoke(Input)
+                             If Not Input.IsEmptyString Then
+                                 Dim lv$ = Input.Split("-").LastOrDefault
+                                 If Not lv.IsEmptyString Then
+                                     Input = Input.Replace($"-{lv}", String.Empty)
+                                     Input &= $".{lv}"
+                                 End If
+                             End If
+                             Return Input
+                         End Function}
+        Friend ReadOnly Property FileRegExExt As New RParams(FileUrlRegexDefault, 0, Nothing, InputForbidRemover)
+        Friend ReadOnly Property FileRegExExt2 As New RParams("([^/]+?)(?=(\Z|&))", 0, Nothing, InputForbidRemover)
+        Friend ReadOnly Property FileExistsRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 2)
+        Friend ReadOnly Property TempListAddParams As New ListAddParams(LAP.NotContainsOnly) With {.Comparer = New FComparer(Of UserMedia)(Function(x, y) x.URL = y.URL)}
+    End Module
+End Namespace

SCrawler/API/LPSG/SiteSettings.vb

@@ -0,0 +1,37 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.LPSG
+    <Manifest("AndyProgram_LPSG")>
+    Friend Class SiteSettings : Inherits Base.SiteSettingsBase
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.LPSGIcon_48
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.LPSGPic_32
+            End Get
+        End Property
+        Friend Sub New()
+            MyBase.New("LPSG", "www.lpsg.com")
+            UrlPatternUser = "https://www.lpsg.com/threads/{0}/"
+            UserRegex = RParams.DMS(".+?lpsg.com/threads/([^/]+)", 1)
+        End Sub
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return If(Responser.Cookies?.Count, 0) > 0
+        End Function
+    End Class
+End Namespace

SCrawler/API/LPSG/UserData.vb

@@ -0,0 +1,105 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Imports Converters = PersonalUtilities.Functions.SymbolsConverter.Converters
+Namespace API.LPSG
+    Friend Class UserData : Inherits UserDataBase
+        Private Const Name_LatestPage As String = "LatestPage"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            If Loading Then
+                LatestPage = Container.Value(Name_LatestPage)
+            Else
+                Container.Add(Name_LatestPage, LatestPage)
+            End If
+        End Sub
+        Private Property LatestPage As String = String.Empty
+        Private Enum Mode : Internal : External : End Enum
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Responser.DeclaredError = EDP.ThrowException
+
+                Dim NextPage$
+                Dim r$
+                Dim _LPage As Func(Of String) = Function() If(LatestPage.IsEmptyString, String.Empty, $"page-{LatestPage}")
+
+                Do
+                    URL = $"https://www.lpsg.com/threads/{Name}/{_LPage.Invoke}"
+                    r = Responser.GetResponse(URL)
+                    UserExists = True
+                    UserSuspended = False
+                    ThrowAny(Token)
+                    If Not r.IsEmptyString Then
+                        NextPage = RegexReplace(r, NextPageRegex)
+                        UpdateMediaList(RegexReplace(r, PhotoRegEx), Mode.Internal)
+                        UpdateMediaList(RegexReplace(r, PhotoRegExExt), Mode.External)
+                        If NextPage = LatestPage Or NextPage.IsEmptyString Then Exit Do Else LatestPage = NextPage
+                    Else
+                        Exit Do
+                    End If
+                Loop
+
+                If _TempMediaList.ListExists And _ContentList.ListExists Then _
+                   _TempMediaList.RemoveAll(Function(m) _ContentList.Exists(Function(mm) mm.URL = m.URL))
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+        Private Sub UpdateMediaList(ByVal l As List(Of String), ByVal m As Mode)
+            If l.ListExists Then
+                Dim f As SFile
+                Dim u$
+                Dim exists As Boolean
+                Dim r As RParams
+                Dim ude As New ErrorsDescriber(EDP.ReturnValue)
+                For Each url$ In l
+                    If Not url.IsEmptyString Then u = SymbolsConverter.Decode(url, {Converters.HTML, Converters.ASCII}, ude) Else u = String.Empty
+                    If Not u.IsEmptyString Then
+                        exists = Not IsEmptyString(RegexReplace(u, FileExistsRegEx))
+                        If m = Mode.Internal Then
+                            r = FileRegEx
+                        Else
+                            r = FileRegExExt
+                            If Not exists Then
+                                r = FileRegExExt2
+                                exists = Not IsEmptyString(RegexReplace(u, FileRegExExt2))
+                            End If
+                        End If
+                        If exists Then
+                            f = CStr(RegexReplace(u, r))
+                            f.Path = MyFile.CutPath.PathNoSeparator
+                            f.Separator = "\"
+                            If f.Extension.IsEmptyString Then f.Extension = "jpg"
+                            _TempMediaList.ListAddValue(New UserMedia With {.Type = UTypes.Picture, .URL = url, .File = f}, TempListAddParams)
+                        End If
+                    End If
+                Next
+            End If
+        End Sub
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            With Responser : .Mode = Responser.Modes.WebClient : .ResetStatus() : End With
+            UseResponserClient = True
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                MyMainLOG = $"{ToStringForLog()}: LPSG not available"
+                Return 1
+            Else
+                Return 0
+            End If
+        End Function
+    End Class
+End Namespace

SCrawler/API/PornHub/Declarations.vb

@@ -0,0 +1,44 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.PornHub
+    Friend Module Declarations
+#Region "Converters"
+        Private ReadOnly UnicodeHexConverter As Func(Of String, String) = Function(Input) SymbolsConverter.UnicodeHex.Decode(Input, EDP.ReturnValue)
+        Friend ReadOnly HtmlConverter As Func(Of String, String) = Function(Input) SymbolsConverter.HTML.Decode(Input, EDP.ReturnValue)
+#End Region
+#Region "Declarations video"
+        Friend ReadOnly RegexVideo_FlashVarsBlock As RParams = RParams.DM("(?<=flashvars_\['[nN]ext[vV]ideo'\];[\r\n]*?)(.+?)(?=;flashvars_\d+?)", 0, EDP.ReturnValue)
+        Friend ReadOnly RegexVideo_FlashVars_Vars As RParams = RParams.DM("var ([\w\d]{10,})=("".+?)(?=(;|\Z))", 0, RegexReturn.List)
+        Friend ReadOnly RegexVideo_FlashVars_Compiler As RParams = RParams.DM("(?<=\*/)([\w\d\S]{10,})", 0, RegexReturn.List)
+        Friend ReadOnly RegexVideo_Video_All As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""",
+                                                                     0, RegexReturn.List, EDP.ReturnValue, UnicodeHexConverter)
+        Friend ReadOnly RegexVideo_Video_Wrong As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""[\w\W\s\r\n]+?(?=\<div class=""videoUploaderBlock)",
+                                                                       0, RegexReturn.List, EDP.ReturnValue, UnicodeHexConverter)
+        Private ReadOnly RegexVideo_Video_Wrong_Option As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""[\w\W\s\r\n]+?", 0, RegexReturn.ListByMatch)
+        Friend ReadOnly RegexVideo_Video_Wrong_Fields As RField() = {New RField(New RFieldOption(1, RegexVideo_Video_Wrong_Option)), New RField(New RFieldOption(2, RegexVideo_Video_Wrong_Option))}
+        Friend ReadOnly RegexVideo_Video_VideoKey As RParams = RParams.DMS("viewkey=([\w\d]+)", 1, EDP.ReturnValue)
+#End Region
+#Region "Declarations M3U8"
+        Friend ReadOnly Regex_M3U8_FirstFileRegEx As RParams = RParams.DM(".+?m3u8.*", 0)
+        Friend ReadOnly Regex_M3U8_FileUrl As RParams = RParams.DMS("((https://([^/]+)/.+?)([^/]+?m3u8))(.*)", 2, EDP.ReturnValue)
+#End Region
+#Region "Declarations GIF"
+        Friend ReadOnly Regex_Gif_Array As RParams = RParams.DM("\<li id=""(gif\d+)"" class=""gifLi.gifVideoBlock""\>", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly Regex_Gif_UrlName As RParams = RParams.DMS("""name"":.*?""([^""]*)""[^\}]+?""contentUrl"":.*?""([^""]+)""", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
+#End Region
+#Region "Declarations photo"
+        Friend ReadOnly Regex_Photo_ModelHub_PhotoBlocks As RParams = RParams.DM("var PHOTOS_ARRAY_(\d+) = \{[\r\n\s]*?(urls:.*?\[[^]]*\])", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly Regex_Photo_PornHub_PhotoBlocks As RParams = RParams.DM("photoAlbumListContainer[\r\n\s\S]+?title=""([^""]+)""[\r\n\s\S]+?a href=""(/album/\d+)""", 0, RegexReturn.List)
+        Friend ReadOnly Regex_Photo_PornHub_AlbumPhotoArr As RParams = RParams.DMS("\<a href=""(/photo/\d+)""", 1, RegexReturn.List, EDP.ReturnValue,
+                                                                                   CType(Function(Input$) If(Input.IsEmptyString, String.Empty, $"https://www.pornhub.com{Input.Trim}"), Func(Of String, String)))
+        Friend ReadOnly Regex_Photo_PornHub_SinglePhoto As RParams = RParams.DMS("(?<!thumbImage.+?)<img src=""(https://[^""]+\d+[^""]+)""", 1, EDP.ReturnValue)
+#End Region
+    End Module
+End Namespace

SCrawler/API/PornHub/M3U8.vb

@@ -0,0 +1,42 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.API.Base.M3U8Declarations
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.PornHub
+    Friend NotInheritable Class M3U8
+        Private Sub New()
+        End Sub
+        Private Shared Function GetUrlsList(ByVal URL As String, ByVal Responser As Responser) As List(Of String)
+            Dim appender$ = RegexReplace(URL, Regex_M3U8_FileUrl)
+            Dim r$ = Responser.GetResponse(URL)
+            If Not r.IsEmptyString Then
+                Dim file$ = RegexReplace(r, Regex_M3U8_FirstFileRegEx)
+                If Not file.IsEmptyString Then
+                    Dim NewUrl$ = M3U8Base.CreateUrl(appender, file)
+                    If Not NewUrl.IsEmptyString Then
+                        r = Responser.GetResponse(NewUrl)
+                        If Not r.IsEmptyString Then
+                            Dim l As List(Of String) = RegexReplace(r, TsFilesRegEx)
+                            If l.ListExists Then
+                                For i% = 0 To l.Count - 1 : l(i) = M3U8Base.CreateUrl(appender, l(i)) : Next
+                                Return l
+                            End If
+                        End If
+                    End If
+                End If
+            End If
+            Return Nothing
+        End Function
+        Friend Shared Function Download(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As SFile
+            Return M3U8Base.Download(GetUrlsList(URL, Responser), Destination, Responser)
+        End Function
+    End Class
+End Namespace

SCrawler/API/PornHub/OptionsForm.Designer.vb

@@ -0,0 +1,118 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.PornHub
+    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
+    Partial Friend Class OptionsForm : Inherits System.Windows.Forms.Form
+        <System.Diagnostics.DebuggerNonUserCode()>
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            Try
+                If disposing AndAlso components IsNot Nothing Then
+                    components.Dispose()
+                End If
+            Finally
+                MyBase.Dispose(disposing)
+            End Try
+        End Sub
+        Private components As System.ComponentModel.IContainer
+        <System.Diagnostics.DebuggerStepThrough()>
+        Private Sub InitializeComponent()
+            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
+            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
+            Me.CH_DOWN_GIFS = New System.Windows.Forms.CheckBox()
+            Me.CH_DOWN_PHOTO_MODELHUB = New System.Windows.Forms.CheckBox()
+            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
+            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
+            CONTAINER_MAIN.ContentPanel.SuspendLayout()
+            CONTAINER_MAIN.SuspendLayout()
+            TP_MAIN.SuspendLayout()
+            Me.SuspendLayout()
+            '
+            'CONTAINER_MAIN
+            '
+            '
+            'CONTAINER_MAIN.ContentPanel
+            '
+            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(278, 52)
+            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            CONTAINER_MAIN.LeftToolStripPanelVisible = False
+            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
+            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
+            CONTAINER_MAIN.RightToolStripPanelVisible = False
+            CONTAINER_MAIN.Size = New System.Drawing.Size(278, 77)
+            CONTAINER_MAIN.TabIndex = 0
+            CONTAINER_MAIN.TopToolStripPanelVisible = False
+            '
+            'TP_MAIN
+            '
+            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            TP_MAIN.ColumnCount = 1
+            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.Controls.Add(Me.CH_DOWN_GIFS, 0, 0)
+            TP_MAIN.Controls.Add(Me.CH_DOWN_PHOTO_MODELHUB, 0, 1)
+            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            TP_MAIN.Location = New System.Drawing.Point(0, 0)
+            TP_MAIN.Name = "TP_MAIN"
+            TP_MAIN.RowCount = 3
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.Size = New System.Drawing.Size(278, 52)
+            TP_MAIN.TabIndex = 0
+            '
+            'CH_DOWN_GIFS
+            '
+            Me.CH_DOWN_GIFS.AutoSize = True
+            Me.CH_DOWN_GIFS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_GIFS.Location = New System.Drawing.Point(4, 4)
+            Me.CH_DOWN_GIFS.Name = "CH_DOWN_GIFS"
+            Me.CH_DOWN_GIFS.Size = New System.Drawing.Size(270, 19)
+            Me.CH_DOWN_GIFS.TabIndex = 0
+            Me.CH_DOWN_GIFS.Text = "Download gifs"
+            Me.CH_DOWN_GIFS.UseVisualStyleBackColor = True
+            '
+            'CH_DOWN_PHOTO_MODELHUB
+            '
+            Me.CH_DOWN_PHOTO_MODELHUB.AutoSize = True
+            Me.CH_DOWN_PHOTO_MODELHUB.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_PHOTO_MODELHUB.Location = New System.Drawing.Point(4, 30)
+            Me.CH_DOWN_PHOTO_MODELHUB.Name = "CH_DOWN_PHOTO_MODELHUB"
+            Me.CH_DOWN_PHOTO_MODELHUB.Size = New System.Drawing.Size(270, 19)
+            Me.CH_DOWN_PHOTO_MODELHUB.TabIndex = 1
+            Me.CH_DOWN_PHOTO_MODELHUB.Text = "Download photo only from ModelHub"
+            Me.CH_DOWN_PHOTO_MODELHUB.UseVisualStyleBackColor = True
+            '
+            'OptionsForm
+            '
+            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
+            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
+            Me.ClientSize = New System.Drawing.Size(278, 77)
+            Me.Controls.Add(CONTAINER_MAIN)
+            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
+            Me.KeyPreview = True
+            Me.MaximizeBox = False
+            Me.MaximumSize = New System.Drawing.Size(294, 116)
+            Me.MinimizeBox = False
+            Me.MinimumSize = New System.Drawing.Size(294, 116)
+            Me.Name = "OptionsForm"
+            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
+            Me.Text = "Options"
+            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
+            CONTAINER_MAIN.ResumeLayout(False)
+            CONTAINER_MAIN.PerformLayout()
+            TP_MAIN.ResumeLayout(False)
+            TP_MAIN.PerformLayout()
+            Me.ResumeLayout(False)
+
+        End Sub
+        Private WithEvents CH_DOWN_GIFS As CheckBox
+        Private WithEvents CH_DOWN_PHOTO_MODELHUB As CheckBox
+    End Class
+End Namespace

SCrawler/API/PornHub/OptionsForm.resx

@@ -120,4 +120,7 @@
   <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
+  <metadata name="TP_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
 </root>

SCrawler/API/PornHub/OptionsForm.vb

@@ -0,0 +1,34 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Forms
+Namespace API.PornHub
+    Friend Class OptionsForm
+        Private WithEvents MyDefs As DefaultFormOptions
+        Private ReadOnly MyExchangeOptions As UserExchangeOptions
+        Friend Sub New(ByRef ExchangeOptions As UserExchangeOptions)
+            InitializeComponent()
+            MyExchangeOptions = ExchangeOptions
+            MyDefs = New DefaultFormOptions(Me, Settings.Design)
+        End Sub
+        Private Sub MyForm_Load(sender As Object, e As EventArgs) Handles Me.Load
+            With MyDefs
+                .MyViewInitialize(True)
+                .AddOkCancelToolbar()
+                CH_DOWN_GIFS.Checked = MyExchangeOptions.DownloadGifs
+                CH_DOWN_PHOTO_MODELHUB.Checked = MyExchangeOptions.DownloadPhotoOnlyFromModelHub
+                .EndLoaderOperations()
+            End With
+        End Sub
+        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
+            MyExchangeOptions.DownloadGifs = CH_DOWN_GIFS.Checked
+            MyExchangeOptions.DownloadPhotoOnlyFromModelHub = CH_DOWN_PHOTO_MODELHUB.Checked
+            MyDefs.CloseForm()
+        End Sub
+    End Class
+End Namespace

SCrawler/API/PornHub/SiteSettings.vb

@@ -0,0 +1,121 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.PornHub
+    <Manifest("AndyProgram_PornHub"), SavedPosts, SpecialForm(False), SeparatedTasks(1)>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.PornHubIcon_16
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.PornHubPic_16
+            End Get
+        End Property
+        Private ReadOnly Property CurlPathExists As Boolean
+        <PropertyOption(ControlText:="Download GIF", ControlToolTip:="Default for new users", ThreeStates:=True), PXML>
+        Friend ReadOnly Property DownloadGifs As PropertyValue
+        <PropertyOption(ControlText:="Download GIFs as mp4", ControlToolTip:="Download gifs in 'mp4' format instead of native 'webm'"), PXML>
+        Friend ReadOnly Property DownloadGifsAsMp4 As PropertyValue
+        <PropertyOption(ControlText:="Photo ModelHub only",
+                        ControlToolTip:="Download photo only from ModelHub. Prornstar photos hosted on PornHub itself will not be downloaded." & vbCr &
+                                        "Attention! Downloading photos hosted on PornHub is a very heavy job."), PXML>
+        Friend ReadOnly Property DownloadPhotoOnlyFromModelHub As PropertyValue
+        <PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
+        Friend ReadOnly Property SavedPostsUserName As PropertyValue
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("PornHub", "pornhub.com")
+            Responser.CurlPath = $"cURL\curl.exe"
+            Responser.CurlArgumentsRight = "--ssl-no-revoke"
+            CurlPathExists = Responser.CurlPath.Exists
+            Responser.DeclaredError = EDP.ThrowException
+
+            DownloadGifsAsMp4 = New PropertyValue(True)
+            DownloadGifs = New PropertyValue(CInt(CheckState.Indeterminate), GetType(Integer))
+            DownloadPhotoOnlyFromModelHub = New PropertyValue(True)
+            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
+
+            UrlPatternUser = "https://www.pornhub.com/{0}/{1}"
+            UserRegex = RParams.DMS("pornhub.com/([^/]+)/([^/]+).*?", 0, RegexReturn.ListByMatch)
+            ImageVideoContains = "pornhub"
+        End Sub
+#End Region
+#Region "GetInstance, GetSpecialData"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            If What = ISiteSettings.Download.SavedPosts Then
+                Return New UserData With {
+                    .IsSavedPosts = True,
+                    .VideoPageModel = UserData.VideoPageModels.Favorite,
+                    .PersonType = UserData.PersonTypeUser,
+                    .User = New UserInfo With {.Name = $"{UserData.PersonTypeUser}_{CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}"}
+                }
+            Else
+                Return New UserData
+            End If
+        End Function
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            If Available(ISiteSettings.Download.Main, True) Then
+                Using resp As Responser = Responser.Copy
+                    Dim spf$ = String.Empty
+                    Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
+                    Dim m As UserMedia = UserData.GetVideoInfo(URL, resp, f)
+                    If m.State = UserMedia.States.Downloaded Then
+                        m.SpecialFolder = f
+                        Return {m}
+                    End If
+                End Using
+            End If
+            Return Nothing
+        End Function
+#End Region
+#Region "Downloading"
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return Settings.UseM3U8 And CurlPathExists And (Not What = ISiteSettings.Download.SavedPosts OrElse ACheck(SavedPostsUserName.Value))
+        End Function
+#End Region
+#Region "IsMyUser"
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            Try
+                If Not UserURL.IsEmptyString Then
+                    Dim alist As List(Of String) = RegexReplace(UserURL.ToLower, UserRegex)
+                    If alist.ListExists(3) Then Return New ExchangeOptions(Site, $"{alist(1)}_{alist(2)}")
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.PornHub.SiteSettings.IsMyUser({UserURL})]", New ExchangeOptions)
+            End Try
+        End Function
+#End Region
+#Region "GetUserUrl, GetUserPostUrl"
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
+            With DirectCast(User, UserData) : Return String.Format(UrlPatternUser, .PersonType, .NameTrue) : End With
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return Media.URL_BASE
+        End Function
+#End Region
+#Region "User options"
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse Not TypeOf Options Is UserExchangeOptions Then Options = New UserExchangeOptions(Me)
+            If OpenForm Then
+                Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
+            End If
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/PornHub/UserData.vb

@@ -0,0 +1,657 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Namespace API.PornHub
+    Friend Class UserData : Inherits UserDataBase
+        Private Const UrlPattern As String = "https://www.pornhub.com/{0}"
+#Region "Declarations"
+#Region "XML names"
+        Private Const Name_PersonType As String = "PersonType"
+        Private Const Name_NameTrue As String = "NameTrue"
+        Private Const Name_VideoPageModel As String = "VideoPageModel"
+        Private Const Name_PhotoPageModel As String = "PhotoPageModel"
+        Private Const Name_DownloadGifs As String = "DownloadGifs"
+        Private Const Name_DownloadPhotoOnlyFromModelHub As String = "DownloadPhotoOnlyFromModelHub"
+#End Region
+#Region "Structures"
+        Private Structure FlashVar : Implements IRegExCreator
+            Friend Name As String
+            Friend Value As String
+            Public Shared Widening Operator CType(ByVal Name As String) As FlashVar
+                Return New FlashVar With {.Name = Name}
+            End Operator
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    Name = ParamsArray(0)
+                    Value = ParamsArray(1)
+                    If Not Value.IsEmptyString Then Value = Value.Replace(""" + """, String.Empty).Replace("""", String.Empty).StringTrim
+                End If
+                Return Me
+            End Function
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                Return CType(Obj, FlashVar).Name = Name
+            End Function
+        End Structure
+        Private Structure UserVideo : Implements IRegExCreator
+            Friend URL As String
+            Friend ID As String
+            Friend Title As String
+            Friend Function ToUserMedia() As UserMedia
+                Return New UserMedia(URL, UTypes.VideoPre) With {
+                    .File = If(Title.IsEmptyString, .File, New SFile($"{Title}.mp4")),
+                    .Post = ID
+                }
+            End Function
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists Then
+                    URL = ParamsArray(0)
+                    ID = RegexReplace(URL, RegexVideo_Video_VideoKey)
+                    URL = String.Format(UrlPattern, URL.TrimStart("/"))
+                    Title = HtmlConverter(ParamsArray(1)).StringRemoveWinForbiddenSymbols.StringTrim
+                End If
+                Return Me
+            End Function
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                Return DirectCast(Obj, UserVideo).URL = URL
+            End Function
+        End Structure
+        Private Structure PhotoBlock : Implements IRegExCreator
+            Friend AlbumID As String
+            Friend Data As String
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    AlbumID = ParamsArray(0)
+                    Data = ParamsArray(1).StringTrim
+                End If
+                Return Me
+            End Function
+        End Structure
+#End Region
+#Region "Enums"
+        Friend Enum VideoPageModels As Integer
+            [Default] = 0
+            ConcatPage = 1
+            Favorite = 2
+            Undefined = -1
+        End Enum
+        Private Enum PhotoPageModels As Integer
+            Undefined = 0
+            PornHubPage = 1
+            ModelHubPage = 2
+        End Enum
+#End Region
+#Region "Constants"
+        Private Const PersonTypeModel As String = "model"
+        Friend Const PersonTypeUser As String = "users"
+#End Region
+#Region "Person"
+        Friend Property PersonType As String
+        Friend Property NameTrue As String
+        Private _FriendlyName As String = String.Empty
+        Friend Overrides Property FriendlyName As String
+            Get
+                If _FriendlyName.IsEmptyString Then Return NameTrue Else Return _FriendlyName
+            End Get
+            Set(ByVal n As String)
+                _FriendlyName = n
+            End Set
+        End Property
+#End Region
+#Region "Advanced fields"
+        Friend Property VideoPageModel As VideoPageModels = VideoPageModels.Undefined
+        Private Property PhotoPageModel As PhotoPageModels = PhotoPageModels.Undefined
+        Friend Property DownloadGifs As Boolean
+        Friend Property DownloadPhotoOnlyFromModelHub As Boolean = True
+#End Region
+#Region "ExchangeOptions"
+        Friend Overrides Function ExchangeOptionsGet() As Object
+            Return New UserExchangeOptions(Me)
+        End Function
+        Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
+            If Not Obj Is Nothing AndAlso TypeOf Obj Is UserExchangeOptions Then
+                With DirectCast(Obj, UserExchangeOptions)
+                    DownloadGifs = .DownloadGifs
+                    DownloadPhotoOnlyFromModelHub = .DownloadPhotoOnlyFromModelHub
+                End With
+            End If
+        End Sub
+#End Region
+        Private ReadOnly Property MySettings As SiteSettings
+            Get
+                Return DirectCast(HOST.Source, SiteSettings)
+            End Get
+        End Property
+#End Region
+#Region "Initializer, loader"
+        Friend Sub New()
+            UseInternalM3U8Function = True
+        End Sub
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            With Container
+                Dim SetNames As Action = Sub()
+                                             If Not Name.IsEmptyString And NameTrue.IsEmptyString Then
+                                                 Dim n$() = Name.Split("_")
+                                                 If n.ListExists(2) Then
+                                                     NameTrue = Name.Replace($"{n(0)}_", String.Empty)
+                                                     PersonType = n(0)
+                                                     If (PersonType = PersonTypeModel Or PersonType = PersonTypeUser) And
+                                                        VideoPageModel = VideoPageModels.Undefined Then VideoPageModel = VideoPageModels.Default
+                                                 End If
+                                             End If
+                                         End Sub
+                If Loading Then
+                    PersonType = .Value(Name_PersonType)
+                    NameTrue = .Value(Name_NameTrue)
+                    VideoPageModel = .Value(Name_VideoPageModel).FromXML(Of Integer)(VideoPageModels.Undefined)
+                    PhotoPageModel = .Value(Name_PhotoPageModel).FromXML(Of Integer)(PhotoPageModels.Undefined)
+                    DownloadGifs = .Value(Name_DownloadGifs).FromXML(Of Integer)(False)
+                    DownloadPhotoOnlyFromModelHub = .Value(Name_DownloadPhotoOnlyFromModelHub).FromXML(Of Boolean)(True)
+                    SetNames.Invoke()
+                Else
+                    SetNames.Invoke()
+                    .Add(Name_PersonType, PersonType)
+                    .Add(Name_NameTrue, NameTrue)
+                    .Add(Name_VideoPageModel, CInt(VideoPageModel))
+                    .Add(Name_PhotoPageModel, CInt(PhotoPageModel))
+                    .Add(Name_DownloadGifs, DownloadGifs.BoolToInteger)
+                    .Add(Name_DownloadPhotoOnlyFromModelHub, DownloadPhotoOnlyFromModelHub.BoolToInteger)
+                End If
+            End With
+        End Sub
+#End Region
+#Region "Downloading"
+#Region "Download override"
+        Private Const DataDownloaded As Integer = -10
+        Private Const DataDownloaded_NotFound As Integer = -20
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Try
+                Responser.ResetStatus()
+                If PersonType = PersonTypeUser Then Responser.Mode = Responser.Modes.Curl
+
+                If IsSavedPosts Then VideoPageModel = VideoPageModels.Favorite
+
+                Dim page% = 1
+                Dim __continue As Boolean = True
+                Dim __videoDone As Boolean = False
+                Dim d%
+                If DownloadVideos Then
+                    If PersonType = PersonTypeUser Then Responser.Mode = Responser.Modes.Curl : Responser.Method = "POST"
+                    If VideoPageModel = VideoPageModels.Undefined Then
+                        __continue = False
+                        d = DownloadUserVideos(page, Token)
+                        Select Case d
+                            Case DataDownloaded : __continue = True : page += 1
+                            Case 1 : VideoPageModel = VideoPageModels.ConcatPage
+                            Case EXCEPTION_OPERATION_CANCELED : ThrowAny(Token)
+                            Case DataDownloaded_NotFound : __videoDone = True
+                        End Select
+                        If Not __continue And Not __videoDone Then
+                            d = DownloadUserVideos(page, Token)
+                            Select Case d
+                                Case DataDownloaded : __continue = True : page += 1
+                                Case 1 : VideoPageModel = VideoPageModels.Undefined
+                                Case EXCEPTION_OPERATION_CANCELED : ThrowAny(Token)
+                                Case DataDownloaded_NotFound : __videoDone = True
+                            End Select
+                        End If
+                    End If
+                    If __continue And Not __videoDone Then
+                        Do While DownloadUserVideos(page, Token) = DataDownloaded And page < 100 : page += 1 : Loop
+                    End If
+                    If _TempMediaList.Count > 0 Then _TempMediaList.RemoveAll(Function(m) Not m.Type = UTypes.m3u8 And Not m.Type = UTypes.VideoPre)
+                End If
+
+                Responser.Method = "GET"
+                If DownloadGifs And Not IsSavedPosts Then DownloadUserGifs(Token)
+                If DownloadImages Then DownloadUserPhotos(Token)
+            Finally
+                Responser.Mode = Responser.Modes.Default
+                Responser.Method = "GET"
+            End Try
+        End Sub
+#End Region
+#Region "Download video"
+        Private ReadOnly Property VideoPageType As String
+            Get
+                Select Case VideoPageModel
+                    Case VideoPageModels.Default : Return "/videos/upload"
+                    Case VideoPageModels.Favorite : Return "/videos/favorites/"
+                    Case Else : Return String.Empty
+                End Select
+            End Get
+        End Property
+        Private ReadOnly Property VideoPageAppender As String
+            Get
+                Return If(PersonType = PersonTypeUser, "ajax?o=newest&page=", String.Empty)
+            End Get
+        End Property
+        Private Overloads Function DownloadUserVideos(ByVal Page As Integer, ByVal Token As CancellationToken) As Integer
+            Const VideoUrlPattern$ = "https://www.pornhub.com/{0}/{1}{2}{3}"
+            Const HtmlPageNotFoundVideo$ = "<span>Error Page Not Found</span>"
+            Dim URL$ = String.Empty
+            Try
+                Dim p$
+                If PersonType = PersonTypeUser Then
+                    p = Page
+                Else
+                    p = IIf(Page = 1, String.Empty, $"?page={Page}")
+                End If
+
+                URL = $"{String.Format(VideoUrlPattern, PersonType, NameTrue, VideoPageType, VideoPageAppender)}{p}"
+                ThrowAny(Token)
+
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    If PersonType = PersonTypeUser And r.Contains(HtmlPageNotFoundVideo) Then Return DataDownloaded_NotFound
+                    Dim l As List(Of UserVideo) = RegexFields(Of UserVideo)(r, {RegexVideo_Video_All}, {1, 2})
+                    Dim lw As List(Of UserVideo) = Nothing
+                    If Not PersonType = PersonTypeUser Then lw = RegexFields(Of UserVideo)(r, {RegexVideo_Video_Wrong}, RegexVideo_Video_Wrong_Fields)
+                    If l.ListExists Then
+                        If lw.ListExists Then l.ListWithRemove(lw)
+                        If l.Count > 0 Then
+                            Dim lBefore% = l.Count
+                            l.RemoveAll(Function(ByVal uv As UserVideo) As Boolean
+                                            If Not _TempPostsList.Contains(uv.ID) Then
+                                                _TempPostsList.Add(uv.ID)
+                                                Return False
+                                            Else
+                                                Return True
+                                            End If
+                                        End Function)
+                            If l.Count > 0 Then _TempMediaList.ListAddList(l.Select(Function(uv) uv.ToUserMedia))
+                            If l.Count = lBefore And l.Count > 0 Then Return DataDownloaded
+                        End If
+                    End If
+                End If
+                Return DataDownloaded_NotFound
+            Catch regex_ex As RegexFieldsTextBecameNullException
+                If PersonType = PersonTypeUser Or IsSavedPosts Then
+                    Return DataDownloaded_NotFound
+                Else
+                    Return ProcessException(regex_ex, Token, $"videos downloading error [{URL}]")
+                End If
+            Catch ex As Exception
+                Return ProcessException(ex, Token, $"videos downloading error [{URL}]")
+            End Try
+        End Function
+#End Region
+#Region "Download GIF"
+        Private Sub DownloadUserGifs(ByVal Token As CancellationToken)
+            Dim URL$ = $"https://www.pornhub.com/{PersonType}/{NameTrue}/gifs"
+            Try
+                ThrowAny(Token)
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    Dim n$
+                    Dim m As UserMedia = Nothing
+                    Dim l As List(Of RegexMatchStruct) = RegexFields(Of RegexMatchStruct)(r, {Regex_Gif_Array}, {1})
+                    Dim l2 As List(Of String) = Nothing
+                    Dim l3 As List(Of String) = Nothing
+                    If l.ListExists Then l2 = l.Select(Function(ll) $"gif/{ll.Arr(0).Replace("gif", String.Empty)}").ToList
+                    If l2.ListExists Then
+                        For Each gif$ In l2
+                            If Not _TempPostsList.Contains(gif) Then
+                                _TempPostsList.Add(gif)
+                                URL = $"https://www.pornhub.com/{gif}"
+                                m = New UserMedia(URL, UTypes.Video) With {.Post = gif, .SpecialFolder = "GIFs\"}
+                                ThrowAny(Token)
+                                Try
+                                    r = Responser.GetResponse(URL)
+                                    If Not r.IsEmptyString Then
+                                        If l3.ListExists Then l3.Clear() : l3 = Nothing
+                                        l3 = RegexReplace(r, Regex_Gif_UrlName)
+                                        If l3.ListExists(3) Then
+                                            m.URL = l3(2)
+                                            m.File = m.URL
+                                            n = HtmlConverter(l3(1)).StringRemoveWinForbiddenSymbols.StringTrim
+                                            If MySettings.DownloadGifsAsMp4.Value Then m.File.Extension = "mp4"
+                                            If Not n.IsEmptyString Then m.File.Name = n
+                                        End If
+                                    End If
+                                Catch gif_down_ex As Exception
+                                    m.State = UserMedia.States.Missing
+                                End Try
+                                _TempMediaList.ListAddValue(m)
+                            End If
+                        Next
+                    End If
+                    If l.ListExists Then l.Clear()
+                    If l2.ListExists Then l2.Clear()
+                    If l3.ListExists Then l3.Clear()
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"gifs downloading error [{URL}]")
+            End Try
+        End Sub
+#End Region
+#Region "Download photo"
+        Private Const PhotoUrlPattern_ModelHub As String = "https://www.modelhub.com/{0}/photos"
+        Private Const PhotoUrlPattern_PornHub As String = "https://www.pornhub.com/{0}/{1}/photos"
+        Private Sub DownloadUserPhotos(ByVal Token As CancellationToken)
+            Try
+                If IsSavedPosts Then
+                    DownloadUserPhotos_SavedPosts(Token)
+                ElseIf PersonType = PersonTypeModel Then
+                    If PhotoPageModel = PhotoPageModels.Undefined Then
+                        If DownloadUserPhotos_ModelHub(Token) Then PhotoPageModel = PhotoPageModels.ModelHubPage
+                        ThrowAny(Token)
+                        If PhotoPageModel = PhotoPageModels.Undefined AndAlso Not DownloadPhotoOnlyFromModelHub AndAlso
+                           DownloadUserPhotos_PornHub(Token) Then PhotoPageModel = PhotoPageModels.PornHubPage
+                    Else
+                        Select Case PhotoPageModel
+                            Case PhotoPageModels.ModelHubPage : DownloadUserPhotos_ModelHub(Token)
+                            Case PhotoPageModels.PornHubPage : If Not DownloadPhotoOnlyFromModelHub Then DownloadUserPhotos_PornHub(Token)
+                        End Select
+                    End If
+                ElseIf Not DownloadPhotoOnlyFromModelHub Then
+                    DownloadUserPhotos_PornHub(Token)
+                End If
+                ThrowAny(Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, "photos downloading error")
+            End Try
+        End Sub
+        Private Function DownloadUserPhotos_ModelHub(ByVal Token As CancellationToken) As Boolean
+            Dim URL$ = String.Empty
+            Try
+                Dim jErr As New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue)
+                Dim albumName$
+                If PersonType = PersonTypeModel Then
+                    URL = String.Format(PhotoUrlPattern_ModelHub, NameTrue)
+                    Dim r$ = Responser.GetResponse(URL)
+                    If Not r.IsEmptyString Then
+                        Dim l As List(Of PhotoBlock) = RegexFields(Of PhotoBlock)(r, {Regex_Photo_ModelHub_PhotoBlocks}, {1, 2})
+                        If l.ListExists Then l.RemoveAll(Function(ll) ll.Data.IsEmptyString)
+                        If l.ListExists Then
+                            Dim albumRegex As RParams = RParams.DMS("", 1, EDP.ReturnValue)
+                            For Each block As PhotoBlock In l
+                                If Not _TempPostsList.Contains(block.AlbumID) Then _TempPostsList.Add(block.AlbumID) Else Continue For
+                                albumRegex.Pattern = "<li id=""" & block.AlbumID & """ class=""modelBox"">[\r\n\s]*?<div class=""modelPhoto"">[\r\n\s]*?\<[^\>]*?alt=""([^""]*)"""
+                                albumName = StringTrim(RegexReplace(r, albumRegex))
+                                If albumName.IsEmptyString Then albumName = block.AlbumID
+                                Using j As EContainer = JsonDocument.Parse("{" & block.Data & "}", jErr)
+                                    If Not j Is Nothing Then
+                                        If If(j("urls")?.Count, 0) > 0 Then
+                                            _TempMediaList.ListAddList(j("urls").Select(Function(jj) _
+                                                                       New UserMedia(jj.ItemF({0}).XmlIfNothingValue, UTypes.Picture) With {
+                                                                                     .SpecialFolder = $"Albums\{albumName}\"}), LNC)
+                                        End If
+                                    End If
+                                End Using
+                            Next
+                            l.Clear()
+                        End If
+                    End If
+                End If
+                Return True
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Overloads Function DownloadUserPhotos_PornHub(ByVal Token As CancellationToken) As Boolean
+            Try
+                Dim albumName$
+                Dim page%
+                Dim r$ = Responser.GetResponse(String.Format(PhotoUrlPattern_PornHub, PersonType, NameTrue))
+                If Not r.IsEmptyString Then
+                    Dim l As List(Of PhotoBlock) = RegexFields(Of PhotoBlock)(r, {Regex_Photo_PornHub_PhotoBlocks}, {2, 1})
+                    If l.ListExists Then l.RemoveAll(Function(ll) ll.AlbumID.IsEmptyString)
+                    If l.ListExists Then
+                        For Each block As PhotoBlock In l
+                            If Not _TempPostsList.Contains(block.AlbumID) Then _TempPostsList.Add(block.AlbumID) Else Continue For
+                            albumName = block.Data
+                            If albumName.IsEmptyString Then
+                                albumName = block.AlbumID.Split("/").LastOrDefault.StringTrim
+                            Else
+                                albumName = HtmlConverter(albumName).StringRemoveWinForbiddenSymbols.StringTrim
+                            End If
+                            page = 1
+                            Do While DownloadUserPhotos_PornHub(page, block.AlbumID, albumName, Token) : page += 1 : Loop
+                        Next
+                        l.Clear()
+                    End If
+                End If
+                Return True
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Overloads Function DownloadUserPhotos_PornHub(ByVal Page As Integer, ByVal AlbumID As String, ByVal AlbumName As String,
+                                                              ByVal Token As CancellationToken) As Boolean
+            Try
+                Dim r$ = Responser.GetResponse($"https://www.pornhub.com{AlbumID}{IIf(Page = 1, String.Empty, $"?page={Page}")}")
+                If Not r.IsEmptyString Then
+                    Dim l As List(Of String) = RegexReplace(r, Regex_Photo_PornHub_AlbumPhotoArr)
+                    If l.ListExists Then l.RemoveAll(Function(_url) _url.IsEmptyString)
+                    If l.ListExists Then
+                        For Each url$ In l
+                            ThrowAny(Token)
+                            Try
+                                r = Responser.GetResponse(url)
+                                If Not r.IsEmptyString Then
+                                    url = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
+                                    If Not url.IsEmptyString Then _
+                                       _TempMediaList.ListAddValue(New UserMedia(url, UTypes.Picture) With {.SpecialFolder = $"Albums\{AlbumName}\"}, LNC)
+                                End If
+                            Catch
+                            End Try
+                        Next
+                        l.Clear()
+                        Return True
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Function DownloadUserPhotos_SavedPosts(ByVal Token As CancellationToken) As Boolean
+            Const HtmlPageNotFoundPhoto$ = "Page Not Found"
+            Dim URL$ = $"https://www.pornhub.com/{PersonType}/{NameTrue}/photos/favorites"
+            Try
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    If r.Contains(HtmlPageNotFoundPhoto) Then Return False
+                    Dim urls As List(Of String) = RegexReplace(r, Regex_Photo_PornHub_AlbumPhotoArr)
+                    If urls.ListExists Then
+                        Dim NewUrl$
+                        Dim m As UserMedia
+                        Dim l2 As List(Of UserMedia) = urls.Select(Function(__url) New UserMedia(__url, UTypes.Picture) With {
+                                                                                                 .Post = __url.Split("/").LastOrDefault}).ToList
+                        urls.Clear()
+                        If l2.ListExists Then l2.RemoveAll(Function(media) media.URL.IsEmptyString)
+                        If l2.ListExists Then
+                            Dim lBefore% = l2.Count
+                            If _TempPostsList.Count > 0 Then l2.RemoveAll(Function(media) _TempPostsList.Contains(media.Post.ID))
+                            If l2.Count > 0 Then
+                                For i% = 0 To l2.Count - 1
+                                    m = l2(i)
+                                    ThrowAny(Token)
+                                    Try
+                                        r = Responser.GetResponse(m.URL)
+                                        If Not r.IsEmptyString Then
+                                            NewUrl = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
+                                            If Not NewUrl.IsEmptyString Then
+                                                m.URL = NewUrl
+                                                m.File = NewUrl
+                                                _TempPostsList.ListAddValue(m.Post.ID, LNC)
+                                            Else
+                                                Throw New Exception
+                                            End If
+                                        End If
+                                    Catch
+                                        m.State = UserMedia.States.Missing
+                                    End Try
+                                    _TempMediaList.ListAddValue(m, LNC)
+                                Next
+                            End If
+                            Return l2.Count = lBefore
+                        End If
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                Return ProcessException(ex, Token, $"photos downloading error [{URL}]")
+            End Try
+        End Function
+#End Region
+#End Region
+#Region "ReparseVideo"
+        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            Const ERR_NEW_URL$ = "ERR_NEW_URL"
+            Dim URL$ = String.Empty
+            Try
+                If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(tm) tm.Type = UTypes.VideoPre) Then
+                    Dim m As UserMedia
+                    Dim r$, NewUrl$
+                    For i% = _TempMediaList.Count - 1 To 0 Step -1
+                        If _TempMediaList(i).Type = UTypes.VideoPre Then
+                            m = _TempMediaList(i)
+                            ThrowAny(Token)
+                            Try
+                                URL = m.URL
+                                r = Responser.Curl(URL)
+                                If Not r.IsEmptyString Then
+                                    NewUrl = CreateVideoURL(r)
+                                    If NewUrl.IsEmptyString Then
+                                        Throw New Exception With {.HelpLink = ERR_NEW_URL}
+                                    Else
+                                        m.URL = NewUrl
+                                        m.Type = UTypes.m3u8
+                                        _TempMediaList(i) = m
+                                    End If
+                                Else
+                                    _TempMediaList.RemoveAt(i)
+                                End If
+                            Catch mid_ex As Exception
+                                If mid_ex.HelpLink = ERR_NEW_URL OrElse DownloadingException(mid_ex, "") = 1 Then
+                                    m.State = UserMedia.States.Missing
+                                    _TempMediaList(i) = m
+                                Else
+                                    _TempMediaList.RemoveAt(i)
+                                End If
+                            End Try
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "video reparsing error", False)
+            End Try
+        End Sub
+#End Region
+#Region "ReparseMissing"
+        Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
+            Dim rList As New List(Of Integer)
+            Try
+                If ContentMissingExists Then
+                    Dim m As UserMedia
+                    Dim r$
+                    Dim eCurl As New ErrorsDescriber(EDP.ReturnValue)
+                    For i% = 0 To _ContentList.Count - 1
+                        m = _ContentList(i)
+                        If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then
+                            ThrowAny(Token)
+                            r = Responser.Curl(m.URL_BASE, eCurl)
+                            If Not r.IsEmptyString Then
+                                Dim NewUrl$ = CreateVideoURL(r)
+                                If Not NewUrl.IsEmptyString Then
+                                    m.URL = NewUrl
+                                    _TempMediaList.ListAddValue(m, LNC)
+                                    rList.Add(i)
+                                End If
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "missing data downloading error")
+            Finally
+                If rList.Count > 0 Then
+                    For i% = rList.Count - 1 To 0 Step -1 : _ContentList.RemoveAt(rList(i)) : Next
+                    rList.Clear()
+                End If
+            End Try
+        End Sub
+#End Region
+#Region "Download content"
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+            Return M3U8.Download(URL, Responser, DestinationFile)
+        End Function
+#End Region
+#Region "CreateVideoURL"
+        Private Shared Function CreateVideoURL(ByVal r As String) As String
+            Try
+                Dim OutStr$ = String.Empty
+                If Not r.IsEmptyString Then
+                    Dim _VarBlock$ = RegexReplace(r, RegexVideo_FlashVarsBlock)
+                    If Not _VarBlock.IsEmptyString Then
+                        Dim vars As List(Of FlashVar) = RegexFields(Of FlashVar)(_VarBlock, {RegexVideo_FlashVars_Vars}, {1, 2})
+                        Dim compiler As List(Of String) = RegexReplace(_VarBlock, RegexVideo_FlashVars_Compiler)
+                        If vars.ListExists And compiler.ListExists Then
+                            Dim v$
+                            Dim i%
+                            For Each var$ In compiler
+                                i = vars.IndexOf(var)
+                                If i >= 0 Then
+                                    v = vars(i).Value
+                                    If Not v.IsEmptyString Then OutStr &= v
+                                End If
+                            Next
+                        End If
+                    End If
+                End If
+                Return OutStr
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.PornHub.UserData.CreateVideoURL]", String.Empty)
+            End Try
+        End Function
+#End Region
+#Region "Standalone downloader"
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As UserMedia
+            Try
+                Dim r$ = Responser.Curl(URL)
+                If Not r.IsEmptyString Then
+                    Dim NewUrl$ = CreateVideoURL(r)
+                    If Not NewUrl.IsEmptyString Then
+                        Dim f As SFile = M3U8.Download(NewUrl, Responser, Destination)
+                        If Not f.IsEmptyString Then Return New UserMedia With {.State = UserMedia.States.Downloaded}
+                    End If
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"PornHub standalone download error: [{URL}]", New UserMedia)
+            End Try
+        End Function
+#End Region
+#Region "Exceptions"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String,
+                                                          Optional ByVal FromPE As Boolean = False, Optional ByVal EObj As Object = Nothing) As Integer
+            If Responser.Status = Net.WebExceptionStatus.ConnectionClosed Then
+                Return 1
+            ElseIf Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                Return 2
+            Else
+                Return 0
+            End If
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/PornHub/UserExchangeOptions.vb

@@ -0,0 +1,23 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.PornHub
+    Friend Class UserExchangeOptions
+        Friend Property DownloadGifs As Boolean
+        Friend Property DownloadPhotoOnlyFromModelHub As Boolean
+        Friend Sub New(ByVal u As UserData)
+            DownloadGifs = u.DownloadGifs
+            DownloadPhotoOnlyFromModelHub = u.DownloadPhotoOnlyFromModelHub
+        End Sub
+        Friend Sub New(ByVal s As SiteSettings)
+            Dim v As CheckState = CInt(s.DownloadGifs.Value)
+            DownloadGifs = Not v = CheckState.Unchecked
+            DownloadPhotoOnlyFromModelHub = s.DownloadPhotoOnlyFromModelHub.Value
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Reddit/Channel.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -236,8 +236,19 @@ Namespace API.Reddit
             Return If(Name.IsEmptyString, ID, Name)
         End Function
         Friend Sub Delete()
-            File.Delete(, SFODelete.DeleteToRecycleBin)
-            FilePosts.Delete(, SFODelete.DeleteToRecycleBin)
+            Dim f As SFile = ChannelsCollection.ChannelsDeletedPath
+            With File
+                f.Name = .Name
+                f.Extension = .Extension
+                .Copy(f,, True, SFODelete.DeleteToRecycleBin)
+                .Delete(, SFODelete.DeleteToRecycleBin)
+            End With
+            With FilePosts
+                f.Name = .Name
+                f.Extension = .Extension
+                .Copy(f,, True, SFODelete.DeleteToRecycleBin)
+                .Delete(, SFODelete.DeleteToRecycleBin)
+            End With
         End Sub
         Friend Sub DownloadData(ByVal Token As CancellationToken, Optional ByVal SkipExists As Boolean = True,
                                 Optional ByVal p As MyProgress = Nothing)

SCrawler/API/Reddit/ChannelsCollection.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,10 +6,10 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
 Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Forms.Toolbars
-Imports SCrawler.API.Base
-Imports System.Threading
 Namespace API.Reddit
     Friend Class ChannelsCollection : Implements ICollection(Of Channel), IMyEnumerator(Of Channel), IChannelLimits, IDisposable
         Friend Shared ReadOnly Property ChannelsPath As SFile
@@ -17,6 +17,11 @@ Namespace API.Reddit
                 Return $"{SettingsFolderName}\Channels\"
             End Get
         End Property
+        Friend Shared ReadOnly Property ChannelsDeletedPath As SFile
+            Get
+                Return $"{SettingsFolderName}\ChannelsDeleted\"
+            End Get
+        End Property
         Friend Shared ReadOnly Property ChannelsPathCache As SFile
             Get
                 Return $"{Settings.GlobalPath.Value.PathWithSeparator}_CachedData\"

SCrawler/API/Reddit/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler/API/Reddit/IRedditView.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or

SCrawler/API/Reddit/M3U8.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -8,7 +8,7 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
 Imports SCrawler.API.Reddit.M3U8_Declarations
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Reddit
     Namespace M3U8_Declarations
@@ -130,7 +130,7 @@ Namespace API.Reddit
                     End If
                     If CachePath.Exists(SFO.Path) Then
                         Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
-                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ThrowException) 'EDP.ReturnValue
+                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ThrowException)
                         Dim i%
                         Dim eFiles As New List(Of SFile)
                         Dim dFile As SFile = CachePath

SCrawler/API/Reddit/RedditViewSettingsForm.Designer.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -27,7 +27,6 @@ Namespace API.Reddit
             Dim TP_VIEW_MODE As System.Windows.Forms.TableLayoutPanel
             Dim LBL_VIEW_MODE As System.Windows.Forms.Label
             Dim LBL_PERIOD As System.Windows.Forms.Label
-            Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(RedditViewSettingsForm))
             Me.OPT_VIEW_MODE_NEW = New System.Windows.Forms.RadioButton()
             Me.OPT_VIEW_MODE_HOT = New System.Windows.Forms.RadioButton()
             Me.OPT_VIEW_MODE_TOP = New System.Windows.Forms.RadioButton()
@@ -266,7 +265,7 @@ Namespace API.Reddit
             Me.ClientSize = New System.Drawing.Size(477, 112)
             Me.Controls.Add(CONTAINER_MAIN)
             Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
-            Me.Icon = CType(resources.GetObject("$this.Icon"), System.Drawing.Icon)
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.RedditIcon_128
             Me.KeyPreview = True
             Me.MaximizeBox = False
             Me.MaximumSize = New System.Drawing.Size(493, 151)

SCrawler/API/Reddit/RedditViewSettingsForm.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -68,16 +68,7 @@ Namespace API.Reddit
             End With
             MyDefs.CloseForm()
         End Sub
-        Private Sub OPT_VIEW_MODE_NEW_CheckedChanged(sender As Object, e As EventArgs) Handles OPT_VIEW_MODE_NEW.CheckedChanged
-            ChangePeriodEnabled()
-        End Sub
-        Private Sub OPT_VIEW_MODE_HOT_CheckedChanged(sender As Object, e As EventArgs) Handles OPT_VIEW_MODE_HOT.CheckedChanged
-            ChangePeriodEnabled()
-        End Sub
-        Private Sub OPT_VIEW_MODE_TOP_CheckedChanged(sender As Object, e As EventArgs) Handles OPT_VIEW_MODE_TOP.CheckedChanged
-            ChangePeriodEnabled()
-        End Sub
-        Private Sub ChangePeriodEnabled()
+        Private Sub ChangePeriodEnabled() Handles OPT_VIEW_MODE_NEW.CheckedChanged, OPT_VIEW_MODE_HOT.CheckedChanged, OPT_VIEW_MODE_TOP.CheckedChanged
             TP_PERIOD.Enabled = OPT_VIEW_MODE_TOP.Checked
         End Sub
     End Class

SCrawler/API/Reddit/SiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -9,47 +9,38 @@
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Tools.WEB
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports DownDetector = SCrawler.API.Base.DownDetector
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Reddit
-    <Manifest(RedditSiteKey), UseClassAsIs, SavedPosts, SpecialForm(False)>
+    <Manifest(RedditSiteKey), SavedPosts, SpecialForm(False)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
         Friend Overrides ReadOnly Property Icon As Icon
             Get
-                Return My.Resources.RedditIcon
+                Return My.Resources.SiteResources.RedditIcon_128
             End Get
         End Property
         Friend Overrides ReadOnly Property Image As Image
             Get
-                Return My.Resources.RedditPic512
+                Return My.Resources.SiteResources.RedditPic_512
             End Get
         End Property
-        <PropertyOption(ControlText:="Saved posts user"), PXML("SavedPostsUserName")>
+        <PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
         Friend ReadOnly Property SavedPostsUserName As PropertyValue
         <PropertyOption(ControlText:="Use M3U8", ControlToolTip:="Use M3U8 or mp4 for Reddit videos"), PXML>
         Friend ReadOnly Property UseM3U8 As PropertyValue
-        Friend Overrides ReadOnly Property Responser As Response
         Friend Sub New()
-            MyBase.New(RedditSite)
-            Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
-
+            MyBase.New(RedditSite, "reddit.com")
             With Responser
-                If .File.Exists Then
-                    .LoadSettings()
-                Else
-                    .CookiesDomain = "reddit.com"
-                    .Cookies = New CookieKeeper(.CookiesDomain)
-                    .Decoders.Add(SymbolsConverter.Converters.Unicode)
-                    .SaveSettings()
-                End If
+                If .Decoders.Count = 0 OrElse Not .Decoders.Contains(SymbolsConverter.Converters.Unicode) Then _
+                   .Decoders.Add(SymbolsConverter.Converters.Unicode) : .SaveSettings()
             End With
             SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
             UseM3U8 = New PropertyValue(True)
             UrlPatternUser = "https://www.reddit.com/user/{0}/"
             UrlPatternChannel = "https://www.reddit.com/r/{0}/"
-            ImageVideoContains = "redgifs"
+            ImageVideoContains = "reddit.com"
+            UserRegex = RParams.DM("[htps:/]{7,8}.*?reddit.com/([user]{1,4})/([^/]+)", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
         End Sub
         Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
             Select Case What
@@ -65,19 +56,9 @@ Namespace API.Reddit
             End Select
             Return Nothing
         End Function
-        Private ReadOnly RedditRegEx1 As RParams = RParams.DMS("[htps:/]{7,8}.*?reddit.com/user/([^/]+)", 1)
-        Private ReadOnly RedditRegEx2 As RParams = RParams.DMS(".?u/([^/]+)", 1)
-        Private ReadOnly RedditChannelRegEx1 As RParams = RParams.DMS("[htps:/]{7,8}.*?reddit.com/r/([^/]+)", 1)
-        Private ReadOnly RedditChannelRegEx2 As RParams = RParams.DMS(".?r/([^/]+)", 1)
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
-            Dim s$
-            Dim c% = 0
-            For Each r As RParams In {RedditRegEx1, RedditRegEx2, RedditChannelRegEx1, RedditChannelRegEx2}
-                s = RegexReplace(UserURL, r)
-                If Not s.IsEmptyString Then Return New ExchangeOptions(Site, s, c > 1)
-                c += 1
-            Next
-            Return Nothing
+            Dim l As List(Of String) = RegexReplace(UserURL, UserRegex)
+            If l.ListExists(3) Then Return New ExchangeOptions(Site, l(2), l(1) = "r") Else Return Nothing
         End Function
         Friend Overrides Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
             Try
@@ -89,20 +70,32 @@ Namespace API.Reddit
                         If Silent Then
                             Return False
                         Else
-                            Return MsgBoxE({"Over the past hour, Reddit has received an average of " &
-                                            avg.NumToString(New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}) & " outage reports:" & vbCr &
-                                            dl.ListToString(vbCr) & vbCr & vbCr &
-                                            "Do you want to continue parsing Reddit data?", "There are outage reports on Reddit"}, vbYesNo) = vbYes
+                            If MsgBoxE({"Over the past hour, Reddit has received an average of " &
+                                        avg.NumToString(New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}) & " outage reports:" & vbCr &
+                                        dl.ListToString(vbCr) & vbCr & vbCr &
+                                        "Do you want to continue parsing Reddit data?", "There are outage reports on Reddit"}, vbYesNo) = vbYes Then
+                                UpdateRedGifsToken()
+                                Return True
+                            Else
+                                Return False
+                            End If
                         End If
                     End If
                 End If
+                UpdateRedGifsToken()
                 Return True
             Catch ex As Exception
                 Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Reddit.SiteSettings.Available]", True)
             End Try
         End Function
-        Friend Overrides Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
-            Return UserData.GetVideoInfo(URL, Responser)
+        Private Sub UpdateRedGifsToken()
+            DirectCast(Settings(RedGifs.RedGifsSiteKey).Source, RedGifs.SiteSettings).UpdateTokenIfRequired()
+        End Sub
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            Dim spf$ = String.Empty
+            Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
+            f = $"{f.PathWithSeparator}OptionalPath\"
+            Return UserData.GetVideoInfo(URL, Responser, f, spf)
         End Function
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
             If Options Is Nothing OrElse Not TypeOf Options Is RedditViewExchange Then Options = New RedditViewExchange
@@ -110,5 +103,8 @@ Namespace API.Reddit
                 Using f As New RedditViewSettingsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return $"https://www.reddit.com/comments/{Media.Post.ID.Split("_").LastOrDefault}/"
+        End Function
     End Class
 End Namespace

SCrawler/API/Reddit/UserData.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,15 +6,16 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.XML
-Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.ImageRenderer
-Imports PersonalUtilities.Tools.WEB
-Imports PersonalUtilities.Tools.WebDocuments.JSON
 Imports System.Net
 Imports System.Threading
 Imports SCrawler.API.Base
 Imports SCrawler.API.Reddit.RedditViewExchange
+Imports SCrawler.Plugin.Hosts
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.ImageRenderer
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports CView = SCrawler.API.Reddit.IRedditView.View
@@ -135,7 +136,7 @@ Namespace API.Reddit
             _CrossPosts.Clear()
             If Not IsSavedPosts AndAlso (IsChannel AndAlso Not ChannelInfo Is Nothing) Then
                 If Not Responser Is Nothing Then Responser.Dispose()
-                Responser = New Response
+                Responser = New Responser
                 Responser.Copy(MySiteSettings.Responser)
                 ChannelPostsNames.ListAddList(ChannelInfo.PostsAll.Select(Function(p) p.ID), LNC)
                 If Not ViewMode = CView.New Then ChannelPostsNames.ListAddList(ChannelInfo.PostsNames, LNC)
@@ -151,6 +152,8 @@ Namespace API.Reddit
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             _TotalPostsDownloaded = 0
             If IsSavedPosts Then
+                'TODO: Reddit saved posts: remove Unicode converter?
+                Responser.DecodersError = EDP.ReturnValue
                 DownloadDataChannel(String.Empty, Token)
             ElseIf IsChannel Then
                 If ChannelInfo Is Nothing Then
@@ -163,6 +166,8 @@ Namespace API.Reddit
                     End If
                     If DownloadTopCount.HasValue Then DownloadLimitCount = DownloadTopCount
                 End If
+                If SaveToCache AndAlso Not Responser.Decoders.Contains(SymbolsConverter.Converters.HTML) Then _
+                   Responser.Decoders.Add(SymbolsConverter.Converters.HTML)
                 DownloadDataChannel(String.Empty, Token)
                 If ChannelInfo Is Nothing Then _TempPostsList.ListAddList(_TempMediaList.Select(Function(m) m.Post.ID), LNC)
             Else
@@ -173,6 +178,8 @@ Namespace API.Reddit
 #Region "Download Functions (User, Channel)"
         Private _TotalPostsDownloaded As Integer = 0
         Private ReadOnly _CrossPosts As List(Of String)
+        Private Const SiteGfycatKey As String = "gfycat"
+        Private Const SiteRedGifsKey As String = "redgifs"
         Private Sub DownloadDataUser(ByVal POST As String, ByVal Token As CancellationToken)
             Const CPRI$ = "crosspostRootId"
             Const CPPI$ = "crosspostParentId"
@@ -227,12 +234,15 @@ Namespace API.Reddit
                                                 Continue For
                                             End If
                                             If nn.Contains("created") Then PostDate = nn("created").Value Else PostDate = String.Empty
-                                            If DownloadToDate.HasValue AndAlso Not CheckDatesLimit(PostDate, DateTrueProvider(IsChannel)) Then Exit Sub
+                                            Select Case CheckDatesLimit(PostDate, DateTrueProvider(IsChannel))
+                                                Case DateResult.Skip : Continue For
+                                                Case DateResult.Exit : Exit Sub
+                                            End Select
 
                                             _ItemsBefore = _TempMediaList.Count
                                             added = True
                                             s = nn.ItemF({"source", "url"})
-                                            If s.XmlIfNothingValue("/").StringContains({"redgifs.com", "gfycat.com"}) Then
+                                            If s.XmlIfNothingValue("/").StringContains({$"{SiteRedGifsKey}.com", $"{SiteGfycatKey}.com"}) Then
                                                 _TempMediaList.ListAddValue(MediaFromData(UTypes.VideoPre, s.Value, _PostID(), PostDate,, IsChannel), LNC)
                                             ElseIf Not CreateImgurMedia(s.XmlIfNothingValue, _PostID(), PostDate,, IsChannel) Then
                                                 s = nn.ItemF({"media"}).XmlIfNothing
@@ -264,7 +274,7 @@ Namespace API.Reddit
                                                 If Not s.IsEmptyString AndAlso TryFile(s.Value) Then
                                                     With s.Value.ToLower
                                                         Select Case True
-                                                            Case .Contains("redgifs"), .Contains("gfycat") : tmpType = UTypes.VideoPre
+                                                            Case .Contains(SiteRedGifsKey), .Contains(SiteGfycatKey) : tmpType = UTypes.VideoPre
                                                             Case .Contains("m3u8") : If Settings.UseM3U8 Then tmpType = UTypes.m3u8
                                                             Case .Contains(".gif") And TryFile(s.Value) : tmpType = UTypes.GIF
                                                             Case TryFile(s.Value) : tmpType = UTypes.Picture
@@ -322,7 +332,7 @@ Namespace API.Reddit
 
                                         If ChannelPostsNames.Contains(PostID) Then
                                             If ViewMode = CView.New Then ExistsDetected = True Else NewPostDetected = True 'bypass
-                                            Continue For 'Exit Sub
+                                            Continue For
                                         End If
                                         If DownloadLimitCount.HasValue AndAlso _TotalPostsDownloaded >= DownloadLimitCount.Value Then Exit Sub
                                         If Not DownloadLimitPost.IsEmptyString AndAlso DownloadLimitPost = PostID Then Exit Sub
@@ -370,16 +380,15 @@ Namespace API.Reddit
                                         ElseIf Not s.Value({"media", "reddit_video"}, "fallback_url").IsEmptyString Then
                                             tmpUrl = s.Value({"media", "reddit_video"}, "fallback_url")
                                             If SaveToCache Then
-                                                tmpUrl = s.Value("thumbnail")
+                                                tmpUrl = GetVideoRedditPreview(s)
                                                 If Not tmpUrl.IsEmptyString Then
-                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, tmpUrl, PostID, PostDate, _UserID, IsChannel), LNC)
+                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, tmpUrl, PostID, PostDate, _UserID, IsChannel, False), LNC)
                                                     _TotalPostsDownloaded += 1
                                                 End If
                                             ElseIf UseM3U8 AndAlso Not s.Value({"media", "reddit_video"}, "hls_url").IsEmptyString Then
                                                 _TempMediaList.ListAddValue(MediaFromData(UTypes.m3u8, s.Value({"media", "reddit_video"}, "hls_url"),
                                                                                           PostID, PostDate, _UserID, IsChannel), LNC)
                                             Else
-                                                '_TempMediaList.ListAddValue(MediaFromData(UTypes.VideoPre + UTypes.m3u8, tmpUrl, PostID, PostDate, _UserID, IsChannel), LNC)
                                                 _TempMediaList.ListAddValue(MediaFromData(UTypes.Video, tmpUrl, PostID, PostDate, _UserID, IsChannel), LNC)
                                                 _TotalPostsDownloaded += 1
                                             End If
@@ -409,19 +418,6 @@ Namespace API.Reddit
         End Sub
 #End Region
 #Region "Download Base Functions"
-        Private Function ImgurPicture(ByVal Source As EContainer, ByVal Value As String) As String
-            Try
-                Dim e As EContainer = Source({"source", "url"}).XmlIfNothing
-                If Not e.IsEmptyString AndAlso e.Value.ToLower.Contains("imgur") Then
-                    Return e.Value
-                Else
-                    Return Value
-                End If
-            Catch ex As Exception
-                LogError(ex, "[ImgurPicture]")
-                Return Value
-            End Try
-        End Function
         Private Function CreateImgurMedia(ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
                                           Optional ByVal _UserID As String = "", Optional ByVal IsChannel As Boolean = False) As Boolean
             If Not _URL.IsEmptyString AndAlso _URL.Contains("imgur") Then
@@ -440,8 +436,33 @@ Namespace API.Reddit
                 ElseIf _URL.Contains(".gif") Then
                     _TempMediaList.ListAddValue(MediaFromData(UTypes.GIF, _URL, PostID, PostDate, _UserID, IsChannel), LNC)
                 Else
-                    If Not TryFile(_URL) Then _URL &= ".jpg"
-                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, _URL, PostID, PostDate, _UserID, IsChannel), LNC)
+                    Dim obj As IEnumerable(Of UserMedia) = Imgur.Envir.GetVideoInfo(_URL, EDP.ReturnValue)
+                    If Not obj.ListExists Then
+                        If Not TryFile(_URL) Then _URL &= ".jpg"
+                        _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, _URL, PostID, PostDate, _UserID, IsChannel), LNC)
+                    Else
+                        Dim ut As UTypes
+                        Dim m As UserMedia
+                        For Each data As UserMedia In obj
+                            With data
+                                If Not .URL.IsEmptyString Then
+                                    If Not .File.IsEmptyString Then
+                                        Select Case .File.Extension
+                                            Case "jpg", "png", "jpeg" : ut = UTypes.Picture
+                                            Case "gifv" : ut = IIf(SaveToCache, UTypes.Picture, UTypes.Video)
+                                            Case "mp4" : ut = UTypes.Video
+                                            Case "gif" : ut = UTypes.GIF
+                                            Case Else : ut = UTypes.Picture : .File.Extension = "jpg"
+                                        End Select
+                                        m = MediaFromData(ut, _URL, PostID, PostDate, _UserID, IsChannel)
+                                        m.URL = .URL
+                                        m.File = .File.File
+                                        _TempMediaList.ListAddValue(m, LNC)
+                                    End If
+                                End If
+                            End With
+                        Next
+                    End If
                 End If
                 Return True
             Else
@@ -470,21 +491,70 @@ Namespace API.Reddit
                 Return False
             End Try
         End Function
+        Private Function GetVideoRedditPreview(ByVal Node As EContainer) As String
+            Try
+                If Not Node Is Nothing Then
+                    Dim n As EContainer = Node.ItemF({"preview", "images", 0})
+                    Dim DestNode$() = Nothing
+                    If If(n?.Count, 0) > 0 Then
+                        If If(n("resolutions")?.Count, 0) > 0 Then
+                            DestNode = {"resolutions"}
+                        ElseIf If(n({"variants", "nsfw", "resolutions"})?.Count, 0) > 0 Then
+                            DestNode = {"variants", "nsfw", "resolutions"}
+                        End If
+                        If Not DestNode Is Nothing Then
+                            With n(DestNode)
+                                Dim sl As List(Of Sizes) = .Select(Function(e) New Sizes(e.Value("width"), e.Value("url"))).
+                                                            ListWithRemove(Function(ss) ss.HasError Or ss.Data.IsEmptyString)
+                                If sl.ListExists Then
+                                    Dim s As Sizes
+                                    sl.Sort()
+                                    s = sl.First
+                                    sl.Clear()
+                                    Return s.Data
+                                End If
+                            End With
+                        End If
+                    End If
+                End If
+                Return String.Empty
+            Catch ex As Exception
+                ProcessException(ex, Nothing, "reddit video preview parsing error", False)
+                Return String.Empty
+            End Try
+        End Function
         Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            Dim RedGifsResponser As Responser = Nothing
             Try
                 ThrowAny(Token)
                 Const v2 As UTypes = UTypes.VideoPre + UTypes.m3u8
                 If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(p) p.Type = UTypes.VideoPre Or p.Type = v2) Then
                     Dim r$, v$
                     Dim e As New ErrorsDescriber(EDP.ReturnValue)
-                    Dim m As UserMedia
+                    Dim m As UserMedia, m2 As UserMedia
+                    Dim RedGifsHost As SettingsHost = Settings(RedGifs.RedGifsSiteKey)
+                    RedGifsResponser = RedGifsHost.Responser.Copy
                     For i% = _TempMediaList.Count - 1 To 0 Step -1
                         ThrowAny(Token)
                         If _TempMediaList(i).Type = UTypes.VideoPre Or _TempMediaList(i).Type = v2 Then
                             m = _TempMediaList(i)
                             If _TempMediaList(i).Type = UTypes.VideoPre Then
-                                If m.URL.Contains("gfycat.com") Then
+                                If m.URL.Contains($"{SiteGfycatKey}.com") Then
                                     r = Gfycat.Envir.GetVideo(m.URL)
+                                ElseIf m.URL.Contains(SiteRedGifsKey) Then
+                                    m2 = RedGifs.UserData.GetDataFromUrlId(m.URL, False, RedGifsResponser, RedGifsHost)
+                                    If m2.State = UStates.Missing Then
+                                        m.State = UStates.Missing
+                                        _ContentList.Add(m)
+                                        _TempMediaList.RemoveAt(i)
+                                    ElseIf m2.State = RedGifs.UserData.DataGone Then
+                                        _TempMediaList.RemoveAt(i)
+                                    Else
+                                        m2.URL_BASE = m.URL
+                                        m2.Post = m.Post
+                                        _TempMediaList(i) = m2
+                                    End If
+                                    Continue For
                                 Else
                                     r = Responser.GetResponse(m.URL,, e)
                                 End If
@@ -505,33 +575,126 @@ Namespace API.Reddit
                 End If
             Catch ex As Exception
                 ProcessException(ex, Token, "video reparsing error", False)
+            Finally
+                If Not RedGifsResponser Is Nothing Then RedGifsResponser.Dispose()
             End Try
         End Sub
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Response) As IEnumerable(Of UserMedia)
+        Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
+            Dim rList As New List(Of Integer)
+            Dim RedGifsResponser As Responser = Nothing
             Try
-                If Not URL.IsEmptyString AndAlso URL.Contains("redgifs") Then
+                If Not ChannelInfo Is Nothing Or SaveToCache Then Exit Sub
+                If ContentMissingExists Then
+                    Dim RedGifsHost As SettingsHost = Settings(RedGifs.RedGifsSiteKey)
+                    RedGifsResponser = RedGifsHost.Responser.Copy
+                    Dim m As UserMedia, m2 As UserMedia
+                    For i% = 0 To _ContentList.Count - 1
+                        m = _ContentList(i)
+                        If m.State = UStates.Missing AndAlso Not m.Post.ID.IsEmptyString Then
+                            ThrowAny(Token)
+                            If Not m.URL.IsEmptyString AndAlso m.URL.Contains(SiteRedGifsKey) Then
+                                m2 = RedGifs.UserData.GetDataFromUrlId(m.URL, False, RedGifsResponser, RedGifsHost)
+                                If m2.State = RedGifs.UserData.DataGone Then
+                                    rList.Add(i)
+                                ElseIf Not m2.Type = UTypes.Undefined And Not m2.State = UStates.Missing Then
+                                    m.Type = m2.Type
+                                    m.File = m2.File
+                                    m.URL_BASE = m.URL
+                                    m.URL = m2.URL
+                                    rList.Add(i)
+                                    _TempMediaList.ListAddValue(m, LNC)
+                                End If
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "missing data downloading error")
+            Finally
+                If Not RedGifsResponser Is Nothing Then RedGifsResponser.Dispose()
+                If rList.Count > 0 Then
+                    For i% = rList.Count - 1 To 0 Step -1 : _ContentList.RemoveAt(rList(i)) : Next
+                    rList.Clear()
+                End If
+            End Try
+        End Sub
+        Private Sub ParsePost(ByVal URL As String)
+            Try
+                If Not URL.IsEmptyString Then
+                    Dim __id$ = RegexReplace(URL, RParams.DMS("comments/([^/]+)", 1, EDP.ReturnValue))
+                    If Not __id.IsEmptyString Then
+                        URL = $"https://www.reddit.com/comments/{__id.Split("_").LastOrDefault}/.json"
+                        Dim r$ = Responser.GetResponse(URL,, EDP.ReturnValue)
+                        If Not r.IsEmptyString Then
+                            Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
+                                With j.ItemF({0, "data", "children", 0, "data"})
+                                    If .ListExists Then
+                                        If .Contains({"media"}, "reddit_video") Then
+                                            With .Item({"media"}, "reddit_video")
+                                                If UseM3U8 AndAlso .Item("hls_url").XmlIfNothingValue("/").ToLower.Contains("m3u8") Then
+                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.m3u8, .Value("hls_url"), __id, String.Empty), LNC)
+                                                ElseIf Not UseM3U8 AndAlso .Item("fallback_url").XmlIfNothingValue("/").ToLower.Contains("mp4") Then
+                                                    _TempMediaList.ListAddValue(MediaFromData(UTypes.Video, .Value("fallback_url"), __id, String.Empty), LNC)
+                                                End If
+                                            End With
+                                        ElseIf Not .Value("url").IsEmptyString Then
+                                            If .Value("url").StringContains({$"{SiteRedGifsKey}.com", $"{SiteGfycatKey}.com"}) Then
+                                                _TempMediaList.ListAddValue(MediaFromData(UTypes.VideoPre, .Value("url"), __id, String.Empty), LNC)
+                                            Else
+                                                CreateImgurMedia(.Value("url"), __id, String.Empty)
+                                            End If
+                                        End If
+                                    End If
+                                End With
+                            End Using
+                        End If
+                    End If
+                End If
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"API.Reddit.ParsePost({URL})")
+            End Try
+        End Sub
+        Private Class AbsProgress : Inherits PersonalUtilities.Forms.Toolbars.MyProgress
+            Public Overrides Sub Perform(Optional ByVal Value As Double = 1)
+            End Sub
+        End Class
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Responser, ByVal f As SFile, ByVal SpecialFolder As String) As IEnumerable(Of UserMedia)
+            Try
+                If Not URL.IsEmptyString Then
                     Using r As New UserData
-                        r._TempMediaList.Add(MediaFromData(UTypes.VideoPre, URL, String.Empty, String.Empty,, False))
-                        r.Responser = New Response
+                        r.SetEnvironment(Settings(RedditSiteKey), Nothing, False, False)
+                        r.Responser = New Responser
                         r.Responser.Copy(resp)
-                        r.ReparseVideo(Nothing)
-                        If r._TempMediaList.ListExists Then Return {r._TempMediaList(0)}
+                        r.ParsePost(URL)
+                        If r._TempMediaList.Count > 0 Then
+                            r.ReparseVideo(Nothing)
+                            If r._TempMediaList.Count > 0 Then
+                                r._ContentNew.AddRange(r._TempMediaList)
+                                r.Progress = New AbsProgress
+                                r.User.File.Path = f.Path
+                                r.SeparateVideoFolder = False
+                                r.DownloadContent(Nothing)
+                                If r._ContentNew.Exists(Function(c) c.State = UStates.Downloaded) Then _
+                                   Return {New UserMedia With {.State = UStates.Downloaded, .SpecialFolder = SpecialFolder}}
+                            End If
+                        End If
                     End Using
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, "Video searching error")
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Reddit.UserData.GetVideoInfo({URL})]")
             End Try
         End Function
 #End Region
 #Region "Structure creator"
         Protected Shared Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
-                                                Optional ByVal _UserID As String = "", Optional ByVal IsChannel As Boolean = False) As UserMedia
+                                                Optional ByVal _UserID As String = "", Optional ByVal IsChannel As Boolean = False,
+                                                Optional ByVal ReplacePreview As Boolean = True) As UserMedia
             If _URL.IsEmptyString And t = UTypes.Picture Then Return Nothing
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
             Dim m As New UserMedia(_URL, t) With {.Post = New UserPost With {.ID = PostID, .UserID = _UserID}}
             If t = UTypes.Picture Or t = UTypes.GIF Then m.File = UrlToFile(m.URL) Else m.File = Nothing
-            If m.URL.Contains("preview") Then m.URL = $"https://i.redd.it/{m.File.File}"
+            If ReplacePreview And m.URL.Contains("preview") Then m.URL = $"https://i.redd.it/{m.File.File}"
             If Not PostDate.IsEmptyString Then m.Post.Date = AConvert(Of Date)(PostDate, DateTrueProvider(IsChannel), Nothing) Else m.Post.Date = Nothing
             Return m
         End Function
@@ -551,6 +714,7 @@ Namespace API.Reddit
         End Function
 #End Region
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            Dim RedGifsResponser As Responser = Nothing
             Try
                 Const _RFN$ = "RedditVideo"
                 Const RFN$ = _RFN & "{0}"
@@ -560,7 +724,10 @@ Namespace API.Reddit
                 If _ContentNew.Count > 0 Then
                     _ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
                     If _ContentNew.Count > 0 Then
+                        RedGifsResponser = Settings(RedGifs.RedGifsSiteKey).Responser.Copy
                         MyFile.Exists(SFO.Path)
+                        Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
+                        Dim IsImgurStuff As Boolean
                         Dim MyDir$
                         If Not IsSavedPosts AndAlso (IsChannel And SaveToCache And Not ChannelInfo Is Nothing) Then
                             MyDir = ChannelInfo.CachePath.PathNoSeparator
@@ -579,6 +746,7 @@ Namespace API.Reddit
                         Dim vsf As Boolean = SeparateVideoFolderF
                         Dim UseMD5 As Boolean = Not IsChannel Or (Not cached And Settings.ChannelsRegularCheckMD5)
                         Dim bDP As New ErrorsDescriber(EDP.None)
+                        Dim RGRERROR As New ErrorsDescriber(EDP.ThrowException)
                         Dim ImgurUrls As New List(Of String)
                         Dim TryBytes As Func(Of String, Imaging.ImageFormat, String) =
                             Function(ByVal __URL As String, ByVal ImgFormat As Imaging.ImageFormat) As String
@@ -628,8 +796,8 @@ Namespace API.Reddit
                                                                        End Function
                         Dim m$
                         Using w As New WebClient
-                            If vsf Then SFileShares.SFileExists($"{MyDir}\Video\", SFO.Path)
-                            Progress.TotalCount += _ContentNew.Count
+                            If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
+                            Progress.Maximum += _ContentNew.Count
                             For i = 0 To _ContentNew.Count - 1
                                 ThrowAny(Token)
                                 v = _ContentNew(i)
@@ -651,6 +819,7 @@ Namespace API.Reddit
 
                                 If (Not m.IsEmptyString AndAlso Not HashList.Contains(m)) Or Not (v.Type = UTypes.Picture Or
                                                                                                   v.Type = UTypes.GIF) Or Not UseMD5 Or ImgurUrls.Count > 0 Then
+                                    IsImgurStuff = ImgurUrls.Count > 0
                                     Do
                                         If Not cached And Not m.IsEmptyString Then HashList.Add(m)
                                         v.MD5 = m
@@ -677,12 +846,14 @@ Namespace API.Reddit
                                                 f = M3U8.Download(v.URL, f)
                                             ElseIf ImgurUrls.Count > 0 Then
                                                 w.DownloadFile(ImgurUrls(0), f.ToString)
+                                            ElseIf v.URL.Contains(SiteRedGifsKey) Then
+                                                RedGifsResponser.DownloadFile(v.URL, f, RGRERROR)
                                             Else
                                                 w.DownloadFile(v.URL, f.ToString)
                                             End If
                                             If Not v.Type = UTypes.m3u8 Or Not f.IsEmptyString Then
                                                 Select Case v.Type
-                                                    Case UTypes.Picture : DownloadedPictures(False) += 1
+                                                    Case UTypes.Picture, UTypes.GIF : DownloadedPictures(False) += 1
                                                     Case UTypes.Video, UTypes.m3u8 : DownloadedVideos(False) += 1
                                                 End Select
                                                 If Not IsChannel Or Not SaveToCache Then
@@ -695,7 +866,11 @@ Namespace API.Reddit
                                                 dCount += 1
                                             End If
                                         Catch wex As Exception
-                                            If Not IsChannel Then ErrorDownloading(f, v.URL)
+                                            If Not IsChannel Then
+                                                If Not IsImgurStuff And MissingErrorsAdd Then ErrorDownloading(f, v.URL)
+                                                v.Attempts += 1
+                                                v.State = UStates.Missing
+                                            End If
                                         End Try
                                         If ImgurUrls.Count > 0 Then ImgurUrls.RemoveAt(0)
                                     Loop While ImgurUrls.Count > 0
@@ -723,20 +898,22 @@ Namespace API.Reddit
                 HasError = True
             End Try
         End Sub
-        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False) As Integer
-            If Responser.StatusCode = HttpStatusCode.NotFound Then
-                UserExists = False
-            ElseIf Responser.StatusCode = HttpStatusCode.Forbidden Then
-                UserSuspended = True
-            ElseIf Responser.StatusCode = HttpStatusCode.BadGateway Or
-                   Responser.StatusCode = HttpStatusCode.ServiceUnavailable Then
-                MyMainLOG = $"[{CInt(Responser.StatusCode)}] Reddit is currently unavailable ({ToString()})"
-            ElseIf Responser.StatusCode = HttpStatusCode.GatewayTimeout Then
-                Return 1
-            Else
-                If Not FromPE Then LogError(ex, Message) : HasError = True
-                Return 0
-            End If
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            With Responser
+                If .StatusCode = HttpStatusCode.NotFound Then
+                    UserExists = False
+                ElseIf .StatusCode = HttpStatusCode.Forbidden Then
+                    UserSuspended = True
+                ElseIf .StatusCode = HttpStatusCode.BadGateway Or .StatusCode = HttpStatusCode.ServiceUnavailable Then
+                    MyMainLOG = $"[{CInt(Responser.StatusCode)}] Reddit is currently unavailable ({ToString()})"
+                ElseIf .StatusCode = HttpStatusCode.GatewayTimeout Then
+                    Return 1
+                Else
+                    If Not FromPE Then LogError(ex, Message) : HasError = True
+                    Return 0
+                End If
+            End With
             Return 1
         End Function
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)

SCrawler/API/Redgifs/Declarations.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,9 +6,14 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.RedGifs
     Friend Module Declarations
+        Friend Const RedGifsSiteKey As String = "AndyProgram_RedGifs"
         Friend Const RedGifsSite As String = "RedGifs"
         Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v, n, e))
+        Friend ReadOnly WatchIDRegex As RParams = RParams.DMS(".+?watch/([^\?&""/]+)", 1, EDP.ReturnValue)
+        Friend ReadOnly ThumbsIDRegex As RParams = RParams.DMS("([^/\?&""]+?)(-\w+?|)\.(mp4|jpg)", 1, EDP.ReturnValue,
+                                                               CType(Function(Input$) Input.StringToLower.StringTrim, Func(Of String, String)))
     End Module
 End Namespace

SCrawler/API/Redgifs/SiteSettings.vb

@@ -1,4 +1,4 @@
-' Copyright (C) 2022  Andy
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
 ' This program is free software: you can redistribute it and/or modify
 ' it under the terms of the GNU General Public License as published by
 ' the Free Software Foundation, either version 3 of the License, or
@@ -6,24 +6,172 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports SCrawler.API.Base
+Imports PersonalUtilities.Forms
+Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Imports UStates = SCrawler.API.Base.UserMedia.States
 Namespace API.RedGifs
-    <Manifest("AndyProgram_RedGifs"), UseClassAsIs>
+    <Manifest(RedGifsSiteKey)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.RedGifsIcon_32
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.RedGifsPic_32
+            End Get
+        End Property
+        <PropertyOption(ControlToolTip:="Bearer token", AllowNull:=False), ControlNumber(1)>
+        Friend ReadOnly Property Token As PropertyValue
+        <PXML> Friend ReadOnly Property TokenLastDateUpdated As PropertyValue
+        Private Const TokenName As String = "authorization"
+#Region "TokenUpdateInterval"
+        <PropertyOption(ControlText:="Token refresh interval", ControlToolTip:="Interval (in minutes) to refresh the token", AllowNull:=False, LeftOffset:=120),
+            PXML, ControlNumber(0)>
+        Friend ReadOnly Property TokenUpdateInterval As PropertyValue
+        Private Class TokenIntervalProvider : Implements IFieldsCheckerProvider
+            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
+            Private Property Name As String Implements IFieldsCheckerProvider.Name
+            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
+            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+                TypeError = False
+                ErrorMessage = String.Empty
+                If Not ACheck(Of Integer)(Value) Then
+                    TypeError = True
+                ElseIf CInt(Value) > 0 Then
+                    Return Value
+                Else
+                    ErrorMessage = $"The value of [{Name}] field must be greater than or equal to 1"
+                End If
+                Return Nothing
+            End Function
+            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
+                Throw New NotImplementedException("[GetFormat] is not available in the context of [TokenIntervalProvider]")
+            End Function
+        End Class
+        <Provider(NameOf(TokenUpdateInterval), FieldsChecker:=True)>
+        Private ReadOnly Property TokenUpdateIntervalProvider As IFormatProvider
+#End Region
+#End Region
+#Region "Initializer"
         Friend Sub New()
             MyBase.New(RedGifsSite, "redgifs.com")
+            Dim t$ = String.Empty
+            With Responser
+                Dim b As Boolean = Not .Mode = Responser.Modes.WebClient
+                .Mode = Responser.Modes.WebClient
+                t = .Headers.Value(TokenName)
+                If b Then .SaveSettings()
+            End With
+            Token = New PropertyValue(t, GetType(String), Sub(v) UpdateResponse(v))
+            TokenLastDateUpdated = New PropertyValue(Now.AddYears(-1), GetType(Date))
+            TokenUpdateInterval = New PropertyValue(60 * 12, GetType(Integer))
+            TokenUpdateIntervalProvider = New TokenIntervalProvider
             UrlPatternUser = "https://www.redgifs.com/users/{0}/"
             UserRegex = RParams.DMS("[htps:/]{7,8}.*?redgifs.com/users/([^/]+)", 1)
             ImageVideoContains = "redgifs"
         End Sub
+#End Region
+#Region "Response updater"
+        Private Sub UpdateResponse(ByVal Value As String)
+            Responser.Headers.Add(TokenName, Value)
+            Responser.SaveSettings()
+        End Sub
+#End Region
+#Region "Token updaters"
+        Friend Function UpdateTokenIfRequired() As Boolean
+            Dim d As Date? = AConvert(Of Date)(TokenLastDateUpdated.Value, AModes.Var, Nothing)
+            If Not d.HasValue OrElse d.Value < Now.AddMinutes(-CInt(TokenUpdateInterval.Value)) Then
+                Return UpdateToken()
+            Else
+                Return True
+            End If
+        End Function
+        <PropertyUpdater(NameOf(Token))>
+        Friend Function UpdateToken() As Boolean
+            Try
+                Dim r$
+                Dim NewToken$ = String.Empty
+                Using resp As New Responser : r = resp.GetResponse("https://api.redgifs.com/v2/auth/temporary",, EDP.ThrowException) : End Using
+                If Not r.IsEmptyString Then
+                    Dim j As EContainer = JsonDocument.Parse(r)
+                    If Not j Is Nothing Then
+                        NewToken = j.Value("token")
+                        j.Dispose()
+                    End If
+                End If
+                If Not NewToken.IsEmptyString Then
+                    Token.Value = $"Bearer {NewToken}"
+                    TokenLastDateUpdated.Value = Now
+                    Return True
+                Else
+                    Return False
+                End If
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.RedGifs.SiteSettings.UpdateToken]", False)
+            End Try
+        End Function
+#End Region
+#Region "Update settings"
+        Private _LastTokenValue As String = String.Empty
+        Friend Overrides Sub BeginEdit()
+            _LastTokenValue = AConvert(Of String)(Token.Value, AModes.Var, String.Empty)
+            MyBase.BeginEdit()
+        End Sub
+        Friend Overrides Sub Update()
+            Dim NewToken$ = AConvert(Of String)(Token.Value, AModes.Var, String.Empty)
+            If Not _LastTokenValue = NewToken Then TokenLastDateUpdated.Value = Now
+            MyBase.Update()
+        End Sub
+        Friend Overrides Sub EndEdit()
+            _LastTokenValue = String.Empty
+            MyBase.EndEdit()
+        End Sub
+#End Region
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
             Return New UserData
         End Function
-        Friend Overrides Function GetSpecialDataF(ByVal URL As String) As IEnumerable(Of UserMedia)
-            Return Reddit.UserData.GetVideoInfo(URL, Nothing)
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            If BaseAuthExists() Then
+                Using resp As Responser = Responser.Copy
+                    Dim m As UserMedia = UserData.GetDataFromUrlId(URL, False, resp, Settings(RedGifsSiteKey))
+                    If Not m.State = UStates.Missing And Not m.State = UserData.DataGone And (m.Type = UTypes.Picture Or m.Type = UTypes.Video) Then
+                        Try
+                            Dim spf$ = String.Empty
+                            Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
+                            If f.IsEmptyString Then
+                                f = m.File.File
+                            Else
+                                f.Name = m.File.Name
+                                f.Extension = m.File.Extension
+                            End If
+                            resp.DownloadFile(m.URL, f, EDP.ThrowException)
+                            m.State = UStates.Downloaded
+                            m.SpecialFolder = spf
+                            Return {m}
+                        Catch ex As Exception
+                            ErrorsDescriber.Execute(EDP.SendInLog, ex, $"Redgifs standalone download error: [{URL}]")
+                        End Try
+                    End If
+                End Using
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return $"https://www.redgifs.com/watch/{Media.Post.ID}"
+        End Function
+        Friend Overrides Function BaseAuthExists() As Boolean
+            Return UpdateTokenIfRequired() AndAlso ACheck(Token.Value)
         End Function
     End Class
 End Namespace