2022.12.26.0

UserMedia: fixed plugin bugs
Instagram: updated algo and settings; update responser settings based on site response
PornHub: fixed bug in SiteSettings; fixed typos
RedGis: fixed downloading user profiles
XVideos: fixed user profile opening
UserDataBind: fixed multiple collection removing issue
DownloadedInfoForm: fixed user focusing
UserCreatorForm: add user name to form header if user exists
ListImagesLoader: changed loading algo
MainFrame: added channels button to tray context menu
Added ffmpeg fox x86
Fixed typos

.gitignore

@@ -34,6 +34,7 @@ bld/
 [Ll]og/
 [Ll]ogs/
 ffmpeg/
+cURL/
 Info/
 Hidden/
 

Changelog.md

@@ -1,3 +1,60 @@
+# 2022.12.26.0
+
+*2022-12-26*
+
+**ATTENTION!**
+
+**Instagram requirements changed. Headers and cookies are now required to download Timeline, Stories and Saved posts; hash to download tagged posts. Please update your credentials.**
+
+**Instagram tagged posts no longer provide the total amount of tagged posts. I've corrected the tagged posts notification, but now I can't tell how many requests will be spent on downloading tagged posts. And from now on, one request will be spent on downloading each tagged post, because Instagram doesn't provide complete information about the tagged post with the site's response. In this case, if the number of tagged posts is 1000, 1000 requests will be spent. Be careful when downloading them. I highly recommend that you forcefully disable the downloading of tagged posts for a while.**
+
+- Added
+  - Updated user loading algorithm
+  - Channels button to tray context menu
+  - (Request #96) Add FFmpeg to x86 version
+- Fixed
+  - PornHub wrong behavior when downloading images
+  - Unable open XVideos user profile
+  - Cannot delete multiple collections at once
+  - Can't focus user from the download info form
+  - Instagram downloader not working
+  - (Issue #69) **RedGifs data is not downloading**. Again.
+  - Minor bugs
+
+# 2022.11.16.0
+
+*2022-11-16*
+
+**ATTENTION! This version makes changes to the base SCrawler user configuration file. Since you started using this version, you still can downgrade. BUT! Once you add a virtual collection or a virtual user to a collection, you won't be able to downgrade without losing data.**
+
+- Added
+  - **PornHub**
+  - **XHamster**
+  - An ability to download saved XVIDEOS posts
+  - Download indicator. While downloading, the rainbow tray icon changed to a blue arrow.
+  - Collections: the ability to edit a collection using a form
+  - Collections: the ability to create a **`virtual collection`** and add a **`virtual user`** to a real collection
+  - Collections: an easier way to added users to a collection
+  - Collections: an easier way to create collections
+  - Added icons for channels form context menu buttons
+  - More convenient change of user labels from the context menu of the user list
+  - Notifications: complete transition from default notifications to ToastNotifications
+  - Notifications: when you click on the notification that some of the channels are downloaded, the channels form opens
+  - Notifications: when you click on the notification that all users are downloaded, the main window form opens
+  - Notifications: when you click on the notification that the saved posts are downloaded, the saved posts form opens
+  - Import users
+  - Minor improvements
+- Plugins
+  - Added
+    - `TaskGroup` attribute
+    - `IUserMedia` interface
+  - Changed
+    - `GetUserUrl` and `GetUserPostUrl` functions: `String UserName` and `String UserID` changed to ` IPluginContentProvider User`
+- Fixed
+  - Collections editor: new added collections are still not added to the top of the collections list
+  - Users search form doesn't remember last size
+  - Minor bugs
+
 # 2022.10.23.0
 
 *2022-10-23*

HowToSupport.md

@@ -1,7 +1,7 @@
 Your support is very valuable to me. Any support is greatly appreciated. Your support encourages me to make new features, update the program, add new sites, etc.
 
 You can support the program by:
-  - **Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
+  - **Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
   - :heavy_dollar_sign: make a donation on this site: https://ko-fi.com/andyprogram
   - :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
   - :speech_balloon: tell your friends about the program

ProgramsComparison.md

@@ -68,7 +68,7 @@ https://github.com/RipMeApp/ripme
 | **Free options** | The program is completely free | The program is completely free, but site limits are not declared |
 | Operating Systems | Windows 10+ | Windows, MacOS, Linux |
 | Select want content type to download | Yes | Yes |
-| Suported sites | 6 internal and any site using plugins | 86+ sites (declared) |
+| Suported sites | 9 internal and any site using plugins | 86+ sites (declared) |
 | Other sites support | **Yes** | No |
 | Still supported | **Yes** | **No (last release date May 4, 2021)** |
 

README.md

@@ -1,12 +1,15 @@
-# :rainbow_flag: Social networks crawler :rainbow_flag:
+# :rainbow_flag: Social networks crawler :rainbow_flag: :christmas_tree:
+
+# :christmas_tree: Happy new year :christmas_tree:
 
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest)
 [![GitHub license](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
+[![GitHub all releases](https://img.shields.io/github/downloads/aandyprogram/scrawler/total?label=Total%20downloads)](https://github.com/AAndyProgram/SCrawler/releases)
 [![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
 [![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
 [![How to support](https://img.shields.io/badge/HowToSupport-green)](HowToSupport.md)
 
-A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram, TikTok, RedGifs, XVIDEOS, LPSG).
+A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, LPSG).
 
 **If you like SCrawler, please like the program on [this site]( https://alternativeto.net/software/scrawler/about/)**
 
@@ -14,20 +17,23 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 
 [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
 
-**Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
+**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
 
 ![Main window](ProgramScreenshots/MainWindow.png)
 ![Channels window](ProgramScreenshots/Channels.png)
 
 # What can program do:
 - Download pictures and videos from users' profiles and subreddits:
-  - Reddit images, galleries of images, videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
+  - Reddit images, galleries of images, videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**)), saved posts;
   - Redgifs videos (https://www.redgifs.com/);
-  - Twitter images and videos;
+  - Twitter images and videos, saved (bookmarked) posts;
-  - Instagram images and videos, tagged posts, stories;
+  - Instagram images and videos, tagged posts, stories, saved posts;
   - TikTok videos ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
   - Imgur images, galleries and videos;
   - Gfycat videos;
+  - PornHub images, videos, save (liked) posts;
+  - XHamster images, videos, saved posts;
+  - XVIDEOS videos;
   - [Other](#supported-sites) supported sites
 - Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
 - Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
@@ -57,7 +63,9 @@ Do you like this program? Consider adding to my coffee fund by making a donation
 - Imgur
 - Gfycat
 - LPSG
-- XVIDEOS
+- **PornHub**
+- **XHamster**
+- **XVIDEOS**
 - [Other sites](Plugins.md)
 
 **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
@@ -105,6 +113,8 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
   - [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
   - [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
   - [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
+  - [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
+  - [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
   - [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
   - [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
 

SCrawler.PluginProvider/Attributes/Attributes.vb

@@ -132,13 +132,26 @@ Namespace Plugin.Attributes
         ''' Predefined task counter.<br/>
         ''' <see cref="TaskCounter"/> will take precedence if it is defined.
         ''' </param>
-        Public Sub New(Optional ByVal JobsCount As Integer = -1)
+        Public Sub New(Optional ByVal TasksCount As Integer = -1)
-            TasksCount = JobsCount
+            Me.TasksCount = TasksCount
         End Sub
     End Class
     ''' <summary>A property attribute that specifies how many users should be downloaded at the same time in one thread</summary>
     <AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskCounter : Inherits Attribute
     End Class
+    ''' <remarks>
+    ''' This attribute cannot be combined with <see cref="SeparatedTasks"/>.
+    ''' If set to <see cref="SeparatedTasks"/>, this attribute will be ignored
+    ''' </remarks>
+    ''' <inheritdoc cref="SeparatedTasks"/>
+    <AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskGroup : Inherits Attribute
+        Public ReadOnly Name As String
+        ''' <summary>Initialize a new TaskGroup attribute.</summary>
+        ''' <param name="Name">Group name</param>
+        Public Sub New(ByVal Name As String)
+            Me.Name = Name
+        End Sub
+    End Class
     ''' <summary>This attribute indicates that the plugin has a SavedPosts environment</summary>
     <AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class SavedPosts : Inherits Attribute
     End Class

SCrawler.PluginProvider/Interfaces/IPluginContentProvider.vb

@@ -17,9 +17,9 @@ Namespace Plugin
         Property ID As String
         Property ParseUserMediaOnly As Boolean
         Property UserDescription As String
-        Property ExistingContentList As List(Of PluginUserMedia)
+        Property ExistingContentList As List(Of IUserMedia)
         Property TempPostsList As List(Of String)
-        Property TempMediaList As List(Of PluginUserMedia)
+        Property TempMediaList As List(Of IUserMedia)
         Property UserExists As Boolean
         Property UserSuspended As Boolean
         Property IsSavedPosts As Boolean

SCrawler.PluginProvider/Interfaces/ISiteSettings.vb

@@ -18,12 +18,12 @@ Namespace Plugin
         ReadOnly Property Image As Image
         ReadOnly Property Site As String
         Property Logger As ILogProvider
-        Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String
+        Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
         Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
         Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
         Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
         Function GetInstance(ByVal What As Download) As IPluginContentProvider
-        Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
+        Function GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String
 #Region "XML Support"
         Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))
 #End Region

SCrawler.PluginProvider/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2022.10.23.0")>
+<Assembly: AssemblyVersion("2022.12.26.0")>
-<Assembly: AssemblyFileVersion("2022.10.23.0")>
+<Assembly: AssemblyFileVersion("2022.12.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.PluginProvider/Objects/PluginUserMedia.vb

@@ -7,25 +7,46 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Namespace Plugin
-    Public Structure PluginUserMedia
+    Public Enum UserMediaTypes As Integer
-        Enum Types As Integer
+        Undefined = 0
-            Undefined = 0
+        [Picture] = 1
-            [Picture] = 1
+        [Video] = 2
-            [Video] = 2
+        [Text] = 3
-            [Text] = 3
+        VideoPre = 10
-            VideoPre = 10
+        GIF = 50
-            GIF = 50
+        m3u8 = 100
-            m3u8 = 100
+    End Enum
-        End Enum
+    Public Enum UserMediaStates As Integer
-        Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : Missing = 4 : End Enum
+        Unknown = 0
-        Public ContentType As Integer
+        Tried = 1
-        Public URL As String
+        Downloaded = 2
-        Public MD5 As String
+        Skipped = 3
-        Public File As String
+        Missing = 4
-        Public DownloadState As Integer
+    End Enum
-        Public PostID As String
+    Public Structure PluginUserMedia : Implements IUserMedia
-        Public PostDate As Date?
+        Public Property ContentType As Integer Implements IUserMedia.ContentType
-        Public SpecialFolder As String
+        Public Property URL As String Implements IUserMedia.URL
-        Public Attempts As Integer
+        Public Property URL_BASE As String Implements IUserMedia.URL_BASE
+        Public Property MD5 As String Implements IUserMedia.MD5
+        Public Property File As String Implements IUserMedia.File
+        Public Property DownloadState As Integer Implements IUserMedia.DownloadState
+        Public Property PostID As String Implements IUserMedia.PostID
+        Public Property PostDate As Date? Implements IUserMedia.PostDate
+        Public Property SpecialFolder As String Implements IUserMedia.SpecialFolder
+        Public Property Attempts As Integer Implements IUserMedia.Attempts
+        Public Property [Object] As Object Implements IUserMedia.Object
     End Structure
+    Public Interface IUserMedia
+        Property ContentType As Integer
+        Property URL As String
+        Property URL_BASE As String
+        Property MD5 As String
+        Property File As String
+        Property DownloadState As Integer
+        Property PostID As String
+        Property PostDate As Date?
+        Property SpecialFolder As String
+        Property Attempts As Integer
+        Property [Object] As Object
+    End Interface
 End Namespace

SCrawler/API/Base/M3U8Base.vb

@@ -0,0 +1,64 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.Base
+    Namespace M3U8Declarations
+        Friend Module M3U8Defaults
+            Friend ReadOnly TsFilesRegEx As RParams = RParams.DM(".+?\.ts[^\r\n]*", 0, RegexReturn.List)
+        End Module
+    End Namespace
+    Friend NotInheritable Class M3U8Base
+        Private Sub New()
+        End Sub
+        Friend Shared Function CreateUrl(ByVal Appender As String, ByVal File As String) As String
+            File = File.StringTrimStart("/")
+            If File.StartsWith("http") Then
+                Return File
+            Else
+                If File.StartsWith("hls/") And Appender.Contains("hls/") Then _
+                   Appender = LinkFormatterSecure(Appender.Replace("https://", String.Empty).Split("/").First)
+                Return $"{Appender.StringTrimEnd("/")}/{File}"
+            End If
+        End Function
+        Friend Shared Function Download(ByVal URLs As List(Of String), ByVal DestinationFile As SFile, Optional ByVal Responser As Responser = Nothing) As SFile
+            Dim CachePath As SFile = Nothing
+            Try
+                If URLs.ListExists Then
+                    Dim ConcatFile As SFile = DestinationFile
+                    If ConcatFile.Name.IsEmptyString Then ConcatFile.Name = "PlayListFile"
+                    ConcatFile.Extension = "mp4"
+                    CachePath = $"{DestinationFile.PathWithSeparator}_Cache\{SFile.GetDirectories($"{DestinationFile.PathWithSeparator}_Cache\",,, EDP.ReturnValue).ListIfNothing.Count + 1}\"
+                    If CachePath.Exists(SFO.Path) Then
+                        Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
+                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ReturnValue)
+                        Dim i%
+                        Dim eFiles As New List(Of SFile)
+                        Dim dFile As SFile = CachePath
+                        dFile.Extension = "ts"
+                        Using w As New DownloadObjects.WebClient2(Responser)
+                            For i = 0 To URLs.Count - 1
+                                dFile.Name = $"ConPart_{i}"
+                                w.DownloadFile(URLs(i), dFile)
+                                eFiles.Add(dFile)
+                            Next
+                        End Using
+                        DestinationFile = FFMPEG.ConcatenateFiles(eFiles, Settings.FfmpegFile, ConcatFile, p, EDP.ThrowException)
+                        eFiles.Clear()
+                        Return DestinationFile
+                    End If
+                End If
+                Return Nothing
+            Finally
+                CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
+            End Try
+        End Function
+    End Class
+End Namespace

SCrawler/API/Base/ProfileSaved.vb

@@ -7,8 +7,8 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports PersonalUtilities.Forms.Toolbars
 Imports SCrawler.Plugin.Hosts
+Imports PersonalUtilities.Forms.Toolbars
 Imports PDownload = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
     Friend NotInheritable Class ProfileSaved
@@ -52,6 +52,7 @@ Namespace API.Base
                 ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Base.ProfileSaved.Download({HOST.Key})]")
             Finally
                 HOST.DownloadDone(PDownload.SavedPosts)
+                MainFrameObj.UpdateLogButton()
             End Try
         End Sub
     End Class

SCrawler/API/Base/SiteSettingsBase.vb

@@ -7,7 +7,8 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Cookies
 Imports SCrawler.Plugin
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
@@ -16,8 +17,8 @@ Namespace API.Base
         Friend Overridable ReadOnly Property Icon As Icon Implements ISiteSettings.Icon
         Friend Overridable ReadOnly Property Image As Image Implements ISiteSettings.Image
         Private Property Logger As ILogProvider = LogConnector Implements ISiteSettings.Logger
-        Friend Overridable ReadOnly Property Responser As Response
+        Friend Overridable ReadOnly Property Responser As Responser
-        Private Property IResponserContainer_Responser As Response Implements IResponserContainer.Responser
+        Private Property IResponserContainer_Responser As Responser Implements IResponserContainer.Responser
             Get
                 Return Responser
             End Get
@@ -29,7 +30,7 @@ Namespace API.Base
         End Sub
         Friend Sub New(ByVal SiteName As String, ByVal CookiesDomain As String)
             Site = SiteName
-            Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
+            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml")
             With Responser
                 If .File.Exists Then
                     If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
@@ -40,6 +41,7 @@ Namespace API.Base
                     .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
                     .SaveSettings()
                 End If
+                If .CookiesDomain.IsEmptyString Then .CookiesDomain = CookiesDomain
             End With
         End Sub
 #Region "XML"
@@ -74,15 +76,18 @@ Namespace API.Base
 #Region "User info"
         Protected UrlPatternUser As String = String.Empty
         Protected UrlPatternChannel As String = String.Empty
-        Friend Overridable Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String Implements ISiteSettings.GetUserUrl
+        Friend Overridable Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String Implements ISiteSettings.GetUserUrl
             If Channel Then
-                If Not UrlPatternChannel.IsEmptyString Then Return String.Format(UrlPatternChannel, UserName)
+                If Not UrlPatternChannel.IsEmptyString Then Return String.Format(UrlPatternChannel, User.Name)
             Else
-                If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, UserName)
+                If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, User.Name)
             End If
             Return String.Empty
         End Function
-        Friend Overridable Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
+        Private Function ISiteSettings_GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String Implements ISiteSettings.GetUserPostUrl
+            Return GetUserPostUrl(User, Media)
+        End Function
+        Friend Overridable Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
             Return String.Empty
         End Function
         Protected UserRegex As RParams = Nothing
@@ -94,7 +99,7 @@ Namespace API.Base
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]")
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]", New ExchangeOptions)
             End Try
         End Function
         Protected ImageVideoContains As String = String.Empty

SCrawler/API/Base/Structures.vb

@@ -6,12 +6,13 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Base
     Friend Module Structures
-        Friend Structure UserMedia : Implements IEquatable(Of UserMedia), IEContainerProvider
+        Friend Structure UserMedia : Implements IUserMedia, IEquatable(Of UserMedia), IEContainerProvider
 #Region "XML Names"
             Friend Const Name_MediaNode As String = "MediaData"
             Private Const Name_MediaType As String = "Type"
@@ -48,6 +49,97 @@ Namespace API.Base
             ''' SomeFolder\SomeFolder2
             ''' </summary>
             Friend SpecialFolder As String
+            Friend [Object] As Object
+#Region "Interface Support"
+            Private Property IUserMedia_Type As Integer Implements IUserMedia.ContentType
+                Get
+                    Return Type
+                End Get
+                Set(ByVal Type As Integer)
+                    Me.Type = Type
+                End Set
+            End Property
+            Private Property IUserMedia_URL_BASE As String Implements IUserMedia.URL_BASE
+                Get
+                    Return URL_BASE
+                End Get
+                Set(ByVal URL_BASE As String)
+                    Me.URL_BASE = URL_BASE
+                End Set
+            End Property
+            Private Property IUserMedia_URL As String Implements IUserMedia.URL
+                Get
+                    Return URL
+                End Get
+                Set(ByVal URL As String)
+                    Me.URL = URL
+                End Set
+            End Property
+            Private Property IUserMedia_MD5 As String Implements IUserMedia.MD5
+                Get
+                    Return MD5
+                End Get
+                Set(ByVal MD5 As String)
+                    Me.MD5 = MD5
+                End Set
+            End Property
+            Private Property IUserMedia_File As String Implements IUserMedia.File
+                Get
+                    Return File
+                End Get
+                Set(ByVal File As String)
+                    Me.File = File
+                End Set
+            End Property
+            Private Property IUserMedia_State As Integer Implements IUserMedia.DownloadState
+                Get
+                    Return State
+                End Get
+                Set(ByVal State As Integer)
+                    Me.State = State
+                End Set
+            End Property
+            Private Property IUserMedia_PostID As String Implements IUserMedia.PostID
+                Get
+                    Return Post.ID
+                End Get
+                Set(ByVal PostID As String)
+                    Post = New UserPost(PostID, Post.Date)
+                End Set
+            End Property
+            Private Property IUserMedia_PostDate As Date? Implements IUserMedia.PostDate
+                Get
+                    Return Post.Date
+                End Get
+                Set(ByVal PostDate As Date?)
+                    Post = New UserPost(Post.ID, PostDate)
+                End Set
+            End Property
+            Private Property IUserMedia_SpecialFolder As String Implements IUserMedia.SpecialFolder
+                Get
+                    Return SpecialFolder
+                End Get
+                Set(ByVal SpecialFolder As String)
+                    Me.SpecialFolder = SpecialFolder
+                End Set
+            End Property
+            Private Property IUserMedia_Attempts As Integer Implements IUserMedia.Attempts
+                Get
+                    Return Attempts
+                End Get
+                Set(ByVal Attempts As Integer)
+                    Me.Attempts = Attempts
+                End Set
+            End Property
+            Private Property IUserMedia_Object As Object Implements IUserMedia.Object
+                Get
+                    Return Me.Object
+                End Get
+                Set(ByVal Obj As Object)
+                    Me.Object = Obj
+                End Set
+            End Property
+#End Region
             Friend Sub New(ByVal URL As String)
                 Me.URL = URL
                 URL_BASE = URL
@@ -58,16 +150,17 @@ Namespace API.Base
                 Me.New(URL)
                 Me.Type = Type
             End Sub
-            Friend Sub New(ByVal m As Plugin.PluginUserMedia)
+            Friend Sub New(ByVal m As IUserMedia)
                 [Type] = m.ContentType
                 URL = m.URL
-                URL_BASE = URL
+                URL_BASE = m.URL_BASE
                 MD5 = m.MD5
                 File = m.File
                 Post = New UserPost With {.ID = m.PostID, .[Date] = m.PostDate}
                 State = m.DownloadState
                 SpecialFolder = m.SpecialFolder
                 Attempts = m.Attempts
+                Me.Object = m.Object
             End Sub
             Friend Sub New(ByVal e As EContainer, ByVal UserInstance As IUserData)
                 Type = e.Attribute(Name_MediaType).Value.FromXML(Of Integer)(CInt(Types.Undefined))
@@ -117,19 +210,6 @@ Namespace API.Base
             Public Overrides Function ToString() As String
                 Return URL
             End Function
-            Friend Function PluginUserMedia() As Plugin.PluginUserMedia
-                Return New Plugin.PluginUserMedia With {
-                    .ContentType = Type,
-                    .DownloadState = State,
-                    .File = File,
-                    .MD5 = MD5,
-                    .URL = URL,
-                    .SpecialFolder = SpecialFolder,
-                    .PostID = Post.ID,
-                    .PostDate = Post.Date,
-                    .Attempts = Attempts
-                }
-            End Function
             Friend Overloads Function Equals(ByVal Other As UserMedia) As Boolean Implements IEquatable(Of UserMedia).Equals
                 Return URL = Other.URL
             End Function
@@ -154,17 +234,31 @@ Namespace API.Base
             ''' <summary>Post ID</summary>
             Friend ID As String
             Friend [Date] As Date?
-#Region "Channel compatible fields"
             Friend UserID As String
             Friend CachedFile As SFile
+#Region "Initializers"
+            Public Sub New(ByVal ID As String)
+                Me.ID = ID
+            End Sub
+            Public Sub New(ByVal [Date] As Date?)
+                Me.Date = [Date]
+            End Sub
+            Public Sub New(ByVal ID As String, ByVal [Date] As Date?)
+                Me.ID = ID
+                Me.Date = [Date]
+            End Sub
+            Public Shared Widening Operator CType(ByVal ID As String) As UserPost
+                Return New UserPost(ID)
+            End Operator
+            Public Shared Widening Operator CType(ByVal Post As UserPost) As String
+                Return Post.ID
+            End Operator
 #End Region
-            Friend Function GetImage(ByVal s As Size, ByVal e As ErrorsDescriber, ByVal NullArg As Image) As Image
+#Region "ToString"
-                If Not CachedFile.IsEmptyString Then
+            Public Overrides Function ToString() As String
-                    Return If(PersonalUtilities.Tools.ImageRenderer.GetImage(SFile.GetBytes(CachedFile), s, e), NullArg.Clone)
+                Return ID
-                Else
-                    Return NullArg.Clone
-                End If
             End Function
+#End Region
 #Region "IEquatable, IComparable Support"
             Friend Overloads Function Equals(ByVal Other As UserPost) As Boolean Implements IEquatable(Of UserPost).Equals
                 Return ID = Other.ID

SCrawler/API/Base/UserDataBase.vb

@@ -10,7 +10,7 @@ Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Imports System.IO
 Imports System.Net
 Imports System.Threading
@@ -94,9 +94,16 @@ Namespace API.Base
         End Sub
 #End Region
 #Region "XML Declarations"
-        Private Const Name_Site As String = "Site"
+        Private Const Name_Site As String = UserInfo.Name_Site
-        Private Const Name_IsChannel As String = "IsChannel"
+        Private Const Name_Plugin As String = UserInfo.Name_Plugin
-        Private Const Name_UserName As String = "UserName"
+        Private Const Name_IsChannel As String = UserInfo.Name_IsChannel
+        Friend Const Name_UserName As String = "UserName"
+        Private Const Name_Model_User As String = UserInfo.Name_Model_User
+        Private Const Name_Model_Collection As String = UserInfo.Name_Model_Collection
+        Private Const Name_Merged As String = UserInfo.Name_Merged
+        Private Const Name_SpecialPath As String = UserInfo.Name_SpecialPath
+        Private Const Name_SpecialCollectionPath As String = UserInfo.Name_SpecialCollectionPath
+
         Private Const Name_UserExists As String = "UserExists"
         Private Const Name_UserSuspended As String = "UserSuspended"
         Private Const Name_FriendlyName As String = "FriendlyName"
@@ -108,8 +115,8 @@ Namespace API.Base
         Private Const Name_CreatedByChannel As String = "CreatedByChannel"
 
         Private Const Name_SeparateVideoFolder As String = "SeparateVideoFolder"
-        Private Const Name_CollectionName As String = "Collection"
+        Private Const Name_CollectionName As String = UserInfo.Name_Collection
-        Private Const Name_LabelsName As String = "Labels"
+        Friend Const Name_LabelsName As String = "Labels"
 
         Private Const Name_ReadyForDownload As String = "ReadyForDownload"
         Private Const Name_DownloadImages As String = "DownloadImages"
@@ -122,7 +129,7 @@ Namespace API.Base
         Private Const Name_ScriptUse As String = "ScriptUse"
         Private Const Name_ScriptData As String = "ScriptData"
 
-        Private Const Name_DataMerging As String = "DataMerging"
+        <Obsolete("Use 'Name_Merged'", False)> Friend Const Name_DataMerging As String = "DataMerging"
 #End Region
 #Region "Declarations"
 #Region "Host, Site, Progress, Self"
@@ -156,6 +163,21 @@ Namespace API.Base
         End Property
         Friend Overridable Property ID As String = String.Empty Implements IContentProvider.ID, IPluginContentProvider.ID
         Friend Overridable Property FriendlyName As String = String.Empty Implements IContentProvider.FriendlyName
+        Friend ReadOnly Property UserModel As UsageModel Implements IUserData.UserModel
+            Get
+                Return User.UserModel
+            End Get
+        End Property
+        Friend Overridable ReadOnly Property CollectionModel As UsageModel Implements IUserData.CollectionModel
+            Get
+                Return User.CollectionModel
+            End Get
+        End Property
+        Friend Overridable ReadOnly Property IsVirtual As Boolean Implements IUserData.IsVirtual
+            Get
+                Return UserModel = UsageModel.Virtual
+            End Get
+        End Property
 #End Region
 #Region "Description"
         Friend Property UserDescription As String = String.Empty Implements IContentProvider.Description, IPluginContentProvider.UserDescription
@@ -231,7 +253,7 @@ Namespace API.Base
         Protected Function GetNullPicture(ByVal MaxHeigh As XML.Base.XMLValue(Of Integer)) As Bitmap
             Return New Bitmap(CInt(DivideWithZeroChecking(MaxHeigh.Value, 100) * 75), MaxHeigh.Value)
         End Function
-        Protected Function GetPicture(Of T)(Optional ByVal ReturnNullImageOnNothing As Boolean = True, Optional ByVal GetToast As Boolean = False) As T
+        Friend Function GetPicture(Of T)(Optional ByVal ReturnNullImageOnNothing As Boolean = True, Optional ByVal GetToast As Boolean = False) As T
             Dim rsfile As Boolean = GetType(T) Is GetType(SFile)
             Dim f As SFile = Nothing
             Dim p As UserImage = Nothing
@@ -335,7 +357,7 @@ BlockNullPicture:
         Friend Overridable Sub ChangeCollectionName(ByVal NewName As String, ByVal UpdateSettings As Boolean)
             Dim u As UserInfo = User
             u.CollectionName = NewName
-            u.IncludedInCollection = Not NewName.IsEmptyString
+            u.UpdateUserFile()
             User = u
             If UpdateSettings Then Settings.UpdateUsersList(User)
         End Sub
@@ -455,7 +477,10 @@ BlockNullPicture:
         End Property
         Friend Overridable Function GetUserInformation() As String
             Dim OutStr$ = $"User: {Name} (site: {Site}"
-            If IncludedInCollection Then OutStr &= $"; collection: {CollectionName}"
+            If IncludedInCollection Then
+                OutStr &= $"; collection: {CollectionName}"
+                If CollectionModel = UsageModel.Default And UserModel = UsageModel.Virtual Then OutStr &= "; virtual"
+            End If
             OutStr &= ")"
             OutStr.StringAppendLine($"Labels: {Labels.ListToString}")
             OutStr.StringAppendLine($"Path: {MyFile.CutPath.Path}")
@@ -494,9 +519,9 @@ BlockNullPicture:
         Private Property IPluginContentProvider_Thrower As IThrower Implements IPluginContentProvider.Thrower
         Private Property IPluginContentProvider_LogProvider As ILogProvider Implements IPluginContentProvider.LogProvider
         Friend Property ExternalPlugin As IPluginContentProvider
-        Private Property IPluginContentProvider_ExistingContentList As List(Of PluginUserMedia) Implements IPluginContentProvider.ExistingContentList
+        Private Property IPluginContentProvider_ExistingContentList As List(Of IUserMedia) Implements IPluginContentProvider.ExistingContentList
         Private Property IPluginContentProvider_TempPostsList As List(Of String) Implements IPluginContentProvider.TempPostsList
-        Private Property IPluginContentProvider_TempMediaList As List(Of PluginUserMedia) Implements IPluginContentProvider.TempMediaList
+        Private Property IPluginContentProvider_TempMediaList As List(Of IUserMedia) Implements IPluginContentProvider.TempMediaList
         Private Property IPluginContentProvider_SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
         Private Property IPluginContentProvider_DataPath As String Implements IPluginContentProvider.DataPath
         Private Sub IPluginContentProvider_XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String))) Implements IPluginContentProvider.XmlFieldsSet
@@ -620,7 +645,7 @@ BlockNullPicture:
                     With DirectCast(u, UserDataBase)
                         If Not .User.Plugin.IsEmptyString Then
                             uName = .User.Name
-                            Return Settings(.User.Plugin).GetUserPostUrl(.ID, PostData.Post.ID)
+                            Return Settings(.User.Plugin).GetUserPostUrl(.Self, PostData)
                         End If
                     End With
                 End If
@@ -657,7 +682,14 @@ BlockNullPicture:
                         LastUpdated = AConvert(Of Date)(x.Value(Name_LastUpdated), ADateTime.Formats.BaseDateTime, Nothing)
                         ScriptUse = x.Value(Name_ScriptUse).FromXML(Of Boolean)(False)
                         ScriptData = x.Value(Name_ScriptData)
-                        DataMerging = x.Value(Name_DataMerging).FromXML(Of Boolean)(False)
+                        'TODELETE: UserDataBase remove old 'merge' constant
+#Disable Warning BC40000
+                        If x.Contains(Name_DataMerging) Then
+                            DataMerging = x.Value(Name_DataMerging).FromXML(Of Boolean)(False)
+                        Else
+                            DataMerging = x.Value(Name_Merged).FromXML(Of Boolean)(False)
+                        End If
+#Enable Warning
                         ChangeCollectionName(x.Value(Name_CollectionName), False)
                         Labels.ListAddList(x.Value(Name_LabelsName).StringToList(Of String, List(Of String))("|", EDP.ReturnValue), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
                         LoadUserInformation_OptionalFields(x, True)
@@ -675,7 +707,13 @@ BlockNullPicture:
                 MyFile.Exists(SFO.Path)
                 Using x As New XmlFile With {.Name = "User"}
                     x.Add(Name_Site, Site)
+                    x.Add(Name_Plugin, HOST.Key)
                     x.Add(Name_UserName, User.Name)
+                    x.Add(Name_IsChannel, IsChannel.BoolToInteger)
+                    x.Add(Name_Model_User, CInt(UserModel))
+                    x.Add(Name_Model_Collection, CInt(CollectionModel))
+                    x.Add(Name_SpecialPath, User.SpecialPath)
+                    x.Add(Name_SpecialCollectionPath, User.SpecialCollectionPath)
                     x.Add(Name_UserExists, UserExists.BoolToInteger)
                     x.Add(Name_UserSuspended, UserSuspended.BoolToInteger)
                     x.Add(Name_UserID, ID)
@@ -700,7 +738,7 @@ BlockNullPicture:
                     x.Add(Name_ScriptData, ScriptData)
                     x.Add(Name_CollectionName, CollectionName)
                     x.Add(Name_LabelsName, Labels.ListToString("|", EDP.ReturnValue))
-                    x.Add(Name_DataMerging, DataMerging.BoolToInteger)
+                    x.Add(Name_Merged, DataMerging.BoolToInteger)
 
                     LoadUserInformation_OptionalFields(x, False)
 
@@ -748,7 +786,7 @@ BlockNullPicture:
 #Region "Open site, folder"
         Friend Overridable Sub OpenSite(Optional ByVal e As ErrorsDescriber = Nothing) Implements IContentProvider.OpenSite
             Try
-                Dim URL$ = HOST.Source.GetUserUrl(Name, IsChannel)
+                Dim URL$ = HOST.Source.GetUserUrl(Me, IsChannel)
                 If Not URL.IsEmptyString Then Process.Start(URL)
             Catch ex As Exception
                 If Not e.Exists Then e = New ErrorsDescriber(EDP.ShowAllMsg)
@@ -806,7 +844,7 @@ BlockNullPicture:
         End Function
 #End Region
 #Region "Download functions and options"
-        Protected Responser As Response
+        Protected Responser As Responser
         Protected UseResponserClient As Boolean = False
         Friend Overridable Sub DownloadData(ByVal Token As CancellationToken) Implements IContentProvider.DownloadData
             Dim Canceled As Boolean = False
@@ -815,9 +853,9 @@ BlockNullPicture:
                 UpdateDataFiles()
                 UserDescriptionReset()
                 If Not Responser Is Nothing Then Responser.Dispose()
-                Responser = New Response
+                Responser = New Responser
                 If Not HOST.Responser Is Nothing Then Responser.Copy(HOST.Responser)
-                'TODO: remove
+                'TODO: UserDataBase remove [Responser.DecodersError]
                 Responser.DecodersError = New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue) With {
                     .DeclaredMessage = New MMessage($"SymbolsConverter error: [{ToStringForLog()}]", ToStringForLog())}
 
@@ -831,6 +869,7 @@ BlockNullPicture:
                 DownloadedVideos(False) = 0
                 _TempMediaList.Clear()
                 _TempPostsList.Clear()
+                LatestData.Clear()
                 Dim __SaveData As Boolean = Not CreatedByChannel Or Not Settings.FromChannelDownloadTopUse
 
                 LoadContentInformation()
@@ -862,7 +901,7 @@ BlockNullPicture:
                 DownloadContent(Token)
                 ThrowIfDisposed()
 
-                LatestData.ListAddList(_ContentNew.Where(_downContent), LNC)
+                If IncludeInTheFeed Then LatestData.ListAddList(_ContentNew.Where(_downContent), LNC)
                 Dim mcb& = If(ContentMissingExists, _ContentList.LongCount(Function(c) MissingFinder(c)), 0)
                 _ContentList.ListAddList(_ContentNew.Where(Function(c) _downContent(c) Or MissingFinder(c)), LNC)
                 Dim mca& = If(ContentMissingExists, _ContentList.LongCount(Function(c) MissingFinder(c)), 0)
@@ -887,7 +926,7 @@ BlockNullPicture:
                 ThrowIfDisposed()
                 If UpPic Or EnvirChanged.Invoke Then OnUserUpdated()
             Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                MyMainLOG = $"{Site} - {Name}: downloading canceled"
+                MyMainLOG = $"{ToStringForLog()}: downloading canceled"
                 Canceled = True
             Catch dex As ObjectDisposedException When Disposed
                 Canceled = True
@@ -926,11 +965,7 @@ BlockNullPicture:
         Protected Overridable Sub ReparseMissing(ByVal Token As CancellationToken)
         End Sub
         Protected MustOverride Sub DownloadContent(ByVal Token As CancellationToken)
-        Private NotInheritable Class OptionalWebClient : Implements IDisposable
+        Private NotInheritable Class OptionalWebClient : Inherits DownloadObjects.WebClient2
-            Private ReadOnly WC As WebClient
-            Private ReadOnly RC As Response
-            Private ReadOnly RCERROR As New ErrorsDescriber(EDP.ThrowException)
-            Private ReadOnly UseResponserClient As Boolean
             Friend Sub New(ByRef Source As UserDataBase)
                 UseResponserClient = Source.UseResponserClient
                 If UseResponserClient Then
@@ -939,28 +974,6 @@ BlockNullPicture:
                     WC = New WebClient
                 End If
             End Sub
-            Friend Sub DownloadFile(ByVal URL As String, ByVal File As String)
-                If UseResponserClient Then
-                    RC.DownloadFile(URL, File, RCERROR)
-                Else
-                    WC.DownloadFile(URL, File)
-                End If
-            End Sub
-#Region "IDisposable Support"
-            Private disposedValue As Boolean = False
-            Protected Sub Dispose(ByVal disposing As Boolean)
-                If Not disposedValue And disposing And Not WC Is Nothing Then WC.Dispose()
-                disposedValue = True
-            End Sub
-            Protected Overrides Sub Finalize()
-                Dispose(False)
-                MyBase.Finalize()
-            End Sub
-            Friend Sub Dispose() Implements IDisposable.Dispose
-                Dispose(True)
-                GC.SuppressFinalize(Me)
-            End Sub
-#End Region
         End Class
         Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
             Try
@@ -1068,9 +1081,12 @@ BlockNullPicture:
         Protected Overridable Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
             Return Nothing
         End Function
+        Protected Const EXCEPTION_OPERATION_CANCELED As Integer = -1
         ''' <param name="RDE">Request DownloadingException</param>
         ''' <returns>0 - exit</returns>
-        Protected Function ProcessException(ByVal ex As Exception, ByVal Token As CancellationToken, ByVal Message As String, Optional ByVal RDE As Boolean = True, Optional ByVal EObj As Object = Nothing) As Integer
+        Protected Function ProcessException(ByVal ex As Exception, ByVal Token As CancellationToken, ByVal Message As String,
+                                            Optional ByVal RDE As Boolean = True, Optional ByVal EObj As Object = Nothing,
+                                            Optional ByVal ThrowEx As Boolean = True) As Integer
             If Not ((TypeOf ex Is OperationCanceledException And Token.IsCancellationRequested) Or
                     (TypeOf ex Is ObjectDisposedException And Disposed)) Then
                 If RDE Then
@@ -1078,6 +1094,9 @@ BlockNullPicture:
                     If v = 0 Then LogError(ex, Message) : HasError = True
                     Return v
                 End If
+            Else
+                'URGENT: UserDataBase.ProcessException [Throw ex]
+                If ThrowEx Then Throw ex Else Return EXCEPTION_OPERATION_CANCELED
             End If
             Return 0
         End Function
@@ -1124,7 +1143,7 @@ BlockNullPicture:
         End Sub
 #End Region
 #Region "Delete, Move, Merge, Copy"
-        Friend Overridable Function Delete(Optional ByVal Multiple As Boolean = False) As Integer Implements IUserData.Delete
+        Friend Overridable Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer Implements IUserData.Delete
             Dim f As SFile = SFile.GetPath(MyFile.CutPath.Path)
             If f.Exists(SFO.Path, False) AndAlso (User.Merged OrElse f.Delete(SFO.Path, Settings.DeleteMode)) Then
                 If Not IncludedInCollection Then MainFrameObj.ImageHandler(Me, False)
@@ -1138,44 +1157,51 @@ BlockNullPicture:
                 Return 0
             End If
         End Function
-        Friend Overridable Function MoveFiles(ByVal __CollectionName As String) As Boolean Implements IUserData.MoveFiles
+        Friend Overridable Function MoveFiles(ByVal __CollectionName As String, ByVal __SpecialCollectionPath As SFile) As Boolean Implements IUserData.MoveFiles
             Dim UserBefore As UserInfo = User
             Dim Removed As Boolean = True
             Dim _TurnBack As Boolean = False
             Try
                 Dim f As SFile
+                Dim v As Boolean = IsVirtual
                 If IncludedInCollection Then
                     Settings.Users.Add(Me)
                     Removed = False
                     User.CollectionName = String.Empty
-                    User.IncludedInCollection = False
+                    User.SpecialCollectionPath = String.Empty
+                    User.UserModel = UsageModel.Default
+                    User.CollectionModel = UsageModel.Default
                 Else
                     Settings.Users.Remove(Me)
                     Removed = True
                     User.CollectionName = __CollectionName
-                    User.IncludedInCollection = True
+                    User.SpecialCollectionPath = __SpecialCollectionPath
-                    User.SpecialPath = Nothing
+                    If Not IsVirtual Then User.SpecialPath = Nothing
                 End If
                 _TurnBack = True
                 User.UpdateUserFile()
-                f = User.File.CutPath(, EDP.ThrowException)
+
-                If f.Exists(SFO.Path, False) Then
+                If Not v Then
-                    If If(SFile.GetFiles(f,, SearchOption.AllDirectories), New List(Of SFile)).Count > 0 AndAlso
+                    f = User.File.CutPath(, EDP.ThrowException)
-                       MsgBoxE({$"Destination directory [{f.Path}] already exists and contains files!" & vbCr &
+                    If f.Exists(SFO.Path, False) Then
-                                "By continuing, this directory and all files will be deleted",
+                        If If(SFile.GetFiles(f,, SearchOption.AllDirectories), New List(Of SFile)).Count > 0 AndAlso
-                                "Destination directory is not empty!"}, MsgBoxStyle.Exclamation,,, {"Delete", "Cancel"}) = 1 Then
+                           MsgBoxE({$"Destination directory [{f.Path}] already exists and contains files!" & vbCr &
-                        MsgBoxE("Operation canceled", MsgBoxStyle.Exclamation)
+                                    "By continuing, this directory and all files will be deleted",
-                        User = UserBefore
+                                    "Destination directory is not empty!"}, MsgBoxStyle.Exclamation,,, {"Delete", "Cancel"}) = 1 Then
-                        If Removed Then Settings.Users.Add(Me) Else Settings.Users.Remove(Me)
+                            MsgBoxE("Operation canceled", MsgBoxStyle.Exclamation)
-                        _TurnBack = False
+                            User = UserBefore
-                        Return False
+                            If Removed Then Settings.Users.Add(Me) Else Settings.Users.Remove(Me)
+                            _TurnBack = False
+                            Return False
+                        End If
+                        f.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException)
                     End If
-                    f.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException)
+                    f.CutPath.Exists(SFO.Path)
+                    Directory.Move(UserBefore.File.CutPath(, EDP.ThrowException).Path, f.Path)
+                    If Not ScriptData.IsEmptyString AndAlso ScriptData.Contains(UserBefore.File.PathNoSeparator) Then _
+                       ScriptData = ScriptData.Replace(UserBefore.File.PathNoSeparator, MyFile.PathNoSeparator)
                 End If
-                f.CutPath.Exists(SFO.Path)
+
-                Directory.Move(UserBefore.File.CutPath(, EDP.ThrowException).Path, f.Path)
-                If Not ScriptData.IsEmptyString AndAlso ScriptData.Contains(UserBefore.File.PathNoSeparator) Then _
-                   ScriptData = ScriptData.Replace(UserBefore.File.PathNoSeparator, MyFile.PathNoSeparator)
                 Settings.UsersList.Remove(UserBefore)
                 Settings.UpdateUsersList(User)
                 UpdateUserInformation()
@@ -1411,6 +1437,7 @@ BlockNullPicture:
         End Sub
 #End Region
     End Class
+#Region "Base interfaces"
     Friend Interface IContentProvider
         ReadOnly Property Site As String
         Property Name As String
@@ -1433,6 +1460,9 @@ BlockNullPicture:
         ReadOnly Property IsCollection As Boolean
         Property CollectionName As String
         ReadOnly Property IncludedInCollection As Boolean
+        ReadOnly Property UserModel As UsageModel
+        ReadOnly Property CollectionModel As UsageModel
+        ReadOnly Property IsVirtual As Boolean
         ReadOnly Property Labels As List(Of String)
 #End Region
         ReadOnly Property IsChannel As Boolean
@@ -1464,8 +1494,8 @@ BlockNullPicture:
         ''' 2 - Collection removed<br/>
         ''' 3 - Collection split
         ''' </summary>
-        Function Delete(Optional ByVal Multiple As Boolean = False) As Integer
+        Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer
-        Function MoveFiles(ByVal CollectionName As String) As Boolean
+        Function MoveFiles(ByVal CollectionName As String, ByVal SpecialCollectionPath As SFile) As Boolean
         Function CopyFiles(ByVal DestinationPath As SFile, Optional ByVal e As ErrorsDescriber = Nothing) As Boolean
         Sub OpenFolder()
         ReadOnly Property Self As IUserData
@@ -1488,4 +1518,5 @@ BlockNullPicture:
         Property SkipExistsUsers As Boolean
         Property SaveToCache As Boolean
     End Interface
+#End Region
 End Namespace

SCrawler/API/BaseObjects/DomainEnvir.vb

@@ -0,0 +1,86 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Forms
+Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
+Namespace API.BaseObjects
+    Friend Interface IDomainContainer
+        ReadOnly Property Icon As Icon
+        ReadOnly Property Site As String
+        ReadOnly Property Domains As List(Of String)
+        ReadOnly Property DomainsTemp As List(Of String)
+        ReadOnly Property DomainsDefault As String
+        ReadOnly Property DomainsSettingProp As Plugin.PropertyValue
+        Property DomainsChanged As Boolean
+        Property Initialized As Boolean
+        Property DomainsUpdateInProgress As Boolean
+        Property DomainsUpdatedBySite As Boolean
+        Sub UpdateDomains()
+    End Interface
+    Friend NotInheritable Class DomainContainer
+        Private Sub New()
+        End Sub
+        Friend Shared Sub EndInit(ByVal s As IDomainContainer)
+            If ACheck(s.DomainsSettingProp.Value) Then s.Domains.ListAddList(CStr(s.DomainsSettingProp.Value).Split("|"), LAP.NotContainsOnly)
+        End Sub
+        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer)
+            UpdateDomains(s, Nothing, True)
+        End Sub
+        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer, ByVal NewDomains As IEnumerable(Of String), ByVal Internal As Boolean)
+            With s
+                If Not .Initialized Or (.DomainsUpdatedBySite And Not Internal) Then Exit Sub
+                If Not .DomainsUpdateInProgress Then
+                    .DomainsUpdateInProgress = True
+                    .Domains.ListAddList(.DomainsDefault.Split("|"), LAP.NotContainsOnly)
+                    .Domains.ListAddList(NewDomains, LAP.NotContainsOnly)
+                    .DomainsSettingProp.Value = .Domains.ListToString("|")
+                    If Not Internal Then .DomainsUpdatedBySite = True
+                    .DomainsUpdateInProgress = False
+                End If
+            End With
+        End Sub
+        Friend Shared Sub Update(ByVal s As IDomainContainer)
+            With s
+                If .DomainsChanged Then
+                    .Domains.Clear()
+                    .Domains.ListAddList(.DomainsTemp, LAP.NotContainsOnly)
+                    .UpdateDomains()
+                End If
+            End With
+        End Sub
+        Friend Shared Sub EndEdit(ByVal s As IDomainContainer)
+            s.DomainsTemp.ListAddList(s.Domains, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
+            s.DomainsChanged = False
+        End Sub
+        Friend Shared Sub OpenSettingsForm(ByVal s As IDomainContainer)
+            Dim __add As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e) e.ValueNew = InputBoxE($"Enter a new domain using the pattern [{s.Site}.com]:", "New domain").IfNullOrEmptyE(Nothing)
+            Dim __delete As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e)
+                                                                           Dim n$ = AConvert(Of String)(e.ValueCurrent, AModes.Var, String.Empty)
+                                                                           e.Result = MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
+                                                                                              "Removing domains"}, vbYesNo) = vbYes
+                                                                       End Sub
+            Using f As New SimpleListForm(Of String)(If(s.DomainsChanged, s.DomainsTemp, s.Domains), Settings.Design) With {
+                .Buttons = {ADB.Add, ADB.Delete},
+                .Mode = SimpleListFormModes.Remaining,
+                .FormText = s.Site,
+                .Icon = s.Icon,
+                .LocationOnly = True,
+                .Size = New Size(400, 330),
+                .DesignXMLNode = s.Site
+            }
+                AddHandler f.AddClick, __add
+                AddHandler f.DeleteClick, __delete
+                f.ShowDialog()
+                If f.DialogResult = DialogResult.OK Then
+                    s.DomainsChanged = True
+                    s.DomainsTemp.ListAddList(f.DataResult, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
+                End If
+            End Using
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Imgur/Envir.vb

@@ -11,7 +11,7 @@ Imports SCrawler.API.Base
 Imports SCrawler.API.Imgur.Declarations
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Namespace API.Imgur
     Namespace Declarations
         Friend Module Imgur_Declarations

SCrawler/API/Instagram/Declarations.vb

@@ -7,11 +7,49 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Clients.EventArguments
+Imports PersonalUtilities.Tools.Web.Cookies
 Namespace API.Instagram
     Friend Module Declarations
         Friend Const InstagramSite As String = "Instagram"
         Friend Const InstagramSiteKey As String = "AndyProgram_Instagram"
         Friend ReadOnly FilesPattern As RParams = RParams.DMS(".+?([^/\?]+?\.[\w\d]{3,4})(?=(\?|\Z))", 1, EDP.ReturnValue)
         Friend ReadOnly Property DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
+        Friend Sub UpdateResponser(ByVal Source As IResponse, ByRef Destination As Responser)
+            Const r_wwwClaimName$ = "x-ig-set-www-claim"
+            Const r_tokenName$ = "csrftoken"
+            If Not Source Is Nothing Then
+                Dim isInternal As Boolean = TypeOf Source Is WebDataResponse
+                Dim wwwClaimName$, tokenName$
+                If isInternal Then
+                    wwwClaimName = r_wwwClaimName
+                    tokenName = r_tokenName
+                Else
+                    wwwClaimName = SiteSettings.Header_IG_WWW_CLAIM
+                    tokenName = SiteSettings.Header_CSRF_TOKEN
+                End If
+                Dim wwwClaim$ = String.Empty
+                Dim token$ = String.Empty
+                With Source
+                    If isInternal Then
+                        If .HeadersExists Then wwwClaim = .Headers.Value(wwwClaimName)
+                        If .CookiesExists Then token = If(.Cookies.FirstOrDefault(Function(c) c.Name = tokenName)?.Value, String.Empty)
+                    Else
+                        If .HeadersExists Then
+                            wwwClaim = .Headers.Value(wwwClaimName)
+                            token = .Headers.Value(tokenName)
+                        End If
+                    End If
+                End With
+
+                If Not wwwClaim.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_IG_WWW_CLAIM, wwwClaim)
+                If Not token.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_CSRF_TOKEN, token)
+                If Not isInternal Then
+                    Destination.Cookies.Update(Source.Cookies, CookieKeeper.UpdateModes.ReplaceByNameAll, False, EDP.SendInLog)
+                    Destination.SaveSettings()
+                End If
+            End If
+        End Sub
     End Module
 End Namespace

SCrawler/API/Instagram/SettingsExchangeOptions.vb

@@ -1,23 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.Instagram
-    Friend Structure SettingsExchangeOptions
-        Friend DownloadTimeline As Boolean
-        Friend DownloadStoriesTagged As Boolean
-        Friend DownloadSaved As Boolean
-        Friend Changed As Boolean
-        Friend Sub New(ByVal Source As SiteSettings)
-            With Source
-                DownloadTimeline = .DownloadTimeline
-                DownloadStoriesTagged = .DownloadStoriesTagged
-                DownloadSaved = .DownloadSaved
-            End With
-        End Sub
-    End Structure
-End Namespace

SCrawler/API/Instagram/SiteSettings.vb

@@ -13,9 +13,11 @@ Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Cookies
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Instagram
-    <Manifest(InstagramSiteKey), SeparatedTasks(1), SavedPosts, SpecialForm(False), SpecialForm(True)>
+    <Manifest(InstagramSiteKey), SeparatedTasks(1), SavedPosts, SpecialForm(False)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
 #Region "Declarations"
 #Region "Images"
@@ -77,28 +79,23 @@ Namespace API.Instagram
         End Class
 #End Region
 #Region "Authorization properties"
-        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash", IsAuth:=True, AllowNull:=False), PXML("InstaHash"), ControlNumber(0)>
+        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash for tagged posts", IsAuth:=True), PXML("InstaHash"), ControlNumber(0)>
-        Friend ReadOnly Property Hash As PropertyValue
+        Friend ReadOnly Property HashTagged As PropertyValue
-        Private Const HashSavedPosts_Text As String = "Hash 2"
+        <PropertyOption(ControlText:="x-csrftoken", IsAuth:=True, AllowNull:=False), ControlNumber(2)>
-        <PropertyOption(ControlText:=HashSavedPosts_Text, ControlToolTip:="Instagram session hash for saved posts", IsAuth:=True), PXML("InstaHashSavedPosts"), ControlNumber(1)>
-        Friend ReadOnly Property HashSavedPosts As PropertyValue
-        <PropertyOption(ControlText:="x-csrftoken", ControlToolTip:="Instagram token for tagged data", IsAuth:=True, AllowNull:=False), ControlNumber(2)>
         Friend ReadOnly Property CSRF_TOKEN As PropertyValue
         <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True, AllowNull:=False), ControlNumber(3)>
         Friend Property IG_APP_ID As PropertyValue
         <PropertyOption(ControlText:="x-ig-www-claim", IsAuth:=True, AllowNull:=False), ControlNumber(4)>
         Friend Property IG_WWW_CLAIM As PropertyValue
-        Private Const SavedPostsUserName_Text As String = "Saved posts user"
-        <PropertyOption(ControlText:=SavedPostsUserName_Text, IsAuth:=True), PXML("SavedPostsUserName"), ControlNumber(5)>
-        Friend ReadOnly Property SavedPostsUserName As PropertyValue
         Friend Overrides Function BaseAuthExists() As Boolean
-            Return If(Responser.Cookies?.Count, 0) > 0 And ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value) And ACheck(CSRF_TOKEN.Value)
+            Return Responser.CookiesExists And ACheck(IG_APP_ID.Value) And ACheck(IG_WWW_CLAIM.Value) And ACheck(CSRF_TOKEN.Value)
         End Function
         Private Const Header_IG_APP_ID As String = "x-ig-app-id"
-        Private Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
+        Friend Const Header_IG_WWW_CLAIM As String = "x-ig-www-claim"
-        Private Const Header_CSRF_TOKEN As String = "x-csrftoken"
+        Friend Const Header_CSRF_TOKEN As String = "x-csrftoken"
+        Private _FieldsChangerSuspended As Boolean = False
         Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
-            If Not PropName.IsEmptyString Then
+            If Not _FieldsChangerSuspended And Not PropName.IsEmptyString Then
                 Dim f$ = String.Empty
                 Select Case PropName
                     Case NameOf(IG_APP_ID) : f = Header_IG_APP_ID
@@ -106,7 +103,7 @@ Namespace API.Instagram
                     Case NameOf(CSRF_TOKEN) : f = Header_CSRF_TOKEN
                 End Select
                 If Not f.IsEmptyString Then
-                    If Responser.Headers.Count > 0 AndAlso Responser.Headers.ContainsKey(f) Then Responser.Headers.Remove(f)
+                    Responser.Headers.Remove(f)
                     If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
                     Responser.SaveSettings()
                 End If
@@ -114,33 +111,34 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "Download properties"
-        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(6)>
+        <PropertyOption(ControlText:="Request timer", AllowNull:=False), PXML("RequestsWaitTimer"), ControlNumber(8)>
         Friend ReadOnly Property RequestsWaitTimer As PropertyValue
         <Provider(NameOf(RequestsWaitTimer), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerProvider As IFormatProvider
-        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(7)>
+        <PropertyOption(ControlText:="Request timer counter", AllowNull:=False, LeftOffset:=120), PXML("RequestsWaitTimerTaskCount"), ControlNumber(9)>
         Friend ReadOnly Property RequestsWaitTimerTaskCount As PropertyValue
         <Provider(NameOf(RequestsWaitTimerTaskCount), FieldsChecker:=True)>
         Private ReadOnly Property RequestsWaitTimerTaskCountProvider As IFormatProvider
-        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(8)>
+        <PropertyOption(ControlText:="Posts limit timer", AllowNull:=False), PXML("SleepTimerOnPostsLimit"), ControlNumber(10)>
         Friend ReadOnly Property SleepTimerOnPostsLimit As PropertyValue
         <Provider(NameOf(SleepTimerOnPostsLimit), FieldsChecker:=True)>
         Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
-        <PropertyOption(ControlText:="Get stories"), PXML, ControlNumber(9)>
+        <PropertyOption(ControlText:="Get stories"), PXML, ControlNumber(11)>
         Friend ReadOnly Property GetStories As PropertyValue
-        <PropertyOption(ControlText:="Get tagged photos"), PXML, ControlNumber(10)>
+        <PropertyOption(ControlText:="Get tagged photos"), PXML, ControlNumber(12)>
         Friend ReadOnly Property GetTagged As PropertyValue
         <PropertyOption(ControlText:="Tagged notify limit",
                         ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
-                        "-1 to disable"), PXML, ControlNumber(11)>
+                        "-1 to disable"), PXML, ControlNumber(13)>
         Friend ReadOnly Property TaggedNotifyLimit As PropertyValue
         <Provider(NameOf(TaggedNotifyLimit), FieldsChecker:=True)>
         Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
 #End Region
 #Region "Download ready"
-        Friend ReadOnly Property DownloadTimeline As XMLValue(Of Boolean)
+        <PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline, stories and saved posts"), PXML, ControlNumber(6)>
-        Friend ReadOnly Property DownloadStoriesTagged As XMLValue(Of Boolean)
+        Friend ReadOnly Property DownloadTimeline As PropertyValue
-        Friend ReadOnly Property DownloadSaved As XMLValue(Of Boolean)
+        <PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts"), PXML, ControlNumber(7)>
+        Friend ReadOnly Property DownloadTagged As PropertyValue
 #End Region
 #Region "429 bypass"
         Private ReadOnly Property DownloadingErrorDate As XMLValue(Of Date)
@@ -202,12 +200,13 @@ Namespace API.Instagram
 
             With Responser
                 If .Headers.Count > 0 Then
-                    With .Headers
+                    token = .Headers.Value(Header_CSRF_TOKEN)
-                        If .ContainsKey(Header_CSRF_TOKEN) Then token = .Item(Header_CSRF_TOKEN)
+                    app_id = .Headers.Value(Header_IG_APP_ID)
-                        If .ContainsKey(Header_IG_APP_ID) Then app_id = .Item(Header_IG_APP_ID)
+                    www_claim = .Headers.Value(Header_IG_WWW_CLAIM)
-                        If .ContainsKey(Header_IG_WWW_CLAIM) Then www_claim = .Item(Header_IG_WWW_CLAIM)
-                    End With
                 End If
+                .CookiesExtractMode = Responser.CookiesExtractModes.Response
+                .CookiesUpdateMode = CookieKeeper.UpdateModes.ReplaceByNameAll
+                .CookiesExtractedAutoSave = False
                 If Not .Cookies Is Nothing Then
                     .Cookies.ChangedAllowInternalDrop = False
                     .Cookies.Changed = False
@@ -216,17 +215,13 @@ Namespace API.Instagram
 
             Dim n() As String = {SettingsCLS.Name_Node_Sites, Site.ToString}
 
-            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
+            HashTagged = New PropertyValue(String.Empty, GetType(String))
-
-            Hash = New PropertyValue(String.Empty, GetType(String))
-            HashSavedPosts = New PropertyValue(String.Empty, GetType(String))
             CSRF_TOKEN = New PropertyValue(token, GetType(String), Sub(v) ChangeResponserFields(NameOf(CSRF_TOKEN), v))
             IG_APP_ID = New PropertyValue(app_id, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_APP_ID), v))
             IG_WWW_CLAIM = New PropertyValue(www_claim, GetType(String), Sub(v) ChangeResponserFields(NameOf(IG_WWW_CLAIM), v))
 
-            DownloadTimeline = New XMLValue(Of Boolean)("DownloadTimeline", True, _XML, n)
+            DownloadTimeline = New PropertyValue(True)
-            DownloadStoriesTagged = New XMLValue(Of Boolean)("DownloadStoriesTagged", True, _XML, n)
+            DownloadTagged = New PropertyValue(False)
-            DownloadSaved = New XMLValue(Of Boolean)("DownloadSaved", True, _XML, n)
 
             RequestsWaitTimer = New PropertyValue(1000)
             RequestsWaitTimerProvider = New TimersChecker(100)
@@ -253,7 +248,6 @@ Namespace API.Instagram
         End Sub
         Private Structure LatestValues
             Friend Hash As String
-            Friend Hash2 As String
             Friend Token As String
             Friend AppID As String
             Friend WwwClaim As String
@@ -261,80 +255,41 @@ Namespace API.Instagram
             Friend Sub New(ByVal Source As SiteSettings)
                 Exists = True
                 With Source
-                    Hash = AConvert(Of String)(.Hash.Value, String.Empty)
+                    Hash = AConvert(Of String)(.HashTagged.Value, String.Empty)
-                    Hash2 = AConvert(Of String)(.HashSavedPosts.Value, String.Empty)
+                    With .Responser
-                    With .Responser.Headers
+                        Token = .Headers.Value(Header_CSRF_TOKEN)
-                        If .ContainsKey(Header_CSRF_TOKEN) Then Token = .Item(Header_CSRF_TOKEN)
+                        AppID = .Headers.Value(Header_IG_APP_ID)
-                        If .ContainsKey(Header_IG_APP_ID) Then AppID = .Item(Header_IG_APP_ID)
+                        WwwClaim = .Headers.Value(Header_IG_WWW_CLAIM)
-                        If .ContainsKey(Header_IG_WWW_CLAIM) Then WwwClaim = .Item(Header_IG_WWW_CLAIM)
                     End With
                 End With
             End Sub
         End Structure
         Private LV As LatestValues = Nothing
-        Private ASO As SettingsExchangeOptions = Nothing
         Friend Overrides Sub BeginEdit()
             LV = New LatestValues(Me)
-            ASO = Nothing
             MyBase.BeginEdit()
         End Sub
         Friend Overrides Sub EndEdit()
             LV = Nothing
-            ASO = Nothing
             MyBase.EndEdit()
         End Sub
-        Friend Overrides Sub Update()
+        Friend Overloads Overrides Sub Update()
             If LV.Exists Then
                 Dim __lv As New LatestValues(Me)
                 If If(Responser.Cookies?.Count, 0) > 0 Then
-                    Dim _cookiesChanged As Boolean = If(Responser.Cookies?.Changed, False)
+                    Dim _cookiesChanged As Boolean = If(Responser.Cookies?.Changed, False) And Responser.CookiesExists
-                    If Not DownloadTimeline AndAlso (_cookiesChanged Or
+                    Dim _tokensChanged As Boolean = (Not __lv.Token.IsEmptyString And Not __lv.WwwClaim.IsEmptyString And Not __lv.AppID.IsEmptyString) And
-                       (Not LV.Hash = __lv.Hash And Not __lv.Hash.IsEmptyString)) Then DownloadTimeline.Value = True
+                        (Not LV.Token = __lv.Token Or Not LV.WwwClaim = __lv.WwwClaim Or Not LV.AppID = __lv.AppID)
-                    If Not DownloadSaved AndAlso (_cookiesChanged Or (Not LV.Hash2 = __lv.Hash2 And Not __lv.Hash2.IsEmptyString)) Then DownloadSaved.Value = True
+                    If _cookiesChanged Or _tokensChanged Then DownloadTimeline.Value = True
-                    If Not DownloadStoriesTagged AndAlso (
+                    If Not __lv.Hash.IsEmptyString And (_cookiesChanged Or _tokensChanged Or Not LV.Hash = __lv.Hash) Then DownloadTagged.Value = True
-                           _cookiesChanged Or (
-                                (Not LV.Hash = __lv.Hash Or Not LV.Token = __lv.Token Or Not LV.AppID = __lv.AppID Or Not LV.WwwClaim = __lv.WwwClaim) And
-                                (Not __lv.Hash.IsEmptyString And Not __lv.Token.IsEmptyString And Not __lv.AppID.IsEmptyString And Not __lv.WwwClaim.IsEmptyString)
-                            )) Then DownloadStoriesTagged.Value = True
                 End If
             End If
-            If ASO.Changed Then
-                DownloadTimeline.Value = ASO.DownloadTimeline
-                DownloadStoriesTagged.Value = ASO.DownloadStoriesTagged
-                DownloadSaved.Value = ASO.DownloadSaved
-            End If
             LV = Nothing
-            ASO = Nothing
             If Not Responser.Cookies Is Nothing Then Responser.Cookies.Changed = False
             MyBase.Update()
         End Sub
 #End Region
 #Region "PropertiesDataChecker"
-        <PropertiesDataChecker({NameOf(Hash), NameOf(HashSavedPosts)})>
-        Private Function CheckHashControls(ByVal p As IEnumerable(Of PropertyData)) As Boolean
-            If p.ListExists(2) Then
-                Dim h$ = String.Empty
-                Dim hsp$ = String.Empty
-                For Each pp As PropertyData In p
-                    Select Case pp.Name
-                        Case NameOf(Hash) : h = AConvert(Of String)(pp.Value, String.Empty)
-                        Case NameOf(HashSavedPosts) : hsp = AConvert(Of String)(pp.Value, String.Empty)
-                    End Select
-                Next
-                If h.IsEmptyString And hsp.IsEmptyString Then
-                    Return True
-                Else
-                    If h = hsp Then
-                        MsgBoxE({"InstaHash for saved posts must be different from InstaHash!", "InstaHash are equal"}, vbCritical)
-                        Return False
-                    Else
-                        Return True
-                    End If
-                End If
-            Else
-                Return False
-            End If
-        End Function
         <PropertiesDataChecker({NameOf(TaggedNotifyLimit)})>
         Private Function CheckNotifyLimit(ByVal p As IEnumerable(Of PropertyData)) As Boolean
             If p.ListExists Then
@@ -353,37 +308,6 @@ Namespace API.Instagram
             End If
             Return False
         End Function
-        <PropertiesDataChecker({NameOf(HashSavedPosts), NameOf(SavedPostsUserName)})>
-        Private Function CheckSavedOptions(ByVal p As IEnumerable(Of PropertyData)) As Boolean
-            If p.ListExists Then
-                Const MsgTitle$ = "Saved posts credentials"
-                Dim __hash$ = String.Empty
-                Dim __name$ = String.Empty
-                Dim _OptionlErrorText$ = $"For download saved posts, you must to set both [{HashSavedPosts_Text}] and [{SavedPostsUserName_Text}]."
-                For i% = 0 To p.Count - 1
-                    Select Case p(i).Name
-                        Case NameOf(HashSavedPosts) : __hash = p(i).Value
-                        Case NameOf(SavedPostsUserName) : __name = p(i).Value
-                    End Select
-                Next
-                If __hash = __name Then
-                    If __hash.IsEmptyString Then
-                        Return True
-                    Else
-                        MsgBoxE({$"[{HashSavedPosts_Text}] and [{SavedPostsUserName_Text}] for saved posts cannot be equal!", MsgTitle}, vbCritical)
-                    End If
-                Else
-                    If __hash.IsEmptyString Then
-                        MsgBoxE({$"[{HashSavedPosts_Text}] not set.{vbCr}{_OptionlErrorText}", MsgTitle}, vbCritical)
-                    ElseIf __name.IsEmptyString Then
-                        MsgBoxE({$"[{SavedPostsUserName_Text}] not set.{vbCr}{_OptionlErrorText}", MsgTitle}, vbCritical)
-                    Else
-                        Return True
-                    End If
-                End If
-            End If
-            Return False
-        End Function
 #End Region
 #Region "Plugin functions"
         Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
@@ -391,7 +315,7 @@ Namespace API.Instagram
                 Case Download.Main : Return New UserData
                 Case Download.SavedPosts
                     Dim u As New UserData
-                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}
+                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = Site}
                     Return u
             End Select
             Return Nothing
@@ -399,13 +323,7 @@ Namespace API.Instagram
 #Region "Downloading"
         Friend Property SkipUntilNextSession As Boolean = False
         Friend Overrides Function ReadyToDownload(ByVal What As Download) As Boolean
-            If ActiveJobs < 2 AndAlso Not SkipUntilNextSession AndAlso ReadyForDownload AndAlso BaseAuthExists() Then
+            Return ActiveJobs < 2 AndAlso Not SkipUntilNextSession AndAlso ReadyForDownload AndAlso BaseAuthExists() AndAlso DownloadTimeline.Value
-                Select Case What
-                    Case Download.Main : Return ACheck(Hash.Value) And DownloadTimeline
-                    Case Download.SavedPosts : Return ACheck(HashSavedPosts.Value) And DownloadSaved
-                End Select
-            End If
-            Return False
         End Function
         Private ActiveJobs As Integer = 0
         Private _NextWNM As UserData.WNM = UserData.WNM.Notify
@@ -434,6 +352,10 @@ Namespace API.Instagram
                 _NextTagged = .TaggedCheckSession
                 LastDownloadDate.Value = Now
                 LastRequestsCount.Value = .RequestsCount
+                _FieldsChangerSuspended = True
+                IG_WWW_CLAIM.Value = Responser.Headers.Value(Header_IG_WWW_CLAIM)
+                CSRF_TOKEN.Value = Responser.Headers.Value(Header_CSRF_TOKEN)
+                _FieldsChangerSuspended = False
             End With
         End Sub
         Friend Overrides Sub DownloadDone(ByVal What As Download)
@@ -453,12 +375,14 @@ Namespace API.Instagram
                 Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
-        Friend Overrides Sub OpenSettingsForm()
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Using f As New AdditionalSettingsForm(If(ASO.Changed, ASO, New SettingsExchangeOptions(Me)))
+            Try
-                f.ShowDialog()
+                Dim code$ = DirectCast(User, UserData).GetPostCodeById(Media.Post.ID)
-                If f.DialogResult = DialogResult.OK Then ASO = f.MyParameters
+                If Not code.IsEmptyString Then Return $"https://instagram.com/p/{code}/" Else Return String.Empty
-            End Using
+            Catch ex As Exception
-        End Sub
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "Can't open user's post", String.Empty)
+            End Try
+        End Function
 #End Region
     End Class
 End Namespace

SCrawler/API/Instagram/UserData.vb

@@ -9,10 +9,11 @@
 Imports System.Net
 Imports System.Threading
 Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.Messaging
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports SCrawler.API.Base
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Instagram
@@ -25,12 +26,48 @@ Namespace API.Instagram
         Private Const Name_TaggedChecked As String = "TaggedChecked"
 #End Region
 #Region "Declarations"
+        Private Structure PostKV : Implements IEContainerProvider
+            Private Const Name_Code As String = "Code"
+            Private Const Name_Section As String = "Section"
+            Friend Code As String
+            Friend ID As String
+            Friend Section As Sections
+            Friend Sub New(ByVal _Section As Sections)
+                Section = _Section
+            End Sub
+            Friend Sub New(ByVal _Code As String, ByVal _ID As String, ByVal _Section As Sections)
+                Code = _Code
+                ID = _ID
+                Section = _Section
+            End Sub
+            Private Sub New(ByVal e As EContainer)
+                Code = e.Attribute(Name_Code)
+                Section = e.Attribute(Name_Section)
+                ID = e.Value
+            End Sub
+            Public Shared Widening Operator CType(ByVal e As EContainer) As PostKV
+                Return New PostKV(e)
+            End Operator
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                If Not IsNothing(Obj) AndAlso TypeOf Obj Is PostKV Then
+                    With DirectCast(Obj, PostKV)
+                        Return Code = .Code And ID = .ID And Section = .Section
+                    End With
+                Else
+                    Return False
+                End If
+            End Function
+            Private Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer
+                Return New EContainer("Post", ID, {New EAttribute(Name_Section, CInt(Section)), New EAttribute(Name_Code, Code)})
+            End Function
+        End Structure
         Private ReadOnly Property MySiteSettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
             End Get
         End Property
-        Private ReadOnly _SavedPostsIDs As New List(Of String)
+        Private ReadOnly PostsKVIDs As List(Of PostKV)
+        Private ReadOnly PostsToReparse As List(Of PostKV)
         Private LastCursor As String = String.Empty
         Private FirstLoadingDone As Boolean = False
         Friend Property GetStories As Boolean
@@ -51,6 +88,8 @@ Namespace API.Instagram
 #End Region
 #Region "Initializer, loader"
         Friend Sub New()
+            PostsKVIDs = New List(Of PostKV)
+            PostsToReparse = New List(Of PostKV)
         End Sub
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
             If Loading Then
@@ -78,48 +117,127 @@ Namespace API.Instagram
                 End If
                 Throw New ExitException
             End Sub
-            Friend Sub New()
-            End Sub
-            Friend Sub New(ByRef CompleteArg As Boolean)
-                CompleteArg = True
-            End Sub
         End Class
+        Private Sub LoadSavePostsKV(ByVal Load As Boolean)
+            Dim x As XmlFile
+            Dim f As SFile = MyFilePosts
+            If Not f.IsEmptyString Then
+                f.Name &= "_KV"
+                f.Extension = "xml"
+                If Load Then
+                    PostsKVIDs.Clear()
+                    x = New XmlFile(f, Protector.Modes.All, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                    x.LoadData()
+                    If x.Count > 0 Then PostsKVIDs.ListAddList(x, LAP.IgnoreICopier)
+                    x.Dispose()
+                Else
+                    x = New XmlFile With {.AllowSameNames = True}
+                    x.AddRange(PostsKVIDs)
+                    x.Name = "Posts"
+                    x.Save(f, EDP.SendInLog)
+                    x.Dispose()
+                End If
+            End If
+        End Sub
+        Private Overloads Function PostKvExists(ByVal pkv As PostKV) As Boolean
+            Return PostKvExists(pkv.ID, False, pkv.Section) OrElse PostKvExists(pkv.Code, True, pkv.Section)
+        End Function
+        Private Overloads Function PostKvExists(ByVal PostCodeId As String, ByVal IsCode As Boolean, ByVal Section As Sections) As Boolean
+            If Not PostCodeId.IsEmptyString And PostsKVIDs.Count > 0 Then
+                If PostsKVIDs.FindIndex(Function(p) p.Section = Section AndAlso If(IsCode, p.Code = PostCodeId, p.ID = PostCodeId)) >= 0 Then
+                    Return True
+                ElseIf Not IsCode Then
+                    Return _TempPostsList.Contains(GetPostIdBySection(PostCodeId, Section)) Or
+                           _TempPostsList.Contains(PostCodeId.Replace($"_{ID}", String.Empty)) Or
+                           _TempPostsList.Contains(GetPostIdBySection(PostCodeId.Replace($"_{ID}", String.Empty), Section))
+                End If
+            End If
+            Return False
+        End Function
+        Friend Function GetPostCodeById(ByVal PostID As String) As String
+            Try
+                If Not PostID.IsEmptyString Then
+                    Dim f As SFile = MyFilePosts
+                    If Not f.IsEmptyString Then
+                        f.Name &= "_KV"
+                        f.Extension = "xml"
+                        Dim l As List(Of PostKV) = Nothing
+                        Using x As New XmlFile(f, Protector.Modes.All, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                            x.LoadData()
+                            l.ListAddList(x, LAP.IgnoreICopier)
+                        End Using
+                        Dim code$ = String.Empty
+                        If l.ListExists Then
+                            Dim i% = l.FindIndex(Function(p) p.ID = PostID)
+                            If i >= 0 Then code = l(i).Code
+                            l.Clear()
+                        End If
+                        Return code
+                    End If
+                End If
+                Return String.Empty
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{ToStringForLog()}: Cannot find post code by ID ({PostID})", String.Empty)
+            End Try
+        End Function
+        Private Function GetPostIdBySection(ByVal ID As String, ByVal Section As Sections) As String
+            If Section = Sections.Timeline Then
+                Return ID
+            Else
+                Return $"{Section}_{ID}"
+            End If
+        End Function
+        Private _DownloadingInProgress As Boolean = False
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             Dim s As Sections = Sections.Timeline
+            Dim errorFound As Boolean = False
             Try
+                LoadSavePostsKV(True)
+                _DownloadingInProgress = True
+                AddHandler Responser.ResponseReceived, AddressOf Responser_ResponseReceived
                 ThrowAny(Token)
-                _InstaHash = String.Empty
                 HasError = False
-                Dim fc As Boolean = IIf(IsSavedPosts, MySiteSettings.DownloadSaved.Value, MySiteSettings.DownloadTimeline.Value)
+                If CBool(MySiteSettings.DownloadTimeline.Value) And Not LastCursor.IsEmptyString Then
-                If fc And Not LastCursor.IsEmptyString Then
                     s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
                     DownloadData(LastCursor, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
-                If fc And Not HasError Then
+                If CBool(MySiteSettings.DownloadTimeline.Value) And Not HasError Then
                     s = IIf(IsSavedPosts, Sections.SavedPosts, Sections.Timeline)
                     DownloadData(String.Empty, s, Token)
                     ThrowAny(Token)
                     If Not HasError Then FirstLoadingDone = True
                 End If
                 If FirstLoadingDone Then LastCursor = String.Empty
-                If IsSavedPosts Then
+                If MySiteSettings.BaseAuthExists() Then
-                    If MySiteSettings.DownloadSaved Then s = Sections.SavedPosts : DownloadPosts(Token)
+                    If CBool(MySiteSettings.DownloadTimeline.Value) And GetStories Then s = Sections.Stories : DownloadData(String.Empty, s, Token)
-                ElseIf MySiteSettings.BaseAuthExists() Then
+                    If CBool(MySiteSettings.DownloadTagged.Value) And ACheck(MySiteSettings.HashTagged.Value) And GetTaggedData Then s = Sections.Tagged : DownloadData(String.Empty, s, Token)
-                    DownloadedTags = 0
-                    If MySiteSettings.DownloadStoriesTagged And GetStories Then s = Sections.Stories : DownloadData(String.Empty, s, Token)
-                    If MySiteSettings.DownloadStoriesTagged And GetTaggedData Then s = Sections.Tagged : DownloadData(String.Empty, s, Token)
                 End If
                 If WaitNotificationMode = WNM.SkipTemp Or WaitNotificationMode = WNM.SkipCurrent Then WaitNotificationMode = WNM.Notify
             Catch eex As ExitException
             Catch ex As Exception
-                ProcessException(ex, Token, "[API.Instagram.UserData.DownloadDataF]", False, s)
+                errorFound = True
+                Throw ex
             Finally
                 E560Thrown = False
+                UpdateResponser()
+                If Not errorFound Then LoadSavePostsKV(False)
             End Try
         End Sub
-        Private _InstaHash As String = String.Empty
+        Private Sub UpdateResponser()
+            Try
+                If _DownloadingInProgress AndAlso Not Responser Is Nothing AndAlso Not Responser.Disposed Then
+                    _DownloadingInProgress = False
+                    RemoveHandler Responser.ResponseReceived, AddressOf Responser_ResponseReceived
+                    Declarations.UpdateResponser(Responser, MySiteSettings.Responser)
+                End If
+            Catch
+            End Try
+        End Sub
+        Private Sub Responser_ResponseReceived(ByVal Sender As Object, ByVal e As EventArguments.WebDataResponse)
+            Declarations.UpdateResponser(e, Responser)
+        End Sub
         Private Enum Sections : Timeline : Tagged : Stories : SavedPosts : End Enum
         Private Const StoriesFolder As String = "Stories"
         Private Const TaggedFolder As String = "Tagged"
@@ -145,7 +263,7 @@ Namespace API.Instagram
                                               "What do you want to do?", "Waiting for Instagram download...",
                                               {
                                                New MsgBoxButton("Wait") With {.ToolTip = "Wait and ask again when the error is found."},
-                                               New MsgBoxButton("Wait (disable current") With {.ToolTip = "Wait and skip future prompts while downloading the current profile."},
+                                               New MsgBoxButton("Wait (disable current)") With {.ToolTip = "Wait and skip future prompts while downloading the current profile."},
                                                New MsgBoxButton("Abort") With {.ToolTip = "Abort operation"},
                                                New MsgBoxButton("Wait (disable all)") With {.ToolTip = "Wait and skip future prompts while downloading the current session."}
                                               },
@@ -180,13 +298,11 @@ Namespace API.Instagram
 #Region "Tags"
         Private TaggedChecked As Boolean = False
         Friend TaggedCheckSession As Boolean = True
-        Private DownloadedTags As Integer = 0
         Private DownloadTagsLimit As Integer? = Nothing
-        Private ReadOnly Property TaggedLimitsNotifications(Optional ByVal v As Integer? = Nothing) As Boolean
+        Private ReadOnly Property TaggedLimitsNotifications(ByVal v As Integer) As Boolean
             Get
                 Return Not TaggedChecked AndAlso TaggedCheckSession AndAlso
-                       CInt(MySiteSettings.TaggedNotifyLimit.Value) > 0 AndAlso
+                       CInt(MySiteSettings.TaggedNotifyLimit.Value) > 0 AndAlso v > CInt(MySiteSettings.TaggedNotifyLimit.Value)
-                       (Not v.HasValue OrElse v.Value > CInt(MySiteSettings.TaggedNotifyLimit.Value))
             End Get
         End Property
         Private Function SetTagsLimit(ByVal Max As Integer, ByVal p As ANumbers) As DialogResult
@@ -219,8 +335,9 @@ Namespace API.Instagram
         End Function
         Private Function TaggedContinue(ByVal TaggedCount As Integer) As DialogResult
             Dim agi As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
-            Dim msg As New MMessage($"The number of tagged posts by user [{ToString()}] is {TaggedCount.NumToString(agi)}" & vbCr &
+            Dim msg As New MMessage($"The number of already downloaded tagged posts by user [{ToString()}] is {TaggedCount.NumToString(agi)}" & vbCr &
-                                    $"This is about {(TaggedCount / 12).RoundUp.NumToString(agi)} requests." & vbCr &
+                                    "There is currently no way to know how many posts exist." & vbCr &
+                                    "One request will be spent per post." & vbCr &
                                     "The tagged data download operation can take a long time.",
                                     "Too much tagged data",
                                     {
@@ -252,40 +369,46 @@ Namespace API.Instagram
             Dim URL$ = String.Empty
             Dim StoriesList As List(Of String) = Nothing
             Dim StoriesRequested As Boolean = False
-            Dim _DownloadComplete As Boolean = False
+            Dim dValue% = 1
             LastCursor = Cursor
             Try
-                Do While Not _DownloadComplete
+                Do While dValue = 1
                     ThrowAny(Token)
                     If Not Ready() Then Thread.Sleep(10000) : ThrowAny(Token) : Continue Do
                     ReconfigureAwaiter()
 
                     Try
-                        Dim n As EContainer, nn As EContainer, node As EContainer
+                        Dim n As EContainer, nn As EContainer
                         Dim HasNextPage As Boolean = False
-                        Dim Pinned As Boolean
                         Dim EndCursor$ = String.Empty
                         Dim PostID$ = String.Empty, PostDate$ = String.Empty, SpecFolder$ = String.Empty
-                        Dim TaggedCount%
+                        Dim PostIDKV As PostKV
                         Dim ENode() As Object = Nothing
                         NextRequest(True)
 
                         'Check environment
-                        If Cursor.IsEmptyString And _InstaHash.IsEmptyString Then _
+                        If Not IsSavedPosts Then
-                           _InstaHash = CStr(If(IsSavedPosts, MySiteSettings.HashSavedPosts, MySiteSettings.Hash).Value)
+                            If ID.IsEmptyString Then GetUserId()
-                        If ID.IsEmptyString Then GetUserId()
+                            If ID.IsEmptyString Then Throw New ArgumentException("User ID is not detected", "ID")
-                        If ID.IsEmptyString Then Throw New ArgumentException("User ID is not detected", "ID")
+                        End If
+
 
                         'Create query
                         Select Case Section
-                            Case Sections.Timeline, Sections.SavedPosts
+                            Case Sections.Timeline
+                                URL = $"https://www.instagram.com/api/v1/feed/user/{Name}/username/?count=50" &
+                                        If(Cursor.IsEmptyString, String.Empty, $"&max_id={Cursor}")
+                                ENode = Nothing
+                            Case Sections.SavedPosts
+                                SavedPostsDownload(String.Empty, Token)
+                                Exit Sub
+                            Case Sections.Tagged
+                                Dim h$ = AConvert(Of String)(MySiteSettings.HashTagged.Value, String.Empty)
+                                If h.IsEmptyString Then Throw New ExitException
                                 Dim vars$ = "{""id"":" & ID & ",""first"":50,""after"":""" & Cursor & """}"
                                 vars = SymbolsConverter.ASCII.EncodeSymbolsOnly(vars)
-                                URL = $"https://www.instagram.com/graphql/query/?query_hash={_InstaHash}&variables={vars}"
+                                URL = $"https://www.instagram.com/graphql/query/?query_hash={h}&variables={vars}"
                                 ENode = {"data", "user", 0}
-                            Case Sections.Tagged
-                                URL = $"https://i.instagram.com/api/v1/usertags/{ID}/feed/?count=50&max_id={Cursor}"
-                                ENode = {"items"}
                                 SpecFolder = TaggedFolder
                             Case Sections.Stories
                                 If Not StoriesRequested Then
@@ -303,7 +426,7 @@ Namespace API.Instagram
                                 If StoriesList.ListExists Then
                                     Continue Do
                                 Else
-                                    Throw New ExitException(_DownloadComplete)
+                                    Throw New ExitException
                                 End If
                         End Select
 
@@ -316,78 +439,73 @@ Namespace API.Instagram
                         'Parsing
                         If Not r.IsEmptyString Then
                             Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
-                                n = j.ItemF(ENode).XmlIfNothing
+                                n = If(ENode Is Nothing, j, j.ItemF(ENode)).XmlIfNothing
                                 If n.Count > 0 Then
                                     Select Case Section
-                                        Case Sections.Timeline, Sections.SavedPosts
+                                        Case Sections.Timeline
-                                            If n.Contains("page_info") Then
+                                            With n
-                                                With n("page_info")
+                                                HasNextPage = .Value("more_available").FromXML(Of Boolean)(False)
-                                                    HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
+                                                EndCursor = .Value("next_max_id")
-                                                    EndCursor = .Value("end_cursor")
+                                                If If(.Item("items")?.Count, 0) > 0 Then
-                                                End With
+                                                    If Not DefaultParser(.Item("items"), Section, Token) Then Throw New ExitException
-                                            End If
+                                                Else
-                                            n = n("edges").XmlIfNothing
+                                                    HasNextPage = False
-                                            If n.Count > 0 Then
+                                                End If
-                                                For Each nn In n
+                                            End With
-                                                    ThrowAny(Token)
-                                                    node = nn(0).XmlIfNothing
-                                                    If IsSavedPosts Then
-                                                        PostID = node.Value("shortcode")
-                                                        If Not PostID.IsEmptyString AndAlso _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
-                                                    End If
-                                                    PostID = node.Value("id")
-                                                    Pinned = CBool(If(node("pinned_for_users")?.Count, 0))
-                                                    If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) And Not Pinned Then Throw New ExitException(_DownloadComplete)
-                                                    _TempPostsList.Add(PostID)
-                                                    PostDate = node.Value("taken_at_timestamp")
-                                                    If IsSavedPosts Then
-                                                        _SavedPostsIDs.Add(PostID)
-                                                    Else
-                                                        Select Case CheckDatesLimit(PostDate, DateProvider)
-                                                            Case DateResult.Skip : Continue For
-                                                            Case DateResult.Exit : If Not Pinned Then Throw New ExitException(_DownloadComplete)
-                                                        End Select
-                                                        ObtainMedia(node, PostID, PostDate, SpecFolder)
-                                                    End If
-                                                Next
-                                            End If
                                         Case Sections.Tagged
-                                            HasNextPage = j.Value("more_available").FromXML(Of Boolean)(False)
+                                            With n
-                                            EndCursor = j.Value("next_max_id")
+                                                If .Contains("page_info") Then
-                                            For Each nn In n
+                                                    With .Item("page_info")
-                                                PostID = $"Tagged_{nn.Value("id")}"
+                                                        HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
-                                                If Not PostID.IsEmptyString And _TempPostsList.Contains(PostID) Then Throw New ExitException(_DownloadComplete)
+                                                        EndCursor = .Value("end_cursor")
-                                                _TempPostsList.Add(PostID)
+                                                    End With
-                                                ObtainMedia2(nn, PostID, SpecFolder)
+                                                Else
-                                                DownloadedTags += 1
+                                                    HasNextPage = False
-                                                If DownloadTagsLimit.HasValue AndAlso DownloadedTags >= DownloadTagsLimit.Value Then Throw New ExitException(_DownloadComplete)
+                                                End If
-                                            Next
+                                                If If(.Item("edges")?.Count, 0) > 0 Then
-                                            If TaggedLimitsNotifications Then
+                                                    For Each nn In .Item("edges")
-                                                TaggedCount = j.Value("total_count").FromXML(Of Integer)(0)
+                                                        PostIDKV = New PostKV(Section)
+                                                        If nn.Count > 0 AndAlso nn(0).Count > 0 Then
+                                                            With nn(0)
+                                                                PostIDKV = New PostKV(.Value("shortcode"), .Value("id"), Section)
+                                                                If PostKvExists(PostIDKV) Then
+                                                                    Throw New ExitException
+                                                                Else
+                                                                    If Not DownloadTagsLimit.HasValue OrElse PostsToReparse.Count + 1 < DownloadTagsLimit.Value Then
+                                                                        _TempPostsList.Add(GetPostIdBySection(PostIDKV.ID, Section))
+                                                                        PostsKVIDs.ListAddValue(PostIDKV, LAP.NotContainsOnly)
+                                                                        PostsToReparse.ListAddValue(PostIDKV, LNC)
+                                                                    ElseIf DownloadTagsLimit.HasValue OrElse PostsToReparse.Count + 1 >= DownloadTagsLimit.Value Then
+                                                                        Throw New ExitException
+                                                                    End If
+                                                                End If
+                                                            End With
+                                                        End If
+                                                    Next
+                                                Else
+                                                    HasNextPage = False
+                                                End If
+                                            End With
+                                            If TaggedLimitsNotifications(PostsToReparse.Count) Then
                                                 TaggedChecked = True
-                                                If TaggedLimitsNotifications(TaggedCount) AndAlso
+                                                If TaggedContinue(PostsToReparse.Count) = DialogResult.Cancel Then Throw New ExitException
-                                                   TaggedContinue(TaggedCount) = DialogResult.Cancel Then Throw New ExitException(_DownloadComplete)
                                             End If
                                     End Select
                                 Else
-                                    If j.Value("status") = "ok" AndAlso j({"data", "user"}).XmlIfNothing.Count = 0 AndAlso
+                                    If j.Value("status") = "ok" AndAlso If(j("items")?.Count, 0) = 0 AndAlso
                                        _TempMediaList.Count = 0 AndAlso Section = Sections.Timeline Then _
-                                       UserExists = False : Throw New ExitException(_DownloadComplete)
+                                       UserExists = False : Throw New ExitException
                                 End If
                             End Using
                         Else
-                            Throw New ExitException(_DownloadComplete)
+                            Throw New ExitException
                         End If
-                        _DownloadComplete = True
+                        dValue = 0
                         If HasNextPage And Not EndCursor.IsEmptyString Then DownloadData(EndCursor, Section, Token)
                     Catch eex As ExitException
                         Throw eex
-                    Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                        Exit Do
-                    Catch dex As ObjectDisposedException When Disposed
-                        Exit Do
                     Catch ex As Exception
-                        If DownloadingException(ex, $"data downloading error [{URL}]", False, Section) = 1 Then Continue Do Else Exit Do
+                        dValue = ProcessException(ex, Token, $"data downloading error [{URL}]",, Section, False)
                     End Try
                 Loop
             Catch eex2 As ExitException
@@ -400,10 +518,10 @@ Namespace API.Instagram
         End Sub
         Private Sub DownloadPosts(ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
-            Dim _DownloadComplete As Boolean = False
+            Dim dValue% = 1
             Dim _Index% = 0
             Try
-                Do While Not _DownloadComplete
+                Do While dValue = 1
                     ThrowAny(Token)
                     If Not Ready() Then Thread.Sleep(10000) : ThrowAny(Token) : Continue Do
                     ReconfigureAwaiter()
@@ -411,13 +529,11 @@ Namespace API.Instagram
                     Try
                         Dim r$
                         Dim j As EContainer, jj As EContainer
-                        Dim _MediaObtained As Boolean
+                        If PostsToReparse.Count > 0 And _Index <= PostsToReparse.Count - 1 Then
-                        If _SavedPostsIDs.Count > 0 And _Index <= _SavedPostsIDs.Count - 1 Then
                             Dim e As New ErrorsDescriber(EDP.ThrowException)
-                            For i% = _Index To _SavedPostsIDs.Count - 1
+                            For i% = _Index To PostsToReparse.Count - 1
                                 _Index = i
-                                'URL = $"https://instagram.com/p/{_SavedPostsIDs(i)}/?__a=1"
+                                URL = $"https://www.instagram.com/api/v1/media/{PostsToReparse(i).ID}/info/"
-                                URL = $"https://i.instagram.com/api/v1/media/{_SavedPostsIDs(i)}/info/"
                                 ThrowAny(Token)
                                 NextRequest(((i + 1) Mod 5) = 0)
                                 ThrowAny(Token)
@@ -427,17 +543,9 @@ Namespace API.Instagram
                                 If Not r.IsEmptyString Then
                                     j = JsonDocument.Parse(r)
                                     If Not j Is Nothing Then
-                                        _MediaObtained = False
+                                        If If(j("items")?.Count, 0) > 0 Then
-                                        If j.Contains({"graphql", "shortcode_media"}) Then
-                                            With j({"graphql", "shortcode_media"}).XmlIfNothing
-                                                If .Count > 0 Then ObtainMedia(.Self, _SavedPostsIDs(i), String.Empty, String.Empty) : _MediaObtained = True
-                                            End With
-                                        End If
-                                        If Not _MediaObtained AndAlso j.Contains("items") Then
                                             With j("items")
-                                                If .Count > 0 Then
+                                                For Each jj In .Self : ObtainMedia(jj, PostsToReparse(i).ID) : Next
-                                                    For Each jj In .Self : ObtainMedia2(jj, _SavedPostsIDs(i)) : Next
-                                                End If
                                             End With
                                         End If
                                         j.Dispose()
@@ -445,24 +553,79 @@ Namespace API.Instagram
                                 End If
                             Next
                         End If
-                        _DownloadComplete = True
+                        dValue = 0
                     Catch eex As ExitException
                         Throw eex
-                    Catch oex As OperationCanceledException When Token.IsCancellationRequested
-                        Exit Do
-                    Catch dex As ObjectDisposedException When Disposed
-                        Exit Do
                     Catch ex As Exception
-                        If DownloadingException(ex, $"downloading saved posts error [{URL}]", False, Sections.SavedPosts) = 1 Then Continue Do Else Exit Do
+                        dValue = ProcessException(ex, Token, $"downloading posts error [{URL}]",, Sections.Tagged, False)
                     End Try
                 Loop
             Catch eex2 As ExitException
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
-                ProcessException(DoEx, Token, $"downloading saved posts error [{URL}]",, Sections.SavedPosts)
+                ProcessException(DoEx, Token, $"downloading posts error [{URL}]",, Sections.Tagged)
             End Try
         End Sub
+        Private Sub SavedPostsDownload(ByVal Cursor As String, ByVal Token As CancellationToken)
+            Dim URL$ = $"https://www.instagram.com/api/v1/feed/saved/posts/?max_id={Cursor}"
+            Dim HasNextPage As Boolean = False
+            Dim NextCursor$ = String.Empty
+            ThrowAny(Token)
+            Dim r$ = Responser.GetResponse(URL)
+            Dim nodes As IEnumerable(Of EContainer) = Nothing
+            If Not r.IsEmptyString Then
+                Using e As EContainer = JsonDocument.Parse(r)
+                    If If(e?.Count, 0) > 0 Then
+                        With e
+                            HasNextPage = .Value("more_available").FromXML(Of Boolean)(False)
+                            NextCursor = .Value("next_max_id")
+                            If .Contains("items") Then nodes = (From ee As EContainer In .Item("items") Where ee.Count > 0 Select ee(0))
+                        End With
+                        If nodes.ListExists Then
+                            DefaultParser(nodes, Sections.SavedPosts, Token)
+                            If HasNextPage And Not NextCursor.IsEmptyString Then SavedPostsDownload(NextCursor, Token)
+                        End If
+                    End If
+                End Using
+            End If
+        End Sub
+        Private Function DefaultParser(ByVal Items As IEnumerable(Of EContainer), ByVal Section As Sections, ByVal Token As CancellationToken,
+                                       Optional ByVal SpecFolder As String = Nothing) As Boolean
+            ThrowAny(Token)
+            If Items.Count > 0 Then
+                Dim PostIDKV As PostKV
+                Dim Pinned As Boolean
+                Dim PostDate$
+                If SpecFolder.IsEmptyString Then
+                    Select Case Section
+                        Case Sections.Tagged : SpecFolder = TaggedFolder
+                        Case Sections.Stories : SpecFolder = StoriesFolder
+                        Case Else : SpecFolder = String.Empty
+                    End Select
+                End If
+                For Each nn In Items
+                    With nn
+                        PostIDKV = New PostKV(.Value("code"), .Value("id"), Section)
+                        Pinned = .Contains("timeline_pinned_user_ids")
+                        If PostKvExists(PostIDKV) And Not Pinned Then Return False
+                        _TempPostsList.Add(PostIDKV.ID)
+                        PostsKVIDs.ListAddValue(PostIDKV, LAP.NotContainsOnly)
+                        PostDate = .Value("taken_at")
+                        If Not IsSavedPosts Then
+                            Select Case CheckDatesLimit(PostDate, DateProvider)
+                                Case DateResult.Skip : Continue For
+                                Case DateResult.Exit : If Not Pinned Then Return False
+                            End Select
+                        End If
+                        ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate)
+                    End With
+                Next
+                Return True
+            Else
+                Return False
+            End If
+        End Function
 #End Region
 #Region "Code ID converters"
         Private Shared Function CodeToID(ByVal Code As String) As String
@@ -485,31 +648,14 @@ Namespace API.Instagram
         End Function
 #End Region
 #Region "Obtain Media"
-        Private Sub ObtainMedia(ByVal node As EContainer, ByVal PostID As String, ByVal PostDate As String, ByVal SpecFolder As String)
+        Private Sub ObtainMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
-            Dim CreateMedia As Action(Of EContainer) =
-                Sub(ByVal e As EContainer)
-                    Dim t As UTypes = If(e.Value("is_video").FromXML(Of Boolean)(False), UTypes.Video, UTypes.Picture)
-                    Dim tmpValue$
-                    If t = UTypes.Picture Then
-                        tmpValue = e.Value("display_url")
-                    Else
-                        tmpValue = e.Value("video_url")
-                    End If
-                    If Not tmpValue.IsEmptyString Then _TempMediaList.ListAddValue(MediaFromData(t, tmpValue, PostID, PostDate, SpecFolder), LNC)
-                End Sub
-            If node.Contains({"edge_sidecar_to_children", "edges"}) Then
-                For Each edge As EContainer In node({"edge_sidecar_to_children", "edges"}) : CreateMedia(edge("node").XmlIfNothing) : Next
-            Else
-                CreateMedia(node)
-            End If
-        End Sub
-        Private Sub ObtainMedia2(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
                                  Optional ByVal DateObj As String = Nothing)
             Try
                 Dim img As Predicate(Of EContainer) = Function(_img) Not _img.Name.IsEmptyString AndAlso _img.Name.StartsWith("image_versions") AndAlso _img.Count > 0
                 Dim vid As Predicate(Of EContainer) = Function(_vid) Not _vid.Name.IsEmptyString AndAlso _vid.Name.StartsWith("video_versions") AndAlso _vid.Count > 0
                 Dim ss As Func(Of EContainer, Sizes) = Function(_ss) New Sizes(_ss.Value("width"), _ss.Value("url"))
                 Dim mDate As Func(Of EContainer, String) = Function(ByVal elem As EContainer) As String
+                                                               If Not DateObj.IsEmptyString Then Return DateObj
                                                                If elem.Contains("taken_at") Then
                                                                    Return elem.Value("taken_at")
                                                                ElseIf elem.Contains("imported_taken_at") Then
@@ -518,7 +664,7 @@ Namespace API.Instagram
                                                                    Dim ev$ = elem.Value("device_timestamp")
                                                                    If Not ev.IsEmptyString Then
                                                                        If ev.Length > 10 Then
-                                                                           Return elem.Value("device_timestamp").Substring(0, 10)
+                                                                           Return ev.Substring(0, 10)
                                                                        Else
                                                                            Return ev
                                                                        End If
@@ -574,7 +720,7 @@ Namespace API.Instagram
                                 DateObj = mDate(n)
                                 With n("carousel_media").XmlIfNothing
                                     If .Count > 0 Then
-                                        For Each d In .Self : ObtainMedia2(d, PostID, SpecialFolder, DateObj) : Next
+                                        For Each d In .Self : ObtainMedia(d, PostID, SpecialFolder, DateObj) : Next
                                     End If
                                 End With
                         End Select
@@ -587,22 +733,6 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "GetUserId"
-        <Obsolete> Private Sub GetUserId_Old()
-            Try
-                Dim r$ = Responser.GetResponse($"https://www.instagram.com/{Name}/?__a=1",, EDP.ThrowException)
-                If Not r.IsEmptyString Then
-                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
-                        ID = j({"graphql", "user"}, "id").XmlIfNothingValue
-                    End Using
-                End If
-            Catch ex As Exception
-                If Responser.StatusCode = HttpStatusCode.NotFound Or Responser.StatusCode = HttpStatusCode.BadRequest Then
-                    Throw ex
-                Else
-                    LogError(ex, "get Instagram user id")
-                End If
-            End Try
-        End Sub
         Private Sub GetUserId()
             Try
                 Dim r$ = Responser.GetResponse($"https://i.instagram.com/api/v1/users/web_profile_info/?username={Name}",, EDP.ThrowException)
@@ -651,7 +781,7 @@ Namespace API.Instagram
                                                 pid = storyID & s.Value("id")
                                                 If Not _TempPostsList.Contains(pid) Then
                                                     ThrowAny(Token)
-                                                    ObtainMedia2(s, pid, sFolder)
+                                                    ObtainMedia(s, pid, sFolder)
                                                     _TempPostsList.Add(pid)
                                                 End If
                                             Next
@@ -730,8 +860,7 @@ Namespace API.Instagram
                 Dim s As Sections = DirectCast(Section, Sections)
                 Select Case s
                     Case Sections.Timeline : MySiteSettings.DownloadTimeline.Value = False
-                    Case Sections.SavedPosts : MySiteSettings.DownloadSaved.Value = False
+                    Case Else : MySiteSettings.DownloadTagged.Value = False
-                    Case Else : MySiteSettings.DownloadStoriesTagged.Value = False
                 End Select
                 MyMainLOG = $"[{s}] downloading is disabled until you update your credentials".ToUpper
             End If
@@ -749,7 +878,7 @@ Namespace API.Instagram
         End Function
 #End Region
 #Region "Standalone downloader"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Response) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Responser) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString AndAlso URL.Contains("instagram.com") Then
                     Dim PID$ = RegexReplace(URL, RParams.DMS(".*?instagram.com/p/([_\w\d]+)", 1))
@@ -757,9 +886,9 @@ Namespace API.Instagram
                     If Not PID.IsEmptyString Then
                         Using t As New UserData
                             t.SetEnvironment(Settings(InstagramSiteKey), Nothing, False, False)
-                            t.Responser = New Response
+                            t.Responser = New Responser
                             t.Responser.Copy(r)
-                            t._SavedPostsIDs.Add(PID)
+                            t.PostsToReparse.Add(New PostKV With {.ID = PID})
                             t.DownloadPosts(Nothing)
                             Return ListAddList(Nothing, t._TempMediaList)
                         End Using
@@ -773,7 +902,13 @@ Namespace API.Instagram
 #End Region
 #Region "IDisposable Support"
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            If Not disposedValue And disposing Then _SavedPostsIDs.Clear()
+            If Not disposedValue Then
+                UpdateResponser()
+                If disposing Then
+                    PostsKVIDs.Clear()
+                    PostsToReparse.Clear()
+                End If
+            End If
             MyBase.Dispose(disposing)
         End Sub
 #End Region

SCrawler/API/LPSG/UserData.vb

@@ -10,6 +10,7 @@ Imports System.Threading
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports Converters = PersonalUtilities.Functions.SymbolsConverter.Converters
 Namespace API.LPSG
@@ -27,7 +28,7 @@ Namespace API.LPSG
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
             Try
-                Responser.Error = EDP.ThrowException
+                Responser.DeclaredError = EDP.ThrowException
 
                 Dim NextPage$
                 Dim r$
@@ -87,7 +88,7 @@ Namespace API.LPSG
             End If
         End Sub
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
-            With Responser : .UseWebClient = True : .UseWebClientCookies = True : .ResetError() : End With
+            With Responser : .Mode = Responser.Modes.WebClient : .ResetStatus() : End With
             UseResponserClient = True
             DownloadContentDefault(Token)
         End Sub

SCrawler/API/PornHub/Declarations.vb

@@ -0,0 +1,44 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.PornHub
+    Friend Module Declarations
+#Region "Converters"
+        Private ReadOnly UnicodeHexConverter As Func(Of String, String) = Function(Input) SymbolsConverter.UnicodeHex.Decode(Input, EDP.ReturnValue)
+        Friend ReadOnly HtmlConverter As Func(Of String, String) = Function(Input) SymbolsConverter.HTML.Decode(Input, EDP.ReturnValue)
+#End Region
+#Region "Declarations video"
+        Friend ReadOnly RegexVideo_FlashVarsBlock As RParams = RParams.DM("(?<=flashvars_\['[nN]ext[vV]ideo'\];[\r\n]*?)(.+?)(?=;flashvars_\d+?)", 0, EDP.ReturnValue)
+        Friend ReadOnly RegexVideo_FlashVars_Vars As RParams = RParams.DM("var ([\w\d]{10,})=("".+?)(?=(;|\Z))", 0, RegexReturn.List)
+        Friend ReadOnly RegexVideo_FlashVars_Compiler As RParams = RParams.DM("(?<=\*/)([\w\d\S]{10,})", 0, RegexReturn.List)
+        Friend ReadOnly RegexVideo_Video_All As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""",
+                                                                     0, RegexReturn.List, EDP.ReturnValue, UnicodeHexConverter)
+        Friend ReadOnly RegexVideo_Video_Wrong As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""[\w\W\s\r\n]+?(?=\<div class=""videoUploaderBlock)",
+                                                                       0, RegexReturn.List, EDP.ReturnValue, UnicodeHexConverter)
+        Private ReadOnly RegexVideo_Video_Wrong_Option As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""[\w\W\s\r\n]+?", 0, RegexReturn.ListByMatch)
+        Friend ReadOnly RegexVideo_Video_Wrong_Fields As RField() = {New RField(New RFieldOption(1, RegexVideo_Video_Wrong_Option)), New RField(New RFieldOption(2, RegexVideo_Video_Wrong_Option))}
+        Friend ReadOnly RegexVideo_Video_VideoKey As RParams = RParams.DMS("viewkey=([\w\d]+)", 1, EDP.ReturnValue)
+#End Region
+#Region "Declarations M3U8"
+        Friend ReadOnly Regex_M3U8_FirstFileRegEx As RParams = RParams.DM(".+?m3u8.*", 0)
+        Friend ReadOnly Regex_M3U8_FileUrl As RParams = RParams.DMS("((https://([^/]+)/.+?)([^/]+?m3u8))(.*)", 2, EDP.ReturnValue)
+#End Region
+#Region "Declarations GIF"
+        Friend ReadOnly Regex_Gif_Array As RParams = RParams.DM("\<li id=""(gif\d+)"" class=""gifLi.gifVideoBlock""\>", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly Regex_Gif_UrlName As RParams = RParams.DMS("""name"":.*?""([^""]*)""[^\}]+?""contentUrl"":.*?""([^""]+)""", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
+#End Region
+#Region "Declarations photo"
+        Friend ReadOnly Regex_Photo_ModelHub_PhotoBlocks As RParams = RParams.DM("var PHOTOS_ARRAY_(\d+) = \{[\r\n\s]*?(urls:.*?\[[^]]*\])", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly Regex_Photo_PornHub_PhotoBlocks As RParams = RParams.DM("photoAlbumListContainer[\r\n\s\S]+?title=""([^""]+)""[\r\n\s\S]+?a href=""(/album/\d+)""", 0, RegexReturn.List)
+        Friend ReadOnly Regex_Photo_PornHub_AlbumPhotoArr As RParams = RParams.DMS("\<a href=""(/photo/\d+)""", 1, RegexReturn.List, EDP.ReturnValue,
+                                                                                   CType(Function(Input$) If(Input.IsEmptyString, String.Empty, $"https://www.pornhub.com{Input.Trim}"), Func(Of String, String)))
+        Friend ReadOnly Regex_Photo_PornHub_SinglePhoto As RParams = RParams.DMS("(?<!thumbImage.+?)<img src=""(https://[^""]+\d+[^""]+)""", 1, EDP.ReturnValue)
+#End Region
+    End Module
+End Namespace

SCrawler/API/PornHub/M3U8.vb

@@ -0,0 +1,42 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.API.Base.M3U8Declarations
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.PornHub
+    Friend NotInheritable Class M3U8
+        Private Sub New()
+        End Sub
+        Private Shared Function GetUrlsList(ByVal URL As String, ByVal Responser As Responser) As List(Of String)
+            Dim appender$ = RegexReplace(URL, Regex_M3U8_FileUrl)
+            Dim r$ = Responser.GetResponse(URL)
+            If Not r.IsEmptyString Then
+                Dim file$ = RegexReplace(r, Regex_M3U8_FirstFileRegEx)
+                If Not file.IsEmptyString Then
+                    Dim NewUrl$ = M3U8Base.CreateUrl(appender, file)
+                    If Not NewUrl.IsEmptyString Then
+                        r = Responser.GetResponse(NewUrl)
+                        If Not r.IsEmptyString Then
+                            Dim l As List(Of String) = RegexReplace(r, TsFilesRegEx)
+                            If l.ListExists Then
+                                For i% = 0 To l.Count - 1 : l(i) = M3U8Base.CreateUrl(appender, l(i)) : Next
+                                Return l
+                            End If
+                        End If
+                    End If
+                End If
+            End If
+            Return Nothing
+        End Function
+        Friend Shared Function Download(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As SFile
+            Return M3U8Base.Download(GetUrlsList(URL, Responser), Destination, Responser)
+        End Function
+    End Class
+End Namespace

SCrawler/API/PornHub/OptionsForm.Designer.vb

@@ -6,9 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Namespace API.Instagram
+Namespace API.PornHub
     <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
-    Partial Friend Class AdditionalSettingsForm : Inherits System.Windows.Forms.Form
+    Partial Friend Class OptionsForm : Inherits System.Windows.Forms.Form
         <System.Diagnostics.DebuggerNonUserCode()>
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)
             Try
@@ -24,9 +24,8 @@ Namespace API.Instagram
         Private Sub InitializeComponent()
             Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
             Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
-            Me.CH_DOWN_TIME = New System.Windows.Forms.CheckBox()
+            Me.CH_DOWN_GIFS = New System.Windows.Forms.CheckBox()
-            Me.CH_DOWN_TAG = New System.Windows.Forms.CheckBox()
+            Me.CH_DOWN_PHOTO_MODELHUB = New System.Windows.Forms.CheckBox()
-            Me.CH_DOWN_SAVED = New System.Windows.Forms.CheckBox()
             CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
             TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
             CONTAINER_MAIN.ContentPanel.SuspendLayout()
@@ -40,13 +39,13 @@ Namespace API.Instagram
             'CONTAINER_MAIN.ContentPanel
             '
             CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(234, 78)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(278, 52)
             CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
             CONTAINER_MAIN.LeftToolStripPanelVisible = False
             CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
             CONTAINER_MAIN.Name = "CONTAINER_MAIN"
             CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(234, 103)
+            CONTAINER_MAIN.Size = New System.Drawing.Size(278, 77)
             CONTAINER_MAIN.TabIndex = 0
             CONTAINER_MAIN.TopToolStripPanelVisible = False
             '
@@ -55,71 +54,56 @@ Namespace API.Instagram
             TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
             TP_MAIN.ColumnCount = 1
             TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 20.0!))
+            TP_MAIN.Controls.Add(Me.CH_DOWN_GIFS, 0, 0)
-            TP_MAIN.Controls.Add(Me.CH_DOWN_TIME, 0, 0)
+            TP_MAIN.Controls.Add(Me.CH_DOWN_PHOTO_MODELHUB, 0, 1)
-            TP_MAIN.Controls.Add(Me.CH_DOWN_TAG, 0, 1)
-            TP_MAIN.Controls.Add(Me.CH_DOWN_SAVED, 0, 2)
             TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
             TP_MAIN.Location = New System.Drawing.Point(0, 0)
             TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 4
+            TP_MAIN.RowCount = 3
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(234, 78)
+            TP_MAIN.Size = New System.Drawing.Size(278, 52)
             TP_MAIN.TabIndex = 0
             '
-            'CH_DOWN_TIME
+            'CH_DOWN_GIFS
             '
-            Me.CH_DOWN_TIME.AutoSize = True
+            Me.CH_DOWN_GIFS.AutoSize = True
-            Me.CH_DOWN_TIME.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_GIFS.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_TIME.Location = New System.Drawing.Point(4, 4)
+            Me.CH_DOWN_GIFS.Location = New System.Drawing.Point(4, 4)
-            Me.CH_DOWN_TIME.Name = "CH_DOWN_TIME"
+            Me.CH_DOWN_GIFS.Name = "CH_DOWN_GIFS"
-            Me.CH_DOWN_TIME.Size = New System.Drawing.Size(226, 19)
+            Me.CH_DOWN_GIFS.Size = New System.Drawing.Size(270, 19)
-            Me.CH_DOWN_TIME.TabIndex = 0
+            Me.CH_DOWN_GIFS.TabIndex = 0
-            Me.CH_DOWN_TIME.Text = "Download Timeline"
+            Me.CH_DOWN_GIFS.Text = "Download gifs"
-            Me.CH_DOWN_TIME.UseVisualStyleBackColor = True
+            Me.CH_DOWN_GIFS.UseVisualStyleBackColor = True
             '
-            'CH_DOWN_TAG
+            'CH_DOWN_PHOTO_MODELHUB
             '
-            Me.CH_DOWN_TAG.AutoSize = True
+            Me.CH_DOWN_PHOTO_MODELHUB.AutoSize = True
-            Me.CH_DOWN_TAG.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_DOWN_PHOTO_MODELHUB.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_TAG.Location = New System.Drawing.Point(4, 30)
+            Me.CH_DOWN_PHOTO_MODELHUB.Location = New System.Drawing.Point(4, 30)
-            Me.CH_DOWN_TAG.Name = "CH_DOWN_TAG"
+            Me.CH_DOWN_PHOTO_MODELHUB.Name = "CH_DOWN_PHOTO_MODELHUB"
-            Me.CH_DOWN_TAG.Size = New System.Drawing.Size(226, 19)
+            Me.CH_DOWN_PHOTO_MODELHUB.Size = New System.Drawing.Size(270, 19)
-            Me.CH_DOWN_TAG.TabIndex = 1
+            Me.CH_DOWN_PHOTO_MODELHUB.TabIndex = 1
-            Me.CH_DOWN_TAG.Text = "Download Stories and Tagged data"
+            Me.CH_DOWN_PHOTO_MODELHUB.Text = "Download photo only from ModelHub"
-            Me.CH_DOWN_TAG.UseVisualStyleBackColor = True
+            Me.CH_DOWN_PHOTO_MODELHUB.UseVisualStyleBackColor = True
             '
-            'CH_DOWN_SAVED
+            'OptionsForm
-            '
-            Me.CH_DOWN_SAVED.AutoSize = True
-            Me.CH_DOWN_SAVED.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_SAVED.Location = New System.Drawing.Point(4, 56)
-            Me.CH_DOWN_SAVED.Name = "CH_DOWN_SAVED"
-            Me.CH_DOWN_SAVED.Size = New System.Drawing.Size(226, 19)
-            Me.CH_DOWN_SAVED.TabIndex = 2
-            Me.CH_DOWN_SAVED.Text = "Download saved posts"
-            Me.CH_DOWN_SAVED.UseVisualStyleBackColor = True
-            '
-            'AdditionalSettingsForm
             '
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
             Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(234, 103)
+            Me.ClientSize = New System.Drawing.Size(278, 77)
             Me.Controls.Add(CONTAINER_MAIN)
             Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
             Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
             Me.KeyPreview = True
             Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(250, 142)
+            Me.MaximumSize = New System.Drawing.Size(294, 116)
             Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(250, 142)
+            Me.MinimumSize = New System.Drawing.Size(294, 116)
-            Me.Name = "AdditionalSettingsForm"
+            Me.Name = "OptionsForm"
-            Me.ShowInTaskbar = False
             Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Additional settings"
+            Me.Text = "Options"
             CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
             CONTAINER_MAIN.ResumeLayout(False)
             CONTAINER_MAIN.PerformLayout()
@@ -128,8 +112,7 @@ Namespace API.Instagram
             Me.ResumeLayout(False)
 
         End Sub
-        Private WithEvents CH_DOWN_TIME As CheckBox
+        Private WithEvents CH_DOWN_GIFS As CheckBox
-        Private WithEvents CH_DOWN_TAG As CheckBox
+        Private WithEvents CH_DOWN_PHOTO_MODELHUB As CheckBox
-        Private WithEvents CH_DOWN_SAVED As CheckBox
     End Class
 End Namespace

SCrawler/API/PornHub/OptionsForm.vb

@@ -7,34 +7,27 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports PersonalUtilities.Forms
-Namespace API.Instagram
+Namespace API.PornHub
-    Friend Class AdditionalSettingsForm
+    Friend Class OptionsForm
         Private WithEvents MyDefs As DefaultFormOptions
-        Friend Property MyParameters As SettingsExchangeOptions
+        Private ReadOnly MyExchangeOptions As UserExchangeOptions
-        Friend Sub New(ByVal Parameters As SettingsExchangeOptions)
+        Friend Sub New(ByRef ExchangeOptions As UserExchangeOptions)
             InitializeComponent()
-            MyParameters = Parameters
+            MyExchangeOptions = ExchangeOptions
             MyDefs = New DefaultFormOptions(Me, Settings.Design)
         End Sub
         Private Sub MyForm_Load(sender As Object, e As EventArgs) Handles Me.Load
             With MyDefs
                 .MyViewInitialize(True)
                 .AddOkCancelToolbar()
-                With MyParameters
+                CH_DOWN_GIFS.Checked = MyExchangeOptions.DownloadGifs
-                    CH_DOWN_TIME.Checked = .DownloadTimeline
+                CH_DOWN_PHOTO_MODELHUB.Checked = MyExchangeOptions.DownloadPhotoOnlyFromModelHub
-                    CH_DOWN_TAG.Checked = .DownloadStoriesTagged
-                    CH_DOWN_SAVED.Checked = .DownloadSaved
-                End With
                 .EndLoaderOperations()
             End With
         End Sub
         Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
-            MyParameters = New SettingsExchangeOptions With {
+            MyExchangeOptions.DownloadGifs = CH_DOWN_GIFS.Checked
-                .DownloadTimeline = CH_DOWN_TIME.Checked,
+            MyExchangeOptions.DownloadPhotoOnlyFromModelHub = CH_DOWN_PHOTO_MODELHUB.Checked
-                .DownloadStoriesTagged = CH_DOWN_TAG.Checked,
-                .DownloadSaved = CH_DOWN_SAVED.Checked,
-                .Changed = True
-            }
             MyDefs.CloseForm()
         End Sub
     End Class

SCrawler/API/PornHub/SiteSettings.vb

@@ -0,0 +1,120 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.PornHub
+    <Manifest("AndyProgram_PornHub"), SavedPosts, SpecialForm(False), SeparatedTasks(1)>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.PornHubIcon_16
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.PornHubPic_16
+            End Get
+        End Property
+        Private ReadOnly Property CurlPathExists As Boolean
+        <PropertyOption(ControlText:="Download GIF", ControlToolTip:="Default for new users", ThreeStates:=True), PXML>
+        Friend ReadOnly Property DownloadGifs As PropertyValue
+        <PropertyOption(ControlText:="Download GIFs as mp4", ControlToolTip:="Download gifs in 'mp4' format instead of native 'webm'"), PXML>
+        Friend ReadOnly Property DownloadGifsAsMp4 As PropertyValue
+        <PropertyOption(ControlText:="Photo ModelHub only",
+                        ControlToolTip:="Download photo only from ModelHub. Prornstar photos hosted on PornHub itself will not be downloaded." & vbCr &
+                                        "Attention! Downloading photos hosted on PornHub is a very heavy job."), PXML>
+        Friend ReadOnly Property DownloadPhotoOnlyFromModelHub As PropertyValue
+        <PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
+        Friend ReadOnly Property SavedPostsUserName As PropertyValue
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("PornHub", "pornhub.com")
+            Responser.CurlPath = $"cURL\curl.exe"
+            CurlPathExists = Responser.CurlPath.Exists
+            Responser.DeclaredError = EDP.ThrowException
+
+            DownloadGifsAsMp4 = New PropertyValue(True)
+            DownloadGifs = New PropertyValue(CInt(CheckState.Indeterminate), GetType(Integer))
+            DownloadPhotoOnlyFromModelHub = New PropertyValue(True)
+            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
+
+            UrlPatternUser = "https://www.pornhub.com/{0}/{1}"
+            UserRegex = RParams.DMS("pornhub.com/([^/]+)/([^/]+).*?", 0, RegexReturn.ListByMatch)
+            ImageVideoContains = "pornhub"
+        End Sub
+#End Region
+#Region "GetInstance, GetSpecialData"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            If What = ISiteSettings.Download.SavedPosts Then
+                Return New UserData With {
+                    .IsSavedPosts = True,
+                    .VideoPageModel = UserData.VideoPageModels.Favorite,
+                    .PersonType = UserData.PersonTypeUser,
+                    .User = New UserInfo With {.Name = $"{UserData.PersonTypeUser}_{CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}"}
+                }
+            Else
+                Return New UserData
+            End If
+        End Function
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            If Available(ISiteSettings.Download.Main, True) Then
+                Using resp As Responser = Responser.Copy
+                    Dim spf$ = String.Empty
+                    Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
+                    Dim m As UserMedia = UserData.GetVideoInfo(URL, resp, f)
+                    If m.State = UserMedia.States.Downloaded Then
+                        m.SpecialFolder = f
+                        Return {m}
+                    End If
+                End Using
+            End If
+            Return Nothing
+        End Function
+#End Region
+#Region "Downloading"
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return Settings.UseM3U8 And CurlPathExists And (Not What = ISiteSettings.Download.SavedPosts OrElse ACheck(SavedPostsUserName.Value))
+        End Function
+#End Region
+#Region "IsMyUser"
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            Try
+                If Not UserURL.IsEmptyString Then
+                    Dim alist As List(Of String) = RegexReplace(UserURL.ToLower, UserRegex)
+                    If alist.ListExists(3) Then Return New ExchangeOptions(Site, $"{alist(1)}_{alist(2)}")
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.PornHub.SiteSettings.IsMyUser({UserURL})]", New ExchangeOptions)
+            End Try
+        End Function
+#End Region
+#Region "GetUserUrl, GetUserPostUrl"
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
+            With DirectCast(User, UserData) : Return String.Format(UrlPatternUser, .PersonType, .NameTrue) : End With
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return Media.URL_BASE
+        End Function
+#End Region
+#Region "User options"
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse Not TypeOf Options Is UserExchangeOptions Then Options = New UserExchangeOptions(Me)
+            If OpenForm Then
+                Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
+            End If
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/PornHub/UserData.vb

@@ -0,0 +1,656 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Namespace API.PornHub
+    Friend Class UserData : Inherits UserDataBase
+        Private Const UrlPattern As String = "https://www.pornhub.com/{0}"
+#Region "Declarations"
+#Region "XML names"
+        Private Const Name_PersonType As String = "PersonType"
+        Private Const Name_NameTrue As String = "NameTrue"
+        Private Const Name_VideoPageModel As String = "VideoPageModel"
+        Private Const Name_PhotoPageModel As String = "PhotoPageModel"
+        Private Const Name_DownloadGifs As String = "DownloadGifs"
+        Private Const Name_DownloadPhotoOnlyFromModelHub As String = "DownloadPhotoOnlyFromModelHub"
+#End Region
+#Region "Structures"
+        Private Structure FlashVar : Implements IRegExCreator
+            Friend Name As String
+            Friend Value As String
+            Public Shared Widening Operator CType(ByVal Name As String) As FlashVar
+                Return New FlashVar With {.Name = Name}
+            End Operator
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    Name = ParamsArray(0)
+                    Value = ParamsArray(1)
+                    If Not Value.IsEmptyString Then Value = Value.Replace(""" + """, String.Empty).Replace("""", String.Empty).StringTrim
+                End If
+                Return Me
+            End Function
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                Return CType(Obj, FlashVar).Name = Name
+            End Function
+        End Structure
+        Private Structure UserVideo : Implements IRegExCreator
+            Friend URL As String
+            Friend ID As String
+            Friend Title As String
+            Friend Function ToUserMedia() As UserMedia
+                Return New UserMedia(URL, UTypes.VideoPre) With {
+                    .File = If(Title.IsEmptyString, .File, New SFile($"{Title}.mp4")),
+                    .Post = ID
+                }
+            End Function
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists Then
+                    URL = ParamsArray(0)
+                    ID = RegexReplace(URL, RegexVideo_Video_VideoKey)
+                    URL = String.Format(UrlPattern, URL.TrimStart("/"))
+                    Title = HtmlConverter(ParamsArray(1)).StringRemoveWinForbiddenSymbols.StringTrim
+                End If
+                Return Me
+            End Function
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                Return DirectCast(Obj, UserVideo).URL = URL
+            End Function
+        End Structure
+        Private Structure PhotoBlock : Implements IRegExCreator
+            Friend AlbumID As String
+            Friend Data As String
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    AlbumID = ParamsArray(0)
+                    Data = ParamsArray(1).StringTrim
+                End If
+                Return Me
+            End Function
+        End Structure
+#End Region
+#Region "Enums"
+        Friend Enum VideoPageModels As Integer
+            [Default] = 0
+            ConcatPage = 1
+            Favorite = 2
+            Undefined = -1
+        End Enum
+        Private Enum PhotoPageModels As Integer
+            Undefined = 0
+            PornHubPage = 1
+            ModelHubPage = 2
+        End Enum
+#End Region
+#Region "Constants"
+        Private Const PersonTypeModel As String = "model"
+        Friend Const PersonTypeUser As String = "users"
+#End Region
+#Region "Person"
+        Friend Property PersonType As String
+        Friend Property NameTrue As String
+        Private _FriendlyName As String = String.Empty
+        Friend Overrides Property FriendlyName As String
+            Get
+                If _FriendlyName.IsEmptyString Then Return NameTrue Else Return _FriendlyName
+            End Get
+            Set(ByVal n As String)
+                _FriendlyName = n
+            End Set
+        End Property
+#End Region
+#Region "Advanced fields"
+        Friend Property VideoPageModel As VideoPageModels = VideoPageModels.Undefined
+        Private Property PhotoPageModel As PhotoPageModels = PhotoPageModels.Undefined
+        Friend Property DownloadGifs As Boolean
+        Friend Property DownloadPhotoOnlyFromModelHub As Boolean = True
+#End Region
+#Region "ExchangeOptions"
+        Friend Overrides Function ExchangeOptionsGet() As Object
+            Return New UserExchangeOptions(Me)
+        End Function
+        Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
+            If Not Obj Is Nothing AndAlso TypeOf Obj Is UserExchangeOptions Then
+                With DirectCast(Obj, UserExchangeOptions)
+                    DownloadGifs = .DownloadGifs
+                    DownloadPhotoOnlyFromModelHub = .DownloadPhotoOnlyFromModelHub
+                End With
+            End If
+        End Sub
+#End Region
+        Private ReadOnly Property MySettings As SiteSettings
+            Get
+                Return DirectCast(HOST.Source, SiteSettings)
+            End Get
+        End Property
+#End Region
+#Region "Initializer, loader"
+        Friend Sub New()
+            UseInternalM3U8Function = True
+        End Sub
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            With Container
+                Dim SetNames As Action = Sub()
+                                             If Not Name.IsEmptyString And NameTrue.IsEmptyString Then
+                                                 Dim n$() = Name.Split("_")
+                                                 If n.ListExists(2) Then
+                                                     NameTrue = Name.Replace($"{n(0)}_", String.Empty)
+                                                     PersonType = n(0)
+                                                     If (PersonType = PersonTypeModel Or PersonType = PersonTypeUser) And
+                                                        VideoPageModel = VideoPageModels.Undefined Then VideoPageModel = VideoPageModels.Default
+                                                 End If
+                                             End If
+                                         End Sub
+                If Loading Then
+                    PersonType = .Value(Name_PersonType)
+                    NameTrue = .Value(Name_NameTrue)
+                    VideoPageModel = .Value(Name_VideoPageModel).FromXML(Of Integer)(VideoPageModels.Undefined)
+                    PhotoPageModel = .Value(Name_PhotoPageModel).FromXML(Of Integer)(PhotoPageModels.Undefined)
+                    DownloadGifs = .Value(Name_DownloadGifs).FromXML(Of Integer)(False)
+                    DownloadPhotoOnlyFromModelHub = .Value(Name_DownloadPhotoOnlyFromModelHub).FromXML(Of Boolean)(True)
+                    SetNames.Invoke()
+                Else
+                    SetNames.Invoke()
+                    .Add(Name_PersonType, PersonType)
+                    .Add(Name_NameTrue, NameTrue)
+                    .Add(Name_VideoPageModel, CInt(VideoPageModel))
+                    .Add(Name_PhotoPageModel, CInt(PhotoPageModel))
+                    .Add(Name_DownloadGifs, DownloadGifs.BoolToInteger)
+                    .Add(Name_DownloadPhotoOnlyFromModelHub, DownloadPhotoOnlyFromModelHub.BoolToInteger)
+                End If
+            End With
+        End Sub
+#End Region
+#Region "Downloading"
+#Region "Download override"
+        Private Const DataDownloaded As Integer = -10
+        Private Const DataDownloaded_NotFound As Integer = -20
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Try
+                Responser.ResetStatus()
+                If PersonType = PersonTypeUser Then Responser.Mode = Responser.Modes.Curl
+
+                If IsSavedPosts Then VideoPageModel = VideoPageModels.Favorite
+
+                Dim page% = 1
+                Dim __continue As Boolean = True
+                Dim __videoDone As Boolean = False
+                Dim d%
+                If DownloadVideos Then
+                    If PersonType = PersonTypeUser Then Responser.Mode = Responser.Modes.Curl : Responser.Method = "POST"
+                    If VideoPageModel = VideoPageModels.Undefined Then
+                        __continue = False
+                        d = DownloadUserVideos(page, Token)
+                        Select Case d
+                            Case DataDownloaded : __continue = True : page += 1
+                            Case 1 : VideoPageModel = VideoPageModels.ConcatPage
+                            Case EXCEPTION_OPERATION_CANCELED : ThrowAny(Token)
+                            Case DataDownloaded_NotFound : __videoDone = True
+                        End Select
+                        If Not __continue And Not __videoDone Then
+                            d = DownloadUserVideos(page, Token)
+                            Select Case d
+                                Case DataDownloaded : __continue = True : page += 1
+                                Case 1 : VideoPageModel = VideoPageModels.Undefined
+                                Case EXCEPTION_OPERATION_CANCELED : ThrowAny(Token)
+                                Case DataDownloaded_NotFound : __videoDone = True
+                            End Select
+                        End If
+                    End If
+                    If __continue And Not __videoDone Then
+                        Do While DownloadUserVideos(page, Token) = DataDownloaded And page < 100 : page += 1 : Loop
+                    End If
+                End If
+
+                Responser.Method = "GET"
+                If DownloadGifs And Not IsSavedPosts Then DownloadUserGifs(Token)
+                If DownloadImages Then DownloadUserPhotos(Token)
+            Finally
+                Responser.Mode = Responser.Modes.Default
+                Responser.Method = "GET"
+            End Try
+        End Sub
+#End Region
+#Region "Download video"
+        Private ReadOnly Property VideoPageType As String
+            Get
+                Select Case VideoPageModel
+                    Case VideoPageModels.Default : Return "/videos/upload"
+                    Case VideoPageModels.Favorite : Return "/videos/favorites/"
+                    Case Else : Return String.Empty
+                End Select
+            End Get
+        End Property
+        Private ReadOnly Property VideoPageAppender As String
+            Get
+                Return If(PersonType = PersonTypeUser, "ajax?o=newest&page=", String.Empty)
+            End Get
+        End Property
+        Private Overloads Function DownloadUserVideos(ByVal Page As Integer, ByVal Token As CancellationToken) As Integer
+            Const VideoUrlPattern$ = "https://www.pornhub.com/{0}/{1}{2}{3}"
+            Const HtmlPageNotFoundVideo$ = "<span>Error Page Not Found</span>"
+            Dim URL$ = String.Empty
+            Try
+                Dim p$
+                If PersonType = PersonTypeUser Then
+                    p = Page
+                Else
+                    p = IIf(Page = 1, String.Empty, $"?page={Page}")
+                End If
+
+                URL = $"{String.Format(VideoUrlPattern, PersonType, NameTrue, VideoPageType, VideoPageAppender)}{p}"
+                ThrowAny(Token)
+
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    If PersonType = PersonTypeUser And r.Contains(HtmlPageNotFoundVideo) Then Return DataDownloaded_NotFound
+                    Dim l As List(Of UserVideo) = RegexFields(Of UserVideo)(r, {RegexVideo_Video_All}, {1, 2})
+                    Dim lw As List(Of UserVideo) = Nothing
+                    If Not PersonType = PersonTypeUser Then RegexFields(Of UserVideo)(r, {RegexVideo_Video_Wrong}, RegexVideo_Video_Wrong_Fields)
+                    If l.ListExists Then
+                        If lw.ListExists Then l.ListWithRemove(lw)
+                        If l.Count > 0 Then
+                            Dim lBefore% = l.Count
+                            l.RemoveAll(Function(ByVal uv As UserVideo) As Boolean
+                                            If Not _TempPostsList.Contains(uv.ID) Then
+                                                _TempPostsList.Add(uv.ID)
+                                                Return False
+                                            Else
+                                                Return True
+                                            End If
+                                        End Function)
+                            If l.Count > 0 Then _TempMediaList.ListAddList(l.Select(Function(uv) uv.ToUserMedia))
+                            If l.Count = lBefore And l.Count > 0 Then Return DataDownloaded
+                        End If
+                    End If
+                End If
+                Return DataDownloaded_NotFound
+            Catch regex_ex As RegexFieldsTextBecameNullException
+                If PersonType = PersonTypeUser Or IsSavedPosts Then
+                    Return DataDownloaded_NotFound
+                Else
+                    Return ProcessException(regex_ex, Token, $"videos downloading error [{URL}]")
+                End If
+            Catch ex As Exception
+                Return ProcessException(ex, Token, $"videos downloading error [{URL}]")
+            End Try
+        End Function
+#End Region
+#Region "Download GIF"
+        Private Sub DownloadUserGifs(ByVal Token As CancellationToken)
+            Dim URL$ = $"https://www.pornhub.com/{PersonType}/{NameTrue}/gifs"
+            Try
+                ThrowAny(Token)
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    Dim n$
+                    Dim m As UserMedia = Nothing
+                    Dim l As List(Of RegexMatchStruct) = RegexFields(Of RegexMatchStruct)(r, {Regex_Gif_Array}, {1})
+                    Dim l2 As List(Of String) = Nothing
+                    Dim l3 As List(Of String) = Nothing
+                    If l.ListExists Then l2 = l.Select(Function(ll) $"gif/{ll.Arr(0).Replace("gif", String.Empty)}").ToList
+                    If l2.ListExists Then
+                        For Each gif$ In l2
+                            If Not _TempPostsList.Contains(gif) Then
+                                _TempPostsList.Add(gif)
+                                URL = $"https://www.pornhub.com/{gif}"
+                                m = New UserMedia(URL, UTypes.Video) With {.Post = gif, .SpecialFolder = "GIFs\"}
+                                ThrowAny(Token)
+                                Try
+                                    r = Responser.GetResponse(URL)
+                                    If Not r.IsEmptyString Then
+                                        If l3.ListExists Then l3.Clear() : l3 = Nothing
+                                        l3 = RegexReplace(r, Regex_Gif_UrlName)
+                                        If l3.ListExists(3) Then
+                                            m.URL = l3(2)
+                                            m.File = m.URL
+                                            n = HtmlConverter(l3(1)).StringRemoveWinForbiddenSymbols.StringTrim
+                                            If MySettings.DownloadGifsAsMp4.Value Then m.File.Extension = "mp4"
+                                            If Not n.IsEmptyString Then m.File.Name = n
+                                        End If
+                                    End If
+                                Catch gif_down_ex As Exception
+                                    m.State = UserMedia.States.Missing
+                                End Try
+                                _TempMediaList.ListAddValue(m)
+                            End If
+                        Next
+                    End If
+                    If l.ListExists Then l.Clear()
+                    If l2.ListExists Then l2.Clear()
+                    If l3.ListExists Then l3.Clear()
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"gifs downloading error [{URL}]")
+            End Try
+        End Sub
+#End Region
+#Region "Download photo"
+        Private Const PhotoUrlPattern_ModelHub As String = "https://www.modelhub.com/{0}/photos"
+        Private Const PhotoUrlPattern_PornHub As String = "https://www.pornhub.com/{0}/{1}/photos"
+        Private Sub DownloadUserPhotos(ByVal Token As CancellationToken)
+            Try
+                If IsSavedPosts Then
+                    DownloadUserPhotos_SavedPosts(Token)
+                ElseIf PersonType = PersonTypeModel Then
+                    If PhotoPageModel = PhotoPageModels.Undefined Then
+                        If DownloadUserPhotos_ModelHub(Token) Then PhotoPageModel = PhotoPageModels.ModelHubPage
+                        ThrowAny(Token)
+                        If PhotoPageModel = PhotoPageModels.Undefined AndAlso Not DownloadPhotoOnlyFromModelHub AndAlso
+                           DownloadUserPhotos_PornHub(Token) Then PhotoPageModel = PhotoPageModels.PornHubPage
+                    Else
+                        Select Case PhotoPageModel
+                            Case PhotoPageModels.ModelHubPage : DownloadUserPhotos_ModelHub(Token)
+                            Case PhotoPageModels.PornHubPage : If Not DownloadPhotoOnlyFromModelHub Then DownloadUserPhotos_PornHub(Token)
+                        End Select
+                    End If
+                ElseIf Not DownloadPhotoOnlyFromModelHub Then
+                    DownloadUserPhotos_PornHub(Token)
+                End If
+                ThrowAny(Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, "photos downloading error")
+            End Try
+        End Sub
+        Private Function DownloadUserPhotos_ModelHub(ByVal Token As CancellationToken) As Boolean
+            Dim URL$ = String.Empty
+            Try
+                Dim jErr As New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue)
+                Dim albumName$
+                If PersonType = PersonTypeModel Then
+                    URL = String.Format(PhotoUrlPattern_ModelHub, NameTrue)
+                    Dim r$ = Responser.GetResponse(URL)
+                    If Not r.IsEmptyString Then
+                        Dim l As List(Of PhotoBlock) = RegexFields(Of PhotoBlock)(r, {Regex_Photo_ModelHub_PhotoBlocks}, {1, 2})
+                        If l.ListExists Then l.RemoveAll(Function(ll) ll.Data.IsEmptyString)
+                        If l.ListExists Then
+                            Dim albumRegex As RParams = RParams.DMS("", 1, EDP.ReturnValue)
+                            For Each block As PhotoBlock In l
+                                If Not _TempPostsList.Contains(block.AlbumID) Then _TempPostsList.Add(block.AlbumID) Else Continue For
+                                albumRegex.Pattern = "<li id=""" & block.AlbumID & """ class=""modelBox"">[\r\n\s]*?<div class=""modelPhoto"">[\r\n\s]*?\<[^\>]*?alt=""([^""]*)"""
+                                albumName = StringTrim(RegexReplace(r, albumRegex))
+                                If albumName.IsEmptyString Then albumName = block.AlbumID
+                                Using j As EContainer = JsonDocument.Parse("{" & block.Data & "}", jErr)
+                                    If Not j Is Nothing Then
+                                        If If(j("urls")?.Count, 0) > 0 Then
+                                            _TempMediaList.ListAddList(j("urls").Select(Function(jj) _
+                                                                       New UserMedia(jj.ItemF({0}).XmlIfNothingValue, UTypes.Picture) With {
+                                                                                     .SpecialFolder = $"Albums\{albumName}\"}), LNC)
+                                        End If
+                                    End If
+                                End Using
+                            Next
+                            l.Clear()
+                        End If
+                    End If
+                End If
+                Return True
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Overloads Function DownloadUserPhotos_PornHub(ByVal Token As CancellationToken) As Boolean
+            Try
+                Dim albumName$
+                Dim page%
+                Dim r$ = Responser.GetResponse(String.Format(PhotoUrlPattern_PornHub, PersonType, NameTrue))
+                If Not r.IsEmptyString Then
+                    Dim l As List(Of PhotoBlock) = RegexFields(Of PhotoBlock)(r, {Regex_Photo_PornHub_PhotoBlocks}, {2, 1})
+                    If l.ListExists Then l.RemoveAll(Function(ll) ll.AlbumID.IsEmptyString)
+                    If l.ListExists Then
+                        For Each block As PhotoBlock In l
+                            If Not _TempPostsList.Contains(block.AlbumID) Then _TempPostsList.Add(block.AlbumID) Else Continue For
+                            albumName = block.Data
+                            If albumName.IsEmptyString Then
+                                albumName = block.AlbumID.Split("/").LastOrDefault.StringTrim
+                            Else
+                                albumName = HtmlConverter(albumName).StringRemoveWinForbiddenSymbols.StringTrim
+                            End If
+                            page = 1
+                            Do While DownloadUserPhotos_PornHub(page, block.AlbumID, albumName, Token) : page += 1 : Loop
+                        Next
+                        l.Clear()
+                    End If
+                End If
+                Return True
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Overloads Function DownloadUserPhotos_PornHub(ByVal Page As Integer, ByVal AlbumID As String, ByVal AlbumName As String,
+                                                              ByVal Token As CancellationToken) As Boolean
+            Try
+                Dim r$ = Responser.GetResponse($"https://www.pornhub.com{AlbumID}{IIf(Page = 1, String.Empty, $"?page={Page}")}")
+                If Not r.IsEmptyString Then
+                    Dim l As List(Of String) = RegexReplace(r, Regex_Photo_PornHub_AlbumPhotoArr)
+                    If l.ListExists Then l.RemoveAll(Function(_url) _url.IsEmptyString)
+                    If l.ListExists Then
+                        For Each url$ In l
+                            ThrowAny(Token)
+                            Try
+                                r = Responser.GetResponse(url)
+                                If Not r.IsEmptyString Then
+                                    url = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
+                                    If Not url.IsEmptyString Then _
+                                       _TempMediaList.ListAddValue(New UserMedia(url, UTypes.Picture) With {.SpecialFolder = $"Albums\{AlbumName}\"}, LNC)
+                                End If
+                            Catch
+                            End Try
+                        Next
+                        l.Clear()
+                        Return True
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                ThrowAny(Token)
+                Return False
+            End Try
+        End Function
+        Private Function DownloadUserPhotos_SavedPosts(ByVal Token As CancellationToken) As Boolean
+            Const HtmlPageNotFoundPhoto$ = "Page Not Found"
+            Dim URL$ = $"https://www.pornhub.com/{PersonType}/{NameTrue}/photos/favorites"
+            Try
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    If r.Contains(HtmlPageNotFoundPhoto) Then Return False
+                    Dim urls As List(Of String) = RegexReplace(r, Regex_Photo_PornHub_AlbumPhotoArr)
+                    If urls.ListExists Then
+                        Dim NewUrl$
+                        Dim m As UserMedia
+                        Dim l2 As List(Of UserMedia) = urls.Select(Function(__url) New UserMedia(__url, UTypes.Picture) With {
+                                                                                                 .Post = __url.Split("/").LastOrDefault}).ToList
+                        urls.Clear()
+                        If l2.ListExists Then l2.RemoveAll(Function(media) media.URL.IsEmptyString)
+                        If l2.ListExists Then
+                            Dim lBefore% = l2.Count
+                            If _TempPostsList.Count > 0 Then l2.RemoveAll(Function(media) _TempPostsList.Contains(media.Post.ID))
+                            If l2.Count > 0 Then
+                                For i% = 0 To l2.Count - 1
+                                    m = l2(i)
+                                    ThrowAny(Token)
+                                    Try
+                                        r = Responser.GetResponse(m.URL)
+                                        If Not r.IsEmptyString Then
+                                            NewUrl = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
+                                            If Not NewUrl.IsEmptyString Then
+                                                m.URL = NewUrl
+                                                m.File = NewUrl
+                                                _TempPostsList.ListAddValue(m.Post.ID, LNC)
+                                            Else
+                                                Throw New Exception
+                                            End If
+                                        End If
+                                    Catch
+                                        m.State = UserMedia.States.Missing
+                                    End Try
+                                    _TempMediaList.ListAddValue(m, LNC)
+                                Next
+                            End If
+                            Return l2.Count = lBefore
+                        End If
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                Return ProcessException(ex, Token, $"photos downloading error [{URL}]")
+            End Try
+        End Function
+#End Region
+#End Region
+#Region "ReparseVideo"
+        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            Const ERR_NEW_URL$ = "ERR_NEW_URL"
+            Dim URL$ = String.Empty
+            Try
+                If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(tm) tm.Type = UTypes.VideoPre) Then
+                    Dim m As UserMedia
+                    Dim r$, NewUrl$
+                    For i% = _TempMediaList.Count - 1 To 0 Step -1
+                        If _TempMediaList(i).Type = UTypes.VideoPre Then
+                            m = _TempMediaList(i)
+                            ThrowAny(Token)
+                            Try
+                                URL = m.URL
+                                r = Responser.Curl(URL)
+                                If Not r.IsEmptyString Then
+                                    NewUrl = CreateVideoURL(r)
+                                    If NewUrl.IsEmptyString Then
+                                        Throw New Exception With {.HelpLink = ERR_NEW_URL}
+                                    Else
+                                        m.URL = NewUrl
+                                        m.Type = UTypes.m3u8
+                                        _TempMediaList(i) = m
+                                    End If
+                                Else
+                                    _TempMediaList.RemoveAt(i)
+                                End If
+                            Catch mid_ex As Exception
+                                If mid_ex.HelpLink = ERR_NEW_URL OrElse DownloadingException(mid_ex, "") = 1 Then
+                                    m.State = UserMedia.States.Missing
+                                    _TempMediaList(i) = m
+                                Else
+                                    _TempMediaList.RemoveAt(i)
+                                End If
+                            End Try
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "video reparsing error", False)
+            End Try
+        End Sub
+#End Region
+#Region "ReparseMissing"
+        Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
+            Dim rList As New List(Of Integer)
+            Try
+                If ContentMissingExists Then
+                    Dim m As UserMedia
+                    Dim r$
+                    Dim eCurl As New ErrorsDescriber(EDP.ReturnValue)
+                    For i% = 0 To _ContentList.Count - 1
+                        m = _ContentList(i)
+                        If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then
+                            ThrowAny(Token)
+                            r = Responser.Curl(m.URL_BASE, eCurl)
+                            If Not r.IsEmptyString Then
+                                Dim NewUrl$ = CreateVideoURL(r)
+                                If Not NewUrl.IsEmptyString Then
+                                    m.URL = NewUrl
+                                    _TempMediaList.ListAddValue(m, LNC)
+                                    rList.Add(i)
+                                End If
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "missing data downloading error")
+            Finally
+                If rList.Count > 0 Then
+                    For i% = rList.Count - 1 To 0 Step -1 : _ContentList.RemoveAt(rList(i)) : Next
+                    rList.Clear()
+                End If
+            End Try
+        End Sub
+#End Region
+#Region "Download content"
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+            Return M3U8.Download(URL, Responser, DestinationFile)
+        End Function
+#End Region
+#Region "CreateVideoURL"
+        Private Shared Function CreateVideoURL(ByVal r As String) As String
+            Try
+                Dim OutStr$ = String.Empty
+                If Not r.IsEmptyString Then
+                    Dim _VarBlock$ = RegexReplace(r, RegexVideo_FlashVarsBlock)
+                    If Not _VarBlock.IsEmptyString Then
+                        Dim vars As List(Of FlashVar) = RegexFields(Of FlashVar)(_VarBlock, {RegexVideo_FlashVars_Vars}, {1, 2})
+                        Dim compiler As List(Of String) = RegexReplace(_VarBlock, RegexVideo_FlashVars_Compiler)
+                        If vars.ListExists And compiler.ListExists Then
+                            Dim v$
+                            Dim i%
+                            For Each var$ In compiler
+                                i = vars.IndexOf(var)
+                                If i >= 0 Then
+                                    v = vars(i).Value
+                                    If Not v.IsEmptyString Then OutStr &= v
+                                End If
+                            Next
+                        End If
+                    End If
+                End If
+                Return OutStr
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.PornHub.UserData.CreateVideoURL]", String.Empty)
+            End Try
+        End Function
+#End Region
+#Region "Standalone downloader"
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As UserMedia
+            Try
+                Dim r$ = Responser.Curl(URL)
+                If Not r.IsEmptyString Then
+                    Dim NewUrl$ = CreateVideoURL(r)
+                    If Not NewUrl.IsEmptyString Then
+                        Dim f As SFile = M3U8.Download(NewUrl, Responser, Destination)
+                        If Not f.IsEmptyString Then Return New UserMedia With {.State = UserMedia.States.Downloaded}
+                    End If
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"PornHub standalone download error: [{URL}]", New UserMedia)
+            End Try
+        End Function
+#End Region
+#Region "Exceptions"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String,
+                                                          Optional ByVal FromPE As Boolean = False, Optional ByVal EObj As Object = Nothing) As Integer
+            If Responser.Status = Net.WebExceptionStatus.ConnectionClosed Then
+                Return 1
+            ElseIf Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+                Return 2
+            Else
+                Return 0
+            End If
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/PornHub/UserExchangeOptions.vb

@@ -0,0 +1,23 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.PornHub
+    Friend Class UserExchangeOptions
+        Friend Property DownloadGifs As Boolean
+        Friend Property DownloadPhotoOnlyFromModelHub As Boolean
+        Friend Sub New(ByVal u As UserData)
+            DownloadGifs = u.DownloadGifs
+            DownloadPhotoOnlyFromModelHub = u.DownloadPhotoOnlyFromModelHub
+        End Sub
+        Friend Sub New(ByVal s As SiteSettings)
+            Dim v As CheckState = CInt(s.DownloadGifs.Value)
+            DownloadGifs = Not v = CheckState.Unchecked
+            DownloadPhotoOnlyFromModelHub = s.DownloadPhotoOnlyFromModelHub.Value
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Reddit/M3U8.vb

@@ -8,7 +8,7 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
 Imports SCrawler.API.Reddit.M3U8_Declarations
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Reddit
     Namespace M3U8_Declarations

SCrawler/API/Reddit/SiteSettings.vb

@@ -25,7 +25,7 @@ Namespace API.Reddit
                 Return My.Resources.SiteResources.RedditPic_512
             End Get
         End Property
-        <PropertyOption(ControlText:="Saved posts user"), PXML("SavedPostsUserName")>
+        <PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
         Friend ReadOnly Property SavedPostsUserName As PropertyValue
         <PropertyOption(ControlText:="Use M3U8", ControlToolTip:="Use M3U8 or mp4 for Reddit videos"), PXML>
         Friend ReadOnly Property UseM3U8 As PropertyValue
@@ -40,6 +40,7 @@ Namespace API.Reddit
             UrlPatternUser = "https://www.reddit.com/user/{0}/"
             UrlPatternChannel = "https://www.reddit.com/r/{0}/"
             ImageVideoContains = "reddit.com"
+            UserRegex = RParams.DM("[htps:/]{7,8}.*?reddit.com/([user]{1,4})/([^/]+)", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
         End Sub
         Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
             Select Case What
@@ -55,19 +56,9 @@ Namespace API.Reddit
             End Select
             Return Nothing
         End Function
-        Private ReadOnly RedditRegEx1 As RParams = RParams.DMS("[htps:/]{7,8}.*?reddit.com/user/([^/]+)", 1)
-        Private ReadOnly RedditRegEx2 As RParams = RParams.DMS(".?u/([^/]+)", 1)
-        Private ReadOnly RedditChannelRegEx1 As RParams = RParams.DMS("[htps:/]{7,8}.*?reddit.com/r/([^/]+)", 1)
-        Private ReadOnly RedditChannelRegEx2 As RParams = RParams.DMS(".?r/([^/]+)", 1)
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
-            Dim s$
+            Dim l As List(Of String) = RegexReplace(UserURL, UserRegex)
-            Dim c% = 0
+            If l.ListExists(3) Then Return New ExchangeOptions(Site, l(2), l(1) = "r") Else Return Nothing
-            For Each r As RParams In {RedditRegEx1, RedditRegEx2, RedditChannelRegEx1, RedditChannelRegEx2}
-                s = RegexReplace(UserURL, r)
-                If Not s.IsEmptyString Then Return New ExchangeOptions(Site, s, c > 1)
-                c += 1
-            Next
-            Return Nothing
         End Function
         Friend Overrides Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
             Try
@@ -83,7 +74,7 @@ Namespace API.Reddit
                                         avg.NumToString(New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}) & " outage reports:" & vbCr &
                                         dl.ListToString(vbCr) & vbCr & vbCr &
                                         "Do you want to continue parsing Reddit data?", "There are outage reports on Reddit"}, vbYesNo) = vbYes Then
-                                DirectCast(Settings(RedGifs.RedGifsSiteKey).Source, RedGifs.SiteSettings).UpdateTokenIfRequired()
+                                UpdateRedGifsToken()
                                 Return True
                             Else
                                 Return False
@@ -91,11 +82,15 @@ Namespace API.Reddit
                         End If
                     End If
                 End If
+                UpdateRedGifsToken()
                 Return True
             Catch ex As Exception
                 Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Reddit.SiteSettings.Available]", True)
             End Try
         End Function
+        Private Sub UpdateRedGifsToken()
+            DirectCast(Settings(RedGifs.RedGifsSiteKey).Source, RedGifs.SiteSettings).UpdateTokenIfRequired()
+        End Sub
         Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
             Dim spf$ = String.Empty
             Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
@@ -108,8 +103,8 @@ Namespace API.Reddit
                 Using f As New RedditViewSettingsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
-        Friend Overrides Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return $"https://www.reddit.com/comments/{PostID.Split("_").LastOrDefault}/"
+            Return $"https://www.reddit.com/comments/{Media.Post.ID.Split("_").LastOrDefault}/"
         End Function
     End Class
 End Namespace

SCrawler/API/Reddit/UserData.vb

@@ -14,8 +14,8 @@ Imports SCrawler.Plugin.Hosts
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.ImageRenderer
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports CView = SCrawler.API.Reddit.IRedditView.View
@@ -136,7 +136,7 @@ Namespace API.Reddit
             _CrossPosts.Clear()
             If Not IsSavedPosts AndAlso (IsChannel AndAlso Not ChannelInfo Is Nothing) Then
                 If Not Responser Is Nothing Then Responser.Dispose()
-                Responser = New Response
+                Responser = New Responser
                 Responser.Copy(MySiteSettings.Responser)
                 ChannelPostsNames.ListAddList(ChannelInfo.PostsAll.Select(Function(p) p.ID), LNC)
                 If Not ViewMode = CView.New Then ChannelPostsNames.ListAddList(ChannelInfo.PostsNames, LNC)
@@ -152,6 +152,8 @@ Namespace API.Reddit
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             _TotalPostsDownloaded = 0
             If IsSavedPosts Then
+                'TODO: Reddit saved posts: remove Unicode converter?
+                Responser.DecodersError = EDP.ReturnValue
                 DownloadDataChannel(String.Empty, Token)
             ElseIf IsChannel Then
                 If ChannelInfo Is Nothing Then
@@ -522,7 +524,7 @@ Namespace API.Reddit
             End Try
         End Function
         Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
-            Dim RedGifsResponser As Response = Nothing
+            Dim RedGifsResponser As Responser = Nothing
             Try
                 ThrowAny(Token)
                 Const v2 As UTypes = UTypes.VideoPre + UTypes.m3u8
@@ -579,7 +581,7 @@ Namespace API.Reddit
         End Sub
         Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
             Dim rList As New List(Of Integer)
-            Dim RedGifsResponser As Response = Nothing
+            Dim RedGifsResponser As Responser = Nothing
             Try
                 If Not ChannelInfo Is Nothing Or SaveToCache Then Exit Sub
                 If ContentMissingExists Then
@@ -656,11 +658,11 @@ Namespace API.Reddit
             Public Overrides Sub Perform(Optional ByVal Value As Double = 1)
             End Sub
         End Class
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Response, ByVal f As SFile, ByVal SpecialFolder As String) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Responser, ByVal f As SFile, ByVal SpecialFolder As String) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString Then
                     Using r As New UserData
-                        r.Responser = New Response
+                        r.Responser = New Responser
                         r.Responser.Copy(resp)
                         r.ParsePost(URL)
                         If r._TempMediaList.Count > 0 Then
@@ -711,7 +713,7 @@ Namespace API.Reddit
         End Function
 #End Region
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
-            Dim RedGifsResponser As Response = Nothing
+            Dim RedGifsResponser As Responser = Nothing
             Try
                 Const _RFN$ = "RedditVideo"
                 Const RFN$ = _RFN & "{0}"

SCrawler/API/Redgifs/Declarations.vb

@@ -14,6 +14,6 @@ Namespace API.RedGifs
         Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v, n, e))
         Friend ReadOnly WatchIDRegex As RParams = RParams.DMS(".+?watch/([^\?&""/]+)", 1, EDP.ReturnValue)
         Friend ReadOnly ThumbsIDRegex As RParams = RParams.DMS("([^/\?&""]+?)(-\w+?|)\.(mp4|jpg)", 1, EDP.ReturnValue,
-                                                               Function(v) If(CStr(v).IsEmptyString, String.Empty, CStr(v).ToLower.Trim))
+                                                               CType(Function(Input$) Input.StringToLower.StringTrim, Func(Of String, String)))
     End Module
 End Namespace

SCrawler/API/Redgifs/SiteSettings.vb

@@ -11,8 +11,9 @@ Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Cookies
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Namespace API.RedGifs
@@ -32,7 +33,6 @@ Namespace API.RedGifs
         <PropertyOption(AllowNull:=False, ControlText:="Token", ControlToolTip:="Bearer token")>
         Friend Property Token As PropertyValue
         <PXML> Friend Property TokenLastDateUpdated As PropertyValue
-        <DoNotUse> Friend ReadOnly Property NoCredentialsResponser As Response
         Private Const TokenName As String = "authorization"
 #End Region
 #Region "Initializer"
@@ -40,25 +40,11 @@ Namespace API.RedGifs
             MyBase.New(RedGifsSite, "redgifs.com")
             Dim t$ = String.Empty
             With Responser
-                Dim b As Boolean = Not .UseWebClient Or Not .UseWebClientCookies Or Not .UseWebClientAdditionalHeaders
+                Dim b As Boolean = Not .Mode = Responser.Modes.WebClient
-                .UseWebClient = True
+                .Mode = Responser.Modes.WebClient
-                .UseWebClientCookies = True
+                t = .Headers.Value(TokenName)
-                .UseWebClientAdditionalHeaders = True
-                If .Headers.Count > 0 AndAlso .Headers.ContainsKey(TokenName) Then t = .Headers(TokenName)
                 If b Then .SaveSettings()
             End With
-            NoCredentialsResponser = New Response($"{SettingsFolderName}\Responser_{RedGifsSite}_NC.xml") With {
-                .CookiesEncryptKey = SettingsCLS.CookieEncryptKey,
-                .CookiesDomain = "redgifs.com"
-            }
-            With NoCredentialsResponser
-                If .File.Exists Then
-                    .LoadSettings()
-                Else
-                    .Cookies = New CookieKeeper(.CookiesDomain) With {.EncryptKey = SettingsCLS.CookieEncryptKey}
-                    .SaveSettings()
-                End If
-            End With
             Token = New PropertyValue(t, GetType(String), Sub(v) UpdateResponse(v))
             TokenLastDateUpdated = New PropertyValue(Now.AddYears(-1), GetType(Date))
             UrlPatternUser = "https://www.redgifs.com/users/{0}/"
@@ -68,10 +54,8 @@ Namespace API.RedGifs
 #End Region
 #Region "Response updater"
         Private Sub UpdateResponse(ByVal Value As String)
-            With Responser.Headers
+            Responser.Headers.Add(TokenName, Value)
-                If .Count = 0 OrElse Not .ContainsKey(TokenName) Then .Add(TokenName, Value) Else .Item(TokenName) = Value
+            Responser.SaveSettings()
-                Responser.SaveSettings()
-            End With
         End Sub
 #End Region
 #Region "Token updaters"
@@ -88,7 +72,7 @@ Namespace API.RedGifs
             Try
                 Dim r$
                 Dim NewToken$ = String.Empty
-                Using resp As New Response : r = resp.GetResponse("https://api.redgifs.com/v2/auth/temporary",, EDP.ThrowException) : End Using
+                Using resp As New Responser : r = resp.GetResponse("https://api.redgifs.com/v2/auth/temporary",, EDP.ThrowException) : End Using
                 If Not r.IsEmptyString Then
                     Dim j As EContainer = JsonDocument.Parse(r)
                     If Not j Is Nothing Then
@@ -129,7 +113,7 @@ Namespace API.RedGifs
         End Function
         Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
             If BaseAuthExists() Then
-                Using resp As Response = Responser.Copy
+                Using resp As Responser = Responser.Copy
                     Dim m As UserMedia = UserData.GetDataFromUrlId(URL, False, resp, Settings(RedGifsSiteKey))
                     If Not m.State = UStates.Missing And Not m.State = UserData.DataGone And (m.Type = UTypes.Picture Or m.Type = UTypes.Video) Then
                         Try
@@ -153,8 +137,8 @@ Namespace API.RedGifs
             End If
             Return Nothing
         End Function
-        Friend Overrides Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return $"https://www.redgifs.com/watch/{PostID}"
+            Return $"https://www.redgifs.com/watch/{Media.Post.ID}"
         End Function
         Friend Overrides Function BaseAuthExists() As Boolean
             Return UpdateTokenIfRequired() AndAlso ACheck(Token.Value)

SCrawler/API/Redgifs/UserData.vb

@@ -11,8 +11,8 @@ Imports System.Threading
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Namespace API.RedGifs
@@ -34,21 +34,15 @@ Namespace API.RedGifs
         End Sub
 #End Region
 #Region "Download functions"
-        Private NoCredentialsResponser As Response
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
-            Try
+            DownloadData(1, Token)
-                NoCredentialsResponser = MySettings.NoCredentialsResponser.Copy
-                DownloadData(1, Token)
-            Finally
-                NoCredentialsResponser.Dispose()
-            End Try
         End Sub
         Private Overloads Sub DownloadData(ByVal Page As Integer, ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
             Try
                 Dim _page As Func(Of String) = Function() If(Page = 1, String.Empty, $"&page={Page}")
                 URL = $"https://api.redgifs.com/v2/users/{Name}/search?order=recent{_page.Invoke}"
-                Dim r$ = NoCredentialsResponser.GetResponse(URL,, EDP.ThrowException)
+                Dim r$ = Responser.GetResponse(URL,, EDP.ThrowException)
                 Dim postDate$, postID$
                 Dim pTotal% = 0
                 If Not r.IsEmptyString Then
@@ -62,7 +56,7 @@ Namespace API.RedGifs
                                     Case DateResult.Exit : Exit Sub
                                 End Select
                                 postID = g.Value("id")
-                                If Not _TempPostsList.Contains(postID) Then _TempPostsList.Add(postID) Else Exit For
+                                If Not _TempPostsList.Contains(postID) Then _TempPostsList.Add(postID) Else Exit Sub
                                 ObtainMedia(g, postID, postDate)
                             Next
                         End If
@@ -70,7 +64,7 @@ Namespace API.RedGifs
                 End If
                 If pTotal > 0 And Page < pTotal Then DownloadData(Page + 1, Token)
             Catch ex As Exception
-                ProcessException(ex, Token, $"data downloading error [{URL}]",, True)
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
             End Try
         End Sub
 #End Region
@@ -166,7 +160,7 @@ Namespace API.RedGifs
                 Return String.Empty
             End If
         End Function
-        Friend Shared Function GetDataFromUrlId(ByVal Obj As String, ByVal ObjIsID As Boolean, ByVal Responser As Response,
+        Friend Shared Function GetDataFromUrlId(ByVal Obj As String, ByVal ObjIsID As Boolean, ByVal Responser As Responser,
                                                 ByVal Host As Plugin.Hosts.SettingsHost) As UserMedia
             Dim URL$ = String.Empty
             Try
@@ -179,7 +173,7 @@ Namespace API.RedGifs
                     If Host.Source.Available(Plugin.ISiteSettings.Download.Main, True) Then
                         If Responser Is Nothing Then Responser = Host.Responser.Copy
                         URL = String.Format(PostDataUrl, Obj.ToLower)
-                        Dim r$ = Responser.DownloadString(URL, EDP.ThrowException)
+                        Dim r$ = Responser.GetResponse(URL,, EDP.ThrowException)
                         If Not r.IsEmptyString Then
                             Using j As EContainer = JsonDocument.Parse(r)
                                 If Not j Is Nothing Then
@@ -206,8 +200,15 @@ Namespace API.RedGifs
                 If Not Responser Is Nothing AndAlso (Responser.Client.StatusCode = DataGone Or Responser.Client.StatusCode = HttpStatusCode.NotFound) Then
                     Return New UserMedia With {.State = DataGone}
                 Else
-                    Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.RedGifs.UserData.GetDataFromUrlId({URL})]",
+                    Dim m As New UserMedia With {.State = UStates.Missing}
-                                                   New UserMedia With {.State = UStates.Missing})
+                    Dim _errText$ = "API.RedGifs.UserData.GetDataFromUrlId({0})"
+                    If Responser.Client.StatusCode = HttpStatusCode.Unauthorized Then
+                        _errText = $"RedGifs credentials have expired [{CInt(Responser.Client.StatusCode)}]: {_errText}"
+                        MyMainLOG = String.Format(_errText, URL)
+                        Return m
+                    Else
+                        Return ErrorsDescriber.Execute(EDP.SendInLog, ex, String.Format(_errText, URL), m)
+                    End If
                 End If
             End Try
         End Function
@@ -232,18 +233,8 @@ Namespace API.RedGifs
 #Region "Exception"
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
                                                           Optional ByVal EObj As Object = Nothing) As Integer
-            Dim IsNoCredentialsResponser As Boolean = AConvert(Of Boolean)(EObj, False)
+            Dim s As WebExceptionStatus = Responser.Client.Status
-            Dim s As WebExceptionStatus = -1
+            Dim sc As HttpStatusCode = Responser.Client.StatusCode
-            Dim sc As HttpStatusCode = -1
-            If IsNoCredentialsResponser Then
-                If Not NoCredentialsResponser Is Nothing Then
-                    s = NoCredentialsResponser.Status
-                    sc = NoCredentialsResponser.StatusCode
-                End If
-            Else
-                s = Responser.Client.Status
-                sc = Responser.Client.StatusCode
-            End If
             If sc = HttpStatusCode.NotFound Or s = DataGone Then
                 UserExists = False
             ElseIf sc = HttpStatusCode.Unauthorized Then

SCrawler/API/TikTok/UserData.vb

@@ -10,7 +10,7 @@ Imports System.Threading
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Namespace API.TikTok
     Friend Class UserData : Inherits UserDataBase
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
@@ -52,7 +52,7 @@ Namespace API.TikTok
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Response, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
             Try
                 If Not URL.IsEmptyString Then
                     Dim PostId$ = String.Empty
@@ -61,7 +61,7 @@ Namespace API.TikTok
                     Dim r$
                     PostId = RegexEnvir.ExtractPostID(URL)
                     If Not PostId.IsEmptyString Then
-                        Using resp As Response = Responser.Copy() : r = resp.GetResponse(URL,, EDP.ThrowException) : End Using
+                        Using resp As Responser = Responser.Copy() : r = resp.GetResponse(URL,, EDP.ThrowException) : End Using
                         If Not r.IsEmptyString Then
                             If RegexEnvir.GetVideoData(r, PostId, PostURL, PostDate) Then Return {MediaFromData(PostURL, PostId, PostDate)}
                         End If

SCrawler/API/Twitter/Declarations.vb

@@ -6,14 +6,21 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Globalization
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Twitter
     Friend Module Declarations
         Friend Const TwitterSite As String = "Twitter"
-        Friend DateProvider As New ADateTime(ADateTime.Formats.BaseDateTime)
+        Friend ReadOnly DateProvider As ADateTime = GetDateProvider()
         Friend ReadOnly VideoNode As NodeParams() = {New NodeParams("video_info", True, True, True, True, 10)}
         Friend ReadOnly VideoSizeRegEx As RParams = RParams.DMS("\d+x(\d+)", 1, EDP.ReturnValue)
         Friend ReadOnly UserIdRegEx As RParams = RParams.DMS("user_id.:.(\d+)", 1, EDP.ReturnValue)
+        Private Function GetDateProvider() As ADateTime
+            Dim n As DateTimeFormatInfo = CultureInfo.GetCultureInfo("en-us").DateTimeFormat.Clone
+            n.FullDateTimePattern = "ddd MMM dd HH:mm:ss +ffff yyyy"
+            n.TimeSeparator = String.Empty
+            Return New ADateTime(DirectCast(n.Clone, DateTimeFormatInfo)) With {.DateTimeStyle = DateTimeStyles.AssumeUniversal}
+        End Function
     End Module
 End Namespace

SCrawler/API/Twitter/SiteSettings.vb

@@ -9,8 +9,9 @@
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Tools.WEB
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Cookies
 Namespace API.Twitter
     <Manifest("AndyProgram_Twitter"), SavedPosts>
     Friend Class SiteSettings : Inherits SiteSettingsBase
@@ -31,12 +32,12 @@ Namespace API.Twitter
         Private ReadOnly Property Auth As PropertyValue
         <PropertyOption(AllowNull:=False, ControlText:="Token", ControlToolTip:="Set token from [x-csrf-token] response header")>
         Private ReadOnly Property Token As PropertyValue
-        <PropertyOption(ControlText:="Saved posts user name", ControlToolTip:="Personal profile username", LeftOffset:=120), PXML>
+        <PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
         Friend ReadOnly Property SavedPostsUserName As PropertyValue
-        Friend Overrides ReadOnly Property Responser As Response
+        Friend Overrides ReadOnly Property Responser As Responser
         Friend Sub New()
             MyBase.New(TwitterSite)
-            Responser = New Response($"{SettingsFolderName}\Responser_{Site}.xml")
+            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml")
 
             Dim a$ = String.Empty
             Dim t$ = String.Empty
@@ -45,10 +46,8 @@ Namespace API.Twitter
                 If .File.Exists Then
                     If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
                     .LoadSettings()
-                    With .Headers
+                    a = .Headers.Value(Header_Authorization)
-                        If .ContainsKey(Header_Authorization) Then a = .Item(Header_Authorization)
+                    t = .Headers.Value(Header_Token)
-                        If .ContainsKey(Header_Token) Then t = .Item(Header_Token)
-                    End With
                 Else
                     .ContentType = "application/json"
                     .Accept = "*/*"
@@ -56,17 +55,15 @@ Namespace API.Twitter
                     .Cookies = New CookieKeeper(.CookiesDomain) With {.EncryptKey = SettingsCLS.CookieEncryptKey}
                     .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
                     .Decoders.Add(SymbolsConverter.Converters.Unicode)
-                    With .Headers
+                    .Headers.Add("sec-ch-ua", " Not;A Brand"";v=""99"", ""Google Chrome"";v=""91"", ""Chromium"";v=""91""")
-                        .Add("sec-ch-ua", " Not;A Brand"";v=""99"", ""Google Chrome"";v=""91"", ""Chromium"";v=""91""")
+                    .Headers.Add("sec-ch-ua-mobile", "?0")
-                        .Add("sec-ch-ua-mobile", "?0")
+                    .Headers.Add("sec-fetch-dest", "empty")
-                        .Add("sec-fetch-dest", "empty")
+                    .Headers.Add("sec-fetch-mode", "cors")
-                        .Add("sec-fetch-mode", "cors")
+                    .Headers.Add("sec-fetch-site", "same-origin")
-                        .Add("sec-fetch-site", "same-origin")
+                    .Headers.Add(Header_Token, String.Empty)
-                        .Add(Header_Token, String.Empty)
+                    .Headers.Add("x-twitter-active-user", "yes")
-                        .Add("x-twitter-active-user", "yes")
+                    .Headers.Add("x-twitter-auth-type", "OAuth2Session")
-                        .Add("x-twitter-auth-type", "OAuth2Session")
+                    .Headers.Add(Header_Authorization, String.Empty)
-                        .Add(Header_Authorization, String.Empty)
-                    End With
                     .SaveSettings()
                 End If
             End With
@@ -87,7 +84,7 @@ Namespace API.Twitter
                     Case NameOf(Token) : f = Header_Token
                 End Select
                 If Not f.IsEmptyString Then
-                    If Responser.Headers.Count > 0 AndAlso Responser.Headers.ContainsKey(f) Then Responser.Headers.Remove(f)
+                    Responser.Headers.Remove(f)
                     If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
                     Responser.SaveSettings()
                 End If
@@ -103,8 +100,8 @@ Namespace API.Twitter
         Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
             Return UserData.GetVideoInfo(URL, Responser)
         End Function
-        Friend Overrides Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return $"https://twitter.com/{UserID}/status/{PostID}"
+            Return $"https://twitter.com/{User.Name}/status/{Media.Post.ID}"
         End Function
         Friend Overrides Function BaseAuthExists() As Boolean
             Return If(Responser.Cookies?.Count, 0) > 0 And ACheck(Token.Value) And ACheck(Auth.Value)

SCrawler/API/Twitter/UserData.vb

@@ -11,8 +11,8 @@ Imports System.Threading
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Namespace API.Twitter
     Friend Class UserData : Inherits UserDataBase
@@ -263,13 +263,13 @@ Namespace API.Twitter
         End Sub
 #End Region
 #Region "Get video static"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Response) As IEnumerable(Of UserMedia)
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Responser) As IEnumerable(Of UserMedia)
             Try
                 If URL.Contains("twitter") Then
                     Dim PostID$ = RegexReplace(URL, RParams.DM("(?<=/)\d+", 0))
                     If Not PostID.IsEmptyString Then
                         Dim r$
-                        Using rc As Response = resp.Copy() : r = rc.GetResponse(String.Format(SinglePostUrl, PostID),, EDP.ReturnValue) : End Using
+                        Using rc As Responser = resp.Copy() : r = rc.GetResponse(String.Format(SinglePostUrl, PostID),, EDP.ReturnValue) : End Using
                         If Not r.IsEmptyString Then
                             Using j As EContainer = JsonDocument.Parse(r)
                                 If j.ListExists Then

SCrawler/API/UserDataBind.vb

@@ -8,9 +8,10 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
 Imports SCrawler.API.Base
-Imports PersonalUtilities.Tools
+Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.Messaging
+Imports PersonalUtilities.Tools
 Namespace API
     Friend Class UserDataBind : Inherits UserDataBase : Implements ICollection(Of IUserData), IMyEnumerator(Of IUserData)
 #Region "Events"
@@ -20,6 +21,17 @@ Namespace API
 #Region "Declarations"
         Friend ReadOnly Property Collections As List(Of IUserData)
 #Region "Base class overrides"
+        Friend Overrides ReadOnly Property IsVirtual As Boolean
+            Get
+                Return CollectionModel = UsageModel.Virtual
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property CollectionModel As UsageModel
+            Get
+                If Count > 0 Then Return Item(0).CollectionModel Else Return UsageModel.Default
+            End Get
+        End Property
+        Friend Property CurrentlyEdited As Boolean = False
         Private _CollectionName As String = String.Empty
         Friend Overrides Property CollectionName As String
             Get
@@ -80,10 +92,13 @@ Namespace API
         End Sub
         Friend Overrides Function GetUserPicture() As Image
             If Count > 0 Then
-                Return Collections(0).GetPicture
+                Dim img As Image
-            Else
+                For Each u As UserDataBase In Collections
-                Return GetNullPicture(Settings.MaxLargeImageHeight)
+                    img = u.GetPicture(Of Image)(False)
+                    If Not img Is Nothing Then Return img
+                Next
             End If
+            Return GetNullPicture(If(Settings.ViewMode.Value = ViewModes.IconLarge, Settings.MaxLargeImageHeight, Settings.MaxSmallImageHeight))
         End Function
 #End Region
         Friend Overrides ReadOnly Property DownloadedTotal(Optional ByVal Total As Boolean = True) As Integer
@@ -102,7 +117,15 @@ Namespace API
         End Property
         Friend Overrides Property MyFile As SFile
             Get
-                If Count > 0 Then Return Collections(0).File Else Return Nothing
+                If Count > 0 Then
+                    If IsVirtual Then
+                        Return GetRealUserFile.IfNullOrEmpty(Collections(0).File)
+                    Else
+                        Return Collections(0).File
+                    End If
+                Else
+                    Return Nothing
+                End If
             End Get
             Set(ByVal NewFile As SFile)
             End Set
@@ -120,8 +143,8 @@ Namespace API
         End Property
         Friend Overrides Property DataMerging As Boolean
             Get
-                If Count > 0 Then
+                If Count > 0 AndAlso Collections.Exists(RealUser) Then
-                    Return DirectCast(Collections(0), UserDataBase).DataMerging
+                    Return DirectCast(Collections.Find(RealUser), UserDataBase).DataMerging
                 Else
                     Return False
                 End If
@@ -184,6 +207,7 @@ Namespace API
         End Property
         Friend Overrides Function GetUserInformation() As String
             Dim OutStr$ = String.Empty
+            If IsVirtual Then OutStr = "This is a virtual collection."
             If Count > 0 Then Collections.ForEach(Sub(c) OutStr.StringAppendLine(DirectCast(c, UserDataBase).GetUserInformation(), vbNewLine.StringDup(2)))
             Return OutStr
         End Function
@@ -346,12 +370,36 @@ Namespace API
             If Not e.Exists Then e = New ErrorsDescriber(EDP.SendInLog)
             If Count > 0 Then Collections.ForEach(Sub(c) c.OpenSite(e))
         End Sub
+        Private ReadOnly RealUser As Predicate(Of IUserData) = Function(u) u.UserModel = UsageModel.Default
         Friend Overrides Sub OpenFolder()
             Try
-                If Count > 0 Then GlobalOpenPath(Collections(0).File.CutPath(2))
+                If Count > 0 Then
+                    Dim i% = Collections.FindIndex(RealUser)
+                    If i = -1 Then i = 0
+                    If i >= 0 Then
+                        If IsVirtual Or Collections(i).UserModel = UsageModel.Virtual Then
+                            Collections(i).OpenFolder()
+                        Else
+                            GlobalOpenPath(Collections(i).File.CutPath(2))
+                        End If
+                    End If
+                End If
             Catch
             End Try
         End Sub
+        Friend Function GetRealUserFile() As SFile
+            Dim i% = -1
+            If Count > 0 Then i = Collections.FindIndex(RealUser)
+            If i >= 0 Then Return Collections(i).File Else Return Nothing
+        End Function
+        Friend Function GetRealUserSpecialCollectionPath()
+            Dim _SpecialCollectionPath As SFile = Nothing
+            If Count > 0 And Not IsVirtual Then
+                Dim _RealUser As UserDataBase = Collections.Find(RealUser)
+                If Not _RealUser Is Nothing Then _SpecialCollectionPath = _RealUser.User.SpecialCollectionPath
+            End If
+            Return _SpecialCollectionPath
+        End Function
 #End Region
 #Region "ICollection Support"
         Private ReadOnly Property IsReadOnly As Boolean Implements ICollection(Of IUserData).IsReadOnly
@@ -386,8 +434,8 @@ Namespace API
         ''' <exception cref="InvalidOperationException"></exception>
         Friend Overloads Sub Add(ByVal _Item As IUserData) Implements ICollection(Of IUserData).Add
             With _Item
-                If .MoveFiles(CollectionName) Then
+                If .MoveFiles(CollectionName, GetRealUserSpecialCollectionPath()) Then
-                    If DataMerging Then DirectCast(.Self, UserDataBase).MergeData()
+                    If Not _Item.IsVirtual And DataMerging Then DirectCast(.Self, UserDataBase).MergeData()
                     Collections.Add(_Item)
                     With Collections.Last
                         If Count > 1 Then
@@ -445,14 +493,9 @@ Namespace API
         Private Sub ConsolidateScripts()
             If Count > 1 AndAlso ScriptUse Then Collections.ForEach(Sub(c) c.ScriptUse = True)
         End Sub
-        Friend Sub AddRange(ByVal _Items As IEnumerable(Of IUserData))
-            If _Items.ListExists Then
-                For i% = 0 To _Items.Count - 1 : Add(_Items(i)) : Next
-            End If
-        End Sub
 #End Region
 #Region "Move, Merge"
-        Friend Overrides Function MoveFiles(ByVal __CollectionName As String) As Boolean
+        Friend Overrides Function MoveFiles(ByVal __CollectionName As String, ByVal __SpecialCollectionPath As SFile) As Boolean
             Throw New NotImplementedException("Move files is not available in the collection context")
         End Function
         Friend Overloads Sub MergeData(ByVal Merging As Boolean)
@@ -488,52 +531,73 @@ Namespace API
                         "Operation canceled", MsgBoxStyle.Critical)
                 Return False
             Else
-                DirectCast(_Item, UserDataBase).MoveFiles(String.Empty)
+                _Item.MoveFiles(String.Empty, Nothing)
                 MainFrameObj.ImageHandler(_Item)
                 AddRemoveBttDeleteHandler(_Item, False)
                 RaiseEvent OnUserRemoved(_Item)
                 Return Collections.Remove(_Item)
             End If
         End Function
-        Friend Overrides Function Delete(Optional ByVal Multiple As Boolean = False) As Integer
+        Friend Overrides Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer
             If Count > 0 Then
                 Const MsgTitle$ = "Deleting a collection"
-                Dim f As SFile
+                Dim f As SFile = Nothing
+                If Not IsVirtual Then
+                    f = GetRealUserFile()
+                    If Not f.IsEmptyString Then f = f.CutPath(IIf(DataMerging, 1, 2))
+                End If
                 Dim m As New MMessage($"Collection [{CollectionName} (number of profiles: {Count})] may contain data" & vbCr &
                                       "Are you sure you want to delete the collection and all of its files?", MsgTitle,
-                                      {New MsgBoxButton("Delete") With {.ToolTip = "Delete the collection and all files"},
+                                      {New MsgBoxButton("Delete") With {.ToolTip = "Delete the collection and all files", .KeyCode = Keys.Enter},
                                        New MsgBoxButton("Split") With {
                                         .ToolTip = "Users will be removed from the collection and will be displayed in the program as separate users." & vbCr &
-                                                   "All user data will remain."},
+                                                   "All user data will remain.",
+                                        .KeyCode = New ButtonKey(Keys.Enter, True)},
                                        "Cancel"}, vbExclamation)
-                Select Case If(Multiple, 0, MsgBoxE(m).Index)
+                Dim v%
+                If CollectionValue >= 0 Then
+                    Select Case CollectionValue
+                        Case 2 : v = 0
+                        Case 3 : v = 1
+                        Case Else : v = MsgBoxE(m)
+                    End Select
+                ElseIf Multiple Then
+                    v = 0
+                Else
+                    v = MsgBoxE(m)
+                End If
+                Select Case v
                     Case 0
-                        f = Collections(0).File.CutPath(IIf(DataMerging, 1, 2)).PathWithSeparator
-                        Settings.Users.Remove(Me)
                         Collections.ForEach(Sub(c) c.Delete())
-                        Downloader.UserRemove(Me)
+                        If Collections.All(Function(c As UserDataBase) c.Disposed) Then
-                        MainFrameObj.ImageHandler(Me, False)
+                            Settings.Users.Remove(Me)
-                        Collections.ListClearDispose
+                            Downloader.UserRemove(Me)
-                        Dispose(False)
+                            MainFrameObj.ImageHandler(Me, False)
-                        f.Delete(SFO.Path, SFODelete.EmptyOnly + Settings.DeleteMode, EDP.SendInLog)
+                            Collections.ListClearDispose
-                        Return 2
+                            Dispose(False)
+                            If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.EmptyOnly + Settings.DeleteMode, EDP.SendInLog)
+                            Return 2
+                        End If
                     Case 1
                         If DataMerging Then
                             MsgBoxE({$"Collection [{CollectionName}] data merged{vbCr}Unable to split merged collection{vbCr}Operation canceled", MsgTitle}, vbExclamation)
                             Return 0
                         Else
-                            f = Collections(0).File.CutPath(2)
+                            Collections.ForEach(Sub(ByVal c As IUserData)
-                            Settings.Users.Remove(Me)
+                                                    If c.MoveFiles(String.Empty, Nothing) Then
-                            Collections.ForEach(Sub(c)
+                                                        UserListLoader.UpdateUser(Settings.GetUser(c), True)
-                                                    c.MoveFiles(String.Empty)
+                                                        MainFrameObj.ImageHandler(c)
-                                                    MainFrameObj.ImageHandler(c)
+                                                    End If
                                                 End Sub)
-                            Collections.Clear()
+                            If Collections.All(Function(c) c.CollectionName.IsEmptyString) Then
-                            f.Delete(SFO.Path, SFODelete.Default + Settings.DeleteMode, EDP.SendInLog)
+                                Settings.Users.Remove(Me)
-                            Downloader.UserRemove(Me)
+                                Collections.Clear()
-                            MainFrameObj.ImageHandler(Me, False)
+                                If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.Default + Settings.DeleteMode, EDP.SendInLog)
-                            Dispose(False)
+                                Downloader.UserRemove(Me)
-                            Return 3
+                                MainFrameObj.ImageHandler(Me, False)
+                                Dispose(False)
+                                Return 3
+                            End If
                         End If
                     Case Else : If Not Multiple Then MsgBoxE({"Operation canceled", MsgTitle})
                 End Select
@@ -562,9 +626,11 @@ Namespace API
                                      "Deleting a user"}, vbExclamation,,,
                                     {
                                         New MsgBoxButton("Remove") With {
-                                        .ToolTip = "Remove a user from the collection only. All its data will remain. The user will appear in the program."},
+                                            .ToolTip = "Remove a user from the collection only. All its data will remain. The user will appear in the program.",
+                                            .KeyCode = Keys.Enter},
                                         New MsgBoxButton("Delete") With {
-                                        .ToolTip = "Delete a user from the collection and erase their data."},
+                                            .ToolTip = "Delete a user from the collection and erase their data.",
+                                            .KeyCode = New ButtonKey(Keys.Enter, True)},
                                         "Cancel"
                                     }).Index
                     Case 0

SCrawler/API/XVIDEOS/Declarations.vb

@@ -10,10 +10,13 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.XVIDEOS
     Friend Module Declarations
         Friend Const XvideosSiteKey As String = "AndyProgram_XVIDEOS"
-        Friend ReadOnly Property M3U8Regex As RParams = RParams.DM("http.+?.m3u8.*?(?=')", 0)
+        Private ReadOnly HtmlConverter As Func(Of String, String) = Function(Input) SymbolsConverter.HTML.Decode(Input, EDP.ReturnValue)
-        Friend ReadOnly Property VideoTitleRegex As RParams = RParams.DMS("html5player.setVideoTitle\('(.+)(?='\);)", 1)
+        Friend ReadOnly Regex_M3U8 As RParams = RParams.DM("http.+?.m3u8.*?(?=')", 0)
-        Friend ReadOnly Property VideoID As RParams = RParams.DMS(".*?www.xvideos.com/(video\d+).*", 1)
+        Friend ReadOnly Regex_VideoTitle As RParams = RParams.DMS("html5player.setVideoTitle\('(.+)(?='\);)", 1, EDP.ReturnValue, HtmlConverter)
-        Friend ReadOnly Property M3U8Reparse As RParams = RParams.DM("NAME=""(\d+).*?""[\r\n]*?(.+)(?=(|[\r\n]+?))", 0, RegexReturn.List)
+        Friend ReadOnly Regex_VideoID As RParams = RParams.DMS(".*?www.xvideos.com/(video\d+).*", 1)
-        Friend ReadOnly Property M3U8Appender As RParams = RParams.DM("(.+)(?=/.+?\.m3u8.*?)", 0)
+        Friend ReadOnly Regex_M3U8_Reparse As RParams = RParams.DM("NAME=""(\d+).*?""[\r\n]*?(.+)(?=(|[\r\n]+?))", 0, RegexReturn.List)
+        Friend ReadOnly Regex_M3U8_Appender As RParams = RParams.DM("(.+)(?=/.+?\.m3u8.*?)", 0)
+        Friend ReadOnly Regex_SavedVideosPlaylist As RParams = RParams.DM("<div id=""video.+?data-id=""(\d+).+?a href=""([^""]+)"".+?title=""([^""]*)""",
+                                                                          0, RegexReturn.List, EDP.ReturnValue, HtmlConverter)
     End Module
 End Namespace

SCrawler/API/XVIDEOS/M3U8.vb

@@ -7,44 +7,11 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
-Imports PersonalUtilities.Tools.WEB
 Namespace API.XVIDEOS
     Friend NotInheritable Class M3U8
         Private Sub New()
         End Sub
-        Private Shared Function Save(ByVal URLs As List(Of String), ByVal ffmpegFile As SFile, ByVal f As SFile) As SFile
+        Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal f As SFile) As SFile
-            Dim CachePath As SFile = Nothing
-            Try
-                If URLs.ListExists Then
-                    Dim ConcatFile As SFile = f
-                    If ConcatFile.Name.IsEmptyString Then ConcatFile.Name = "PlayListFile"
-                    ConcatFile.Extension = "mp4"
-                    CachePath = $"{f.PathWithSeparator}_Cache\{SFile.GetDirectories($"{f.PathWithSeparator}_Cache\",,, EDP.ReturnValue).ListIfNothing.Count + 1}\"
-                    If CachePath.Exists(SFO.Path) Then
-                        Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
-                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ReturnValue)
-                        Dim i%
-                        Dim eFiles As New List(Of SFile)
-                        Dim dFile As SFile = CachePath
-                        dFile.Extension = "ts"
-                        Using w As New WebClient
-                            For i = 0 To URLs.Count - 1
-                                dFile.Name = $"ConPart_{i}"
-                                w.DownloadFile(URLs(i), dFile)
-                                eFiles.Add(dFile)
-                            Next
-                        End Using
-                        f = FFMPEG.ConcatenateFiles(eFiles, ffmpegFile, ConcatFile, p, EDP.ThrowException)
-                        eFiles.Clear()
-                        Return f
-                    End If
-                End If
-                Return Nothing
-            Finally
-                CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
-            End Try
-        End Function
-        Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal ffmpegFile As SFile, ByVal f As SFile) As SFile
             Try
                 If Not URL.IsEmptyString Then
                     Using w As New WebClient
@@ -52,13 +19,13 @@ Namespace API.XVIDEOS
                         If Not r.IsEmptyString Then
                             Dim l As List(Of String) = ListAddList(Nothing, r.StringFormatLines.StringToList(Of String)(vbNewLine).ListWithRemove(Function(v) v.Trim.StartsWith("#")),
                                                                    New ListAddParams With {.Converter = Function(Input) $"{Appender}/{Input.ToString.Trim}"})
-                            If l.ListExists Then Return Save(l, ffmpegFile, f)
+                            If l.ListExists Then Return Base.M3U8Base.Download(l, f)
                         End If
                     End Using
                 End If
                 Return Nothing
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[M3U8.Download({URL}, {Appender}, {ffmpegFile}, {f})]")
+                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[M3U8.Download({URL}, {Appender}, {f})]")
                 Throw ex
             End Try
         End Function

SCrawler/API/XVIDEOS/SettingsForm.Designer.vb

@@ -1,80 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.XVIDEOS
-    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
-    Partial Friend Class SettingsForm : Inherits System.Windows.Forms.Form
-        <System.Diagnostics.DebuggerNonUserCode()>
-        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            Try
-                If disposing AndAlso components IsNot Nothing Then
-                    components.Dispose()
-                End If
-            Finally
-                MyBase.Dispose(disposing)
-            End Try
-        End Sub
-        Private components As System.ComponentModel.IContainer
-        <System.Diagnostics.DebuggerStepThrough()>
-        Private Sub InitializeComponent()
-            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
-            Me.LIST_DOMAINS = New System.Windows.Forms.ListBox()
-            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            CONTAINER_MAIN.ContentPanel.SuspendLayout()
-            CONTAINER_MAIN.SuspendLayout()
-            Me.SuspendLayout()
-            '
-            'CONTAINER_MAIN
-            '
-            '
-            'CONTAINER_MAIN.ContentPanel
-            '
-            CONTAINER_MAIN.ContentPanel.Controls.Add(Me.LIST_DOMAINS)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 241)
-            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            CONTAINER_MAIN.LeftToolStripPanelVisible = False
-            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
-            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
-            CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(384, 291)
-            CONTAINER_MAIN.TabIndex = 0
-            '
-            'LIST_DOMAINS
-            '
-            Me.LIST_DOMAINS.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.LIST_DOMAINS.FormattingEnabled = True
-            Me.LIST_DOMAINS.Location = New System.Drawing.Point(0, 0)
-            Me.LIST_DOMAINS.Name = "LIST_DOMAINS"
-            Me.LIST_DOMAINS.Size = New System.Drawing.Size(384, 241)
-            Me.LIST_DOMAINS.TabIndex = 0
-            '
-            'SettingsForm
-            '
-            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
-            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(384, 291)
-            Me.Controls.Add(CONTAINER_MAIN)
-            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
-            Me.Icon = Global.SCrawler.My.Resources.SiteResources.XvideosIcon_48
-            Me.KeyPreview = True
-            Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(400, 330)
-            Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(400, 330)
-            Me.Name = "SettingsForm"
-            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Settings"
-            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-            CONTAINER_MAIN.ResumeLayout(False)
-            CONTAINER_MAIN.PerformLayout()
-            Me.ResumeLayout(False)
-
-        End Sub
-        Private WithEvents LIST_DOMAINS As Windows.Forms.ListBox
-    End Class
-End Namespace

SCrawler/API/XVIDEOS/SettingsForm.resx

@@ -1,123 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<root>
-  <!-- 
-    Microsoft ResX Schema 
-    
-    Version 2.0
-    
-    The primary goals of this format is to allow a simple XML format 
-    that is mostly human readable. The generation and parsing of the 
-    various data types are done through the TypeConverter classes 
-    associated with the data types.
-    
-    Example:
-    
-    ... ado.net/XML headers & schema ...
-    <resheader name="resmimetype">text/microsoft-resx</resheader>
-    <resheader name="version">2.0</resheader>
-    <resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
-    <resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
-    <data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
-    <data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
-    <data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
-        <value>[base64 mime encoded serialized .NET Framework object]</value>
-    </data>
-    <data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
-        <value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
-        <comment>This is a comment</comment>
-    </data>
-                
-    There are any number of "resheader" rows that contain simple 
-    name/value pairs.
-    
-    Each data row contains a name, and value. The row also contains a 
-    type or mimetype. Type corresponds to a .NET class that support 
-    text/value conversion through the TypeConverter architecture. 
-    Classes that don't support this are serialized and stored with the 
-    mimetype set.
-    
-    The mimetype is used for serialized objects, and tells the 
-    ResXResourceReader how to depersist the object. This is currently not 
-    extensible. For a given mimetype the value must be set accordingly:
-    
-    Note - application/x-microsoft.net.object.binary.base64 is the format 
-    that the ResXResourceWriter will generate, however the reader can 
-    read any of the formats listed below.
-    
-    mimetype: application/x-microsoft.net.object.binary.base64
-    value   : The object must be serialized with 
-            : System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
-            : and then encoded with base64 encoding.
-    
-    mimetype: application/x-microsoft.net.object.soap.base64
-    value   : The object must be serialized with 
-            : System.Runtime.Serialization.Formatters.Soap.SoapFormatter
-            : and then encoded with base64 encoding.
-
-    mimetype: application/x-microsoft.net.object.bytearray.base64
-    value   : The object must be serialized into a byte array 
-            : using a System.ComponentModel.TypeConverter
-            : and then encoded with base64 encoding.
-    -->
-  <xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
-    <xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
-    <xsd:element name="root" msdata:IsDataSet="true">
-      <xsd:complexType>
-        <xsd:choice maxOccurs="unbounded">
-          <xsd:element name="metadata">
-            <xsd:complexType>
-              <xsd:sequence>
-                <xsd:element name="value" type="xsd:string" minOccurs="0" />
-              </xsd:sequence>
-              <xsd:attribute name="name" use="required" type="xsd:string" />
-              <xsd:attribute name="type" type="xsd:string" />
-              <xsd:attribute name="mimetype" type="xsd:string" />
-              <xsd:attribute ref="xml:space" />
-            </xsd:complexType>
-          </xsd:element>
-          <xsd:element name="assembly">
-            <xsd:complexType>
-              <xsd:attribute name="alias" type="xsd:string" />
-              <xsd:attribute name="name" type="xsd:string" />
-            </xsd:complexType>
-          </xsd:element>
-          <xsd:element name="data">
-            <xsd:complexType>
-              <xsd:sequence>
-                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
-                <xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
-              </xsd:sequence>
-              <xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
-              <xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
-              <xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
-              <xsd:attribute ref="xml:space" />
-            </xsd:complexType>
-          </xsd:element>
-          <xsd:element name="resheader">
-            <xsd:complexType>
-              <xsd:sequence>
-                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
-              </xsd:sequence>
-              <xsd:attribute name="name" type="xsd:string" use="required" />
-            </xsd:complexType>
-          </xsd:element>
-        </xsd:choice>
-      </xsd:complexType>
-    </xsd:element>
-  </xsd:schema>
-  <resheader name="resmimetype">
-    <value>text/microsoft-resx</value>
-  </resheader>
-  <resheader name="version">
-    <value>2.0</value>
-  </resheader>
-  <resheader name="reader">
-    <value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
-  </resheader>
-  <resheader name="writer">
-    <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
-  </resheader>
-  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
-</root>

SCrawler/API/XVIDEOS/SettingsForm.vb

@@ -1,70 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Forms
-Imports PersonalUtilities.Forms.Toolbars
-Namespace API.XVIDEOS
-    Friend Class SettingsForm
-        Private Const SettingsDesignXmlNode As String = "XvideosSettingsForm"
-        Private WithEvents MyDefs As DefaultFormOptions
-        Private ReadOnly Property Source As SiteSettings
-        Friend Sub New(ByRef s As SiteSettings)
-            InitializeComponent()
-            Source = s
-            MyDefs = New DefaultFormOptions(Me, Settings.Design)
-        End Sub
-        Private Sub SettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
-            With MyDefs
-                If Not Settings.Design.Contains(SettingsDesignXmlNode) Then Settings.Design.Add(SettingsDesignXmlNode, String.Empty)
-                .MyViewInitialize(Me, Settings.Design(SettingsDesignXmlNode), True)
-                .AddEditToolbar()
-                .AddOkCancelToolbar()
-                If Source.Domains.Count > 0 Then Source.Domains.ForEach(Sub(d) LIST_DOMAINS.Items.Add(d))
-                .EndLoaderOperations()
-            End With
-        End Sub
-        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
-            Source.Domains.Clear()
-            With LIST_DOMAINS
-                If .Items.Count > 0 Then
-                    For Each i In .Items : Source.Domains.Add(i.ToString) : Next
-                End If
-            End With
-            Source.UpdateDomains()
-            MyDefs.CloseForm()
-        End Sub
-        Private Sub MyDefs_ButtonAddClick(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonAddClick
-            Dim nd$ = InputBoxE("Enter a new domain using the pattern [xvideos.com]:", "New domain")
-            If Not nd.IsEmptyString Then
-                If Not LIST_DOMAINS.Items.Contains(nd) Then
-                    LIST_DOMAINS.Items.Add(nd)
-                Else
-                    MsgBoxE($"The domain [{nd}] already added")
-                End If
-            End If
-        End Sub
-        Private Sub MyDefs_ButtonDeleteClickE(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonDeleteClickE
-            Const MsgTitle$ = "Removing domains"
-            If _LatestSelected.ValueBetween(0, LIST_DOMAINS.Items.Count - 1) Then
-                Dim n$ = LIST_DOMAINS.Items(_LatestSelected)
-                If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?", MsgTitle}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
-                    LIST_DOMAINS.Items.RemoveAt(_LatestSelected)
-                    MsgBoxE({$"Domain [{n}] removed", MsgTitle})
-                Else
-                    MsgBoxE({"Operation canceled", MsgTitle})
-                End If
-            Else
-                MsgBoxE({"No domain selected", MsgTitle}, vbExclamation)
-            End If
-        End Sub
-        Private _LatestSelected As Integer = -1
-        Private Sub LIST_DOMENS_SelectedIndexChanged(sender As Object, e As EventArgs) Handles LIST_DOMAINS.SelectedIndexChanged
-            _LatestSelected = LIST_DOMAINS.SelectedIndex
-        End Sub
-    End Class
-End Namespace

SCrawler/API/XVIDEOS/SiteSettings.vb

@@ -7,15 +7,16 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports SCrawler.API.Base
+Imports SCrawler.API.BaseObjects
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Namespace API.XVIDEOS
-    <Manifest(XvideosSiteKey), SpecialForm(True)>
+    <Manifest(XvideosSiteKey), SavedPosts, SpecialForm(True), TaskGroup(SettingsCLS.TaskStackNamePornSite)>
-    Friend Class SiteSettings : Inherits SiteSettingsBase
+    Friend Class SiteSettings : Inherits SiteSettingsBase : Implements IDomainContainer
-#Region "Images"
+#Region "Declarations"
-        Friend Overrides ReadOnly Property Icon As Icon
+        Friend Overrides ReadOnly Property Icon As Icon Implements IDomainContainer.Icon
             Get
                 Return My.Resources.SiteResources.XvideosIcon_48
             End Get
@@ -25,58 +26,88 @@ Namespace API.XVIDEOS
                 Return My.Resources.SiteResources.XvideosPic_32
             End Get
         End Property
+#Region "Domains"
+        Private ReadOnly Property IDomainContainer_Site As String Implements IDomainContainer.Site
+            Get
+                Return Site
+            End Get
+        End Property
+        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue Implements IDomainContainer.DomainsSettingProp
+        Friend ReadOnly Property Domains As List(Of String) Implements IDomainContainer.Domains
+        Private ReadOnly Property DomainsTemp As List(Of String) Implements IDomainContainer.DomainsTemp
+        Private Property DomainsChanged As Boolean = False Implements IDomainContainer.DomainsChanged
+        Private ReadOnly Property DomainsDefault As String = "xvideos.com|xnxx.com" Implements IDomainContainer.DomainsDefault
 #End Region
-#Region "Declarations"
-        <PXML("Domains")> Private Property SiteDomains As PropertyValue
         <PropertyOption(ControlText:="Download UHD", ControlToolTip:="Download UHD (4K) content"), PXML>
-        Public Property DownloadUHD As PropertyValue
+        Friend Property DownloadUHD As PropertyValue
-        Friend ReadOnly Property Domains As List(Of String)
+        Private Property Initialized As Boolean = False Implements IDomainContainer.Initialized
-        Private Const DomainsDefault As String = "xvideos.com|xnxx.com"
+        <PropertyOption(ControlText:="Playlist of saved videos",
-        Private _Initialized As Boolean = False
+                        ControlToolTip:="Your personal videos playlist to download as 'saved posts'. " & vbCr &
+                                        "This playlist must be private (Visibility = 'Only me'). It also required cookies." & vbCr &
+                                        "This playlist must be entered by pattern: https://www.xvideos.com/favorite/01234567/playlistname.",
+                        LeftOffset:=130), PXML>
+        Friend ReadOnly Property SavedVideosPlaylist As PropertyValue
 #End Region
 #Region "Initializer"
         Friend Sub New()
             MyBase.New("XVIDEOS", "www.xvideos.com")
+            Responser.DeclaredError = EDP.ThrowException
             Domains = New List(Of String)
+            DomainsTemp = New List(Of String)
             SiteDomains = New PropertyValue(DomainsDefault, GetType(String), Sub(s) UpdateDomains())
             DownloadUHD = New PropertyValue(False)
+            SavedVideosPlaylist = New PropertyValue(String.Empty, GetType(String))
+            UrlPatternUser = "https://xvideos.com/{0}"
         End Sub
         Friend Overrides Sub EndInit()
-            _Initialized = True
+            Initialized = True
-            UpdateDomains()
+            DomainContainer.EndInit(Me)
+            DomainsTemp.ListAddList(Domains)
         End Sub
 #End Region
-#Region "Update"
+#Region "Edit"
-        Private _DomainsUpdateInProgress As Boolean = False
+        Private Property DomainsUpdateInProgress As Boolean = False Implements IDomainContainer.DomainsUpdateInProgress
-        Friend Sub UpdateDomains()
+        Private Property DomainsUpdatedBySite As Boolean = False Implements IDomainContainer.DomainsUpdatedBySite
-            If Not _Initialized Then Exit Sub
+        Friend Sub UpdateDomains() Implements IDomainContainer.UpdateDomains
-            If Not _DomainsUpdateInProgress Then
+            DomainContainer.UpdateDomains(Me)
-                _DomainsUpdateInProgress = True
-                If Not ACheck(SiteDomains.Value) Then SiteDomains.Value = DomainsDefault
-                Domains.ListAddList(CStr(SiteDomains.Value).Split("|"), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
-                Domains.ListAddList(DomainsDefault.Split("|"), LAP.NotContainsOnly)
-                SiteDomains.Value = Domains.ListToString("|")
-                _DomainsUpdateInProgress = False
-            End If
         End Sub
         Friend Overrides Sub Update()
-            UpdateDomains()
+            DomainContainer.Update(Me)
             Responser.SaveSettings()
         End Sub
+        Friend Overrides Sub EndEdit()
+            DomainContainer.EndEdit(Me)
+            MyBase.EndEdit()
+        End Sub
+        Friend Overrides Sub OpenSettingsForm()
+            DomainContainer.OpenSettingsForm(Me)
+        End Sub
 #End Region
 #Region "Download"
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
-            Return New UserData
+            If What = ISiteSettings.Download.SavedPosts Then
+                Return New UserData With {.IsSavedPosts = True, .User = New UserInfo With {.Name = "XVIDEOS"}}
+            Else
+                Return New UserData
+            End If
         End Function
         Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
-            Return Settings.UseM3U8
+            If Settings.UseM3U8 Then
+                If What = ISiteSettings.Download.SavedPosts Then
+                    Return ACheck(SavedVideosPlaylist.Value) And If(Responser.Cookies?.Count, 0) > 0
+                Else
+                    Return True
+                End If
+            Else
+                Return False
+            End If
         End Function
 #End Region
 #Region "User: get, check"
-        Friend Overrides Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
-            Dim user$ = UserName.Split("_").FirstOrDefault
+            Dim __user$ = User.Name.Split("_").FirstOrDefault
-            user &= $"/{UserName.Replace($"{user}_", String.Empty)}"
+            __user &= $"/{User.Name.Replace($"{__user}_", String.Empty)}"
-            Return user
+            Return String.Format(UrlPatternUser, __user)
         End Function
         Private Const UserRegexDefault As String = "/(profiles|[\w]*?[-]{0,1}channels)/([^/]+)(\Z|.*?)"
         Private Const URD As String = ".*?{0}{1}"
@@ -84,9 +115,10 @@ Namespace API.XVIDEOS
             If Not UserURL.IsEmptyString Then
                 If Domains.Count > 0 Then
                     Dim uName$, uOpt$, fStr$
+                    Dim uErr As New ErrorsDescriber(EDP.ReturnValue)
                     For i% = 0 To Domains.Count - 1
                         fStr = String.Format(URD, Domains(i), UserRegexDefault)
-                        uName = RegexReplace(UserURL, RParams.DMS(fStr, 2))
+                        uName = RegexReplace(UserURL, RParams.DMS(fStr, 2, uErr))
                         If Not uName.IsEmptyString Then
                             uOpt = RegexReplace(UserURL, RParams.DMS(fStr, 1))
                             If Not uOpt.IsEmptyString Then Return New ExchangeOptions(Site, $"{uOpt}_{uName}")
@@ -97,11 +129,6 @@ Namespace API.XVIDEOS
             Return Nothing
         End Function
 #End Region
-#Region "Settings"
-        Friend Overrides Sub OpenSettingsForm()
-            Using f As New SettingsForm(Me) : f.ShowDialog() : End Using
-        End Sub
-#End Region
 #Region "Get special data"
         Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
             If Not URL.IsEmptyString And Domains.Count > 0 Then
@@ -115,7 +142,7 @@ Namespace API.XVIDEOS
                 Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
                 f.Name = "video"
                 f.Extension = "mp4"
-                Using resp As Response = Responser.Copy
+                Using resp As Responser = Responser.Copy
                     Using user As New UserData With {.HOST = Settings(XvideosSiteKey)}
                         DirectCast(user, UserDataBase).User.File = f
                         Dim p As UserMedia = user.Download(URL, resp, DownloadUHD.Value, String.Empty)

SCrawler/API/XVIDEOS/UserData.vb

@@ -10,12 +10,29 @@ Imports System.Threading
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.WebDocuments.JSON
+Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.XVIDEOS
     Friend Class UserData : Inherits UserDataBase
+        Private Structure PlayListVideo : Implements IRegExCreator
+            Friend ID As String
+            Friend URL As String
+            Friend Title As String
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(3) Then
+                    ID = ParamsArray(0)
+                    URL = ParamsArray(1)
+                    If Not URL.IsEmptyString Then URL = $"https://www.xvideos.com/{URL.StringTrimStart("/")}"
+                    Title = ParamsArray(2).StringRemoveWinForbiddenSymbols.StringTrim
+                End If
+                Return Me
+            End Function
+            Friend Function ToUserMedia() As UserMedia
+                Return New UserMedia(URL, UTypes.VideoPre) With {.Object = Me, .PictureOption = Title, .Post = ID}
+            End Function
+        End Structure
         Private ReadOnly Property MySettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
@@ -28,71 +45,76 @@ Namespace API.XVIDEOS
             UseInternalM3U8Function = True
         End Sub
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            If Not Settings.UseM3U8 Then
+                'TODELETE: XVideos m3u8 delete after debug ffmpeg x86
+                'If Not Settings.OS64 Then
+                '    MyMainLOG = $"XVIDEOS [{ToStringForLog()}]: The plugin only works with x64 OS."
+                'Else
+                '    'MyMainLOG = $"{ToStringForLog()}: File [ffmpeg.exe] not found"
+                'End If
+                MyMainLOG = $"{ToStringForLog()}: File [ffmpeg.exe] not found"
+                Exit Sub
+            End If
+            If IsSavedPosts Then
+                If Not ACheck(MySettings.SavedVideosPlaylist.Value) Then Throw New ArgumentNullException("SavedVideosPlaylist", "Playlist of saved videos cannot be null")
+                DownloadSavedVideos(Token)
+            Else
+                DownloadUserVideo(Token)
+            End If
+        End Sub
+        Private Sub DownloadUserVideo(ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
             Try
-                If Not Settings.UseM3U8 Then
-                    If Not Settings.OS64 Then
-                        MyMainLOG = $"XVIDEOS [{ToStringForLog()}]: The plugin only works with x64 OS."
-                    Else
-                        MyMainLOG = $"{ToStringForLog()}: File [ffmpeg.exe] not found"
-                    End If
-                    Exit Sub
-                End If
-
                 Dim NextPage% = 0
                 Dim r$
+                Dim j As EContainer = Nothing
                 Dim jj As EContainer
-                Dim e As ErrorsDescriber = EDP.ThrowException
+                Dim user$ = MySettings.GetUserUrl(Me, False)
-                Dim user$ = MySettings.GetUserUrl(Name, False)
                 Dim p As UserMedia
                 Dim EnvirSet As Boolean = False
 
                 Do
                     ThrowAny(Token)
                     URL = $"https://www.xvideos.com/{user}/videos/new/{If(NextPage = 0, String.Empty, NextPage)}"
-                    r = Responser.GetResponse(URL,, e)
+                    r = Responser.GetResponse(URL)
                     If Not r.IsEmptyString Then
                         If Not EnvirSet Then UserExists = True : UserSuspended = False : EnvirSet = True
-                        With JsonDocument.Parse(r).XmlIfNothing
+                        j = JsonDocument.Parse(r).XmlIfNothing
+                        With j
                             If .Contains("videos") Then
                                 With .Item("videos")
                                     If .Count > 0 Then
                                         NextPage += 1
                                         For Each jj In .Self
                                             p = New UserMedia With {
-                                                .Post = New UserPost With {.ID = jj.Value("id")},
+                                                .Post = jj.Value("id"),
-                                                .URL = $"https://www.xvideos.com{jj.Value("u")}"
+                                                .URL = $"https://www.xvideos.com/{jj.Value("u").StringTrimStart("/")}"
                                             }
                                             If Not p.Post.ID.IsEmptyString And Not jj.Value("u").IsEmptyString Then
                                                 If Not _TempPostsList.Contains(p.Post.ID) Then
                                                     _TempPostsList.Add(p.Post.ID)
                                                     _TempMediaList.Add(p)
                                                 Else
-                                                    .Dispose()
                                                     Exit Do
                                                 End If
                                             End If
                                         Next
-                                    Else
+                                        Continue Do
-                                        .Dispose()
-                                        Exit Do
                                     End If
                                 End With
-                            Else
-                                .Dispose()
-                                Exit Do
                             End If
-                            .Dispose()
                         End With
-                    Else
-                        Exit Do
                     End If
-                Loop
+                    If Not j Is Nothing Then j.Dispose()
+                    Exit Do
+                Loop While NextPage < 100
+
+                If Not j Is Nothing Then j.Dispose()
 
                 If _TempMediaList.Count > 0 Then
                     For i% = 0 To _TempMediaList.Count - 1
                         ThrowAny(Token)
-                        With _TempMediaList(i) : _TempMediaList(i) = GetVideoData(.URL, Responser, MySettings.DownloadUHD.Value, .Post.ID) : End With
+                        _TempMediaList(i) = GetVideoData(_TempMediaList(i), Responser, MySettings.DownloadUHD.Value)
                     Next
                     _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
                 End If
@@ -108,38 +130,80 @@ Namespace API.XVIDEOS
                 If _TempMediaList.ListExists Then _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
             End Try
         End Sub
-        Private Function GetVideoData(ByVal URL As String, ByVal resp As Response, ByVal DownloadUHD As Boolean, ByVal ID As String) As UserMedia
+        Private Sub DownloadSavedVideos(ByVal Token As CancellationToken)
+            Dim URL$ = MySettings.SavedVideosPlaylist.Value
             Try
-                If Not URL.IsEmptyString Then
+                Dim NextPage% = 0
-                    Dim r$ = resp.GetResponse(URL,, EDP.ThrowException)
+                Dim __continue As Boolean = True
+                Dim r$
+                Dim data As List(Of PlayListVideo)
+                Dim i%
+                Do
+                    ThrowAny(Token)
+                    URL = $"{MySettings.SavedVideosPlaylist.Value}{If(NextPage = 0, String.Empty, $"/{NextPage}")}"
+                    r = Responser.GetResponse(URL,, EDP.ReturnValue)
+                    If Responser.HasError Then
+                        If Responser.StatusCode = Net.HttpStatusCode.NotFound And NextPage > 0 Then Exit Do
+                        Throw New Exception(Responser.ErrorText, Responser.ErrorException)
+                    End If
+                    NextPage += 1
                     If Not r.IsEmptyString Then
-                        Dim m$ = RegexReplace(r, M3U8Regex)
+                        data = RegexFields(Of PlayListVideo)(r, {Regex_SavedVideosPlaylist}, {1, 2, 3}, EDP.ReturnValue)
-                        If Not m.IsEmptyString Then
+                        If data.ListExists Then
-                            Dim appender$ = RegexReplace(m, M3U8Appender)
+                            If data.RemoveAll(Function(d) _TempPostsList.Contains(d.ID)) > 0 Then __continue = False
-                            Dim t$ = RegexReplace(r, VideoTitleRegex)
+                            If data.ListExists Then
-                            r = resp.GetResponse(m,, EDP.ThrowException)
+                                _TempPostsList.ListAddList(data.Select(Function(d) d.ID), LNC)
+                                i = _TempMediaList.Count
+                                _TempMediaList.ListAddList(data.Select(Function(d) d.ToUserMedia()), LNC)
+                                If _TempMediaList.Count = i Or Not __continue Then Exit Do Else Continue Do
+                            End If
+                        End If
+                    End If
+                    Exit Do
+                Loop While NextPage < 100 And __continue
+
+                If _TempMediaList.Count > 0 Then
+                    For i% = 0 To _TempMediaList.Count - 1
+                        ThrowAny(Token)
+                        _TempMediaList(i) = GetVideoData(_TempMediaList(i), Responser, MySettings.DownloadUHD.Value)
+                    Next
+                    _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+        Private Function GetVideoData(ByVal Media As UserMedia, ByVal resp As Responser, ByVal DownloadUHD As Boolean) As UserMedia
+            Try
+                If Not Media.URL.IsEmptyString Then
+                    Dim r$ = resp.GetResponse(Media.URL)
+                    If Not r.IsEmptyString Then
+                        Dim NewUrl$ = RegexReplace(r, Regex_M3U8)
+                        If Not NewUrl.IsEmptyString Then
+                            Dim appender$ = RegexReplace(NewUrl, Regex_M3U8_Appender)
+                            Dim t$ = If(Media.PictureOption.IsEmptyString, RegexReplace(r, Regex_VideoTitle), Media.PictureOption)
+                            r = resp.GetResponse(NewUrl)
                             If Not r.IsEmptyString Then
-                                Dim ls As List(Of Sizes) = RegexFields(Of Sizes)(r, {M3U8Reparse}, {1, 2})
+                                Dim ls As List(Of Sizes) = RegexFields(Of Sizes)(r, {Regex_M3U8_Reparse}, {1, 2})
                                 If ls.ListExists And Not DownloadUHD Then ls.RemoveAll(Function(v) Not v.Value.ValueBetween(1, 1080))
                                 If ls.ListExists Then
                                     ls.Sort()
-                                    m = $"{appender}/{ls(0).Data}"
+                                    NewUrl = $"{appender}/{ls(0).Data.StringTrimStart("/")}"
                                     ls.Clear()
-                                    Dim pID$ = ID
+                                    Dim pID$ = Media.Post.ID
-                                    If pID.IsEmptyString Then pID = RegexReplace(r, VideoID)
+                                    If pID.IsEmptyString Then pID = RegexReplace(r, Regex_VideoID)
                                     If pID.IsEmptyString Then pID = "0"
 
-                                    If Not t.IsEmptyString Then t = t.StringRemoveWinForbiddenSymbols(" ")
+                                    t = t.StringRemoveWinForbiddenSymbols.StringTrim
                                     If t.IsEmptyString Then
                                         t = pID
                                     Else
                                         If t.Length > 100 Then t = Left(t, 100)
                                     End If
-                                    If Not m.IsEmptyString Then
+                                    If Not NewUrl.IsEmptyString Then
-                                        Return New UserMedia With {
+                                        Return New UserMedia(NewUrl, UTypes.m3u8) With {
-                                            .Type = UTypes.m3u8,
+                                            .Post = pID,
-                                            .Post = New UserPost With {.ID = pID},
+                                            .URL_BASE = Media.URL,
-                                            .URL = m,
                                             .File = $"{t}.mp4",
                                             .PictureOption = appender
                                         }
@@ -151,18 +215,18 @@ Namespace API.XVIDEOS
                 End If
                 Return Nothing
             Catch ex As Exception
-                LogError(ex, $"[XVIDEOS.UserData.GetVideoData({URL})]")
+                LogError(ex, $"[XVIDEOS.UserData.GetVideoData({Media.URL})]")
                 Return Nothing
             End Try
         End Function
-        Friend Function Download(ByVal URL As String, ByVal resp As Response, ByVal DownloadUHD As Boolean, ByVal ID As String)
+        Friend Function Download(ByVal URL As String, ByVal resp As Responser, ByVal DownloadUHD As Boolean, ByVal ID As String)
-            Dim m As UserMedia = GetVideoData(URL, resp, DownloadUHD, ID)
+            Dim m As UserMedia = GetVideoData(New UserMedia(URL, UTypes.VideoPre) With {.Post = ID}, resp, DownloadUHD)
             If Not m.URL.IsEmptyString Then
                 Dim f As SFile = m.File
                 f.Path = MyFile.PathNoSeparator
                 m.State = UStates.Tried
                 Try
-                    f = M3U8.Download(m.URL, m.PictureOption, Settings.FfmpegFile, f)
+                    f = M3U8.Download(m.URL, m.PictureOption, f)
                     m.File = f
                     m.State = UStates.Downloaded
                 Catch ex As Exception
@@ -175,7 +239,7 @@ Namespace API.XVIDEOS
             DownloadContentDefault(Token)
         End Sub
         Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
-            Return M3U8.Download(Media.URL, Media.PictureOption, Settings.FfmpegFile, DestinationFile)
+            Return M3U8.Download(Media.URL, Media.PictureOption, DestinationFile)
         End Function
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
                                                           Optional ByVal EObj As Object = Nothing) As Integer

SCrawler/API/Xhamster/Declarations.vb

@@ -0,0 +1,19 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+'Imports System.Globalization
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.Xhamster
+    Friend Module Declarations
+        Friend Const XhamsterSiteKey As String = "AndyProgram_XHamster"
+        Friend ReadOnly HtmlScript As RParams = RParams.DMS("\<script id='initials-script'\>window.initials=(\{.+?\});\</script\>", 1, EDP.ReturnValue,
+                                                            CType(Function(Input$) Input.StringTrim, Func(Of String, String)))
+        Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
+        Friend ReadOnly FirstM3U8FileRegEx As RParams = RParams.DM("RESOLUTION=\d+x(\d+).*?[\r\n]+?([^#]*?\.m3u8.*)", 0, RegexReturn.List)
+    End Module
+End Namespace

SCrawler/API/Xhamster/M3U8.vb

@@ -0,0 +1,79 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.API.Base.M3U8Declarations
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.Xhamster
+    Friend NotInheritable Class M3U8
+        Private Sub New()
+        End Sub
+        Private Shared Function ParseFirstM3U8(ByVal URL As String, ByVal Responser As Responser, ByVal UHD As Boolean) As String
+            Dim r$, d$
+            Dim _DataObtained As Boolean = False
+            For i% = 0 To 1
+                Try
+                    Responser.UseGZipStream = i
+                    r = Responser.GetResponse(URL.Replace("\", String.Empty))
+                    If Not r.IsEmptyString Then
+                        r = r.StringFormatLines
+                        Dim sList As List(Of Sizes) = RegexFields(Of Sizes)(r, {FirstM3U8FileRegEx}, {1, 2})
+                        If sList.ListExists Then _DataObtained = True : sList.RemoveAll(Function(sv) sv.HasError Or sv.Data.IsEmptyString Or
+                                                                                                     sv.Value = 0 Or (Not UHD And sv.Value > 1080))
+                        If sList.ListExists Then
+                            sList.Sort()
+                            d = sList.First.Data.Trim
+                            If Not d.IsEmptyString Then Return d
+                        End If
+                    End If
+                Catch
+                End Try
+                If _DataObtained Then Exit For
+            Next
+            Return String.Empty
+        End Function
+        Private Shared Function ParseSecondM3U8(ByVal URL As String, ByVal Responser As Responser, ByVal Appender As String) As List(Of String)
+            Dim r$
+            Dim l As List(Of String)
+            For i% = 0 To 1
+                Try
+                    Responser.UseGZipStream = i
+                    r = Responser.GetResponse(URL)
+                    If Not r.IsEmptyString Then
+                        l = RegexReplace(r, TsFilesRegEx)
+                        If l.ListExists Then
+                            For indx% = 0 To l.Count - 1 : l(indx) = M3U8Base.CreateUrl(Appender, l(indx)) : Next
+                            Return l
+                        End If
+                    End If
+                Catch
+                End Try
+            Next
+            Return Nothing
+        End Function
+        Private Shared Function ObtainUrls(ByVal URL As String, ByVal Responser As Responser, ByVal UHD As Boolean) As List(Of String)
+            Try
+                Dim file$ = ParseFirstM3U8(URL, Responser, UHD)
+                If Not file.IsEmptyString Then
+                    Responser.UseGZipStream = False
+                    Dim appender$ = URL.Replace(URL.Split("/").LastOrDefault, String.Empty)
+                    URL = M3U8Base.CreateUrl(appender, file)
+                    Dim l As List(Of String) = ParseSecondM3U8(URL, Responser, appender)
+                    If l.ListExists Then Return l
+                End If
+                Return Nothing
+            Finally
+                Responser.UseGZipStream = False
+            End Try
+        End Function
+        Friend Shared Function Download(ByVal Media As UserMedia, ByVal Responser As Responser, ByVal UHD As Boolean) As SFile
+            Return M3U8Base.Download(ObtainUrls(Media.URL, Responser, UHD), Media.File, Responser)
+        End Function
+    End Class
+End Namespace

SCrawler/API/Xhamster/SiteSettings.vb

@@ -0,0 +1,155 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.API.BaseObjects
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace API.Xhamster
+    <Manifest(XhamsterSiteKey), SavedPosts, SpecialForm(True), TaskGroup(SettingsCLS.TaskStackNamePornSite)>
+    Friend Class SiteSettings : Inherits SiteSettingsBase : Implements IDomainContainer
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon Implements IDomainContainer.Icon
+            Get
+                Return My.Resources.SiteResources.XhamsterIcon_32
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.XhamsterPic_32
+            End Get
+        End Property
+#Region "Domains"
+        Private ReadOnly Property IDomainContainer_Site As String Implements IDomainContainer.Site
+            Get
+                Return Site
+            End Get
+        End Property
+        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue Implements IDomainContainer.DomainsSettingProp
+        Friend ReadOnly Property Domains As List(Of String) Implements IDomainContainer.Domains
+        Private ReadOnly Property DomainsTemp As List(Of String) Implements IDomainContainer.DomainsTemp
+        Private Property DomainsChanged As Boolean = False Implements IDomainContainer.DomainsChanged
+        Friend ReadOnly Property DomainsUpdated As Boolean
+            Get
+                Return DomainsUpdatedBySite
+            End Get
+        End Property
+        Private ReadOnly Property DomainsDefault As String = "xhamster.com" Implements IDomainContainer.DomainsDefault
+#End Region
+        <PropertyOption(ControlText:="Download UHD", ControlToolTip:="Download UHD (4K) content"), PXML>
+        Friend Property DownloadUHD As PropertyValue
+        Private Property Initialized As Boolean = False Implements IDomainContainer.Initialized
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("XHamster", "xhamster.com")
+
+            Responser.DeclaredError = EDP.ThrowException
+
+            Domains = New List(Of String)
+            DomainsTemp = New List(Of String)
+            SiteDomains = New PropertyValue(DomainsDefault, GetType(String), Sub(s) UpdateDomains())
+            DownloadUHD = New PropertyValue(False)
+
+            UrlPatternUser = "https://xhamster.com/users/{0}"
+            UserRegex = RParams.DMS("xhamster.com/users/([^/]+).*?", 1)
+            ImageVideoContains = "xhamster"
+        End Sub
+        Friend Overrides Sub EndInit()
+            Initialized = True
+            DomainContainer.EndInit(Me)
+            DomainsTemp.ListAddList(Domains)
+            MyBase.EndInit()
+        End Sub
+#End Region
+#Region "UpdateDomains"
+        Private Property DomainsUpdateInProgress As Boolean = False Implements IDomainContainer.DomainsUpdateInProgress
+        Private Property DomainsUpdatedBySite As Boolean = False Implements IDomainContainer.DomainsUpdatedBySite
+        Friend Overloads Sub UpdateDomains() Implements IDomainContainer.UpdateDomains
+            DomainContainer.UpdateDomains(Me)
+        End Sub
+        Friend Overloads Sub UpdateDomains(ByVal NewDomains As IEnumerable(Of String), ByVal Internal As Boolean)
+            DomainContainer.UpdateDomains(Me, NewDomains, Internal)
+        End Sub
+#End Region
+#Region "Edit"
+        Friend Overrides Sub Update()
+            DomainContainer.Update(Me)
+            Responser.SaveSettings()
+            MyBase.Update()
+        End Sub
+        Friend Overrides Sub EndEdit()
+            DomainContainer.EndEdit(Me)
+            MyBase.EndEdit()
+        End Sub
+        Friend Overrides Sub OpenSettingsForm()
+            DomainContainer.OpenSettingsForm(Me)
+        End Sub
+#End Region
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            If What = ISiteSettings.Download.SavedPosts Then
+                Return New UserData With {.IsSavedPosts = True, .User = New UserInfo With {.Name = "xhamster"}}
+            Else
+                Return New UserData
+            End If
+        End Function
+        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
+            If Available(ISiteSettings.Download.Main, True) Then
+                Using resp As Responser = Responser.Copy
+                    Dim spf$ = String.Empty
+                    Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
+                    Dim m As UserMedia = UserData.GetVideoInfo(URL, resp, f)
+                    If m.State = UserMedia.States.Downloaded Then
+                        m.SpecialFolder = f
+                        Return {m}
+                    End If
+                End Using
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            If Settings.UseM3U8 AndAlso MyBase.Available(What, Silent) Then
+                If What = ISiteSettings.Download.SavedPosts Then
+                    Return Responser.CookiesExists
+                Else
+                    Return True
+                End If
+            Else
+                Return False
+            End If
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Return Media.URL_BASE
+        End Function
+#Region "Is my user/data"
+        Private Const UserRegexDefault As String = "{0}/users/([^/]+).*?"
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            Dim b As ExchangeOptions = MyBase.IsMyUser(UserURL)
+            If b.Exists Then Return b
+            If Not UserURL.IsEmptyString And Domains.Count > 0 Then
+                Dim uName$, fStr$
+                Dim uErr As New ErrorsDescriber(EDP.ReturnValue)
+                For i% = 0 To Domains.Count - 1
+                    fStr = String.Format(UserRegexDefault, Domains(i))
+                    uName = RegexReplace(UserURL, RParams.DMS(fStr, 1, uErr))
+                    If Not uName.IsEmptyString Then Return New ExchangeOptions(Site, uName)
+                Next
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
+            If Not URL.IsEmptyString And Domains.Count > 0 Then
+                If Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions With {.UserName = URL, .Exists = True}
+            End If
+            Return Nothing
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/Xhamster/UserData.vb

@@ -0,0 +1,344 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Namespace API.Xhamster
+    Friend Class UserData : Inherits UserDataBase
+#Region "Declarations"
+        Private ReadOnly Property MySettings As SiteSettings
+            Get
+                Return DirectCast(HOST.Source, SiteSettings)
+            End Get
+        End Property
+        Private Structure ExchObj
+            Friend IsPhoto As Boolean
+        End Structure
+        Private ReadOnly _TempPhotoData As List(Of UserMedia)
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+        End Sub
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            UseInternalM3U8Function = True
+            _TempPhotoData = New List(Of UserMedia)
+        End Sub
+#End Region
+#Region "Download base functions"
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            _TempPhotoData.Clear()
+            If DownloadVideos Then DownloadData(1, True, Token)
+            If DownloadImages Then
+                DownloadData(1, False, Token)
+                ReparsePhoto(Token)
+            End If
+        End Sub
+        Private Overloads Sub DownloadData(ByVal Page As Integer, ByVal IsVideo As Boolean, ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Dim MaxPage% = -1
+                Dim Type As UTypes = IIf(IsVideo, UTypes.VideoPre, UTypes.Picture)
+                Dim mPages$ = IIf(IsVideo, "maxVideoPages", "maxPhotoPages")
+                Dim listNode$()
+                Dim m As UserMedia
+
+                If IsSavedPosts Then
+                    URL = $"https://xhamster.com/my/favorites/{IIf(IsVideo, "videos", "photos-and-galleries")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
+                    listNode = If(IsVideo, {"favoriteVideoListComponent", "models"}, {"favoritesGalleriesAndPhotosCollection"})
+                Else
+                    URL = $"https://xhamster.com/users/{Name}/{IIf(IsVideo, "videos", "photos")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
+                    listNode = {If(IsVideo, "userVideoCollection", "userGalleriesCollection")}
+                End If
+                ThrowAny(Token)
+
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then r = RegexReplace(r, HtmlScript)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
+                        If j.Count > 0 Then
+                            If Not MySettings.DomainsUpdated AndAlso j.Contains("trustURLs") Then _
+                               MySettings.UpdateDomains(j("trustURLs").Select(Function(d) d(0).XmlIfNothingValue), False)
+
+                            MaxPage = j.Value(mPages).FromXML(Of Integer)(-1)
+
+                            With j(listNode)
+                                If .ListExists Then
+                                    For Each e As EContainer In .Self
+                                        m = ExtractMedia(e, Type)
+                                        If Not m.URL.IsEmptyString Then
+                                            If m.File.IsEmptyString Then Continue For
+
+                                            If m.Post.Date.HasValue Then
+                                                Select Case CheckDatesLimit(m.Post.Date.Value, Nothing)
+                                                    Case DateResult.Skip : Continue For
+                                                    Case DateResult.Exit : Exit Sub
+                                                End Select
+                                            End If
+
+                                            If IsVideo AndAlso Not _TempPostsList.Contains(m.Post.ID) Then
+                                                _TempPostsList.Add(m.Post.ID)
+                                                _TempMediaList.ListAddValue(m, LNC)
+                                            ElseIf Not IsVideo Then
+                                                If DirectCast(m.Object, ExchObj).IsPhoto Then
+                                                    If Not m.Post.ID.IsEmptyString AndAlso Not _TempPostsList.Contains(m.Post.ID) Then
+                                                        _TempPostsList.Add(m.Post.ID)
+                                                        _TempMediaList.ListAddValue(m, LNC)
+                                                    End If
+                                                Else
+                                                    _TempPhotoData.ListAddValue(m, LNC)
+                                                End If
+                                            Else
+                                                Exit Sub
+                                            End If
+                                        End If
+                                    Next
+                                End If
+                            End With
+                        End If
+                    End Using
+                End If
+
+                If MaxPage > 0 AndAlso Page < MaxPage Then DownloadData(Page + 1, IsVideo, Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+#End Region
+#Region "Reparse video, photo"
+        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(tm) tm.Type = UTypes.VideoPre) Then
+                    Dim m As UserMedia, m2 As UserMedia
+                    For i% = _TempMediaList.Count - 1 To 0 Step -1
+                        If _TempMediaList(i).Type = UTypes.VideoPre Then
+                            m = _TempMediaList(i)
+                            If Not m.URL_BASE.IsEmptyString Then
+                                m2 = Nothing
+                                If GetM3U8(m2, m.URL_BASE, Responser) Then
+                                    m2.URL_BASE = m.URL_BASE
+                                    _TempMediaList(i) = m2
+                                Else
+                                    m.State = UserMedia.States.Missing
+                                    _TempMediaList(i) = m
+                                End If
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "video reparsing error", False)
+            End Try
+        End Sub
+        Private Overloads Sub ReparsePhoto(ByVal Token As CancellationToken)
+            If _TempPhotoData.Count > 0 Then
+                For i% = 0 To _TempPhotoData.Count - 1 : ReparsePhoto(i, 1, Token) : Next
+                _TempPhotoData.Clear()
+            End If
+        End Sub
+        Private Overloads Sub ReparsePhoto(ByVal Index As Integer, ByVal Page As Integer, ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Dim MaxPage% = -1
+                Dim m As UserMedia
+                Dim sm As UserMedia = _TempPhotoData(Index)
+
+                URL = $"{sm.URL}{IIf(Page = 1, String.Empty, $"/{Page}")}"
+                ThrowAny(Token)
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then r = RegexReplace(r, HtmlScript)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
+                        If j.Count > 0 Then
+                            MaxPage = j.Value({"pagination"}, "maxPage").FromXML(Of Integer)(-1)
+                            With j({"photosGalleryModel"}, "photos")
+                                If .ListExists Then
+                                    For Each e In .Self
+                                        m = ExtractMedia(e, UTypes.Picture, "imageURL", False, sm.Post.Date)
+                                        m.URL_BASE = sm.URL
+                                        If Not m.URL.IsEmptyString Then
+                                            m.Post.ID = $"{sm.Post.ID}_{m.Post.ID}"
+                                            m.SpecialFolder = sm.SpecialFolder
+                                            If Not _TempPostsList.Contains(m.Post.ID) Then
+                                                _TempPostsList.Add(m.Post.ID)
+                                                _TempMediaList.ListAddValue(m, LNC)
+                                            Else
+                                                Exit Sub
+                                            End If
+                                        End If
+                                    Next
+                                End If
+                            End With
+                        End If
+                    End Using
+                End If
+
+                If MaxPage > 0 AndAlso Page < MaxPage Then ReparsePhoto(Index, Page + 1, Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, "photo reparsing error", False)
+            End Try
+        End Sub
+#End Region
+#Region "Reparse missing"
+        Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
+            Dim rList As New List(Of Integer)
+            Try
+                If ContentMissingExists Then
+                    Dim m As UserMedia, m2 As UserMedia
+                    For i% = 0 To _ContentList.Count - 1
+                        m = _ContentList(i)
+                        If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then
+                            ThrowAny(Token)
+                            m2 = Nothing
+                            If GetM3U8(m2, m.URL_BASE, Responser) Then
+                                m2.URL_BASE = m.URL_BASE
+                                _TempMediaList.ListAddValue(m2, LNC)
+                                rList.Add(i)
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "missing data downloading error")
+            Finally
+                If rList.Count > 0 Then
+                    For i% = rList.Count - 1 To 0 Step -1 : _ContentList.RemoveAt(rList(i)) : Next
+                    rList.Clear()
+                End If
+            End Try
+        End Sub
+#End Region
+#Region "GetM3U8"
+        Private Overloads Function GetM3U8(ByRef m As UserMedia, ByVal URL As String, ByVal Responser As Responser,
+                                           Optional ByVal e As ErrorsDescriber = Nothing) As Boolean
+            Try
+                If Not URL.IsEmptyString Then
+                    Dim r$ = Responser.GetResponse(URL)
+                    If Not r.IsEmptyString Then r = RegexReplace(r, HtmlScript)
+                    If Not r.IsEmptyString Then
+                        Using j As EContainer = JsonDocument.Parse(r)
+                            If j.ListExists Then
+                                m = ExtractMedia(j("videoModel"), UTypes.VideoPre)
+                                m.URL_BASE = URL
+                                Return GetM3U8(m, j)
+                            End If
+                        End Using
+                    End If
+                End If
+                Return False
+            Catch ex As Exception
+                If Not e.Exists Then e = EDP.ReturnValue
+                Return ErrorsDescriber.Execute(e, ex, $"[{ToStringForLog()}]: API.Xhamster.GetM3U8({URL})", False)
+            End Try
+        End Function
+        Private Overloads Function GetM3U8(ByRef m As UserMedia, ByVal j As EContainer) As Boolean
+            Dim url$ = j.Value({"xplayerSettings", "sources", "hls"}, "url")
+            If Not url.IsEmptyString Then m.URL = url : m.Type = UTypes.m3u8 : Return True
+            Return False
+        End Function
+#End Region
+#Region "Standalone downloader"
+        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, ByVal Path As SFile) As UserMedia
+            Try
+                Using u As New UserData With {.Responser = Responser, .HOST = Settings(XhamsterSiteKey)}
+                    Dim m As UserMedia = Nothing
+                    If u.GetM3U8(m, URL, Responser, EDP.ThrowException) Then
+                        m.File.Path = Path.Path
+                        Dim f As SFile = u.DownloadM3U8(m.URL, m, m.File)
+                        If Not f.IsEmptyString Then
+                            m.File = f
+                            m.State = UserMedia.States.Downloaded
+                            Return m
+                        End If
+                    End If
+                End Using
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"XHamster standalone download error: [{URL}]", New UserMedia)
+            End Try
+        End Function
+#End Region
+#Region "Download data"
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overloads Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+            Media.File = DestinationFile
+            Return M3U8.Download(Media, Responser, MySettings.DownloadUHD.Value)
+        End Function
+#End Region
+#Region "Create media"
+        Private Shared Function ExtractMedia(ByVal j As EContainer, ByVal t As UTypes, Optional ByVal UrlNode As String = "pageURL",
+                                             Optional ByVal DetectGalery As Boolean = True, Optional ByVal PostDate As Date? = Nothing) As UserMedia
+            If Not j Is Nothing Then
+                Dim m As New UserMedia(j.Value(UrlNode).Replace("\", String.Empty), t) With {
+                    .Post = New UserPost With {
+                        .ID = j.Value("id"),
+                        .Date = AConvert(Of Date)(j.Value("created"), DateProvider, Nothing)
+                    },
+                    .PictureOption = j.Value("title").StringRemoveWinForbiddenSymbols,
+                    .Object = New ExchObj
+                }
+                If PostDate.HasValue Then m.Post.Date = PostDate
+                Dim setSpecialFolder As Boolean = False
+                Dim processFile As Boolean = True
+                Dim ext$ = "mp4"
+                If t = UTypes.Picture Then
+                    ext = "jpg"
+                    If (Not DetectGalery OrElse j.Contains("galleryId")) AndAlso Not j.Value("imageURL").IsEmptyString Then
+                        m.Object = New ExchObj With {.IsPhoto = True}
+                        m.URL = j.Value("imageURL")
+                        m.URL_BASE = m.URL
+                        If DetectGalery Then m.Post.ID = $"{j.Value("galleryId")}_{m.Post.ID}"
+                        m.File = m.URL
+                        m.File.Separator = "\"
+                        processFile = m.File.File.IsEmptyString
+                    Else
+                        setSpecialFolder = True
+                    End If
+                End If
+                If Not m.URL.IsEmptyString Then
+                    If m.Post.ID.IsEmptyString Then m.Post.ID = m.URL.Split("/").LastOrDefault
+                    If m.PictureOption.IsEmptyString Then m.PictureOption = j.Value("titleLocalized").StringRemoveWinForbiddenSymbols
+                    If m.PictureOption.IsEmptyString Then m.PictureOption = m.Post.ID
+                    If setSpecialFolder Then m.SpecialFolder = m.PictureOption
+
+                    If processFile Then
+                        If Not m.PictureOption.IsEmptyString Then
+                            m.File = $"{m.PictureOption}.{ext}"
+                        ElseIf Not m.Post.ID.IsEmptyString Then
+                            m.File = $"{m.Post.ID}.{ext}"
+                        End If
+                    End If
+                    m.File.Separator = "\"
+                End If
+                Return m
+            Else
+                Return Nothing
+            End If
+        End Function
+#End Region
+#Region "Exception"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            Return If(Responser.Status = Net.WebExceptionStatus.ConnectionClosed, 1, 0)
+        End Function
+#End Region
+#Region "Idisposable support"
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            If Not disposedValue And disposing Then _TempPhotoData.Clear()
+            MyBase.Dispose(disposing)
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/Channels/ChannelViewForm.Designer.vb

@@ -140,24 +140,28 @@ Partial Friend Class ChannelViewForm : Inherits System.Windows.Forms.Form
         '
         'BTT_C_OPEN_USER
         '
+        Me.BTT_C_OPEN_USER.Image = Global.SCrawler.My.Resources.Resources.GlobePic_32
         Me.BTT_C_OPEN_USER.Name = "BTT_C_OPEN_USER"
         Me.BTT_C_OPEN_USER.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_OPEN_USER.Text = "Open user"
         '
         'BTT_C_OPEN_POST
         '
+        Me.BTT_C_OPEN_POST.Image = Global.SCrawler.My.Resources.Resources.GlobePic_32
         Me.BTT_C_OPEN_POST.Name = "BTT_C_OPEN_POST"
         Me.BTT_C_OPEN_POST.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_OPEN_POST.Text = "Open post"
         '
         'BTT_C_OPEN_PICTURE
         '
+        Me.BTT_C_OPEN_PICTURE.Image = Global.SCrawler.My.Resources.Resources.PicturePic_32
         Me.BTT_C_OPEN_PICTURE.Name = "BTT_C_OPEN_PICTURE"
         Me.BTT_C_OPEN_PICTURE.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_OPEN_PICTURE.Text = "Open picture"
         '
         'BTT_C_OPEN_FOLDER
         '
+        Me.BTT_C_OPEN_FOLDER.Image = Global.SCrawler.My.Resources.Resources.FolderPic_32
         Me.BTT_C_OPEN_FOLDER.Name = "BTT_C_OPEN_FOLDER"
         Me.BTT_C_OPEN_FOLDER.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_OPEN_FOLDER.Text = "Open folder"
@@ -165,6 +169,7 @@ Partial Friend Class ChannelViewForm : Inherits System.Windows.Forms.Form
         'BTT_C_REMOVE_FROM_SELECTED
         '
         Me.BTT_C_REMOVE_FROM_SELECTED.AutoToolTip = True
+        Me.BTT_C_REMOVE_FROM_SELECTED.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
         Me.BTT_C_REMOVE_FROM_SELECTED.Name = "BTT_C_REMOVE_FROM_SELECTED"
         Me.BTT_C_REMOVE_FROM_SELECTED.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_REMOVE_FROM_SELECTED.Text = "Remove user from selected"
@@ -172,6 +177,7 @@ Partial Friend Class ChannelViewForm : Inherits System.Windows.Forms.Form
         '
         'BTT_C_ADD_TO_BLACKLIST
         '
+        Me.BTT_C_ADD_TO_BLACKLIST.Image = Global.SCrawler.My.Resources.Resources.DBPic_32
         Me.BTT_C_ADD_TO_BLACKLIST.Name = "BTT_C_ADD_TO_BLACKLIST"
         Me.BTT_C_ADD_TO_BLACKLIST.Size = New System.Drawing.Size(305, 22)
         Me.BTT_C_ADD_TO_BLACKLIST.Text = "Add/Remove this user to/from the BlackList"

SCrawler/Channels/ChannelViewForm.vb

@@ -581,7 +581,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
             End If
         End If
     End Sub
-    Private Sub CMB_CHANNELS_ActionSelectedItemChanged(ByVal _Item As ListViewItem) Handles CMB_CHANNELS.ActionSelectedItemChanged
+    Private Sub CMB_CHANNELS_ActionSelectedItemChanged(ByVal Sender As Object, ByVal e As EventArgs, ByVal Item As ListViewItem) Handles CMB_CHANNELS.ActionSelectedItemChanged
         SetLimitsByChannel()
         Dim c As Channel = GetCurrentChannel()
         If Not c Is Nothing Then MyRange.Source = c
@@ -814,6 +814,14 @@ Friend Class ChannelViewForm : Implements IChannelLimits
     End Sub
 #End Region
 #Region "MyRange"
+    Private ReadOnly GetListImage_Error As New ErrorsDescriber(EDP.ReturnValue)
+    Private Function GetListImage(ByVal Post As UserPost, ByVal s As Size, ByVal NullArg As Image) As Image
+        If Not Post.CachedFile.IsEmptyString Then
+            Return If(ImageRenderer.GetImage(SFile.GetBytes(Post.CachedFile), s, GetListImage_Error), NullArg.Clone)
+        Else
+            Return NullArg.Clone
+        End If
+    End Function
     Private Sub MyRange_IndexChanged(ByVal Sender As Object, ByVal e As EventArgs) Handles MyRange.IndexChanged
         Try
             If MyDefs.Initializing Then Exit Sub
@@ -825,11 +833,10 @@ Friend Class ChannelViewForm : Implements IChannelLimits
                 If .Count > 0 Then
                     Dim s As Size = GetImageSize()
                     Dim NullImage As Image = New Bitmap(s.Width, s.Height)
-                    Dim ie As New ErrorsDescriber(EDP.ReturnValue)
                     For i% = 0 To .Count - 1
                         p = .Item(i)
                         With p
-                            LIST_POSTS.LargeImageList.Images.Add(.GetImage(s, ie, NullImage))
+                            LIST_POSTS.LargeImageList.Images.Add(GetListImage(p, s, NullImage))
                             LIST_POSTS.Items.Add(New ListViewItem(.UserID, i) With {.Tag = p.ID})
                             With LIST_POSTS.Items(LIST_POSTS.Items.Count - 1)
                                 If PendingUsers.Contains(.Text) Then .Checked = True

SCrawler/Download/ActiveDownloadingProgress.Designer.vb

@@ -22,7 +22,6 @@ Namespace DownloadObjects
         Private components As System.ComponentModel.IContainer
         <System.Diagnostics.DebuggerStepThrough()>
         Private Sub InitializeComponent()
-            Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(ActiveDownloadingProgress))
             Me.TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
             Me.SuspendLayout()
             '
@@ -36,8 +35,8 @@ Namespace DownloadObjects
             Me.TP_MAIN.Location = New System.Drawing.Point(0, 0)
             Me.TP_MAIN.Name = "TP_MAIN"
             Me.TP_MAIN.RowCount = 1
-            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 64.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 66.0!))
-            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 64.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 66.0!))
             Me.TP_MAIN.Size = New System.Drawing.Size(434, 61)
             Me.TP_MAIN.TabIndex = 0
             '
@@ -47,12 +46,13 @@ Namespace DownloadObjects
             Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
             Me.ClientSize = New System.Drawing.Size(434, 61)
             Me.Controls.Add(Me.TP_MAIN)
-            Me.Icon = Global.SCrawler.My.Resources.ArrowDownIcon_Blue_24
+            Me.Icon = Global.SCrawler.My.Resources.Resources.ArrowDownIcon_Blue_24
             Me.KeyPreview = True
             Me.MinimumSize = New System.Drawing.Size(450, 100)
             Me.Name = "ActiveDownloadingProgress"
             Me.Text = "Active downloading progress"
             Me.ResumeLayout(False)
+
         End Sub
         Private WithEvents TP_MAIN As TableLayoutPanel
     End Class

SCrawler/Download/ActiveDownloadingProgress.vb

@@ -42,6 +42,7 @@ Namespace DownloadObjects
         End Sub
         Private Sub Downloader_Reconfigured()
             Const RowHeight% = 30
+            Const LowestValue% = 39
             Dim a As Action = Sub()
                                   With TP_MAIN
                                       If .Controls.Count > 0 Then
@@ -65,13 +66,18 @@ Namespace DownloadObjects
                                                   .Controls.Add(JobsList.Last.Get, 0, .RowStyles.Count - 1)
                                               End With
                                           Next
-                                          TP_MAIN.RowStyles.Add(New RowStyle(SizeType.Percent, 100))
+                                          TP_MAIN.RowStyles.Add(New RowStyle(SizeType.AutoSize))
                                           TP_MAIN.RowCount += 1
+
+                                          Dim s As Size = Size
+                                          Dim ss As Size = Screen.PrimaryScreen.WorkingArea.Size
+                                          Dim c% = TP_MAIN.RowStyles.Count - 1
+                                          s.Height = c * RowHeight + LowestValue + (PaddingE.GetOf({TP_MAIN}).Vertical(c) / c).RoundDown - c
+                                          If s.Height > ss.Height Then s.Height = ss.Height
+                                          MinimumSize = Nothing
+                                          Size = s
+                                          MinimumSize = New Size(MinWidth, s.Height)
                                       End If
-                                      Dim s As Size = Size
-                                      s.Height = TP_MAIN.RowStyles.Count * RowHeight + PaddingE.GetOf({TP_MAIN}).Vertical(TP_MAIN.RowStyles.Count) - TP_MAIN.RowStyles.Count * 2
-                                      MinimumSize = New Size(MinWidth, s.Height)
-                                      Size = s
                                   End With
                                   TP_MAIN.Refresh()
                               End Sub

SCrawler/Download/Automation/AutoDownloader.vb

@@ -7,7 +7,6 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports SCrawler.API
 Imports SCrawler.API.Base
 Imports SCrawler.DownloadObjects.Groups
 Imports PersonalUtilities.Functions.XML
@@ -421,21 +420,32 @@ Namespace DownloadObjects
                 LastDownloadDate = LastDownloadDate.AddMinutes(Timer)
             End If
         End Sub
+        Private _SpecialDelayUse As Boolean = False
+        Private _SpecialDelayTime As Date? = Nothing
         Private Sub Checker()
             Try
                 Dim _StartDownload As Boolean
                 While (Not _StopRequested Or Downloader.Working) And Not Mode = Modes.None
                     If LastDownloadDate.AddMinutes(Timer) < Now And _StartTime.AddMinutes(StartupDelay) < Now And
-                       Not Downloader.Working And Not IsPaused And Not _StopRequested And Not Mode = Modes.None Then
+                       Not IsPaused And Not _StopRequested And Not Mode = Modes.None Then
-                        _StartDownload = False
+                        If Downloader.Working Then
-                        If Settings.Automation.Count = 1 Then
+                            _SpecialDelayUse = True
-                            _StartDownload = True
-                        ElseIf Index = -1 Then
-                            _StartDownload = True
                         Else
-                            _StartDownload = NextExecutionDate.AddMilliseconds(1000 * (Index + 1)).Ticks <= Now.Ticks
+                            If _SpecialDelayUse And Not _SpecialDelayTime.HasValue Then _SpecialDelayTime = Now.AddSeconds(10)
+                            If Not _SpecialDelayUse OrElse (_SpecialDelayTime.HasValue AndAlso _SpecialDelayTime.Value < Now) Then
+                                _SpecialDelayUse = False
+                                _SpecialDelayTime = Nothing
+                                _StartDownload = False
+                                If Settings.Automation.Count = 1 Then
+                                    _StartDownload = True
+                                ElseIf Index = -1 Then
+                                    _StartDownload = True
+                                Else
+                                    _StartDownload = NextExecutionDate.AddMilliseconds(1000 * (Index + 1)).Ticks <= Now.Ticks
+                                End If
+                                If _StartDownload Then Download()
+                            End If
                         End If
-                        If _StartDownload Then Download()
                     End If
                     Thread.Sleep(500)
                 End While
@@ -461,18 +471,25 @@ Namespace DownloadObjects
                 Dim DownloadedUsersCount% = 0
                 Dim simple As Boolean = ShowSimpleNotification And ShowNotifications
                 Dim notify As Action = Sub()
-                                           With Downloader.Downloaded
+                                           Try
-                                               If ShowNotifications And .Count > 0 Then .ForEach(Sub(ByVal u As IUserData)
+                                               With Downloader.Downloaded
-                                                                                                     If Keys.Contains(u.Key) Then
+                                                   If ShowNotifications And .Count > 0 Then
-                                                                                                         If simple Then
+                                                       For indx% = 0 To .Count - 1
-                                                                                                             DownloadedUsersCount += 1
+                                                           With .Item(indx)
-                                                                                                         Else
+                                                               If Keys.Contains(.Key) Then
-                                                                                                             ShowNotification(u)
+                                                                   If simple Then
-                                                                                                         End If
+                                                                       DownloadedUsersCount += 1
-                                                                                                         Keys.Remove(u.Key)
+                                                                   Else
-                                                                                                     End If
+                                                                       ShowNotification(.Self)
-                                                                                                 End Sub)
+                                                                   End If
-                                           End With
+                                                                   Keys.Remove(.Key)
+                                                               End If
+                                                           End With
+                                                       Next
+                                                   End If
+                                               End With
+                                           Catch n_ex As Exception
+                                           End Try
                                        End Sub
                 Select Case Mode
                     Case Modes.All

SCrawler/Download/DownloadedInfoForm.vb

@@ -150,8 +150,8 @@ Namespace DownloadObjects
         Private Sub BTT_FIND_Click(sender As Object, e As EventArgs) Handles BTT_FIND.Click
             Try
                 If _LatestSelected.ValueBetween(0, LIST_DOWN.Items.Count - 1) AndAlso _LatestSelected.ValueBetween(0, Downloader.Downloaded.Count - 1) Then
-                    Dim i% = Settings.Users.IndexOf(_TempUsersList(_LatestSelected))
+                    Dim u As IUserData = Settings.GetUser(_TempUsersList(_LatestSelected), True)
-                    If i >= 0 Then RaiseEvent UserFind(Settings.Users(i).Key)
+                    If Not u Is Nothing Then RaiseEvent UserFind(u.Key)
                 End If
             Catch ex As Exception
             End Try

SCrawler/Download/Feed/DownloadFeedForm.vb

@@ -157,7 +157,7 @@ Namespace DownloadObjects
                     Using chooser As New SimpleListForm(Of SFile)(fList, Settings.Design) With {
                         .FormText = "Sessions",
                         .Icon = My.Resources.ArrowDownIcon_Blue_24,
-                        .Mode = SimpleListForm(Of SFile).Modes.CheckedItems,
+                        .Mode = SimpleListFormModes.CheckedItems,
                         .Provider = New CustomProvider(Function(v, d, p, n, ee) DirectCast(v, SFile).File)
                     }
                         chooser.ClearButtons()

SCrawler/Download/Feed/FeedMedia.vb

@@ -115,6 +115,7 @@ Namespace DownloadObjects
                         Case UserMedia.Types.Video, UserMedia.Types.m3u8
                             infoType = UserMedia.Types.Video
                             MyVideo = New FeedVideo(File) With {.Tag = File, .Dock = DockStyle.Fill, .ContextMenuStrip = CONTEXT_DATA}
+                            If MyVideo.HasError Then HasError = True
                             TP_MAIN.Controls.Add(MyVideo, 0, 1)
                             BTT_CONTEXT_OPEN_MEDIA.Text &= " video"
                             BTT_CONTEXT_DELETE.Text &= " video"

SCrawler/Download/Feed/FeedVideo.vb

@@ -10,7 +10,7 @@ Imports LibVLCSharp
 Imports System.Threading
 Imports System.ComponentModel
 Imports PersonalUtilities.Tools
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web
 Imports VLCState = LibVLCSharp.Shared.VLCState
 Namespace DownloadObjects
     <ToolboxItem(False), DesignTimeVisible(False)>
@@ -38,6 +38,7 @@ Namespace DownloadObjects
         Private VideoLengthMs As Integer = 0
         Private VideoLengthStr As String
         Private MediaFile As SFile = Nothing
+        Friend ReadOnly HasError As Boolean = False
         Public Sub New()
             InitializeComponent()
         End Sub
@@ -63,7 +64,6 @@ Namespace DownloadObjects
                                 MyVideo.BackgroundImageLayout = ImageLayout.Zoom
                             End If
                         Catch img_set_ex As Exception
-                            'TODELETE: FeedVideo set BackgroundImage error
                             ErrorsDescriber.Execute(EDP.SendInLog, img_set_ex, "Error setting background image for media player." & vbCr &
                                                                                $"File: {File}{vbCr}Image: {f}")
                         End Try
@@ -71,8 +71,8 @@ Namespace DownloadObjects
                 End If
                 UpdateButtons()
             Catch ex As Exception
-                'TODELETE: FeedVideo initialization error
                 ErrorsDescriber.Execute(EDP.SendInLog, ex, $"Media player initialization error({File})")
+                HasError = True
             End Try
         End Sub
         Private _Disposed As Boolean = False

SCrawler/Download/Groups/DownloadGroup.vb

@@ -6,7 +6,6 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports SCrawler.API
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base

SCrawler/Download/TDownloader.vb

@@ -138,6 +138,7 @@ Namespace DownloadObjects
                 Return If(Pool.Count = 0, 0, Pool.Sum(Function(j) j.Count))
             End Get
         End Property
+        Friend Property Suspended As Boolean = False
 #End Region
 #Region "Automation Support"
         Private _AutoDownloaderTasks As Integer = 0
@@ -166,9 +167,10 @@ Namespace DownloadObjects
             End Property
             Friend ReadOnly Property Name As String
                 Get
-                    Return Hosts(0).Name
+                    If Not GroupName.IsEmptyString Then Return GroupName Else Return Hosts(0).Name
                 End Get
             End Property
+            Friend ReadOnly Property GroupName As String
             Friend ReadOnly Property TaskCount As Integer
                 Get
                     Return Hosts(0).TaskCount
@@ -190,6 +192,10 @@ Namespace DownloadObjects
                 Keys = New List(Of String)
                 [Type] = JobType
             End Sub
+            Friend Sub New(ByVal JobType As Download, ByVal GroupName As String)
+                Me.New(JobType)
+                Me.GroupName = GroupName
+            End Sub
             Public Overloads Function Add(ByVal User As IUserData, ByVal _IncludedInTheFeed As Boolean) As Boolean
                 With DirectCast(User, UserDataBase)
                     If Keys.Count > 0 Then
@@ -274,6 +280,7 @@ Namespace DownloadObjects
 #Region "Pool"
         Friend Sub ReconfPool()
             If Pool.Count = 0 OrElse Not Pool.Exists(Function(j) j.Working Or j.Count > 0) Then
+                Dim i%
                 Pool.ListClearDispose
                 If Settings.Plugins.Count > 0 Then
                     Pool.Add(New Job(Download.Main))
@@ -281,6 +288,15 @@ Namespace DownloadObjects
                         If p.Settings.IsSeparatedTasks Then
                             Pool.Add(New Job(Download.Main))
                             Pool.Last.AddHost(p.Settings)
+                        ElseIf Not p.Settings.TaskGroupName.IsEmptyString Then
+                            i = -1
+                            If Pool.Count > 0 Then i = Pool.FindIndex(Function(pt) pt.GroupName = p.Settings.TaskGroupName)
+                            If i >= 0 Then
+                                Pool(i).AddHost(p.Settings)
+                            Else
+                                Pool.Add(New Job(Download.Main, p.Settings.TaskGroupName))
+                                Pool.Last.AddHost(p.Settings)
+                            End If
                         Else
                             Pool(0).AddHost(p.Settings)
                         End If
@@ -315,7 +331,7 @@ Namespace DownloadObjects
                 MyProgressForm.DisableProgressChange = False
                 Do While Pool.Exists(Function(p) p.Count > 0 Or p.Working)
                     For Each j As Job In Pool
-                        If j.Count > 0 And Not j.Working Then j.Start(New ThreadStart(Sub() StartDownloading(j)))
+                        If j.Count > 0 And Not j.Working And Not Suspended Then j.Start(New ThreadStart(Sub() StartDownloading(j)))
                     Next
                     Thread.Sleep(200)
                 Loop

SCrawler/Download/WebClient2.vb

@@ -0,0 +1,50 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Net
+Imports PersonalUtilities.Tools.Web.Clients
+Namespace DownloadObjects
+    Friend Class WebClient2 : Implements IDisposable
+        Protected WC As WebClient
+        Protected RC As Responser
+        Private ReadOnly RCERROR As New ErrorsDescriber(EDP.ThrowException)
+        Protected UseResponserClient As Boolean
+        Friend Sub New()
+        End Sub
+        Friend Sub New(ByVal Responser As Responser)
+            If Not Responser Is Nothing Then
+                RC = Responser
+                UseResponserClient = True
+            Else
+                WC = New WebClient
+            End If
+        End Sub
+        Friend Sub DownloadFile(ByVal URL As String, ByVal File As String)
+            If UseResponserClient Then
+                RC.DownloadFile(URL, File, RCERROR)
+            Else
+                WC.DownloadFile(URL, File)
+            End If
+        End Sub
+#Region "IDisposable Support"
+        Private disposedValue As Boolean = False
+        Protected Overridable Sub Dispose(ByVal disposing As Boolean)
+            If Not disposedValue And disposing And Not WC Is Nothing Then WC.Dispose()
+            disposedValue = True
+        End Sub
+        Protected Overrides Sub Finalize()
+            Dispose(False)
+            MyBase.Finalize()
+        End Sub
+        Friend Sub Dispose() Implements IDisposable.Dispose
+            Dispose(True)
+            GC.SuppressFinalize(Me)
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/Editors/CollectionEditorForm.vb

@@ -24,8 +24,9 @@ Namespace Editors
                     .MyViewInitialize()
                     .AddOkCancelToolbar()
                     Collections.ListAddList(Settings.LastCollections)
-                    Collections.ListAddList((From c In Settings.Users Where c.IsCollection Select c.CollectionName), LAP.NotContainsOnly, EDP.ThrowException)
+                    Dim ecol As List(Of String) = ListAddList(Nothing, (From c In Settings.Users Where c.IsCollection Select c.CollectionName), LAP.NotContainsOnly)
-                    If Collections.ListExists Then Collections.Sort() : CMB_COLLECTIONS.Items.AddRange(From c In Collections Select New ListItem(c))
+                    If ecol.ListExists Then ecol.Sort() : Collections.ListAddList(ecol, LAP.NotContainsOnly) : ecol.Clear()
+                    If Collections.Count > 0 Then CMB_COLLECTIONS.Items.AddRange(Collections.Select(Function(c) New ListItem(c)))
                     If Not Collection.IsEmptyString And Collections.Contains(Collection) Then CMB_COLLECTIONS.SelectedIndex = Collections.IndexOf(Collection)
                     .DelegateClosingChecker = False
                     .EndLoaderOperations()
@@ -55,8 +56,8 @@ Namespace Editors
         Private Sub CMB_COLLECTIONS_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As EventArgs) Handles CMB_COLLECTIONS.ActionOnButtonClick
             If Sender.DefaultButton = ActionButton.DefaultButtons.Add Then AddNewCollection()
         End Sub
-        Private Sub CMB_COLLECTIONS_ActionOnListDoubleClick(ByVal _Item As ListViewItem) Handles CMB_COLLECTIONS.ActionOnListDoubleClick
+        Private Sub CMB_COLLECTIONS_ActionOnListDoubleClick(ByVal Sender As Object, ByVal e As EventArgs, ByVal Item As ListViewItem) Handles CMB_COLLECTIONS.ActionOnListDoubleClick
-            _Item.Selected = True
+            Item.Selected = True
             MyDefs_ButtonOkClick()
         End Sub
         Private Sub AddNewCollection()

SCrawler/Editors/GlobalSettingsForm.vb

@@ -221,7 +221,7 @@ Namespace Editors
         End Sub
         Private Sub TXT_GLOBAL_PATH_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As EventArgs) Handles TXT_GLOBAL_PATH.ActionOnButtonClick
             If Sender.DefaultButton = ADB.Open Then
-                Dim f As SFile = SFile.SelectPath(Settings.GlobalPath.Value)
+                Dim f As SFile = SFile.SelectPath(Settings.GlobalPath.Value).IfNullOrEmpty(Settings.GlobalPath.Value)
                 If Not f.IsEmptyString Then TXT_GLOBAL_PATH.Text = f
             End If
         End Sub

SCrawler/Editors/LabelsForm.vb

@@ -9,7 +9,6 @@
 Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Forms.Controls
 Imports PersonalUtilities.Forms.Controls.Base
-Imports PersonalUtilities.Functions.Messaging
 Friend Class LabelsForm
     Private WithEvents MyDefs As DefaultFormOptions
     Friend ReadOnly Property LabelsList As List(Of String)
@@ -26,8 +25,6 @@ Friend Class LabelsForm
         End Get
     End Property
     Private _AnyLabelAdd As Boolean = False
-    Friend Property MultiUser As Boolean = False
-    Friend Property MultiUserClearExists As Boolean = False
     Friend Property WithDeleteButton As Boolean = False
     Private ReadOnly AddNoParsed As Boolean = False
     Friend Sub New(ByVal LabelsArr As IEnumerable(Of String), Optional ByVal AddNoParsed As Boolean = False)
@@ -47,12 +44,14 @@ Friend Class LabelsForm
                 .MyViewInitialize()
                 .AddOkCancelToolbar()
                 .MyOkCancel.BTT_DELETE.Visible = WithDeleteButton
-                If Source.Count > 0 Then
+                Dim s As List(Of String) = ListAddList(Nothing, Source).ListAddList(LabelsList, LAP.NotContainsOnly)
+                If s.ListExists Then
                     Dim items As New List(Of Integer)
+                    s.Sort()
                     CMB_LABELS.BeginUpdate()
-                    For i% = 0 To Source.Count - 1
+                    For i% = 0 To s.Count - 1
-                        If LabelsList.Contains(Source(i)) Then items.Add(i)
+                        If LabelsList.Contains(s(i)) Then items.Add(i)
-                        CMB_LABELS.Items.Add(Source(i))
+                        CMB_LABELS.Items.Add(s(i))
                     Next
                     If Not _Source Is Nothing Then CMB_LABELS.Buttons.Clear()
                     CMB_LABELS.EndUpdate()
@@ -72,24 +71,11 @@ Friend Class LabelsForm
     End Sub
     Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
         Try
-            If MultiUser Then
-                Dim m As New MMessage("You are changing labels for more one user" & vbNewLine & "What do you want to do?",
-                                      "MultiUser labels changing",
-                                      {New MsgBoxButton("Replace exists") With {.ToolTip = "Per user: all existing labels will be removed and replaced with these labels"},
-                                       New MsgBoxButton("Add to exists") With {.ToolTip = "Per user: these labels will be add to existing labels"},
-                                       New MsgBoxButton("Cancel")},
-                                      MsgBoxStyle.Exclamation)
-                Select Case MsgBoxE(m).Index
-                    Case 0 : MultiUserClearExists = True
-                    Case 1 : MultiUserClearExists = False
-                    Case 2 : Exit Sub
-                End Select
-            End If
             LabelsList.ListAddList(CMB_LABELS.Items.CheckedItems.Select(Function(l) CStr(l.Value(0))), LAP.ClearBeforeAdd, LAP.NotContainsOnly)
             If _AnyLabelAdd And _Source Is Nothing Then Settings.Labels.Update()
             MyDefs.CloseForm()
         Catch ex As Exception
-            ErrorsDescriber.Execute(EDP.LogMessageValue, ex, "Choosing labels")
+            ErrorsDescriber.Execute(EDP.LogMessageValue, ex, "Label selection")
         End Try
     End Sub
     Private Sub MyDefs_ButtonDeleteClickOC(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonDeleteClickOC

SCrawler/Editors/SiteEditorForm.vb

@@ -11,8 +11,7 @@ Imports SCrawler.Plugin.Hosts
 Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Forms.Controls
 Imports PersonalUtilities.Forms.Controls.Base
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Cookies
-Imports CookieControl = PersonalUtilities.Tools.WEB.CookieListForm.CookieControl
 Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
 Namespace Editors
     Friend Class SiteEditorForm
@@ -205,7 +204,7 @@ Namespace Editors
         End Sub
         Private Sub ChangePath(ByVal Sender As ActionButton, ByVal PathValue As SFile, ByRef CNT As TextBoxExtended)
             If Sender.DefaultButton = ADB.Open Then
-                Dim f As SFile = SFile.SelectPath(PathValue)
+                Dim f As SFile = SFile.SelectPath(PathValue).IfNullOrEmpty(PathValue)
                 If Not f.IsEmptyString Then CNT.Text = f
             End If
         End Sub
@@ -213,12 +212,13 @@ Namespace Editors
             Select Case Sender.DefaultButton
                 Case ADB.Edit
                     If Not Host.Responser Is Nothing Then
-                        Using f As New CookieListForm(Host.Responser) With {
+                        Using f As New CookieListForm With {.DesignXML = Settings.Design, .UseGrid = False}
-                            .MyDesignXML = Settings.Design,
+                            f.SetCollection(Host.Responser.Cookies)
-                            .DisableControls = CookieControl.AddFromInternal + CookieControl.AuthorizeProgram + CookieControl.OpenBrowser
-                        }
                             f.ShowDialog()
-                            MyDefs.MyOkCancel.EnableOK = True
+                            If f.DialogResult = DialogResult.OK Then
+                                f.GetCollection(Host.Responser)
+                                MyDefs.MyOkCancel.EnableOK = True
+                            End If
                         End Using
                         SetCookieText()
                     End If

SCrawler/Editors/UserCreatorForm.Designer.vb

@@ -23,99 +23,147 @@ Namespace Editors
         <System.Diagnostics.DebuggerStepThrough()>
         Private Sub InitializeComponent()
             Me.components = New System.ComponentModel.Container()
-            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
+            Dim TT_MAIN As System.Windows.Forms.ToolTip
-            Dim TP_SITE As System.Windows.Forms.TableLayoutPanel
+            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
             Dim ActionButton1 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(UserCreatorForm))
             Dim ListColumn1 As PersonalUtilities.Forms.Controls.Base.ListColumn = New PersonalUtilities.Forms.Controls.Base.ListColumn()
             Dim ListColumn2 As PersonalUtilities.Forms.Controls.Base.ListColumn = New PersonalUtilities.Forms.Controls.Base.ListColumn()
-            Dim TP_PARAMS As System.Windows.Forms.TableLayoutPanel
-            Dim TP_OTHER As System.Windows.Forms.TableLayoutPanel
             Dim ActionButton2 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton3 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton4 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
-            Dim TP_DOWN_OPTIONS As System.Windows.Forms.TableLayoutPanel
             Dim ActionButton5 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton6 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton7 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton8 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
-            Dim TT_MAIN As System.Windows.Forms.ToolTip
-            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
-            Me.TXT_USER = New PersonalUtilities.Forms.Controls.TextBoxExtended()
-            Me.CH_IS_CHANNEL = New System.Windows.Forms.CheckBox()
-            Me.CMB_SITE = New PersonalUtilities.Forms.Controls.ComboBoxExtended()
-            Me.BTT_OTHER_SETTINGS = New System.Windows.Forms.Button()
-            Me.CH_TEMP = New System.Windows.Forms.CheckBox()
-            Me.CH_FAV = New System.Windows.Forms.CheckBox()
             Me.CH_PARSE_USER_MEDIA = New System.Windows.Forms.CheckBox()
             Me.CH_READY_FOR_DOWN = New System.Windows.Forms.CheckBox()
+            Me.BTT_OTHER_SETTINGS = New System.Windows.Forms.Button()
+            Me.TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
+            Me.TXT_USER = New PersonalUtilities.Forms.Controls.TextBoxExtended()
+            Me.TP_SITE = New System.Windows.Forms.TableLayoutPanel()
+            Me.CH_IS_CHANNEL = New System.Windows.Forms.CheckBox()
+            Me.CMB_SITE = New PersonalUtilities.Forms.Controls.ComboBoxExtended()
+            Me.TP_TEMP_FAV = New System.Windows.Forms.TableLayoutPanel()
+            Me.CH_TEMP = New System.Windows.Forms.CheckBox()
+            Me.CH_FAV = New System.Windows.Forms.CheckBox()
+            Me.TP_READY_USERMEDIA = New System.Windows.Forms.TableLayoutPanel()
             Me.TXT_DESCR = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TXT_USER_FRIENDLY = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TP_ADD_BY_LIST = New System.Windows.Forms.TableLayoutPanel()
             Me.CH_ADD_BY_LIST = New System.Windows.Forms.CheckBox()
             Me.CH_AUTO_DETECT_SITE = New System.Windows.Forms.CheckBox()
             Me.TXT_LABELS = New PersonalUtilities.Forms.Controls.TextBoxExtended()
+            Me.TP_DOWN_IMG_VID = New System.Windows.Forms.TableLayoutPanel()
             Me.CH_DOWN_IMAGES = New System.Windows.Forms.CheckBox()
             Me.CH_DOWN_VIDEOS = New System.Windows.Forms.CheckBox()
             Me.TXT_SPEC_FOLDER = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TXT_SCRIPT = New PersonalUtilities.Forms.Controls.TextBoxExtended()
-            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
-            TP_SITE = New System.Windows.Forms.TableLayoutPanel()
-            TP_PARAMS = New System.Windows.Forms.TableLayoutPanel()
-            TP_OTHER = New System.Windows.Forms.TableLayoutPanel()
-            TP_DOWN_OPTIONS = New System.Windows.Forms.TableLayoutPanel()
             TT_MAIN = New System.Windows.Forms.ToolTip(Me.components)
             CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            TP_MAIN.SuspendLayout()
+            CONTAINER_MAIN.ContentPanel.SuspendLayout()
+            CONTAINER_MAIN.SuspendLayout()
+            Me.TP_MAIN.SuspendLayout()
             CType(Me.TXT_USER, System.ComponentModel.ISupportInitialize).BeginInit()
-            TP_SITE.SuspendLayout()
+            Me.TP_SITE.SuspendLayout()
             CType(Me.CMB_SITE, System.ComponentModel.ISupportInitialize).BeginInit()
-            TP_PARAMS.SuspendLayout()
+            Me.TP_TEMP_FAV.SuspendLayout()
-            TP_OTHER.SuspendLayout()
+            Me.TP_READY_USERMEDIA.SuspendLayout()
             CType(Me.TXT_DESCR, System.ComponentModel.ISupportInitialize).BeginInit()
             CType(Me.TXT_USER_FRIENDLY, System.ComponentModel.ISupportInitialize).BeginInit()
             Me.TP_ADD_BY_LIST.SuspendLayout()
             CType(Me.TXT_LABELS, System.ComponentModel.ISupportInitialize).BeginInit()
-            TP_DOWN_OPTIONS.SuspendLayout()
+            Me.TP_DOWN_IMG_VID.SuspendLayout()
             CType(Me.TXT_SPEC_FOLDER, System.ComponentModel.ISupportInitialize).BeginInit()
             CType(Me.TXT_SCRIPT, System.ComponentModel.ISupportInitialize).BeginInit()
-            CONTAINER_MAIN.ContentPanel.SuspendLayout()
-            CONTAINER_MAIN.SuspendLayout()
             Me.SuspendLayout()
             '
+            'CH_PARSE_USER_MEDIA
+            '
+            Me.CH_PARSE_USER_MEDIA.AutoSize = True
+            Me.CH_PARSE_USER_MEDIA.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_PARSE_USER_MEDIA.Location = New System.Drawing.Point(229, 4)
+            Me.CH_PARSE_USER_MEDIA.Name = "CH_PARSE_USER_MEDIA"
+            Me.CH_PARSE_USER_MEDIA.Size = New System.Drawing.Size(219, 20)
+            Me.CH_PARSE_USER_MEDIA.TabIndex = 0
+            Me.CH_PARSE_USER_MEDIA.Text = "Get user media only"
+            TT_MAIN.SetToolTip(Me.CH_PARSE_USER_MEDIA, "For twitter only!" & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "If checked then user media only will be downloaded." & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "Otherwise" &
+        " all media (include comments and retwits) will be downloaded.")
+            Me.CH_PARSE_USER_MEDIA.UseVisualStyleBackColor = True
+            '
+            'CH_READY_FOR_DOWN
+            '
+            Me.CH_READY_FOR_DOWN.AutoSize = True
+            Me.CH_READY_FOR_DOWN.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_READY_FOR_DOWN.Location = New System.Drawing.Point(4, 4)
+            Me.CH_READY_FOR_DOWN.Name = "CH_READY_FOR_DOWN"
+            Me.CH_READY_FOR_DOWN.Size = New System.Drawing.Size(218, 20)
+            Me.CH_READY_FOR_DOWN.TabIndex = 1
+            Me.CH_READY_FOR_DOWN.Text = "Ready for download"
+            TT_MAIN.SetToolTip(Me.CH_READY_FOR_DOWN, "Can be downloaded by [Download All]")
+            Me.CH_READY_FOR_DOWN.UseVisualStyleBackColor = True
+            '
+            'BTT_OTHER_SETTINGS
+            '
+            Me.BTT_OTHER_SETTINGS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.BTT_OTHER_SETTINGS.Location = New System.Drawing.Point(329, 2)
+            Me.BTT_OTHER_SETTINGS.Margin = New System.Windows.Forms.Padding(1)
+            Me.BTT_OTHER_SETTINGS.Name = "BTT_OTHER_SETTINGS"
+            Me.BTT_OTHER_SETTINGS.Size = New System.Drawing.Size(121, 24)
+            Me.BTT_OTHER_SETTINGS.TabIndex = 2
+            Me.BTT_OTHER_SETTINGS.Text = "Options (F2)"
+            TT_MAIN.SetToolTip(Me.BTT_OTHER_SETTINGS, "Other settings")
+            Me.BTT_OTHER_SETTINGS.UseVisualStyleBackColor = True
+            '
+            'CONTAINER_MAIN
+            '
+            '
+            'CONTAINER_MAIN.ContentPanel
+            '
+            CONTAINER_MAIN.ContentPanel.Controls.Add(Me.TP_MAIN)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(454, 461)
+            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            CONTAINER_MAIN.LeftToolStripPanelVisible = False
+            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
+            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
+            CONTAINER_MAIN.RightToolStripPanelVisible = False
+            CONTAINER_MAIN.Size = New System.Drawing.Size(454, 461)
+            CONTAINER_MAIN.TabIndex = 0
+            CONTAINER_MAIN.TopToolStripPanelVisible = False
+            '
             'TP_MAIN
             '
-            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_MAIN.ColumnCount = 1
+            Me.TP_MAIN.ColumnCount = 1
-            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Controls.Add(Me.TXT_USER, 0, 0)
+            Me.TP_MAIN.Controls.Add(Me.TXT_USER, 0, 0)
-            TP_MAIN.Controls.Add(TP_SITE, 0, 3)
+            Me.TP_MAIN.Controls.Add(Me.TP_SITE, 0, 3)
-            TP_MAIN.Controls.Add(TP_PARAMS, 0, 4)
+            Me.TP_MAIN.Controls.Add(Me.TP_TEMP_FAV, 0, 4)
-            TP_MAIN.Controls.Add(TP_OTHER, 0, 6)
+            Me.TP_MAIN.Controls.Add(Me.TP_READY_USERMEDIA, 0, 6)
-            TP_MAIN.Controls.Add(Me.TXT_DESCR, 0, 10)
+            Me.TP_MAIN.Controls.Add(Me.TXT_DESCR, 0, 10)
-            TP_MAIN.Controls.Add(Me.TXT_USER_FRIENDLY, 0, 1)
+            Me.TP_MAIN.Controls.Add(Me.TXT_USER_FRIENDLY, 0, 1)
-            TP_MAIN.Controls.Add(Me.TP_ADD_BY_LIST, 0, 7)
+            Me.TP_MAIN.Controls.Add(Me.TP_ADD_BY_LIST, 0, 7)
-            TP_MAIN.Controls.Add(Me.TXT_LABELS, 0, 8)
+            Me.TP_MAIN.Controls.Add(Me.TXT_LABELS, 0, 8)
-            TP_MAIN.Controls.Add(TP_DOWN_OPTIONS, 0, 5)
+            Me.TP_MAIN.Controls.Add(Me.TP_DOWN_IMG_VID, 0, 5)
-            TP_MAIN.Controls.Add(Me.TXT_SPEC_FOLDER, 0, 2)
+            Me.TP_MAIN.Controls.Add(Me.TXT_SPEC_FOLDER, 0, 2)
-            TP_MAIN.Controls.Add(Me.TXT_SCRIPT, 0, 9)
+            Me.TP_MAIN.Controls.Add(Me.TXT_SCRIPT, 0, 9)
-            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_MAIN.Location = New System.Drawing.Point(0, 0)
+            Me.TP_MAIN.Location = New System.Drawing.Point(0, 0)
-            TP_MAIN.Name = "TP_MAIN"
+            Me.TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 11
+            Me.TP_MAIN.RowCount = 11
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 26.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 26.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 26.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 26.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(454, 461)
+            Me.TP_MAIN.Size = New System.Drawing.Size(454, 461)
-            TP_MAIN.TabIndex = 0
+            Me.TP_MAIN.TabIndex = 0
             '
             'TXT_USER
             '
@@ -130,22 +178,22 @@ Namespace Editors
             '
             'TP_SITE
             '
-            TP_SITE.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_SITE.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_SITE.ColumnCount = 3
+            Me.TP_SITE.ColumnCount = 3
-            TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 79.0!))
+            Me.TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 79.0!))
-            TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 104.0!))
+            Me.TP_SITE.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 123.0!))
-            TP_SITE.Controls.Add(Me.CH_IS_CHANNEL, 0, 0)
+            Me.TP_SITE.Controls.Add(Me.CH_IS_CHANNEL, 0, 0)
-            TP_SITE.Controls.Add(Me.CMB_SITE, 1, 0)
+            Me.TP_SITE.Controls.Add(Me.CMB_SITE, 1, 0)
-            TP_SITE.Controls.Add(Me.BTT_OTHER_SETTINGS, 2, 0)
+            Me.TP_SITE.Controls.Add(Me.BTT_OTHER_SETTINGS, 2, 0)
-            TP_SITE.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_SITE.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_SITE.Location = New System.Drawing.Point(1, 88)
+            Me.TP_SITE.Location = New System.Drawing.Point(1, 88)
-            TP_SITE.Margin = New System.Windows.Forms.Padding(0)
+            Me.TP_SITE.Margin = New System.Windows.Forms.Padding(0)
-            TP_SITE.Name = "TP_SITE"
+            Me.TP_SITE.Name = "TP_SITE"
-            TP_SITE.RowCount = 1
+            Me.TP_SITE.RowCount = 1
-            TP_SITE.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_SITE.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_SITE.Size = New System.Drawing.Size(452, 28)
+            Me.TP_SITE.Size = New System.Drawing.Size(452, 28)
-            TP_SITE.TabIndex = 3
+            Me.TP_SITE.TabIndex = 3
             '
             'CH_IS_CHANNEL
             '
@@ -177,39 +225,27 @@ Namespace Editors
             Me.CMB_SITE.Location = New System.Drawing.Point(84, 3)
             Me.CMB_SITE.Margin = New System.Windows.Forms.Padding(3, 2, 3, 3)
             Me.CMB_SITE.Name = "CMB_SITE"
-            Me.CMB_SITE.Size = New System.Drawing.Size(259, 22)
+            Me.CMB_SITE.Size = New System.Drawing.Size(240, 22)
             Me.CMB_SITE.TabIndex = 1
             Me.CMB_SITE.TextBoxBorderStyle = System.Windows.Forms.BorderStyle.FixedSingle
             '
-            'BTT_OTHER_SETTINGS
+            'TP_TEMP_FAV
             '
-            Me.BTT_OTHER_SETTINGS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_TEMP_FAV.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            Me.BTT_OTHER_SETTINGS.Location = New System.Drawing.Point(348, 2)
+            Me.TP_TEMP_FAV.ColumnCount = 2
-            Me.BTT_OTHER_SETTINGS.Margin = New System.Windows.Forms.Padding(1)
+            Me.TP_TEMP_FAV.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            Me.BTT_OTHER_SETTINGS.Name = "BTT_OTHER_SETTINGS"
+            Me.TP_TEMP_FAV.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            Me.BTT_OTHER_SETTINGS.Size = New System.Drawing.Size(102, 24)
+            Me.TP_TEMP_FAV.Controls.Add(Me.CH_TEMP, 0, 0)
-            Me.BTT_OTHER_SETTINGS.TabIndex = 2
+            Me.TP_TEMP_FAV.Controls.Add(Me.CH_FAV, 1, 0)
-            Me.BTT_OTHER_SETTINGS.Text = "Options (F2)"
+            Me.TP_TEMP_FAV.Dock = System.Windows.Forms.DockStyle.Fill
-            TT_MAIN.SetToolTip(Me.BTT_OTHER_SETTINGS, "Other settings")
+            Me.TP_TEMP_FAV.Location = New System.Drawing.Point(1, 117)
-            Me.BTT_OTHER_SETTINGS.UseVisualStyleBackColor = True
+            Me.TP_TEMP_FAV.Margin = New System.Windows.Forms.Padding(0)
-            '
+            Me.TP_TEMP_FAV.Name = "TP_TEMP_FAV"
-            'TP_PARAMS
+            Me.TP_TEMP_FAV.RowCount = 1
-            '
+            Me.TP_TEMP_FAV.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_PARAMS.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_TEMP_FAV.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
-            TP_PARAMS.ColumnCount = 2
+            Me.TP_TEMP_FAV.Size = New System.Drawing.Size(452, 28)
-            TP_PARAMS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
+            Me.TP_TEMP_FAV.TabIndex = 4
-            TP_PARAMS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_PARAMS.Controls.Add(Me.CH_TEMP, 0, 0)
-            TP_PARAMS.Controls.Add(Me.CH_FAV, 1, 0)
-            TP_PARAMS.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_PARAMS.Location = New System.Drawing.Point(1, 117)
-            TP_PARAMS.Margin = New System.Windows.Forms.Padding(0)
-            TP_PARAMS.Name = "TP_PARAMS"
-            TP_PARAMS.RowCount = 1
-            TP_PARAMS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_PARAMS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
-            TP_PARAMS.Size = New System.Drawing.Size(452, 28)
-            TP_PARAMS.TabIndex = 4
             '
             'CH_TEMP
             '
@@ -233,48 +269,23 @@ Namespace Editors
             Me.CH_FAV.Text = "Favorite"
             Me.CH_FAV.UseVisualStyleBackColor = True
             '
-            'TP_OTHER
+            'TP_READY_USERMEDIA
             '
-            TP_OTHER.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_READY_USERMEDIA.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_OTHER.ColumnCount = 2
+            Me.TP_READY_USERMEDIA.ColumnCount = 2
-            TP_OTHER.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
+            Me.TP_READY_USERMEDIA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_OTHER.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
+            Me.TP_READY_USERMEDIA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_OTHER.Controls.Add(Me.CH_PARSE_USER_MEDIA, 1, 0)
+            Me.TP_READY_USERMEDIA.Controls.Add(Me.CH_PARSE_USER_MEDIA, 1, 0)
-            TP_OTHER.Controls.Add(Me.CH_READY_FOR_DOWN, 0, 0)
+            Me.TP_READY_USERMEDIA.Controls.Add(Me.CH_READY_FOR_DOWN, 0, 0)
-            TP_OTHER.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_READY_USERMEDIA.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_OTHER.Location = New System.Drawing.Point(1, 175)
+            Me.TP_READY_USERMEDIA.Location = New System.Drawing.Point(1, 175)
-            TP_OTHER.Margin = New System.Windows.Forms.Padding(0)
+            Me.TP_READY_USERMEDIA.Margin = New System.Windows.Forms.Padding(0)
-            TP_OTHER.Name = "TP_OTHER"
+            Me.TP_READY_USERMEDIA.Name = "TP_READY_USERMEDIA"
-            TP_OTHER.RowCount = 1
+            Me.TP_READY_USERMEDIA.RowCount = 1
-            TP_OTHER.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_READY_USERMEDIA.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_OTHER.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
+            Me.TP_READY_USERMEDIA.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
-            TP_OTHER.Size = New System.Drawing.Size(452, 28)
+            Me.TP_READY_USERMEDIA.Size = New System.Drawing.Size(452, 28)
-            TP_OTHER.TabIndex = 6
+            Me.TP_READY_USERMEDIA.TabIndex = 6
-            '
-            'CH_PARSE_USER_MEDIA
-            '
-            Me.CH_PARSE_USER_MEDIA.AutoSize = True
-            Me.CH_PARSE_USER_MEDIA.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_PARSE_USER_MEDIA.Location = New System.Drawing.Point(229, 4)
-            Me.CH_PARSE_USER_MEDIA.Name = "CH_PARSE_USER_MEDIA"
-            Me.CH_PARSE_USER_MEDIA.Size = New System.Drawing.Size(219, 20)
-            Me.CH_PARSE_USER_MEDIA.TabIndex = 0
-            Me.CH_PARSE_USER_MEDIA.Text = "Get user media only"
-            TT_MAIN.SetToolTip(Me.CH_PARSE_USER_MEDIA, "For twitter only!" & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "If checked then user media only will be downloaded." & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "Otherwise" &
-        " all media (include comments and retwits) will be downloaded.")
-            Me.CH_PARSE_USER_MEDIA.UseVisualStyleBackColor = True
-            '
-            'CH_READY_FOR_DOWN
-            '
-            Me.CH_READY_FOR_DOWN.AutoSize = True
-            Me.CH_READY_FOR_DOWN.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_READY_FOR_DOWN.Location = New System.Drawing.Point(4, 4)
-            Me.CH_READY_FOR_DOWN.Name = "CH_READY_FOR_DOWN"
-            Me.CH_READY_FOR_DOWN.Size = New System.Drawing.Size(218, 20)
-            Me.CH_READY_FOR_DOWN.TabIndex = 1
-            Me.CH_READY_FOR_DOWN.Text = "Ready for download"
-            TT_MAIN.SetToolTip(Me.CH_READY_FOR_DOWN, "Can be downloaded by [Download All]")
-            Me.CH_READY_FOR_DOWN.UseVisualStyleBackColor = True
             '
             'TXT_DESCR
             '
@@ -361,23 +372,23 @@ Namespace Editors
             Me.TXT_LABELS.TabIndex = 8
             Me.TXT_LABELS.TextBoxReadOnly = True
             '
-            'TP_DOWN_OPTIONS
+            'TP_DOWN_IMG_VID
             '
-            TP_DOWN_OPTIONS.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_DOWN_IMG_VID.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_DOWN_OPTIONS.ColumnCount = 2
+            Me.TP_DOWN_IMG_VID.ColumnCount = 2
-            TP_DOWN_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
+            Me.TP_DOWN_IMG_VID.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_DOWN_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
+            Me.TP_DOWN_IMG_VID.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_DOWN_OPTIONS.Controls.Add(Me.CH_DOWN_IMAGES, 0, 0)
+            Me.TP_DOWN_IMG_VID.Controls.Add(Me.CH_DOWN_IMAGES, 0, 0)
-            TP_DOWN_OPTIONS.Controls.Add(Me.CH_DOWN_VIDEOS, 1, 0)
+            Me.TP_DOWN_IMG_VID.Controls.Add(Me.CH_DOWN_VIDEOS, 1, 0)
-            TP_DOWN_OPTIONS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_DOWN_IMG_VID.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_DOWN_OPTIONS.Location = New System.Drawing.Point(1, 146)
+            Me.TP_DOWN_IMG_VID.Location = New System.Drawing.Point(1, 146)
-            TP_DOWN_OPTIONS.Margin = New System.Windows.Forms.Padding(0)
+            Me.TP_DOWN_IMG_VID.Margin = New System.Windows.Forms.Padding(0)
-            TP_DOWN_OPTIONS.Name = "TP_DOWN_OPTIONS"
+            Me.TP_DOWN_IMG_VID.Name = "TP_DOWN_IMG_VID"
-            TP_DOWN_OPTIONS.RowCount = 1
+            Me.TP_DOWN_IMG_VID.RowCount = 1
-            TP_DOWN_OPTIONS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_DOWN_IMG_VID.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_DOWN_OPTIONS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
+            Me.TP_DOWN_IMG_VID.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 27.0!))
-            TP_DOWN_OPTIONS.Size = New System.Drawing.Size(452, 28)
+            Me.TP_DOWN_IMG_VID.Size = New System.Drawing.Size(452, 28)
-            TP_DOWN_OPTIONS.TabIndex = 5
+            Me.TP_DOWN_IMG_VID.TabIndex = 5
             '
             'CH_DOWN_IMAGES
             '
@@ -442,22 +453,6 @@ Namespace Editors
             Me.TXT_SCRIPT.Size = New System.Drawing.Size(446, 22)
             Me.TXT_SCRIPT.TabIndex = 9
             '
-            'CONTAINER_MAIN
-            '
-            '
-            'CONTAINER_MAIN.ContentPanel
-            '
-            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(454, 461)
-            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            CONTAINER_MAIN.LeftToolStripPanelVisible = False
-            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
-            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
-            CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(454, 461)
-            CONTAINER_MAIN.TabIndex = 0
-            CONTAINER_MAIN.TopToolStripPanelVisible = False
-            '
             'UserCreatorForm
             '
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -474,28 +469,28 @@ Namespace Editors
             Me.Name = "UserCreatorForm"
             Me.ShowInTaskbar = False
             Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Create User"
+            Me.Text = "Create user"
-            TP_MAIN.ResumeLayout(False)
+            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
+            CONTAINER_MAIN.ResumeLayout(False)
+            CONTAINER_MAIN.PerformLayout()
+            Me.TP_MAIN.ResumeLayout(False)
             CType(Me.TXT_USER, System.ComponentModel.ISupportInitialize).EndInit()
-            TP_SITE.ResumeLayout(False)
+            Me.TP_SITE.ResumeLayout(False)
-            TP_SITE.PerformLayout()
+            Me.TP_SITE.PerformLayout()
             CType(Me.CMB_SITE, System.ComponentModel.ISupportInitialize).EndInit()
-            TP_PARAMS.ResumeLayout(False)
+            Me.TP_TEMP_FAV.ResumeLayout(False)
-            TP_PARAMS.PerformLayout()
+            Me.TP_TEMP_FAV.PerformLayout()
-            TP_OTHER.ResumeLayout(False)
+            Me.TP_READY_USERMEDIA.ResumeLayout(False)
-            TP_OTHER.PerformLayout()
+            Me.TP_READY_USERMEDIA.PerformLayout()
             CType(Me.TXT_DESCR, System.ComponentModel.ISupportInitialize).EndInit()
             CType(Me.TXT_USER_FRIENDLY, System.ComponentModel.ISupportInitialize).EndInit()
             Me.TP_ADD_BY_LIST.ResumeLayout(False)
             Me.TP_ADD_BY_LIST.PerformLayout()
             CType(Me.TXT_LABELS, System.ComponentModel.ISupportInitialize).EndInit()
-            TP_DOWN_OPTIONS.ResumeLayout(False)
+            Me.TP_DOWN_IMG_VID.ResumeLayout(False)
-            TP_DOWN_OPTIONS.PerformLayout()
+            Me.TP_DOWN_IMG_VID.PerformLayout()
             CType(Me.TXT_SPEC_FOLDER, System.ComponentModel.ISupportInitialize).EndInit()
             CType(Me.TXT_SCRIPT, System.ComponentModel.ISupportInitialize).EndInit()
-            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-            CONTAINER_MAIN.ResumeLayout(False)
-            CONTAINER_MAIN.PerformLayout()
             Me.ResumeLayout(False)
 
         End Sub
@@ -517,5 +512,10 @@ Namespace Editors
         Private WithEvents CMB_SITE As PersonalUtilities.Forms.Controls.ComboBoxExtended
         Private WithEvents BTT_OTHER_SETTINGS As Button
         Private WithEvents TXT_SCRIPT As PersonalUtilities.Forms.Controls.TextBoxExtended
+        Private WithEvents TP_SITE As TableLayoutPanel
+        Private WithEvents TP_MAIN As TableLayoutPanel
+        Private WithEvents TP_TEMP_FAV As TableLayoutPanel
+        Private WithEvents TP_READY_USERMEDIA As TableLayoutPanel
+        Private WithEvents TP_DOWN_IMG_VID As TableLayoutPanel
     End Class
 End Namespace

SCrawler/Editors/UserCreatorForm.resx

@@ -117,10 +117,13 @@
   <resheader name="writer">
     <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
   </resheader>
-  <metadata name="TP_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+  <metadata name="TT_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
-  <metadata name="TP_SITE.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+  <metadata name="TT_MAIN.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
+    <value>17, 17</value>
+  </metadata>
+  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
   <assembly alias="System.Drawing" name="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
@@ -214,18 +217,6 @@
         AAAASUVORK5CYII=
 </value>
   </data>
-  <metadata name="TT_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
-  <metadata name="TT_MAIN.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
-    <value>17, 17</value>
-  </metadata>
-  <metadata name="TP_PARAMS.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
-  <metadata name="TP_OTHER.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
   <data name="ActionButton2.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
@@ -253,9 +244,6 @@
         AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
 </value>
   </data>
-  <metadata name="TP_DOWN_OPTIONS.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
   <data name="ActionButton5.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
@@ -294,7 +282,4 @@
         AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
 </value>
   </data>
-  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
 </root>

SCrawler/Editors/UserCreatorForm.vb

@@ -10,6 +10,8 @@ Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Forms.Controls
 Imports PersonalUtilities.Forms.Controls.Base
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools
+Imports SCrawler.API
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Hosts
@@ -19,7 +21,10 @@ Namespace Editors
         Private WithEvents MyDef As DefaultFormOptions
         Friend Property User As UserInfo
         Private Property UserInstance As IUserData
+        Private ReadOnly UserIsCollection As Boolean = False
 #Region "User options"
+        ''' <summary>COLLECTION EDITING ONLY</summary>
+        Friend Property CollectionName As String = String.Empty
         Friend Property StartIndex As Integer = -1
         Friend ReadOnly Property UserTemporary As Boolean
             Get
@@ -107,10 +112,36 @@ Namespace Editors
             If Not _Instance Is Nothing Then
                 UserInstance = _Instance
                 User = DirectCast(UserInstance, UserDataBase).User
+                UserIsCollection = TypeOf UserInstance Is UserDataBind
+                If UserIsCollection Then
+                    With DirectCast(UserInstance, UserDataBind) : .CurrentlyEdited = True : CollectionName = .CollectionName : End With
+                End If
             End If
         End Sub
 #End Region
 #Region "Form handlers"
+        Private Class CollectionNameFieldProvider : Implements IFieldsCheckerProvider
+            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
+            Private Property Name As String Implements IFieldsCheckerProvider.Name
+            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
+            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+                If ACheck(Value) Then
+                    If Settings.Users.Exists(Function(u) u.IsCollection AndAlso u.CollectionName = CStr(Value) AndAlso
+                                                         Not DirectCast(u, UserDataBind).CurrentlyEdited) Then
+                        ErrorMessage = $"A collection named [{Value}] already exist"
+                        Return Nothing
+                    Else
+                        Return Value
+                    End If
+                Else
+                    Return Nothing
+                End If
+            End Function
+            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
+                Throw New NotImplementedException("[GetFormat] is not available in the 'CollectionNameFieldProvider'")
+            End Function
+        End Class
         Private Sub UserCreatorForm_Load(sender As Object, e As EventArgs) Handles Me.Load
             Try
                 With MyDef
@@ -122,52 +153,125 @@ Namespace Editors
                         .Items.AddRange(Settings.Plugins.Select(Function(p) New ListItem({p.Key, p.Name})))
                         .EndUpdate(True)
                     End With
-                    If User.Name.IsEmptyString Then
+
-                        CH_READY_FOR_DOWN.Checked = True
+                    Dim NameFieldProvider As IFormatProvider = Nothing
-                        CH_TEMP.Checked = Settings.DefaultTemporary
+
-                        CH_DOWN_IMAGES.Checked = Settings.DefaultDownloadImages
+                    If UserIsCollection Then
-                        CH_DOWN_VIDEOS.Checked = Settings.DefaultDownloadVideos
+                        Icon = If(ImageRenderer.GetIcon(My.Resources.DBPic_32, EDP.ReturnValue), Icon)
-                        TXT_SCRIPT.Checked = Settings.ScriptData.Attribute
+                        Text = $"Collection: {UserInstance.CollectionName}"
-                        SetParamsBySite()
+
-                    Else
+                        TXT_USER.CaptionText = "Collection name"
-                        TP_ADD_BY_LIST.Enabled = False
+                        TXT_USER.Text = UserInstance.CollectionName
-                        TXT_USER.Text = User.Name
+                        TXT_USER.Buttons.AddRange({ADB.Refresh, ADB.Clear})
-                        TXT_SPEC_FOLDER.Text = User.SpecialPath
+                        TXT_USER.Buttons.UpdateButtonsPositions()
-                        Dim i% = Settings.Plugins.FindIndex(Function(p) p.Key = User.Plugin)
+                        TXT_SPEC_FOLDER.Buttons.Clear()
-                        If i >= 0 Then CMB_SITE.SelectedIndex = i
+                        TXT_SPEC_FOLDER.TextBoxReadOnly = True
-                        SetParamsBySite()
+                        TXT_SPEC_FOLDER.Buttons.UpdateButtonsPositions()
-                        CH_IS_CHANNEL.Enabled = False
+
-                        CMB_SITE.Enabled = False
+                        With TP_MAIN
-                        CH_IS_CHANNEL.Checked = User.IsChannel
+                            .Controls.Clear()
-                        If Not UserInstance Is Nothing Then
+                            .RowStyles.Clear()
-                            TXT_USER.Enabled = False
+                            .RowCount = 0
-                            TXT_SPEC_FOLDER.TextBoxReadOnly = True
+                            With .RowStyles
-                            TXT_SPEC_FOLDER.Buttons.Clear()
+                                .Add(New RowStyle(SizeType.Absolute, 28))
-                            TXT_SPEC_FOLDER.Buttons.UpdateButtonsPositions()
+                                .Add(New RowStyle(SizeType.Absolute, 28))
-                            With UserInstance
+                                .Add(New RowStyle(SizeType.Absolute, 28))
-                                TXT_USER_FRIENDLY.Text = .FriendlyName
+                                .Add(New RowStyle(SizeType.Absolute, 28))
-                                CH_FAV.Checked = .Favorite
+                                .Add(New RowStyle(SizeType.Absolute, 28))
-                                CH_TEMP.Checked = .Temporary
+                                .Add(New RowStyle(SizeType.Absolute, 26))
-                                CH_PARSE_USER_MEDIA.Checked = .ParseUserMediaOnly
+                                .Add(New RowStyle(SizeType.Percent, 100))
-                                CH_READY_FOR_DOWN.Checked = .ReadyForDownload
-                                CH_DOWN_IMAGES.Checked = .DownloadImages
-                                CH_DOWN_VIDEOS.Checked = .DownloadVideos
-                                TXT_SCRIPT.Checked = .ScriptUse
-                                TXT_SCRIPT.Text = .ScriptData
-                                TXT_DESCR.Text = .Description.StringFormatLines
-                                UserLabels.ListAddList(.Labels)
-                                If UserLabels.ListExists Then TXT_LABELS.Text = UserLabels.ListToString
                             End With
-                            CH_ADD_BY_LIST.Enabled = False
+                            .RowCount = .RowStyles.Count
-                        Else
+                            With .Controls
+                                .Add(TXT_USER, 0, 0)
+                                .Add(TXT_SPEC_FOLDER, 0, 1)
+                                .Add(TP_TEMP_FAV, 0, 2)
+                                .Add(TP_DOWN_IMG_VID, 0, 3)
+                                .Add(TP_READY_USERMEDIA, 0, 4)
+                                .Add(TXT_LABELS, 0, 5)
+                                .Add(TXT_DESCR, 0, 6)
+                            End With
+                            .Refresh()
+                            .Update()
+                        End With
+
+                        TXT_DESCR.TextBoxReadOnly = True
+                        TXT_DESCR.Buttons.Clear()
+                        TXT_DESCR.Buttons.UpdateButtonsPositions()
+
+                        CH_TEMP.ThreeState = True
+                        CH_FAV.ThreeState = True
+                        CH_DOWN_IMAGES.ThreeState = True
+                        CH_DOWN_VIDEOS.ThreeState = True
+                        CH_READY_FOR_DOWN.ThreeState = True
+                        CH_PARSE_USER_MEDIA.ThreeState = True
+
+                        With DirectCast(UserInstance, UserDataBind)
+                            Dim state As Func(Of Boolean, Func(Of IUserData, Boolean, Boolean), CheckState) =
+                                Function(v, p) If(.All(Function(pp) p.Invoke(pp, v)), If(v, CheckState.Checked, CheckState.Unchecked), CheckState.Indeterminate)
+                            TXT_SPEC_FOLDER.Text = DirectCast(.Item(0), UserDataBase).User.SpecialCollectionPath.ToString
+                            CH_TEMP.CheckState = state(.Item(0).Temporary, Function(p, v) p.Temporary = v)
+                            CH_FAV.CheckState = state(.Item(0).Favorite, Function(p, v) p.Favorite = v)
+                            CH_DOWN_IMAGES.CheckState = state(.Item(0).DownloadImages, Function(p, v) p.DownloadImages = v)
+                            CH_DOWN_VIDEOS.CheckState = state(.Item(0).DownloadVideos, Function(p, v) p.DownloadVideos = v)
+                            CH_READY_FOR_DOWN.CheckState = state(.Item(0).ReadyForDownload, Function(p, v) p.ReadyForDownload = v)
+                            CH_PARSE_USER_MEDIA.CheckState = state(.Item(0).ParseUserMediaOnly, Function(p, v) p.ParseUserMediaOnly = v)
+                            TXT_DESCR.Text = .GetUserInformation.StringFormatLines
+                            UserLabels.ListAddList(.Labels)
+                            If UserLabels.ListExists Then TXT_LABELS.Text = UserLabels.ListToString
+                        End With
+
+                        NameFieldProvider = New CollectionNameFieldProvider
+                    Else
+                        If User.Name.IsEmptyString Then
+                            CH_READY_FOR_DOWN.Checked = True
                             CH_TEMP.Checked = Settings.DefaultTemporary
-                            CH_READY_FOR_DOWN.Checked = Not Settings.DefaultTemporary
                             CH_DOWN_IMAGES.Checked = Settings.DefaultDownloadImages
                             CH_DOWN_VIDEOS.Checked = Settings.DefaultDownloadVideos
+                            TXT_SCRIPT.Checked = Settings.ScriptData.Attribute
+                            SetParamsBySite()
+                        Else
+                            TP_ADD_BY_LIST.Enabled = False
+                            TXT_USER.Text = User.Name
+                            TXT_SPEC_FOLDER.Text = User.SpecialPath
+                            Dim i% = Settings.Plugins.FindIndex(Function(p) p.Key = User.Plugin)
+                            If i >= 0 Then CMB_SITE.SelectedIndex = i
+                            SetParamsBySite()
+                            CH_IS_CHANNEL.Enabled = False
+                            CMB_SITE.Enabled = False
+                            CH_IS_CHANNEL.Checked = User.IsChannel
+                            If Not UserInstance Is Nothing Then
+                                Text = $"User: {UserInstance.Name}"
+                                If Not UserInstance.FriendlyName.IsEmptyString Then Text &= $" ({UserInstance.FriendlyName})"
+                                TXT_USER.Enabled = False
+                                TXT_SPEC_FOLDER.TextBoxReadOnly = True
+                                TXT_SPEC_FOLDER.Buttons.Clear()
+                                TXT_SPEC_FOLDER.Buttons.UpdateButtonsPositions()
+                                With UserInstance
+                                    TXT_USER_FRIENDLY.Text = .FriendlyName
+                                    CH_FAV.Checked = .Favorite
+                                    CH_TEMP.Checked = .Temporary
+                                    CH_PARSE_USER_MEDIA.Checked = .ParseUserMediaOnly
+                                    CH_READY_FOR_DOWN.Checked = .ReadyForDownload
+                                    CH_DOWN_IMAGES.Checked = .DownloadImages
+                                    CH_DOWN_VIDEOS.Checked = .DownloadVideos
+                                    TXT_SCRIPT.Checked = .ScriptUse
+                                    TXT_SCRIPT.Text = .ScriptData
+                                    TXT_DESCR.Text = .Description.StringFormatLines
+                                    UserLabels.ListAddList(.Labels)
+                                    If UserLabels.ListExists Then TXT_LABELS.Text = UserLabels.ListToString
+                                End With
+                                CH_ADD_BY_LIST.Enabled = False
+                            Else
+                                CH_TEMP.Checked = Settings.DefaultTemporary
+                                CH_READY_FOR_DOWN.Checked = Not Settings.DefaultTemporary
+                                CH_DOWN_IMAGES.Checked = Settings.DefaultDownloadImages
+                                CH_DOWN_VIDEOS.Checked = Settings.DefaultDownloadVideos
+                            End If
                         End If
                     End If
                     .MyFieldsChecker = New FieldsChecker
-                    .MyFieldsCheckerE.AddControl(Of String)(TXT_USER, TXT_USER.CaptionText)
+                    .MyFieldsCheckerE.AddControl(Of String)(TXT_USER, TXT_USER.CaptionText,, NameFieldProvider)
                     .MyFieldsChecker.EndLoaderOperations()
                     .EndLoaderOperations()
                 End With
@@ -186,64 +290,82 @@ Namespace Editors
         End Sub
         Private Sub UserCreatorForm_Disposed(sender As Object, e As EventArgs) Handles Me.Disposed
             UserLabels.Clear()
+            If UserIsCollection And Not UserInstance Is Nothing Then DirectCast(UserInstance, UserDataBind).CurrentlyEdited = False
         End Sub
 #End Region
 #Region "Ok, Cancel"
         Private Sub MyDef_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDef.ButtonOkClick
-            If Not CH_ADD_BY_LIST.Checked Then
+            If UserIsCollection Then
                 If MyDef.MyFieldsChecker.AllParamsOK Then
-                    Dim s As SettingsHost = GetSiteByCheckers()
+                    With UserInstance
-                    If Not s Is Nothing Then
+                        If Not CH_TEMP.CheckState = CheckState.Indeterminate Then .Temporary = CH_TEMP.Checked
-                        Dim tmpUser As UserInfo = User.Clone
+                        If Not CH_FAV.CheckState = CheckState.Indeterminate Then .Favorite = CH_FAV.Checked
-                        With tmpUser
+                        If Not CH_DOWN_IMAGES.CheckState = CheckState.Indeterminate Then .DownloadImages = CH_DOWN_IMAGES.Checked
-                            .Name = TXT_USER.Text
+                        If Not CH_DOWN_VIDEOS.CheckState = CheckState.Indeterminate Then .DownloadVideos = CH_DOWN_VIDEOS.Checked
-                            .SpecialPath = SpecialPath(s)
+                        If Not CH_READY_FOR_DOWN.CheckState = CheckState.Indeterminate Then .ReadyForDownload = CH_READY_FOR_DOWN.Checked
-                            .Site = s.Name
+                        If Not CH_PARSE_USER_MEDIA.CheckState = CheckState.Indeterminate Then .ParseUserMediaOnly = CH_PARSE_USER_MEDIA.Checked
-                            .Plugin = s.Key
+                        DirectCast(UserInstance, UserDataBind).Collections.ForEach(Sub(u) u.Labels.ListAddList(UserLabels, LAP.ClearBeforeAdd, LAP.NotContainsOnly))
-                            .IsChannel = CH_IS_CHANNEL.Checked
+                        CollectionName = TXT_USER.Text
-                            .UpdateUserFile()
+                        .UpdateUserInformation()
-                        End With
+                    End With
-                        User = tmpUser
+                    GoTo CloseForm
-                        Dim ScriptText$ = TXT_SCRIPT.Text
-                        If Not ScriptText.IsEmptyString Then
-                            Dim f As SFile = ScriptText
-                            If Not SFile.IsDirectory(ScriptText) And Not UserInstance Is Nothing Then
-                                With DirectCast(UserInstance, UserDataBase) : f.Path = .MyFile.Path : End With
-                            End If
-                            TXT_SCRIPT.Text = f
-                        End If
-                        If Not UserInstance Is Nothing Then
-                            With DirectCast(UserInstance, UserDataBase)
-                                .User = User
-                                .FriendlyName = TXT_USER_FRIENDLY.Text
-                                .Favorite = CH_FAV.Checked
-                                .Temporary = CH_TEMP.Checked
-                                .ReadyForDownload = CH_READY_FOR_DOWN.Checked
-                                .DownloadImages = CH_DOWN_IMAGES.Checked
-                                .DownloadVideos = CH_DOWN_VIDEOS.Checked
-                                .UserDescription = TXT_DESCR.Text
-                                If Not MyExchangeOptions Is Nothing Then .ExchangeOptionsSet(MyExchangeOptions)
-                                Dim l As New ListAddParams(LAP.NotContainsOnly + LAP.ClearBeforeAdd)
-                                If .IsCollection Then
-                                    With DirectCast(UserInstance, API.UserDataBind)
-                                        If .Count > 0 Then .Collections.ForEach(Sub(c) c.Labels.ListAddList(UserLabels, l))
-                                    End With
-                                Else
-                                    .Labels.ListAddList(UserLabels, LAP.NotContainsOnly, LAP.ClearBeforeAdd)
-                                End If
-                                .ParseUserMediaOnly = CH_PARSE_USER_MEDIA.Checked
-                                .ScriptUse = TXT_SCRIPT.Checked
-                                .ScriptData = TXT_SCRIPT.Text
-                                .UpdateUserInformation()
-                            End With
-                        End If
-                        GoTo CloseForm
-                    Else
-                        MsgBoxE("User site not selected", MsgBoxStyle.Exclamation)
-                    End If
                 End If
             Else
-                If CreateUsersByList() Then GoTo CloseForm
+                If Not CH_ADD_BY_LIST.Checked Then
+                    If MyDef.MyFieldsChecker.AllParamsOK Then
+                        Dim s As SettingsHost = GetSiteByCheckers()
+                        If Not s Is Nothing Then
+                            Dim tmpUser As UserInfo = User.Clone
+                            With tmpUser
+                                .Name = TXT_USER.Text
+                                .SpecialPath = SpecialPath(s)
+                                .Site = s.Name
+                                .Plugin = s.Key
+                                .IsChannel = CH_IS_CHANNEL.Checked
+                                .UpdateUserFile()
+                            End With
+                            User = tmpUser
+                            Dim ScriptText$ = TXT_SCRIPT.Text
+                            If Not ScriptText.IsEmptyString Then
+                                Dim f As SFile = ScriptText
+                                If Not SFile.IsDirectory(ScriptText) And Not UserInstance Is Nothing Then
+                                    With DirectCast(UserInstance, UserDataBase) : f.Path = .MyFile.Path : End With
+                                End If
+                                TXT_SCRIPT.Text = f
+                            End If
+                            If Not UserInstance Is Nothing Then
+                                With DirectCast(UserInstance, UserDataBase)
+                                    .User = User
+                                    .FriendlyName = TXT_USER_FRIENDLY.Text
+                                    .Favorite = CH_FAV.Checked
+                                    .Temporary = CH_TEMP.Checked
+                                    .ReadyForDownload = CH_READY_FOR_DOWN.Checked
+                                    .DownloadImages = CH_DOWN_IMAGES.Checked
+                                    .DownloadVideos = CH_DOWN_VIDEOS.Checked
+                                    .UserDescription = TXT_DESCR.Text
+                                    If Not MyExchangeOptions Is Nothing Then .ExchangeOptionsSet(MyExchangeOptions)
+                                    Dim l As New ListAddParams(LAP.NotContainsOnly + LAP.ClearBeforeAdd)
+                                    If .IsCollection Then
+                                        With DirectCast(UserInstance, API.UserDataBind)
+                                            If .Count > 0 Then .Collections.ForEach(Sub(c) c.Labels.ListAddList(UserLabels, l))
+                                        End With
+                                    Else
+                                        .Labels.ListAddList(UserLabels, LAP.NotContainsOnly, LAP.ClearBeforeAdd)
+                                    End If
+                                    .ParseUserMediaOnly = CH_PARSE_USER_MEDIA.Checked
+                                    .ScriptUse = TXT_SCRIPT.Checked
+                                    .ScriptData = TXT_SCRIPT.Text
+                                    .UpdateUserInformation()
+                                End With
+                            End If
+                            GoTo CloseForm
+                        Else
+                            MsgBoxE("User site not selected", MsgBoxStyle.Exclamation)
+                        End If
+                    End If
+                Else
+                    If CreateUsersByList() Then GoTo CloseForm
+                End If
             End If
             Exit Sub
 CloseForm:
@@ -257,7 +379,7 @@ CloseForm:
         Private _TextChangeInvoked As Boolean = False
         Private Sub TXT_USER_ActionOnTextChanged(ByVal Sender As Object, ByVal e As EventArgs) Handles TXT_USER.ActionOnTextChanged
             Try
-                If Not _TextChangeInvoked Then
+                If Not _TextChangeInvoked And Not UserIsCollection Then
                     _TextChangeInvoked = True
                     If Not CH_ADD_BY_LIST.Checked Then
                         Dim s As ExchangeOptions = GetSiteByText(TXT_USER.Text)
@@ -282,7 +404,10 @@ CloseForm:
             Catch
             End Try
         End Sub
-        Private Sub CMB_SITE_ActionSelectedItemChanged(ByVal Item As ListViewItem) Handles CMB_SITE.ActionSelectedItemChanged
+        Private Sub TXT_USER_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As ActionButtonEventArgs) Handles TXT_USER.ActionOnButtonClick
+            If UserIsCollection AndAlso Sender.DefaultButton = ADB.Refresh Then TXT_USER.Text = UserInstance.CollectionName
+        End Sub
+        Private Sub CMB_SITE_ActionSelectedItemChanged(ByVal Sender As Object, ByVal e As EventArgs, ByVal Item As ListViewItem) Handles CMB_SITE.ActionSelectedItemChanged
             CH_IS_CHANNEL.Checked = False
             MyExchangeOptions = Nothing
             SetParamsBySite()
@@ -299,7 +424,7 @@ CloseForm:
             If Sender.DefaultButton = ADB.Open Then
                 Dim f As SFile = Nothing
                 If Not TXT_SPEC_FOLDER.Text.IsEmptyString Then f = $"{TXT_SPEC_FOLDER.Text}\"
-                f = SFile.SelectPath(f, True)
+                f = SFile.SelectPath(f)
                 If Not f.IsEmptyString Then TXT_SPEC_FOLDER.Text = f.PathWithSeparator
             End If
         End Sub
@@ -382,7 +507,8 @@ CloseForm:
                                 End If
 
                                 If Not s Is Nothing Then
-                                    tmpUser = New UserInfo(uu, s,,, __sf(uu, s)) With {.IsChannel = _IsChannel}
+                                    tmpUser = New UserInfo(uu, s) With {.SpecialPath = __sf(uu, s), .IsChannel = _IsChannel}
+                                    tmpUser.UpdateUserFile()
                                     uid = -1
                                     If Settings.UsersList.Count > 0 Then uid = Settings.UsersList.IndexOf(tmpUser)
                                     If uid < 0 And Not UsersForCreate.Contains(tmpUser) Then

SCrawler/EncryptCookies.vb

@@ -6,11 +6,11 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Namespace EncryptCookies
     Friend Module EncryptFunction
         Friend CookiesEncrypted As Boolean = False
-        Friend Sub ValidateCookiesEncrypt(ByRef Responser As Response)
+        Friend Sub ValidateCookiesEncrypt(ByRef Responser As Responser)
             If Not Responser Is Nothing Then
                 Dim b As Boolean = False
                 With Responser

SCrawler/ListImagesLoader.vb

@@ -6,11 +6,12 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Threading
 Imports SCrawler.API
 Imports SCrawler.API.Base
 Friend Class ListImagesLoader
     Private ReadOnly Property MyList As ListView
-    Private Class UserOption : Implements IComparable(Of UserOption)
+    Private Structure UserOption : Implements IComparable(Of UserOption)
         Friend ReadOnly User As IUserData
         Friend ReadOnly LVI As ListViewItem
         Friend Index As Integer
@@ -19,93 +20,106 @@ Friend Class ListImagesLoader
                 Return LVI.Name
             End Get
         End Property
-        Friend [Image] As Image
+        Friend Sub New(ByVal u As IUserData, ByVal l As ListView)
-        Friend Sub New(ByVal u As IUserData, ByVal l As ListView, ByVal GetImage As Boolean)
             User = u
             LVI = u.GetLVI(l)
             Index = u.Index
-            If GetImage Then Image = u.GetPicture
-        End Sub
-        Friend Sub UpdateImage()
-            Image = User.GetPicture
         End Sub
         Friend Function CompareTo(ByVal Other As UserOption) As Integer Implements IComparable(Of UserOption).CompareTo
             Return Index.CompareTo(Other.Index)
         End Function
-    End Class
+    End Structure
     Friend Sub New(ByRef l As ListView)
         MyList = l
     End Sub
+    Private UserDataList As List(Of UserOption)
+    Private UpdateInProgress As Boolean = False
+    Private ImageThread As Thread
+    Private Sub UpdateImages()
+        If UserDataList.ListExists And Not If(ImageThread?.IsAlive, False) Then
+            ImageThread = New Thread(New ThreadStart(Sub()
+                                                         Dim ar As IAsyncResult = Nothing
+                                                         Dim a As Action = Sub()
+                                                                               If UserDataList.ListExists Then
+                                                                                   For i% = 0 To UserDataList.Count - 1
+                                                                                       With UserDataList(i).User
+                                                                                           Select Case Settings.ViewMode.Value
+                                                                                               Case View.LargeIcon : MyList.LargeImageList.Images.Add(.Key, .GetPicture())
+                                                                                               Case View.SmallIcon : MyList.SmallImageList.Images.Add(.Key, .GetPicture())
+                                                                                           End Select
+                                                                                       End With
+                                                                                       Application.DoEvents()
+                                                                                   Next
+                                                                                   UserDataList.Clear()
+                                                                                   GC.Collect()
+                                                                               End If
+                                                                               If Not ar Is Nothing Then MyList.EndInvoke(ar)
+                                                                           End Sub
+                                                         If MyList.InvokeRequired Then
+                                                             ar = MyList.BeginInvoke(a)
+                                                         Else
+                                                             a.Invoke
+                                                         End If
+                                                     End Sub)) With {.IsBackground = True}
+            ImageThread.SetApartmentState(ApartmentState.MTA)
+            ImageThread.Start()
+        End If
+    End Sub
     Friend Sub Update()
-        Dim a As Action = Sub()
+        If Not UpdateInProgress Then
-                              With MyList
+            UpdateInProgress = True
-                                  .Items.Clear()
+            Dim a As Action = Sub()
-                                  If Not .LargeImageList Is Nothing Then .LargeImageList.Images.Clear()
+                                  With MyList
-                                  .LargeImageList = New ImageList
+                                      .Items.Clear()
-                                  If Not .SmallImageList Is Nothing Then .SmallImageList.Images.Clear()
+                                      If Not .LargeImageList Is Nothing Then .LargeImageList.Images.Clear()
-                                  .SmallImageList = New ImageList
+                                      .LargeImageList = New ImageList
-                                  If Settings.ViewModeIsPicture Then
+                                      If Not .SmallImageList Is Nothing Then .SmallImageList.Images.Clear()
-                                      .LargeImageList.ColorDepth = ColorDepth.Depth32Bit
+                                      .SmallImageList = New ImageList
-                                      .SmallImageList.ColorDepth = ColorDepth.Depth32Bit
+                                      If Settings.ViewModeIsPicture Then
-                                      .LargeImageList.ImageSize = New Size(DivideWithZeroChecking(Settings.MaxLargeImageHeight.Value, 100) * 75, Settings.MaxLargeImageHeight.Value)
+                                          .LargeImageList.ColorDepth = ColorDepth.Depth32Bit
-                                      .SmallImageList.ImageSize = New Size(DivideWithZeroChecking(Settings.MaxSmallImageHeight.Value, 100) * 75, Settings.MaxSmallImageHeight.Value)
+                                          .SmallImageList.ColorDepth = ColorDepth.Depth32Bit
-                                  End If
+                                          .LargeImageList.ImageSize = New Size(DivideWithZeroChecking(Settings.MaxLargeImageHeight.Value, 100) * 75, Settings.MaxLargeImageHeight.Value)
-                              End With
+                                          .SmallImageList.ImageSize = New Size(DivideWithZeroChecking(Settings.MaxSmallImageHeight.Value, 100) * 75, Settings.MaxSmallImageHeight.Value)
-                          End Sub
+                                      End If
-        If MyList.InvokeRequired Then MyList.Invoke(a) Else a.Invoke
+                                  End With
-        If Settings.Users.Count > 0 Then
+                              End Sub
-            Settings.Users.Sort()
+            If MyList.InvokeRequired Then MyList.Invoke(a) Else a.Invoke
-            Dim v As View = Settings.ViewMode.Value
+            If Settings.Users.Count > 0 Then
-            Dim i%
+                Settings.Users.Sort()
+                Dim v As View = Settings.ViewMode.Value
 
-            With MyList
+                With MyList
-                MyList.BeginUpdate()
+                    MyList.BeginUpdate()
 
-                If Settings.FastProfilesLoading Then
+                    If Settings.FastProfilesLoading Then
-                    Settings.Users.ListReindex
+                        Settings.Users.ListReindex
-                    Dim UData As List(Of UserOption)
 
-                    If Settings.ViewModeIsPicture Then
+                        UserDataList = (From u As IUserData In Settings.Users Where u.FitToAddParams Select New UserOption(u, MyList)).ListIfNothing
-                        UData = GetUsersWithImages()
+                        If UserDataList.ListExists Then UserDataList.Sort()
-                        If UData.ListExists Then
+
-                            UData.Sort()
+                        If UserDataList.ListExists Then
-                            Select Case v
+                            .Items.AddRange(UserDataList.Select(Function(u) u.LVI).ToArray)
-                                Case View.LargeIcon : .LargeImageList.Images.AddRange(UData.Select(Function(u) u.Image).ToArray)
+                            If Settings.ViewModeIsPicture Then MyList.EndUpdate() : UpdateImages() Else UserDataList.Clear()
-                                Case View.SmallIcon : .SmallImageList.Images.AddRange(UData.Select(Function(u) u.Image).ToArray)
-                            End Select
                         End If
                     Else
-                        UData = (From u As IUserData In Settings.Users Where u.FitToAddParams Select New UserOption(u, MyList, False)).ListIfNothing
+                        Dim t As New List(Of Task)
-                        If UData.ListExists Then UData.Sort()
+                        For Each User As IUserData In Settings.Users
-                    End If
+                            If User.FitToAddParams Then
-
+                                If Settings.ViewModeIsPicture Then
-                    If UData.ListExists Then
+                                    t.Add(Task.Run(Sub() UpdateUser(User, True)))
-                        If Settings.ViewModeIsPicture Then
+                                Else
-                            For i = 0 To UData.Count - 1
+                                    UpdateUser(User, True)
-                                Select Case v
+                                End If
-                                    Case View.LargeIcon : .LargeImageList.Images.SetKeyName(i, UData(i).Key)
-                                    Case View.SmallIcon : .SmallImageList.Images.SetKeyName(i, UData(i).Key)
-                                End Select
-                            Next
-                        End If
-                        .Items.AddRange(UData.Select(Function(u) u.LVI).ToArray)
-                        UData.Clear()
-                    End If
-                Else
-                    Dim t As New List(Of Task)
-                    For Each User As IUserData In Settings.Users
-                        If User.FitToAddParams Then
-                            If Settings.ViewModeIsPicture Then
-                                t.Add(Task.Run(Sub() UpdateUser(User, True)))
-                            Else
-                                UpdateUser(User, True)
                             End If
-                        End If
+                        Next
-                    Next
+                        If t.Count > 0 Then Task.WhenAll(t.ToArray) : t.Clear()
-                    If t.Count > 0 Then Task.WhenAll(t.ToArray) : t.Clear()
+                    End If
-                End If
+                End With
-            End With
+                MyList.EndUpdate()
-            MyList.EndUpdate()
+            End If
+            UpdateInProgress = False
+        Else
+            MsgBoxE({"The user list is currently being updated. Please wait for the update operation to complete and try again.", "Update user list"}, vbExclamation)
         End If
     End Sub
     Friend Sub UpdateUser(ByVal User As IUserData, ByVal Add As Boolean)
@@ -172,24 +186,4 @@ Friend Class ListImagesLoader
             Return False
         End If
     End Function
-    Private Function GetUsersWithImages() As List(Of UserOption)
-        Try
-            Dim t As New List(Of Task)
-            Dim l As New List(Of UserOption)
-            t.AddRange(From u As IUserData In Settings.Users Where u.FitToAddParams Select Task.Run(Sub() l.Add(New UserOption(u, MyList, True))))
-            If t.Count > 0 Then Task.WaitAll(t.ToArray)
-            If l.Count > 0 Then
-                For i% = 0 To l.Count - 1
-                    If l(i) Is Nothing Then Throw New ArgumentNullException("UserOption", $"One of the UserOptions [{i} / {l.Count - 1}] is null.")
-                    If l(i).Image Is Nothing Then l(i).UpdateImage()
-                Next
-            End If
-            Return l
-        Catch ex As Exception
-            Return ErrorsDescriber.Execute(EDP.LogMessageValue, ex,
-                                           "Image fast loading error." & vbCr &
-                                           "Click the ""Refresh"" button to manually refresh the user list." & vbCr &
-                                           "[ListImagesLoader.GetUsersWithImages]")
-        End Try
-    End Function
 End Class

SCrawler/MainFrame.Designer.vb

@@ -124,6 +124,7 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         Me.BTT_TRAY_SHOW_HIDE = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_CLOSE = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_CLOSE_NO_SCRIPT = New System.Windows.Forms.ToolStripMenuItem()
+        Me.BTT_TRAY_CHANNELS = New System.Windows.Forms.ToolStripMenuItem()
         SEP_1 = New System.Windows.Forms.ToolStripSeparator()
         SEP_2 = New System.Windows.Forms.ToolStripSeparator()
         CONTEXT_SEP_1 = New System.Windows.Forms.ToolStripSeparator()
@@ -754,6 +755,7 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         '
         'BTT_CONTEXT_GROUPS
         '
+        Me.BTT_CONTEXT_GROUPS.Image = Global.SCrawler.My.Resources.Resources.TagPic_24
         Me.BTT_CONTEXT_GROUPS.Name = "BTT_CONTEXT_GROUPS"
         Me.BTT_CONTEXT_GROUPS.Size = New System.Drawing.Size(221, 22)
         Me.BTT_CONTEXT_GROUPS.Text = "Change labels"
@@ -824,9 +826,9 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         '
         'TRAY_CONTEXT
         '
-        Me.TRAY_CONTEXT.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_TRAY_PAUSE_AUTOMATION, Me.BTT_TRAY_SILENT_MODE, Me.BTT_TRAY_FEED_SHOW, TRAY_SEP_1, Me.BTT_TRAY_SHOW_HIDE, TRAY_SEP_2, Me.BTT_TRAY_CLOSE, Me.BTT_TRAY_CLOSE_NO_SCRIPT})
+        Me.TRAY_CONTEXT.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_TRAY_PAUSE_AUTOMATION, Me.BTT_TRAY_SILENT_MODE, Me.BTT_TRAY_FEED_SHOW, Me.BTT_TRAY_CHANNELS, TRAY_SEP_1, Me.BTT_TRAY_SHOW_HIDE, TRAY_SEP_2, Me.BTT_TRAY_CLOSE, Me.BTT_TRAY_CLOSE_NO_SCRIPT})
         Me.TRAY_CONTEXT.Name = "TRAY_CONTEXT"
-        Me.TRAY_CONTEXT.Size = New System.Drawing.Size(171, 148)
+        Me.TRAY_CONTEXT.Size = New System.Drawing.Size(171, 170)
         '
         'BTT_TRAY_PAUSE_AUTOMATION
         '
@@ -883,6 +885,13 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         Me.BTT_TRAY_CLOSE_NO_SCRIPT.ToolTipText = "Close the program without executing the script"
         Me.BTT_TRAY_CLOSE_NO_SCRIPT.Visible = False
         '
+        'BTT_TRAY_CHANNELS
+        '
+        Me.BTT_TRAY_CHANNELS.Name = "BTT_TRAY_CHANNELS"
+        Me.BTT_TRAY_CHANNELS.Size = New System.Drawing.Size(170, 22)
+        Me.BTT_TRAY_CHANNELS.Text = "Channels"
+        Me.BTT_TRAY_CHANNELS.Image = Global.SCrawler.My.Resources.SiteResources.RedditPic_512
+        '
         'MainFrame
         '
         Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -986,4 +995,5 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
     Friend WithEvents BTT_DOWN_AUTOMATION_PAUSE As ToolStripMenuItem
     Private WithEvents BTT_TRAY_FEED_SHOW As ToolStripMenuItem
     Friend WithEvents MENU_DOWN_ALL As ToolStripDropDownButton
+    Private WithEvents BTT_TRAY_CHANNELS As ToolStripMenuItem
 End Class

SCrawler/MainFrame.vb

@@ -7,7 +7,6 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports System.Globalization
 Imports System.ComponentModel
 Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Functions.Messaging
@@ -21,9 +20,10 @@ Public Class MainFrame
 #Region "Declarations"
     Private MyView As FormView
     Private WithEvents MyActivator As FormActivator
+    Private WithEvents BTT_IMPORT_USERS As ToolStripMenuItem
     Private ReadOnly _VideoDownloadingMode As Boolean = False
-    Private MyChannels As ChannelViewForm
+    Friend MyChannels As ChannelViewForm
-    Private MySavedPosts As DownloadSavedPostsForm
+    Friend MySavedPosts As DownloadSavedPostsForm
     Private MyMissingPosts As MissingPostsForm
     Private MyFeed As DownloadFeedForm
     Private MySearch As UserSearchForm
@@ -32,10 +32,6 @@ Public Class MainFrame
 #Region "Initializer"
     Public Sub New()
         InitializeComponent()
-        Dim n As DateTimeFormatInfo = CultureInfo.GetCultureInfo("en-us").DateTimeFormat.Clone
-        n.FullDateTimePattern = "ddd MMM dd HH:mm:ss +ffff yyyy"
-        n.TimeSeparator = String.Empty
-        Twitter.DateProvider = New ADateTime(DirectCast(n.Clone, DateTimeFormatInfo)) With {.DateTimeStyle = DateTimeStyles.AssumeUniversal}
         Settings = New SettingsCLS
         With Settings.Plugins
             If .Count > 0 Then
@@ -44,6 +40,8 @@ Public Class MainFrame
                 Next
             End If
         End With
+        BTT_IMPORT_USERS = New ToolStripMenuItem With {.Text = "Import", .Image = My.Resources.UsersIcon_32.ToBitmap}
+        MENU_SETTINGS.DropDownItems.AddRange({New ToolStripSeparator, BTT_IMPORT_USERS})
         Dim Args() As String = Environment.GetCommandLineArgs
         If Args.ListExists(2) AndAlso Args(1) = "v" Then
             Using f As New VideosDownloaderForm With {.IsStandalone = True} : f.ShowDialog() : End Using
@@ -252,7 +250,6 @@ CloseResume:
 #Region "List refill, update"
     Friend Sub RefillList()
         UserListLoader.Update()
-        GC.Collect()
     End Sub
     Private Sub UserListUpdate(ByVal User As IUserData, ByVal Add As Boolean)
         UserListLoader.UpdateUser(User, Add)
@@ -278,6 +275,35 @@ CloseResume:
             End Using
         End With
     End Sub
+    Private Sub BTT_IMPORT_USERS_Click(sender As Object, e As EventArgs) Handles BTT_IMPORT_USERS.Click
+        Const MsgTitle$ = "Import users"
+        Try
+            Dim file As SFile = Nothing
+            Dim _OriginalLocations As Boolean = False
+            Select Case MsgBoxE({"Where do you want to import users from?" & vbCr & vbCr &
+                                 "This feature is not for importing users from the site. It's more like searching for missing users.", MsgTitle}, vbQuestion,,,
+                                {"Select path", New MsgBoxButton("Current", "All plugin paths will be checked"), "Cancel"}).Index
+                Case 0 : file = SFile.SelectPath
+                Case 1 : _OriginalLocations = True
+                Case Else : MsgBoxE({"Operation canceled", MsgTitle}) : Exit Sub
+            End Select
+            If Not file.IsEmptyString Or _OriginalLocations Then
+                Using import As New UserFinder(file)
+                    With import
+                        .Find(_OriginalLocations)
+                        If .Count > 0 Then
+                            .Verify()
+                            .Dialog()
+                        Else
+                            MsgBoxE({"No users found", MsgTitle})
+                        End If
+                    End With
+                End Using
+            End If
+        Catch ex As Exception
+            ErrorsDescriber.Execute(EDP.LogMessageValue, ex, MsgTitle)
+        End Try
+    End Sub
 #End Region
 #Region "Add, Edit, Delete, Refresh"
     Private Sub OnUsersAddedHandler(ByVal StartIndex As Integer)
@@ -353,7 +379,7 @@ CloseResume:
         If MyFeed Is Nothing Then MyFeed = New DownloadFeedForm : AddHandler Downloader.FeedFilesChanged, AddressOf MyFeed.Downloader_FilesChanged
         If MyFeed.Visible Then MyFeed.BringToFront() Else MyFeed.Show()
     End Sub
-    Private Sub BTT_CHANNELS_Click(sender As Object, e As EventArgs) Handles BTT_CHANNELS.Click
+    Private Sub BTT_CHANNELS_Click(sender As Object, e As EventArgs) Handles BTT_CHANNELS.Click, BTT_TRAY_CHANNELS.Click
         If MyChannels Is Nothing Then
             MyChannels = New ChannelViewForm
             AddHandler MyChannels.OnUsersAdded, AddressOf OnUsersAddedHandler
@@ -718,23 +744,47 @@ CloseResume:
         End If
     End Sub
     Private Sub BTT_CONTEXT_GROUPS_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_GROUPS.Click
+        Const MsgTitle$ = "Label change"
         Try
             Dim users As List(Of IUserData) = GetSelectedUserArray()
             If users.ListExists Then
                 Dim l As List(Of String) = ListAddList(Nothing, users.SelectMany(Function(u) u.Labels), LAP.NotContainsOnly)
-                Using f As New LabelsForm(l) With {.MultiUser = True}
+                Using f As New LabelsForm(l) With {.WithDeleteButton = l.Count > 0}
                     f.ShowDialog()
                     If f.DialogResult = DialogResult.OK Then
-                        Dim _lp As LAP = LAP.NotContainsOnly
+                        Dim labels As List(Of String) = f.LabelsList
-                        If f.MultiUserClearExists Then _lp += LAP.ClearBeforeAdd
+                        Dim lp As New ListAddParams(LAP.NotContainsOnly)
-                        Dim lp As New ListAddParams(_lp)
+                        Dim a As Action(Of IUserData) = Sub(u) u.Labels.ListAddList(labels, lp)
+                        Dim cMsg As New MMessage("Operation canceled", MsgTitle)
+                        If labels.ListExists Then
+                            Select Case MsgBoxE(New MMessage($"What do you want to do with the selected labels?{vbCr}Selected labels:{vbCr}{labels.ListToString(vbCr)}",
+                                                             MsgTitle,
+                                                             {
+                                                                New MsgBoxButton("Replace", "All existing labels will be removed and replaced with these labels"),
+                                                                New MsgBoxButton("Add", "These labels will be added to the existing ones"),
+                                                                New MsgBoxButton("Remove", "These labels will be removed from the existing ones"),
+                                                                "Cancel"
+                                                             }, vbExclamation) With {.ButtonsPerRow = 2}).Index
+                                Case 0 : lp.ClearBeforeAdd = True
+                                Case 1 : lp.ClearBeforeAdd = False
+                                Case 2 : a = Sub(u) u.Labels.ListDisposeRemove(labels)
+                                Case Else : cMsg.Show() : Exit Sub
+                            End Select
+                        Else
+                            If MsgBoxE({"Are you sure you want to remove all labels?", MsgTitle}, vbExclamation + vbYesNo) = vbYes Then
+                                a = Sub(u) u.Labels.Clear()
+                            Else
+                                cMsg.Show()
+                                Exit Sub
+                            End If
+                        End If
                         users.ForEach(Sub(ByVal u As IUserData)
                                           If u.IsCollection Then
                                               With DirectCast(u, UserDataBind)
-                                                  If .Count > 0 Then .Collections.ForEach(Sub(uu) uu.Labels.ListAddList(f.LabelsList, lp))
+                                                  If .Count > 0 Then .Collections.ForEach(a)
                                               End With
                                           Else
-                                              u.Labels.ListAddList(f.LabelsList, lp)
+                                              a.Invoke(u)
                                           End If
                                           u.UpdateUserInformation()
                                       End Sub)
@@ -818,11 +868,48 @@ CloseResume:
                             MainFrameObj.CollectionHandler(DirectCast(.Users.Last, UserDataBind))
                             userCollection = .Users.Last
                         End If
+
+                        Dim __modelUser As UsageModel = -1
+                        Dim __modelCollection As UsageModel = -1
+                        Dim __ModelAskForDecision As Boolean = False
+                        If Not Added Then __modelCollection = userCollection.CollectionModel
+                        If Added Then
+                            __ModelAskForDecision = True
+                        ElseIf userCollection.CollectionModel = UsageModel.Virtual Then
+                            __modelUser = UsageModel.Virtual
+                            __modelCollection = UsageModel.Virtual
+                        Else
+                            __ModelAskForDecision = True
+                        End If
+                        If __ModelAskForDecision Then
+                            Select Case MsgBoxE({"How do you want to add users to the collection?", MsgTitle}, vbQuestion,,,
+                                                {
+                                                    New MsgBoxButton("Default", "User files will be moved to the collection") With {.KeyCode = Keys.Enter},
+                                                    New MsgBoxButton("Virtual", "The user will be included in the collection, but user files will not be moved") With {
+                                                                     .KeyCode = New ButtonKey(Keys.Enter, True)}
+                                                }).Index
+                                Case 0
+                                    __modelUser = UsageModel.Default
+                                    If __modelCollection = -1 Then __modelCollection = UsageModel.Default
+                                Case 1
+                                    __modelUser = UsageModel.Virtual
+                                    If __modelCollection = -1 Then __modelCollection = UsageModel.Virtual
+                            End Select
+                        End If
+                        If __modelUser = -1 Or __modelCollection = -1 Then
+                            MsgBoxE({$"Some parameters cannot be processed:{vbCr}" &
+                                     $"UserModel: {CInt(__modelUser)}{vbCr}CollectionModel: {CInt(__modelCollection)}{vbCr}" &
+                                     "Operation canceled", MsgTitle}, vbCritical)
+                            Exit Sub
+                        End If
+
                         Dim __added_users As New List(Of IUserData)
                         Dim __added_users_not As New List(Of IUserData)
-                        For Each user As IUserData In users
+                        For Each user As UserDataBase In users
                             If Not user.IsCollection Then
                                 Try
+                                    user.User.UserModel = __modelUser
+                                    user.User.CollectionModel = __modelCollection
                                     userCollection.Add(user)
                                     RemoveUserFromList(user)
                                     UserListUpdate(userCollection, Added)
@@ -862,15 +949,17 @@ CloseResume:
         End If
     End Sub
     Private Sub BTT_CONTEXT_COL_MERGE_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_COL_MERGE.Click
+        Const MsgTitle$ = "Merging files"
         Dim user As IUserData = GetSelectedUser()
         If Not user Is Nothing Then
             If user.IsCollection Then
                 If DirectCast(user, UserDataBind).DataMerging Then
-                    MsgBoxE("Collection files are already merged")
+                    MsgBoxE({"Collection files are already merged", MsgTitle})
+                ElseIf user.IsVirtual Then
+                    MsgBoxE({"The action cannot be performed. This is a virtual collection.", MsgTitle}, vbCritical)
                 Else
-                    If MsgBoxE({"Do you really want to merge collection files into one folder?" & vbNewLine &
+                    If MsgBoxE({"Are you sure you want to merge the collection files into one folder?" & vbNewLine &
-                                "This action is not turnable!", "Merging files"},
+                                "This action is not turnable!", MsgTitle}, vbExclamation + vbYesNo) = vbYes Then
-                               MsgBoxStyle.Exclamation + MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
                         DirectCast(user, UserDataBind).DataMerging = True
                     End If
                 End If
@@ -880,83 +969,118 @@ CloseResume:
         End If
     End Sub
     Private Sub BTT_CONTEXT_CHANGE_FOLDER_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_CHANGE_FOLDER.Click
+        Const MsgTitle$ = "Change user folder"
         Try
+            If Downloader.Working Then
+                MsgBoxE({"Some users are currently downloading." & vbCr &
+                         "You cannot change paths while downloading." & vbCr &
+                         "Wait until the download is complete.", MsgTitle}, vbCritical)
+                Exit Sub
+            Else
+                Downloader.Suspended = True
+            End If
             Dim users As List(Of IUserData) = GetSelectedUserArray()
             If users.ListExists Then
                 If users.Count = 1 Then
                     Dim CutOption% = 1
                     Dim _IsCollection As Boolean = False
+                    Dim CurrDir As SFile
+                    Dim colName$ = String.Empty
                     With users(0)
                         If .IsCollection Then
                             _IsCollection = True
                             With DirectCast(.Self, UserDataBind)
                                 If .Count = 0 Then
                                     Throw New ArgumentOutOfRangeException("Collection", "Collection is empty")
+                                ElseIf .IsVirtual Then
+                                    MsgBoxE({"This is a virtual collection." & vbCr &
+                                             "The virtual collection path cannot be changed." & vbCr &
+                                             "To change the paths of users included in a virtual collection, " &
+                                             "you must split the collection and then change the user paths.", MsgTitle}, vbCritical)
+                                    Exit Sub
                                 Else
-                                    With DirectCast(.Collections(0), UserDataBase)
+                                    CurrDir = .GetRealUserFile
-                                        If Not .User.Merged Then CutOption = 2
+                                    If CurrDir.IsEmptyString Then
-                                    End With
+                                        MsgBoxE({"Non-virtual users not found", MsgTitle}, vbCritical)
+                                        Exit Sub
+                                    End If
+                                    CurrDir = CurrDir.CutPath(IIf(.DataMerging, 3, 2))
+                                    colName = CurrDir.PathFolders.LastOrDefault
+                                    Dim vu As IEnumerable(Of IUserData) = .Where(Function(vuu) vuu.UserModel = UsageModel.Virtual)
+                                    If vu.ListExists Then
+                                        If MsgBoxE({"This collection contains virtual users." & vbCr &
+                                                    "If you continue, the virtual user paths will not be changed." & vbCr &
+                                                    "The following users have been added to the collection in virtual mode:" & vbCr &
+                                                    vu.ListToStringE(vbCr, GetUserListProvider(False)), MsgTitle},
+                                                   vbExclamation,,, {"Continue", "Cancel"}) = 1 Then MsgBoxE({"Operation canceled", MsgTitle}) : Exit Sub
+                                    End If
                                 End If
                             End With
+                        Else
+                            CurrDir = .Self.File.CutPath(1)
                         End If
-                    End With
 
-                    Dim CurrDir As SFile = users(0).File.CutPath(CutOption)
+                        Dim NewDest As SFile = SFile.SelectPath(CurrDir, $"Select a new destination for {IIf(_IsCollection, "collection", "user")} [{ .Self}]")
-                    Dim NewDest As SFile = SFile.GetPath(InputBoxE($"Enter a new destination for user [{users(0)}]", "Change user folder", CurrDir.Path))
+                        If Not NewDest.IsEmptyString Then
-                    If Not NewDest.IsEmptyString Then
+                            NewDest = $"{NewDest.PathWithSeparator}{colName}\"
-                        If MsgBoxE({$"You are changing the user's [{users(0)}] destination" & vbCr &
+                            If MsgBoxE({$"You are changing the user's [{ .Self}] destination" & vbCr &
-                                    $"Current destination: {CurrDir.PathNoSeparator}" & vbCr &
+                                        $"Current destination: {CurrDir.PathNoSeparator}" & vbCr &
-                                    $"New destination: {NewDest.Path}",
+                                        $"New destination: {NewDest.PathNoSeparator}",
-                                    "Changing user destination"}, MsgBoxStyle.Exclamation,,, {"Confirm", "Cancel"}) = 0 Then
+                                        MsgTitle}, MsgBoxStyle.Exclamation,,, {"Confirm", "Cancel"}) = 0 Then
-                            If Not NewDest.IsEmptyString AndAlso
+                                If Not NewDest.IsEmptyString AndAlso
-                               (Not NewDest.Exists(SFO.Path, False) OrElse
+                                   (Not NewDest.Exists(SFO.Path, False) OrElse
-                                    (
+                                        (
-                                        SFile.GetFiles(NewDest,, IO.SearchOption.AllDirectories, EDP.ThrowException).ListIfNothing.Count = 0 AndAlso
+                                            SFile.GetFiles(NewDest,, IO.SearchOption.AllDirectories, EDP.ThrowException).ListIfNothing.Count = 0 AndAlso
-                                        NewDest.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException) AndAlso
+                                            NewDest.Delete(SFO.Path, Settings.DeleteMode, EDP.ThrowException) AndAlso
-                                        Not NewDest.Exists(SFO.Path, False)
+                                            Not NewDest.Exists(SFO.Path, False)
-                                    )
+                                        )
-                               ) Then
+                                   ) Then
-                                NewDest.CutPath.Exists(SFO.Path)
+                                    If SFile.Move(CurrDir, NewDest, SFO.Path,,, EDP.ShowMainMsg + EDP.ReturnValue) Then
-                                IO.Directory.Move(CurrDir.Path, NewDest.Path)
+                                        Dim ApplyChanges As Action(Of IUserData) = Sub(ByVal __user As IUserData)
-                                Dim ApplyChanges As Action(Of IUserData) = Sub(ByVal __user As IUserData)
+                                                                                       With DirectCast(__user, UserDataBase)
-                                                                               With DirectCast(__user, UserDataBase)
+                                                                                           Dim u As UserInfo = .User
-                                                                                   Dim u As UserInfo = .User.Clone
+                                                                                           Settings.UsersList.Remove(u)
-                                                                                   Settings.UsersList.Remove(u)
+                                                                                           If _IsCollection Then
-                                                                                   Dim d As SFile = Nothing
+                                                                                               u.SpecialCollectionPath = NewDest
-                                                                                   If _IsCollection Then d = SFile.GetPath($"{NewDest.PathWithSeparator}{u.File.PathFolders(1).LastOrDefault}")
+                                                                                           Else
-                                                                                   If d.IsEmptyString Then d = NewDest
+                                                                                               u.SpecialPath = NewDest
-                                                                                   u.SpecialPath = d.PathWithSeparator
+                                                                                           End If
-                                                                                   u.UpdateUserFile()
+                                                                                           u.UpdateUserFile()
-                                                                                   Settings.UpdateUsersList(u)
+                                                                                           Settings.UsersList.Add(u)
-                                                                                   .User = u.Clone
+                                                                                           .User = u
-                                                                                   .UpdateUserInformation()
+                                                                                           .UpdateUserInformation()
-                                                                               End With
+                                                                                       End With
-                                                                           End Sub
+                                                                                   End Sub
-                                If users(0).IsCollection Then
+                                        If .Self.IsCollection Then
-                                    With DirectCast(users(0), UserDataBind)
+                                            With DirectCast(.Self, UserDataBind)
-                                        For Each user In .Collections : ApplyChanges(user) : Next
+                                                For Each user In .Collections : ApplyChanges(user) : Next
-                                    End With
+                                            End With
+                                        Else
+                                            ApplyChanges(.Self)
+                                        End If
+                                        Settings.UpdateUsersList()
+                                        MsgBoxE({"User data has been moved", MsgTitle})
+                                    End If
                                 Else
-                                    ApplyChanges(users(0))
+                                    MsgBoxE({$"Unable to move user data to new destination [{NewDest}]{vbCr}Operation canceled", MsgTitle}, MsgBoxStyle.Critical)
                                 End If
-                                MsgBoxE($"User data has been moved")
                             Else
-                                MsgBoxE($"Unable to move user data to new destination [{NewDest}]{vbCr}Operation canceled", MsgBoxStyle.Critical)
+                                MsgBoxE({"Operation canceled", MsgTitle})
                             End If
                         Else
-                            MsgBoxE("Operation canceled")
+                            MsgBoxE({$"You have not entered a new destination{vbCr}Operation canceled", MsgTitle}, MsgBoxStyle.Exclamation)
                         End If
-                    Else
+                    End With
-                        MsgBoxE("You have not entered a new destination" & vbCr & "Operation canceled", MsgBoxStyle.Exclamation)
-                    End If
                 Else
-                    MsgBoxE("You have selected multiple users. You can change the folder only for one user!", MsgBoxStyle.Critical)
+                    MsgBoxE({"You have selected multiple users. You can change the folder only for one user!", MsgTitle}, MsgBoxStyle.Critical)
                 End If
             Else
-                MsgBoxE("No one user selected", MsgBoxStyle.Exclamation)
+                MsgBoxE({"No one user selected", MsgTitle}, MsgBoxStyle.Exclamation)
             End If
         Catch ex As Exception
             ErrorsDescriber.Execute(EDP.ShowAllMsg, ex, "Error while moving user")
+        Finally
+            Downloader.Suspended = False
         End Try
     End Sub
 #End Region
@@ -987,7 +1111,7 @@ CloseResume:
 #Region "6 - information"
     Private Sub BTT_CONTEXT_INFO_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_INFO.Click
         Dim user As IUserData = GetSelectedUser()
-        If Not user Is Nothing Then MsgBoxE(DirectCast(user, UserDataBase).GetUserInformation())
+        If Not user Is Nothing Then MsgBoxE(New MMessage(DirectCast(user, UserDataBase).GetUserInformation(), "User information") With {.Editable = True})
     End Sub
 #End Region
     Private Sub USER_CONTEXT_VisibleChanged(sender As Object, e As EventArgs) Handles USER_CONTEXT.VisibleChanged
@@ -1169,7 +1293,7 @@ CancelDownloadingOperation:
             Exit Sub
 ResumeDownloadingOperation:
             Dim uStr$ = If(users.Count = 1, String.Empty, users.ListToStringE(vbNewLine, GetUserListProvider(True)))
-            Dim fStr$ = $"({IIf(IncludeInTheFeed, "included in", "excluded from")} the feed)"
+            Dim fStr$ = $" ({IIf(IncludeInTheFeed, "included in", "excluded from")} the feed)"
             If users.Count = 1 OrElse MsgBoxE({$"You have selected {users.Count} user profiles" & vbCr &
                                                $"Do you want to download them all{fStr}?{vbNewLine.StringDup(2)}" &
                                                $"Selected users:{vbNewLine}{uStr}", "Multiple users selected"},
@@ -1184,16 +1308,61 @@ ResumeDownloadingOperation:
         End If
     End Sub
     Private Sub EditSelectedUser()
+        Const MsgTitle$ = "User update"
         Dim user As IUserData = GetSelectedUser()
         If Not user Is Nothing Then
             On Error Resume Next
-            If user.IsCollection Then
+            If Not user.IsCollection OrElse DirectCast(user, UserDataBind).Count > 0 Then
-                If USER_CONTEXT.Visible Then USER_CONTEXT.Hide()
+                If user.IsCollection And USER_CONTEXT.Visible Then USER_CONTEXT.Hide()
-                MsgBoxE($"This is collection!{vbNewLine}Collection editing not allowed!", vbExclamation)
-            Else
                 Using f As New UserCreatorForm(user)
                     f.ShowDialog()
-                    If f.DialogResult = DialogResult.OK Then UserListUpdate(user, False)
+                    If f.DialogResult = DialogResult.OK Then
+                        Dim NeedToUpdate As Boolean = True
+                        If user.IsCollection Then
+                            If user.IsCollection And Not user.CollectionName = f.CollectionName Then
+                                If Not user.IsVirtual AndAlso Downloader.Working Then
+                                    MsgBoxE({"Some users are currently downloading." & vbCr &
+                                             "You cannot change collection name while downloading." & vbCr &
+                                             "Wait until the download is complete.", MsgTitle}, vbCritical)
+                                    Exit Sub
+                                Else
+                                    If Not user.IsVirtual Then
+                                        Dim colFile As SFile = DirectCast(user, UserDataBind).GetRealUserFile
+                                        If Not colFile.IsEmptyString Then
+                                            colFile = colFile.CutPath(IIf(DirectCast(user, UserDataBind).DataMerging, 1, 2))
+                                            If Not colFile.IsEmptyString Then
+                                                Dim nf As SFile = $"{colFile.CutPath(1).PathWithSeparator}{f.CollectionName}".CSFilePS
+                                                If Not SFile.Rename(colFile, New SFile With {.Path = f.CollectionName}, SFO.Path,
+                                                                    New ErrorsDescriber(True, False, False, New SFile)).IsEmptyString Then
+                                                    RemoveUserFromList(user)
+                                                    Dim __user As UserInfo
+                                                    For Each ColUser As UserDataBase In DirectCast(user, UserDataBind).Collections
+                                                        __user = ColUser.User
+                                                        Settings.UsersList.Remove(__user)
+                                                        __user.CollectionName = f.CollectionName
+                                                        If Not __user.SpecialCollectionPath.IsEmptyString Then __user.SpecialCollectionPath = nf
+                                                        __user.UpdateUserFile()
+                                                        ColUser.User = __user
+                                                        Settings.UsersList.Add(__user)
+                                                    Next
+                                                    user.UpdateUserInformation()
+                                                    UserListUpdate(user, True)
+                                                    NeedToUpdate = False
+                                                End If
+                                            End If
+                                        End If
+                                    Else
+                                        RemoveUserFromList(user)
+                                        user.CollectionName = f.CollectionName
+                                        user.UpdateUserInformation()
+                                        UserListUpdate(user, True)
+                                        NeedToUpdate = False
+                                    End If
+                                End If
+                            End If
+                        End If
+                        If NeedToUpdate Then UserListUpdate(user, False)
+                    End If
                 End Using
             End If
         End If
@@ -1206,19 +1375,26 @@ ResumeDownloadingOperation:
                 Dim userProvider As IFormatProvider = GetUserListProvider(True)
                 Dim ugn As Func(Of IUserData, String) = Function(u) AConvert(Of String)(u, userProvider)
                 Dim m As New MMessage(users.ListToStringE(vbNewLine, userProvider), "Users deleting",
-                                      {New MsgBoxButton("Delete and ban") With {.ToolTip = "Users and their data will be deleted and added to the blacklist"},
+                                      {New MsgBoxButton("Delete and ban") With {
+                                            .ToolTip = "Users and their data will be deleted and added to the blacklist",
+                                            .KeyCode = Keys.Enter},
                                        New MsgBoxButton("Delete user only and ban") With {
                                             .ToolTip = "Users will be deleted and added to the blacklist (user data will not be deleted)"},
                                        New MsgBoxButton("Delete and ban with reason") With {
-                                            .ToolTip = "Users and their data will be deleted and added to the blacklist with set a reason to delete"},
+                                            .ToolTip = "Users and their data will be deleted and added to the blacklist with set a reason to delete",
+                                            .KeyCode = New ButtonKey(Keys.Enter,, True)},
                                        New MsgBoxButton("Delete user only and ban with reason") With {
                                             .ToolTip = "Users will be deleted and added to the blacklist with set a reason to delete (user data will not be deleted)"},
-                                       New MsgBoxButton("Delete") With {.ToolTip = "Delete users and their data"},
+                                       New MsgBoxButton("Delete") With {
+                                            .ToolTip = "Delete users and their data",
+                                            .KeyCode = New ButtonKey(Keys.Enter, True)},
                                        New MsgBoxButton("Delete user only") With {.ToolTip = "Delete users but keep data"}, "Cancel"},
                                       MsgBoxStyle.Exclamation) With {.ButtonsPerRow = 2, .ButtonsPlacing = MMessage.ButtonsPlacings.StartToEnd}
                 m.Text = $"The following users ({users.Count}) will be deleted:{vbNewLine}{m.Text}"
                 Dim result% = MsgBoxE(m)
                 If result < 6 Then
+                    Dim collectionResult% = -1
+                    Dim tmpResult%
                     Dim IsMultiple As Boolean = users.Count > 1
                     Dim removedUsers As New List(Of String)
                     Dim keepData As Boolean = Not (result Mod 2) = 0
@@ -1253,7 +1429,9 @@ ResumeDownloadingOperation:
                             removedUsers.Add(ugn(user))
                             user.Dispose()
                         Else
-                            If user.Delete(IsMultiple) > 0 Then
+                            tmpResult = user.Delete(IsMultiple, collectionResult)
+                            If user.IsCollection And collectionResult = -1 Then collectionResult = tmpResult
+                            If tmpResult > 0 Then
                                 If banUser Then Settings.BlackList.ListAddValue(New UserBan(user.Name, reason), l) : b = True
                                 RemoveUserFromList(user)
                                 removedUsers.Add(ugn(user))
@@ -1273,8 +1451,8 @@ ResumeDownloadingOperation:
                         m.Text = "No one user deleted!"
                         m.Style = MsgBoxStyle.Critical
                     Else
-                        m.Text = $"The following users were deleted:{vbNewLine}{removedUsers.ListToStringE(vbNewLine, userProvider)}{vbNewLine.StringDup(2)}"
+                        m.Text = $"The following users were deleted:{vbNewLine}{removedUsers.ListToString(vbNewLine)}{vbNewLine.StringDup(2)}"
-                        m.Text &= $"The following users were NOT deleted:{vbNewLine}{leftUsers.ListToStringE(vbNewLine, userProvider)}"
+                        m.Text &= $"The following users were NOT deleted:{vbNewLine}{leftUsers.ListToString(vbNewLine)}"
                         m.Style = MsgBoxStyle.Exclamation
                     End If
                     If b Then Settings.UpdateBlackList()
@@ -1294,7 +1472,7 @@ ResumeDownloadingOperation:
             If users.ListExists Then
                 Dim f As SFile = Settings.LastCopyPath
                 Dim _select_path As Func(Of Boolean) = Function() As Boolean
-                                                           f = SFile.SelectPath(f, True)
+                                                           f = SFile.SelectPath(f)
                                                            If f.Exists(SFO.Path, False) Then
                                                                Return MsgBoxE({$"Are you sure you want to copy the data to the selected folder?{vbCr}{f}",
                                                                               MsgTitle}, vbQuestion + vbYesNo) = vbYes

SCrawler/MainFrameObjects.vb

@@ -9,6 +9,7 @@
 Imports SCrawler.API
 Imports SCrawler.API.Base
 Imports PersonalUtilities.Tools.Notifications
+Imports NotifyObj = SCrawler.SettingsCLS.NotificationObjects
 Friend Class MainFrameObjects
     Friend ReadOnly Property MF As MainFrame
     Private WithEvents Notificator As NotificationsManager
@@ -63,16 +64,28 @@ Friend Class MainFrameObjects
 #End Region
 #Region "Notifications"
     Private Const NotificationInternalKey As String = "NotificationInternalKey"
-    Friend Sub ShowNotification(ByVal Sender As SettingsCLS.NotificationObjects, ByVal Message As String)
+    Friend Sub ShowNotification(ByVal Sender As NotifyObj, ByVal Message As String)
         If Settings.ProcessNotification(Sender) Then
-            Using n As New Notification(Message) With {.Key = NotificationInternalKey} : n.Show() : End Using
+            Using n As New Notification(Message) With {.Key = $"{NotificationInternalKey}_{Sender}"} : n.Show() : End Using
         End If
     End Sub
     Friend Sub ClearNotifications()
         Notificator.Clear()
     End Sub
     Private Sub Notificator_OnClicked(ByVal Key As String) Handles Notificator.OnClicked
-        If Key = NotificationInternalKey OrElse Settings.Automation Is Nothing OrElse Not Settings.Automation.NotificationClicked(Key) Then Focus(True)
+        If Not Key.IsEmptyString Then
+            If Key.StartsWith(NotificationInternalKey) Then
+                Select Case Key
+                    Case $"{NotificationInternalKey}_{NotifyObj.Channels}" : MF.MyChannels.FormShowS()
+                    Case $"{NotificationInternalKey}_{NotifyObj.SavedPosts}" : MF.MySavedPosts.FormShowS()
+                    Case Else : Focus(True)
+                End Select
+            ElseIf Settings.Automation Is Nothing OrElse Not Settings.Automation.NotificationClicked(Key) Then
+                Focus(True)
+            Else
+                Focus(True)
+            End If
+        End If
     End Sub
 #End Region
 End Class

SCrawler/MainMod.vb

@@ -10,7 +10,7 @@ Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web
 Imports SCrawler.API
 Imports SCrawler.API.Base
 Imports SCrawler.Plugin.Hosts
@@ -93,6 +93,10 @@ Friend Module MainMod
         Replace = 1
         Add = 2
     End Enum
+    Friend Enum UsageModel As Integer
+        [Default] = 0
+        Virtual = 1
+    End Enum
     Friend Downloader As TDownloader
     Friend InfoForm As DownloadedInfoForm
     Friend VideoDownloader As VideosDownloaderForm

SCrawler/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2022.10.23.0")>
+<Assembly: AssemblyVersion("2022.12.26.0")>
-<Assembly: AssemblyFileVersion("2022.10.23.0")>
+<Assembly: AssemblyFileVersion("2022.12.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/My Project/Resources.Designer.vb

@@ -370,6 +370,16 @@ Namespace My.Resources
             End Get
         End Property
         
+        '''<summary>
+        '''  Looks up a localized resource of type System.Drawing.Bitmap.
+        '''</summary>
+        Friend ReadOnly Property TagPic_24() As System.Drawing.Bitmap
+            Get
+                Dim obj As Object = ResourceManager.GetObject("TagPic_24", resourceCulture)
+                Return CType(obj,System.Drawing.Bitmap)
+            End Get
+        End Property
+        
         '''<summary>
         '''  Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
         '''</summary>

SCrawler/My Project/Resources.resx

@@ -214,4 +214,7 @@
   <data name="TagIcon_32" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>..\Content\Icons\TagIcon_32.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
   </data>
+  <data name="TagPic_24" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>..\Content\Pictures\TagPic_24.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
+  </data>
 </root>

SCrawler/PluginsEnvironment/Hosts/PluginHost.vb

@@ -81,6 +81,8 @@ Namespace Plugin.Hosts
                     New PluginHost(New API.RedGifs.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids),
                     New PluginHost(New API.TikTok.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids),
                     New PluginHost(New API.LPSG.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids),
+                    New PluginHost(New API.PornHub.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids),
+                    New PluginHost(New API.Xhamster.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids),
                     New PluginHost(New API.XVIDEOS.SiteSettings, _XML, GlobalPath, _Temp, _Imgs, _Vids)}
         End Function
         Friend Shared Function GetPluginsHosts(ByRef _XML As XmlFile, ByVal GlobalPath As SFile,

SCrawler/PluginsEnvironment/Hosts/PropertyValueHost.vb

@@ -50,7 +50,7 @@ Namespace Plugin.Hosts
                     End With
                     If Not .ControlToolTip.IsEmptyString And Not TT Is Nothing Then TT.SetToolTip(Control, .ControlToolTip)
                 Else
-                    If Type Is GetType(Boolean) Then
+                    If Type Is GetType(Boolean) Or .ThreeStates Then
                         Control = New CheckBox
                         If Not .ControlToolTip.IsEmptyString And Not TT Is Nothing Then TT.SetToolTip(Control, .ControlToolTip)
                         DirectCast(Control, CheckBox).ThreeState = .ThreeStates

SCrawler/PluginsEnvironment/Hosts/SettingsHost.vb

@@ -11,7 +11,7 @@ Imports SCrawler.API.Base
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
-Imports PersonalUtilities.Tools.WEB
+Imports PersonalUtilities.Tools.Web.Clients
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace Plugin.Hosts
     Friend Class SettingsHost
@@ -71,15 +71,16 @@ Namespace Plugin.Hosts
                         Dim i% = PropList.FindIndex(Function(p) p.IsTaskCounter)
                         If i >= 0 Then Return CInt(PropList(i).Value)
                     End If
-                    If _TaskCountDefined.HasValue Then Return _TaskCountDefined.Value
+                    If _TaskCountDefined.HasValue AndAlso _TaskCountDefined.Value > 0 Then Return _TaskCountDefined.Value
                 End If
                 Return Settings.MaxUsersJobsCount
             End Get
         End Property
+        Friend ReadOnly Property TaskGroupName As String = String.Empty
         Friend ReadOnly Property HasSpecialOptions As Boolean = False
         Private ReadOnly _ResponserGetMethod As MethodInfo
         Private ReadOnly _ResponserIsContainer As Boolean = False
-        Friend ReadOnly Property Responser As Response
+        Friend ReadOnly Property Responser As Responser
             Get
                 If Not _ResponserGetMethod Is Nothing Then
                     Return _ResponserGetMethod.Invoke(Source, Nothing)
@@ -156,6 +157,8 @@ Namespace Plugin.Hosts
                         With DirectCast(a, SeparatedTasks)
                             If .TasksCount > 0 Then _TaskCountDefined = .TasksCount
                         End With
+                    ElseIf TypeOf a Is TaskGroup Then
+                        TaskGroupName = DirectCast(a, TaskGroup).Name
                     ElseIf TypeOf a Is SavedPosts Then
                         IsSavedPostsCompatible = True
                     ElseIf TypeOf a Is SpecialForm Then
@@ -187,7 +190,7 @@ Namespace Plugin.Hosts
                     If m.MemberType = MemberTypes.Property Then
                         PropList.Add(New PropertyValueHost(Source, m))
                         With DirectCast(m, PropertyInfo)
-                            If .PropertyType Is GetType(Response) AndAlso m.GetCustomAttribute(Of DoNotUse)() Is Nothing Then _ResponserGetMethod = .GetMethod
+                            If .PropertyType Is GetType(Responser) AndAlso m.GetCustomAttribute(Of DoNotUse)() Is Nothing Then _ResponserGetMethod = .GetMethod
                         End With
                     End If
                     With m.GetCustomAttributes()
@@ -291,8 +294,8 @@ Namespace Plugin.Hosts
             If Not um Is Nothing Then
                 If TypeOf um Is IEnumerable(Of UserMedia) Then
                     Return um
-                ElseIf TypeOf um Is IEnumerable(Of PluginUserMedia) Then
+                ElseIf TypeOf um Is IEnumerable(Of IUserMedia) Then
-                    Return um.ToObjectsList.ListCast(Of UserMedia)(New ListAddParams With {.Converter = Function(v) New UserMedia(DirectCast(v, PluginUserMedia))})
+                    Return um.ToObjectsList.ListCast(Of UserMedia)(New ListAddParams With {.Converter = Function(v) New UserMedia(DirectCast(v, IUserMedia))})
                 End If
             End If
             Return Nothing
@@ -309,8 +312,8 @@ Namespace Plugin.Hosts
                 Throw New ArgumentNullException("IPluginContentProvider", $"Plugin [{Key}] does not provide user instance")
             End If
         End Function
-        Friend Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String
+        Friend Function GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String
-            Return Source.GetUserPostUrl(UserID, PostID)
+            Return Source.GetUserPostUrl(User, Media)
         End Function
         Private _AvailableValue As Boolean = True
         Private _AvailableAsked As Boolean = False

SCrawler/PluginsEnvironment/Hosts/UserDataHost.vb

@@ -10,8 +10,8 @@ Imports System.Threading
 Imports System.Reflection
 Imports PersonalUtilities.Functions.XML
 Imports SCrawler.API.Base
-Imports UStates = SCrawler.Plugin.PluginUserMedia.States
+Imports UStates = SCrawler.Plugin.UserMediaStates
-Imports UTypes = SCrawler.Plugin.PluginUserMedia.Types
+Imports UTypes = SCrawler.Plugin.UserMediaTypes
 Namespace Plugin.Hosts
     Friend Class UserDataHost : Inherits UserDataBase
         Private ReadOnly UseInternalDownloader As Boolean
@@ -56,11 +56,11 @@ Namespace Plugin.Hosts
                 .DownloadDateFrom = DownloadDateFrom
                 .DownloadDateTo = DownloadDateTo
 
-                .ExistingContentList = New List(Of PluginUserMedia)
+                .ExistingContentList = New List(Of IUserMedia)
-                .TempMediaList = New List(Of PluginUserMedia)
+                .TempMediaList = New List(Of IUserMedia)
                 .TempPostsList = New List(Of String)
 
-                If _ContentList.Count > 0 Then ExternalPlugin.ExistingContentList = _ContentList.Select(Function(u) u.PluginUserMedia).ToList
+                If _ContentList.Count > 0 Then ExternalPlugin.ExistingContentList = _ContentList.ListCast(Of IUserMedia)
                 ExternalPlugin.TempPostsList = ListAddList(Nothing, _TempPostsList)
 
                 .GetMedia()
@@ -81,8 +81,8 @@ Namespace Plugin.Hosts
             Else
                 With ExternalPlugin
                     If .TempMediaList.ListExists Then .TempMediaList.Clear()
-                    .TempMediaList = New List(Of PluginUserMedia)
+                    .TempMediaList = New List(Of IUserMedia)
-                    .TempMediaList.ListAddList(_ContentNew.Select(Function(c) c.PluginUserMedia()))
+                    .TempMediaList.ListAddList(_ContentNew)
                     .Download()
                     _ContentNew.Clear()
                     If .TempMediaList.ListExists Then

SCrawler/SCrawler.vbproj

@@ -158,17 +158,13 @@
     <Import Include="System.Threading.Tasks" />
   </ItemGroup>
   <ItemGroup>
+    <Compile Include="API\BaseObjects\DomainEnvir.vb" />
     <Compile Include="API\Base\Declarations.vb" />
     <Compile Include="API\Base\DownDetector.vb" />
+    <Compile Include="API\Base\M3U8Base.vb" />
     <Compile Include="API\Base\ProfileSaved.vb" />
     <Compile Include="API\Base\SiteSettingsBase.vb" />
     <Compile Include="API\Base\Structures.vb" />
-    <Compile Include="API\Instagram\AdditionalSettingsForm.Designer.vb">
-      <DependentUpon>AdditionalSettingsForm.vb</DependentUpon>
-    </Compile>
-    <Compile Include="API\Instagram\AdditionalSettingsForm.vb">
-      <SubType>Form</SubType>
-    </Compile>
     <Compile Include="API\Instagram\EditorExchangeOptions.vb" />
     <Compile Include="API\Instagram\OptionsForm.Designer.vb">
       <DependentUpon>OptionsForm.vb</DependentUpon>
@@ -176,10 +172,20 @@
     <Compile Include="API\Instagram\OptionsForm.vb">
       <SubType>Form</SubType>
     </Compile>
-    <Compile Include="API\Instagram\SettingsExchangeOptions.vb" />
     <Compile Include="API\LPSG\Declarations.vb" />
     <Compile Include="API\LPSG\SiteSettings.vb" />
     <Compile Include="API\LPSG\UserData.vb" />
+    <Compile Include="API\PornHub\Declarations.vb" />
+    <Compile Include="API\PornHub\M3U8.vb" />
+    <Compile Include="API\PornHub\OptionsForm.Designer.vb">
+      <DependentUpon>OptionsForm.vb</DependentUpon>
+    </Compile>
+    <Compile Include="API\PornHub\OptionsForm.vb">
+      <SubType>Form</SubType>
+    </Compile>
+    <Compile Include="API\PornHub\SiteSettings.vb" />
+    <Compile Include="API\PornHub\UserData.vb" />
+    <Compile Include="API\PornHub\UserExchangeOptions.vb" />
     <Compile Include="API\Reddit\RedditViewSettingsForm.Designer.vb">
       <DependentUpon>RedditViewSettingsForm.vb</DependentUpon>
     </Compile>
@@ -190,14 +196,12 @@
     <Compile Include="API\TikTok\Declarations.vb" />
     <Compile Include="API\TikTok\SiteSettings.vb" />
     <Compile Include="API\TikTok\UserData.vb" />
+    <Compile Include="API\Xhamster\Declarations.vb" />
+    <Compile Include="API\Xhamster\M3U8.vb" />
+    <Compile Include="API\Xhamster\SiteSettings.vb" />
+    <Compile Include="API\Xhamster\UserData.vb" />
     <Compile Include="API\XVIDEOS\Declarations.vb" />
     <Compile Include="API\XVIDEOS\M3U8.vb" />
-    <Compile Include="API\XVIDEOS\SettingsForm.Designer.vb">
-      <DependentUpon>SettingsForm.vb</DependentUpon>
-    </Compile>
-    <Compile Include="API\XVIDEOS\SettingsForm.vb">
-      <SubType>Form</SubType>
-    </Compile>
     <Compile Include="API\XVIDEOS\SiteSettings.vb" />
     <Compile Include="API\XVIDEOS\UserData.vb" />
     <Compile Include="Download\ActiveDownloadingProgress.Designer.vb">
@@ -258,6 +262,7 @@
     <Compile Include="Download\Automation\SchedulerEditorForm.vb">
       <SubType>Form</SubType>
     </Compile>
+    <Compile Include="Download\WebClient2.vb" />
     <Compile Include="EncryptCookies.vb" />
     <Compile Include="GlobalSuppressions.vb" />
     <Compile Include="MainFrameObjects.vb" />
@@ -385,6 +390,7 @@
       <SubType>Component</SubType>
     </Compile>
     <Compile Include="UserBan.vb" />
+    <Compile Include="UserFinder.vb" />
     <Compile Include="UserImage.vb" />
     <Compile Include="Download\VideosDownloaderForm.Designer.vb">
       <DependentUpon>VideosDownloaderForm.vb</DependentUpon>
@@ -401,18 +407,15 @@
     </Compile>
   </ItemGroup>
   <ItemGroup>
-    <EmbeddedResource Include="API\Instagram\AdditionalSettingsForm.resx">
-      <DependentUpon>AdditionalSettingsForm.vb</DependentUpon>
-    </EmbeddedResource>
     <EmbeddedResource Include="API\Instagram\OptionsForm.resx">
       <DependentUpon>OptionsForm.vb</DependentUpon>
     </EmbeddedResource>
+    <EmbeddedResource Include="API\PornHub\OptionsForm.resx">
+      <DependentUpon>OptionsForm.vb</DependentUpon>
+    </EmbeddedResource>
     <EmbeddedResource Include="API\Reddit\RedditViewSettingsForm.resx">
       <DependentUpon>RedditViewSettingsForm.vb</DependentUpon>
     </EmbeddedResource>
-    <EmbeddedResource Include="API\XVIDEOS\SettingsForm.resx">
-      <DependentUpon>SettingsForm.vb</DependentUpon>
-    </EmbeddedResource>
     <EmbeddedResource Include="Channels\ChannelsStatsForm.resx">
       <DependentUpon>ChannelsStatsForm.vb</DependentUpon>
     </EmbeddedResource>
@@ -512,6 +515,9 @@
     <None Include="Content\Pictures\StarPic_24.png" />
     <None Include="Content\Pictures\StartPic_Green_16.png" />
     <None Include="Content\Pictures\StopPic_32.png" />
+    <None Include="cURL\curl-ca-bundle.crt">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </None>
     <None Include="My Project\app.manifest" />
     <None Include="My Project\Application.myapp">
       <Generator>MyApplicationCodeGenerator</Generator>
@@ -564,6 +570,20 @@
     <None Include="Content\Icons\ArrowDownIcon_Blue_24.ico" />
     <None Include="Content\Pictures\PinPic_32.png" />
     <None Include="Content\Icons\TagIcon_32.ico" />
+    <None Include="Content\Pictures\SitePictures\XhamsterPic_32.png" />
+    <None Include="Content\Icons\SiteIcons\XhamsterIcon_32.ico" />
+    <None Include="Content\Pictures\SitePictures\PornHubPic_16.png" />
+    <None Include="Content\Icons\SiteIcons\PornHubIcon_16.ico" />
+    <None Include="Content\Pictures\TagPic_24.png" />
+    <Content Include="cURL\curl.exe">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+    <Content Include="cURL\libcurl-x64.def">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+    <Content Include="cURL\libcurl-x64.dll">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
     <Content Include="ffmpeg.exe">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </Content>

SCrawler/SettingsCLS.vb

@@ -17,17 +17,20 @@ Imports SCrawler.Plugin.Hosts
 Imports SCrawler.DownloadObjects
 Friend Class SettingsCLS : Implements IDisposable
     Friend Const DefaultMaxDownloadingTasks As Integer = 5
+    Friend Const TaskStackNamePornSite As String = "Porn sites"
     Friend Const Name_Node_Sites As String = "Sites"
     Private Const SitesValuesSeparator As String = ","
     Friend Const CookieEncryptKey As String = "SCrawlerCookiesEncryptKeyword"
     Friend ReadOnly Design As XmlFile
     Private ReadOnly MyXML As XmlFile
-    Friend ReadOnly OS64 As Boolean
+    Private ReadOnly OS64 As Boolean
     Private ReadOnly FfmpegExists As Boolean
     Friend ReadOnly FfmpegFile As SFile
     Friend ReadOnly Property UseM3U8 As Boolean
         Get
-            Return OS64 And FfmpegExists
+            'TODELETE: SETTINGS m3u8 delete after debug ffmpeg x86
+            'Return OS64 And FfmpegExists
+            Return FfmpegExists
         End Get
     End Property
     Private ReadOnly FFMPEGNotification As XMLValue(Of Boolean)
@@ -256,15 +259,14 @@ Friend Class SettingsCLS : Implements IDisposable
                 If UsersList.Count > 0 Then
                     Dim cUsers As List(Of UserInfo) = UsersList.Where(Function(u) u.IncludedInCollection And Not u.Protected).ToList
                     If cUsers.ListExists Then
-                        Dim d As New Dictionary(Of SFile, List(Of UserInfo))
+                        Dim d As New Dictionary(Of String, List(Of UserInfo))
                         cUsers = cUsers.ListForEachCopy(Of List(Of UserInfo))(Function(ByVal f As UserInfo, ByVal f_indx As Integer) As UserInfo
-                                                                                  Dim m% = IIf(f.Merged, 1, 2)
+                                                                                  Dim m% = IIf(f.Merged Or f.IsVirtual, 1, 2)
                                                                                   If Not f.Protected AndAlso SFile.GetPath(f.File.CutPath(m - 1).Path).Exists(SFO.Path, False) Then
-                                                                                      Dim fp As SFile = SFile.GetPath(f.File.CutPath(m).Path)
+                                                                                      If Not d.ContainsKey(f.CollectionName) Then
-                                                                                      If Not d.ContainsKey(fp) Then
+                                                                                          d.Add(f.CollectionName, New List(Of UserInfo) From {f})
-                                                                                          d.Add(fp, New List(Of UserInfo) From {f})
                                                                                       Else
-                                                                                          d(f.File.CutPath(m).Path).Add(f)
+                                                                                          d(f.CollectionName).Add(f)
                                                                                       End If
                                                                                       Return f
                                                                                   Else
@@ -274,8 +276,8 @@ Friend Class SettingsCLS : Implements IDisposable
                                                                               End Function, True)
                         Dim v%
                         If d.Count > 0 Then
-                            For Each kv In d
+                            For Each kv As KeyValuePair(Of String, List(Of UserInfo)) In d
-                                Users.Add(New UserDataBind(kv.Value(0).CollectionName))
+                                Users.Add(New UserDataBind(kv.Key))
                                 MainFrameObj.CollectionHandler(DirectCast(Users(Users.Count - 1), UserDataBind))
                                 For v = 0 To kv.Value.Count - 1 : DirectCast(Users(Users.Count - 1), UserDataBind).Add(kv.Value(v), False) : Next
                             Next

SCrawler/SiteResources.Designer.vb

@@ -104,6 +104,26 @@ Namespace My.Resources
             End Get
         End Property
         
+        '''<summary>
+        '''  Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
+        '''</summary>
+        Friend Shared ReadOnly Property PornHubIcon_16() As System.Drawing.Icon
+            Get
+                Dim obj As Object = ResourceManager.GetObject("PornHubIcon_16", resourceCulture)
+                Return CType(obj,System.Drawing.Icon)
+            End Get
+        End Property
+        
+        '''<summary>
+        '''  Looks up a localized resource of type System.Drawing.Bitmap.
+        '''</summary>
+        Friend Shared ReadOnly Property PornHubPic_16() As System.Drawing.Bitmap
+            Get
+                Dim obj As Object = ResourceManager.GetObject("PornHubPic_16", resourceCulture)
+                Return CType(obj,System.Drawing.Bitmap)
+            End Get
+        End Property
+        
         '''<summary>
         '''  Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
         '''</summary>
@@ -184,6 +204,26 @@ Namespace My.Resources
             End Get
         End Property
         
+        '''<summary>
+        '''  Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
+        '''</summary>
+        Friend Shared ReadOnly Property XhamsterIcon_32() As System.Drawing.Icon
+            Get
+                Dim obj As Object = ResourceManager.GetObject("XhamsterIcon_32", resourceCulture)
+                Return CType(obj,System.Drawing.Icon)
+            End Get
+        End Property
+        
+        '''<summary>
+        '''  Looks up a localized resource of type System.Drawing.Bitmap.
+        '''</summary>
+        Friend Shared ReadOnly Property XhamsterPic_32() As System.Drawing.Bitmap
+            Get
+                Dim obj As Object = ResourceManager.GetObject("XhamsterPic_32", resourceCulture)
+                Return CType(obj,System.Drawing.Bitmap)
+            End Get
+        End Property
+        
         '''<summary>
         '''  Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
         '''</summary>

SCrawler/SiteResources.resx

@@ -130,6 +130,12 @@
   <data name="LPSGPic_32" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>Content\Pictures\SitePictures\LPSGPic_32.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
   </data>
+  <data name="PornHubIcon_16" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>Content\Icons\SiteIcons\PornHubIcon_16.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
+  </data>
+  <data name="PornHubPic_16" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>Content\Pictures\SitePictures\PornHubPic_16.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
+  </data>
   <data name="RedditIcon_128" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>Content\Icons\SiteIcons\RedditIcon_128.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
   </data>
@@ -154,6 +160,12 @@
   <data name="TwitterPic_400" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>Content\Pictures\SitePictures\TwitterPic_400.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
   </data>
+  <data name="XhamsterIcon_32" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>Content\Icons\SiteIcons\XhamsterIcon_32.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
+  </data>
+  <data name="XhamsterPic_32" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>Content\Pictures\SitePictures\XhamsterPic_32.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
+  </data>
   <data name="XvideosIcon_48" type="System.Resources.ResXFileRef, System.Windows.Forms">
     <value>Content\Icons\SiteIcons\XvideosIcon_48.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
   </data>

SCrawler/UserFinder.vb

@@ -0,0 +1,335 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.Messaging
+Imports PersonalUtilities.Forms
+Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
+Friend Class UserFinder : Implements IDisposable
+    Private Structure SkippedUser
+        Friend User As UserInfo
+        Friend Reason As String
+    End Structure
+    Private ReadOnly Paths As List(Of SFile)
+    Private ReadOnly FoundUsers As List(Of UserInfo)
+    Private ReadOnly Added As List(Of UserInfo)
+    Private ReadOnly Skipped As List(Of SkippedUser)
+    Private ReadOnly Duplicates As List(Of UserInfo)
+    Private ReadOnly IgnoredCollections As List(Of String)
+    Private ReadOnly NotRecognized As List(Of SFile)
+    Private OriginalLocations As Boolean = False
+    Private PathStr As String
+    Private Const LabelImported As String = "Imported"
+    Private ReadOnly Labels As List(Of String)
+    Friend ReadOnly Property Count As Integer
+        Get
+            Return FoundUsers.Count
+        End Get
+    End Property
+    Friend Sub New(ByVal Path As SFile)
+        Paths = New List(Of SFile) From {Path}
+        PathStr = vbCr & Path.ToString
+        FoundUsers = New List(Of UserInfo)
+        Added = New List(Of UserInfo)
+        Skipped = New List(Of SkippedUser)
+        Duplicates = New List(Of UserInfo)
+        IgnoredCollections = New List(Of String)
+        NotRecognized = New List(Of SFile)
+        Labels = New List(Of String)
+    End Sub
+    Private Function GetFiles() As List(Of SFile)
+        Dim files As New List(Of SFile)
+        If Paths.Count > 0 Then
+            For Each path As SFile In Paths
+                files.ListAddList(SFile.GetFiles(path, "User_*.xml", IO.SearchOption.AllDirectories, EDP.ReturnValue), LAP.NotContainsOnly)
+            Next
+        End If
+        Return files
+    End Function
+    Friend Function Find(ByVal OriginalLocations As Boolean) As Boolean
+        Try
+            Me.OriginalLocations = OriginalLocations
+            If OriginalLocations Then
+                Paths.Clear()
+                PathStr = String.Empty
+                Paths.ListAddList(Settings.Plugins.Select(Function(p) p.Settings.Path), LAP.NotContainsOnly)
+                Paths.ListAddValue(Settings.CollectionsPathF, LAP.NotContainsOnly)
+                PathStr = vbCr & Paths.ListToString(vbCr)
+            End If
+            FoundUsers.Clear()
+            If Paths.Count > 0 Then
+                Dim files As List(Of SFile) = GetFiles()
+                If files.ListExists Then files.RemoveAll(Function(ff) ff.Name.EndsWith("_Data"))
+                If files.ListExists Then
+                    Dim x As XmlFile
+                    Dim xErr As New ErrorsDescriber(EDP.None)
+                    Dim u As UserInfo
+                    For Each f As SFile In files
+                        x = New XmlFile(f, Protector.Modes.All, False) With {.XmlReadOnly = True}
+                        x.LoadData(xErr)
+                        If Not x.HasError And x.Count > 0 Then
+                            u = New UserInfo With {
+                                .Name = x.Value(UserDataBase.Name_UserName),
+                                .Site = x.Value(UserInfo.Name_Site),
+                                .Plugin = x.Value(UserInfo.Name_Plugin),
+                                .File = f,
+                                .SpecialPath = x.Value(UserInfo.Name_SpecialPath),
+                                .SpecialCollectionPath = x.Value(UserInfo.Name_SpecialCollectionPath),
+                                .UserModel = x.Value(UserInfo.Name_Model_User).FromXML(Of Integer)(UsageModel.Default),
+                                .CollectionModel = x.Value(UserInfo.Name_Model_Collection).FromXML(Of Integer)(UsageModel.Default),
+                                .CollectionName = x.Value(UserInfo.Name_Collection),
+                                .IsChannel = x.Value(UserInfo.Name_IsChannel).FromXML(Of Boolean)(False)
+                            }
+                            'TODELETE: UserFinder remove old 'merge' constant
+#Disable Warning BC40000
+                            If x.Contains(UserDataBase.Name_DataMerging) Then
+                                u.Merged = x.Value(UserDataBase.Name_DataMerging).FromXML(Of Boolean)(False)
+                            Else
+                                u.Merged = x.Value(UserInfo.Name_Merged).FromXML(Of Boolean)(False)
+                            End If
+#Enable Warning
+                            FoundUsers.Add(u)
+                        Else
+                            If x.HasError Then NotRecognized.Add(f)
+                        End If
+                        x.Dispose()
+                    Next
+                End If
+            End If
+            Return Count > 0
+        Catch ex As Exception
+            Return ErrorsDescriber.Execute(EDP.LogMessageValue, ex, $"[UserFinder.Find:{PathStr}]", False)
+        End Try
+    End Function
+    Friend Sub Verify()
+        Try
+            Added.Clear()
+            Skipped.Clear()
+            Duplicates.Clear()
+            IgnoredCollections.Clear()
+            If Count > 0 Then
+                Dim u As UserInfo
+                Dim s As Plugin.Hosts.SettingsHost
+                Dim pIndx%
+                For i% = 0 To Count - 1
+                    u = FoundUsers(i)
+                    s = Nothing
+                    If u.Plugin.IsEmptyString Then
+                        pIndx = Settings.Plugins.FindIndex(Function(pp) pp.Name.ToLower = u.Site.ToLower)
+                        If pIndx >= 0 Then s = Settings.Plugins(pIndx).Settings
+                    Else
+                        s = Settings(u.Plugin)
+                    End If
+                    If Not s Is Nothing Then
+                        u.Plugin = s.Key
+                        If Not OriginalLocations Then
+                            If u.IncludedInCollection And u.UserModel = UsageModel.Default Then
+                                u.SpecialCollectionPath = u.File.CutPath(IIf(u.Merged, 1, 2)).Path.CSFileP
+                            Else
+                                u.SpecialPath = u.File.CutPath(1).Path.CSFileP
+                            End If
+                        End If
+                        u.UpdateUserFile()
+                        If Settings.UsersList.Contains(u) Then
+                            Duplicates.Add(u)
+                        ElseIf u.File.Exists And (u.CollectionName.IsEmptyString OrElse
+                                                  IgnoredCollections.Contains(u.CollectionName.ToLower) OrElse
+                                                  Not Settings.UsersList.Exists(Function(uu) uu.CollectionName.StringToLower = u.CollectionName.ToLower)) Then
+                            Added.Add(u)
+                            If Not IgnoredCollections.Contains(u.CollectionName) Then IgnoredCollections.Add(u.CollectionName)
+                        Else
+                            Skipped.Add(New SkippedUser With {.User = u, .Reason = "file path generation / collection exists"})
+                        End If
+                    Else
+                        Skipped.Add(New SkippedUser With {.User = u, .Reason = "user plugin not recognized"})
+                    End If
+                Next
+            End If
+        Catch ex As Exception
+            ErrorsDescriber.Execute(EDP.LogMessageValue, ex, $"[UserFinder.Verify:{PathStr}]")
+        End Try
+    End Sub
+    Friend Function Dialog() As Boolean
+        Const MsgTitle$ = "Import users"
+        Const DesignNode$ = "ImportUserSelector"
+        Try
+            Dim uStr As Func(Of UserInfo, String) = Function(u) $"{IIf(u.CollectionName.IsEmptyString, String.Empty, $"[{u.CollectionName}]: ")}{u.Site} - {u.Name}"
+            Dim uc As Comparison(Of UserInfo) = Function(ByVal x As UserInfo, ByVal y As UserInfo) As Integer
+                                                    If Not x.CollectionName.IsEmptyString And Not y.CollectionName.IsEmptyString Then
+                                                        Return x.CollectionName.CompareTo(y.CollectionName)
+                                                    ElseIf Not x.CollectionName.IsEmptyString Then
+                                                        Return -1
+                                                    ElseIf Not y.CollectionName.IsEmptyString Then
+                                                        Return 1
+                                                    Else
+                                                        Return uStr(x).CompareTo(uStr(y))
+                                                    End If
+                                                End Function
+            Dim __added$ = String.Empty
+            Dim __dup$ = String.Empty
+            Dim __skipped$ = String.Empty
+            Dim __labelText$
+            If Added.Count > 0 Then Added.Sort(uc) : __added = $"The following users will be added to SCrawler:{vbCr}{Added.Select(uStr).ListToString(vbCr)}"
+            If Duplicates.Count > 0 Then Duplicates.Sort(uc) : __dup = $"The following users already exist In SCrawler and will not be added:{vbCr}{Duplicates.Select(uStr).ListToString(vbCr)}"
+            If Skipped.Count > 0 Then
+                Skipped.Sort(Function(x, y) uc(x.User, y.User))
+                __skipped = $"The following users will not be added to SCrawler{vbCr}{Skipped.Select(Function(u) $"{uStr(u.User)} ({u.Reason})").ListToString(vbCr)}"
+            End If
+            __added = {__added, __dup, __skipped}.ListToString(vbCr.StringDup(2))
+            If Not __added.IsEmptyString Then
+                Using t As New TextSaver("LOGs\ImportUsers.txt") With {.ForceAddDateTimeToFileName = True}
+                    t.Append(__added)
+                    If Added.Count > 0 Then
+                        t.AppendLine(vbNewLine.StringDup(2))
+                        t.AppendLine($"Added:{vbNewLine}{Added.Select(Function(u) u.File.ToString).ListToString(vbNewLine)}")
+                    End If
+                    If Duplicates.Count > 0 Then
+                        t.AppendLine(vbNewLine.StringDup(2))
+                        t.AppendLine($"Duplicates:{vbNewLine}{Duplicates.Select(Function(u) u.File.ToString).ListToString(vbNewLine)}")
+                    End If
+                    If Skipped.Count > 0 Then
+                        t.AppendLine(vbNewLine.StringDup(2))
+                        t.AppendLine($"Duplicates:{vbNewLine}{Skipped.Select(Function(u) u.User.File.ToString).ListToString(vbNewLine)}")
+                    End If
+                    If NotRecognized.Count > 0 Then
+                        t.AppendLine(vbNewLine.StringDup(2))
+                        t.AppendLine($"Not recognized:{vbNewLine}{NotRecognized.ListToString(vbNewLine)}")
+                    End If
+                    t.Save()
+                End Using
+                Dim msg As New MMessage(__added, MsgTitle,, vbQuestion) With {.Editable = True}
+                Dim BttSelect As New MsgBoxButton("Select", "Select users to import") With {
+                    .IsDialogResultButton = False,
+                    .CallBack = Sub(r, m, b)
+                                    If Not Settings.Design.Contains(DesignNode) Then Settings.Design.Add(DesignNode, String.Empty)
+                                    Using f As New SimpleListForm(Of UserInfo)(Added, Settings.Design(DesignNode)) With {
+                                        .Icon = My.Resources.UsersIcon_32,
+                                        .FormText = MsgTitle,
+                                        .Mode = SimpleListFormModes.CheckedItemsAutoCheckAll,
+                                        .ButtonInsertKey = Nothing,
+                                        .Provider = New CustomProvider(Function(v, d, p, n, e) uStr(v))
+                                    }
+                                        If f.ShowDialog() = DialogResult.OK Then
+                                            Added.Clear()
+                                            Added.ListAddList(f.DataResult)
+                                        End If
+                                    End Using
+                                End Sub}
+                msg.Buttons = If(Added.Count > 0, {New MsgBoxButton("Process"), BttSelect, New MsgBoxButton("Cancel")}, Nothing)
+                If MsgBoxE(msg) = 0 Then
+                    If Added.Count > 0 Then
+                        Add()
+                        If Labels.Count = 0 Then
+                            __labelText = String.Empty
+                        ElseIf Labels.Count = 1 Then
+                            __labelText = $"{vbCr}{vbCr}The '{Labels(0)}' label has been added to each user."
+                        Else
+                            __labelText = $"{vbCr}{vbCr}The following labels have been added to each user: {Labels.ListToString}."
+                        End If
+                        MsgBoxE(New MMessage($"Restart SCrawler to take effect.{__labelText}{vbCr}{vbCr}" &
+                                             $"The following users have been added to SCrawler:{vbCr}" &
+                                             Added.Select(uStr).ListToString(vbCr), MsgTitle) With {.Editable = True})
+                        Return True
+                    End If
+                Else
+                    If Added.Count > 0 Then MsgBoxE({"Operation canceled", MsgTitle})
+                End If
+            Else
+                MsgBoxE({"No users found", MsgTitle})
+            End If
+            Return False
+        Catch ex As Exception
+            Return ErrorsDescriber.Execute(EDP.LogMessageValue, ex, $"[UserFinder.Dialog:{PathStr}]", False)
+        End Try
+    End Function
+    Friend Sub Add()
+        Try
+            Labels.Clear()
+            Select Case MsgBoxE({"Do you want to add an 'Imported' label to each user?", "User labels"}, vbQuestion,,,
+                                {"Yes", New MsgBoxButton("Select", "Select labels to add"), "No"}).Index
+                Case 0 : Labels.Add(LabelImported)
+                Case 1
+                    Labels.ListAddList(GetLabels())
+                    If Labels.Count = 0 AndAlso MsgBoxE({"You have not selected any labels." &
+                                                         "Do you want to add an 'Imported' label to each user?", "User labels"},
+                                                        vbExclamation + vbYesNo) = vbYes Then Labels.Add(LabelImported)
+            End Select
+            If Labels.Count > 0 Then
+                Dim x As XmlFile
+                Dim l As List(Of String)
+                Dim lp As New ListAddParams(LAP.NotContainsOnly)
+                For Each u As UserInfo In Added
+                    x = New XmlFile(u.File, Protector.Modes.All)
+                    l = x.Value(UserDataBase.Name_LabelsName).StringToList(Of String, List(Of String))("|", EDP.ReturnValue)
+                    If Not l.ListExists Then l = New List(Of String)
+                    l.ListAddList(Labels, lp)
+                    x.Value(UserDataBase.Name_LabelsName) = l.ListToString("|")
+                    x.UpdateData()
+                    x.Dispose()
+                Next
+            End If
+            Settings.UsersList.AddRange(Added)
+            Settings.UpdateUsersList()
+        Catch ex As Exception
+            ErrorsDescriber.Execute(EDP.LogMessageValue, ex, $"[UserFinder.Add:{PathStr}]")
+        End Try
+    End Sub
+    Private Function GetLabels() As List(Of String)
+        Const DesignNode$ = "ImportUserSelectorLabels"
+        Try
+            Dim __add As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e) e.ValueNew = InputBoxE("Enter a new label name", "New label").IfNullOrEmptyE(Nothing)
+            Dim l As List(Of String) = ListAddList(Nothing, Settings.Labels, LAP.NotContainsOnly).ListAddValue(LabelImported, LAP.NotContainsOnly)
+            If l.Count > 0 Then l.Sort()
+            If Not Settings.Design.Contains(DesignNode) Then Settings.Design.Add(DesignNode, String.Empty)
+            Using f As New SimpleListForm(Of String)(l, Settings.Design(DesignNode)) With {
+                .Icon = My.Resources.TagIcon_32,
+                .FormText = "Labels for imported users",
+                .Mode = SimpleListFormModes.CheckedItems,
+                .Buttons = {ADB.Add}
+            }
+                f.DataSelected.Add(LabelImported)
+                AddHandler f.AddClick, __add
+                If f.ShowDialog() = DialogResult.OK Then
+                    l.Clear()
+                    l.ListAddList(f.DataResult, LAP.NotContainsOnly)
+                    Return l
+                End If
+            End Using
+            Return Nothing
+        Catch ex As Exception
+            Return ErrorsDescriber.Execute(EDP.LogMessageValue, ex, $"[UserFinder.GetLabels:{PathStr}]")
+        End Try
+    End Function
+#Region "IDisposable Support"
+    Private disposedValue As Boolean = False
+    Protected Overridable Sub Dispose(ByVal disposing As Boolean)
+        If Not disposedValue Then
+            If disposing Then
+                Paths.Clear()
+                FoundUsers.Clear()
+                Added.Clear()
+                Skipped.Clear()
+                Duplicates.Clear()
+                IgnoredCollections.Clear()
+                NotRecognized.Clear()
+                Labels.Clear()
+            End If
+            disposedValue = True
+        End If
+    End Sub
+    Protected Overrides Sub Finalize()
+        Dispose(False)
+        MyBase.Finalize()
+    End Sub
+    Friend Sub Dispose() Implements IDisposable.Dispose
+        Dispose(True)
+        GC.SuppressFinalize(Me)
+    End Sub
+#End Region
+End Class