2023.4.28.0

Plugins
IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function
Add IDownloadableMedia interface
Removed 'Channel' option from all functions and enums
ISiteSettings: added GetSingleMediaInstance function
ExchangeOptions: removed 'IsChannel'
UserMediaTypes: added Audio and AudioPre enums
IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums

SCrawler
Add YouTube standalone downloader
Add gallery-dl & yt-dlp support
Remove 'UserInfo' requirement from 'ProfilesSaved'
Update 'SiteSettingsBase' to use domains and Netscape cookies
UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download

Rewritten DomainsContainer
Create a universal settings form and PSettingsArttribute
Gfycat, Imgur: turn these classes into IUserData to download a single object

All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist

LPSG: fix attachments; update converters and regex
Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music
Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support
Twitter: fixed gif path bug; fixed downloading saved posts
PornHub: hide unnecessary errors; photo galleries bug
RedGifs: add 'UserAgent' option

Added icons to download progress

Rename some objects
Completely redesigned standalone downloader form and rewritten its environment
WebClient2: update to use tokens

Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser'

UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found
Remove temporary 'EncryptCookies' module

MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader)
PropertyValueHost: update for option forms compatibility
SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork
UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function
Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler
UserFinder: remove old 'Merge' const; remove channel option
UserInfo: remove channel option

SCrawler/API/Base/Declarations.vb

@@ -8,7 +8,11 @@
 ' but WITHOUT ANY WARRANTY
 Namespace API.Base
     Friend Module Declarations
+        Friend Const UserLabelName As String = "User"
         Friend ReadOnly LNC As New ListAddParams(LAP.NotContainsOnly)
+        Friend ReadOnly UnixDate32Provider As New ADateTime(ADateTime.Formats.Unix32)
+        Friend ReadOnly UnixDate64Provider As New ADateTime(ADateTime.Formats.Unix64)
+        Friend ReadOnly HtmlConverter As Func(Of String, String) = Function(Input) SymbolsConverter.HTML.Decode(Input, EDP.ReturnValue)
         Friend ReadOnly TitleHtmlConverter As Func(Of String, String) =
                Function(Input) SymbolsConverter.HTML.Decode(SymbolsConverter.Convert(Input, EDP.ReturnValue), EDP.ReturnValue).
                                StringRemoveWinForbiddenSymbols().StringTrim()

SCrawler/API/Base/DownDetector.vb

@@ -60,7 +60,7 @@ Namespace API.Base
                 End Using
                 Return l2
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[DownDetector.GetData({Site})]")
+                Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, $"[DownDetector.GetData({Site})]")
             End Try
         End Function
     End Class

SCrawler/API/Base/GDLBatch.vb

@@ -0,0 +1,86 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Tools
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.Base.GDL
+    Friend Module Declarations
+        Private Structure GDLURL : Implements IRegExCreator
+            Private _URL As String
+            Friend ReadOnly Property URL As String
+                Get
+                    Return _URL
+                End Get
+            End Property
+            Public Shared Widening Operator CType(ByVal u As String) As GDLURL
+                Return New GDLURL With {._URL = u}
+            End Operator
+            Public Shared Widening Operator CType(ByVal u As GDLURL) As String
+                Return u.URL
+            End Operator
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    Dim u$ = ParamsArray(0).StringTrim.StringTrimEnd("/"), u2$
+                    If Not u.IsEmptyString Then
+                        u2 = ParamsArray(1).StringTrim
+                        If Not u2.IsEmptyString AndAlso u2.StartsWith("GET", StringComparison.OrdinalIgnoreCase) Then
+                            u2 = u2.Remove(0, 3).StringTrim.StringTrimStart("/")
+                            If Not u2.IsEmptyString Then _URL = $"{u}/{u2}"
+                        End If
+                    End If
+                End If
+                Return Me
+            End Function
+            Public Shared Operator =(ByVal x As GDLURL, ByVal y As GDLURL) As Boolean
+                Return x.URL = y.URL
+            End Operator
+            Public Shared Operator <>(ByVal x As GDLURL, ByVal y As GDLURL) As Boolean
+                Return Not x.URL = y.URL
+            End Operator
+            Public Overrides Function ToString() As String
+                Return URL
+            End Function
+            Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+                Return URL = CType(Obj, String)
+            End Function
+        End Structure
+        Private ReadOnly Property GdlUrlPattern As RParams = RParams.DM(GDLBatch.UrlLibStart.Replace("[", "\[").Replace("]", "\]") &
+                                                                        "([^""]+?)""(GET [^""]+)""", 0, EDP.ReturnValue)
+        Friend Function GetUrlsFromGalleryDl(ByVal Batch As BatchExecutor, ByVal Command As String) As List(Of String)
+            Dim urls As New List(Of String)
+            Dim u As GDLURL
+            With Batch
+                .Execute(Command)
+                If .ErrorOutputData.Count > 0 Then
+                    For Each eValue$ In .ErrorOutputData
+                        u = RegexFields(Of GDLURL)(eValue, {GdlUrlPattern}, {1, 2}, EDP.ReturnValue).ListIfNothing.FirstOrDefault
+                        If Not u.URL.IsEmptyString Then urls.ListAddValue(u, LNC)
+                    Next
+                End If
+            End With
+            Return urls
+        End Function
+    End Module
+    Friend Class GDLBatch : Inherits BatchExecutor
+        Friend Property TempPostsList As List(Of String)
+        Friend Const UrlLibStart As String = "[urllib3.connectionpool][debug]"
+        Friend Const UrlTextStart As String = UrlLibStart & " https"
+        Friend Sub New()
+            MyBase.New(True)
+            ChangeDirectory(Settings.GalleryDLFile.File)
+        End Sub
+        Protected Overrides Async Sub OutputDataReceiver(ByVal Sender As Object, ByVal e As DataReceivedEventArgs)
+            MyBase.OutputDataReceiver(Sender, e)
+            Await Validate(e.Data)
+        End Sub
+        Protected Overridable Async Function Validate(ByVal Value As String) As Task
+            If Await Task.Run(Of Boolean)(Function() Not Value.IsEmptyString AndAlso
+                                                     TempPostsList.Exists(Function(v) Value.Contains(v))) Then Kill(EDP.None)
+        End Function
+    End Class
+End Namespace

SCrawler/API/Base/M3U8Base.vb

@@ -6,9 +6,12 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.RegularExpressions
+Imports System.Threading
+Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Web
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Forms.Toolbars
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Base
     Namespace M3U8Declarations
         Friend Module M3U8Defaults
@@ -16,6 +19,7 @@ Namespace API.Base
         End Module
     End Namespace
     Friend NotInheritable Class M3U8Base
+        Friend Const TempCacheFolderName As String = "tmpCache"
         Private Sub New()
         End Sub
         Friend Shared Function CreateUrl(ByVal Appender As String, ByVal File As String) As String
@@ -28,36 +32,40 @@ Namespace API.Base
                 Return $"{Appender.StringTrimEnd("/")}/{File}"
             End If
         End Function
-        Friend Shared Function Download(ByVal URLs As List(Of String), ByVal DestinationFile As SFile, Optional ByVal Responser As Responser = Nothing) As SFile
-            Dim CachePath As SFile = Nothing
+        Friend Shared Function Download(ByVal URLs As List(Of String), ByVal DestinationFile As SFile, Optional ByVal Responser As Responser = Nothing,
+                                        Optional ByVal Token As CancellationToken = Nothing, Optional ByVal Progress As MyProgress = Nothing) As SFile
+            Dim Cache As CacheKeeper = Nothing
             Try
                 If URLs.ListExists Then
                     Dim ConcatFile As SFile = DestinationFile
                     If ConcatFile.Name.IsEmptyString Then ConcatFile.Name = "PlayListFile"
                     ConcatFile.Extension = "mp4"
-                    CachePath = $"{DestinationFile.PathWithSeparator}_Cache\{SFile.GetDirectories($"{DestinationFile.PathWithSeparator}_Cache\",,, EDP.ReturnValue).ListIfNothing.Count + 1}\"
-                    If CachePath.Exists(SFO.Path) Then
-                        Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
-                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ReturnValue)
+                    Cache = New CacheKeeper($"{DestinationFile.PathWithSeparator}_{TempCacheFolderName}\")
+                    Dim cache2 As CacheKeeper = Cache.NewInstance
+                    If cache2.RootDirectory.Exists(SFO.Path) Then
+                        Dim progressExists As Boolean = Not Progress Is Nothing
+                        If progressExists Then Progress.Maximum += URLs.Count
+                        Dim p As SFileNumbers = SFileNumbers.Default(ConcatFile.Name)
+                        ConcatFile = SFile.IndexReindex(ConcatFile,,, p, EDP.ReturnValue)
                         Dim i%
-                        Dim eFiles As New List(Of SFile)
-                        Dim dFile As SFile = CachePath
+                        Dim dFile As SFile = cache2.RootDirectory
                         dFile.Extension = "ts"
                         Using w As New DownloadObjects.WebClient2(Responser)
                             For i = 0 To URLs.Count - 1
+                                If progressExists Then Progress.Perform()
+                                Token.ThrowIfCancellationRequested()
                                 dFile.Name = $"ConPart_{i}"
                                 w.DownloadFile(URLs(i), dFile)
-                                eFiles.Add(dFile)
+                                cache2.AddFile(dFile, True)
                             Next
                         End Using
-                        DestinationFile = FFMPEG.ConcatenateFiles(eFiles, Settings.FfmpegFile, ConcatFile, p, EDP.ThrowException)
-                        eFiles.Clear()
+                        DestinationFile = FFMPEG.ConcatenateFiles(cache2, Settings.FfmpegFile.File, ConcatFile, Settings.CMDEncoding, p, EDP.ThrowException)
                         Return DestinationFile
                     End If
                 End If
                 Return Nothing
             Finally
-                CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
+                Cache.DisposeIfReady
             End Try
         End Function
     End Class

SCrawler/API/Base/ProfileSaved.vb

@@ -18,20 +18,17 @@ Namespace API.Base
             HOST = h
             Progress = Bar
         End Sub
-        Friend Sub Download(ByVal Token As CancellationToken)
+        Friend Sub Download(ByVal Token As CancellationToken, ByVal Multiple As Boolean)
             Try
                 If HOST.Source.ReadyToDownload(PDownload.SavedPosts) Then
-                    If HOST.Available(PDownload.SavedPosts, False) Then
+                    If HOST.Available(PDownload.SavedPosts, Multiple) Then
                         HOST.DownloadStarted(PDownload.SavedPosts)
                         Dim u As New UserInfo With {.Plugin = HOST.Key, .Site = HOST.Name, .SpecialPath = HOST.SavedPostsPath}
                         Using user As IUserData = HOST.GetInstance(PDownload.SavedPosts, Nothing, False, False)
-                            If Not user Is Nothing AndAlso Not user.Name.IsEmptyString Then
-                                u.Name = user.Name
+                            If Not user Is Nothing Then
                                 With DirectCast(user, UserDataBase)
-                                    With .User : u.IsChannel = .IsChannel : u.UpdateUserFile() : End With
-                                    .User = u
-                                    .LoadUserInformation()
                                     .IsSavedPosts = True
+                                    .LoadUserInformation()
                                     .Progress = Progress
                                     If Not .FileExists Then .UpdateUserInformation()
                                 End With
@@ -49,7 +46,7 @@ Namespace API.Base
                 End If
             Catch ex As Exception
                 Progress.InformationTemporary = $"{HOST.Name} downloading error"
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Base.ProfileSaved.Download({HOST.Key})]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[API.Base.ProfileSaved.Download({HOST.Key})]")
             Finally
                 HOST.DownloadDone(PDownload.SavedPosts)
                 MainFrameObj.UpdateLogButton()

SCrawler/API/Base/SiteSettingsBase.vb

@@ -6,10 +6,9 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.RegularExpressions
-Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.Web.Cookies
 Imports SCrawler.Plugin
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Functions.RegularExpressions
 Imports Download = SCrawler.Plugin.ISiteSettings.Download
 Namespace API.Base
     Friend MustInherit Class SiteSettingsBase : Implements ISiteSettings, IResponserContainer
@@ -18,6 +17,23 @@ Namespace API.Base
         Friend Overridable ReadOnly Property Image As Image Implements ISiteSettings.Image
         Private Property Logger As ILogProvider = LogConnector Implements ISiteSettings.Logger
         Friend Overridable ReadOnly Property Responser As Responser
+        Friend ReadOnly Property CookiesNetscapeFile As SFile
+        Protected CheckNetscapeCookiesOnEndInit As Boolean = False
+        Private _UseNetscapeCookies As Boolean = False
+        Protected Property UseNetscapeCookies As Boolean
+            Get
+                Return _UseNetscapeCookies
+            End Get
+            Set(ByVal use As Boolean)
+                Dim b As Boolean = Not _UseNetscapeCookies = use
+                _UseNetscapeCookies = use
+                If Not Responser Is Nothing Then
+                    Responser.Cookies.ChangedAllowInternalDrop = Not _UseNetscapeCookies
+                    Responser.Cookies.Changed = False
+                End If
+                If b And _UseNetscapeCookies Then Update_SaveCookiesNetscape()
+            End Set
+        End Property
         Private Property IResponserContainer_Responser As Responser Implements IResponserContainer.Responser
             Get
                 Return Responser
@@ -27,20 +43,15 @@ Namespace API.Base
         Friend MustOverride Function GetInstance(ByVal What As Download) As IPluginContentProvider Implements ISiteSettings.GetInstance
         Friend Sub New(ByVal SiteName As String)
             Site = SiteName
+            CookiesNetscapeFile = $"{SettingsFolderName}\Responser_{Site}_Cookies_Netscape.txt"
         End Sub
         Friend Sub New(ByVal SiteName As String, ByVal CookiesDomain As String)
-            Site = SiteName
-            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml")
+            Me.New(SiteName)
+            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml") With {.DeclaredError = EDP.ThrowException}
             With Responser
-                If .File.Exists Then
-                    If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
-                    .LoadSettings()
-                Else
-                    .CookiesDomain = CookiesDomain
-                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
-                    .SaveSettings()
-                End If
-                If .CookiesDomain.IsEmptyString Then .CookiesDomain = CookiesDomain
+                .CookiesDomain = CookiesDomain
+                .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                If .File.Exists Then .LoadSettings() Else .SaveSettings()
             End With
         End Sub
 #Region "XML"
@@ -51,17 +62,47 @@ Namespace API.Base
         Friend Overridable Sub BeginInit() Implements ISiteSettings.BeginInit
         End Sub
         Friend Overridable Sub EndInit() Implements ISiteSettings.EndInit
-            EncryptCookies.ValidateCookiesEncrypt(Responser)
             If Not DefaultUserAgent.IsEmptyString And Not Responser Is Nothing Then Responser.UserAgent = DefaultUserAgent
+            If CheckNetscapeCookiesOnEndInit Then Update_SaveCookiesNetscape(, True)
         End Sub
+#End Region
+#Region "Update, Edit"
         Friend Overridable Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
         End Sub
         Friend Overridable Sub EndUpdate() Implements ISiteSettings.EndUpdate
         End Sub
+        Protected _SiteEditorFormOpened As Boolean = False
         Friend Overridable Sub BeginEdit() Implements ISiteSettings.BeginEdit
+            _SiteEditorFormOpened = True
         End Sub
         Friend Overridable Sub EndEdit() Implements ISiteSettings.EndEdit
+            If _SiteEditorFormOpened Then DomainsReset()
+            _SiteEditorFormOpened = False
         End Sub
+        Friend Overridable Sub Update() Implements ISiteSettings.Update
+            If _SiteEditorFormOpened Then
+                If UseNetscapeCookies Then Update_SaveCookiesNetscape()
+                DomainsApply()
+            End If
+            If Not Responser Is Nothing Then Responser.SaveSettings()
+        End Sub
+        Protected Sub Update_SaveCookiesNetscape(Optional ByVal Force As Boolean = False, Optional ByVal IsInit As Boolean = False)
+            If Not Responser Is Nothing Then
+                With Responser
+                    If .Cookies.Changed Or Force Or IsInit Then
+                        If IsInit And CookiesNetscapeFile.Exists Then Exit Sub
+                        If .CookiesExists Then .Cookies.SaveNetscapeFile(CookiesNetscapeFile) Else CookiesNetscapeFile.Delete()
+                        .Cookies.Changed = False
+                    End If
+                End With
+            End If
+        End Sub
+#Region "Specialized"
+        Protected Overridable Sub DomainsApply()
+        End Sub
+        Protected Overridable Sub DomainsReset()
+        End Sub
+#End Region
 #End Region
 #Region "Before and After Download"
         Friend Overridable Sub DownloadStarted(ByVal What As Download) Implements ISiteSettings.DownloadStarted
@@ -75,20 +116,15 @@ Namespace API.Base
 #End Region
 #Region "User info"
         Protected UrlPatternUser As String = String.Empty
-        Protected UrlPatternChannel As String = String.Empty
-        Friend Overridable Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String Implements ISiteSettings.GetUserUrl
-            If Channel Then
-                If Not UrlPatternChannel.IsEmptyString Then Return String.Format(UrlPatternChannel, User.Name)
-            Else
-                If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, User.Name)
-            End If
+        Friend Overridable Function GetUserUrl(ByVal User As IPluginContentProvider) As String Implements ISiteSettings.GetUserUrl
+            If Not UrlPatternUser.IsEmptyString Then Return String.Format(UrlPatternUser, User.Name)
             Return String.Empty
         End Function
         Private Function ISiteSettings_GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String Implements ISiteSettings.GetUserPostUrl
             Return GetUserPostUrl(User, Media)
         End Function
         Friend Overridable Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return String.Empty
+            Return Media.URL_BASE.IfNullOrEmpty(Media.URL)
         End Function
         Protected UserRegex As RParams = Nothing
         Friend Overridable Function IsMyUser(ByVal UserURL As String) As ExchangeOptions Implements ISiteSettings.IsMyUser
@@ -99,43 +135,40 @@ Namespace API.Base
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]", New ExchangeOptions)
+                Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, $"[API.Base.SiteSettingsBase.IsMyUser({UserURL})]", New ExchangeOptions)
             End Try
         End Function
         Protected ImageVideoContains As String = String.Empty
         Friend Overridable Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions Implements ISiteSettings.IsMyImageVideo
             If Not ImageVideoContains.IsEmptyString AndAlso URL.Contains(ImageVideoContains) Then
-                Return New ExchangeOptions With {.Exists = True}
+                Return New ExchangeOptions(Site, String.Empty) With {.Exists = True}
             Else
                 Return Nothing
             End If
         End Function
-        Friend Overridable Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable Implements ISiteSettings.GetSpecialData
-            Return Nothing
+        Private Function ISiteSettings_GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As String) As IDownloadableMedia Implements ISiteSettings.GetSingleMediaInstance
+            Return GetSingleMediaInstance(URL, OutputFile)
         End Function
-        Friend Shared Function GetSpecialDataFile(ByVal Path As String, ByVal AskForPath As Boolean, ByRef SpecFolderObj As String) As SFile
-            Dim f As SFile = Path.CSFileP
-            If f.Name.IsEmptyString Then f.Name = "OutputFile"
-#Disable Warning BC40000
-            If Path.CSFileP.IsEmptyString Or AskForPath Then f = SFile.SaveAs(f, "File destination",,,, EDP.ReturnValue) : SpecFolderObj = f.Path
-#Enable Warning
-            Return f
+        Friend Overridable Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As SFile) As IDownloadableMedia
+            Return New Hosts.DownloadableMediaHost(URL, OutputFile)
         End Function
 #End Region
 #Region "Ready, Available"
+        ''' <returns>True</returns>
         Friend Overridable Function BaseAuthExists() As Boolean
             Return True
         End Function
+        ''' <summary>JOB: leave or remove</summary>
+        ''' <returns>Return BaseAuthExists()</returns>
         Friend Overridable Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
             Return BaseAuthExists()
         End Function
+        ''' <summary>'DownloadData': before processing</summary>
+        ''' <returns>True</returns>
         Friend Overridable Function ReadyToDownload(ByVal What As Download) As Boolean Implements ISiteSettings.ReadyToDownload
             Return True
         End Function
 #End Region
-        Friend Overridable Sub Update() Implements ISiteSettings.Update
-            If Not Responser Is Nothing Then Responser.SaveSettings()
-        End Sub
         Friend Overridable Sub Reset() Implements ISiteSettings.Reset
         End Sub
         Friend Overridable Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean) Implements ISiteSettings.UserOptions

SCrawler/API/Base/Structures.vb

@@ -27,10 +27,12 @@ Namespace API.Base
 #End Region
             Friend Enum Types As Integer
                 Undefined = 0
-                [Picture] = 1
-                [Video] = 2
-                [Text] = 3
+                Picture = 1
+                Video = 2
+                Audio = 200
+                Text = 4
                 VideoPre = 10
+                AudioPre = 215
                 GIF = 50
                 m3u8 = 100
             End Enum
@@ -51,12 +53,12 @@ Namespace API.Base
             Friend SpecialFolder As String
             Friend [Object] As Object
 #Region "Interface Support"
-            Private Property IUserMedia_Type As Integer Implements IUserMedia.ContentType
+            Private Property IUserMedia_Type As UserMediaTypes Implements IUserMedia.ContentType
                 Get
-                    Return Type
+                    Return CInt(Type)
                 End Get
-                Set(ByVal Type As Integer)
-                    Me.Type = Type
+                Set(ByVal Type As UserMediaTypes)
+                    Me.Type = CInt(Type)
                 End Set
             End Property
             Private Property IUserMedia_URL_BASE As String Implements IUserMedia.URL_BASE
@@ -91,12 +93,12 @@ Namespace API.Base
                     Me.File = File
                 End Set
             End Property
-            Private Property IUserMedia_State As Integer Implements IUserMedia.DownloadState
+            Private Property IUserMedia_State As UserMediaStates Implements IUserMedia.DownloadState
                 Get
-                    Return State
+                    Return CInt(State)
                 End Get
-                Set(ByVal State As Integer)
-                    Me.State = State
+                Set(ByVal State As UserMediaStates)
+                    Me.State = CInt(State)
                 End Set
             End Property
             Private Property IUserMedia_PostID As String Implements IUserMedia.PostID

SCrawler/API/Base/UserDataBase.vb

@@ -9,6 +9,7 @@
 Imports System.IO
 Imports System.Net
 Imports System.Threading
+Imports System.ComponentModel
 Imports System.Runtime.CompilerServices
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Hosts
@@ -18,6 +19,7 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.ImageRenderer
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Base
@@ -98,7 +100,7 @@ Namespace API.Base
 #Region "XML Declarations"
         Private Const Name_Site As String = UserInfo.Name_Site
         Private Const Name_Plugin As String = UserInfo.Name_Plugin
-        Private Const Name_IsChannel As String = UserInfo.Name_IsChannel
+        Protected Const Name_IsChannel As String = "IsChannel"
         Friend Const Name_UserName As String = "UserName"
         Private Const Name_Model_User As String = UserInfo.Name_Model_User
         Private Const Name_Model_Collection As String = UserInfo.Name_Model_Collection
@@ -108,9 +110,9 @@ Namespace API.Base
 
         Private Const Name_UserExists As String = "UserExists"
         Private Const Name_UserSuspended As String = "UserSuspended"
-        Private Const Name_FriendlyName As String = "FriendlyName"
+        Protected Const Name_FriendlyName As String = "FriendlyName"
         Private Const Name_UserSiteName As String = "UserSiteName"
-        Private Const Name_UserID As String = "UserID"
+        Protected Const Name_UserID As String = "UserID"
         Private Const Name_Description As String = "Description"
         Private Const Name_ParseUserMediaOnly As String = "ParseUserMediaOnly"
         Private Const Name_Temporary As String = "Temporary"
@@ -132,10 +134,12 @@ Namespace API.Base
         Private Const Name_ScriptUse As String = "ScriptUse"
         Private Const Name_ScriptData As String = "ScriptData"
 
-        <Obsolete("Use 'Name_Merged'", False)> Friend Const Name_DataMerging As String = "DataMerging"
+        Protected Const Name_UseMD5Comparison As String = "UseMD5Comparison"
+        Protected Const Name_RemoveExistingDuplicates As String = "RemoveExistingDuplicates"
+        Protected Const Name_StartMD5Checked As String = "StartMD5Checked"
 #End Region
 #Region "Declarations"
-#Region "Host, Site, Progress, Self"
+#Region "Host, Site, Progress"
         Friend Property HOST As SettingsHost Implements IUserData.HOST
         Friend ReadOnly Property Site As String Implements IContentProvider.Site
             Get
@@ -167,15 +171,17 @@ Namespace API.Base
                 Me._UserSuspended = _UserSuspended
             End Set
         End Property
-        Friend Overridable Property Name As String Implements IContentProvider.Name, IPluginContentProvider.Name
+        Private Property IPluginContentProvider_Name As String Implements IPluginContentProvider.Name
+            Get
+                Return Name
+            End Get
+            Set(ByVal NewName As String)
+            End Set
+        End Property
+        Friend Overridable ReadOnly Property Name As String Implements IContentProvider.Name
             Get
                 Return User.Name
             End Get
-            Set(ByVal NewName As String)
-                User.Name = NewName
-                User.UpdateUserFile()
-                Settings.UpdateUsersList(User)
-            End Set
         End Property
         Friend Overridable Property ID As String = String.Empty Implements IContentProvider.ID, IPluginContentProvider.ID
         Protected _FriendlyName As String = String.Empty
@@ -275,11 +281,6 @@ Namespace API.Base
         End Property
 #End Region
 #Region "Channel"
-        Friend Overridable ReadOnly Property IsChannel As Boolean Implements IUserData.IsChannel
-            Get
-                Return User.IsChannel
-            End Get
-        End Property
         Friend Property CreatedByChannel As Boolean = False
 #End Region
 #Region "Images"
@@ -564,7 +565,7 @@ BlockNullPicture:
 #End Region
 #Region "Plugins Support"
         Protected Event ProgressChanged As IPluginContentProvider.ProgressChangedEventHandler Implements IPluginContentProvider.ProgressChanged
-        Protected Event TotalCountChanged As IPluginContentProvider.TotalCountChangedEventHandler Implements IPluginContentProvider.TotalCountChanged
+        Protected Event ProgressMaximumChanged As IPluginContentProvider.ProgressMaximumChangedEventHandler Implements IPluginContentProvider.ProgressMaximumChanged
         Private Property IPluginContentProvider_Settings As ISiteSettings Implements IPluginContentProvider.Settings
             Get
                 Return HOST.Source
@@ -585,9 +586,11 @@ BlockNullPicture:
         Private Function IPluginContentProvider_XmlFieldsGet() As List(Of KeyValuePair(Of String, String)) Implements IPluginContentProvider.XmlFieldsGet
             Return Nothing
         End Function
-        Private Sub IPluginContentProvider_GetMedia() Implements IPluginContentProvider.GetMedia
+        Private Sub IPluginContentProvider_GetMedia(ByVal Token As CancellationToken) Implements IPluginContentProvider.GetMedia
         End Sub
-        Private Sub IPluginContentProvider_Download() Implements IPluginContentProvider.Download
+        Private Sub IPluginContentProvider_Download(ByVal Token As CancellationToken) Implements IPluginContentProvider.Download
+        End Sub
+        Private Sub IPluginContentProvider_DownloadSingleObject(ByVal Data As IDownloadableMedia, ByVal Token As CancellationToken) Implements IPluginContentProvider.DownloadSingleObject
         End Sub
         Friend Overridable Function ExchangeOptionsGet() As Object Implements IPluginContentProvider.ExchangeOptionsGet
             Return Nothing
@@ -598,8 +601,8 @@ BlockNullPicture:
 #End Region
 #Region "IIndexable Support"
         Friend Property Index As Integer = 0 Implements IIndexable.Index
-        Private Function SetIndex(ByVal Obj As Object, ByVal _Index As Integer) As Object Implements IIndexable.SetIndex
-            DirectCast(Obj, UserDataBase).Index = _Index
+        Private Function SetIndex(ByVal Obj As Object, ByVal Index As Integer) As Object Implements IIndexable.SetIndex
+            DirectCast(Obj, UserDataBase).Index = Index
             Return Obj
         End Function
 #End Region
@@ -607,7 +610,7 @@ BlockNullPicture:
         Friend ReadOnly Property LVIKey As String Implements IUserData.Key
             Get
                 If Not _IsCollection Then
-                    Return $"{IIf(IsChannel, "C", String.Empty)}{Site.ToString.ToUpper}_{Name}"
+                    Return $"{Site.ToString.ToUpper}_{Name}"
                 Else
                     Return $"CCCC_{CollectionName}"
                 End If
@@ -658,7 +661,7 @@ BlockNullPicture:
                         Next
                     End If
                 ElseIf Settings.ShowGroups Then
-                    Return Destination.Groups.Item(GetLviGroupName(HOST, Temporary, Favorite, IsCollection, IsChannel))
+                    Return Destination.Groups.Item(GetLviGroupName(HOST, Temporary, Favorite, IsCollection))
                 End If
                 Return Destination.Groups.Item(LabelsKeeper.NoLabeledName)
             Catch ex As Exception
@@ -689,7 +692,7 @@ BlockNullPicture:
         ''' <exception cref="ArgumentOutOfRangeException"></exception>
         Friend Shared Function GetInstance(ByVal u As UserInfo, Optional ByVal _LoadUserInformation As Boolean = True) As IUserData
             If Not u.Plugin.IsEmptyString Then
-                Return Settings(u.Plugin).GetInstance(u.DownloadOption, u, _LoadUserInformation)
+                Return Settings(u.Plugin).GetInstance(ISiteSettings.Download.Main, u, _LoadUserInformation)
             Else
                 Throw New ArgumentOutOfRangeException("Plugin", $"Plugin [{u.Plugin}] information does not recognized by loader")
             End If
@@ -707,7 +710,7 @@ BlockNullPicture:
                 End If
                 Return String.Empty
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"GetPostUrl({uName}, {PostData.Post.ID})", String.Empty)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, $"GetPostUrl({uName}, {PostData.Post.ID})", String.Empty)
             End Try
         End Function
 #End Region
@@ -716,7 +719,7 @@ BlockNullPicture:
         Private _UserInformationLoaded As Boolean = False
         Friend Overridable Sub LoadUserInformation() Implements IUserData.LoadUserInformation
             Try
-                UpdateDataFiles(, True)
+                UpdateDataFiles()
                 If MyFileSettings.Exists Then
                     FileExists = True
                     Using x As New XmlFile(MyFileSettings) With {.XmlReadOnly = True}
@@ -740,14 +743,7 @@ BlockNullPicture:
                         LastUpdated = AConvert(Of Date)(x.Value(Name_LastUpdated), ADateTime.Formats.BaseDateTime, Nothing)
                         ScriptUse = x.Value(Name_ScriptUse).FromXML(Of Boolean)(False)
                         ScriptData = x.Value(Name_ScriptData)
-                        'TODELETE: UserDataBase remove old 'merge' constant
-#Disable Warning BC40000
-                        If x.Contains(Name_DataMerging) Then
-                            DataMerging = x.Value(Name_DataMerging).FromXML(Of Boolean)(False)
-                        Else
-                            DataMerging = x.Value(Name_Merged).FromXML(Of Boolean)(False)
-                        End If
-#Enable Warning
+                        DataMerging = x.Value(Name_Merged).FromXML(Of Boolean)(False)
                         ChangeCollectionName(x.Value(Name_CollectionName), False)
                         Labels.ListAddList(x.Value(Name_LabelsName).StringToList(Of String, List(Of String))("|", EDP.ReturnValue), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
                         LoadUserInformation_OptionalFields(x, True)
@@ -762,13 +758,12 @@ BlockNullPicture:
         End Sub
         Friend Overridable Sub UpdateUserInformation() Implements IUserData.UpdateUserInformation
             Try
-                UpdateDataFiles(True)
+                UpdateDataFiles()
                 MyFileSettings.Exists(SFO.Path)
                 Using x As New XmlFile With {.Name = "User"}
                     x.Add(Name_Site, Site)
                     x.Add(Name_Plugin, HOST.Key)
                     x.Add(Name_UserName, User.Name)
-                    x.Add(Name_IsChannel, IsChannel.BoolToInteger)
                     x.Add(Name_Model_User, CInt(UserModel))
                     x.Add(Name_Model_Collection, CInt(CollectionModel))
                     x.Add(Name_SpecialPath, User.SpecialPath)
@@ -815,7 +810,7 @@ BlockNullPicture:
 #Region "User data"
         Friend Overridable Overloads Sub LoadContentInformation(Optional ByVal Force As Boolean = False)
             Try
-                UpdateDataFiles(, True)
+                UpdateDataFiles()
                 If Not MyFileData.Exists Or (_DataLoaded And Not Force) Then Exit Sub
                 Using x As New XmlFile(MyFileData, Protector.Modes.All, False) With {.XmlReadOnly = True, .AllowSameNames = True}
                     x.LoadData()
@@ -830,7 +825,7 @@ BlockNullPicture:
         End Sub
         Friend Sub UpdateContentInformation()
             Try
-                UpdateDataFiles(True, True)
+                UpdateDataFiles()
                 If MyFileData.IsEmptyString Then Exit Sub
                 MyFileData.Exists(SFO.Path)
                 Using x As New XmlFile With {.AllowSameNames = True, .Name = "Data"}
@@ -846,7 +841,7 @@ BlockNullPicture:
 #Region "Open site, folder"
         Friend Overridable Sub OpenSite(Optional ByVal e As ErrorsDescriber = Nothing) Implements IContentProvider.OpenSite
             Try
-                Dim URL$ = HOST.Source.GetUserUrl(Me, IsChannel)
+                Dim URL$ = HOST.Source.GetUserUrl(Me)
                 If Not URL.IsEmptyString Then Process.Start(URL)
             Catch ex As Exception
                 If Not e.Exists Then e = New ErrorsDescriber(EDP.ShowAllMsg)
@@ -886,7 +881,7 @@ BlockNullPicture:
         Protected Function CheckDatesLimit(ByVal DateObj As Object, ByVal DateProvider As IFormatProvider) As DateResult
             Try
                 If (DownloadDateFrom.HasValue Or DownloadDateTo.HasValue) AndAlso ACheck(DateObj) Then
-                    Dim td As Date? = AConvert(Of Date)(DateObj, DateProvider, Nothing)
+                    Dim td As Date? = AConvert(DateObj, AModes.Var, GetType(Date),, True, Nothing, DateProvider)
                     If td.HasValue Then
                         If td.Value.ValueBetween(_DownloadDateFromF, _DownloadDateToF) Then
                             Return DateResult.Continue
@@ -899,13 +894,14 @@ BlockNullPicture:
                 End If
                 Return DateResult.Continue
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[UserDataBase.CheckDatesLimit({If(TypeOf DateObj Is String, CStr(DateObj), "?")})]", DateResult.Continue)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[UserDataBase.CheckDatesLimit({If(TypeOf DateObj Is String, CStr(DateObj), "?")})]", DateResult.Continue)
             End Try
         End Function
 #End Region
 #Region "Download functions and options"
         Protected Responser As Responser
         Protected UseResponserClient As Boolean = False
+        Protected UseClientTokens As Boolean = False
         Protected _ForceSaveUserData As Boolean = False
         Protected _ForceSaveUserInfo As Boolean = False
         Private _DownloadInProgress As Boolean = False
@@ -915,7 +911,7 @@ BlockNullPicture:
         Private _PictureExists As Boolean
         Private _EnvirInvokeUserUpdated As Boolean = False
         Protected Sub EnvirDownloadSet()
-            UpdateDataFiles(, True)
+            UpdateDataFiles()
             _DownloadInProgress = True
             _DescriptionChecked = False
             _DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
@@ -948,8 +944,8 @@ BlockNullPicture:
                 If Not Responser Is Nothing Then Responser.Dispose()
                 Responser = New Responser
                 If Not HOST.Responser Is Nothing Then Responser.Copy(HOST.Responser)
-                'TODO: UserDataBase remove [Responser.DecodersError]
-                Responser.DecodersError = New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue) With {
+
+                Responser.DecodersError = New ErrorsDescriber(EDP.SendToLog + EDP.ReturnValue) With {
                     .DeclaredMessage = New MMessage($"SymbolsConverter error: [{ToStringForLog()}]", ToStringForLog())}
 
                 Dim _downContent As Func(Of UserMedia, Boolean) = Function(c) c.State = UStates.Downloaded
@@ -981,7 +977,9 @@ BlockNullPicture:
 
                 ReparseVideo(Token)
                 ThrowAny(Token)
-                If IsSavedPosts Then UpdateDataFiles(True)
+
+                If UseMD5Comparison Then ValidateMD5(Token) : ThrowAny(Token)
+
                 If _TempPostsList.Count > 0 And Not DownloadMissingOnly And __SaveData Then _
                    TextSaver.SaveTextToFile(_TempPostsList.ListToString(Environment.NewLine), MyFilePosts, True,, EDP.None)
                 _ContentNew.ListAddList(_TempMediaList, LAP.ClearBeforeAdd)
@@ -1035,19 +1033,14 @@ BlockNullPicture:
                 _ForceSaveUserInfo = False
             End Try
         End Sub
-        Protected Sub UpdateDataFiles(Optional ByVal ForceSaved As Boolean = False, Optional ByVal ValidateContetnt As Boolean = False)
-            'TODELETE: saved posts name compatibility 2023.2.5.0
-            Dim __validateSaved As Func(Of Boolean) = Function() MyFileData.Exists Or MyFilePosts.Exists
-            If Not User.File.IsEmptyString Then
+        Protected Sub UpdateDataFiles()
+            If Not User.File.IsEmptyString OrElse IsSavedPosts Then
                 MyFileSettings = Nothing
                 If IsSavedPosts Then
-                    Dim u As UserInfo = User
-                    u.Name = "SavedPosts"
-                    u.UpdateUserFile()
-                    Dim mfp As SFile = u.File
-                    mfp.Name &= "_Posts"
-                    mfp.Extension = "txt"
-                    If (ValidateContetnt AndAlso mfp.Exists) Or (Not ValidateContetnt AndAlso u.File.Exists) Or ForceSaved Then MyFileSettings = u.File
+                    User = New UserInfo(SettingsHost.SavedPostsFolderName, HOST)
+                    User.File.Path = $"{HOST.SavedPostsPath.PathWithSeparator}{SettingsFolderName}"
+                    MyFileSettings = User.File
+                    MyFileSettings.Name = MyFileSettings.Name.Replace(SettingsHost.SavedPostsFolderName, "SavedPosts")
                 End If
                 If MyFileSettings.IsEmptyString Then MyFileSettings = User.File
                 MyFileData = MyFileSettings
@@ -1060,6 +1053,71 @@ BlockNullPicture:
             End If
         End Sub
         Protected MustOverride Sub DownloadDataF(ByVal Token As CancellationToken)
+#Region "DownloadSingleObject"
+        Protected IsSingleObjectDownload As Boolean = False
+        Friend Overridable Sub DownloadSingleObject(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken) Implements IUserData.DownloadSingleObject
+            Try
+                Data.DownloadState = UserMediaStates.Tried
+                Progress = Data.Progress
+                If Not Responser Is Nothing Then Responser.Dispose()
+                Responser = New Responser
+                If Not HOST Is Nothing AndAlso Not HOST.Responser Is Nothing Then Responser.Copy(HOST.Responser)
+                SeparateVideoFolder = False
+                IsSingleObjectDownload = True
+                UseInternalDownloadFileFunction_UseProgress = True
+                UseInternalM3U8Function_UseProgress = True
+                DownloadSingleObject_GetPosts(Data, Token)
+                DownloadSingleObject_CreateMedia(Data, Token)
+                DownloadSingleObject_Download(Data, Token)
+                DownloadSingleObject_PostProcessing(Data)
+            Catch ex As Exception
+                Data.DownloadState = UserMediaStates.Missing
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"{Site} single data downloader error: {Data.URL}")
+            End Try
+        End Sub
+        Protected Overridable Sub DownloadSingleObject_CreateMedia(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            If _TempMediaList.Count > 0 Then
+                For Each m As UserMedia In _TempMediaList
+                    m.File = DownloadSingleObject_CreateFile(Data, m.File)
+                    _ContentNew.Add(m)
+                Next
+            End If
+        End Sub
+        Protected Overridable Sub DownloadSingleObject_PostProcessing(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, Optional ByVal ResetTitle As Boolean = True)
+            If _ContentNew.Count > 0 Then
+                If _ContentNew.Any(Function(mm) mm.State = UStates.Downloaded) Then
+                    Data.DownloadState = UserMediaStates.Downloaded
+                    If _ContentNew(0).Type = UTypes.Picture Or _ContentNew(0).Type = UTypes.GIF Then
+                        DirectCast(Data, IDownloadableMedia).ThumbnailFile = _ContentNew(0).File
+                    ElseIf Settings.STDownloader_TakeSnapshot And Settings.FfmpegFile.Exists And Not Settings.STDownloader_RemoveDownloadedAutomatically Then
+                        Dim f As SFile = _ContentNew(0).File
+                        Dim ff As SFile = f
+                        ff.Name &= "_thumb"
+                        ff.Extension = "jpg"
+                        f = Web.FFMPEG.TakeSnapshot(f, ff, Settings.FfmpegFile, TimeSpan.FromSeconds(1),,, EDP.LogMessageValue)
+                        If f.Exists Then DirectCast(Data, IDownloadableMedia).ThumbnailFile = f
+                    End If
+                Else
+                    Data.DownloadState = UserMediaStates.Missing
+                End If
+                YouTube.Objects.YouTubeMediaContainerBase.Update(_ContentNew(0), Data)
+                If ResetTitle And Not _ContentNew(0).File.Name.IsEmptyString Then Data.Title = _ContentNew(0).File.Name
+            Else
+                Data.DownloadState = UserMediaStates.Missing
+            End If
+        End Sub
+        Protected Function DownloadSingleObject_CreateFile(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal DFile As SFile) As SFile
+            If Not Data.File.Path.IsEmptyString Then DFile.Path = Data.File.Path
+            If DFile.Name.IsEmptyString Then DFile.Name = "OutputFile"
+            Return DFile
+        End Function
+        Protected Overridable Sub DownloadSingleObject_Download(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            DownloadContent(Token)
+        End Sub
+        Protected Overridable Sub DownloadSingleObject_GetPosts(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken)
+        End Sub
+#End Region
+#Region "ReparseVideo, ReparseMissing"
         Protected Overridable Sub ReparseVideo(ByVal Token As CancellationToken)
         End Sub
         ''' <summary>
@@ -1069,15 +1127,170 @@ BlockNullPicture:
         ''' </summary>
         Protected Overridable Sub ReparseMissing(ByVal Token As CancellationToken)
         End Sub
+#End Region
+#Region "MD5 support"
+        Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
+        Friend Property UseMD5Comparison As Boolean = False
+        Protected Property StartMD5Checked As Boolean = True
+        Friend Property RemoveExistingDuplicates As Boolean = False
+        Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken)
+            Try
+                Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
+                If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
+                    Dim i%
+                    Dim itemsCount% = 0
+                    Dim limit% = If(DownloadTopCount, 0)
+                    Dim data As UserMedia = Nothing
+                    Dim hashList As New Dictionary(Of String, SFile)
+                    Dim f As SFile
+                    Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
+                    Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
+                        Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
+                            Try
+                                Dim ImgFormat As Imaging.ImageFormat = Nothing
+                                Dim hash$ = String.Empty
+                                Dim __isGif As Boolean = False
+                                If __data.Type = UTypes.GIF Then
+                                    ImgFormat = Imaging.ImageFormat.Gif
+                                    __isGif = True
+                                ElseIf Not __data.File.IsEmptyString Then
+                                    ImgFormat = GetImageFormat(__data.File)
+                                End If
+                                If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
+                                If IsUrl Then
+                                    hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL_BASE.IfNullOrEmpty(__data.URL), ErrMD5), ImgFormat, ErrMD5))
+                                Else
+                                    hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+                                End If
+                                If hash.IsEmptyString And Not __isGif Then
+                                    If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
+                                    If IsUrl Then
+                                        hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL_BASE.IfNullOrEmpty(__data.URL), ErrMD5), ImgFormat, ErrMD5))
+                                    Else
+                                        hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+                                    End If
+                                End If
+                                Return hash
+                            Catch
+                                Return String.Empty
+                            End Try
+                        End Function
+                    If Not StartMD5Checked Then
+                        StartMD5Checked = True
+                        If _ContentList.Exists(missingMD5) Then
+                            Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
+                            Dim eIndx%
+                            Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
+                            If RemoveExistingDuplicates Then
+                                RemoveExistingDuplicates = False
+                                _ForceSaveUserInfo = True
+                                If existingFiles.Count > 0 Then
+                                    Dim h$
+                                    For i = existingFiles.Count - 1 To 0 Step -1
+                                        h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
+                                        If Not h.IsEmptyString Then
+                                            If hashList.ContainsKey(h) Then
+                                                MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
+                                                existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
+                                                existingFiles.RemoveAt(i)
+                                            Else
+                                                hashList.Add(h, existingFiles(i))
+                                            End If
+                                        End If
+                                    Next
+                                End If
+                            End If
+                            For i = 0 To _ContentList.Count - 1
+                                data = _ContentList(i)
+                                If (data.Type = UTypes.GIF Or data.Type = UTypes.Picture) Then
+                                    If data.MD5.IsEmptyString Then
+                                        ThrowAny(Token)
+                                        eIndx = existingFiles.FindIndex(eFinder)
+                                        If eIndx >= 0 Then
+                                            data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
+                                            If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
+                                        End If
+                                    End If
+                                    existingFiles.RemoveAll(eFinder)
+                                End If
+                            Next
+                            If existingFiles.Count > 0 Then
+                                For i = 0 To existingFiles.Count - 1
+                                    f = existingFiles(i)
+                                    data = New UserMedia(f.File) With {
+                                        .State = UStates.Downloaded,
+                                        .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
+                                        .File = f
+                                    }
+                                    ThrowAny(Token)
+                                    data.MD5 = __getMD5(data, False)
+                                    If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
+                                Next
+                                existingFiles.Clear()
+                            End If
+                        End If
+                    End If
+
+                    If _ContentList.Count > 0 Then
+                        With _ContentList.Select(Function(d) d.MD5)
+                            If .ListExists Then .ToList.ForEach(Sub(md5value) _
+                               If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
+                        End With
+                    End If
+
+                    For i = _TempMediaList.Count - 1 To 0 Step -1
+                        If limit > 0 And itemsCount >= limit Then
+                            _TempMediaList.RemoveAt(i)
+                        Else
+                            data = _TempMediaList(i)
+                            If missingMD5(data) Then
+                                ThrowAny(Token)
+                                data.MD5 = __getMD5(data, True)
+                                If Not data.MD5.IsEmptyString Then
+                                    If hashList.ContainsKey(data.MD5) Then
+                                        _TempMediaList.RemoveAt(i)
+                                    Else
+                                        hashList.Add(data.MD5, New SFile)
+                                        _TempMediaList(i) = data
+                                        itemsCount += 1
+                                    End If
+                                End If
+                            End If
+                        End If
+                    Next
+                End If
+            Catch iex As ArgumentOutOfRangeException When Disposed
+            Catch ex As Exception
+                ProcessException(ex, Token, "ValidateMD5",, VALIDATE_MD5_ERROR)
+            End Try
+        End Sub
+#End Region
+#Region "DownloadContent"
         Protected MustOverride Sub DownloadContent(ByVal Token As CancellationToken)
         Private NotInheritable Class OptionalWebClient : Inherits DownloadObjects.WebClient2
+            Private ReadOnly Source As UserDataBase
             Friend Sub New(ByRef Source As UserDataBase)
+                Me.Source = Source
                 UseResponserClient = Source.UseResponserClient
                 If UseResponserClient Then
-                    RC = Source.Responser
+                    Client = Source.Responser
                 Else
-                    WC = New WebClient
+                    Client = New RWebClient With {.UseNativeClient = Not Source.IsSingleObjectDownload}
                 End If
+                If Source.IsSingleObjectDownload Then DelegateEvents = True
+            End Sub
+            Private _LastProgressValue As Integer = 0
+            Protected Overrides Sub Client_DownloadProgressChanged(ByVal Sender As Object, ByVal e As DownloadProgressChangedEventArgs)
+                Dim v% = e.ProgressPercentage
+                If v > _LastProgressValue Then
+                    If v > 100 Then v = 100
+                    Source.Progress.Value = v
+                    Source.Progress.Perform(0)
+                End If
+                _LastProgressValue = e.ProgressPercentage
+            End Sub
+            Protected Overrides Sub Client_DownloadFileCompleted(ByVal Sender As Object, ByVal e As AsyncCompletedEventArgs)
+                Source.Progress.Done()
             End Sub
         End Class
         Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
@@ -1090,37 +1303,51 @@ BlockNullPicture:
                     If _ContentNew.Count > 0 Then
                         MyFile.Exists(SFO.Path)
                         Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
-                        Dim MyDir$ = MyFile.CutPath.PathNoSeparator
+                        Dim MyDir$ = DownloadContentDefault_GetRootDir()
                         Dim vsf As Boolean = SeparateVideoFolderF
                         Dim __isVideo As Boolean
+                        Dim __interrupt As Boolean
                         Dim f As SFile
                         Dim v As UserMedia
+                        Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
 
                         Using w As New OptionalWebClient(Me)
                             If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
                             Progress.Maximum += _ContentNew.Count
+                            If IsSingleObjectDownload Then
+                                If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
+                                    Progress.Value = 0
+                                    Progress.Maximum = 100
+                                    Progress.Provider = MyProgressNumberProvider.Percentage
+                                ElseIf _ContentNew(0).Type = UTypes.m3u8 Then
+                                    Progress.Provider = MyProgressNumberProvider.Percentage
+                                Else
+                                    w.DelegateEvents = False
+                                End If
+                            End If
+
                             For i = 0 To _ContentNew.Count - 1
                                 ThrowAny(Token)
                                 v = _ContentNew(i)
                                 v.State = UStates.Tried
                                 If v.File.IsEmptyString Then
-                                    f = v.URL
+                                    f = CreateFileFromUrl(v.URL)
                                 Else
                                     f = v.File
                                 End If
                                 f.Separator = "\"
-                                f.Path = MyDir
+                                If Not IsSingleObjectDownload Then f.Path = MyDir
 
                                 If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
 
-                                If Not v.File.IsEmptyString And Not v.URL.IsEmptyString Then
+                                If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
                                     Try
-                                        __isVideo = v.Type = UTypes.Video Or f.Extension = "mp4"
+                                        __isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
 
                                         If f.Extension.IsEmptyString Then
                                             Select Case v.Type
                                                 Case UTypes.Picture : f.Extension = "jpg"
-                                                Case UTypes.Video : f.Extension = "mp4"
+                                                Case UTypes.Video, UTypes.m3u8 : f.Extension = "mp4"
                                                 Case UTypes.GIF : f.Extension = "gif"
                                             End Select
                                         ElseIf f.Extension = "webp" And Settings.DownloadNativeImageFormat Then
@@ -1138,16 +1365,29 @@ BlockNullPicture:
                                             End If
                                         End If
 
+                                        If __isVideo Then fileNumProvider.FileName = f.Name : f = SFile.IndexReindex(f,,, fileNumProvider)
+
+                                        __interrupt = False
                                         If v.Type = UTypes.m3u8 And UseInternalM3U8Function Then
-                                            f = DownloadM3U8(v.URL, v, f)
+                                            f = DownloadM3U8(v.URL, v, f, Token)
                                             If f.IsEmptyString Then Throw New Exception("M3U8 download failed")
+                                        ElseIf UseInternalDownloadFileFunction AndAlso ValidateDownloadFile(v.URL, v, __interrupt) Then
+                                            f = DownloadFile(v.URL, v, f, Token)
+                                            If f.IsEmptyString Then Throw New Exception("InternalFunc download failed")
                                         Else
-                                            w.DownloadFile(v.URL, f.ToString)
+                                            If UseInternalDownloadFileFunction And __interrupt Then Throw New Exception("InternalFunc download interrupted")
+                                            If UseClientTokens Then
+                                                w.DownloadFile(v.URL, f, Token)
+                                            Else
+                                                w.DownloadFile(v.URL, f)
+                                            End If
                                         End If
 
                                         If __isVideo Then
                                             v.Type = UTypes.Video
                                             DownloadedVideos(False) += 1
+                                        ElseIf v.Type = UTypes.GIF Then
+                                            DownloadedPictures(False) += 1
                                         Else
                                             v.Type = UTypes.Picture
                                             DownloadedPictures(False) += 1
@@ -1155,11 +1395,20 @@ BlockNullPicture:
 
                                         v.File = ChangeFileNameByProvider(f, v)
                                         v.State = UStates.Downloaded
+                                        DownloadContentDefault_PostProcessing(v, f, Token)
                                         dCount += 1
-                                    Catch wex As Exception
-                                        v.Attempts += 1
+                                    Catch woex As OperationCanceledException When Token.IsCancellationRequested
+                                        If f.Exists Then f.Delete()
                                         v.State = UStates.Missing
-                                        If MissingErrorsAdd Then ErrorDownloading(f, v.URL)
+                                        v.Attempts += 1
+                                        _ContentNew(i) = v
+                                        Throw woex
+                                    Catch wex As Exception
+                                        If DownloadContentDefault_ProcessDownloadException() Then
+                                            v.Attempts += 1
+                                            v.State = UStates.Missing
+                                            If MissingErrorsAdd Then ErrorDownloading(f, v.URL)
+                                        End If
                                     End Try
                                 Else
                                     v.State = UStates.Skipped
@@ -1185,9 +1434,30 @@ BlockNullPicture:
             End Try
         End Sub
         Protected UseInternalM3U8Function As Boolean = False
-        Protected Overridable Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+        Protected UseInternalM3U8Function_UseProgress As Boolean = False
+        Protected Overridable Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile,
+                                                    ByVal Token As CancellationToken) As SFile
             Return Nothing
         End Function
+        Protected UseInternalDownloadFileFunction As Boolean = False
+        Protected UseInternalDownloadFileFunction_UseProgress As Boolean = False
+        Protected Overridable Function DownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile,
+                                                    ByVal Token As CancellationToken) As SFile
+            Return Nothing
+        End Function
+        Protected Overridable Function ValidateDownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByRef Interrupt As Boolean) As Boolean
+            Return True
+        End Function
+        Protected Overridable Function DownloadContentDefault_GetRootDir() As String
+            Return MyFile.CutPath(IIf(IsSingleObjectDownload, 0, 1)).PathNoSeparator
+        End Function
+        Protected Overridable Sub DownloadContentDefault_PostProcessing(ByRef m As UserMedia, ByVal File As SFile, ByVal Token As CancellationToken)
+        End Sub
+        Protected Overridable Function DownloadContentDefault_ProcessDownloadException() As Boolean
+            Return True
+        End Function
+#End Region
+#Region "ProcessException"
         Protected Const EXCEPTION_OPERATION_CANCELED As Integer = -1
         ''' <param name="RDE">Request DownloadingException</param>
         ''' <returns>0 - exit</returns>
@@ -1208,14 +1478,19 @@ BlockNullPicture:
         End Function
         ''' <summary>0 - Execute LogError and set HasError</summary>
         Protected MustOverride Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False, Optional ByVal EObj As Object = Nothing) As Integer
-        Protected Function ChangeFileNameByProvider(ByVal f As SFile, ByVal m As UserMedia) As SFile
+#End Region
+#Region "ChangeFileNameByProvider, RunScript"
+        Protected Overridable Function CreateFileFromUrl(ByVal URL As String) As SFile
+            Return New SFile(URL)
+        End Function
+        Protected Overridable Function ChangeFileNameByProvider(ByVal f As SFile, ByVal m As UserMedia) As SFile
             Dim ff As SFile = Nothing
             Try
                 If f.Exists Then
                     If Not Settings.FileReplaceNameByDate.Value = FileNameReplaceMode.None Then
                         ff = f
                         ff.Name = String.Format(FileDateAppenderPattern, f.Name, CStr(AConvert(Of String)(If(m.Post.Date, Now), FileDateAppenderProvider, String.Empty)))
-                        ff = SFile.Indexed_IndexFile(ff,, New NumberedFile(ff))
+                        ff = SFile.IndexReindex(ff,,, New NumberedFile(ff))
                     End If
                     If Not ff.Name.IsEmptyString Then My.Computer.FileSystem.RenameFile(f, ff.File) : Return ff
                 End If
@@ -1238,7 +1513,7 @@ BlockNullPicture:
                     If Not ScriptPattern.IsEmptyString Then
                         If Not ScriptPattern.Contains(spa) Then ScriptPattern &= $" ""{spa}"""
                         Using b As New BatchExecutor With {.RedirectStandardError = True}
-                            b.Execute({String.Format(ScriptPattern, MyFile.CutPath(1).PathNoSeparator)}, EDP.SendInLog + EDP.ThrowException)
+                            b.Execute({String.Format(ScriptPattern, MyFile.CutPath(1).PathNoSeparator)}, EDP.SendToLog + EDP.ThrowException)
                             If b.HasError Or Not b.ErrorOutput.IsEmptyString Then Throw New Exception(b.ErrorOutput, b.ErrorException)
                         End Using
                     End If
@@ -1248,6 +1523,7 @@ BlockNullPicture:
             End Try
         End Sub
 #End Region
+#End Region
 #Region "Delete, Move, Merge, Copy"
         Friend Overridable Function Delete(Optional ByVal Multiple As Boolean = False, Optional ByVal CollectionValue As Integer = -1) As Integer Implements IUserData.Delete
             Dim f As SFile = SFile.GetPath(MyFile.CutPath.Path)
@@ -1359,7 +1635,7 @@ BlockNullPicture:
                     FilesMover.Invoke
                     If SFile.GetFiles(UserBefore.File.CutPath,, SearchOption.AllDirectories,
                                       New ErrorsDescriber(False, False, False, New List(Of SFile))).Count = 0 Then
-                        UserBefore.File.CutPath.Delete(SFO.Path, Settings.DeleteMode, EDP.SendInLog)
+                        UserBefore.File.CutPath.Delete(SFO.Path, Settings.DeleteMode, EDP.SendToLog)
                     End If
                     If Not ScriptData.IsEmptyString AndAlso ScriptData.Contains(UserBefore.File.PathNoSeparator) Then _
                        ScriptData = ScriptData.Replace(UserBefore.File.PathNoSeparator, MyFile.PathNoSeparator)
@@ -1424,7 +1700,7 @@ BlockNullPicture:
 #End Region
 #Region "Errors functions"
         Protected Sub LogError(ByVal ex As Exception, ByVal Message As String)
-            ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{ToStringForLog()}: {Message}")
+            ErrorsDescriber.Execute(EDP.SendToLog, ex, $"{ToStringForLog()}: {Message}")
         End Sub
         Protected Sub ErrorDownloading(ByVal f As SFile, ByVal URL As String)
             If Not f.Exists Then MyMainLOG = $"Error downloading from [{URL}] to [{f}]"
@@ -1546,7 +1822,7 @@ BlockNullPicture:
 #Region "Base interfaces"
     Friend Interface IContentProvider
         ReadOnly Property Site As String
-        Property Name As String
+        ReadOnly Property Name As String
         Property ID As String
         Property FriendlyName As String
         Property Description As String
@@ -1554,6 +1830,7 @@ BlockNullPicture:
         Property Temporary As Boolean
         Sub OpenSite(Optional ByVal e As ErrorsDescriber = Nothing)
         Sub DownloadData(ByVal Token As CancellationToken)
+        Sub DownloadSingleObject(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken)
     End Interface
     Friend Interface IUserData : Inherits IContentProvider, IComparable(Of UserDataBase), IComparable, IEquatable(Of UserDataBase), IIndexable, IDisposable
         Event UserUpdated(ByVal User As IUserData)
@@ -1571,7 +1848,6 @@ BlockNullPicture:
         ReadOnly Property IsVirtual As Boolean
         ReadOnly Property Labels As List(Of String)
 #End Region
-        ReadOnly Property IsChannel As Boolean
         Property Exists As Boolean
         Property Suspended As Boolean
         Property ReadyForDownload As Boolean

SCrawler/API/BaseObjects/DomainEnvir.vb

@@ -1,86 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Forms
-Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
-Namespace API.BaseObjects
-    Friend Interface IDomainContainer
-        ReadOnly Property Icon As Icon
-        ReadOnly Property Site As String
-        ReadOnly Property Domains As List(Of String)
-        ReadOnly Property DomainsTemp As List(Of String)
-        ReadOnly Property DomainsDefault As String
-        ReadOnly Property DomainsSettingProp As Plugin.PropertyValue
-        Property DomainsChanged As Boolean
-        Property Initialized As Boolean
-        Property DomainsUpdateInProgress As Boolean
-        Property DomainsUpdatedBySite As Boolean
-        Sub UpdateDomains()
-    End Interface
-    Friend NotInheritable Class DomainContainer
-        Private Sub New()
-        End Sub
-        Friend Shared Sub EndInit(ByVal s As IDomainContainer)
-            If ACheck(s.DomainsSettingProp.Value) Then s.Domains.ListAddList(CStr(s.DomainsSettingProp.Value).Split("|"), LAP.NotContainsOnly)
-        End Sub
-        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer)
-            UpdateDomains(s, Nothing, True)
-        End Sub
-        Friend Overloads Shared Sub UpdateDomains(ByVal s As IDomainContainer, ByVal NewDomains As IEnumerable(Of String), ByVal Internal As Boolean)
-            With s
-                If Not .Initialized Or (.DomainsUpdatedBySite And Not Internal) Then Exit Sub
-                If Not .DomainsUpdateInProgress Then
-                    .DomainsUpdateInProgress = True
-                    .Domains.ListAddList(.DomainsDefault.Split("|"), LAP.NotContainsOnly)
-                    .Domains.ListAddList(NewDomains, LAP.NotContainsOnly)
-                    .DomainsSettingProp.Value = .Domains.ListToString("|")
-                    If Not Internal Then .DomainsUpdatedBySite = True
-                    .DomainsUpdateInProgress = False
-                End If
-            End With
-        End Sub
-        Friend Shared Sub Update(ByVal s As IDomainContainer)
-            With s
-                If .DomainsChanged Then
-                    .Domains.Clear()
-                    .Domains.ListAddList(.DomainsTemp, LAP.NotContainsOnly)
-                    .UpdateDomains()
-                End If
-            End With
-        End Sub
-        Friend Shared Sub EndEdit(ByVal s As IDomainContainer)
-            s.DomainsTemp.ListAddList(s.Domains, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
-            s.DomainsChanged = False
-        End Sub
-        Friend Shared Sub OpenSettingsForm(ByVal s As IDomainContainer)
-            Dim __add As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e) e.Item = InputBoxE($"Enter a new domain using the pattern [{s.Site}.com]:", "New domain").IfNullOrEmptyE(Nothing)
-            Dim __delete As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e)
-                                                                           Dim n$ = AConvert(Of String)(e.Item, AModes.Var, String.Empty)
-                                                                           e.Result = MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
-                                                                                              "Removing domains"}, vbYesNo) = vbYes
-                                                                       End Sub
-            Using f As New SimpleListForm(Of String)(If(s.DomainsChanged, s.DomainsTemp, s.Domains), Settings.Design) With {
-                .Buttons = {ADB.Add, ADB.Delete},
-                .Mode = SimpleListFormModes.Remaining,
-                .FormText = s.Site,
-                .Icon = s.Icon,
-                .LocationOnly = True,
-                .Size = New Size(400, 330),
-                .DesignXMLNodeName = s.Site
-            }
-                AddHandler f.AddClick, __add
-                AddHandler f.DeleteClick, __delete
-                f.ShowDialog()
-                If f.DialogResult = DialogResult.OK Then
-                    s.DomainsChanged = True
-                    s.DomainsTemp.ListAddList(f.DataResult, LAP.ClearBeforeAdd, LAP.NotContainsOnly)
-                End If
-            End Using
-        End Sub
-    End Class
-End Namespace

SCrawler/API/BaseObjects/DomainsContainer.vb

@@ -0,0 +1,108 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports PersonalUtilities.Forms
+Imports PersonalUtilities.Tools
+Imports ADB = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons
+Namespace API.Base
+    Friend Class DomainsContainer : Implements IEnumerable(Of String), IMyEnumerator(Of String)
+        Friend Event DomainsUpdated(ByVal Sender As DomainsContainer)
+        Friend ReadOnly Property Domains As List(Of String)
+        Friend ReadOnly Property DomainsTemp As List(Of String)
+        Friend ReadOnly Property DomainsDefault As String
+        Friend Property Changed As Boolean
+        Private DomainsUpdateInProgress As Boolean = False
+        Friend Property UpdatedBySite As Boolean
+        Protected ReadOnly Property Instance As ISiteSettings
+        Friend Property DestinationProp As PropertyValue
+        Default Friend ReadOnly Property Item(ByVal Index As Integer) As String Implements IMyEnumerator(Of String).MyEnumeratorObject
+            Get
+                Return Domains(Index)
+            End Get
+        End Property
+        Friend ReadOnly Property Count As Integer Implements IMyEnumerator(Of String).MyEnumeratorCount
+            Get
+                Return Domains.Count
+            End Get
+        End Property
+        Friend Sub New(ByVal _Instance As ISiteSettings, ByVal DefaultValue As String)
+            Domains = New List(Of String)
+            DomainsTemp = New List(Of String)
+            Instance = _Instance
+            DomainsDefault = DefaultValue
+            If Not DomainsDefault.IsEmptyString Then Domains.ListAddList(CStr(DomainsDefault).Split("|"), LAP.NotContainsOnly)
+        End Sub
+        Friend Sub PopulateInitialDomains(ByVal InitialValue As String)
+            If Not InitialValue.IsEmptyString Then Domains.ListAddList(CStr(InitialValue).Split("|"), LAP.NotContainsOnly)
+        End Sub
+        Public Overrides Function ToString() As String
+            Return Domains.ListToString("|")
+        End Function
+        Friend Sub Add(ByVal NewDomains As IEnumerable(Of String), ByVal UpdateBySite As Boolean)
+            If Not DomainsUpdateInProgress Then
+                DomainsUpdateInProgress = True
+                Domains.ListAddList(NewDomains, LAP.NotContainsOnly)
+                If UpdateBySite Then Me.UpdatedBySite = True
+                Save()
+                DomainsUpdateInProgress = False
+                RaiseEvent DomainsUpdated(Me)
+            End If
+        End Sub
+        Friend Overridable Function Apply() As Boolean
+            If Changed Then
+                Domains.Clear()
+                Domains.ListAddList(DomainsTemp, LAP.NotContainsOnly)
+                Save()
+                RaiseEvent DomainsUpdated(Me)
+                Return True
+            Else
+                Return False
+            End If
+        End Function
+        Friend Overridable Sub Save()
+            If Not DestinationProp Is Nothing Then DestinationProp.Value = ToString()
+        End Sub
+        Friend Overridable Sub Reset()
+            Changed = False
+            DomainsTemp.Clear()
+        End Sub
+        Friend Overridable Sub OpenSettingsForm()
+            Dim __add As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e) e.Item = InputBoxE($"Enter a new domain using the pattern [{Instance.Site}.com]:", "New domain").IfNullOrEmptyE(Nothing)
+            Dim __delete As EventHandler(Of SimpleListFormEventArgs) = Sub(sender, e)
+                                                                           Dim n$ = AConvert(Of String)(e.Item, AModes.Var, String.Empty)
+                                                                           e.Result = MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
+                                                                                              "Removing domains"}, vbYesNo) = vbYes
+                                                                       End Sub
+            Using f As New SimpleListForm(Of String)(If(Changed, DomainsTemp, Domains), Settings.Design) With {
+                .Buttons = {ADB.Add, ADB.Delete},
+                .Mode = SimpleListFormModes.Remaining,
+                .FormText = Instance.Site,
+                .Icon = Instance.Icon,
+                .LocationOnly = True,
+                .Size = New Size(400, 330),
+                .DesignXMLNodeName = $"{Instance.Site}_DomainsForm"
+            }
+                AddHandler f.AddClick, __add
+                AddHandler f.DeleteClick, __delete
+                f.ShowDialog()
+                If f.DialogResult = DialogResult.OK Then
+                    Changed = True
+                    DomainsTemp.Clear()
+                    DomainsTemp.ListAddList(f.DataResult, LAP.NotContainsOnly)
+                End If
+            End Using
+        End Sub
+        Private Function GetEnumerator() As IEnumerator(Of String) Implements IEnumerable(Of String).GetEnumerator
+            Return New MyEnumerator(Of String)(Me)
+        End Function
+        Private Function IEnumerable_GetEnumerator() As IEnumerator Implements IEnumerable.GetEnumerator
+            Return GetEnumerator()
+        End Function
+    End Class
+End Namespace

SCrawler/API/BaseObjects/InternalSettingsForm.Designer.vb

@@ -0,0 +1,89 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Base
+    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
+    Partial Friend Class InternalSettingsForm : Inherits System.Windows.Forms.Form
+        <System.Diagnostics.DebuggerNonUserCode()>
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            Try
+                If disposing AndAlso components IsNot Nothing Then
+                    components.Dispose()
+                End If
+            Finally
+                MyBase.Dispose(disposing)
+            End Try
+        End Sub
+        Private components As System.ComponentModel.IContainer
+        <System.Diagnostics.DebuggerStepThrough()>
+        Private Sub InitializeComponent()
+            Me.components = New System.ComponentModel.Container()
+            Me.CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
+            Me.TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
+            Me.TT_MAIN = New System.Windows.Forms.ToolTip(Me.components)
+            Me.CONTAINER_MAIN.ContentPanel.SuspendLayout()
+            Me.CONTAINER_MAIN.SuspendLayout()
+            Me.SuspendLayout()
+            '
+            'CONTAINER_MAIN
+            '
+            '
+            'CONTAINER_MAIN.ContentPanel
+            '
+            Me.CONTAINER_MAIN.ContentPanel.Controls.Add(Me.TP_MAIN)
+            Me.CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(184, 0)
+            Me.CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CONTAINER_MAIN.LeftToolStripPanelVisible = False
+            Me.CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
+            Me.CONTAINER_MAIN.Name = "CONTAINER_MAIN"
+            Me.CONTAINER_MAIN.RightToolStripPanelVisible = False
+            Me.CONTAINER_MAIN.Size = New System.Drawing.Size(184, 25)
+            Me.CONTAINER_MAIN.TabIndex = 0
+            Me.CONTAINER_MAIN.TopToolStripPanelVisible = False
+            '
+            'TP_MAIN
+            '
+            Me.TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            Me.TP_MAIN.ColumnCount = 1
+            Me.TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 20.0!))
+            Me.TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TP_MAIN.Location = New System.Drawing.Point(0, 0)
+            Me.TP_MAIN.Name = "TP_MAIN"
+            Me.TP_MAIN.RowCount = 1
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 160.0!))
+            Me.TP_MAIN.Size = New System.Drawing.Size(184, 0)
+            Me.TP_MAIN.TabIndex = 0
+            '
+            'InternalSettingsForm
+            '
+            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
+            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
+            Me.ClientSize = New System.Drawing.Size(184, 25)
+            Me.Controls.Add(Me.CONTAINER_MAIN)
+            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
+            Me.KeyPreview = True
+            Me.MaximizeBox = False
+            Me.MinimizeBox = False
+            Me.Name = "InternalSettingsForm"
+            Me.ShowInTaskbar = False
+            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
+            Me.Text = "Settings"
+            Me.CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
+            Me.CONTAINER_MAIN.ResumeLayout(False)
+            Me.CONTAINER_MAIN.PerformLayout()
+            Me.ResumeLayout(False)
+
+        End Sub
+
+        Private WithEvents TP_MAIN As TableLayoutPanel
+        Private WithEvents TT_MAIN As ToolTip
+        Private WithEvents CONTAINER_MAIN As ToolStripContainer
+    End Class
+End Namespace

SCrawler/API/BaseObjects/InternalSettingsForm.resx

@@ -117,10 +117,7 @@
   <resheader name="writer">
     <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
   </resheader>
-  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
-  </metadata>
-  <metadata name="TP_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
-    <value>False</value>
+  <metadata name="TT_MAIN.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
+    <value>17, 17</value>
   </metadata>
 </root>

SCrawler/API/BaseObjects/InternalSettingsForm.vb

@@ -0,0 +1,256 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Reflection
+Imports System.ComponentModel
+Imports PersonalUtilities.Forms
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Namespace API.Base
+    Friend Class InternalSettingsForm
+        Private WithEvents MyDefs As DefaultFormOptions
+        Private ReadOnly Property MySettingsInstance As ISiteSettings
+        Private ReadOnly Property MyObject As Object
+        Private ReadOnly IsSettingsForm As Boolean = True
+        Private ReadOnly Property MyMembers As List(Of MemberOption)
+        ''' <summary>Default: 200</summary>
+        Friend Property MinimumWidth As Integer = 200
+        Private Class MemberOption : Inherits Hosts.PropertyValueHost : Implements IDisposable
+            Friend ToolTip As String
+            Friend Caption As String
+            Friend ThreeState As Boolean = False
+            Friend AllowNull As Boolean = True
+            Friend Provider As Type
+            Friend Overrides Property Type As Type
+                Get
+                    Return _Type
+                End Get
+                Set(ByVal t As Type)
+                    MyBase.Type = t
+                End Set
+            End Property
+            Friend Overrides ReadOnly Property Name As String
+                Get
+                    Return Member.Name
+                End Get
+            End Property
+            Friend Overrides Property LeftOffset As Integer
+                Get
+                    Return If(_LeftOffset, 0)
+                End Get
+                Set(ByVal NewOffset As Integer)
+                    MyBase.LeftOffset = NewOffset
+                End Set
+            End Property
+            Private ReadOnly _MinimumWidth As Integer? = Nothing
+            Friend ReadOnly Property Width As Integer
+                Get
+                    Return LeftOffset + If(_MinimumWidth, 0) + If(TypeOf Control Is CheckBox, 0, 200) +
+                           PaddingE.GetOf({Control}).Horizontal(2) + MeasureText(Caption, Control.Font).Width
+                End Get
+            End Property
+            Friend OptName As String = String.Empty
+            Friend Sub New(ByRef PropertySource As Object, ByVal m As MemberInfo, ByVal ps As PSettingAttribute, ByVal po As PropertyOption)
+                Source = PropertySource
+                Member = m
+                _Type = Member.GetMemberType
+                _Value = Member.GetMemberValue(PropertySource)
+                With ps
+                    ToolTip = .ToolTip
+                    Caption = .Caption
+                    ThreeState = .ThreeState
+                    AllowNull = .AllowNull
+                    Provider = .Provider
+                    _LeftOffset = .LeftOffsetGet
+                    ControlNumber = .Number
+                    _MinimumWidth = .MinimumWidth
+                End With
+                If Not po Is Nothing Then
+                    With po
+                        OptName = po.Name
+                        If ToolTip.IsEmptyString Then ToolTip = .ControlToolTip
+                        If Caption.IsEmptyString Then Caption = .ControlText
+                    End With
+                End If
+            End Sub
+            Protected Overrides ReadOnly Property Control_IsInformationLabel As Boolean
+                Get
+                    Return False
+                End Get
+            End Property
+            Protected Overrides ReadOnly Property Control_ThreeStates As Boolean
+                Get
+                    Return ThreeState
+                End Get
+            End Property
+            Protected Overrides ReadOnly Property Control_Caption As String
+                Get
+                    Return Caption
+                End Get
+            End Property
+            Protected Overrides ReadOnly Property Control_ToolTip As String
+                Get
+                    Return ToolTip
+                End Get
+            End Property
+            Friend Overloads Sub CreateControl(ByVal f As FieldsChecker, ByVal TT As ToolTip)
+                CreateControl(TT)
+                If Not Provider Is Nothing Then f.AddControl(Control, Caption, Type, AllowNull, Activator.CreateInstance(Provider))
+            End Sub
+#Region "IDisposable Support"
+            Private disposedValue As Boolean = False
+            Protected Overridable Overloads Sub Dispose(ByVal disposing As Boolean)
+                If Not disposedValue Then
+                    If disposing Then Control.Dispose()
+                    Control = Nothing
+                    disposedValue = True
+                End If
+            End Sub
+            Protected Overrides Sub Finalize()
+                Dispose(False)
+                MyBase.Finalize()
+            End Sub
+            Friend Overloads Sub Dispose() Implements IDisposable.Dispose
+                Dispose(True)
+                GC.SuppressFinalize(Me)
+            End Sub
+#End Region
+        End Class
+        Friend Sub New(ByVal Obj As Object, ByVal s As ISiteSettings, ByVal _IsSettingsForm As Boolean)
+            InitializeComponent()
+            MyDefs = New DefaultFormOptions(Me, Settings.Design)
+            MyMembers = New List(Of MemberOption)
+
+            MyObject = Obj
+            MySettingsInstance = s
+            IsSettingsForm = _IsSettingsForm
+            If _IsSettingsForm Then
+                Text = "Settings"
+            Else
+                Text = "Options"
+            End If
+            Icon = s.Icon
+        End Sub
+        Private Sub InternalSettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
+            Try
+                With MyDefs
+                    .MyViewInitialize(True)
+                    .AddOkCancelToolbar()
+
+                    Dim members As IEnumerable(Of MemberInfo)
+                    Dim member As MemberInfo
+                    Dim attr As PSettingAttribute
+                    Dim opt As PropertyOption
+                    Dim providersMembersSettings As IEnumerable(Of MemberInfo)
+                    Dim providersMembersObj As IEnumerable(Of MemberInfo)
+                    Dim providersPredicate As Func(Of MemberInfo, Boolean) = Function(m) m.MemberType = MemberTypes.Property AndAlso
+                                                                                         m.GetMemberCustomAttributes(Of Provider).ListExists
+                    Dim m1 As MemberInfo, m2 As MemberInfo
+                    Dim tmpObj As Object
+
+                    members = GetObjectMembers(MyObject, Function(m) (m.MemberType = MemberTypes.Field Or m.MemberType = MemberTypes.Property) AndAlso
+                                                                     Not m.GetCustomAttribute(Of PSettingAttribute) Is Nothing)
+                    providersMembersSettings = GetObjectMembers(MySettingsInstance, providersPredicate)
+                    providersMembersObj = GetObjectMembers(MyObject, providersPredicate)
+
+                    If members.ListExists Then
+                        For Each member In members
+                            attr = member.GetMemberCustomAttribute(Of PSettingAttribute)
+                            If Not attr Is Nothing AndAlso
+                               (attr.Address = SettingAddress.Both OrElse
+                                (
+                                    (IsSettingsForm And attr.Address = SettingAddress.Settings) Or
+                                    (Not IsSettingsForm And attr.Address = SettingAddress.User)
+                                )
+                               ) Then
+                                opt = Nothing
+
+                                If Not attr.NameAssoc.IsEmptyString Then
+                                    m1 = GetObjectMembers(MyObject, Function(m) m.Name = attr.NameAssocInstance).FirstOrDefault
+                                    If Not m1 Is Nothing AndAlso (m1.MemberType = MemberTypes.Property Or
+                                                                  m1.MemberType = MemberTypes.Field Or
+                                                                  m1.MemberType = MemberTypes.Method) Then
+                                        tmpObj = m1.GetMemberValue(MyObject)
+                                        If Not tmpObj Is Nothing Then
+                                            m2 = GetObjectMembers(tmpObj, Function(m) m.Name = attr.NameAssoc).FirstOrDefault
+                                            If Not m2 Is Nothing Then opt = m2.GetMemberCustomAttribute(Of PropertyOption)
+                                        End If
+                                    End If
+                                End If
+
+                                MyMembers.Add(New MemberOption(MyObject, member, attr, opt))
+                            End If
+                        Next
+                    End If
+
+                    .MyFieldsCheckerE = New FieldsChecker
+
+                    If MyMembers.Count > 0 Then
+
+                        Dim prov As IEnumerable(Of Provider)
+                        Dim _prov As Provider
+                        Dim si% = -1
+                        Dim i%
+                        For Each provEnum In {providersMembersObj, providersMembersSettings}
+                            si += 1
+                            If provEnum.ListExists Then
+                                For Each member In provEnum
+                                    prov = member.GetMemberCustomAttributes(Of Provider)
+                                    If prov.ListExists Then
+                                        For Each _prov In prov
+                                            i = MyMembers.FindIndex(Function(m) If(si = 0, m.Name, m.OptName) = _prov.Name)
+                                            If i >= 0 Then MyMembers(i).SetProvider(member.GetMemberValue(If(si = 0, MyObject, CObj(MySettingsInstance))), _prov)
+                                        Next
+                                    End If
+                                Next
+                            End If
+                        Next
+
+                        TP_MAIN.RowStyles.Clear()
+                        TP_MAIN.RowCount = 0
+                        For i% = 0 To MyMembers.Count - 1
+                            With MyMembers(i)
+                                .CreateControl(MyDefs.MyFieldsCheckerE, TT_MAIN)
+                                TP_MAIN.RowStyles.Add(New RowStyle(SizeType.Absolute, .ControlHeight))
+                                TP_MAIN.RowCount += 1
+                                TP_MAIN.Controls.Add(.Control, 0, TP_MAIN.RowStyles.Count - 1)
+                            End With
+                        Next
+                    Else
+                        Throw New ArgumentOutOfRangeException("Members", "Settings instance does not contain settings members")
+                    End If
+
+                    .MyFieldsChecker.EndLoaderOperations()
+
+                    Dim s As Size = Size
+                    s.Height += (MyMembers.Sum(Function(m) m.ControlHeight) +
+                                (PaddingE.GetOf({TP_MAIN},,,,,, 0).Vertical(MyMembers.Count - 1) / 2).RoundDown + MyMembers.Count - 1)
+                    s.Width = MyMembers.Max(Function(m) m.Width) + PaddingE.GetOf({TP_MAIN, CONTAINER_MAIN, CONTAINER_MAIN.ContentPanel, Me}, False).Horizontal(2)
+                    If MinimumWidth > 0 And s.Width < MinimumWidth Then s.Width = MinimumWidth
+                    Size = s
+                    MinimumSize = s
+                    MaximumSize = s
+
+                    .EndLoaderOperations()
+                End With
+            Catch ex As Exception
+                MyDefs.InvokeLoaderError(ex)
+            End Try
+        End Sub
+        Private Sub InternalSettingsForm_Closing(sender As Object, e As CancelEventArgs) Handles Me.Closing
+            TP_MAIN.Controls.Clear()
+            MyMembers.ListClearDispose
+        End Sub
+        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
+            If MyDefs.MyFieldsChecker.AllParamsOK Then
+                MyMembers.ForEach(Sub(m) m.UpdateValueByControl())
+                MyDefs.CloseForm()
+            End If
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Gfycat/Envir.vb

@@ -7,12 +7,70 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
+Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
+Imports SCrawler.Plugin.Hosts
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Gfycat
-    Friend NotInheritable Class Envir
-        Private Sub New()
+    Friend NotInheritable Class Envir : Inherits UserDataBase
+        Friend Const SiteKey As String = "AndyProgram_Gfycat"
+        Friend Const SiteName As String = "Gfycat"
+        Friend Sub New()
         End Sub
+#Region "UserDataBase Support"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As PersonalUtilities.Functions.XML.XmlFile, ByVal Loading As Boolean)
+        End Sub
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+        End Sub
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            SeparateVideoFolder = False
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            Return 0
+        End Function
+#End Region
+#Region "DownloadSingleObject"
+        Private _IsRedGifs As Boolean = False
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim urlVideo$ = GetVideo(Data.URL)
+            If Not urlVideo.IsEmptyString Then
+                If urlVideo.Contains("redgifs.com") Then
+                    _IsRedGifs = True
+                    DirectCast(Settings(RedGifs.RedGifsSiteKey).Source, RedGifs.SiteSettings).UpdateTokenIfRequired()
+                    Dim newData As IYouTubeMediaContainer = Settings(RedGifs.RedGifsSiteKey).GetSingleMediaInstance(urlVideo, Data.File)
+                    If Not newData Is Nothing Then
+                        newData.Progress = Data.Progress
+                        newData.Download(Data.UseCookies, Token)
+                        YouTubeMediaContainerBase.Update(newData, Data)
+                        DirectCast(Data, DownloadableMediaHost).ExchangeData(newData, Data)
+                        With DirectCast(Data, YouTubeMediaContainerBase)
+                            .Site = RedGifs.RedGifsSite
+                            .SiteKey = RedGifs.RedGifsSiteKey
+                            .SiteIcon = Settings(RedGifs.RedGifsSiteKey).Source.Image
+                        End With
+                    Else
+                        Throw New Exception($"Unable to get RedGifs instance{vbCr}{Data.URL}{vbCr}{urlVideo}")
+                    End If
+                Else
+                    Dim m As New UserMedia(urlVideo, UserMedia.Types.Video) With {.URL_BASE = Data.URL}
+                    m.File.Path = Data.File.Path
+                    _TempMediaList.Add(m)
+                End If
+            End If
+        End Sub
+        Protected Overrides Sub DownloadSingleObject_CreateMedia(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            If Not _IsRedGifs Then MyBase.DownloadSingleObject_CreateMedia(Data, Token)
+        End Sub
+        Protected Overrides Sub DownloadSingleObject_PostProcessing(ByVal Data As IYouTubeMediaContainer, Optional ByVal ResetTitle As Boolean = True)
+            If Not _IsRedGifs Then MyBase.DownloadSingleObject_PostProcessing(Data, ResetTitle)
+        End Sub
+        Protected Overrides Sub DownloadSingleObject_Download(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            If Not _IsRedGifs Then MyBase.DownloadSingleObject_Download(Data, Token)
+        End Sub
+#End Region
         Friend Shared Function GetVideo(ByVal URL As String) As String
             Try
                 Dim r$
@@ -33,14 +91,22 @@ Namespace API.Gfycat
                 Dim e As EDP = EDP.ReturnValue
                 If TypeOf ex Is WebException Then
                     Dim obj As HttpWebResponse = TryCast(DirectCast(ex, WebException).Response, HttpWebResponse)
-                    If Not If(obj?.StatusCode, HttpStatusCode.OK) = HttpStatusCode.NotFound Then e += EDP.SendInLog
+                    If Not If(obj?.StatusCode, HttpStatusCode.OK) = HttpStatusCode.NotFound Then e += EDP.SendToLog
                 End If
                 Return ErrorsDescriber.Execute(e, ex, $"[API.Gfycat.Envir.GetVideo({URL})]", String.Empty)
             End Try
         End Function
-        Friend Shared Function GetVideoInfo(ByVal URL As String) As IEnumerable(Of UserMedia)
-            Dim u$ = GetVideo(URL)
-            Return If(u.IsEmptyString, Nothing, {New UserMedia(u, UserMedia.Types.Video)})
+        Friend Shared Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As SFile) As IYouTubeMediaContainer
+            If Not URL.IsEmptyString AndAlso URL.Contains("gfycat") Then
+                Return New DownloadableMediaHost(URL, OutputFile) With {
+                    .Instance = New Envir,
+                    .Site = SiteName,
+                    .SiteKey = SiteKey,
+                    .SiteIcon = Nothing
+                }
+            Else
+                Return Nothing
+            End If
         End Function
     End Class
 End Namespace

SCrawler/API/Imgur/Envir.vb

@@ -7,8 +7,11 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
+Imports System.Threading
 Imports SCrawler.API.Base
 Imports SCrawler.API.Imgur.Declarations
+Imports SCrawler.API.YouTube.Objects
+Imports SCrawler.Plugin.Hosts
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Documents.JSON
@@ -18,8 +21,28 @@ Namespace API.Imgur
             Friend ReadOnly PostRegex As RParams = RParams.DMS("/([^/]+?)(|#.*?|\.[\w]{0,4})(|\?.*?)\Z", 1)
         End Module
     End Namespace
-    Friend NotInheritable Class Envir
-        Private Sub New()
+    Friend NotInheritable Class Envir : Inherits UserDataBase
+        Friend Const SiteKey As String = "AndyProgram_Imgur"
+        Friend Const SiteName As String = "Imgur"
+        Friend Sub New()
+        End Sub
+#Region "UserDataBase Support"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+        End Sub
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+        End Sub
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            SeparateVideoFolder = False
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            Return 0
+        End Function
+#End Region
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim videos As IEnumerable(Of UserMedia) = GetVideoInfo(Data.URL, EDP.SendToLog)
+            If videos.ListExists Then _TempMediaList.AddRange(videos)
         End Sub
         Friend Shared Function GetGallery(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As List(Of String)
             Try
@@ -47,7 +70,7 @@ Namespace API.Imgur
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return DownloadingException(ex, $"[API.Imgur.Envir.GetGallery({URL})]", Nothing, e)
+                Return DownloadingException_Internal(ex, $"[API.Imgur.Envir.GetGallery({URL})]", Nothing, e)
             End Try
         End Function
         Friend Shared Function GetImage(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As String
@@ -64,7 +87,7 @@ Namespace API.Imgur
                 End If
                 Return String.Empty
             Catch ex As Exception
-                Return DownloadingException(ex, $"[API.Imgur.Envir.GetImage({URL})]", String.Empty, e)
+                Return DownloadingException_Internal(ex, $"[API.Imgur.Envir.GetImage({URL})]", String.Empty, e)
             End Try
         End Function
         Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
@@ -81,11 +104,23 @@ Namespace API.Imgur
                 Return Nothing
             Catch ex As Exception
                 If Not e.Exists Then e = EDP.LogMessageValue
-                Return ErrorsDescriber.Execute(e, ex, "Imgur standalone downloader: fetch media error")
+                Return ErrorsDescriber.Execute(e, ex, $"[API.Imgur.Envir.GetVideoInfo({URL})]: fetch media error")
             End Try
         End Function
-        Private Shared Function DownloadingException(ByVal ex As Exception, ByVal Message As String,
-                                                     ByVal NullArg As Object, ByVal e As ErrorsDescriber) As Object
+        Friend Shared Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As SFile) As IYouTubeMediaContainer
+            If Not URL.IsEmptyString AndAlso URL.Contains("imgur.com") Then
+                Return New DownloadableMediaHost(URL, OutputFile) With {
+                    .Instance = New Envir,
+                    .Site = SiteName,
+                    .SiteKey = SiteKey,
+                    .SiteIcon = Nothing
+                }
+            Else
+                Return Nothing
+            End If
+        End Function
+        Private Shared Function DownloadingException_Internal(ByVal ex As Exception, ByVal Message As String,
+                                                              ByVal NullArg As Object, ByVal e As ErrorsDescriber) As Object
             If TypeOf ex Is WebException Then
                 Dim obj As HttpWebResponse = TryCast(DirectCast(ex, WebException).Response, HttpWebResponse)
                 If Not obj Is Nothing Then
@@ -97,7 +132,7 @@ Namespace API.Imgur
                     End If
                 End If
             End If
-            If Not e.Exists Then e = New ErrorsDescriber(EDP.ReturnValue + EDP.SendInLog)
+            If Not e.Exists Then e = New ErrorsDescriber(EDP.ReturnValue + EDP.SendToLog)
             Return ErrorsDescriber.Execute(e, ex, Message, NullArg)
         End Function
     End Class

SCrawler/API/Instagram/Declarations.vb

@@ -6,16 +6,15 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Cookies
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Clients.EventArguments
-Imports PersonalUtilities.Tools.Web.Cookies
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Instagram
     Friend Module Declarations
         Friend Const InstagramSite As String = "Instagram"
         Friend Const InstagramSiteKey As String = "AndyProgram_Instagram"
         Friend ReadOnly FilesPattern As RParams = RParams.DMS(".+?([^/\?]+?\.[\w\d]{3,4})(?=(\?|\Z))", 1, EDP.ReturnValue)
-        Friend ReadOnly Property DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
         Friend Sub UpdateResponser(ByVal Source As IResponse, ByRef Destination As Responser)
             Const r_wwwClaimName$ = "x-ig-set-www-claim"
             Const r_tokenName$ = "csrftoken"
@@ -46,7 +45,7 @@ Namespace API.Instagram
                 If Not wwwClaim.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_IG_WWW_CLAIM, wwwClaim)
                 If Not token.IsEmptyString Then Destination.Headers.Add(SiteSettings.Header_CSRF_TOKEN, token)
                 If Not isInternal Then
-                    Destination.Cookies.Update(Source.Cookies, CookieKeeper.UpdateModes.ReplaceByNameAll, False, EDP.SendInLog)
+                    Destination.Cookies.Update(Source.Cookies, CookieKeeper.UpdateModes.ReplaceByNameAll, False, EDP.SendToLog)
                     Destination.SaveSettings()
                 End If
             End If

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -6,14 +6,24 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
-Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
 Namespace API.Instagram
     Friend Class EditorExchangeOptions
+        <PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
         Friend Property GetTimeline As Boolean
+        <PSetting(Caption:="Get stories", ToolTip:="Download user stories")>
         Friend Property GetStories As Boolean
+        <PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")>
         Friend Property GetTagged As Boolean
-        Friend Sub New(ByVal h As ISiteSettings)
-            With DirectCast(h, SiteSettings)
+        Friend Sub New(ByVal u As UserData)
+            With u
+                GetTimeline = .GetTimeline
+                GetStories = .GetStories
+                GetTagged = .GetTaggedData
+            End With
+        End Sub
+        Friend Sub New(ByVal s As SiteSettings)
+            With s
                 GetTimeline = CBool(.GetTimeline.Value)
                 GetStories = CBool(.GetStories.Value)
                 GetTagged = CBool(.GetTagged.Value)

SCrawler/API/Instagram/OptionsForm.Designer.vb

@@ -1,135 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.Instagram
-    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
-    Partial Friend Class OptionsForm : Inherits System.Windows.Forms.Form
-        <System.Diagnostics.DebuggerNonUserCode()>
-        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            Try
-                If disposing AndAlso components IsNot Nothing Then
-                    components.Dispose()
-                End If
-            Finally
-                MyBase.Dispose(disposing)
-            End Try
-        End Sub
-        Private components As System.ComponentModel.IContainer
-        <System.Diagnostics.DebuggerStepThrough()>
-        Private Sub InitializeComponent()
-            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
-            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
-            Me.CH_GET_STORIES = New System.Windows.Forms.CheckBox()
-            Me.CH_GET_TAGGED = New System.Windows.Forms.CheckBox()
-            Me.CH_GET_TIMELINE = New System.Windows.Forms.CheckBox()
-            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
-            CONTAINER_MAIN.ContentPanel.SuspendLayout()
-            CONTAINER_MAIN.SuspendLayout()
-            TP_MAIN.SuspendLayout()
-            Me.SuspendLayout()
-            '
-            'CONTAINER_MAIN
-            '
-            '
-            'CONTAINER_MAIN.ContentPanel
-            '
-            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(260, 79)
-            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            CONTAINER_MAIN.LeftToolStripPanelVisible = False
-            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
-            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
-            CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(260, 104)
-            CONTAINER_MAIN.TabIndex = 0
-            CONTAINER_MAIN.TopToolStripPanelVisible = False
-            '
-            'TP_MAIN
-            '
-            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_MAIN.ColumnCount = 1
-            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Controls.Add(Me.CH_GET_STORIES, 0, 1)
-            TP_MAIN.Controls.Add(Me.CH_GET_TAGGED, 0, 2)
-            TP_MAIN.Controls.Add(Me.CH_GET_TIMELINE, 0, 0)
-            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_MAIN.Location = New System.Drawing.Point(0, 0)
-            TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 4
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(260, 79)
-            TP_MAIN.TabIndex = 0
-            '
-            'CH_GET_STORIES
-            '
-            Me.CH_GET_STORIES.AutoSize = True
-            Me.CH_GET_STORIES.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_GET_STORIES.Location = New System.Drawing.Point(4, 30)
-            Me.CH_GET_STORIES.Name = "CH_GET_STORIES"
-            Me.CH_GET_STORIES.Size = New System.Drawing.Size(252, 19)
-            Me.CH_GET_STORIES.TabIndex = 1
-            Me.CH_GET_STORIES.Text = "Get stories"
-            Me.CH_GET_STORIES.UseVisualStyleBackColor = True
-            '
-            'CH_GET_TAGGED
-            '
-            Me.CH_GET_TAGGED.AutoSize = True
-            Me.CH_GET_TAGGED.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_GET_TAGGED.Location = New System.Drawing.Point(4, 56)
-            Me.CH_GET_TAGGED.Name = "CH_GET_TAGGED"
-            Me.CH_GET_TAGGED.Size = New System.Drawing.Size(252, 19)
-            Me.CH_GET_TAGGED.TabIndex = 2
-            Me.CH_GET_TAGGED.Text = "Get tagged data"
-            Me.CH_GET_TAGGED.UseVisualStyleBackColor = True
-            '
-            'CH_GET_TIMELINE
-            '
-            Me.CH_GET_TIMELINE.AutoSize = True
-            Me.CH_GET_TIMELINE.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_GET_TIMELINE.Location = New System.Drawing.Point(4, 4)
-            Me.CH_GET_TIMELINE.Name = "CH_GET_TIMELINE"
-            Me.CH_GET_TIMELINE.Size = New System.Drawing.Size(252, 19)
-            Me.CH_GET_TIMELINE.TabIndex = 0
-            Me.CH_GET_TIMELINE.Text = "Get Timeline"
-            Me.CH_GET_TIMELINE.UseVisualStyleBackColor = True
-            '
-            'OptionsForm
-            '
-            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
-            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(260, 104)
-            Me.Controls.Add(CONTAINER_MAIN)
-            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
-            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
-            Me.KeyPreview = True
-            Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(276, 143)
-            Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(276, 143)
-            Me.Name = "OptionsForm"
-            Me.ShowInTaskbar = False
-            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Options"
-            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-            CONTAINER_MAIN.ResumeLayout(False)
-            CONTAINER_MAIN.PerformLayout()
-            TP_MAIN.ResumeLayout(False)
-            TP_MAIN.PerformLayout()
-            Me.ResumeLayout(False)
-
-        End Sub
-
-        Private WithEvents CH_GET_STORIES As CheckBox
-        Private WithEvents CH_GET_TAGGED As CheckBox
-        Private WithEvents CH_GET_TIMELINE As CheckBox
-    End Class
-End Namespace

SCrawler/API/Instagram/OptionsForm.vb

@@ -1,40 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Forms
-Namespace API.Instagram
-    Friend Class OptionsForm
-        Private WithEvents MyDefs As DefaultFormOptions
-        Private ReadOnly Property MyExchangeOptions As EditorExchangeOptions
-        Friend Sub New(ByRef ExchangeOptions As EditorExchangeOptions)
-            InitializeComponent()
-            MyExchangeOptions = ExchangeOptions
-            MyDefs = New DefaultFormOptions(Me, Settings.Design)
-        End Sub
-        Private Sub OptionsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
-            With MyDefs
-                .MyViewInitialize(True)
-                .AddOkCancelToolbar()
-                With MyExchangeOptions
-                    CH_GET_TIMELINE.Checked = .GetTimeline
-                    CH_GET_STORIES.Checked = .GetStories
-                    CH_GET_TAGGED.Checked = .GetTagged
-                End With
-                .EndLoaderOperations()
-            End With
-        End Sub
-        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
-            With MyExchangeOptions
-                .GetTimeline = CH_GET_TIMELINE.Checked
-                .GetStories = CH_GET_STORIES.Checked
-                .GetTagged = CH_GET_TAGGED.Checked
-            End With
-            MyDefs.CloseForm()
-        End Sub
-    End Class
-End Namespace

SCrawler/API/Instagram/SiteSettings.vb

@@ -34,49 +34,39 @@ Namespace API.Instagram
         End Property
 #End Region
 #Region "Providers"
-        Private Class TimersChecker : Implements IFieldsCheckerProvider
-            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
-            Private Property Name As String Implements IFieldsCheckerProvider.Name
-            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
+        Private Class TimersChecker : Inherits FieldsCheckerProviderBase
             Private ReadOnly LVProvider As New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral}
             Private ReadOnly _LowestValue As Integer
             Friend Sub New(ByVal LowestValue As Integer)
                 _LowestValue = LowestValue
             End Sub
-            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
-                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
                 TypeError = False
                 ErrorMessage = String.Empty
                 If Not ACheck(Of Integer)(Value) Then
                     TypeError = True
                 ElseIf CInt(Value) < _LowestValue Then
                     ErrorMessage = $"The value of [{Name}] field must be greater than or equal to {_LowestValue.NumToString(LVProvider)}"
+                    HasError = True
                 Else
                     Return Value
                 End If
                 Return Nothing
             End Function
-            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("[GetFormat] is not available in the context of [TimersChecker]")
-            End Function
         End Class
-        Private Class TaggedNotifyLimitChecker : Implements IFieldsCheckerProvider
-            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
-            Private Property Name As String Implements IFieldsCheckerProvider.Name
-            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
-            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
-                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+        Private Class TaggedNotifyLimitChecker : Inherits FieldsCheckerProviderBase
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
                 Dim v% = AConvert(Of Integer)(Value, -10)
                 If v > 0 Or v = -1 Then
                     Return Value
                 Else
                     ErrorMessage = $"The value of [{Name}] field must be greater than 0 or equal to -1"
+                    HasError = True
                     Return Nothing
                 End If
             End Function
-            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("[GetFormat] is not available in the context of [TaggedNotifyLimitChecker]")
-            End Function
         End Class
 #End Region
 #Region "Authorization properties"
@@ -270,17 +260,11 @@ Namespace API.Instagram
             Return False
         End Function
 #End Region
-#Region "Plugin functions"
+#Region "GetInstance"
         Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
-            Select Case What
-                Case Download.Main : Return New UserData
-                Case Download.SavedPosts
-                    Dim u As New UserData
-                    DirectCast(u, UserDataBase).User = New UserInfo With {.Name = Site}
-                    Return u
-            End Select
-            Return Nothing
+            Return New UserData
         End Function
+#End Region
 #Region "Downloading"
         Friend Property SkipUntilNextSession As Boolean = False
         Friend Overrides Function ReadyToDownload(ByVal What As Download) As Boolean
@@ -328,13 +312,11 @@ Namespace API.Instagram
             SkipUntilNextSession = False
         End Sub
 #End Region
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            Return UserData.GetVideoInfo(URL, Responser)
-        End Function
+#Region "UserOptions, GetUserPostUrl"
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
             If Options Is Nothing OrElse Not TypeOf Options Is EditorExchangeOptions Then Options = New EditorExchangeOptions(Me)
             If OpenForm Then
-                Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
             End If
         End Sub
         Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
@@ -342,7 +324,7 @@ Namespace API.Instagram
                 Dim code$ = DirectCast(User, UserData).GetPostCodeById(Media.Post.ID)
                 If Not code.IsEmptyString Then Return $"https://instagram.com/p/{code}/" Else Return String.Empty
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "Can't open user's post", String.Empty)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "Can't open user's post", String.Empty)
             End Try
         End Function
 #End Region

SCrawler/API/Instagram/UserData.vb

@@ -8,13 +8,14 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
 Imports System.Threading
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.Messaging
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Documents.JSON
-Imports SCrawler.API.Base
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Instagram
     Friend Class UserData : Inherits UserDataBase
@@ -77,7 +78,7 @@ Namespace API.Instagram
 #End Region
 #Region "Exchange options"
         Friend Overrides Function ExchangeOptionsGet() As Object
-            Return New EditorExchangeOptions(HOST.Source) With {.GetTimeline = GetTimeline, .GetStories = GetStories, .GetTagged = GetTaggedData}
+            Return New EditorExchangeOptions(Me)
         End Function
         Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
             If Not Obj Is Nothing AndAlso TypeOf Obj Is EditorExchangeOptions Then
@@ -139,7 +140,7 @@ Namespace API.Instagram
                     x = New XmlFile With {.AllowSameNames = True}
                     x.AddRange(PostsKVIDs)
                     x.Name = "Posts"
-                    x.Save(f, EDP.SendInLog)
+                    x.Save(f, EDP.SendToLog)
                     x.Dispose()
                 End If
             End If
@@ -182,7 +183,7 @@ Namespace API.Instagram
                 End If
                 Return String.Empty
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"{ToStringForLog()}: Cannot find post code by ID ({PostID})", String.Empty)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, $"{ToStringForLog()}: Cannot find post code by ID ({PostID})", String.Empty)
             End Try
         End Function
         Private Function GetPostIdBySection(ByVal ID As String, ByVal Section As Sections) As String
@@ -621,7 +622,7 @@ Namespace API.Instagram
                             PostsKVIDs.ListAddValue(PostIDKV, LNC)
                             PostDate = .Value("taken_at")
                             If Not IsSavedPosts Then
-                                Select Case CheckDatesLimit(PostDate, DateProvider)
+                                Select Case CheckDatesLimit(PostDate, UnixDate32Provider)
                                     Case DateResult.Skip : Continue For
                                     Case DateResult.Exit : If Not Pinned Then Return False
                                 End Select
@@ -637,7 +638,7 @@ Namespace API.Instagram
         End Function
 #End Region
 #Region "Code ID converters"
-        Private Shared Function CodeToID(ByVal Code As String) As String
+        Private Function CodeToID(ByVal Code As String) As String
             Const CodeSymbols$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
             Try
                 If Not Code.IsEmptyString Then
@@ -652,13 +653,13 @@ Namespace API.Instagram
                     Return String.Empty
                 End If
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[API.Instagram.UserData.CodeToID({Code})", String.Empty)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[API.Instagram.UserData.CodeToID({Code})", String.Empty)
             End Try
         End Function
 #End Region
 #Region "Obtain Media"
         Private Sub ObtainMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
-                                 Optional ByVal DateObj As String = Nothing)
+                                Optional ByVal DateObj As String = Nothing)
             Try
                 Dim img As Predicate(Of EContainer) = Function(_img) Not _img.Name.IsEmptyString AndAlso _img.Name.StartsWith("image_versions") AndAlso _img.Count > 0
                 Dim vid As Predicate(Of EContainer) = Function(_vid) Not _vid.Name.IsEmptyString AndAlso _vid.Name.StartsWith("video_versions") AndAlso _vid.Count > 0
@@ -737,7 +738,7 @@ Namespace API.Instagram
                     l.Clear()
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "API.Instagram.ObtainMedia2")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "API.Instagram.ObtainMedia2")
             End Try
         End Sub
 #End Region
@@ -898,38 +899,25 @@ Namespace API.Instagram
         End Sub
 #End Region
 #Region "Create media"
-        Private Shared Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
-                                              Optional ByVal SpecialFolder As String = Nothing) As UserMedia
+        Private Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
+                                       Optional ByVal SpecialFolder As String = Nothing) As UserMedia
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
             Dim m As New UserMedia(_URL, t) With {.Post = New UserPost With {.ID = PostID}}
             If Not m.URL.IsEmptyString Then m.File = CStr(RegexReplace(m.URL, FilesPattern))
-            If Not PostDate.IsEmptyString Then m.Post.Date = AConvert(Of Date)(PostDate, DateProvider, Nothing) Else m.Post.Date = Nothing
+            If Not PostDate.IsEmptyString Then m.Post.Date = AConvert(Of Date)(PostDate, UnixDate32Provider, Nothing) Else m.Post.Date = Nothing
             m.SpecialFolder = SpecialFolder
             Return m
         End Function
 #End Region
 #Region "Standalone downloader"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal r As Responser) As IEnumerable(Of UserMedia)
-            Try
-                If Not URL.IsEmptyString AndAlso URL.Contains("instagram.com") Then
-                    Dim PID$ = RegexReplace(URL, RParams.DMS(".*?instagram.com/p/([_\w\d]+)", 1))
-                    If Not PID.IsEmptyString AndAlso Not ACheck(Of Long)(PID) Then PID = CodeToID(PID)
-                    If Not PID.IsEmptyString Then
-                        Using t As New UserData
-                            t.SetEnvironment(Settings(InstagramSiteKey), Nothing, False, False)
-                            t.Responser = New Responser
-                            t.Responser.Copy(r)
-                            t.PostsToReparse.Add(New PostKV With {.ID = PID})
-                            t.DownloadPosts(Nothing)
-                            Return ListAddList(Nothing, t._TempMediaList)
-                        End Using
-                    End If
-                End If
-                Return Nothing
-            Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, $"Instagram standalone downloader: fetch media error ({URL})")
-            End Try
-        End Function
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim PID$ = RegexReplace(Data.URL, RParams.DMS(".*?instagram.com/p/([_\w\d]+)", 1))
+            If Not PID.IsEmptyString AndAlso Not ACheck(Of Long)(PID) Then PID = CodeToID(PID)
+            If Not PID.IsEmptyString Then
+                PostsToReparse.Add(New PostKV With {.ID = PID})
+                DownloadPosts(Token)
+            End If
+        End Sub
 #End Region
 #Region "IDisposable Support"
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)

SCrawler/API/LPSG/Declarations.vb

@@ -10,28 +10,41 @@ Imports SCrawler.API.Base
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.LPSG
     Friend Module Declarations
-        Friend ReadOnly Property PhotoRegEx As RParams = RParams.DM("(https://www.lpsg.com/attachments)(.+?)(?="")", 0, RegexReturn.List)
+        Friend ReadOnly Property PhotoRegEx As RParams =
+            RParams.DM("(?<=(https://www.lpsg.com|)/attachments/)([^/]+?[-\.]{1}(jpg|jpeg|gif|png|webm)\.?\d*)(?=/?"")", 0, RegexReturn.List,
+                       CType(Function(Input$) If(Input.IsEmptyString, String.Empty, $"https://www.lpsg.com/attachments/{Input.StringTrimStart("/")}"),
+                             Func(Of String, String)))
         Friend ReadOnly Property PhotoRegExExt As New RParams("img.data.src=""(/proxy[^""]+?)""", Nothing, 1, RegexReturn.List) With {
-            .Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
+                                                              .Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
         Friend ReadOnly Property NextPageRegex As RParams = RParams.DMS("<link rel=""next"" href=""(.+?/page-(\d+))""", 2)
         Private Const FileUrlRegexDefault As String = "([^/]+?)(jpg|jpeg|gif|png|webm)"
         Private ReadOnly InputFReplacer As New ErrorsDescriber(EDP.ReturnValue)
         Private ReadOnly InputForbidRemover As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, Input, Input.StringRemoveWinForbiddenSymbols(, InputFReplacer))
-        Friend ReadOnly Property FileRegEx As New RParams(FileUrlRegexDefault, Nothing, 0) With {
-            .Converter = Function(ByVal Input As String) As String
-                             Input = InputForbidRemover.Invoke(Input)
-                             If Not Input.IsEmptyString Then
-                                 Dim lv$ = Input.Split("-").LastOrDefault
-                                 If Not lv.IsEmptyString Then
-                                     Input = Input.Replace($"-{lv}", String.Empty)
-                                     Input &= $".{lv}"
-                                 End If
-                             End If
-                             Return Input
-                         End Function}
-        Friend ReadOnly Property FileRegExExt As New RParams(FileUrlRegexDefault, 0, Nothing, InputForbidRemover)
-        Friend ReadOnly Property FileRegExExt2 As New RParams("([^/]+?)(?=(\Z|&))", 0, Nothing, InputForbidRemover)
+        Private ReadOnly FileRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 0, RegexReturn.ListByMatch, InputFReplacer)
+#Disable Warning IDE0060
+        Friend Function FileRegExF(ByVal Input As String, ByVal Index As Integer) As String
+#Enable Warning
+            If Not Input.IsEmptyString Then
+                Dim l As List(Of String) = RegexReplace(Input, FileRegEx)
+                If l.ListExists(3) Then
+                    Dim ext$ = l(2)
+                    Dim f$ = l(1).StringTrim("-", ".")
+                    Input = $"{f}.{ext}"
+                End If
+            End If
+            Return Input
+        End Function
+        Private ReadOnly FileRegExExt As RParams = RParams.DM(FileUrlRegexDefault, 0, InputForbidRemover, InputFReplacer)
+        Private ReadOnly FileRegExExt2 As RParams = RParams.DM("([^/]+?)(?=(\Z|&))", 0, InputForbidRemover, InputFReplacer)
+        Friend Function FileRegExExtF(ByVal Input As String, ByVal Index As Integer) As String
+            If Index = 0 Then
+                Return RegexReplace(Input, FileRegExExt)
+            Else
+                Return RegexReplace(Input, FileRegExExt2)
+            End If
+        End Function
         Friend ReadOnly Property FileExistsRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 2)
         Friend ReadOnly Property TempListAddParams As New ListAddParams(LAP.NotContainsOnly) With {.Comparer = New FComparer(Of UserMedia)(Function(x, y) x.URL = y.URL)}
+        Friend ReadOnly Property ContentTitleRegEx As RParams = RParams.DMS("meta property=.og:title..content=""([^""]+)""", 1, InputFReplacer)
     End Module
 End Namespace

SCrawler/API/LPSG/SiteSettings.vb

@@ -25,7 +25,7 @@ Namespace API.LPSG
         Friend Sub New()
             MyBase.New("LPSG", "www.lpsg.com")
             UrlPatternUser = "https://www.lpsg.com/threads/{0}/"
-            UserRegex = RParams.DMS(".+?lpsg.com/threads/([^/]+)", 1)
+            UserRegex = RParams.DMS(".+?lpsg.com/threads/[^/]+?\.(\d+)", 1, EDP.ReturnValue)
         End Sub
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
             Return New UserData

SCrawler/API/LPSG/UserData.vb

@@ -32,6 +32,7 @@ Namespace API.LPSG
 
                 Dim NextPage$
                 Dim r$
+                Dim titleChecked As Boolean = False
                 Dim _LPage As Func(Of String) = Function() If(LatestPage.IsEmptyString, String.Empty, $"page-{LatestPage}")
 
                 Do
@@ -41,6 +42,14 @@ Namespace API.LPSG
                     UserSuspended = False
                     ThrowAny(Token)
                     If Not r.IsEmptyString Then
+                        If UserSiteName.IsEmptyString And Not titleChecked Then
+                            UserSiteName = RegexReplace(r, ContentTitleRegEx)
+                            If Not UserSiteName.IsEmptyString Then
+                                _ForceSaveUserInfo = True
+                                If FriendlyName.IsEmptyString Then FriendlyName = UserSiteName
+                            End If
+                        End If
+                        titleChecked = True
                         NextPage = RegexReplace(r, NextPageRegex)
                         UpdateMediaList(RegexReplace(r, PhotoRegEx), Mode.Internal)
                         UpdateMediaList(RegexReplace(r, PhotoRegExExt), Mode.External)
@@ -61,23 +70,24 @@ Namespace API.LPSG
                 Dim f As SFile
                 Dim u$
                 Dim exists As Boolean
-                Dim r As RParams
+                Dim r As Func(Of String, Integer, String)
+                Dim indx% = 0
                 Dim ude As New ErrorsDescriber(EDP.ReturnValue)
                 For Each url$ In l
                     If Not url.IsEmptyString Then u = SymbolsConverter.Decode(url, {Converters.HTML, Converters.ASCII}, ude) Else u = String.Empty
                     If Not u.IsEmptyString Then
                         exists = Not IsEmptyString(RegexReplace(u, FileExistsRegEx))
                         If m = Mode.Internal Then
-                            r = FileRegEx
+                            r = AddressOf FileRegExF
                         Else
-                            r = FileRegExExt
+                            r = AddressOf FileRegExExtF
                             If Not exists Then
-                                r = FileRegExExt2
-                                exists = Not IsEmptyString(RegexReplace(u, FileRegExExt2))
+                                indx = 1
+                                exists = Not IsEmptyString(FileRegExExtF(u, 1))
                             End If
                         End If
                         If exists Then
-                            f = CStr(RegexReplace(u, r))
+                            f = r.Invoke(u, indx)
                             f.Path = MyFile.CutPath.PathNoSeparator
                             f.Separator = "\"
                             If f.Extension.IsEmptyString Then f.Extension = "jpg"

SCrawler/API/Mastodon/Credentials.vb

@@ -0,0 +1,48 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
+Namespace API.Mastodon
+    Friend Structure Credentials : Implements IEContainerProvider
+        Private Const Name_Bearer As String = "Bearer"
+        Private Const Name_Csrf As String = "Csrf"
+        Friend Domain As String
+        Friend Bearer As String
+        Friend Csrf As String
+        Friend ReadOnly Property Exists As Boolean
+            Get
+                Return Not Domain.IsEmptyString And Not Bearer.IsEmptyString And Not Csrf.IsEmptyString
+            End Get
+        End Property
+        Private Sub New(ByVal e As EContainer)
+            Domain = e.Value
+            Bearer = e.Attribute(Name_Bearer)
+            Csrf = e.Attribute(Name_Csrf)
+        End Sub
+        Public Shared Widening Operator CType(ByVal e As EContainer) As Credentials
+            Return New Credentials(e)
+        End Operator
+        Public Shared Widening Operator CType(ByVal Domain As String) As Credentials
+            Return New Credentials With {.Domain = Domain}
+        End Operator
+        Private Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer
+            Return New EContainer("Item", Domain, {New EAttribute(Name_Bearer, Bearer), New EAttribute(Name_Csrf, Csrf)})
+        End Function
+        Public Overrides Function Equals(ByVal Obj As Object) As Boolean
+            If Not IsNothing(Obj) Then
+                If TypeOf Obj Is Credentials Then
+                    Return Domain = DirectCast(Obj, Credentials).Domain
+                Else
+                    Return Domain = CStr(Obj)
+                End If
+            End If
+            Return False
+        End Function
+    End Structure
+End Namespace

SCrawler/API/Mastodon/Declarations.vb

@@ -0,0 +1,14 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Mastodon
+    Friend Module Declarations
+        Friend Const MastodonSiteKey As String = "AndyProgram_Mastodon"
+        Friend ReadOnly DateProvider As New ADateTime("O")
+    End Module
+End Namespace

SCrawler/API/Mastodon/MastodonDomains.vb

@@ -0,0 +1,65 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.XML
+Namespace API.Mastodon
+    Friend Class MastodonDomains : Inherits DomainsContainer
+        Friend ReadOnly Property Credentials As List(Of Credentials)
+        Friend ReadOnly Property CredentialsTemp As List(Of Credentials)
+        Private ReadOnly CredentialsFile As SFile = $"{SettingsFolderName}\Responser_Mastodon_DomainsCredentials.xml"
+        Friend Sub New(ByVal _Instance As ISiteSettings, ByVal DefaultValue As String)
+            MyBase.New(_Instance, DefaultValue)
+            Credentials = New List(Of Credentials)
+            CredentialsTemp = New List(Of Credentials)
+            If CredentialsFile.Exists Then
+                Using x As New XmlFile(CredentialsFile,, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                    x.LoadData()
+                    If x.Count > 0 Then Credentials.ListAddList(x, LAP.IgnoreICopier)
+                End Using
+            End If
+        End Sub
+        Friend Overrides Function Apply() As Boolean
+            If Changed Then
+                Credentials.Clear()
+                If CredentialsTemp.Count > 0 Then Credentials.AddRange(CredentialsTemp)
+                CredentialsTemp.Clear()
+            End If
+            Return MyBase.Apply()
+        End Function
+        Friend Overrides Sub Save()
+            If Credentials.Count > 0 Then
+                Using x As New XmlFile With {.AllowSameNames = True}
+                    x.AddRange(Credentials)
+                    x.Name = "DomainsCredentials"
+                    x.Save(CredentialsFile)
+                End Using
+            Else
+                CredentialsFile.Delete(,, EDP.None)
+            End If
+            MyBase.Save()
+        End Sub
+        Friend Overrides Sub Reset()
+            CredentialsTemp.Clear()
+            MyBase.Reset()
+        End Sub
+        Friend Overrides Sub OpenSettingsForm()
+            Using f As New SettingsForm(Instance)
+                f.ShowDialog()
+                If f.DialogResult = DialogResult.OK Then
+                    Changed = True
+                    CredentialsTemp.Clear()
+                    If f.MyCredentials.Count > 0 Then CredentialsTemp.AddRange(f.MyCredentials)
+                    DomainsTemp.Clear()
+                    If f.MyDomains.Count > 0 Then DomainsTemp.ListAddList(f.MyDomains, LAP.NotContainsOnly)
+                End If
+            End Using
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Mastodon/SettingsForm.Designer.vb

@@ -0,0 +1,165 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Namespace API.Mastodon
+    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
+    Partial Friend Class SettingsForm : Inherits System.Windows.Forms.Form
+        <System.Diagnostics.DebuggerNonUserCode()>
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            Try
+                If disposing AndAlso components IsNot Nothing Then
+                    components.Dispose()
+                End If
+            Finally
+                MyBase.Dispose(disposing)
+            End Try
+        End Sub
+        Private components As System.ComponentModel.IContainer
+        <System.Diagnostics.DebuggerStepThrough()>
+        Private Sub InitializeComponent()
+            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
+            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
+            Dim ActionButton1 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(SettingsForm))
+            Dim ActionButton2 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim ActionButton3 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim ActionButton4 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim ActionButton5 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim ActionButton6 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Me.CMB_DOMAINS = New PersonalUtilities.Forms.Controls.ComboBoxExtended()
+            Me.TXT_AUTH = New PersonalUtilities.Forms.Controls.TextBoxExtended()
+            Me.TXT_TOKEN = New PersonalUtilities.Forms.Controls.TextBoxExtended()
+            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
+            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
+            CONTAINER_MAIN.ContentPanel.SuspendLayout()
+            CONTAINER_MAIN.SuspendLayout()
+            TP_MAIN.SuspendLayout()
+            CType(Me.CMB_DOMAINS, System.ComponentModel.ISupportInitialize).BeginInit()
+            CType(Me.TXT_AUTH, System.ComponentModel.ISupportInitialize).BeginInit()
+            CType(Me.TXT_TOKEN, System.ComponentModel.ISupportInitialize).BeginInit()
+            Me.SuspendLayout()
+            '
+            'CONTAINER_MAIN
+            '
+            '
+            'CONTAINER_MAIN.ContentPanel
+            '
+            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
+            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 361)
+            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            CONTAINER_MAIN.LeftToolStripPanelVisible = False
+            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
+            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
+            CONTAINER_MAIN.RightToolStripPanelVisible = False
+            CONTAINER_MAIN.Size = New System.Drawing.Size(384, 361)
+            CONTAINER_MAIN.TabIndex = 0
+            CONTAINER_MAIN.TopToolStripPanelVisible = False
+            '
+            'TP_MAIN
+            '
+            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
+            TP_MAIN.ColumnCount = 1
+            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 20.0!))
+            TP_MAIN.Controls.Add(Me.CMB_DOMAINS, 0, 0)
+            TP_MAIN.Controls.Add(Me.TXT_AUTH, 0, 1)
+            TP_MAIN.Controls.Add(Me.TXT_TOKEN, 0, 2)
+            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
+            TP_MAIN.Location = New System.Drawing.Point(0, 0)
+            TP_MAIN.Name = "TP_MAIN"
+            TP_MAIN.RowCount = 3
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
+            TP_MAIN.Size = New System.Drawing.Size(384, 361)
+            TP_MAIN.TabIndex = 0
+            '
+            'CMB_DOMAINS
+            '
+            ActionButton1.BackgroundImage = CType(resources.GetObject("ActionButton1.BackgroundImage"), System.Drawing.Image)
+            ActionButton1.Name = "Add"
+            ActionButton1.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Add
+            ActionButton2.BackgroundImage = CType(resources.GetObject("ActionButton2.BackgroundImage"), System.Drawing.Image)
+            ActionButton2.Name = "Delete"
+            ActionButton2.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Delete
+            ActionButton3.BackgroundImage = CType(resources.GetObject("ActionButton3.BackgroundImage"), System.Drawing.Image)
+            ActionButton3.Name = "Clear"
+            ActionButton3.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            ActionButton4.BackgroundImage = CType(resources.GetObject("ActionButton4.BackgroundImage"), System.Drawing.Image)
+            ActionButton4.Name = "ArrowDown"
+            ActionButton4.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.ArrowDown
+            ActionButton4.Visible = False
+            Me.CMB_DOMAINS.Buttons.Add(ActionButton1)
+            Me.CMB_DOMAINS.Buttons.Add(ActionButton2)
+            Me.CMB_DOMAINS.Buttons.Add(ActionButton3)
+            Me.CMB_DOMAINS.Buttons.Add(ActionButton4)
+            Me.CMB_DOMAINS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CMB_DOMAINS.ListDropDownStyle = PersonalUtilities.Forms.Controls.ComboBoxExtended.ListMode.Simple
+            Me.CMB_DOMAINS.Location = New System.Drawing.Point(4, 4)
+            Me.CMB_DOMAINS.Name = "CMB_DOMAINS"
+            Me.CMB_DOMAINS.Size = New System.Drawing.Size(378, 296)
+            Me.CMB_DOMAINS.TabIndex = 0
+            '
+            'TXT_AUTH
+            '
+            ActionButton5.BackgroundImage = CType(resources.GetObject("ActionButton5.BackgroundImage"), System.Drawing.Image)
+            ActionButton5.Name = "Clear"
+            ActionButton5.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            Me.TXT_AUTH.Buttons.Add(ActionButton5)
+            Me.TXT_AUTH.CaptionText = "Auth"
+            Me.TXT_AUTH.CaptionToolTipEnabled = True
+            Me.TXT_AUTH.CaptionToolTipText = "Bearer token"
+            Me.TXT_AUTH.CaptionWidth = 50.0R
+            Me.TXT_AUTH.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TXT_AUTH.Location = New System.Drawing.Point(4, 306)
+            Me.TXT_AUTH.Name = "TXT_AUTH"
+            Me.TXT_AUTH.Size = New System.Drawing.Size(376, 22)
+            Me.TXT_AUTH.TabIndex = 1
+            '
+            'TXT_TOKEN
+            '
+            ActionButton6.BackgroundImage = CType(resources.GetObject("ActionButton6.BackgroundImage"), System.Drawing.Image)
+            ActionButton6.Name = "Clear"
+            ActionButton6.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            Me.TXT_TOKEN.Buttons.Add(ActionButton6)
+            Me.TXT_TOKEN.CaptionText = "Token"
+            Me.TXT_TOKEN.CaptionToolTipEnabled = True
+            Me.TXT_TOKEN.CaptionToolTipText = "csrf token"
+            Me.TXT_TOKEN.CaptionWidth = 50.0R
+            Me.TXT_TOKEN.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TXT_TOKEN.Location = New System.Drawing.Point(4, 335)
+            Me.TXT_TOKEN.Name = "TXT_TOKEN"
+            Me.TXT_TOKEN.Size = New System.Drawing.Size(376, 22)
+            Me.TXT_TOKEN.TabIndex = 2
+            '
+            'SettingsForm
+            '
+            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
+            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
+            Me.ClientSize = New System.Drawing.Size(384, 361)
+            Me.Controls.Add(CONTAINER_MAIN)
+            Me.Icon = Global.SCrawler.My.Resources.SiteResources.MastodonIcon_48
+            Me.MinimumSize = New System.Drawing.Size(400, 400)
+            Me.Name = "SettingsForm"
+            Me.ShowInTaskbar = False
+            Me.Text = "Mastodon domains"
+            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
+            CONTAINER_MAIN.ResumeLayout(False)
+            CONTAINER_MAIN.PerformLayout()
+            TP_MAIN.ResumeLayout(False)
+            CType(Me.CMB_DOMAINS, System.ComponentModel.ISupportInitialize).EndInit()
+            CType(Me.TXT_AUTH, System.ComponentModel.ISupportInitialize).EndInit()
+            CType(Me.TXT_TOKEN, System.ComponentModel.ISupportInitialize).EndInit()
+            Me.ResumeLayout(False)
+
+        End Sub
+        Private WithEvents CMB_DOMAINS As PersonalUtilities.Forms.Controls.ComboBoxExtended
+        Private WithEvents TXT_AUTH As PersonalUtilities.Forms.Controls.TextBoxExtended
+        Private WithEvents TXT_TOKEN As PersonalUtilities.Forms.Controls.TextBoxExtended
+    End Class
+End Namespace

SCrawler/API/Mastodon/SettingsForm.resx

@@ -0,0 +1,292 @@
+<?xml version="1.0" encoding="utf-8"?>
+<root>
+  <!-- 
+    Microsoft ResX Schema 
+    
+    Version 2.0
+    
+    The primary goals of this format is to allow a simple XML format 
+    that is mostly human readable. The generation and parsing of the 
+    various data types are done through the TypeConverter classes 
+    associated with the data types.
+    
+    Example:
+    
+    ... ado.net/XML headers & schema ...
+    <resheader name="resmimetype">text/microsoft-resx</resheader>
+    <resheader name="version">2.0</resheader>
+    <resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
+    <resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
+    <data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
+    <data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
+    <data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
+        <value>[base64 mime encoded serialized .NET Framework object]</value>
+    </data>
+    <data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+        <value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
+        <comment>This is a comment</comment>
+    </data>
+                
+    There are any number of "resheader" rows that contain simple 
+    name/value pairs.
+    
+    Each data row contains a name, and value. The row also contains a 
+    type or mimetype. Type corresponds to a .NET class that support 
+    text/value conversion through the TypeConverter architecture. 
+    Classes that don't support this are serialized and stored with the 
+    mimetype set.
+    
+    The mimetype is used for serialized objects, and tells the 
+    ResXResourceReader how to depersist the object. This is currently not 
+    extensible. For a given mimetype the value must be set accordingly:
+    
+    Note - application/x-microsoft.net.object.binary.base64 is the format 
+    that the ResXResourceWriter will generate, however the reader can 
+    read any of the formats listed below.
+    
+    mimetype: application/x-microsoft.net.object.binary.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
+            : and then encoded with base64 encoding.
+    
+    mimetype: application/x-microsoft.net.object.soap.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Soap.SoapFormatter
+            : and then encoded with base64 encoding.
+
+    mimetype: application/x-microsoft.net.object.bytearray.base64
+    value   : The object must be serialized into a byte array 
+            : using a System.ComponentModel.TypeConverter
+            : and then encoded with base64 encoding.
+    -->
+  <xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
+    <xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
+    <xsd:element name="root" msdata:IsDataSet="true">
+      <xsd:complexType>
+        <xsd:choice maxOccurs="unbounded">
+          <xsd:element name="metadata">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" />
+              </xsd:sequence>
+              <xsd:attribute name="name" use="required" type="xsd:string" />
+              <xsd:attribute name="type" type="xsd:string" />
+              <xsd:attribute name="mimetype" type="xsd:string" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="assembly">
+            <xsd:complexType>
+              <xsd:attribute name="alias" type="xsd:string" />
+              <xsd:attribute name="name" type="xsd:string" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="data">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+                <xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
+              <xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
+              <xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="resheader">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" />
+            </xsd:complexType>
+          </xsd:element>
+        </xsd:choice>
+      </xsd:complexType>
+    </xsd:element>
+  </xsd:schema>
+  <resheader name="resmimetype">
+    <value>text/microsoft-resx</value>
+  </resheader>
+  <resheader name="version">
+    <value>2.0</value>
+  </resheader>
+  <resheader name="reader">
+    <value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <resheader name="writer">
+    <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
+  <metadata name="TP_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
+  <assembly alias="System.Drawing" name="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
+  <data name="ActionButton1.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6
+        JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAADmUlE
+        QVRIS62WWWxMURjHL220JW1HausmlFrDFKUhnUGH6bRFzJ2idImlC0Vp2mlji1A8iNhCPIjIRES8EU+W
+        h2oEtbSDTk3HNNM7S01VKsXjkb/vXBo3k1Ee7sMvmZzzzf//ne/+z50RAAxL1MUIG4G/YAv3HSVhF5Vw
+        IYNdz3LadVj9RgdTB+HQYYPHIJuE1ocSdlEJFzG+1bPRLQLinglIeCkg+XUkKvz56hnkOfQs/rmA8S9H
+        YEp7FDI64tAQtKhnsMapZ7zzNHsUFnbGY4VzIk70l6hnIH4wsDR7NBZ3apDrSqL5T8eFgUr1DLZ78lim
+        Q4N8VzK29MxEpZSBa4M16hnU+c3M9CEFpdJsVHsXos63DDcHrf9nQEXD5VymwW/5USLNwl5vJhp7dTgW
+        NML2pR7jbsUMS+KdMTa5Q8NQxinfBU4dRFcOyjy52OtbhwOBDTgZLKPPmTgY0ON4MBdNfSbYBupxY8Aq
+        G10dqMG5/nIc7ytGQ6CQRliAamkTN/g1Ai4e95Qy3iogpX0UtBRDnhRzdxq2SXOxz5eFQ70rScCEU335
+        ssGxj0YS06HSm4GN3ekwdE2C1hGH1LZR0JDOJof5jwHvnIvzTa0jlooTYfktvt+fhcOBHDQFTWRgxJGP
+        ObAGsulZLMLWnjlY756K5c4JmNcRi6T2SGheCIihS2l5ozAo6NRhMolnUAcGV6IcwwqvFrX+JTjYuwKH
+        SfRAYDms/mzs9y1GFe2VSnOw1j0FejqpLN4WCX4ZufiIBwLMLxQGm12rsLQzgWKYgmLPLNTQw6ynpDSS
+        IBet8y+TqaVRVdFIeJrWuCcj+/0EzH43BomvIhBLI45uFiDcJ+6QwROFwa6+Amb9bGFNg6Xs9Ncd7Oy3
+        Knb2eyU7/20nu9y/m136tIvEl6BC0qKoZwby3alo9JVhj7T5R7m/kJVIIityi8zyXmTiW+I10SqyIQNb
+        uIgNwYuuf25kFd75KPKkI49OmUWnrfYWyXv/wBb2cijhhVf6a9lGei65XclYRDd6mj0GWz2iLBJaH0rY
+        RSVc5Eywmhm7kuQXHX+bJlBStrh+zTi0PpSwi0q4yNFAOVvgiEcKJWUsxZn/NhT+znlofShhF5VwkRpv
+        MUtti4KGYjj6sYCIh5QSu4oG27stjItHU+cjeQzvkcFzFQ2KnSKLoc4FukDCXeI2GbSoaFD4ziyPxNxK
+        0AUyNxOP1DOwcaG/8I+/LRB+At7psBnyDBG0AAAAAElFTkSuQmCC
+</value>
+  </data>
+  <data name="ActionButton2.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABGdBTUEAALGPC/xhBQAAABl0RVh0U29m
+        dHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAVoSURBVEhLhZVrTJNXGMdfrtNSQIoadKRz2o0CorU3
+        WkDIVBRaaGNbwAteh+AARRQlitEYTTRekiX7sH3YPmyZH9wtziybigLRCWTaCW5sCBWhlrb0Ci9zSxbo
+        2f+UliGX7SS/tO85z/k9T57zXhhCCPO7Wh3VIhB83JKQ0Nu4bNlHm5YseZ1hmHC69n+Y5HLFcz7/ft/S
+        pY+vr1hhwL4oEBJcZ0x793If5uZ+1VNfT/qvXCHP6+p8tzMymqRxcW8hMGKqbDo9MlmWddu2AfbiRTJ6
+        +TIZKC52fyAUVi2JiYkLJmGaBYIPnx4+TPrOnCH9p08TC4LNx46RWwrF/ZXR0W/PleRZZuY669atZvbS
+        JcJiL9vQQEZPnSKmwkLPjcTE97GPB8KZlvh4C5X31dWRgRMniAVBtvPnyWB9ve+2XP7jmtjYpOlJTOnp
+        G60lJRZaOZWPQs4ePUpGUZh3xw7SnJDQhT0KEM3c5fOv9paVkX4kMAPL8ePEig1D584RG9rVpFS2rY6J
+        EQaTmKTSjbbiYsvIhQuERTGjKIrFvtHaWjK8fz9plsudexYu/BLxKsBj9ALBGzel0vt9e/b4XiBoENhQ
+        zRDOxIWWOY4cIS0KRZs4Nja5QyLJtRoM1pGzZ/0tYVExi/ayNTVkBPJ76enuJA7nM4j3gVWAHjgTIYqL
+        E96SStvMu3YR64EDxF5dTYYOHSJOJPNA5Kiu9rUrlZ1mrdbCnjzpr5jFGotYtqpqQi6TuVM4nKvwlYHU
+        gDzU31OMSGl8fPJtsbjVsn27z15RQRzAVVlJ3BB4kcx78CAZQbUjVIxrFtd+OdrbmpHhEXG5VE4rTwHz
+        wMRdFDw4jEgFj5dyRyRqsxYVEcfu3cQFPPv2ES8qHEbCYRzgsFZLvO+8Q7xKJXGDVoXCK46Ovob95YBW
+        Ph/8+xwE/wSTyHi81OZVq9qsGs2Ye8sW4srPJy6JhDgTE4kzOpo4IyKIMyyMOLhcX9Py5R4lj0cPtAKs
+        BBwwKfc7p174J5BEhHY9FIk6bBDaIRuiQkDFfsLDSbdU+pdBKPwe8e+BNDBD7vdNn6BYd+6stK5da7bP
+        nz9TDujcoEAw1lJY+CyFz9dCHDubnDJjwltRccS5fr3TjurnlIMBYE5NJY8Nhq7SrCwREsz6xL9y4S4v
+        b3Bt2uSyR0XNkDvQe9ouKu8HvaGh5FfQIxL5OgyG30qUStqmGUkm/3jKy0+48vLcs1XuiI8nL/Ly/rYl
+        JfmovCcgN4JW+l8iGe8oKuoqzcyckSQob3CpVB47l+sXv9KWxYtJt0r1x9ns7HZjQYHNnJxMfoH0EXgA
+        7oFm0CmTjRsNhs6Na9bQF+Tkq57xlJXVu9Rqz9Bs8kWLSG9BwcsqieQONlXnpaaWdul0z7rR+6C8CTSC
+        m8Aol4+36/XGT7VaevCRIIRx6/WWoQULZq2cyveLxY0IrAT0IHm1OTmZT3Q6U2da2qT8B/Ad+BZ05OSM
+        GXW6p4hdBiIZZ1FRt5vPn6vyuwiqCsj9Xyq6qXbDBkWnXm/6OS3NN1X+dUgIeZSdPXZPoxlEXC6IY9pL
+        S7faNBqXC9Iplf95YBb5ZF+RpGbdunQcbO/D1avJ9YC8LT19/Iv8/BeqpKRPEDORAGNeY3HxSYtG43Eq
+        FL5etfpljUzWhPlZ5VOTlGVliR+hHUbs+0mpHP9GpRqM5XAuY20zmGgRRohYKIx9rNd/3qfTOa7l5uLu
+        C63BvARw6fp0eRCMyBslJe8+2bx58EFhoVMlFNJvgQ4kgggQEgykvV0ApEAd+J3z8Z8KxmuA3pr0zikA
+        b4LJZ2FqYBigFdOPNf0NC679Fxi0OPr+XxiAJgwURph/AJfOQQebMR8TAAAAAElFTkSuQmCC
+</value>
+  </data>
+  <data name="ActionButton3.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
+        xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go
+        tbbB43rK5xSAQq1VYFtmeQBoqZTSreVZvgTknM8yyyjA/qodsDF9gspD2Bj6B+DH+NqzhQQAG+POMnSX
+        AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
+</value>
+  </data>
+  <data name="ActionButton4.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAYAAAD0eNT6AAAABGdBTUEAALGPC/xhBQAAE65JREFUeF7t
+        3X2sJWddB/DdLi2lQG2hdOHuvfM887J7Cxca4ELTQMDWKigIFpBAEAgi9g+CJpJo9Q8NJhgBiYZIYspL
+        GlAKCkhEC4KgQlsLQkqhKi/lrYWWlxaw3dLddrerz/Q89+7dc2fbfTn3npf5fJJv2rS758z85nnOzJz5
+        nZktAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMK3O3r79wVUIz65jfGNVxI/VIX69CvGO9M//a9P+e8o3B/8v
+        vKn9s+3fyX8dAJgmaWd+fl3E96Wd/E9XdvZHkfbvXNa+Rn45AGCS3bvjj/E/h3box5OrmxjPyy8PAEyS
+        XXO7zqhCeH/HDnwUOdCE+J6zdux4eH47YIrEGE8uy/Ls9Bnx/LooL0oH9b9Th/I1TVG+rCqKC+q6Xsh/
+        FJgmO8vy6WknfdPQTnsjckMdwlPy2wITLO3wF6si/lGas1ekuXvX0Fzuyg9S3psOCl6qDwimQB3ji9Ok
+        3btmEm907kpnEa/Mbw9Mlq1pB/6cdHZ/ZcfcPZrcXoXyrVVVFfl1gUmSdsS/libqPUMTd5NSvjktwrbB
+        kgDjVi1UT26K+Nnu+XrMuaud60uPWHpIfhtg3JqyfEaanHcPTdZNTRPCPy4uLj40LxIwBudt2fKAtOP/
+        0zQnN+5koIg3tpca81sC49J+LZcm5a3rJulYEq6LSV40YBOFEB6V5uFV6+flRiTsSwf9r81vDYzBCSO4
+        vjfq/KAuiqfm5QM2QRPjuWnubUbz71DCn6W33zpYCmDT1EX5m92Tcuy5q47xFXkxgQ3UduqnOXfn0Bzc
+        xJSvz4sCbIb2pzlp8v1w/WScnKSzkjekRT1hsMTAKC0vL5/Ydud3zb1NT1FelBcL2GiDm3d0TMTJy0ea
+        pjk1LzYwAu3NvtLc+uTQXBtn7tYYCJtja/vQno5JOJFpQrzWb4hhNJoQnpjm1Q3D82wCcnNRFKfnxQQ2
+        Qttk1zH5JjzhFmcIcHzyzb6O5aFem5J0sP/OvKjARmg7b7sm3xRkT3vDorwawJHb1t6Ep2NOTVoOtDch
+        yssMjFr6IPh8x8SbnsT4lrQamgPhCMzPzz+sifHjnXNpMnN5XnRglJaWlk5KE2z/0ISbxnzQQ0bgvlXz
+        1ePSXPnG0NyZ+DRF8Zi8CsCo7Azh0V0TbkrzRc2B0G3wIJ9429CcmZLce4MgYJTyff87JtzU5uayLM/J
+        qwcM7vD5+jQ3DgzNlWnKDXldgFFJZwW/2jHZpj1727uZ5VWE3mofqJXmw4eG5sdUpqqqXXm1gFGoQnhJ
+        12SbgRxoYvzjtIruK04vxRjPSvPgK0PzYmqTPqtemVcNGIU6xgu7JtusJH1ovH9ubu6UvLrQC2ncPyuN
+        /58Mz4fpTvnmvHrAKJQL5dO6J9ssJXxucWFhLq8yzLKtaUf5h2ncb9zz+8eUKsYP53UERmHX/PyOrsk2
+        g7nJDUWYZUuPWHpIE8oPdIz92UiMn86rCoxIOmOYta8KD5uftk2Peb1hZtTzdVOHcF3HmJ+ZVCF+Ia8u
+        MCppcl0+PNlmOG1zYPtYYc2BzIQ0np+ZxvWPh8b5LObqvMrAqEzRo4BHmctijCfnEsBUqkP5u2ksz8Kd
+        PI8g5SfyagOj0jbIpQk2c01DR5Brmh3NfC4DTI324LWO8V0dY3pm48mAsEGm7OEgo0sRb9wZ4+NzGWDi
+        lWUZ0ti9Zt1YnvUU8fdyCYBRmsFbAh9xqhDvqEN4Xi4FTKz8s93vD4/hPiSdpJyXywCMWPtrgKuGJ12P
+        ck/6gPmDXAuYOHVR/lY6UN3XMXb7kDv17MAGqhaqJ6WJ1sdegDUJ726a5oG5JDB2917vL+Kl3eO1N/lQ
+        LgewUdIO8E0dk69vubosy+25JDA2bYNuFeJnOsZovxLjhbkkwEZZXl4+0QfOvfl2Ogg4O5cFNl1dFE9N
+        4/B7Q+Oyj7mh/VzKZQE2UtM0j6iL+LWOidizhN3OPBiHuigvSmPwrvVjsn9pQnh1LguwGQa3Fo3fHp6M
+        Pcw97c1WcllgQy0tLZ2UDr7/qmMc9jJNiF/WkwNjMHhQ0GzfX/yIU8RLfRCxkdq+kzTfrugcf/3MgZ1l
+        +fRcHmCztU8Yq2P8h47J2cdcpTmQjdCE8IQ0vnzjdkjKP8nlAcZoWxXin3dP0n4l1eGb9UL92FwXOG51
+        Ub48ja09w2Otz2nvTJpKs21QIWDs0lnKb6TJqTEphN3NQvncXBY4VtvSju4N3WOs17l6cXHxoblGwKRo
+        r8mlHeAtHZO2b9mfDohem8sCR2XX3K4z0hj65NCYklSTGONpuUzApNlVFFWaqP81NHF7mvD2tnM7lwbu
+        V/vwqTR2vrV+LPU7VSjf4ff+MAU0B65NeWVd12fm0sBhpTnzosHDp7rGUV8T9lVFvDiXCJgSrmEezDea
+        onhMrgsM25rmyuvSODkwNG56nvZyYvi5XCNg2mgOXM3tVVH9ci4L3KtpmlN9W7Y+VYhfiEkuEzCt8n3L
+        fzA8yXuY/b7OZEVZlovt3ew6xknf8965ublTcpmAaac5cG3C2zQ09Vv7bVAaC/+7fmz0Og6QYVZpDlyT
+        GD/dPlgpl4b+2Nru5NIYuGfdmOhxmhB/VBblL+QaATNKc+DBfH1nCI/OdWHGtTewSdv874fGgIT4xfYb
+        wlwmYNZpDlzNbVUIz85lYUblJ2i6BDacGP/u7O3bH5zLBPSF5sDV7K+L+Nu5LMyYtJP7xbSNfzy0zfue
+        A+03gak8WwdVAnpHc+CaxHiJ5sCZsnK9f/+6bd3v3JZ2/r+SawT0mebAg0kfjB93v/Pp136t3X693bWN
+        e56v6nsBhmkOXE24Ph0EnJXrwpSp63qhDuXnu7dtn1P+U1VVP5PLBHAozYGDtD+LchvU6TN4Iqa+lqGs
+        XO8/YVAlgMPQHLiSsC+dNb0ml4UJVxflRWm73b1+O/Y5YXcVwvNziQDun+bANYnxkvO2bHlALg0TJsZ4
+        cl3ESzu3Xa8Trm+KYimXCeDIaQ48mKqIH9McOHl2zc/vaIr42a5t1vN8tCiK03OZAI6J5sCVFPFr7QNk
+        cl0Ys3yp6nvrtlO/s3K9f9ugSgDHSXPgILk58PxcFsYkX+93J8s1qUK8oynKF+YSAYyO5sCVhH3pgOjV
+        uSxsoqZpHpjq//bu7dLjFPHGND+Xc5kARk9z4JrE+JZUEl+1bpLFhYW5VPf/WLcd5N/ruj4zlwlg42gO
+        PCQfdXOVjdeE8MRU6xuGai9uXw2MgebA1YTrFkMoc10YsaYoX5rqfOf6uvc6e9LO/xW5RACbT3Pgam5N
+        B0Q/m8vCCLT3XnCQ2ZXwnWqhenIuE8D4aA5czV3OykZj19yuM1I9PzlUXwnhirIst+cyAYyf5sA1GTQH
+        uu/6MdoZ4+NTHb+1rq59j+v9wKTSHHhIPtI0zam5NByhNH5enGr306Fa9j1720ttuUQAE0tz4Epi/FJM
+        cl24b8ZNd25KdTk31whg8mkOXEm4pX1EbS4LHebn5x+WdnIf765fr3NVCOFRuUwA00Nz4Gr21kX58lwW
+        1qjmq8el+nxjqF4S4yVLS0sn5TIBTB/NgWuiOfAQTVE+J9XltnV16nXCvqqIF+cSAUw3zYGH5INnb9/+
+        4Fyavtra7uRSLe4Zqk3f88MmxvNyjQBmhiavg/liVVVFrkuvLC4uPjSt/4eG6iEhXlOWZchlApg9mgNX
+        c3P6wD8nl6UXqvlqZ1rv/xmqQ+/ThPJv5ufnH5TLBDC7NAeuZm97n/tclplWhfCstL4/GVr/nsf1fqCH
+        NAeu5kB7aSSVZFabA13v786tVVFckGsE0C+aAw8mnSG/f25u7pRcmpnQbt8mlB/oWt8+pwnxWk+PBNAc
+        uJoqxC/MSnNgs7BQ1yFc17WePc97Z+1AD+C4aA5czU3T/qjXdED3zLQePx5ar75nf77ev3VQJQBWaQ5c
+        zZ4qhJfkskyVuigvapvbOtapt2lC/FFTls/IJQKgi+bA1aw0B07FGWOM8eQ6xnd1rEe/E+OX2jGdywTA
+        fdEcuCYx/u2kXzNudjTz6az/c53L3+NUMX7YI6EBjp7mwIO5pt3J5rpMlHKhfFpavu8PLW/fM+s/7QTY
+        eJoDV/PduiiWc1kmwuB6f7x7aDn7ntvrGC/MJQLgeGgOHKQK8Y46hOflsoxN0zQPbEJ8Z9cy9jxf3RnC
+        o3OZABgFzYGrab9efl0qyViaAxcXFubSgchnOpar77k8xnhaLhMAo6Q5cE2K+L7NfoBMHcJT0nvfvG5Z
+        +h3X+wE2iebAg7m6LMvtuS4bKl/v14txSMLuKoQX5BIBsBk0B64kfCfV4om5LCN33pYtD3DA1ZVwfb1Q
+        PzaXCYDNpDlwJWH3RnSe75rbdUZ6/X9b/369z0eLojg9lwmAcdAcuJqV5sCRaEJ4QnrNbw+9R9+zcr1/
+        26BKAIyV5sBDcll7W95cmmPSPocgvc6dQ6/b9+xpivJluUQATBDNgQdz1TE2B6phV4p446TdhAmAIens
+        9ZXpQ1tz4KA58Am5LPfrrB07Hp7+zr90v1Z/UxXxU3Vdn5nLBMAk0xy4krC7WSifm8tyWGVZnl2F+M3u
+        1+hxYrxkeXn5xFwmAKaB5sDV7E9nsRfnsqyTdnIvGtxiuPPv9jV7Ul1ekUsEwLTRHHgwVSjfsbS0dFIu
+        TWtr+6uB9P8ODP/Znue7ZVmek2sEwBTT2Laa8sr2enb7jHoHRl0JV8QYH5nHDQCzwJ0DV/P1tKO7vuO/
+        9zpVKN/qej/AjNIcKB3Z24TyVXmIADCrNAfKmtzUxHhuHhoAzDrNgZJyVQjhUXlIANAjmgN7m/Du471d
+        MgBTzp0D+5Sw777uiQBAz2gO7EPCLSnn500OAAOaA2c615RlGfKmBoBDaQ6cvTQhvmd+fv5BeRMDwGFp
+        DpyJuN4PwDHQHDjVubUqigvypgSAo6M5cPrShHjtYghl3oQAcGw0B05Rivi+ubm5U/KmA4Djozlw4rM/
+        X+/fOthiADA6mgMnME2IP2rK8hl5GwHAxtAcOFH5SozxrLxpAGBjaQ4cf6oYP9w0zal5kwDA5tAcOLYc
+        aC/FpE1wwmBLAMAm0xy46bk91fvCXH4AGCvNgZuRIn6tKYrH5JoDwGTQHLihuTzGeFouNQBMFs2BI4/r
+        /QBMB82Bo0rYXYXwglxWAJh8mgOPN+H6eqF+bC4nAEwVzYHHkiL+c1EUp+caAsB00hx4FInxLalk2waV
+        A4AppznwfrOnLsqX53IBwOzQHHiYFPHGaqF6Ui4TAMwezYGHpirip+q6PjOXBwBmmubANjFesry8fGKu
+        CQD0Q4+bA/dWMf56LgMA9E8PmwO/W5blOXn1AaC/+tMcWF4ZY3xkXm0AYOabA2O8ZGlp6aS8ugDAGrPY
+        HLi3CeWr8voBAIczQ82BN6UDmnPzagEA92f6mwPLz1dVVeTVAQCO1LQ2B1Yh/PX8/PyD8moAAEdrupoD
+        w76qiBfnRQcAjtMUNAeGW1LOz8sLAIzKBDcHXlOWZciLCQCM2gQ2B142Nzd3Sl48AGCjTEhz4H7X+wFg
+        k425OfDWqqh+Pi8KALDJtqWDgDemHfKBoR30hqUJ8dqY5PcHAMalKcrnpJ3z94Z31qNO+/t+1/sBYIKk
+        k/LT6hD+Mu2oR/4rgXTW/+X02r+U3woAmDTtz/GaIv5F2nH/ZHhHfpS5J+Vf01n/S9LLbhu8OgAw0dpb
+        8TYL5XPTmfvb0o78v/MOvWtHvybtzXzKT1Qx/n5d1wv5pQCAaXXvAUFRLLXd+3WMFzZF+cKUl7X/rIri
+        gsWFhbn8RwEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+        AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA6LEtW/4flgYiLD1qeX0A
+        AAAASUVORK5CYII=
+</value>
+  </data>
+  <data name="ActionButton5.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
+        xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go
+        tbbB43rK5xSAQq1VYFtmeQBoqZTSreVZvgTknM8yyyjA/qodsDF9gspD2Bj6B+DH+NqzhQQAG+POMnSX
+        AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
+</value>
+  </data>
+  <data name="ActionButton6.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
+        xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go
+        tbbB43rK5xSAQq1VYFtmeQBoqZTSreVZvgTknM8yyyjA/qodsDF9gspD2Bj6B+DH+NqzhQQAG+POMnSX
+        AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
+</value>
+  </data>
+</root>

SCrawler/API/Mastodon/SettingsForm.vb

@@ -0,0 +1,154 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Forms
+Imports PersonalUtilities.Forms.Controls.Base
+Namespace API.Mastodon
+    Friend Class SettingsForm
+        Private WithEvents MyDefs As DefaultFormOptions
+        Friend ReadOnly Property MyCredentials As List(Of Credentials)
+        Friend ReadOnly Property MyDomains As List(Of String)
+        Friend Sub New(ByVal s As SiteSettings)
+            InitializeComponent()
+            MyCredentials = New List(Of Credentials)
+            If s.Domains.Credentials.Count > 0 Then MyCredentials.AddRange(s.Domains.Credentials)
+            MyDomains = New List(Of String)
+            MyDomains.ListAddList(s.Domains)
+            MyDefs = New DefaultFormOptions(Me, Settings.Design)
+        End Sub
+        Private Sub MyForm_Load(sender As Object, e As EventArgs) Handles Me.Load
+            With MyDefs
+                .MyView = New FormView(Me, Settings.Design, "MastodonSettingsForm")
+                .MyView.Import()
+                .MyView.SetFormSize()
+                .AddOkCancelToolbar()
+                RefillList()
+                .EndLoaderOperations()
+            End With
+        End Sub
+        Private Sub SettingsForm_Disposed(sender As Object, e As EventArgs) Handles Me.Disposed
+            MyCredentials.Clear()
+            MyDomains.Clear()
+        End Sub
+        Private Sub RefillList()
+            CMB_DOMAINS.Items.Clear()
+            If MyDomains.Count > 0 Then
+                MyDomains.Sort()
+                CMB_DOMAINS.BeginUpdate()
+                CMB_DOMAINS.Items.AddRange(MyDomains.Select(Function(d) New ListItem(d)))
+                CMB_DOMAINS.EndUpdate(True)
+            End If
+        End Sub
+        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
+            ApplyCredentials()
+            If MyCredentials.Count > 0 Then MyCredentials.RemoveAll(Function(c) c.Domain.IsEmptyString Or c.Bearer.IsEmptyString Or c.Csrf.IsEmptyString)
+            If MyDomains.Count > 0 Then
+                If MyCredentials.Count > 0 Then
+                    MyCredentials.RemoveAll(Function(c) Not MyDomains.Contains(c.Domain))
+                Else
+                    MyCredentials.Clear()
+                End If
+            End If
+            MyDefs.CloseForm()
+        End Sub
+        Private Sub CMB_DOMAINS_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As ActionButtonEventArgs) Handles CMB_DOMAINS.ActionOnButtonClick
+            Try
+                Dim d$
+                Dim i% = -1
+                Select Case e.DefaultButton
+                    Case ActionButton.DefaultButtons.Add
+                        d = InputBoxE("Enter a new domain using the pattern [mastodon.social]:", "New domain")
+                        If Not d.IsEmptyString Then
+                            If MyDomains.Count > 0 Then i = MyDomains.IndexOf(d)
+                            If i >= 0 Then
+                                MsgBoxE({$"Domain '{d}' already exists", "Add domain"}, vbExclamation)
+                                If i <= CMB_DOMAINS.Count - 1 Then CMB_DOMAINS.SelectedIndex = i
+                            Else
+                                ApplyCredentials()
+                                ClearCredentials()
+                                MyDomains.Add(d)
+                                _Suspended = True
+                                RefillList()
+                                _Suspended = False
+                                i = MyDomains.IndexOf(d)
+                                If i.ValueBetween(0, CMB_DOMAINS.Count - 1) Then
+                                    CMB_DOMAINS.SelectedIndex = i
+                                Else
+                                    _LatestSelected = -1
+                                    _CurrentCredentialsIndex = -1
+                                    _CurrentDomain = String.Empty
+                                End If
+                            End If
+                        End If
+                    Case ActionButton.DefaultButtons.Delete
+                        If _LatestSelected >= 0 Then
+                            d = CMB_DOMAINS.Items(_LatestSelected).Value(0)
+                            If Not d.IsEmptyString AndAlso MsgBoxE({$"Are you sure you want to delete the [{d}] domain?",
+                                                                   "Removing domains"}, vbYesNo) = vbYes Then
+                                i = MyDomains.IndexOf(d)
+                                Dim l% = _LatestSelected
+                                If i >= 0 Then
+                                    ClearCredentials()
+                                    MyDomains.RemoveAt(i)
+                                    _Suspended = True
+                                    RefillList()
+                                    _Suspended = False
+                                    If (l - 1).ValueBetween(0, CMB_DOMAINS.Count - 1) Then
+                                        CMB_DOMAINS.SelectedIndex = l - 1
+                                    Else
+                                        _LatestSelected = -1
+                                        _CurrentCredentialsIndex = -1
+                                        _CurrentDomain = String.Empty
+                                    End If
+                                End If
+                            End If
+                        End If
+                End Select
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.LogMessageValue, ex, "API.Mastodon.SettingsForm.ActionButtonClick")
+            End Try
+        End Sub
+        Private _LatestSelected As Integer = -1
+        Private _CurrentCredentialsIndex As Integer = -1
+        Private _CurrentDomain As String = String.Empty
+        Private _Suspended As Boolean = False
+        Private Sub CMB_DOMAINS_ActionSelectedItemChanged(ByVal Sender As Object, ByVal e As EventArgs, ByVal Item As ListViewItem) Handles CMB_DOMAINS.ActionSelectedItemChanged
+            If Not MyDefs.Initializing And Not _Suspended Then
+                Dim DropCredentials As Boolean = True
+                If Not Item Is Nothing Then
+                    ApplyCredentials()
+                    _LatestSelected = Item.Index
+                    _CurrentDomain = Item.Text
+                    If MyCredentials.Count > 0 And Not _CurrentDomain.IsEmptyString Then
+                        _CurrentCredentialsIndex = MyCredentials.IndexOf(_CurrentDomain)
+                        If _CurrentCredentialsIndex >= 0 Then
+                            With MyCredentials(_CurrentCredentialsIndex) : TXT_AUTH.Text = .Bearer : TXT_TOKEN.Text = .Csrf : End With
+                            DropCredentials = False
+                        End If
+                    Else
+                        _CurrentCredentialsIndex = -1
+                    End If
+                End If
+                If DropCredentials Then ClearCredentials()
+            End If
+        End Sub
+        Private Sub ClearCredentials()
+            TXT_AUTH.Clear()
+            TXT_TOKEN.Clear()
+        End Sub
+        Private Sub ApplyCredentials()
+            Try
+                If _LatestSelected >= 0 And Not _CurrentDomain.IsEmptyString Then
+                    Dim c As New Credentials With {.Domain = _CurrentDomain, .Bearer = TXT_AUTH.Text, .Csrf = TXT_TOKEN.Text}
+                    If _CurrentCredentialsIndex.ValueBetween(0, MyCredentials.Count - 1) Then MyCredentials(_CurrentCredentialsIndex) = c Else MyCredentials.Add(c)
+                End If
+            Catch ex As Exception
+            End Try
+        End Sub
+    End Class
+End Namespace

SCrawler/API/Mastodon/SiteSettings.vb

@@ -0,0 +1,214 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports TS = SCrawler.API.Twitter.SiteSettings
+Namespace API.Mastodon
+    <Manifest(MastodonSiteKey), SavedPosts, SpecialForm(True), SpecialForm(False)>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.MastodonIcon_48
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.MastodonPic_48
+            End Get
+        End Property
+#Region "Domains"
+        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue
+        Friend ReadOnly Property Domains As MastodonDomains
+        <PXML> Private ReadOnly Property DomainsLastUpdateDate As PropertyValue
+#End Region
+#Region "Auth"
+        <PropertyOption(IsAuth:=True, AllowNull:=False, ControlText:="My domain",
+                        ControlToolTip:="Your account domain without 'https://' (for example, 'mastodon.social')"), PXML>
+        Friend ReadOnly Property MyDomain As PropertyValue
+        <PropertyOption(AllowNull:=False, IsAuth:=True, ControlText:="Authorization",
+                        ControlToolTip:="Set authorization from [authorization] response header. This field must start from [Bearer] key word")>
+        Friend ReadOnly Property Auth As PropertyValue
+        <PropertyOption(AllowNull:=False, IsAuth:=True, ControlText:="Token", ControlToolTip:="Set token from [x-csrf-token] response header")>
+        Friend ReadOnly Property Token As PropertyValue
+        Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
+            If Not PropName.IsEmptyString Then
+                Dim f$ = String.Empty
+                Select Case PropName
+                    Case NameOf(Auth) : f = TS.Header_Authorization
+                    Case NameOf(Token) : f = TS.Header_Token
+                End Select
+                If Not f.IsEmptyString Then
+                    Responser.Headers.Remove(f)
+                    If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value))
+                    Responser.SaveSettings()
+                End If
+            End If
+        End Sub
+#End Region
+#Region "Other properties"
+        <PropertyOption(IsAuth:=False, ControlText:=TS.GifsDownload_Text), PXML>
+        Friend ReadOnly Property GifsDownload As PropertyValue
+        <PropertyOption(IsAuth:=False, ControlText:=TS.GifsSpecialFolder_Text, ControlToolTip:=TS.GifsSpecialFolder_ToolTip), PXML>
+        Friend ReadOnly Property GifsSpecialFolder As PropertyValue
+        <PropertyOption(IsAuth:=False, ControlText:=TS.GifsPrefix_Text, ControlToolTip:=TS.GifsPrefix_ToolTip), PXML>
+        Friend ReadOnly Property GifsPrefix As PropertyValue
+        <Provider(NameOf(GifsSpecialFolder), Interaction:=True), Provider(NameOf(GifsPrefix), Interaction:=True)>
+        Private ReadOnly Property GifStringChecker As IFormatProvider
+        <PropertyOption(IsAuth:=False, ControlText:=TS.UseMD5Comparison_Text, ControlToolTip:=TS.UseMD5Comparison_ToolTip), PXML>
+        Friend ReadOnly Property UseMD5Comparison As PropertyValue
+        <PropertyOption(IsAuth:=False, ControlText:="User related to my domain",
+                        ControlToolTip:="Open user profiles and user posts through my domain."), PXML>
+        Friend ReadOnly Property UserRelatedToMyDomain As PropertyValue
+#End Region
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("Mastodon", "mastodon.social")
+
+            Domains = New MastodonDomains(Me, "mastodon.social")
+            SiteDomains = New PropertyValue(Domains.DomainsDefault, GetType(String))
+            Domains.DestinationProp = SiteDomains
+            DomainsLastUpdateDate = New PropertyValue(Now.AddYears(-1))
+
+            Auth = New PropertyValue(Responser.Headers.Value(TS.Header_Authorization), GetType(String), Sub(v) ChangeResponserFields(NameOf(Auth), v))
+            Token = New PropertyValue(Responser.Headers.Value(TS.Header_Token), GetType(String), Sub(v) ChangeResponserFields(NameOf(Token), v))
+
+            GifsDownload = New PropertyValue(True)
+            GifsSpecialFolder = New PropertyValue(String.Empty, GetType(String))
+            GifsPrefix = New PropertyValue("GIF_")
+            GifStringChecker = New TS.GifStringProvider
+            UseMD5Comparison = New PropertyValue(False)
+            MyDomain = New PropertyValue(String.Empty, GetType(String))
+            UserRelatedToMyDomain = New PropertyValue(False)
+
+            UserRegex = RParams.DMS("", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
+        End Sub
+        Friend Overrides Sub EndInit()
+            Domains.PopulateInitialDomains(SiteDomains.Value)
+            If CDate(DomainsLastUpdateDate.Value).AddDays(7) < Now Then UpdateServersList()
+            MyBase.EndInit()
+        End Sub
+#End Region
+#Region "GetInstance"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+#End Region
+#Region "Domains Support"
+        Protected Overrides Sub DomainsApply()
+            Domains.Apply()
+            MyBase.DomainsApply()
+        End Sub
+        Protected Overrides Sub DomainsReset()
+            Domains.Reset()
+            MyBase.DomainsReset()
+        End Sub
+        Friend Overrides Sub OpenSettingsForm()
+            Domains.OpenSettingsForm()
+        End Sub
+#End Region
+#Region "Update"
+        Friend Overrides Sub Update()
+            If _SiteEditorFormOpened Then
+                Dim tf$ = GifsSpecialFolder.Value
+                If Not tf.IsEmptyString Then tf = tf.StringTrim("\") : GifsSpecialFolder.Value = tf
+            End If
+            MyBase.Update()
+        End Sub
+#End Region
+#Region "UserOptions"
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse (Not TypeOf Options Is Twitter.EditorExchangeOptions OrElse
+                                          Not DirectCast(Options, Twitter.EditorExchangeOptions).SiteKey = MastodonSiteKey) Then _
+               Options = New Twitter.EditorExchangeOptions(Me) With {.SiteKey = MastodonSiteKey}
+            If OpenForm Then
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
+            End If
+        End Sub
+#End Region
+#Region "Download"
+        Friend Overrides Function BaseAuthExists() As Boolean
+            Return ACheck(Token.Value) And ACheck(Auth.Value) And ACheck(MyDomain.Value)
+        End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            If What = ISiteSettings.Download.SavedPosts Or What = ISiteSettings.Download.SingleObject Then
+                If Not ACheck(MyDomain.Value) Then MyMainLOG = "Mastodon account domain not set" : Return False
+            Else
+                If CDate(DomainsLastUpdateDate.Value).AddDays(7) < Now Then UpdateServersList()
+            End If
+            Return MyBase.Available(What, Silent)
+        End Function
+#End Region
+#Region "GetUserUrl, GetUserPostUrl"
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
+            If Not ACheck(MyDomain.Value) Then Return String.Empty
+            With DirectCast(User, UserData)
+                If UserRelatedToMyDomain.Value Then
+                    If MyDomain.Value = .UserDomain Then
+                        Return $"https://{ .UserDomain}/@{ .TrueName}"
+                    Else
+                        Return $"https://{MyDomain.Value}/@{ .TrueName}@{ .UserDomain}"
+                    End If
+                Else
+                    Return $"https://{ .UserDomain}/@{ .TrueName}"
+                End If
+            End With
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            If Not ACheck(MyDomain.Value) Then Return String.Empty
+            Return $"{GetUserUrl(User)}/{Media.Post.ID}"
+        End Function
+#End Region
+#Region "IsMyUser, IsMyImageVideo"
+        Private Const UserRegexDefault As String = "https?://{0}/@([^/@]+)@?([^/]*)"
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            If Domains.Count > 0 Then
+                Dim l As List(Of String)
+                For Each domain$ In Domains
+                    UserRegex.Pattern = String.Format(UserRegexDefault, domain)
+                    l = RegexReplace(UserURL, UserRegex)
+                    If l.ListExists(2) Then Return New ExchangeOptions(Site, $"{l(2).IfNullOrEmpty(domain)}@{l(1)}")
+                Next
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
+            If Not URL.IsEmptyString And Domains.Count > 0 Then
+                If Domains.Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions(Site, URL) With {.Exists = True}
+            End If
+            Return Nothing
+        End Function
+#End Region
+#Region "UpdateServersList"
+        Private Sub UpdateServersList()
+            Try
+                Dim r$ = GetWebString("https://api.joinmastodon.org/servers?language=&category=&region=&ownership=&registrations=",, EDP.ThrowException)
+                If Not r.IsEmptyString Then
+                    Dim j As EContainer = JsonDocument.Parse(r, EDP.ReturnValue)
+                    If If(j?.Count, 0) > 0 Then
+                        Domains.Domains.ListAddList(j.Select(Function(e) e.Value("domain")), LAP.NotContainsOnly, EDP.ReturnValue)
+                        Domains.Domains.Sort()
+                        Domains.Save()
+                        j.Dispose()
+                    End If
+                End If
+                DomainsLastUpdateDate.Value = Now
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.Mastodon.SiteSettings.UpdateServersList]")
+            End Try
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/Mastodon/UserData.vb

@@ -0,0 +1,287 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports UTypes = SCrawler.API.Base.UserMedia.Types
+Namespace API.Mastodon
+    Friend Class UserData : Inherits Twitter.UserData
+#Region "XML names"
+        Private Const Name_UserDomain As String = "UserDomain"
+        Private Const Name_TrueName As String = "TrueName"
+#End Region
+#Region "Declarations"
+        Private _UserDomain As String = String.Empty
+        Friend Property UserDomain As String
+            Get
+                Return _UserDomain.IfNullOrEmpty(MySettings.MyDomain.Value)
+            End Get
+            Set(ByVal d As String)
+                _UserDomain = d
+            End Set
+        End Property
+        Friend Property TrueName As String = String.Empty
+        Private ReadOnly Property MySettings As SiteSettings
+            Get
+                Return HOST.Source
+            End Get
+        End Property
+        Private MyCredentials As Credentials
+        Private Sub ResetCredentials()
+            MyCredentials = Nothing
+            With MySettings
+                Dim setDef As Boolean = True
+                If Not IsSavedPosts Then
+                    If ACheck(.MyDomain.Value) AndAlso UserDomain = .MyDomain.Value Then
+                        setDef = True
+                    ElseIf .Domains.Credentials.Count > 0 Then
+                        Dim i% = .Domains.Credentials.IndexOf(UserDomain)
+                        If i >= 0 Then
+                            MyCredentials = .Domains.Credentials(i)
+                            setDef = Not MyCredentials.Exists
+                        End If
+                    End If
+                End If
+                If setDef Then MyCredentials = New Credentials With {.Domain = UserDomain, .Bearer = MySettings.Auth.Value, .Csrf = MySettings.Token.Value}
+            End With
+            With MyCredentials
+                Responser.Headers.Add(Twitter.SiteSettings.Header_Authorization, .Bearer)
+                Responser.Headers.Add(Twitter.SiteSettings.Header_Token, .Csrf)
+            End With
+        End Sub
+#End Region
+#Region "LoadUserInformation"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            MyBase.LoadUserInformation_OptionalFields(Container, Loading)
+            Dim obtainNames As Action = Sub()
+                                            If _UserDomain.IsEmptyString And Not Name.IsEmptyString Then
+                                                Dim l$() = Name.Split("@")
+                                                If l.ListExists(2) Then
+                                                    _UserDomain = l(0)
+                                                    TrueName = l(1)
+                                                Else
+                                                    _UserDomain = MySettings.MyDomain.Value
+                                                    TrueName = Name
+                                                End If
+                                                If FriendlyName.IsEmptyString Then FriendlyName = TrueName
+                                            End If
+                                        End Sub
+            If Loading Then
+                _UserDomain = Container.Value(Name_UserDomain)
+                TrueName = Container.Value(Name_TrueName)
+                obtainNames.Invoke
+            Else
+                obtainNames.Invoke
+                Container.Add(Name_UserDomain, _UserDomain)
+                Container.Add(Name_TrueName, TrueName)
+                Container.Value(Name_FriendlyName) = FriendlyName
+            End If
+        End Sub
+#End Region
+#Region "Download functions"
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            ResetCredentials()
+            DownloadData(String.Empty, Token)
+        End Sub
+        Private Overloads Sub DownloadData(ByVal POST As String, ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Dim PostID$ = String.Empty
+                Dim PostDate$
+                Dim s As EContainer, ss As EContainer
+                Dim NewPostDetected As Boolean = False
+                Dim ExistsDetected As Boolean = False
+
+                If IsSavedPosts Then
+                    URL = $"https://{MySettings.MyDomain.Value}/api/v1/bookmarks"
+                    If Not POST.IsEmptyString Then URL &= $"?max_id={POST}"
+                Else
+                    If POST.IsEmptyString And ID.IsEmptyString Then
+                        ObtainUserID()
+                        If ID.IsEmptyString Then Throw New ArgumentNullException("ID", "Unable to get user ID") With {.HelpLink = 1}
+                    End If
+                    URL = $"https://{MyCredentials.Domain}/api/v1/accounts/{ID}/statuses?"
+                    If ParseUserMediaOnly Then URL &= "only_media=true&"
+                    URL &= "limit=40"
+                    If Not POST.IsEmptyString Then URL &= $"&max_id={POST}"
+                End If
+
+                ThrowAny(Token)
+                Dim r$ = Responser.GetResponse(URL)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r)
+                        If If(j?.Count, 0) > 0 Then
+                            For Each jj As EContainer In j
+                                With jj
+                                    If Not IsSavedPosts And POST.IsEmptyString And Not .Item("account") Is Nothing Then
+                                        With .Item("account")
+                                            If .Value("id") = ID Then
+                                                UserSiteNameUpdate(.Value("display_name"))
+                                                UserDescriptionUpdate(.Value("note"))
+                                                Dim __getImage As Action(Of String) = Sub(ByVal img As String)
+                                                                                          If Not img.IsEmptyString Then
+                                                                                              Dim __imgFile As SFile = img
+                                                                                              If Not __imgFile.Name.IsEmptyString Then
+                                                                                                  If __imgFile.Extension.IsEmptyString Then __imgFile.Extension = "jpg"
+                                                                                                  __imgFile.Path = MyFile.CutPath.Path
+                                                                                                  If Not __imgFile.Exists Then GetWebFile(img, __imgFile, EDP.None)
+                                                                                              End If
+                                                                                          End If
+                                                                                      End Sub
+                                                __getImage.Invoke(.Value("header").IfNullOrEmpty(.Value("header_static")))
+                                                __getImage.Invoke(.Value("avatar").IfNullOrEmpty(.Value("avatar_static")))
+                                            End If
+                                        End With
+                                    End If
+
+                                    PostID = .Value("id")
+                                    PostDate = .Value("created_at")
+
+                                    If Not IsSavedPosts And Not PostDate.IsEmptyString Then
+                                        Select Case CheckDatesLimit(PostDate, DateProvider)
+                                            Case DateResult.Skip : Continue For
+                                            Case DateResult.Exit : Exit Sub
+                                        End Select
+                                    End If
+
+                                    If Not _TempPostsList.Contains(PostID) Then
+                                        NewPostDetected = True
+                                        _TempPostsList.Add(PostID)
+                                    Else
+                                        ExistsDetected = True
+                                        Continue For
+                                    End If
+
+                                    If IsSavedPosts OrElse (Not ParseUserMediaOnly OrElse
+                                                            (If(.Item("reblog")?.Count, 0) = 0 OrElse .Value({"reblog", "account"}, "id") = ID)) Then
+                                        If If(.Item("media_attachments")?.Count, 0) > 0 Then
+                                            s = .Item("media_attachments")
+                                        Else
+                                            s = .Item({"reblog", "account"}, "media_attachments")
+                                        End If
+                                        If s.ListExists Then
+                                            For Each ss In s : ObtainMedia(ss, PostID, PostDate) : Next
+                                        End If
+                                    End If
+                                End With
+                            Next
+                        End If
+                    End Using
+                End If
+
+                If POST.IsEmptyString And ExistsDetected Then Exit Sub
+                If Not PostID.IsEmptyString And NewPostDetected Then DownloadData(PostID, Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error{IIf(IsSavedPosts, " (Saved Posts)", String.Empty)} [{URL}]")
+            End Try
+        End Sub
+        Private Sub ObtainMedia(ByVal e As EContainer, ByVal PostID As String, ByVal PostDate As String, Optional ByVal BaseUrl As String = Nothing)
+            Dim t As UTypes = UTypes.Undefined
+            Select Case e.Value("type")
+                Case "video" : t = UTypes.Video
+                Case "image" : t = UTypes.Picture
+                Case "gifv" : t = UTypes.GIF
+            End Select
+            If Not t = UTypes.Undefined Then
+                Dim m As New UserMedia(e.Value("url"), t) With {
+                    .Post = New UserPost(PostID, AConvert(Of Date)(PostDate, DateProvider, Nothing, EDP.ReturnValue)),
+                    .URL_BASE = BaseUrl.IfNullOrEmpty(MySettings.GetUserPostUrl(Me, m))
+                }
+                If Not t = UTypes.GIF Or GifsDownload Then
+                    If t = UTypes.GIF Then
+                        If Not GifsSpecialFolder.IsEmptyString Then m.SpecialFolder = GifsSpecialFolder
+                        If Not GifsPrefix.IsEmptyString Then m.File.Name = $"{GifsPrefix}{m.File.Name}"
+                    End If
+                    If Not m.URL.IsEmptyString Then _TempMediaList.ListAddValue(m, LNC)
+                End If
+            End If
+        End Sub
+        Private Sub ObtainUserID()
+            Try
+                If ID.IsEmptyString Then
+                    Dim url$ = $"https://{MyCredentials.Domain}/api/v1/accounts/lookup?acct="
+                    If Not UserDomain.IsEmptyString Then
+                        If UserDomain = MyCredentials.Domain Then
+                            url &= $"@{TrueName}"
+                        Else
+                            url &= $"@{TrueName}@{UserDomain}"
+                        End If
+                    Else
+                        url &= $"@{TrueName}"
+                    End If
+                    Dim r$ = Responser.GetResponse(url)
+                    If Not r.IsEmptyString Then
+                        Using j As EContainer = JsonDocument.Parse(r)
+                            If Not j Is Nothing Then ID = j.Value("id")
+                        End Using
+                    End If
+                End If
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"API.Mastodon.UserData.ObtainUserID({ToStringForLog()})")
+            End Try
+        End Sub
+        Private Function GetSinglePostPattern(ByVal Domain As String) As String
+            Return $"https://{Domain}/api/v1/statuses/" & "{0}"
+        End Function
+        Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
+            SinglePostUrl = GetSinglePostPattern(MyCredentials.Domain)
+            MyBase.ReparseMissing(Token)
+        End Sub
+#End Region
+#Region "DownloadSingleObject"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim PostID$ = RegexReplace(Data.URL, RParams.DM("(?<=/)\d+", 0, EDP.ReturnValue))
+            If Not PostID.IsEmptyString Then
+                ResetCredentials()
+                Dim pattern$
+                If Not ACheck(MySettings.MyDomain.Value) Then
+                    Throw New ArgumentNullException("Mastodon domain", "Mastodon domain not set")
+                Else
+                    pattern = GetSinglePostPattern(MySettings.MyDomain.Value)
+                End If
+                Dim r$ = Responser.GetResponse(String.Format(pattern, PostID),, EDP.ReturnValue)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r)
+                        If j.ListExists AndAlso j.Contains("media_attachments") Then
+                            For Each jj As EContainer In j("media_attachments") : ObtainMedia(jj, PostID, String.Empty, Data.URL) : Next
+                        End If
+                    End Using
+                End If
+            End If
+        End Sub
+#End Region
+#Region "Exception"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            If TypeOf ex Is ArgumentNullException AndAlso Not ex.HelpLink.IsEmptyString And ex.HelpLink = 1 Then
+                Return 0
+            Else
+                If Responser.Status = Net.WebExceptionStatus.NameResolutionFailure Then
+                    MyMainLOG = $"User domain ({UserDomain}) not found: {ToStringForLog()}"
+                    Return 1
+                ElseIf Responser.StatusCode = Net.HttpStatusCode.NotFound Then
+                    UserExists = False
+                    Return 1
+                ElseIf Responser.StatusCode = Net.HttpStatusCode.Unauthorized Then
+                    MyMainLOG = $"{ToStringForLog()}: account credentials have expired"
+                    Return 2
+                ElseIf Responser.StatusCode = Net.HttpStatusCode.Gone Then
+                    UserSuspended = True
+                    Return 1
+                Else
+                    Return 0
+                End If
+            End If
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/PathPlugin/SiteSettings.vb

@@ -12,9 +12,10 @@ Imports SCrawler.Plugin.Attributes
 Namespace API.PathPlugin
     <Manifest(PluginKey)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
+        Private ReadOnly _Icon As Icon = Nothing
         Friend Overrides ReadOnly Property Icon As Icon
             Get
-                Return PersonalUtilities.Tools.ImageRenderer.GetIcon(PersonalUtilities.My.Resources.FolderOpenPic_Orange_16, EDP.ReturnValue)
+                Return _Icon
             End Get
         End Property
         Friend Overrides ReadOnly Property Image As Image
@@ -24,6 +25,7 @@ Namespace API.PathPlugin
         End Property
         Friend Sub New()
             MyBase.New(PluginName)
+            _Icon = PersonalUtilities.Tools.ImageRenderer.GetIcon(PersonalUtilities.My.Resources.FolderOpenPic_Orange_16, EDP.ReturnValue)
         End Sub
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
             Return New UserData
@@ -42,7 +44,7 @@ Namespace API.PathPlugin
         Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
             Return Nothing
         End Function
-        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
             Return String.Empty
         End Function
     End Class

SCrawler/API/Pinterest/Declarations.vb

@@ -0,0 +1,21 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Globalization
+Namespace API.Pinterest
+    Friend Module Declarations
+        Friend ReadOnly DateProvider As ADateTime = GetDateProvider()
+        Private Function GetDateProvider() As ADateTime
+            Dim n As DateTimeFormatInfo = CultureInfo.GetCultureInfo("en-us").DateTimeFormat.Clone
+            n.FullDateTimePattern = "ddd dd MMM yyyy HH:mm:ss"
+            n.TimeSeparator = String.Empty
+            'Sat, 01 Jan 2000 01:10:15 +0000
+            Return New ADateTime(DirectCast(n.Clone, DateTimeFormatInfo)) With {.DateTimeStyle = DateTimeStyles.AssumeUniversal}
+        End Function
+    End Module
+End Namespace

SCrawler/API/Pinterest/SiteSettings.vb

@@ -0,0 +1,101 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Forms
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.Pinterest
+    <Manifest("AndyProgram_Pinterest"), SavedPosts, SeparatedTasks>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.PinterestIcon_32
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.PinterestPic_48
+            End Get
+        End Property
+        Private Class ConcurrentDownloadsValidator : Inherits FieldsCheckerProviderBase
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
+                Dim v% = AConvert(Of Integer)(Value, -1)
+                Dim defV% = Settings.MaxUsersJobsCount
+                If v.ValueBetween(1, defV) Then
+                    Return Value
+                Else
+                    ErrorMessage = $"The number of concurrent downloads must be greater than 0 and equal to or less than {defV} (global limit)."
+                    HasError = True
+                    Return Nothing
+                End If
+            End Function
+        End Class
+        <Provider(NameOf(ConcurrentDownloads), FieldsChecker:=True)>
+        Private ReadOnly Property ConcurrentDownloadsProvider As IFormatProvider
+        <PXML, PropertyOption(ControlText:="Concurrent downloads", ControlToolTip:="The number of concurrent downloads.", LeftOffset:=120), TaskCounter>
+        Friend ReadOnly Property ConcurrentDownloads As PropertyValue
+        <PXML, PropertyOption(ControlText:="Saved posts user", ControlToolTip:="Personal profile username")>
+        Friend ReadOnly Property SavedPostsUserName As PropertyValue
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("Pinterest", "pinterest.com")
+            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
+            ConcurrentDownloads = New PropertyValue(1)
+            ConcurrentDownloadsProvider = New ConcurrentDownloadsValidator
+            CheckNetscapeCookiesOnEndInit = True
+            UseNetscapeCookies = True
+            UserRegex = RParams.DMS("https?://w{0,3}.?[^/]*?.?pinterest.com/([^/]+)/?(?(_)|([^/]*))", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
+        End Sub
+#End Region
+#Region "GetInstance, Available"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return Settings.GalleryDLFile.Exists And (Not What = ISiteSettings.Download.SavedPosts OrElse
+                                                     (Responser.CookiesExists And ACheck(SavedPostsUserName.Value)))
+        End Function
+#End Region
+#Region "IsMyUser, IsMyImageVideo, GetUserUrl, GetUserPostUrl"
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            If Not UserURL.IsEmptyString Then
+                Dim l As List(Of String) = RegexReplace(UserURL, UserRegex)
+                If l.ListExists(3) Then
+                    Dim n$ = l(1)
+                    If Not l(2).IsEmptyString Then n &= $"@{l(2)}"
+                    Return New ExchangeOptions(Site, n) With {.Exists = True}
+                End If
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
+            Return IsMyUser(URL)
+        End Function
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
+            With DirectCast(User, UserData)
+                Dim n$ = .TrueUserName
+                Dim c$ = .TrueBoardName
+                If Not c.IsEmptyString Then c &= "/"
+                Return $"https://www.pinterest.com/{n}/{c}"
+            End With
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            If Not Media.Post.ID.IsEmptyString Then
+                Return $"https://www.pinterest.com/pin/{Media.Post.ID}/"
+            Else
+                Return String.Empty
+            End If
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/Pinterest/UserData.vb

@@ -0,0 +1,330 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports SCrawler.API.Base.GDL
+Imports SCrawler.API.YouTube.Objects
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Namespace API.Pinterest
+    Friend Class UserData : Inherits UserDataBase
+#Region "XML names"
+        Private Const Name_IsUser As String = "IsUser"
+        Private Const Name_TrueUserName As String = "TrueUserName"
+        Private Const Name_TrueBoardName As String = "TrueBoardName"
+#End Region
+#Region "Structures"
+        Private Structure BoardInfo
+            Friend ID As String
+            Friend Title As String
+            Friend URL As String
+            Friend Description As String
+            Friend UserID As String
+            Friend UserTitle As String
+        End Structure
+#End Region
+#Region "Declarations"
+        Private ReadOnly Property MySettings As SiteSettings
+            Get
+                Return HOST.Source
+            End Get
+        End Property
+        Friend Property TrueUserName As String
+        Friend Property TrueBoardName As String
+        Friend Property IsUser As Boolean
+#End Region
+#Region "Load"
+        Private Function ReconfUserName() As Boolean
+            If TrueUserName.IsEmptyString Then
+                Dim n$() = Name.Split("@")
+                If n.ListExists Then
+                    TrueUserName = n(0)
+                    IsUser = True
+                    If n.Length > 1 Then TrueBoardName = n(1) : IsUser = False
+                    If Not IsSavedPosts And Not IsSingleObjectDownload Then
+                        Dim l$ = IIf(IsUser, UserLabelName, "Board")
+                        Settings.Labels.Add(l)
+                        Labels.ListAddValue(l, LNC)
+                        Labels.Sort()
+                    End If
+                    Return True
+                End If
+            End If
+            Return False
+        End Function
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            With Container
+                If Loading Then
+                    TrueUserName = .Value(Name_TrueUserName)
+                    TrueBoardName = .Value(Name_TrueBoardName)
+                    IsUser = .Value(Name_IsUser).FromXML(Of Boolean)(False)
+                    ReconfUserName()
+                Else
+                    If ReconfUserName() Then .Value(Name_LabelsName) = Labels.ListToString("|", EDP.ReturnValue)
+                    .Add(Name_TrueUserName, TrueUserName)
+                    .Add(Name_TrueBoardName, TrueBoardName)
+                    .Add(Name_IsUser, IsUser.BoolToInteger)
+                End If
+            End With
+        End Sub
+#End Region
+#Region "Download overrides"
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                If IsSavedPosts Then
+                    IsUser = True
+                    TrueUserName = MySettings.SavedPostsUserName.Value
+                    If TrueUserName.IsEmptyString Then Throw New ArgumentNullException("SavedPostsUserName", "Saved posts user not set")
+                End If
+                Dim boards As List(Of BoardInfo)
+                Dim board As BoardInfo
+                Dim b$ = TrueBoardName
+                If Not b.IsEmptyString Then b &= "/"
+                URL = $"https://www.pinterest.com/{TrueUserName}/{b}"
+                If IsUser Then
+                    boards = GetBoards(Token)
+                Else
+                    boards = New List(Of BoardInfo) From {New BoardInfo With {.URL = URL, .ID = ID, .Title = UserSiteName}}
+                End If
+                If boards.ListExists Then
+                    For i% = 0 To boards.Count - 1
+                        ThrowAny(Token)
+                        board = boards(i)
+                        DownloadBoardImages(board, Token)
+                        boards(i) = board
+                    Next
+                    With boards.First
+                        If IsUser Then
+                            If ID.IsEmptyString Then ID = .UserID
+                            UserSiteNameUpdate(.UserTitle)
+                        Else
+                            If ID.IsEmptyString Then ID = .ID
+                            UserSiteNameUpdate(.Title)
+                            UserDescriptionUpdate(.Description)
+                        End If
+                    End With
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+#End Region
+#Region "Get boards, images"
+        Private Function GetBoards(ByVal Token As CancellationToken) As List(Of BoardInfo)
+            Dim URL$ = $"https://www.pinterest.com/{TrueUserName}/"
+            Try
+                Dim boards As New List(Of BoardInfo)
+                Dim b As BoardInfo
+                Dim r$
+                Dim j As EContainer, jj As EContainer
+                Dim rootNode$() = {"resource_response", "data"}
+                Dim jErr As New ErrorsDescriber(EDP.SendToLog + EDP.ReturnValue)
+                Dim urls As List(Of String) = GetDataFromGalleryDL(URL, True)
+                If urls.ListExists Then urls.RemoveAll(Function(__url) Not __url.Contains("BoardsResource/get/"))
+                If urls.ListExists Then
+                    For Each URL In urls
+                        ThrowAny(Token)
+                        r = Responser.GetResponse(URL,, EDP.ReturnValue)
+                        If Not r.IsEmptyString Then
+                            j = JsonDocument.Parse(r, jErr)
+                            If Not j Is Nothing Then
+                                If If(j(rootNode)?.Count, 0) > 0 Then
+                                    For Each jj In j(rootNode)
+                                        b = New BoardInfo With {
+                                            .URL = jj.Value("url"),
+                                            .Title = TitleHtmlConverter(jj.Value("name")),
+                                            .ID = jj.Value("id")
+                                        }
+                                        If Not b.URL.IsEmptyString Then
+                                            b.URL = $"https://www.pinterest.com/{b.URL.StringTrimStart("/").StringTrimEnd("/")}/"
+                                            boards.Add(b)
+                                        End If
+                                    Next
+                                End If
+                                j.Dispose()
+                            End If
+                        End If
+                    Next
+                End If
+                Return boards
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data (gallery-dl boards) downloading error [{URL}]")
+                Return Nothing
+            End Try
+        End Function
+        Private Sub DownloadBoardImages(ByRef Board As BoardInfo, ByVal Token As CancellationToken)
+            Dim bUrl$ = String.Empty
+            Try
+                Dim r$
+                Dim j As EContainer, jj As EContainer
+                Dim u As UserMedia
+                Dim folder$ = If(IsUser, Board.Title.IfNullOrEmpty(Board.ID), String.Empty)
+                Dim titleExists As Boolean = Not Board.Title.IsEmptyString
+                Dim i% = -1
+                Dim jErr As New ErrorsDescriber(EDP.SendToLog + EDP.ReturnValue)
+                Dim rootNode$() = {"resource_response", "data"}
+                Dim images As List(Of Sizes)
+                Dim imgSelector As Func(Of EContainer, Sizes) = Function(img) New Sizes(img.Value("width"), img.Value("url"))
+                Dim fullData As Predicate(Of EContainer) = Function(e) e.Count > 5
+                Dim l As List(Of String) = GetDataFromGalleryDL(Board.URL, False)
+                If l.ListExists Then l.RemoveAll(Function(ll) Not ll.Contains("BoardFeedResource/get/"))
+                If l.ListExists Then
+                    For Each bUrl In l
+                        ThrowAny(Token)
+                        r = Responser.GetResponse(bUrl,, EDP.ReturnValue)
+                        If Not r.IsEmptyString Then
+                            j = JsonDocument.Parse(r, jErr)
+                            If Not j Is Nothing Then
+                                If If(j(rootNode)?.Count, 0) > 0 Then
+                                    For Each jj In j(rootNode)
+                                        With jj
+                                            If .Contains("images") Then
+                                                images = .Item("images").Select(imgSelector).ToList
+                                                If images.Count > 0 Then
+                                                    images.Sort()
+                                                    i += 1
+                                                    u = New UserMedia(images(0).Data) With {
+                                                        .Post = New UserPost(jj.Value("id"), AConvert(Of Date)(jj.Value("created_at"), DateProvider, Nothing)),
+                                                        .Type = UserMedia.Types.Picture,
+                                                        .SpecialFolder = folder
+                                                    }
+                                                    If i = 0 Then
+                                                        If Board.Title.IsEmptyString Or Board.ID.IsEmptyString Then
+                                                            Board.Title = TitleHtmlConverter(.Value({"board"}, "name"))
+                                                            Board.ID = .Value({"board"}, "id")
+                                                        End If
+                                                        Board.UserID = .Value({"board", "owner"}, "id")
+                                                        Board.UserTitle = TitleHtmlConverter(.Value({"board", "owner"}, "full_name"))
+                                                        If Not titleExists And IsUser Then
+                                                            If Not Board.Title.IsEmptyString Then
+                                                                folder = Board.Title
+                                                            ElseIf Not Board.ID.IsEmptyString Then
+                                                                folder = Board.ID
+                                                            End If
+                                                            u.SpecialFolder = folder
+                                                        End If
+                                                    End If
+
+                                                    If Not u.URL.IsEmptyString Then
+                                                        If u.Post.Date.HasValue Then
+                                                            Select Case CheckDatesLimit(u.Post.Date.Value, Nothing)
+                                                                Case DateResult.Skip : _TempPostsList.ListAddValue(u.Post.ID, LNC) : Continue For
+                                                                Case DateResult.Exit : Exit Sub
+                                                            End Select
+                                                        End If
+                                                        If Not _TempPostsList.Contains(u.Post.ID) Then
+                                                            _TempPostsList.ListAddValue(u.Post.ID, LNC)
+                                                            _TempMediaList.ListAddValue(u, LNC)
+                                                        Else
+                                                            Exit For
+                                                        End If
+                                                    End If
+                                                End If
+                                            End If
+                                        End With
+                                    Next
+                                End If
+                                j.Dispose()
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data (gallery-dl images) downloading error [{bUrl}]")
+            End Try
+        End Sub
+#End Region
+#Region "Gallery-DL Support"
+        Private Class GDLBatch : Inherits GDL.GDLBatch
+            Private ReadOnly Property Source As UserData
+            Private ReadOnly IsBoardsRequested As Boolean
+            Friend Sub New(ByRef s As UserData, ByVal IsBoardsRequested As Boolean)
+                MyBase.New
+                Source = s
+                Me.IsBoardsRequested = IsBoardsRequested
+            End Sub
+            Protected Overrides Async Sub OutputDataReceiver(ByVal Sender As Object, ByVal e As DataReceivedEventArgs)
+                If IsBoardsRequested Then
+                    Await Validate(e.Data)
+                Else
+                    MyBase.OutputDataReceiver(Sender, e)
+                    Await Validate(e.Data)
+                End If
+            End Sub
+            Protected Overrides Async Function Validate(ByVal Value As String) As Task
+                If IsBoardsRequested Then
+                    If ErrorOutputData.Count > 0 Then
+                        If Await Task.Run(Of Boolean)(Function() ErrorOutputData.Exists(Function(ee) Not ee.IsEmptyString AndAlso
+                                                                                                     ee.StartsWith(UrlTextStart))) Then Kill(EDP.None)
+                    End If
+                Else
+                    If Await Task.Run(Of Boolean)(Function() Not Value.IsEmptyString AndAlso
+                                                             Source._TempPostsList.Exists(Function(v) Value.Contains(v))) Then Kill(EDP.None)
+                End If
+            End Function
+        End Class
+        Private Function GetDataFromGalleryDL(ByVal URL As String, ByVal IsBoardsRequested As Boolean) As List(Of String)
+            Dim command$ = $"gallery-dl --verbose --simulate "
+            Try
+                If Not URL.IsEmptyString Then
+                    If MySettings.CookiesNetscapeFile.Exists Then command &= $"--cookies ""{MySettings.CookiesNetscapeFile}"" "
+                    command &= URL
+                    Using batch As New GDLBatch(Me, IsBoardsRequested)
+                        Return GetUrlsFromGalleryDl(batch, command)
+                    End Using
+                End If
+                Return Nothing
+            Catch ex As Exception
+                HasError = True
+                LogError(ex, $"GetJson({command})")
+                Return Nothing
+            End Try
+        End Function
+#End Region
+#Region "DownloadContent"
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            DownloadContentDefault(Token)
+        End Sub
+#End Region
+#Region "DownloadSingleObject"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            User = New UserInfo(MySettings.IsMyUser(Data.URL).UserName, HOST)
+            User.File.Path = Data.File.Path
+            SeparateVideoFolder = False
+            ReconfUserName()
+            DownloadDataF(Token)
+            Data.Title = UserSiteName
+            If Data.Title.IsEmptyString Then
+                Data.Title = TrueUserName
+                If Not TrueBoardName.IsEmptyString Then Data.Title &= $"/{TrueBoardName}"
+            End If
+            Dim additPath$ = TitleHtmlConverter(UserSiteName)
+            If additPath.IsEmptyString Then additPath = IIf(IsUser, TrueUserName, TrueBoardName)
+            If Not additPath.IsEmptyString Then
+                Dim f As SFile = User.File
+                f.Path = f.PathWithSeparator & additPath
+                User.File = f
+                f = Data.File
+                f.Path = User.File.Path
+                Data.File = f
+            End If
+        End Sub
+        Protected Overrides Sub DownloadSingleObject_PostProcessing(ByVal Data As IYouTubeMediaContainer, Optional ByVal ResetTitle As Boolean = True)
+            MyBase.DownloadSingleObject_PostProcessing(Data, Data.Title.IsEmptyString Or Not Data.Title = UserSiteName)
+        End Sub
+#End Region
+#Region "Exception"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            Return 0
+        End Function
+    End Class
+#End Region
+End Namespace

SCrawler/API/PornHub/Declarations.vb

@@ -23,6 +23,7 @@ Namespace API.PornHub
         Private ReadOnly RegexVideo_Video_Wrong_Option As RParams = RParams.DM("div class=""thumbnail-info-wrapper clearfix.+?[\r\n\s]*?\<span class=""title.+?[\r\n\s]*?\<a href=""([^""]+?)""[\s]+?title=""([^""]*?)""[\w\W\s\r\n]+?", 0, RegexReturn.ListByMatch)
         Friend ReadOnly RegexVideo_Video_Wrong_Fields As RField() = {New RField(New RFieldOption(1, RegexVideo_Video_Wrong_Option)), New RField(New RFieldOption(2, RegexVideo_Video_Wrong_Option))}
         Friend ReadOnly RegexVideo_Video_VideoKey As RParams = RParams.DMS("viewkey=([\w\d]+)", 1, EDP.ReturnValue)
+        Friend ReadOnly RegexVideoPageTitle As RParams = RParams.DMS("meta (property|name)=""[^:]+?:title"" content=""([^""]+)""", 2, EDP.ReturnValue)
 #End Region
 #Region "Declarations M3U8"
         Friend ReadOnly Regex_M3U8_FirstFileRegEx As RParams = RParams.DM(".+?m3u8.*", 0)
@@ -38,6 +39,7 @@ Namespace API.PornHub
         Friend ReadOnly Regex_Photo_PornHub_AlbumPhotoArr As RParams = RParams.DMS("\<a href=""(/photo/\d+)""", 1, RegexReturn.List, EDP.ReturnValue,
                                                                                    CType(Function(Input$) If(Input.IsEmptyString, String.Empty, $"https://www.pornhub.com{Input.Trim}"), Func(Of String, String)))
         Friend ReadOnly Regex_Photo_PornHub_SinglePhoto As RParams = RParams.DMS("(?<!thumbImage.+?)<img src=""(https://[^""]+\d+[^""]+)""", 1, EDP.ReturnValue)
+        Friend ReadOnly Regex_Photo_File As RParams = RParams.DM("\d+\.[\w]{3,4}", 0, EDP.ReturnValue)
 #End Region
     End Module
 End Namespace

SCrawler/API/PornHub/M3U8.vb

@@ -6,10 +6,12 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Threading
 Imports SCrawler.API.Base
 Imports SCrawler.API.Base.M3U8Declarations
-Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.PornHub
     Friend NotInheritable Class M3U8
         Private Sub New()
@@ -35,8 +37,9 @@ Namespace API.PornHub
             End If
             Return Nothing
         End Function
-        Friend Shared Function Download(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As SFile
-            Return M3U8Base.Download(GetUrlsList(URL, Responser), Destination, Responser)
+        Friend Shared Function Download(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile,
+                                        ByVal Token As CancellationToken, ByVal Progress As MyProgress) As SFile
+            Return M3U8Base.Download(GetUrlsList(URL, Responser), Destination, Responser, Token, Progress)
         End Function
     End Class
 End Namespace

SCrawler/API/PornHub/OptionsForm.Designer.vb

@@ -1,118 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.PornHub
-    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
-    Partial Friend Class OptionsForm : Inherits System.Windows.Forms.Form
-        <System.Diagnostics.DebuggerNonUserCode()>
-        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            Try
-                If disposing AndAlso components IsNot Nothing Then
-                    components.Dispose()
-                End If
-            Finally
-                MyBase.Dispose(disposing)
-            End Try
-        End Sub
-        Private components As System.ComponentModel.IContainer
-        <System.Diagnostics.DebuggerStepThrough()>
-        Private Sub InitializeComponent()
-            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
-            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
-            Me.CH_DOWN_GIFS = New System.Windows.Forms.CheckBox()
-            Me.CH_DOWN_PHOTO_MODELHUB = New System.Windows.Forms.CheckBox()
-            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
-            CONTAINER_MAIN.ContentPanel.SuspendLayout()
-            CONTAINER_MAIN.SuspendLayout()
-            TP_MAIN.SuspendLayout()
-            Me.SuspendLayout()
-            '
-            'CONTAINER_MAIN
-            '
-            '
-            'CONTAINER_MAIN.ContentPanel
-            '
-            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(278, 52)
-            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            CONTAINER_MAIN.LeftToolStripPanelVisible = False
-            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
-            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
-            CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(278, 77)
-            CONTAINER_MAIN.TabIndex = 0
-            CONTAINER_MAIN.TopToolStripPanelVisible = False
-            '
-            'TP_MAIN
-            '
-            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_MAIN.ColumnCount = 1
-            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Controls.Add(Me.CH_DOWN_GIFS, 0, 0)
-            TP_MAIN.Controls.Add(Me.CH_DOWN_PHOTO_MODELHUB, 0, 1)
-            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_MAIN.Location = New System.Drawing.Point(0, 0)
-            TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 3
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(278, 52)
-            TP_MAIN.TabIndex = 0
-            '
-            'CH_DOWN_GIFS
-            '
-            Me.CH_DOWN_GIFS.AutoSize = True
-            Me.CH_DOWN_GIFS.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_GIFS.Location = New System.Drawing.Point(4, 4)
-            Me.CH_DOWN_GIFS.Name = "CH_DOWN_GIFS"
-            Me.CH_DOWN_GIFS.Size = New System.Drawing.Size(270, 19)
-            Me.CH_DOWN_GIFS.TabIndex = 0
-            Me.CH_DOWN_GIFS.Text = "Download gifs"
-            Me.CH_DOWN_GIFS.UseVisualStyleBackColor = True
-            '
-            'CH_DOWN_PHOTO_MODELHUB
-            '
-            Me.CH_DOWN_PHOTO_MODELHUB.AutoSize = True
-            Me.CH_DOWN_PHOTO_MODELHUB.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_PHOTO_MODELHUB.Location = New System.Drawing.Point(4, 30)
-            Me.CH_DOWN_PHOTO_MODELHUB.Name = "CH_DOWN_PHOTO_MODELHUB"
-            Me.CH_DOWN_PHOTO_MODELHUB.Size = New System.Drawing.Size(270, 19)
-            Me.CH_DOWN_PHOTO_MODELHUB.TabIndex = 1
-            Me.CH_DOWN_PHOTO_MODELHUB.Text = "Download photo only from ModelHub"
-            Me.CH_DOWN_PHOTO_MODELHUB.UseVisualStyleBackColor = True
-            '
-            'OptionsForm
-            '
-            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
-            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(278, 77)
-            Me.Controls.Add(CONTAINER_MAIN)
-            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
-            Me.Icon = Global.SCrawler.My.Resources.SiteResources.InstagramIcon_32
-            Me.KeyPreview = True
-            Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(294, 116)
-            Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(294, 116)
-            Me.Name = "OptionsForm"
-            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Options"
-            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-            CONTAINER_MAIN.ResumeLayout(False)
-            CONTAINER_MAIN.PerformLayout()
-            TP_MAIN.ResumeLayout(False)
-            TP_MAIN.PerformLayout()
-            Me.ResumeLayout(False)
-
-        End Sub
-        Private WithEvents CH_DOWN_GIFS As CheckBox
-        Private WithEvents CH_DOWN_PHOTO_MODELHUB As CheckBox
-    End Class
-End Namespace

SCrawler/API/PornHub/OptionsForm.vb

@@ -1,34 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports PersonalUtilities.Forms
-Namespace API.PornHub
-    Friend Class OptionsForm
-        Private WithEvents MyDefs As DefaultFormOptions
-        Private ReadOnly MyExchangeOptions As UserExchangeOptions
-        Friend Sub New(ByRef ExchangeOptions As UserExchangeOptions)
-            InitializeComponent()
-            MyExchangeOptions = ExchangeOptions
-            MyDefs = New DefaultFormOptions(Me, Settings.Design)
-        End Sub
-        Private Sub MyForm_Load(sender As Object, e As EventArgs) Handles Me.Load
-            With MyDefs
-                .MyViewInitialize(True)
-                .AddOkCancelToolbar()
-                CH_DOWN_GIFS.Checked = MyExchangeOptions.DownloadGifs
-                CH_DOWN_PHOTO_MODELHUB.Checked = MyExchangeOptions.DownloadPhotoOnlyFromModelHub
-                .EndLoaderOperations()
-            End With
-        End Sub
-        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
-            MyExchangeOptions.DownloadGifs = CH_DOWN_GIFS.Checked
-            MyExchangeOptions.DownloadPhotoOnlyFromModelHub = CH_DOWN_PHOTO_MODELHUB.Checked
-            MyDefs.CloseForm()
-        End Sub
-    End Class
-End Namespace

SCrawler/API/PornHub/SiteSettings.vb

@@ -25,7 +25,6 @@ Namespace API.PornHub
                 Return My.Resources.SiteResources.PornHubPic_16
             End Get
         End Property
-        Private ReadOnly Property CurlPathExists As Boolean
         <PropertyOption(ControlText:="Download GIF", ControlToolTip:="Default for new users", ThreeStates:=True), PXML>
         Friend ReadOnly Property DownloadGifs As PropertyValue
         <PropertyOption(ControlText:="Download GIFs as mp4", ControlToolTip:="Download gifs in 'mp4' format instead of native 'webm'"), PXML>
@@ -40,10 +39,7 @@ Namespace API.PornHub
 #Region "Initializer"
         Friend Sub New()
             MyBase.New("PornHub", "pornhub.com")
-            Responser.CurlPath = $"cURL\curl.exe"
-            Responser.CurlArgumentsRight = "--ssl-no-revoke"
-            CurlPathExists = Responser.CurlPath.Exists
-            Responser.DeclaredError = EDP.ThrowException
+            With Responser : .CurlSslNoRevoke = True : .CurlInsecure = True : End With
 
             DownloadGifsAsMp4 = New PropertyValue(True)
             DownloadGifs = New PropertyValue(CInt(CheckState.Indeterminate), GetType(Integer))
@@ -55,37 +51,16 @@ Namespace API.PornHub
             ImageVideoContains = "pornhub"
         End Sub
 #End Region
-#Region "GetInstance, GetSpecialData"
+#Region "GetInstance"
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
-            If What = ISiteSettings.Download.SavedPosts Then
-                Return New UserData With {
-                    .IsSavedPosts = True,
-                    .VideoPageModel = UserData.VideoPageModels.Favorite,
-                    .PersonType = UserData.PersonTypeUser,
-                    .User = New UserInfo With {.Name = $"{UserData.PersonTypeUser}_{CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}"}
-                }
-            Else
-                Return New UserData
-            End If
-        End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            If Available(ISiteSettings.Download.Main, True) Then
-                Using resp As Responser = Responser.Copy
-                    Dim spf$ = String.Empty
-                    Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
-                    Dim m As UserMedia = UserData.GetVideoInfo(URL, resp, f)
-                    If m.State = UserMedia.States.Downloaded Then
-                        m.SpecialFolder = f
-                        Return {m}
-                    End If
-                End Using
-            End If
-            Return Nothing
+            Return New UserData
         End Function
 #End Region
 #Region "Downloading"
         Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
-            Return Settings.UseM3U8 And CurlPathExists And (Not What = ISiteSettings.Download.SavedPosts OrElse ACheck(SavedPostsUserName.Value))
+            Responser.CurlPath = Settings.CurlFile
+            Return Settings.UseM3U8 And Settings.CurlFile.Exists And
+                   (Not What = ISiteSettings.Download.SavedPosts OrElse (ACheck(SavedPostsUserName.Value) And Responser.CookiesExists))
         End Function
 #End Region
 #Region "IsMyUser"
@@ -97,23 +72,20 @@ Namespace API.PornHub
                 End If
                 Return Nothing
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"[API.PornHub.SiteSettings.IsMyUser({UserURL})]", New ExchangeOptions)
+                Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, $"[API.PornHub.SiteSettings.IsMyUser({UserURL})]", New ExchangeOptions)
             End Try
         End Function
 #End Region
-#Region "GetUserUrl, GetUserPostUrl"
-        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
+#Region "GetUserUrl"
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
             With DirectCast(User, UserData) : Return String.Format(UrlPatternUser, .PersonType, .NameTrue) : End With
         End Function
-        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return Media.URL_BASE
-        End Function
 #End Region
 #Region "User options"
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
             If Options Is Nothing OrElse Not TypeOf Options Is UserExchangeOptions Then Options = New UserExchangeOptions(Me)
             If OpenForm Then
-                Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
             End If
         End Sub
 #End Region

SCrawler/API/PornHub/UserData.vb

@@ -8,6 +8,7 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
@@ -136,6 +137,7 @@ Namespace API.PornHub
 #Region "Initializer, loader"
         Friend Sub New()
             UseInternalM3U8Function = True
+            UseClientTokens = True
         End Sub
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
             With Container
@@ -179,7 +181,11 @@ Namespace API.PornHub
                 Responser.ResetStatus()
                 If PersonType = PersonTypeUser Then Responser.Mode = Responser.Modes.Curl
 
-                If IsSavedPosts Then VideoPageModel = VideoPageModels.Favorite
+                If IsSavedPosts Then
+                    VideoPageModel = VideoPageModels.Favorite
+                    PersonType = PersonTypeUser
+                    NameTrue = MySettings.SavedPostsUserName.Value
+                End If
 
                 Dim page% = 1
                 Dim __continue As Boolean = True
@@ -295,7 +301,7 @@ Namespace API.PornHub
                 If Not r.IsEmptyString Then
                     Dim n$
                     Dim m As UserMedia = Nothing
-                    Dim l As List(Of RegexMatchStruct) = RegexFields(Of RegexMatchStruct)(r, {Regex_Gif_Array}, {1})
+                    Dim l As List(Of RegexMatchStruct) = RegexFields(Of RegexMatchStruct)(r, {Regex_Gif_Array}, {1}, EDP.ReturnValue)
                     Dim l2 As List(Of String) = Nothing
                     Dim l3 As List(Of String) = Nothing
                     If l.ListExists Then l2 = l.Select(Function(ll) $"gif/{ll.Arr(0).Replace("gif", String.Empty)}").ToList
@@ -336,6 +342,10 @@ Namespace API.PornHub
         End Sub
 #End Region
 #Region "Download photo"
+        Private Function CreatePhotoFile(ByVal URL As String, ByVal File As SFile) As SFile
+            Dim pFile$ = RegexReplace(URL, Regex_Photo_File)
+            If Not pFile.IsEmptyString Then Return New SFile(pFile) Else Return File
+        End Function
         Private Const PhotoUrlPattern_ModelHub As String = "https://www.modelhub.com/{0}/photos"
         Private Const PhotoUrlPattern_PornHub As String = "https://www.pornhub.com/{0}/{1}/photos"
         Private Sub DownloadUserPhotos(ByVal Token As CancellationToken)
@@ -365,7 +375,8 @@ Namespace API.PornHub
         Private Function DownloadUserPhotos_ModelHub(ByVal Token As CancellationToken) As Boolean
             Dim URL$ = String.Empty
             Try
-                Dim jErr As New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue)
+                Dim j As EContainer
+                Dim jErr As New ErrorsDescriber(EDP.SendToLog + EDP.ReturnValue)
                 Dim albumName$
                 If PersonType = PersonTypeModel Then
                     URL = String.Format(PhotoUrlPattern_ModelHub, NameTrue)
@@ -380,15 +391,16 @@ Namespace API.PornHub
                                 albumRegex.Pattern = "<li id=""" & block.AlbumID & """ class=""modelBox"">[\r\n\s]*?<div class=""modelPhoto"">[\r\n\s]*?\<[^\>]*?alt=""([^""]*)"""
                                 albumName = StringTrim(RegexReplace(r, albumRegex))
                                 If albumName.IsEmptyString Then albumName = block.AlbumID
-                                Using j As EContainer = JsonDocument.Parse("{" & block.Data & "}", jErr)
-                                    If Not j Is Nothing Then
-                                        If If(j("urls")?.Count, 0) > 0 Then
-                                            _TempMediaList.ListAddList(j("urls").Select(Function(jj) _
-                                                                       New UserMedia(jj.ItemF({0}).XmlIfNothingValue, UTypes.Picture) With {
-                                                                                     .SpecialFolder = $"Albums\{albumName}\"}), LNC)
-                                        End If
+                                j = JsonDocument.Parse("{" & block.Data & "}", jErr)
+                                If Not j Is Nothing Then
+                                    If If(j("urls")?.Count, 0) > 0 Then
+                                        _TempMediaList.ListAddList(j("urls").Select(Function(jj) _
+                                                                   New UserMedia(jj.ItemF({0}).XmlIfNothingValue, UTypes.Picture) With {
+                                                                                 .SpecialFolder = $"Albums\{albumName}\",
+                                                                                 .File = CreatePhotoFile(.URL, .File)}), LNC)
                                     End If
-                                End Using
+                                    j.Dispose()
+                                End If
                             Next
                             l.Clear()
                         End If
@@ -444,7 +456,9 @@ Namespace API.PornHub
                                 If Not r.IsEmptyString Then
                                     url = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
                                     If Not url.IsEmptyString Then _
-                                       _TempMediaList.ListAddValue(New UserMedia(url, UTypes.Picture) With {.SpecialFolder = $"Albums\{AlbumName}\"}, LNC)
+                                       _TempMediaList.ListAddValue(New UserMedia(url, UTypes.Picture) With {
+                                                                   .SpecialFolder = $"Albums\{AlbumName}\",
+                                                                   .File = CreatePhotoFile(url, .File)}, LNC)
                                 End If
                             Catch
                             End Try
@@ -468,7 +482,7 @@ Namespace API.PornHub
                     If r.Contains(HtmlPageNotFoundPhoto) Then Return False
                     Dim urls As List(Of String) = RegexReplace(r, Regex_Photo_PornHub_AlbumPhotoArr)
                     If urls.ListExists Then
-                        Dim NewUrl$
+                        Dim NewUrl$, pFile$
                         Dim m As UserMedia
                         Dim l2 As List(Of UserMedia) = urls.Select(Function(__url) New UserMedia(__url, UTypes.Picture) With {
                                                                                                  .Post = __url.Split("/").LastOrDefault}).ToList
@@ -487,7 +501,8 @@ Namespace API.PornHub
                                             NewUrl = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto)
                                             If Not NewUrl.IsEmptyString Then
                                                 m.URL = NewUrl
-                                                m.File = NewUrl
+                                                pFile = RegexReplace(NewUrl, Regex_Photo_File)
+                                                If Not pFile.IsEmptyString Then m.File = pFile Else m.File = NewUrl
                                                 _TempPostsList.ListAddValue(m.Post.ID, LNC)
                                             Else
                                                 Throw New Exception
@@ -511,13 +526,17 @@ Namespace API.PornHub
 #End Region
 #End Region
 #Region "ReparseVideo"
-        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+        Protected Overloads Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            ReparseVideo(Token, False)
+        End Sub
+        Protected Overloads Sub ReparseVideo(ByVal Token As CancellationToken, ByVal CreateFileName As Boolean,
+                                             Optional ByRef Data As IYouTubeMediaContainer = Nothing)
             Const ERR_NEW_URL$ = "ERR_NEW_URL"
             Dim URL$ = String.Empty
             Try
                 If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(tm) tm.Type = UTypes.VideoPre) Then
                     Dim m As UserMedia
-                    Dim r$, NewUrl$
+                    Dim r$, NewUrl$, tmpName$
                     For i% = _TempMediaList.Count - 1 To 0 Step -1
                         If _TempMediaList(i).Type = UTypes.VideoPre Then
                             m = _TempMediaList(i)
@@ -532,6 +551,14 @@ Namespace API.PornHub
                                     Else
                                         m.URL = NewUrl
                                         m.Type = UTypes.m3u8
+                                        If CreateFileName Then
+                                            tmpName = RegexReplace(r, RegexVideoPageTitle)
+                                            If Not tmpName.IsEmptyString Then
+                                                If Not Data Is Nothing Then Data.Title = tmpName
+                                                m.File.Name = TitleHtmlConverter(tmpName)
+                                                m.File.Extension = "mp4"
+                                            End If
+                                        End If
                                         _TempMediaList(i) = m
                                     End If
                                 Else
@@ -565,7 +592,7 @@ Namespace API.PornHub
                         m = _ContentList(i)
                         If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then
                             ThrowAny(Token)
-                            r = Responser.Curl(m.URL_BASE, eCurl)
+                            r = Responser.Curl(m.URL_BASE,, eCurl)
                             If Not r.IsEmptyString Then
                                 Dim NewUrl$ = CreateVideoURL(r)
                                 If Not NewUrl.IsEmptyString Then
@@ -591,12 +618,12 @@ Namespace API.PornHub
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
-        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
-            Return M3U8.Download(URL, Responser, DestinationFile)
+        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile
+            Return M3U8.Download(URL, Responser, DestinationFile, Token, If(UseInternalM3U8Function_UseProgress, Progress, Nothing))
         End Function
 #End Region
 #Region "CreateVideoURL"
-        Private Shared Function CreateVideoURL(ByVal r As String) As String
+        Private Function CreateVideoURL(ByVal r As String) As String
             Try
                 Dim OutStr$ = String.Empty
                 If Not r.IsEmptyString Then
@@ -619,26 +646,18 @@ Namespace API.PornHub
                 End If
                 Return OutStr
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.PornHub.UserData.CreateVideoURL]", String.Empty)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.PornHub.UserData.CreateVideoURL]", String.Empty)
             End Try
         End Function
 #End Region
-#Region "Standalone downloader"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, ByVal Destination As SFile) As UserMedia
-            Try
-                Dim r$ = Responser.Curl(URL)
-                If Not r.IsEmptyString Then
-                    Dim NewUrl$ = CreateVideoURL(r)
-                    If Not NewUrl.IsEmptyString Then
-                        Dim f As SFile = M3U8.Download(NewUrl, Responser, Destination)
-                        If Not f.IsEmptyString Then Return New UserMedia With {.State = UserMedia.States.Downloaded}
-                    End If
-                End If
-                Return Nothing
-            Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"PornHub standalone download error: [{URL}]", New UserMedia)
-            End Try
-        End Function
+#Region "DownloadSingleObject"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            _TempMediaList.Add(New UserMedia(Data.URL, UTypes.VideoPre))
+            ReparseVideo(Token, True, Data)
+        End Sub
+        Protected Overrides Sub DownloadSingleObject_PostProcessing(ByVal Data As IYouTubeMediaContainer, Optional ByVal ResetTitle As Boolean = True)
+            MyBase.DownloadSingleObject_PostProcessing(Data, False)
+        End Sub
 #End Region
 #Region "Exceptions"
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String,

SCrawler/API/PornHub/UserExchangeOptions.vb

@@ -6,18 +6,24 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin.Attributes
 Namespace API.PornHub
     Friend Class UserExchangeOptions
+        <PSetting(Caption:="Download gifs")>
         Friend Property DownloadGifs As Boolean
+        <PSetting(NameOf(SiteSettings.DownloadPhotoOnlyFromModelHub), NameOf(MySettings), Caption:="Download photo only from ModelHub")>
         Friend Property DownloadPhotoOnlyFromModelHub As Boolean
+        Private ReadOnly Property MySettings As SiteSettings
         Friend Sub New(ByVal u As UserData)
             DownloadGifs = u.DownloadGifs
             DownloadPhotoOnlyFromModelHub = u.DownloadPhotoOnlyFromModelHub
+            MySettings = u.HOST.Source
         End Sub
         Friend Sub New(ByVal s As SiteSettings)
             Dim v As CheckState = CInt(s.DownloadGifs.Value)
             DownloadGifs = Not v = CheckState.Unchecked
             DownloadPhotoOnlyFromModelHub = s.DownloadPhotoOnlyFromModelHub.Value
+            MySettings = s
         End Sub
     End Class
 End Namespace

SCrawler/API/Reddit/Channel.vb

@@ -258,7 +258,8 @@ Namespace API.Reddit
                     .Progress = p,
                     .SaveToCache = True,
                     .SkipExistsUsers = SkipExists,
-                    .ChannelInfo = Me
+                    .ChannelInfo = Me,
+                    .IsChannel = True
                 }
                     With d
                         .SetEnvironment(HOST, CUser, False)
@@ -306,7 +307,7 @@ Namespace API.Reddit
         Friend Function GetEnumerator() As IEnumerator(Of UserPost) Implements IEnumerable(Of UserPost).GetEnumerator
             Return New MyEnumerator(Of UserPost)(Me)
         End Function
-        Friend Function IEnumerable_GetEnumerator() As IEnumerator Implements IEnumerable.GetEnumerator
+        Private Function IEnumerable_GetEnumerator() As IEnumerator Implements IEnumerable.GetEnumerator
             Return GetEnumerator()
         End Function
 #End Region
@@ -373,7 +374,7 @@ Namespace API.Reddit
                 Dim l As New List(Of String)
                 If Posts.Count > 0 Or PostsLatest.Count > 0 Then l.ListAddList((From p In PostsAll Where Not p.ID.IsEmptyString Select p.ID), LNC)
                 l.ListAddList(PostsNames, LNC)
-                If l.Count > 0 Then TextSaver.SaveTextToFile(l.ListToString("|"), FilePosts, True,, EDP.SendInLog)
+                If l.Count > 0 Then TextSaver.SaveTextToFile(l.ListToString("|"), FilePosts, True,, EDP.SendToLog)
             End If
             Using x As New XmlFile With {.AllowSameNames = True, .Name = "Channel"}
                 x.Add(Name_Name, Name)
@@ -418,7 +419,7 @@ Namespace API.Reddit
                     CountOfAddedUsers.Clear()
                     CountOfLoadedPostsPerSession.Clear()
                     ChannelExistentUserNames.Clear()
-                    CachePath.Delete(SFO.Path, SFODelete.None, EDP.SendInLog)
+                    CachePath.Delete(SFO.Path, SFODelete.None, EDP.SendToLog)
                 End If
                 disposedValue = True
             End If

SCrawler/API/Reddit/ChannelsCollection.vb

@@ -55,7 +55,7 @@ Namespace API.Reddit
                     Return Nothing
                 End If
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Reddit.ChannelsCollection.GetUserFiles]")
+                Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, "[API.Reddit.ChannelsCollection.GetUserFiles]")
             End Try
         End Function
         Friend Sub UpdateUsersStats()

SCrawler/API/Reddit/Declarations.vb

@@ -15,10 +15,12 @@ Namespace API.Reddit
         Friend ReadOnly JsonNodesJson() As NodeParams = {New NodeParams("posts", True, True, True, True, 3)}
         Friend ReadOnly ChannelJsonNodes() As NodeParams = {New NodeParams("data", True, True, True, True, 1),
                                                             New NodeParams("children", True, True, True)}
+        Friend ReadOnly SingleJsonNodes() As NodeParams = {New NodeParams("data", True, True, True, True, 2),
+                                                           New NodeParams("children", True, True, True),
+                                                           New NodeParams("data", True, True, True, True, 1)}
         Friend ReadOnly UrlBasePattern As RParams = RParams.DM("(?<=/)([^/]+?\.[\w]{3,4})(?=(\?|\Z))", 0)
         Friend ReadOnly VideoRegEx As RParams = RParams.DM("http.{0,1}://[^" & Chr(34) & "]+?mp4", 0)
         Private ReadOnly EUR_PROVIDER As New ANumbers(ANumbers.Cultures.EUR)
-        Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicodeJS(v, n, e))
-        Friend ReadOnly DateProviderChannel As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(AConvert(Of Integer)(v, EUR_PROVIDER, v), n, e))
+        Friend ReadOnly UnixDate32ProviderReddit As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnix32(AConvert(Of Integer)(v, EUR_PROVIDER, v), n, e))
     End Module
 End Namespace

SCrawler/API/Reddit/M3U8.vb

@@ -7,8 +7,11 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
+Imports System.Threading
 Imports SCrawler.API.Reddit.M3U8_Declarations
+Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Web
+Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Reddit
     Namespace M3U8_Declarations
@@ -19,7 +22,7 @@ Namespace API.Reddit
             ''' <summary>Audio, Video</summary>
             Friend ReadOnly PlayListRegEx_2 As RParams = RParams.DM("(?<=#EXT-X-BYTERANGE.+?[\r\n]{1,2})(.+)(?=[\r\n]{0,2})", 0, RegexReturn.List)
             Friend ReadOnly PlayListAudioRegEx As RParams = RParams.DM("(HLS_AUDIO_(\d+)[^""]+)", 0, RegexReturn.List)
-            Friend ReadOnly DPED As New ErrorsDescriber(EDP.SendInLog + EDP.ReturnValue)
+            Friend ReadOnly DPED As New ErrorsDescriber(EDP.SendToLog + EDP.ReturnValue)
         End Module
     End Namespace
     Friend NotInheritable Class M3U8 : Implements IDisposable
@@ -52,9 +55,12 @@ Namespace API.Reddit
         Private OutFile As SFile
         Private VideoFile As SFile
         Private AudioFile As SFile
-        Private CachePath As SFile
+        Private ReadOnly Cache As CacheKeeper
+        Private ReadOnly CacheFiles As CacheKeeper
+        Private ReadOnly Property Progress As MyProgress
+        Private ReadOnly ProgressExists As Boolean
 #End Region
-        Private Sub New(ByVal URL As String, ByVal OutFile As SFile)
+        Private Sub New(ByVal URL As String, ByVal OutFile As SFile, ByVal Progress As MyProgress)
             PlayListURL = URL
             BaseURL = RegexReplace(URL, BaseUrlPattern)
             Video = New List(Of String)
@@ -62,7 +68,10 @@ Namespace API.Reddit
             Me.OutFile = OutFile
             Me.OutFile.Name = "PlayListFile"
             Me.OutFile.Extension = "mp4"
-            CachePath = $"{OutFile.PathWithSeparator}_Cache\{SFile.GetDirectories($"{OutFile.PathWithSeparator}_Cache\",,, EDP.ReturnValue).ListIfNothing.Count + 1}\"
+            Me.Progress = Progress
+            ProgressExists = Not Me.Progress Is Nothing
+            Cache = New CacheKeeper($"{OutFile.PathWithSeparator}_{Base.M3U8Base.TempCacheFolderName}\")
+            CacheFiles = Cache.NewInstance
         End Sub
 #Region "Internal functions"
 #Region "GetPlaylistUrls"
@@ -78,7 +87,7 @@ Namespace API.Reddit
                         If Not r.IsEmptyString Then
                             Dim l As New List(Of Resolution)
                             If Type = Types.Video Then
-                                l = RegexFields(Of Resolution)(r, {PlayListRegEx_1}, {6, 4})
+                                l = RegexFields(Of Resolution)(r, {PlayListRegEx_1}, {6, 4}, EDP.ReturnValue)
                             Else
                                 Try
                                     l = RegexFields(Of Resolution)(r, {PlayListAudioRegEx}, {1, 2})
@@ -112,41 +121,44 @@ Namespace API.Reddit
         End Function
 #End Region
 #Region "ConcatData"
-        Private Overloads Sub ConcatData()
-            ConcatData(Video, Types.Video, VideoFile)
-            ConcatData(Audio, Types.Audio, AudioFile)
+        Private Overloads Sub ConcatData(ByVal Token As CancellationToken)
+            ConcatData(Video, Types.Video, VideoFile, Token)
+            ConcatData(Audio, Types.Audio, AudioFile, Token)
             MergeFiles()
         End Sub
-        Private Overloads Sub ConcatData(ByVal Urls As List(Of String), ByVal Type As Types, ByRef TFile As SFile)
+        Private Overloads Sub ConcatData(ByVal Urls As List(Of String), ByVal Type As Types, ByRef TFile As SFile, ByVal Token As CancellationToken)
             Try
+                Token.ThrowIfCancellationRequested()
                 If Urls.ListExists Then
-                    Dim ConcatFile As SFile = OutFile
+                    Dim tmpCache As CacheKeeper = CacheFiles.NewInstance
+                    Dim ConcatFile As SFile = CacheFiles
                     If Type = Types.Audio Then
-                        ConcatFile.Name &= "_AUDIO"
+                        ConcatFile.Name &= "AUDIO"
                         ConcatFile.Extension = "aac"
                     Else
-                        If Audio.Count > 0 Then ConcatFile.Name &= "_VIDEO"
+                        If Audio.Count > 0 Then ConcatFile.Name &= "VIDEO"
                         ConcatFile.Extension = "mp4"
                     End If
-                    If CachePath.Exists(SFO.Path) Then
-                        Dim p As New SFileNumbers(ConcatFile.Name,,, New ANumbers With {.Format = ANumbers.Formats.General})
-                        ConcatFile = SFile.Indexed_IndexFile(ConcatFile,, p, EDP.ThrowException)
+                    If tmpCache.Validate Then
                         Dim i%
-                        Dim eFiles As New List(Of SFile)
-                        Dim dFile As SFile = CachePath
+                        Dim dFile As SFile = tmpCache.RootDirectory
+                        If ProgressExists Then Progress.Maximum += Urls.Count
                         dFile.Extension = New SFile(Urls(0)).Extension
                         If dFile.Extension.IsEmptyString Then dFile.Extension = "ts"
                         Using w As New WebClient
                             For i = 0 To Urls.Count - 1
+                                If ProgressExists Then Progress.Perform()
+                                Token.ThrowIfCancellationRequested()
                                 dFile.Name = $"ConPart_{i}"
                                 w.DownloadFile(Urls(i), dFile)
-                                eFiles.Add(dFile)
+                                tmpCache.AddFile(dFile, True)
                             Next
                         End Using
-                        TFile = FFMPEG.ConcatenateFiles(eFiles, Settings.FfmpegFile, ConcatFile, p, DPED)
-                        eFiles.Clear()
+                        TFile = FFMPEG.ConcatenateFiles(tmpCache, Settings.FfmpegFile.File, ConcatFile, Settings.CMDEncoding,, DPED)
                     End If
                 End If
+            Catch oex As OperationCanceledException When Token.IsCancellationRequested
+                Throw oex
             Catch ex As Exception
                 ErrorsDescriber.Execute(DPED, ex, $"[M3U8.Save({Type})]")
             End Try
@@ -154,25 +166,27 @@ Namespace API.Reddit
 #End Region
         Private Sub MergeFiles()
             Try
+                Dim p As SFileNumbers = SFileNumbers.Default(OutFile.Name)
+                Dim f As SFile = SFile.IndexReindex(OutFile,,, p, EDP.ReturnValue)
                 If Not VideoFile.IsEmptyString And Not AudioFile.IsEmptyString Then
-                    Dim p As New SFileNumbers(OutFile.Name,, RParams.DMS("PlayListFile_(\d*)", 1), New ANumbers With {.Format = ANumbers.Formats.General})
-                    OutFile = FFMPEG.MergeFiles({VideoFile, AudioFile}, Settings.FfmpegFile, OutFile, p, DPED)
+                    OutFile = FFMPEG.MergeFiles({VideoFile, AudioFile}, Settings.FfmpegFile.File, f, Settings.CMDEncoding, p, DPED)
                 Else
-                    OutFile = VideoFile
+                    If f.IsEmptyString Then f = OutFile
+                    If Not SFile.Move(VideoFile, f) Then OutFile = VideoFile
                 End If
             Catch ex As Exception
                 ErrorsDescriber.Execute(DPED, ex, $"[M3U8.MergeFiles]")
             End Try
         End Sub
-        Friend Function Download() As SFile
+        Friend Function Download(ByVal Token As CancellationToken) As SFile
             GetPlaylistUrls()
-            ConcatData()
+            ConcatData(Token)
             Return OutFile
         End Function
 #End Region
 #Region "Statics"
-        Friend Shared Function Download(ByVal URL As String, ByVal f As SFile) As SFile
-            Using m As New M3U8(URL, f) : Return m.Download() : End Using
+        Friend Shared Function Download(ByVal URL As String, ByVal f As SFile, ByVal Token As CancellationToken, ByVal Progress As MyProgress) As SFile
+            Using m As New M3U8(URL, f, Progress) : Return m.Download(Token) : End Using
         End Function
 #End Region
 #Region "IDisposable Support"
@@ -182,7 +196,7 @@ Namespace API.Reddit
                 If disposing Then
                     Video.Clear()
                     Audio.Clear()
-                    CachePath.Delete(SFO.Path, SFODelete.None, DPED)
+                    Cache.Dispose()
                 End If
                 disposedValue = True
             End If

SCrawler/API/Reddit/RedditViewSettingsForm.vb

@@ -18,7 +18,7 @@ Namespace API.Reddit
             MyOptions = opt
             MyDefs = New DefaultFormOptions(Me, Settings.Design)
         End Sub
-        Private Sub ChannelSettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
+        Private Sub RedditViewSettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
             Try
                 Dim n$ = String.Empty
                 If TypeOf MyOptions Is Channel Then

SCrawler/API/Reddit/SiteSettings.vb

@@ -38,31 +38,28 @@ Namespace API.Reddit
             End With
             SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
             UseM3U8 = New PropertyValue(True)
-            UrlPatternUser = "https://www.reddit.com/user/{0}/"
-            UrlPatternChannel = "https://www.reddit.com/r/{0}/"
+            UrlPatternUser = "https://www.reddit.com/{0}/{1}/"
             ImageVideoContains = "reddit.com"
             UserRegex = RParams.DM("[htps:/]{7,8}.*?reddit.com/([user]{1,4})/([^/]+)", 0, RegexReturn.ListByMatch, EDP.ReturnValue)
         End Sub
         Friend Overrides Function GetInstance(ByVal What As Download) As IPluginContentProvider
-            Select Case What
-                Case Download.Main : Return New UserData
-                Case Download.Channel : Return New UserData With {.SaveToCache = False, .SkipExistsUsers = False, .AutoGetLimits = True}
-                Case Download.SavedPosts
-                    Dim u As New UserData With {.IsSavedPosts = True}
-                    DirectCast(u, UserDataBase).User = New UserInfo With {
-                        .Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty)),
-                        .IsChannel = True
-                    }
-                    Return u
-            End Select
-            Return Nothing
+            Return New UserData
         End Function
+        Friend Const ChannelOption As String = "r"
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
             Dim l As List(Of String) = RegexReplace(UserURL, UserRegex)
-            If l.ListExists(3) Then Return New ExchangeOptions(Site, l(2), l(1) = "r") Else Return Nothing
+            If l.ListExists(3) Then
+                Dim n$ = l(2)
+                If Not l(1).IsEmptyString AndAlso l(1) = ChannelOption Then n &= $"@{ChannelOption}"
+                Return New ExchangeOptions(Site, n)
+            Else
+                Return Nothing
+            End If
         End Function
         Friend Overrides Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
             Try
+                Dim trueValue As Boolean = Not What = Download.SavedPosts OrElse (Responser.CookiesExists And ACheck(SavedPostsUserName.Value))
+                If Not trueValue Then Return False
                 Dim dl As List(Of DownDetector.Data) = DownDetector.GetData("reddit")
                 If dl.ListExists Then
                     dl = dl.Take(4).ToList
@@ -76,7 +73,7 @@ Namespace API.Reddit
                                         dl.ListToString(vbCr) & vbCr & vbCr &
                                         "Do you want to continue parsing Reddit data?", "There are outage reports on Reddit"}, vbYesNo) = vbYes Then
                                 UpdateRedGifsToken()
-                                Return True
+                                Return trueValue
                             Else
                                 Return False
                             End If
@@ -84,28 +81,29 @@ Namespace API.Reddit
                     End If
                 End If
                 UpdateRedGifsToken()
-                Return True
+                Return trueValue
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, "[API.Reddit.SiteSettings.Available]", True)
+                Return ErrorsDescriber.Execute(EDP.SendToLog + EDP.ReturnValue, ex, "[API.Reddit.SiteSettings.Available]", True)
             End Try
         End Function
         Private Sub UpdateRedGifsToken()
             DirectCast(Settings(RedGifs.RedGifsSiteKey).Source, RedGifs.SiteSettings).UpdateTokenIfRequired()
         End Sub
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            Dim spf$ = String.Empty
-            Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
-            f = $"{f.PathWithSeparator}OptionalPath\"
-            Return UserData.GetVideoInfo(URL, Responser, f, spf)
-        End Function
         Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
             If Options Is Nothing OrElse Not TypeOf Options Is RedditViewExchange Then Options = New RedditViewExchange
             If OpenForm Then
                 Using f As New RedditViewSettingsForm(Options) : f.ShowDialog() : End Using
             End If
         End Sub
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
+            With DirectCast(User, UserData) : Return String.Format(UrlPatternUser, IIf(.IsChannel, ChannelOption, "user"), .TrueName) : End With
+        End Function
         Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return $"https://www.reddit.com/comments/{Media.Post.ID.Split("_").LastOrDefault}/"
+            If Not Media.Post.ID.IsEmptyString Then
+                Return $"https://www.reddit.com/comments/{Media.Post.ID.Split("_").LastOrDefault}/"
+            Else
+                Return String.Empty
+            End If
         End Function
     End Class
 End Namespace

SCrawler/API/Redgifs/Declarations.vb

@@ -11,7 +11,6 @@ Namespace API.RedGifs
     Friend Module Declarations
         Friend Const RedGifsSiteKey As String = "AndyProgram_RedGifs"
         Friend Const RedGifsSite As String = "RedGifs"
-        Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v, n, e))
         Friend ReadOnly WatchIDRegex As RParams = RParams.DMS(".+?watch/([^\?&""/]+)", 1, EDP.ReturnValue)
         Friend ReadOnly ThumbsIDRegex As RParams = RParams.DMS("([^/\?&""]+?)(-\w+?|)\.(mp4|jpg)", 1, EDP.ReturnValue,
                                                                CType(Function(Input$) Input.StringToLower.StringTrim, Func(Of String, String)))

SCrawler/API/Redgifs/SiteSettings.vb

@@ -14,8 +14,6 @@ Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Documents.JSON
-Imports UTypes = SCrawler.API.Base.UserMedia.Types
-Imports UStates = SCrawler.API.Base.UserMedia.States
 Namespace API.RedGifs
     <Manifest(RedGifsSiteKey)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
@@ -32,18 +30,17 @@ Namespace API.RedGifs
         End Property
         <PropertyOption(ControlToolTip:="Bearer token", AllowNull:=False), ControlNumber(1)>
         Friend ReadOnly Property Token As PropertyValue
+        <PropertyOption, ControlNumber(2)>
+        Private ReadOnly Property UserAgent As PropertyValue
         <PXML> Friend ReadOnly Property TokenLastDateUpdated As PropertyValue
         Private Const TokenName As String = "authorization"
 #Region "TokenUpdateInterval"
         <PropertyOption(ControlText:="Token refresh interval", ControlToolTip:="Interval (in minutes) to refresh the token", AllowNull:=False, LeftOffset:=120),
             PXML, ControlNumber(0)>
         Friend ReadOnly Property TokenUpdateInterval As PropertyValue
-        Private Class TokenIntervalProvider : Implements IFieldsCheckerProvider
-            Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
-            Private Property Name As String Implements IFieldsCheckerProvider.Name
-            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
-            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
-                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+        Private Class TokenIntervalProvider : Inherits FieldsCheckerProviderBase
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
                 TypeError = False
                 ErrorMessage = String.Empty
                 If Not ACheck(Of Integer)(Value) Then
@@ -52,12 +49,10 @@ Namespace API.RedGifs
                     Return Value
                 Else
                     ErrorMessage = $"The value of [{Name}] field must be greater than or equal to 1"
+                    HasError = True
                 End If
                 Return Nothing
             End Function
-            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("[GetFormat] is not available in the context of [TokenIntervalProvider]")
-            End Function
         End Class
         <Provider(NameOf(TokenUpdateInterval), FieldsChecker:=True)>
         Private ReadOnly Property TokenUpdateIntervalProvider As IFormatProvider
@@ -68,12 +63,14 @@ Namespace API.RedGifs
             MyBase.New(RedGifsSite, "redgifs.com")
             Dim t$ = String.Empty
             With Responser
-                Dim b As Boolean = Not .Mode = Responser.Modes.WebClient
                 .Mode = Responser.Modes.WebClient
+                If Not .UserAgentExists Then .UserAgent = ParserUserAgent
+                .ClientWebUseCookies = False
+                .ClientWebUseHeaders = True
                 t = .Headers.Value(TokenName)
-                If b Then .SaveSettings()
             End With
-            Token = New PropertyValue(t, GetType(String), Sub(v) UpdateResponse(v))
+            Token = New PropertyValue(t, GetType(String), Sub(v) UpdateResponse(NameOf(Token), v))
+            UserAgent = New PropertyValue(Responser.UserAgent, GetType(String), Sub(v) UpdateResponse(NameOf(UserAgent), v))
             TokenLastDateUpdated = New PropertyValue(Now.AddYears(-1), GetType(Date))
             TokenUpdateInterval = New PropertyValue(60 * 12, GetType(Integer))
             TokenUpdateIntervalProvider = New TokenIntervalProvider
@@ -83,8 +80,11 @@ Namespace API.RedGifs
         End Sub
 #End Region
 #Region "Response updater"
-        Private Sub UpdateResponse(ByVal Value As String)
-            Responser.Headers.Add(TokenName, Value)
+        Private Sub UpdateResponse(ByVal Name As String, ByVal Value As String)
+            Select Case Name
+                Case NameOf(Token) : Responser.Headers.Add(TokenName, Value)
+                Case NameOf(UserAgent) : Responser.UserAgent = Value
+            End Select
             Responser.SaveSettings()
         End Sub
 #End Region
@@ -101,16 +101,18 @@ Namespace API.RedGifs
         Friend Function UpdateToken() As Boolean
             Try
                 Dim r$
-                Dim NewToken$ = String.Empty
+                Dim NewToken$ = String.Empty, NewAgent$ = String.Empty
                 Using resp As New Responser : r = resp.GetResponse("https://api.redgifs.com/v2/auth/temporary",, EDP.ThrowException) : End Using
                 If Not r.IsEmptyString Then
                     Dim j As EContainer = JsonDocument.Parse(r)
                     If Not j Is Nothing Then
                         NewToken = j.Value("token")
+                        NewAgent = j.Value("agent")
                         j.Dispose()
                     End If
                 End If
                 If Not NewToken.IsEmptyString Then
+                    If Not NewAgent.IsEmptyString Then UserAgent.Value = NewAgent
                     Token.Value = $"Bearer {NewToken}"
                     TokenLastDateUpdated.Value = Now
                     Return True
@@ -118,7 +120,7 @@ Namespace API.RedGifs
                     Return False
                 End If
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.RedGifs.SiteSettings.UpdateToken]", False)
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.RedGifs.SiteSettings.UpdateToken]", False)
             End Try
         End Function
 #End Region
@@ -129,8 +131,10 @@ Namespace API.RedGifs
             MyBase.BeginEdit()
         End Sub
         Friend Overrides Sub Update()
-            Dim NewToken$ = AConvert(Of String)(Token.Value, AModes.Var, String.Empty)
-            If Not _LastTokenValue = NewToken Then TokenLastDateUpdated.Value = Now
+            If _SiteEditorFormOpened Then
+                Dim NewToken$ = AConvert(Of String)(Token.Value, AModes.Var, String.Empty)
+                If Not _LastTokenValue = NewToken Then TokenLastDateUpdated.Value = Now
+            End If
             MyBase.Update()
         End Sub
         Friend Overrides Sub EndEdit()
@@ -141,32 +145,6 @@ Namespace API.RedGifs
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
             Return New UserData
         End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            If BaseAuthExists() Then
-                Using resp As Responser = Responser.Copy
-                    Dim m As UserMedia = UserData.GetDataFromUrlId(URL, False, resp, Settings(RedGifsSiteKey))
-                    If Not m.State = UStates.Missing And Not m.State = UserData.DataGone And (m.Type = UTypes.Picture Or m.Type = UTypes.Video) Then
-                        Try
-                            Dim spf$ = String.Empty
-                            Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
-                            If f.IsEmptyString Then
-                                f = m.File.File
-                            Else
-                                f.Name = m.File.Name
-                                f.Extension = m.File.Extension
-                            End If
-                            resp.DownloadFile(m.URL, f, EDP.ThrowException)
-                            m.State = UStates.Downloaded
-                            m.SpecialFolder = spf
-                            Return {m}
-                        Catch ex As Exception
-                            ErrorsDescriber.Execute(EDP.SendInLog, ex, $"Redgifs standalone download error: [{URL}]")
-                        End Try
-                    End If
-                End Using
-            End If
-            Return Nothing
-        End Function
         Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
             Return $"https://www.redgifs.com/watch/{Media.Post.ID}"
         End Function

SCrawler/API/Redgifs/UserData.vb

@@ -9,6 +9,7 @@
 Imports System.Net
 Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
@@ -42,7 +43,7 @@ Namespace API.RedGifs
             Try
                 Dim _page As Func(Of String) = Function() If(Page = 1, String.Empty, $"&page={Page}")
                 URL = $"https://api.redgifs.com/v2/users/{Name}/search?order=recent{_page.Invoke}"
-                Dim r$ = Responser.GetResponse(URL,, EDP.ThrowException)
+                Dim r$ = Responser.GetResponse(URL)
                 Dim postDate$, postID$
                 Dim pTotal% = 0
                 If Not r.IsEmptyString Then
@@ -51,7 +52,7 @@ Namespace API.RedGifs
                             pTotal = j.Value("pages").FromXML(Of Integer)(0)
                             For Each g As EContainer In j("gifs")
                                 postDate = g.Value("createDate")
-                                Select Case CheckDatesLimit(postDate, DateProvider)
+                                Select Case CheckDatesLimit(postDate, UnixDate32Provider)
                                     Case DateResult.Skip : Continue For
                                     Case DateResult.Exit : Exit Sub
                                 End Select
@@ -106,13 +107,13 @@ Namespace API.RedGifs
                     Dim u As UserMedia
                     Dim j As EContainer
                     For i% = 0 To _ContentList.Count - 1
-                        If _ContentList(i).State = UserMedia.States.Missing Then
+                        If _ContentList(i).State = UStates.Missing Then
                             ThrowAny(Token)
                             u = _ContentList(i)
                             If Not u.Post.ID.IsEmptyString Then
                                 url = String.Format(PostDataUrl, u.Post.ID.ToLower)
                                 Try
-                                    r = Responser.GetResponse(url,, EDP.ThrowException)
+                                    r = Responser.GetResponse(url)
                                     If Not r.IsEmptyString Then
                                         j = JsonDocument.Parse(r)
                                         If Not j Is Nothing Then
@@ -207,20 +208,29 @@ Namespace API.RedGifs
                         MyMainLOG = String.Format(_errText, URL)
                         Return m
                     Else
-                        Return ErrorsDescriber.Execute(EDP.SendInLog, ex, String.Format(_errText, URL), m)
+                        Return ErrorsDescriber.Execute(EDP.SendToLog, ex, String.Format(_errText, URL), m)
                     End If
                 End If
             End Try
         End Function
 #End Region
+#Region "Single data downloader"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim m As UserMedia = GetDataFromUrlId(Data.URL, False, Responser, HOST)
+            If Not m.State = UStates.Missing And Not m.State = DataGone And (m.Type = UTypes.Picture Or m.Type = UTypes.Video) Then
+                m.URL_BASE = MySettings.GetUserPostUrl(Me, m)
+                _TempMediaList.Add(m)
+            End If
+        End Sub
+#End Region
 #Region "Create media"
-        Private Shared Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String,
-                                              ByVal PostDateStr As String, ByVal PostDateDate As Date?, ByVal State As UStates) As UserMedia
+        Private Function MediaFromData(ByVal t As UTypes, ByVal _URL As String, ByVal PostID As String,
+                                       ByVal PostDateStr As String, ByVal PostDateDate As Date?, ByVal State As UStates) As UserMedia
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
             Dim m As New UserMedia(_URL, t) With {.Post = New UserPost With {.ID = PostID}}
             If Not m.URL.IsEmptyString Then m.File = CStr(RegexReplace(m.URL, FilesPattern))
             If Not PostDateStr.IsEmptyString Then
-                m.Post.Date = AConvert(Of Date)(PostDateStr, DateProvider, Nothing)
+                m.Post.Date = AConvert(Of Date)(PostDateStr, UnixDate32Provider, Nothing)
             ElseIf PostDateDate.HasValue Then
                 m.Post.Date = PostDateDate
             Else
@@ -233,8 +243,8 @@ Namespace API.RedGifs
 #Region "Exception"
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
                                                           Optional ByVal EObj As Object = Nothing) As Integer
-            Dim s As WebExceptionStatus = Responser.Client.Status
-            Dim sc As HttpStatusCode = Responser.Client.StatusCode
+            Dim s As WebExceptionStatus = Responser.Status
+            Dim sc As HttpStatusCode = Responser.StatusCode
             If sc = HttpStatusCode.NotFound Or s = DataGone Then
                 UserExists = False
             ElseIf sc = HttpStatusCode.Unauthorized Then

SCrawler/API/ThisVid/Declarations.vb

@@ -0,0 +1,22 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.ThisVid
+    Friend Module Declarations
+        Friend Const ThisVidSiteKey As String = "AndyProgram_ThisVid"
+        Friend ReadOnly RegExNextPage As RParams = RParams.DMS("class=.pagination-next...a class=.selective..href=""([^""]+)""", 1)
+        Friend ReadOnly RegExVideoList As RParams = RParams.DMS("\<a href=""([^""]+)"" title=""[^""]+"" class=""tumbpu""", 1, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly RegExVideoListSavedPosts As RParams = RParams.DMS("\<a href=""([^""]+)"" title=""[^""]+"">[\r\n\s]*<span class=""thumb ", 1, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly RegExAlbumsList As RParams = RParams.DM("(https://thisvid.com/albums/[^/]+/?)"" title=""([^""]*?)"" class=""tumbpu""", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly RegExAlbumsListSaved As RParams = RParams.DM("(https://thisvid.com/albums/[^/]+/?)"" title=""([^""]*?)"">[\r\n\s]*\<span class=""thumb""", 0, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly RegExAlbumID As RParams = RParams.DMS("albumId:.'(\d+)'", 1)
+        Friend ReadOnly RegExAlbumImagesList As RParams = RParams.DMS("""([^""]+?image\d+/?)""", 1, RegexReturn.List, EDP.ReturnValue)
+        Friend ReadOnly RegExAlbumImageUrl As RParams = RParams.DMS("\<img src=""(https?://media.thisvid.com/contents/albums/[^""]+?)""", 1, EDP.ReturnValue)
+    End Module
+End Namespace

SCrawler/API/ThisVid/SiteSettings.vb

@@ -0,0 +1,69 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.ThisVid
+    <Manifest(ThisVidSiteKey), SeparatedTasks(1), SpecialForm(False), SavedPosts>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteResources.ThisVidIcon_16
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteResources.ThisVidPic_16
+            End Get
+        End Property
+        <PXML, PropertyOption(ControlText:="Public videos", ControlToolTip:="Download public videos")>
+        Friend ReadOnly Property DownloadPublic As PropertyValue
+        <PXML, PropertyOption(ControlText:="Private videos", ControlToolTip:="Download private videos")>
+        Friend ReadOnly Property DownloadPrivate As PropertyValue
+        <PXML, PropertyOption(ControlText:="Different folders",
+                              ControlToolTip:="Use different folders to store video files." & vbCr &
+                                              "If true, then public videos will be stored in the 'Public' folder, private - in the 'Private' folder." & vbCr &
+                                              "If false, all videos will be stored in the 'Video' folder.")>
+        Friend ReadOnly Property DifferentFolders As PropertyValue
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New("ThisVid", "thisvid.com")
+            DownloadPublic = New PropertyValue(True)
+            DownloadPrivate = New PropertyValue(True)
+            DifferentFolders = New PropertyValue(True)
+            CheckNetscapeCookiesOnEndInit = True
+            UseNetscapeCookies = True
+            UserRegex = RParams.DMS("thisvid.com/members/(\d+)", 1)
+            UrlPatternUser = "https://thisvid.com/members/{0}/"
+            ImageVideoContains = "https://thisvid.com/videos/"
+        End Sub
+#End Region
+#Region "GetInstance, GetSpecialData"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+#End Region
+#Region "Downloading"
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return Settings.YtdlpFile.Exists And (What = ISiteSettings.Download.SingleObject Or Responser.CookiesExists)
+        End Function
+#End Region
+#Region "UserOptions"
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse Not TypeOf Options Is UserExchangeOptions Then Options = New UserExchangeOptions(Me)
+            If OpenForm Then
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
+            End If
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/ThisVid/UserData.vb

@@ -0,0 +1,332 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools
+Imports PersonalUtilities.Tools.Web.Documents.JSON
+Namespace API.ThisVid
+    Friend Class UserData : Inherits UserDataBase
+#Region "XML names"
+        Private Const Name_DownloadPublic As String = "DownloadPublic"
+        Private Const Name_DownloadPrivate As String = "DownloadPrivate"
+        Private Const Name_DifferentFolders As String = "DifferentFolders"
+#End Region
+#Region "Structures"
+        Private Structure Album : Implements IRegExCreator
+            Friend URL As String
+            Friend Title As String
+            Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray
+                If ParamsArray.ListExists(2) Then
+                    URL = ParamsArray(0)
+                    Title = TitleHtmlConverter(ParamsArray(1))
+                End If
+                Return Me
+            End Function
+        End Structure
+#End Region
+#Region "Declarations"
+        Friend Property DownloadPublic As Boolean = True
+        Friend Property DownloadPrivate As Boolean = True
+        Friend Property DifferentFolders As Boolean = True
+#End Region
+#Region "Loaders"
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            With Container
+                If Loading Then
+                    DownloadPublic = .Value(Name_DownloadPublic).FromXML(Of Boolean)(True)
+                    DownloadPrivate = .Value(Name_DownloadPrivate).FromXML(Of Boolean)(True)
+                    DifferentFolders = .Value(Name_DifferentFolders).FromXML(Of Boolean)(True)
+                Else
+                    .Add(Name_DownloadPublic, DownloadPublic.BoolToInteger)
+                    .Add(Name_DownloadPrivate, DownloadPrivate.BoolToInteger)
+                    .Add(Name_DifferentFolders, DifferentFolders.BoolToInteger)
+                End If
+            End With
+        End Sub
+        Friend Overrides Function ExchangeOptionsGet() As Object
+            Return New UserExchangeOptions(Me)
+        End Function
+        Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
+            If Not Obj Is Nothing AndAlso TypeOf Obj Is UserExchangeOptions Then
+                With DirectCast(Obj, UserExchangeOptions)
+                    DownloadPublic = .DownloadPublic
+                    DownloadPrivate = .DownloadPrivate
+                    DifferentFolders = .DifferentFolders
+                End With
+            End If
+        End Sub
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            UseClientTokens = True
+        End Sub
+#End Region
+#Region "Validation"
+        Private Function IsValid() As Boolean
+            Const ProfileDataPattern$ = "{0}[\r\n\s\W]*:[\r\n\s\W]*\<strong\>[\r\n\s\W]*([^\<]*)[\r\n\s\W]*\</strong"
+            Const DescriptionPattern$ = "span style=""line-height: \d*px;""\>[\r\n\s\W]*([^\<]*)[\r\n\s\W]*\<"
+            Try
+                If Not IsSavedPosts Then
+                    Dim r$ = Responser.GetResponse($"https://thisvid.com/members/{ID}/")
+                    If Not r.IsEmptyString Then
+                        Dim rr As New RParams("", Nothing, 1, EDP.ReturnValue)
+                        Dim __getValue As Func(Of String, Boolean, String) = Function(ByVal member As String, ByVal appendMember As Boolean) As String
+                                                                                 rr.Pattern = String.Format(ProfileDataPattern, member)
+                                                                                 Dim v$ = CStr(RegexReplace(r, rr)).StringTrim
+                                                                                 If Not v.IsEmptyString And appendMember Then v = $"{member}: {v}"
+                                                                                 Return v
+                                                                             End Function
+                        UserSiteNameUpdate(__getValue("Name", False))
+                        If Not UserSiteName.IsEmptyString And FriendlyName.IsEmptyString Then FriendlyName = UserSiteName : _ForceSaveUserData = True
+                        Dim descr$ = String.Empty
+                        descr.StringAppendLine(__getValue("Birth date", True))
+                        descr.StringAppendLine(__getValue("Country", True))
+                        descr.StringAppendLine(__getValue("City", True))
+                        descr.StringAppendLine(__getValue("Gender", True))
+                        descr.StringAppendLine(__getValue("Orientation", True))
+                        descr.StringAppendLine(__getValue("Relationship status", True))
+                        descr.StringAppendLine(__getValue("Favourite category", True))
+                        descr.StringAppendLine(__getValue("My interests", True))
+                        rr.Pattern = DescriptionPattern
+                        descr.StringAppendLine(CStr(RegexReplace(r, rr)).StringTrim)
+                        UserDescriptionUpdate(descr)
+                    Else
+                        Return False
+                    End If
+                End If
+                Return True
+            Catch ex As Exception
+                UserExists = False
+                Return False
+            End Try
+        End Function
+#End Region
+#Region "Download functions"
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            If ID.IsEmptyString Then ID = Name
+            If IsValid() Then
+                If IsSavedPosts Then
+                    DownloadData(1, True, Token)
+                    DownloadData_Images(Token)
+                Else
+                    If DownloadVideos Then
+                        If DownloadPublic Then DownloadData(1, True, Token)
+                        If DownloadPrivate Then DownloadData(1, False, Token)
+                    End If
+                    If DownloadImages Then DownloadData_Images(Token)
+                End If
+            End If
+        End Sub
+        Private Overloads Sub DownloadData(ByVal Page As Integer, ByVal IsPublic As Boolean, ByVal Token As CancellationToken)
+            Dim URL$ = String.Empty
+            Try
+                Dim p$ = IIf(Page = 1, String.Empty, $"{Page}/")
+                If IsSavedPosts Then
+                    URL = $"https://thisvid.com/my_favourite_videos/{p}"
+                Else
+                    URL = $"https://thisvid.com/members/{ID}/{IIf(IsPublic, "public", "private")}_videos/{p}"
+                End If
+                ThrowAny(Token)
+                Dim r$ = Responser.GetResponse(URL)
+                Dim cBefore% = _TempMediaList.Count
+                If Not r.IsEmptyString Then
+                    Dim __SpecialFolder$ = IIf(DifferentFolders, IIf(IsPublic, "Public", "Private"), String.Empty)
+                    Dim l As List(Of String) = RegexReplace(r, If(IsSavedPosts, RegExVideoListSavedPosts, RegExVideoList))
+                    If l.ListExists Then
+                        For Each u$ In l
+                            If Not u.IsEmptyString Then
+                                If Not _TempPostsList.Contains(u) Then
+                                    _TempPostsList.Add(u)
+                                    _TempMediaList.Add(New UserMedia(u) With {.Type = UserMedia.Types.VideoPre, .SpecialFolder = __SpecialFolder})
+                                Else
+                                    Exit Sub
+                                End If
+                            End If
+                        Next
+                    End If
+                End If
+                If Not cBefore = _TempMediaList.Count Then DownloadData(Page + 1, IsPublic, Token)
+            Catch ex As Exception
+                ProcessException(ex, Token, $"videos downloading error [{URL}]")
+            End Try
+        End Sub
+        Private Sub DownloadData_Images(ByVal Token As CancellationToken)
+            Dim __baseUrl$ = If(IsSavedPosts, "https://thisvid.com/my_favourite_albums/", $"https://thisvid.com/members/{ID}/albums/")
+            Dim URL$ = String.Empty
+            Try
+                Dim r$
+                Dim i% = 0
+                Dim __continue As Boolean = False
+                Dim rAlbums As RParams = If(IsSavedPosts, RegExAlbumsListSaved, RegExAlbumsList)
+                Do
+                    i += 1
+                    __continue = False
+                    URL = __baseUrl
+                    If i > 1 Then URL &= $"{i}/"
+                    r = Responser.GetResponse(URL)
+                    If Not r.IsEmptyString() Then
+                        Dim albums As List(Of Album) = RegexFields(Of Album)(r, {rAlbums}, {1, 2}, EDP.ReturnValue)
+                        Dim images As List(Of String)
+                        Dim albumId$, img$, imgUrl$, imgId$
+                        Dim u As UserMedia
+                        Dim rErr As New ErrorsDescriber(EDP.ReturnValue)
+                        __continue = True
+                        If albums.ListExists Then
+                            If albums.Count < 20 Then __continue = False
+                            For Each a As Album In albums
+                                If Not a.URL.IsEmptyString Then
+                                    ThrowAny(Token)
+                                    r = Responser.GetResponse(a.URL,, rErr)
+                                    If Not r.IsEmptyString Then
+                                        albumId = RegexReplace(r, RegExAlbumID)
+                                        If a.Title.IsEmptyString Then a.Title = albumId
+                                        images = RegexReplace(r, RegExAlbumImagesList)
+                                        If images.ListExists Then
+                                            For Each img In images
+                                                ThrowAny(Token)
+                                                r = Responser.GetResponse(img,, rErr)
+                                                If Not r.IsEmptyString Then
+                                                    imgUrl = RegexReplace(r, RegExAlbumImageUrl)
+                                                    If Not imgUrl.IsEmptyString Then
+                                                        u = New UserMedia(imgUrl) With {
+                                                            .SpecialFolder = a.Title,
+                                                            .Type = UserMedia.Types.Picture,
+                                                            .URL_BASE = img
+                                                        }
+                                                        If Not u.File.File.IsEmptyString Then
+                                                            imgId = $"{albumId}_{u.File.Name}"
+                                                            If u.File.Extension.IsEmptyString Then u.File.Extension = "jpg"
+                                                            u.Post = imgId
+                                                            If Not _TempPostsList.Contains(imgId) Then
+                                                                _TempPostsList.Add(imgId)
+                                                                _TempMediaList.Add(u)
+                                                            Else
+                                                                Exit For
+                                                            End If
+                                                        End If
+                                                    End If
+                                                End If
+                                            Next
+                                            images.Clear()
+                                        End If
+                                    End If
+                                End If
+                            Next
+                        Else
+                            Exit Do
+                        End If
+                    End If
+                Loop While __continue
+            Catch ex As Exception
+                ProcessException(ex, Token, $"images downloading error [{URL}]")
+            End Try
+        End Sub
+#End Region
+#Region "ReparseVideo"
+        Protected Overrides Sub ReparseVideo(ByVal Token As CancellationToken)
+            Try
+                If _TempMediaList.Count > 0 Then
+                    Dim u As UserMedia
+                    Dim dirCmd$ = String.Empty
+                    Dim f As SFile = Settings.YtdlpFile.File
+                    Dim n$
+                    Dim cookieFile As SFile = DirectCast(HOST.Source, SiteSettings).CookiesNetscapeFile
+                    Dim command$
+                    Dim e As EContainer
+                    For i% = _TempMediaList.Count - 1 To 0 Step -1
+                        u = _TempMediaList(i)
+                        If u.Type = UserMedia.Types.VideoPre Then
+                            ThrowAny(Token)
+                            command = $"""{f}"" --verbose --dump-json "
+                            If cookieFile.Exists Then command &= $"--no-cookies-from-browser --cookies ""{cookieFile}"" "
+                            command &= u.URL
+                            e = GetJson(command)
+                            If Not e Is Nothing Then
+                                u.URL = e.Value("url")
+                                u.Post = New UserPost(e.Value("id"), ADateTime.ParseUnix32(e.Value("epoch")))
+                                If u.Post.Date.HasValue Then
+                                    Select Case CheckDatesLimit(u.Post.Date.Value, Nothing)
+                                        Case DateResult.Skip : _TempPostsList.ListAddValue(u.Post.ID, LNC) : _TempMediaList.RemoveAt(i) : Continue For
+                                        Case DateResult.Exit : Exit Sub
+                                    End Select
+                                End If
+                                n = TitleHtmlConverter(e.Value("title"))
+                                If Not n.IsEmptyString Then n = n.Replace("ThisVid.com", String.Empty).StringTrim.StringTrimEnd("-").StringTrim
+                                If n.IsEmptyString Then n = u.Post.ID
+                                If n.IsEmptyString Then n = "VideoFile"
+                                u.File = $"{n}.mp4"
+                                If u.URL.IsEmptyString OrElse (Not u.Post.ID.IsEmptyString AndAlso _TempPostsList.Contains(u.Post.ID)) Then
+                                    _TempMediaList.RemoveAt(i)
+                                Else
+                                    u.Type = UserMedia.Types.Video
+                                    _TempPostsList.Add(u.Post.ID)
+                                    _TempMediaList(i) = u
+                                End If
+                                e.Dispose()
+                            End If
+                        End If
+                    Next
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "video reparsing error")
+            End Try
+        End Sub
+#End Region
+#Region "GetJson"
+        Private Function GetJson(ByVal Command As String) As EContainer
+            Try
+                Using b As New BatchExecutor(True)
+                    b.Execute(Command, EDP.ReturnValue)
+                    If b.OutputData.Count > 0 Then
+                        Dim e As EContainer
+                        For Each d$ In b.OutputData
+                            If Not d.IsEmptyString AndAlso d.StartsWith("{") Then
+                                e = JsonDocument.Parse(d, EDP.ReturnValue)
+                                If Not e Is Nothing Then Return e
+                            End If
+                        Next
+                    End If
+                End Using
+                Return Nothing
+            Catch ex As Exception
+                HasError = True
+                LogError(ex, $"GetJson({Command})")
+                Return Nothing
+            End Try
+        End Function
+#End Region
+#Region "DownloadContent"
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            Dim s As Boolean? = SeparateVideoFolder
+            If DifferentFolders Then SeparateVideoFolder = False Else SeparateVideoFolder = Nothing
+            DownloadContentDefault(Token)
+            SeparateVideoFolder = s
+        End Sub
+#End Region
+#Region "Standalone downloader"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            _TempMediaList.Add(New UserMedia(Data.URL) With {.Type = UserMedia.Types.VideoPre})
+            ReparseVideo(Token)
+        End Sub
+#End Region
+#Region "DownloadingException"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            If Responser.StatusCode = Net.HttpStatusCode.NotFound Then
+                Return 1
+            Else
+                Return 0
+            End If
+        End Function
+#End Region
+    End Class
+End Namespace

SCrawler/API/ThisVid/UserExchangeOptions.vb

@@ -0,0 +1,32 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin.Attributes
+Namespace API.ThisVid
+    Friend Class UserExchangeOptions
+        <PSetting(Caption:="Download public videos")>
+        Friend Property DownloadPublic As Boolean = True
+        <PSetting(Caption:="Download private videos")>
+        Friend Property DownloadPrivate As Boolean = True
+        <PSetting(NameOf(SiteSettings.DifferentFolders), NameOf(MySettings), Caption:="Different video folders")>
+        Friend Property DifferentFolders As Boolean = True
+        Private ReadOnly Property MySettings As SiteSettings
+        Friend Sub New(ByVal s As SiteSettings)
+            DownloadPublic = s.DownloadPublic.Value
+            DownloadPrivate = s.DownloadPrivate.Value
+            DifferentFolders = s.DifferentFolders.Value
+            MySettings = s
+        End Sub
+        Friend Sub New(ByVal u As UserData)
+            DownloadPublic = u.DownloadPublic
+            DownloadPrivate = u.DownloadPrivate
+            DifferentFolders = u.DifferentFolders
+            MySettings = u.HOST.Source
+        End Sub
+    End Class
+End Namespace

SCrawler/API/TikTok/Declarations.vb

@@ -10,11 +10,7 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.TikTok
     Friend Module Declarations
         Friend ReadOnly RegexEnvir As New RegexParseEnvir
-        Friend ReadOnly CheckDateProvider As New CustomProvider(Function(v, d, p, n, e)
-                                                                    With DirectCast(v, Date?)
-                                                                        If .HasValue Then Return .Value Else Return Nothing
-                                                                    End With
-                                                                End Function)
+        Friend ReadOnly CheckDateProvider As New CustomProvider(Function(v) IIf(CType(v, Date?).HasValue, CObj(CType(v, Date?).Value), Nothing))
         Friend Class RegexParseEnvir
             Private ReadOnly UrlIdRegex As RParams = RParams.DMS("http[s]?://[w\.]{0,4}tiktok.com/[^/]+?/video/(\d+)", 1, EDP.ReturnValue)
             Private ReadOnly RegexItemsArrPre As RParams = RParams.DMS("ItemList"":\{""user-post"":\{""list"":\[([^\[]+)\]", 1)
@@ -33,7 +29,7 @@ Namespace API.TikTok
                     End If
                     Return Nothing
                 Catch ex As Exception
-                    Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.TikTok.RegexParseEnvir.GetIDList]")
+                    Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.TikTok.RegexParseEnvir.GetIDList]")
                 End Try
             End Function
             Friend Function GetVideoData(ByVal r As String, ByVal ID As String, ByRef URL As String, ByRef [Date] As Date?) As Boolean
@@ -46,12 +42,12 @@ Namespace API.TikTok
                         Dim u$ = RegexReplace(r, VideoPattern)
                         If Not u.IsEmptyString Then URL = SymbolsConverter.Unicode.Decode(u, EDP.ReturnValue)
                         Dim d$ = RegexReplace(r, DatePattern)
-                        If Not d.IsEmptyString Then [Date] = ADateTime.ParseUnicode(d)
+                        If Not d.IsEmptyString Then [Date] = ADateTime.ParseUnix32(d)
                         Return Not URL.IsEmptyString
                     End If
                     Return False
                 Catch ex As Exception
-                    Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[API.TikTok.RegexParseEnvir.GetVideoData]", False)
+                    Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.TikTok.RegexParseEnvir.GetVideoData]", False)
                 End Try
             End Function
             Friend Function ExtractPostID(ByVal URL As String) As String

SCrawler/API/TikTok/SiteSettings.vb

@@ -32,11 +32,12 @@ Namespace API.TikTok
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
             Return New UserData
         End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            Return UserData.GetVideoInfo(URL, Responser)
-        End Function
         Friend Overrides Function BaseAuthExists() As Boolean
             Return Responser.CookiesExists
         End Function
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            'TODO: TikTok disabled
+            Return False
+        End Function
     End Class
 End Namespace

SCrawler/API/TikTok/UserData.vb

@@ -26,7 +26,7 @@ Namespace API.TikTok
                 Dim PostURL$ = String.Empty
                 Dim r$
                 URL = $"https://www.tiktok.com/@{Name}"
-                r = Responser.GetResponse(URL,, EDP.ThrowException)
+                r = Responser.GetResponse(URL)
                 PostIDs = RegexEnvir.GetIDList(r)
                 If PostIDs.ListExists Then
                     For Each __id$ In PostIDs
@@ -52,28 +52,7 @@ Namespace API.TikTok
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
-            Try
-                If Not URL.IsEmptyString Then
-                    Dim PostId$ = String.Empty
-                    Dim PostDate As Date? = Nothing
-                    Dim PostURL$ = String.Empty
-                    Dim r$
-                    PostId = RegexEnvir.ExtractPostID(URL)
-                    If Not PostId.IsEmptyString Then
-                        Using resp As Responser = Responser.Copy() : r = resp.GetResponse(URL,, EDP.ThrowException) : End Using
-                        If Not r.IsEmptyString Then
-                            If RegexEnvir.GetVideoData(r, PostId, PostURL, PostDate) Then Return {MediaFromData(PostURL, PostId, PostDate)}
-                        End If
-                    End If
-                End If
-                Return Nothing
-            Catch ex As Exception
-                If Not e.Exists Then e = New ErrorsDescriber(EDP.ShowMainMsg + EDP.SendInLog)
-                Return ErrorsDescriber.Execute(e, ex, $"TikTok standalone downloader: fetch media error ({URL})")
-            End Try
-        End Function
-        Private Shared Function MediaFromData(ByVal _URL As String, ByVal PostID As String, ByVal PostDate As Date?) As UserMedia
+        Private Function MediaFromData(ByVal _URL As String, ByVal PostID As String, ByVal PostDate As Date?) As UserMedia
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
             Dim m As New UserMedia(_URL, UserMedia.Types.Video) With {.Post = New UserPost With {.ID = PostID}}
             If Not m.URL.IsEmptyString Then m.File = $"{PostID}.mp4"

SCrawler/API/Twitter/Declarations.vb

@@ -16,7 +16,6 @@ Namespace API.Twitter
         Friend ReadOnly DateProvider As ADateTime = GetDateProvider()
         Friend ReadOnly VideoNode As NodeParams() = {New NodeParams("video_info", True, True, True, True, 10)}
         Friend ReadOnly VideoSizeRegEx As RParams = RParams.DMS("\d+x(\d+)", 1, EDP.ReturnValue)
-        Friend ReadOnly UserIdRegEx As RParams = RParams.DMS("user_id.:.(\d+)", 1, EDP.ReturnValue)
         Private Function GetDateProvider() As ADateTime
             Dim n As DateTimeFormatInfo = CultureInfo.GetCultureInfo("en-us").DateTimeFormat.Clone
             n.FullDateTimePattern = "ddd MMM dd HH:mm:ss +ffff yyyy"

SCrawler/API/Twitter/EditorExchangeOptions.vb

@@ -6,20 +6,37 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin.Attributes
 Namespace API.Twitter
     Friend Class EditorExchangeOptions
+        Private Const DefaultOffset As Integer = 100
+        Friend Property SiteKey As String = TwitterSiteKey
+        <PSetting(NameOf(SiteSettings.GifsDownload), NameOf(MySettings), LeftOffset:=DefaultOffset)>
         Friend Property GifsDownload As Boolean
+        <PSetting(NameOf(SiteSettings.GifsSpecialFolder), NameOf(MySettings), LeftOffset:=DefaultOffset)>
         Friend Property GifsSpecialFolder As String
+        <PSetting(NameOf(SiteSettings.GifsPrefix), NameOf(MySettings), LeftOffset:=DefaultOffset)>
         Friend Property GifsPrefix As String
+        <PSetting(NameOf(SiteSettings.UseMD5Comparison), NameOf(MySettings), LeftOffset:=DefaultOffset)>
         Friend Property UseMD5Comparison As Boolean = False
+        <PSetting(Caption:="Remove existing duplicates",
+                  ToolTip:="Existing files will be checked for duplicates and duplicates removed." & vbCr &
+                           "Works only on the first activation 'Use MD5 comparison'.", LeftOffset:=DefaultOffset)>
         Friend Property RemoveExistingDuplicates As Boolean = False
-        Friend Sub New()
-        End Sub
+        Private ReadOnly Property MySettings As Object
         Friend Sub New(ByVal s As SiteSettings)
             GifsDownload = s.GifsDownload.Value
             GifsSpecialFolder = s.GifsSpecialFolder.Value
             GifsPrefix = s.GifsPrefix.Value
             UseMD5Comparison = s.UseMD5Comparison.Value
+            MySettings = s
+        End Sub
+        Friend Sub New(ByVal s As Mastodon.SiteSettings)
+            GifsDownload = s.GifsDownload.Value
+            GifsSpecialFolder = s.GifsSpecialFolder.Value
+            GifsPrefix = s.GifsPrefix.Value
+            UseMD5Comparison = s.UseMD5Comparison.Value
+            MySettings = s
         End Sub
         Friend Sub New(ByVal u As UserData)
             GifsDownload = u.GifsDownload
@@ -27,6 +44,7 @@ Namespace API.Twitter
             GifsPrefix = u.GifsPrefix
             UseMD5Comparison = u.UseMD5Comparison
             RemoveExistingDuplicates = u.RemoveExistingDuplicates
+            MySettings = u.HOST.Source
         End Sub
     End Class
 End Namespace

SCrawler/API/Twitter/OptionsForm.Designer.vb

@@ -1,185 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Namespace API.Twitter
-    <Global.Microsoft.VisualBasic.CompilerServices.DesignerGenerated()>
-    Partial Friend Class OptionsForm : Inherits System.Windows.Forms.Form
-        <System.Diagnostics.DebuggerNonUserCode()>
-        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            Try
-                If disposing AndAlso components IsNot Nothing Then
-                    components.Dispose()
-                End If
-            Finally
-                MyBase.Dispose(disposing)
-            End Try
-        End Sub
-        Private components As System.ComponentModel.IContainer
-        <System.Diagnostics.DebuggerStepThrough()>
-        Private Sub InitializeComponent()
-            Me.components = New System.ComponentModel.Container()
-            Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
-            Dim TP_MAIN As System.Windows.Forms.TableLayoutPanel
-            Dim ActionButton3 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
-            Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(OptionsForm))
-            Dim ActionButton4 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
-            Dim TT_MAIN As System.Windows.Forms.ToolTip
-            Me.CH_DOWN_GIFS = New System.Windows.Forms.CheckBox()
-            Me.TXT_GIF_FOLDER = New PersonalUtilities.Forms.Controls.TextBoxExtended()
-            Me.TXT_GIF_PREFIX = New PersonalUtilities.Forms.Controls.TextBoxExtended()
-            Me.CH_USE_MD5 = New System.Windows.Forms.CheckBox()
-            Me.CH_REMOVE_EXISTING_DUP = New System.Windows.Forms.CheckBox()
-            CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
-            TT_MAIN = New System.Windows.Forms.ToolTip(Me.components)
-            CONTAINER_MAIN.ContentPanel.SuspendLayout()
-            CONTAINER_MAIN.SuspendLayout()
-            TP_MAIN.SuspendLayout()
-            CType(Me.TXT_GIF_FOLDER, System.ComponentModel.ISupportInitialize).BeginInit()
-            CType(Me.TXT_GIF_PREFIX, System.ComponentModel.ISupportInitialize).BeginInit()
-            Me.SuspendLayout()
-            '
-            'CONTAINER_MAIN
-            '
-            '
-            'CONTAINER_MAIN.ContentPanel
-            '
-            CONTAINER_MAIN.ContentPanel.Controls.Add(TP_MAIN)
-            CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(304, 161)
-            CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            CONTAINER_MAIN.LeftToolStripPanelVisible = False
-            CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
-            CONTAINER_MAIN.Name = "CONTAINER_MAIN"
-            CONTAINER_MAIN.RightToolStripPanelVisible = False
-            CONTAINER_MAIN.Size = New System.Drawing.Size(304, 161)
-            CONTAINER_MAIN.TabIndex = 0
-            CONTAINER_MAIN.TopToolStripPanelVisible = False
-            '
-            'TP_MAIN
-            '
-            TP_MAIN.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single]
-            TP_MAIN.ColumnCount = 1
-            TP_MAIN.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Controls.Add(Me.CH_DOWN_GIFS, 0, 0)
-            TP_MAIN.Controls.Add(Me.TXT_GIF_FOLDER, 0, 1)
-            TP_MAIN.Controls.Add(Me.TXT_GIF_PREFIX, 0, 2)
-            TP_MAIN.Controls.Add(Me.CH_USE_MD5, 0, 3)
-            TP_MAIN.Controls.Add(Me.CH_REMOVE_EXISTING_DUP, 0, 4)
-            TP_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
-            TP_MAIN.Location = New System.Drawing.Point(0, 0)
-            TP_MAIN.Name = "TP_MAIN"
-            TP_MAIN.RowCount = 6
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
-            TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_MAIN.Size = New System.Drawing.Size(304, 161)
-            TP_MAIN.TabIndex = 0
-            '
-            'CH_DOWN_GIFS
-            '
-            Me.CH_DOWN_GIFS.AutoSize = True
-            Me.CH_DOWN_GIFS.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_DOWN_GIFS.Location = New System.Drawing.Point(4, 4)
-            Me.CH_DOWN_GIFS.Name = "CH_DOWN_GIFS"
-            Me.CH_DOWN_GIFS.Padding = New System.Windows.Forms.Padding(100, 0, 0, 0)
-            Me.CH_DOWN_GIFS.Size = New System.Drawing.Size(296, 19)
-            Me.CH_DOWN_GIFS.TabIndex = 0
-            Me.CH_DOWN_GIFS.Text = "Download GIFs"
-            Me.CH_DOWN_GIFS.UseVisualStyleBackColor = True
-            '
-            'TXT_GIF_FOLDER
-            '
-            ActionButton3.BackgroundImage = CType(resources.GetObject("ActionButton3.BackgroundImage"), System.Drawing.Image)
-            ActionButton3.Name = "Clear"
-            ActionButton3.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
-            Me.TXT_GIF_FOLDER.Buttons.Add(ActionButton3)
-            Me.TXT_GIF_FOLDER.CaptionText = "GIFs special folder"
-            Me.TXT_GIF_FOLDER.CaptionToolTipText = "Put the GIFs in a special folder"
-            Me.TXT_GIF_FOLDER.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.TXT_GIF_FOLDER.Location = New System.Drawing.Point(4, 30)
-            Me.TXT_GIF_FOLDER.Name = "TXT_GIF_FOLDER"
-            Me.TXT_GIF_FOLDER.Size = New System.Drawing.Size(296, 22)
-            Me.TXT_GIF_FOLDER.TabIndex = 1
-            '
-            'TXT_GIF_PREFIX
-            '
-            ActionButton4.BackgroundImage = CType(resources.GetObject("ActionButton4.BackgroundImage"), System.Drawing.Image)
-            ActionButton4.Name = "Clear"
-            ActionButton4.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
-            Me.TXT_GIF_PREFIX.Buttons.Add(ActionButton4)
-            Me.TXT_GIF_PREFIX.CaptionText = "GIF prefix"
-            Me.TXT_GIF_PREFIX.CaptionToolTipText = "This prefix will be added to the beginning of the filename"
-            Me.TXT_GIF_PREFIX.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.TXT_GIF_PREFIX.Location = New System.Drawing.Point(4, 59)
-            Me.TXT_GIF_PREFIX.Name = "TXT_GIF_PREFIX"
-            Me.TXT_GIF_PREFIX.Size = New System.Drawing.Size(296, 22)
-            Me.TXT_GIF_PREFIX.TabIndex = 2
-            '
-            'CH_USE_MD5
-            '
-            Me.CH_USE_MD5.AutoSize = True
-            Me.CH_USE_MD5.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_USE_MD5.Location = New System.Drawing.Point(4, 88)
-            Me.CH_USE_MD5.Name = "CH_USE_MD5"
-            Me.CH_USE_MD5.Padding = New System.Windows.Forms.Padding(100, 0, 0, 0)
-            Me.CH_USE_MD5.Size = New System.Drawing.Size(296, 19)
-            Me.CH_USE_MD5.TabIndex = 3
-            Me.CH_USE_MD5.Text = "Use MD5 comparison"
-            TT_MAIN.SetToolTip(Me.CH_USE_MD5, "Each image will be checked for existence using MD5")
-            Me.CH_USE_MD5.UseVisualStyleBackColor = True
-            '
-            'CH_REMOVE_EXISTING_DUP
-            '
-            Me.CH_REMOVE_EXISTING_DUP.AutoSize = True
-            Me.CH_REMOVE_EXISTING_DUP.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_REMOVE_EXISTING_DUP.Location = New System.Drawing.Point(4, 114)
-            Me.CH_REMOVE_EXISTING_DUP.Name = "CH_REMOVE_EXISTING_DUP"
-            Me.CH_REMOVE_EXISTING_DUP.Padding = New System.Windows.Forms.Padding(100, 0, 0, 0)
-            Me.CH_REMOVE_EXISTING_DUP.Size = New System.Drawing.Size(296, 19)
-            Me.CH_REMOVE_EXISTING_DUP.TabIndex = 4
-            Me.CH_REMOVE_EXISTING_DUP.Text = "Remove existing duplicates"
-            TT_MAIN.SetToolTip(Me.CH_REMOVE_EXISTING_DUP, "Existing files will be checked for duplicates and duplicates removed." & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "Works only" &
-        " on the first activation 'Use MD5 comparison'.")
-            Me.CH_REMOVE_EXISTING_DUP.UseVisualStyleBackColor = True
-            '
-            'OptionsForm
-            '
-            Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
-            Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
-            Me.ClientSize = New System.Drawing.Size(304, 161)
-            Me.Controls.Add(CONTAINER_MAIN)
-            Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
-            Me.Icon = Global.SCrawler.My.Resources.SiteResources.TwitterIcon_32
-            Me.MaximizeBox = False
-            Me.MaximumSize = New System.Drawing.Size(320, 200)
-            Me.MinimizeBox = False
-            Me.MinimumSize = New System.Drawing.Size(320, 200)
-            Me.Name = "OptionsForm"
-            Me.ShowInTaskbar = False
-            Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
-            Me.Text = "Options"
-            CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
-            CONTAINER_MAIN.ResumeLayout(False)
-            CONTAINER_MAIN.PerformLayout()
-            TP_MAIN.ResumeLayout(False)
-            TP_MAIN.PerformLayout()
-            CType(Me.TXT_GIF_FOLDER, System.ComponentModel.ISupportInitialize).EndInit()
-            CType(Me.TXT_GIF_PREFIX, System.ComponentModel.ISupportInitialize).EndInit()
-            Me.ResumeLayout(False)
-
-        End Sub
-        Private WithEvents CH_DOWN_GIFS As CheckBox
-        Private WithEvents TXT_GIF_FOLDER As PersonalUtilities.Forms.Controls.TextBoxExtended
-        Private WithEvents TXT_GIF_PREFIX As PersonalUtilities.Forms.Controls.TextBoxExtended
-        Private WithEvents CH_USE_MD5 As CheckBox
-        Private WithEvents CH_REMOVE_EXISTING_DUP As CheckBox
-    End Class
-End Namespace

SCrawler/API/Twitter/OptionsForm.vb

@@ -1,81 +0,0 @@
-' Copyright (C) 2023  Andy https://github.com/AAndyProgram
-' This program is free software: you can redistribute it and/or modify
-' it under the terms of the GNU General Public License as published by
-' the Free Software Foundation, either version 3 of the License, or
-' (at your option) any later version.
-'
-' This program is distributed in the hope that it will be useful,
-' but WITHOUT ANY WARRANTY
-Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Forms
-Imports PersonalUtilities.Forms.Controls
-Namespace API.Twitter
-    Friend Class OptionsForm
-        Private WithEvents MyDefs As DefaultFormOptions
-        Private ReadOnly Property MyExchangeOptions As EditorExchangeOptions
-        Private ReadOnly MyGifTextProvider As SiteSettings.GifStringProvider
-        Friend Sub New(ByRef ExchangeOptions As EditorExchangeOptions)
-            InitializeComponent()
-            MyExchangeOptions = ExchangeOptions
-            MyGifTextProvider = New SiteSettings.GifStringProvider
-            MyDefs = New DefaultFormOptions(Me, Settings.Design)
-        End Sub
-        Private Sub OptionsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
-            With MyDefs
-                .MyViewInitialize(True)
-                .AddOkCancelToolbar()
-                With MyExchangeOptions
-                    CH_DOWN_GIFS.Checked = .GifsDownload
-                    TXT_GIF_FOLDER.Text = .GifsSpecialFolder
-                    TXT_GIF_FOLDER.Tag = NameOf(SiteSettings.GifsSpecialFolder)
-                    TXT_GIF_PREFIX.Text = .GifsPrefix
-                    TXT_GIF_PREFIX.Tag = NameOf(SiteSettings.GifsPrefix)
-                    CH_USE_MD5.Checked = .UseMD5Comparison
-                    CH_REMOVE_EXISTING_DUP.Checked = .RemoveExistingDuplicates
-
-                    Try
-                        Dim p As PropertyOption
-                        With Settings(TwitterSiteKey)
-                            p = .PropList.Find(Function(pp) pp.Name = TXT_GIF_FOLDER.Tag).Options
-                            If Not p Is Nothing Then
-                                TXT_GIF_FOLDER.CaptionText = p.ControlText
-                                TXT_GIF_FOLDER.CaptionToolTipText = p.ControlToolTip
-                                TXT_GIF_FOLDER.CaptionToolTipEnabled = Not TXT_GIF_FOLDER.CaptionToolTipText.IsEmptyString
-                            End If
-
-                            p = .PropList.Find(Function(pp) pp.Name = TXT_GIF_PREFIX.Tag).Options
-                            If Not p Is Nothing Then
-                                TXT_GIF_PREFIX.CaptionText = p.ControlText
-                                TXT_GIF_PREFIX.CaptionToolTipText = p.ControlToolTip
-                                TXT_GIF_PREFIX.CaptionToolTipEnabled = Not TXT_GIF_PREFIX.CaptionToolTipText.IsEmptyString
-                            End If
-                        End With
-                    Catch
-                    End Try
-                End With
-                .EndLoaderOperations()
-            End With
-        End Sub
-        Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
-            With MyExchangeOptions
-                .GifsDownload = CH_DOWN_GIFS.Checked
-                .GifsSpecialFolder = TXT_GIF_FOLDER.Text
-                .GifsPrefix = TXT_GIF_PREFIX.Text
-                .UseMD5Comparison = CH_USE_MD5.Checked
-                .RemoveExistingDuplicates = CH_REMOVE_EXISTING_DUP.Checked
-            End With
-            MyDefs.CloseForm()
-        End Sub
-        Private Sub TXT_ActionOnTextChanged(ByVal Sender As TextBoxExtended, ByVal e As EventArgs) Handles TXT_GIF_FOLDER.ActionOnTextChanged,
-                                                                                                           TXT_GIF_PREFIX.ActionOnTextChanged
-            If Not MyDefs.Initializing Then
-                With Sender
-                    MyGifTextProvider.PropertyName = .Tag
-                    Dim s% = .SelectionStart
-                    Dim t$ = AConvert(Of String)(.Text, String.Empty, MyGifTextProvider)
-                    If Not .Text = t Then .Text = t : .Select(s, 0)
-                End With
-            End If
-        End Sub
-    End Class
-End Namespace

SCrawler/API/Twitter/SiteSettings.vb

@@ -11,12 +11,25 @@ Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
-Imports PersonalUtilities.Tools.Web.Cookies
 Namespace API.Twitter
     <Manifest(TwitterSiteKey), SavedPosts, SpecialForm(False)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Token names"
         Friend Const Header_Authorization As String = "authorization"
         Friend Const Header_Token As String = "x-csrf-token"
+#End Region
+#Region "Properties constants"
+        Friend Const GifsSpecialFolder_Text As String = "GIFs special folder"
+        Friend Const GifsSpecialFolder_ToolTip As String = "Put the GIFs in a special folder" & vbCr &
+                                                           "This is a folder name, not an absolute path." & vbCr &
+                                                           "This folder(s) will be created relative to the user's root folder." & vbCr &
+                                                           "Examples:" & vbCr & "SomeFolderName" & vbCr & "SomeFolderName\SomeFolderName2"
+        Friend Const GifsPrefix_Text As String = "GIF prefix"
+        Friend Const GifsPrefix_ToolTip As String = "This prefix will be added to the beginning of the filename"
+        Friend Const GifsDownload_Text As String = "Download GIFs"
+        Friend Const UseMD5Comparison_Text As String = "Use MD5 comparison"
+        Friend Const UseMD5Comparison_ToolTip As String = "Each image will be checked for existence using MD5"
+#End Region
 #Region "Declarations"
         Friend Overrides ReadOnly Property Icon As Icon
             Get
@@ -34,19 +47,13 @@ Namespace API.Twitter
         Private ReadOnly Property Auth As PropertyValue
         <PropertyOption(AllowNull:=False, IsAuth:=True, ControlText:="Token", ControlToolTip:="Set token from [x-csrf-token] response header")>
         Private ReadOnly Property Token As PropertyValue
-        <PropertyOption(IsAuth:=True, ControlText:="Saved posts user", ControlToolTip:="Personal profile username"), PXML>
-        Friend ReadOnly Property SavedPostsUserName As PropertyValue
 #End Region
 #Region "Other properties"
-        <PropertyOption(IsAuth:=False, ControlText:="Download GIFs"), PXML>
+        <PropertyOption(IsAuth:=False, ControlText:=GifsDownload_Text), PXML>
         Friend ReadOnly Property GifsDownload As PropertyValue
-        <PropertyOption(IsAuth:=False, ControlText:="GIFs special folder",
-                        ControlToolTip:="Put the GIFs in a special folder" & vbCr &
-                        "This is a folder name, not an absolute path." & vbCr &
-                        "This folder(s) will be created relative to the user's root folder." & vbCr &
-                        "Examples:" & vbCr & "SomeFolderName" & vbCr & "SomeFolderName\SomeFolderName2"), PXML>
+        <PropertyOption(IsAuth:=False, ControlText:=GifsSpecialFolder_Text, ControlToolTip:=GifsSpecialFolder_ToolTip), PXML>
         Friend ReadOnly Property GifsSpecialFolder As PropertyValue
-        <PropertyOption(IsAuth:=False, ControlText:="GIF prefix", ControlToolTip:="This prefix will be added to the beginning of the filename"), PXML>
+        <PropertyOption(IsAuth:=False, ControlText:=GifsPrefix_Text, ControlToolTip:=GifsPrefix_ToolTip), PXML>
         Friend ReadOnly Property GifsPrefix As PropertyValue
         <Provider(NameOf(GifsSpecialFolder), Interaction:=True), Provider(NameOf(GifsPrefix), Interaction:=True)>
         Private ReadOnly Property GifStringChecker As IFormatProvider
@@ -60,69 +67,18 @@ Namespace API.Twitter
                         v = v.StringRemoveWinForbiddenSymbols
                     Else
                         v = v.StringReplaceSymbols(GetWinForbiddenSymbols.ToList.ListWithRemove("\").ToArray, String.Empty, EDP.ReturnValue)
-                        v = v.StringTrim("\")
                     End If
                 End If
                 Return v
             End Function
             Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("[GetFormat] is not available in the context of [TimersChecker]")
+                Throw New NotImplementedException("[GetFormat] is not available in the context of [GifStringProvider]")
             End Function
         End Class
-        <PropertyOption(IsAuth:=False, ControlText:="Use MD5 comparison", ControlToolTip:="Each image will be checked for existence using MD5"), PXML>
+        <PropertyOption(IsAuth:=False, ControlText:=UseMD5Comparison_Text, ControlToolTip:=UseMD5Comparison_ToolTip), PXML>
         Friend ReadOnly Property UseMD5Comparison As PropertyValue
 #End Region
         Friend Overrides ReadOnly Property Responser As Responser
-#End Region
-        Friend Sub New()
-            MyBase.New(TwitterSite)
-            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml")
-
-            Dim a$ = String.Empty
-            Dim t$ = String.Empty
-
-            With Responser
-                If .File.Exists Then
-                    Dim b As Boolean = .CookiesDomain.IsEmptyString
-                    If EncryptCookies.CookiesEncrypted Then .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
-                    .LoadSettings()
-                    a = .Headers.Value(Header_Authorization)
-                    t = .Headers.Value(Header_Token)
-                    .CookiesDomain = "twitter.com"
-                    If b Then .SaveSettings()
-                Else
-                    .ContentType = "application/json"
-                    .Accept = "*/*"
-                    .CookiesDomain = "twitter.com"
-                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
-                    .Decoders.Add(SymbolsConverter.Converters.Unicode)
-                    .Headers.Add("sec-ch-ua", " Not;A Brand"";v=""99"", ""Google Chrome"";v=""91"", ""Chromium"";v=""91""")
-                    .Headers.Add("sec-ch-ua-mobile", "?0")
-                    .Headers.Add("sec-fetch-dest", "empty")
-                    .Headers.Add("sec-fetch-mode", "cors")
-                    .Headers.Add("sec-fetch-site", "same-origin")
-                    .Headers.Add(Header_Token, String.Empty)
-                    .Headers.Add("x-twitter-active-user", "yes")
-                    .Headers.Add("x-twitter-auth-type", "OAuth2Session")
-                    .Headers.Add(Header_Authorization, String.Empty)
-                    .SaveSettings()
-                End If
-            End With
-
-            Auth = New PropertyValue(a, GetType(String), Sub(v) ChangeResponserFields(NameOf(Auth), v))
-            Token = New PropertyValue(t, GetType(String), Sub(v) ChangeResponserFields(NameOf(Token), v))
-            SavedPostsUserName = New PropertyValue(String.Empty, GetType(String))
-
-            GifsDownload = New PropertyValue(True)
-            GifsSpecialFolder = New PropertyValue(String.Empty, GetType(String))
-            GifsPrefix = New PropertyValue("GIF_")
-            GifStringChecker = New GifStringProvider
-            UseMD5Comparison = New PropertyValue(False)
-
-            UserRegex = RParams.DMS("[htps:/]{7,8}.*?twitter.com/([^/]+)", 1)
-            UrlPatternUser = "https://twitter.com/{0}"
-            ImageVideoContains = "twitter"
-        End Sub
         Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object)
             If Not PropName.IsEmptyString Then
                 Dim f$ = String.Empty
@@ -137,15 +93,59 @@ Namespace API.Twitter
                 End If
             End If
         End Sub
+#End Region
+        Friend Sub New()
+            MyBase.New(TwitterSite)
+            Responser = New Responser($"{SettingsFolderName}\Responser_{Site}.xml") With {.DeclaredError = EDP.ThrowException}
+
+            Dim a$ = String.Empty
+            Dim t$ = String.Empty
+
+            With Responser
+                If .File.Exists Then
+                    .CookiesDomain = "twitter.com"
+                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .LoadSettings()
+                    a = .Headers.Value(Header_Authorization)
+                    t = .Headers.Value(Header_Token)
+                Else
+                    .ContentType = "application/json"
+                    .Accept = "*/*"
+                    .CookiesDomain = "twitter.com"
+                    .CookiesEncryptKey = SettingsCLS.CookieEncryptKey
+                    .Decoders.Add(SymbolsConverter.Converters.Unicode)
+                    .Headers.Add("sec-ch-ua", """Chromium"";v=""112"", ""Google Chrome"";v=""112"", ""Not:A-Brand"";v=""99""")
+                    .Headers.Add("sec-ch-ua-mobile", "?0")
+                    .Headers.Add("sec-fetch-dest", "empty")
+                    .Headers.Add("sec-fetch-mode", "cors")
+                    .Headers.Add("sec-fetch-site", "same-origin")
+                    .Headers.Add(Header_Token, String.Empty)
+                    .Headers.Add("x-twitter-active-user", "yes")
+                    .Headers.Add("x-twitter-auth-type", "OAuth2Session")
+                    .Headers.Add(Header_Authorization, String.Empty)
+                    .SaveSettings()
+                End If
+                .Cookies.ChangedAllowInternalDrop = False
+                .Cookies.Changed = False
+            End With
+
+            Auth = New PropertyValue(a, GetType(String), Sub(v) ChangeResponserFields(NameOf(Auth), v))
+            Token = New PropertyValue(t, GetType(String), Sub(v) ChangeResponserFields(NameOf(Token), v))
+
+            GifsDownload = New PropertyValue(True)
+            GifsSpecialFolder = New PropertyValue(String.Empty, GetType(String))
+            GifsPrefix = New PropertyValue("GIF_")
+            GifStringChecker = New GifStringProvider
+            UseMD5Comparison = New PropertyValue(False)
+
+            UserRegex = RParams.DMS("[htps:/]{7,8}.*?twitter.com/([^/]+)", 1)
+            UrlPatternUser = "https://twitter.com/{0}"
+            ImageVideoContains = "twitter"
+            CheckNetscapeCookiesOnEndInit = True
+            UseNetscapeCookies = True
+        End Sub
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
-            If What = ISiteSettings.Download.SavedPosts Then
-                Return New UserData With {.IsSavedPosts = True, .User = New UserInfo With {.Name = CStr(AConvert(Of String)(SavedPostsUserName.Value, String.Empty))}}
-            Else
-                Return New UserData
-            End If
-        End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            Return UserData.GetVideoInfo(URL, Responser)
+            Return New UserData
         End Function
         Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
             Return $"https://twitter.com/{User.Name}/status/{Media.Post.ID}"
@@ -153,11 +153,31 @@ Namespace API.Twitter
         Friend Overrides Function BaseAuthExists() As Boolean
             Return Responser.CookiesExists And ACheck(Token.Value) And ACheck(Auth.Value)
         End Function
-        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
-            If Options Is Nothing OrElse Not TypeOf Options Is EditorExchangeOptions Then Options = New EditorExchangeOptions(Me)
-            If OpenForm Then
-                Using f As New OptionsForm(Options) : f.ShowDialog() : End Using
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            If MyBase.Available(What, Silent) Then
+                If What = ISiteSettings.Download.SavedPosts Then
+                    Return Settings.GalleryDLFile.Exists
+                Else
+                    Return True
+                End If
+            Else
+                Return False
             End If
+        End Function
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse (Not TypeOf Options Is EditorExchangeOptions OrElse
+                                          Not DirectCast(Options, EditorExchangeOptions).SiteKey = TwitterSiteKey) Then _
+               Options = New EditorExchangeOptions(Me)
+            If OpenForm Then
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
+            End If
+        End Sub
+        Friend Overrides Sub Update()
+            If _SiteEditorFormOpened Then
+                Dim tf$ = GifsSpecialFolder.Value
+                If Not tf.IsEmptyString Then tf = tf.StringTrim("\") : GifsSpecialFolder.Value = tf
+            End If
+            MyBase.Update()
         End Sub
     End Class
 End Namespace

SCrawler/API/Twitter/UserData.vb

@@ -7,39 +7,32 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
-Imports System.Drawing
 Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Documents.JSON
-Imports PersonalUtilities.Tools.ImageRenderer
 Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Twitter
     Friend Class UserData : Inherits UserDataBase
-        Private Const SinglePostUrl As String = "https://api.twitter.com/1.1/statuses/show.json?id={0}&tweet_mode=extended"
+        Protected SinglePostUrl As String = "https://api.twitter.com/1.1/statuses/show.json?id={0}&tweet_mode=extended"
 #Region "XML names"
         Private Const Name_GifsDownload As String = "GifsDownload"
         Private Const Name_GifsSpecialFolder As String = "GifsSpecialFolder"
         Private Const Name_GifsPrefix As String = "GifsPrefix"
-        Private Const Name_UseMD5Comparison As String = "UseMD5Comparison"
-        Private Const Name_RemoveExistingDuplicates As String = "RemoveExistingDuplicates"
-        Private Const Name_StartMD5Checked As String = "StartMD5Checked"
 #End Region
 #Region "Declarations"
-        Friend Property GifsDownload As Boolean
-        Friend Property GifsSpecialFolder As String
-        Friend Property GifsPrefix As String
+        Friend Property GifsDownload As Boolean = True
+        Friend Property GifsSpecialFolder As String = String.Empty
+        Friend Property GifsPrefix As String = String.Empty
         Private ReadOnly _DataNames As List(Of String)
-        Friend Property UseMD5Comparison As Boolean = False
-        Private StartMD5Checked As Boolean = False
-        Friend Property RemoveExistingDuplicates As Boolean = False
 #End Region
 #Region "Exchange options"
         Friend Overrides Function ExchangeOptionsGet() As Object
-            Return New EditorExchangeOptions(Me)
+            Return New EditorExchangeOptions(Me) With {.SiteKey = HOST.Key}
         End Function
         Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
             If Not Obj Is Nothing AndAlso TypeOf Obj Is EditorExchangeOptions Then
@@ -83,45 +76,35 @@ Namespace API.Twitter
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             If IsSavedPosts Then
                 If _ContentList.Count > 0 Then _DataNames.ListAddList(_ContentList.Select(Function(c) c.Post.ID), LAP.ClearBeforeAdd, LAP.NotContainsOnly)
-                DownloadData(String.Empty, Token)
+                DownloadData_SavedPosts(Token)
             Else
                 If _ContentList.Count > 0 Then _DataNames.ListAddList(_ContentList.Select(Function(c) c.File.File), LAP.ClearBeforeAdd, LAP.NotContainsOnly)
                 DownloadData(String.Empty, Token)
-                If UseMD5Comparison Then ValidateMD5(Token)
             End If
         End Sub
         Private Overloads Sub DownloadData(ByVal POST As String, ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
             Try
-                Dim NextCursor$ = String.Empty
-                Dim __NextCursor As Predicate(Of EContainer) = Function(e) e.Value({"content", "operation", "cursor"}, "cursorType") = "Bottom"
                 Dim PostID$ = String.Empty
                 Dim PostDate$
-                Dim nn As EContainer, s As EContainer
+                Dim nn As EContainer
                 Dim NewPostDetected As Boolean = False
                 Dim ExistsDetected As Boolean = False
 
                 Dim UID As Func(Of EContainer, String) = Function(e) e.XmlIfNothing.Item({"user", "id"}).XmlIfNothingValue
 
-                If IsSavedPosts Then
-                    If Name.IsEmptyString Then Throw New ArgumentNullException With {.HelpLink = 1}
-                    URL = $"https://api.twitter.com/2/timeline/bookmark.json?screen_name={Name}&count=200" &
-                           "&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_availability=true"
-                    If Not POST.IsEmptyString Then URL &= $"&cursor={SymbolsConverter.ASCII.EncodeSymbolsOnly(POST)}"
-                Else
-                    URL = $"https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name={Name}&count=200&exclude_replies=false&include_rts=1&tweet_mode=extended"
-                    If Not POST.IsEmptyString Then URL &= $"&max_id={POST}"
-                End If
+                URL = $"https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name={Name}&count=200&exclude_replies=false&include_rts=1&tweet_mode=extended"
+                If Not POST.IsEmptyString Then URL &= $"&max_id={POST}"
 
                 ThrowAny(Token)
-                Dim r$ = Responser.GetResponse(URL,, EDP.ThrowException)
+                Dim r$ = Responser.GetResponse(URL)
                 If Not r.IsEmptyString Then
                     Using w As EContainer = JsonDocument.Parse(r)
                         If w.ListExists Then
 
-                            If Not IsSavedPosts And POST.IsEmptyString And Not w.ItemF({0, "user"}) Is Nothing Then
+                            If POST.IsEmptyString And Not w.ItemF({0, "user"}) Is Nothing Then
                                 With w.ItemF({0, "user"})
-                                    If .Value("screen_name").StringToLower = Name Then
+                                    If .Value("screen_name").StringToLower = Name.ToLower Then
                                         UserSiteNameUpdate(.Value("name"))
                                         UserDescriptionUpdate(.Value("description"))
                                         Dim __getImage As Action(Of String) = Sub(ByVal img As String)
@@ -145,15 +128,10 @@ Namespace API.Twitter
                             For Each nn In If(IsSavedPosts, w({"globalObjects", "tweets"}).XmlIfNothing, w)
                                 ThrowAny(Token)
                                 If nn.Count > 0 Then
-                                    If IsSavedPosts Then
-                                        PostID = nn.Value
-                                        If PostID.IsEmptyString Then PostID = nn.Value("id_str")
-                                    Else
-                                        PostID = nn.Value("id")
-                                        If ID.IsEmptyString Then
-                                            ID = UID(nn)
-                                            If Not ID.IsEmptyString Then UpdateUserInformation()
-                                        End If
+                                    PostID = nn.Value("id")
+                                    If ID.IsEmptyString Then
+                                        ID = UID(nn)
+                                        If Not ID.IsEmptyString Then UpdateUserInformation()
                                     End If
 
                                     'Date Pattern:
@@ -172,32 +150,58 @@ Namespace API.Twitter
                                         Continue For
                                     End If
 
-                                    If IsSavedPosts OrElse Not ParseUserMediaOnly OrElse
-                                        (
-                                            Not nn.Contains("retweeted_status") OrElse
-                                            (Not ID.IsEmptyString AndAlso UID(nn("retweeted_status")) = ID)
-                                        ) Then ObtainMedia(nn, PostID, PostDate)
+                                    If Not ParseUserMediaOnly OrElse
+                                       (Not nn.Contains("retweeted_status") OrElse (Not ID.IsEmptyString AndAlso UID(nn("retweeted_status")) = ID)) Then _
+                                       ObtainMedia(nn, PostID, PostDate)
                                 End If
                             Next
-
-                            If IsSavedPosts Then
-                                s = w.ItemF({"timeline", "instructions", 0, "addEntries", "entries"}).XmlIfNothing
-                                If s.Count > 0 Then NextCursor = If(s.ItemF({__NextCursor})?.Value({"content", "operation", "cursor"}, "value"), String.Empty)
-                            End If
                         End If
                     End Using
 
-                    If IsSavedPosts Then
-                        If Not NextCursor.IsEmptyString And Not NextCursor = POST Then DownloadData(NextCursor, Token)
-                    Else
-                        If POST.IsEmptyString And ExistsDetected Then Exit Sub
-                        If Not PostID.IsEmptyString And NewPostDetected Then DownloadData(PostID, Token)
+                    If POST.IsEmptyString And ExistsDetected Then Exit Sub
+                    If Not PostID.IsEmptyString And NewPostDetected Then DownloadData(PostID, Token)
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
+        End Sub
+        Private Sub DownloadData_SavedPosts(ByVal Token As CancellationToken)
+            Try
+                Dim urls As List(Of String) = GetBookmarksUrlsFromGalleryDL()
+                If urls.ListExists Then
+                    Dim postIds As New List(Of String)
+                    Dim r$
+                    Dim j As EContainer, jj As EContainer
+                    Dim jErr As New ErrorsDescriber(EDP.ReturnValue)
+                    Dim rPattern As RParams = RParams.DM("(?<=tweet-)(\d+)\Z", 0, EDP.ReturnValue)
+                    For Each url$ In urls
+                        r = Responser.GetResponse(url)
+                        If Not r.IsEmptyString Then
+                            j = JsonDocument.Parse(r, jErr)
+                            If Not j Is Nothing Then
+                                jj = j.ItemF({"data", "bookmark_timeline_v2", "timeline", "instructions", 0, "entries"})
+                                If If(jj?.Count, 0) > 0 Then postIds.ListAddList(jj.Select(Function(jj2) CStr(RegexReplace(jj2.Value("entryId"), rPattern))), LNC)
+                                j.Dispose()
+                            End If
+                        End If
+                    Next
+                    If postIds.Count > 0 Then postIds.RemoveAll(Function(pid) pid.IsEmptyString OrElse (_TempPostsList.Contains(pid) Or _DataNames.Contains(pid)))
+                    If postIds.Count > 0 Then
+                        For Each __id$ In postIds
+                            _TempPostsList.Add(__id)
+                            r = Responser.GetResponse(String.Format(SinglePostUrl, __id),, EDP.ReturnValue)
+                            If Not r.IsEmptyString Then
+                                j = JsonDocument.Parse(r, jErr)
+                                If Not j Is Nothing Then
+                                    If j.Count > 0 Then ObtainMedia(j, __id, j.Value("created_at"))
+                                    j.Dispose()
+                                End If
+                            End If
+                        Next
                     End If
                 End If
-            Catch ane As ArgumentNullException When ane.HelpLink = 1
-                MyMainLOG = "Username not set for saved Twitter posts"
             Catch ex As Exception
-                ProcessException(ex, Token, $"data downloading error{IIf(IsSavedPosts, " (Saved Posts)", String.Empty)} [{URL}]")
+                ProcessException(ex, Token, "data downloading error (Saved Posts)")
             End Try
         End Sub
 #End Region
@@ -252,18 +256,24 @@ Namespace API.Twitter
                     If .ListExists Then
                         For Each n As EContainer In .Self
                             If n.Value("type") = "animated_gif" Then
-                                With n({"video_info", "variants"}).XmlIfNothing.ItemF({gifUrl}).XmlIfNothing
-                                    url = .Value("url")
-                                    ff = UrlFile(url)
-                                    If Not ff.IsEmptyString Then
-                                        If GifsDownload And Not _DataNames.Contains(ff) Then
-                                            m = MediaFromData(url, PostID, PostDate,, State, UTypes.Video)
-                                            f = m.File
-                                            If Not f.IsEmptyString And Not GifsPrefix.IsEmptyString Then f.Name = $"{GifsPrefix}{f.Name}" : m.File = f
-                                            If Not GifsSpecialFolder.IsEmptyString Then m.SpecialFolder = $"{GifsSpecialFolder}*"
-                                            _TempMediaList.ListAddValue(m, LNC)
-                                        End If
-                                        Return True
+                                With n({"video_info", "variants"})
+                                    If .ListExists Then
+                                        With .ItemF({gifUrl})
+                                            If .ListExists Then
+                                                url = .Value("url")
+                                                ff = UrlFile(url)
+                                                If Not ff.IsEmptyString Then
+                                                    If GifsDownload And Not _DataNames.Contains(ff) Then
+                                                        m = MediaFromData(url, PostID, PostDate,, State, UTypes.Video)
+                                                        f = m.File
+                                                        If Not f.IsEmptyString And Not GifsPrefix.IsEmptyString Then f.Name = $"{GifsPrefix}{f.Name}" : m.File = f
+                                                        If Not GifsSpecialFolder.IsEmptyString Then m.SpecialFolder = $"{GifsSpecialFolder}*"
+                                                        _TempMediaList.ListAddValue(m, LNC)
+                                                    End If
+                                                    Return True
+                                                End If
+                                            End If
+                                        End With
                                     End If
                                 End With
                             End If
@@ -276,7 +286,7 @@ Namespace API.Twitter
                 Return False
             End Try
         End Function
-        Private Shared Function GetVideoNodeURL(ByVal w As EContainer) As String
+        Private Function GetVideoNodeURL(ByVal w As EContainer) As String
             Dim v As EContainer = w.GetNode(VideoNode)
             If v.ListExists Then
                 Dim l As New List(Of Sizes)
@@ -298,6 +308,18 @@ Namespace API.Twitter
             Return String.Empty
         End Function
 #End Region
+#Region "Gallery-DL Support"
+        Private Function GetBookmarksUrlsFromGalleryDL() As List(Of String)
+            Dim command$ = $"gallery-dl --verbose --simulate --cookies ""{DirectCast(HOST.Source, SiteSettings).CookiesNetscapeFile}"" https://twitter.com/i/bookmarks"
+            Try
+                Using batch As New GDL.GDLBatch With {.TempPostsList = _TempPostsList} : Return GDL.GetUrlsFromGalleryDl(batch, command) : End Using
+            Catch ex As Exception
+                HasError = True
+                LogError(ex, $"GetJson({command})")
+                Return Nothing
+            End Try
+        End Function
+#End Region
 #Region "ReparseMissing"
         Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken)
             Dim rList As New List(Of Integer)
@@ -337,156 +359,19 @@ Namespace API.Twitter
             End Try
         End Sub
 #End Region
-#Region "MD5 support"
-        Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
-        Private Sub ValidateMD5(ByVal Token As CancellationToken)
-            Try
-                Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
-                If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
-                    Dim i%
-                    Dim data As UserMedia = Nothing
-                    Dim hashList As New Dictionary(Of String, SFile)
-                    Dim f As SFile
-                    Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
-                    Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
-                        Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
-                            Try
-                                Dim ImgFormat As Imaging.ImageFormat = Nothing
-                                Dim hash$ = String.Empty
-                                Dim __isGif As Boolean = False
-                                If __data.Type = UTypes.GIF Then
-                                    ImgFormat = Imaging.ImageFormat.Gif
-                                    __isGif = True
-                                ElseIf Not __data.File.IsEmptyString Then
-                                    ImgFormat = GetImageFormat(__data.File)
-                                End If
-                                If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
-                                If IsUrl Then
-                                    hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL_BASE.IfNullOrEmpty(__data.URL), ErrMD5), ImgFormat, ErrMD5))
-                                Else
-                                    hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
-                                End If
-                                If hash.IsEmptyString And Not __isGif Then
-                                    If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
-                                    If IsUrl Then
-                                        hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL_BASE.IfNullOrEmpty(__data.URL), ErrMD5), ImgFormat, ErrMD5))
-                                    Else
-                                        hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
-                                    End If
-                                End If
-                                Return hash
-                            Catch
-                                Return String.Empty
-                            End Try
-                        End Function
-                    If Not StartMD5Checked Then
-                        StartMD5Checked = True
-                        If _ContentList.Exists(missingMD5) Then
-                            Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
-                            Dim eIndx%
-                            Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
-                            If RemoveExistingDuplicates Then
-                                RemoveExistingDuplicates = False
-                                _ForceSaveUserInfo = True
-                                If existingFiles.Count > 0 Then
-                                    Dim h$
-                                    For i = existingFiles.Count - 1 To 0 Step -1
-                                        h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
-                                        If Not h.IsEmptyString Then
-                                            If hashList.ContainsKey(h) Then
-                                                MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
-                                                existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
-                                                existingFiles.RemoveAt(i)
-                                            Else
-                                                hashList.Add(h, existingFiles(i))
-                                            End If
-                                        End If
-                                    Next
-                                End If
-                            End If
-                            For i = 0 To _ContentList.Count - 1
-                                data = _ContentList(i)
-                                If (data.Type = UTypes.GIF Or data.Type = UTypes.Picture) Then
-                                    If data.MD5.IsEmptyString Then
-                                        ThrowAny(Token)
-                                        eIndx = existingFiles.FindIndex(eFinder)
-                                        If eIndx >= 0 Then
-                                            data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
-                                            If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
-                                        End If
-                                    End If
-                                    existingFiles.RemoveAll(eFinder)
-                                End If
-                            Next
-                            If existingFiles.Count > 0 Then
-                                For i = 0 To existingFiles.Count - 1
-                                    f = existingFiles(i)
-                                    data = New UserMedia(f.File) With {
-                                        .State = UStates.Downloaded,
-                                        .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
-                                        .File = f
-                                    }
-                                    ThrowAny(Token)
-                                    data.MD5 = __getMD5(data, False)
-                                    If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
-                                Next
-                                existingFiles.Clear()
-                            End If
-                        End If
-                    End If
-
-                    If _ContentList.Count > 0 Then
-                        With _ContentList.Select(Function(d) d.MD5)
-                            If .ListExists Then .ToList.ForEach(Sub(md5value) _
-                               If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
-                        End With
-                    End If
-
-                    For i = _TempMediaList.Count - 1 To 0 Step -1
-                        data = _TempMediaList(i)
-                        If missingMD5(data) Then
-                            ThrowAny(Token)
-                            data.MD5 = __getMD5(data, True)
-                            If Not data.MD5.IsEmptyString Then
-                                If hashList.ContainsKey(data.MD5) Then
-                                    _TempMediaList.RemoveAt(i)
-                                Else
-                                    hashList.Add(data.MD5, New SFile)
-                                    _TempMediaList(i) = data
-                                End If
-                            End If
-                        End If
-                    Next
+#Region "DownloadSingleObject"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim PostID$ = RegexReplace(Data.URL, RParams.DM("(?<=/)\d+", 0))
+            If Not PostID.IsEmptyString Then
+                Dim r$ = Responser.GetResponse(String.Format(SinglePostUrl, PostID),, EDP.ReturnValue)
+                If Not r.IsEmptyString Then
+                    Using j As EContainer = JsonDocument.Parse(r)
+                        If j.ListExists Then ObtainMedia(j, j.Value("id"), j.Value("created_at"))
+                    End Using
                 End If
-            Catch ex As Exception
-                ProcessException(ex, Token, "ValidateMD5",, VALIDATE_MD5_ERROR)
-            End Try
+            End If
         End Sub
 #End Region
-#Region "Get video static"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal resp As Responser) As IEnumerable(Of UserMedia)
-            Try
-                If URL.Contains("twitter") Then
-                    Dim PostID$ = RegexReplace(URL, RParams.DM("(?<=/)\d+", 0))
-                    If Not PostID.IsEmptyString Then
-                        Dim r$
-                        Using rc As Responser = resp.Copy() : r = rc.GetResponse(String.Format(SinglePostUrl, PostID),, EDP.ReturnValue) : End Using
-                        If Not r.IsEmptyString Then
-                            Using j As EContainer = JsonDocument.Parse(r)
-                                If j.ListExists Then
-                                    Dim u$ = GetVideoNodeURL(j)
-                                    If Not u.IsEmptyString Then Return {MediaFromData(u, PostID, String.Empty,,, UTypes.Video)}
-                                End If
-                            End Using
-                        End If
-                    End If
-                End If
-                Return Nothing
-            Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.ShowMainMsg + EDP.SendInLog, ex, $"Twitter standalone downloader: fetch media error ({URL})")
-            End Try
-        End Function
-#End Region
 #Region "Picture options"
         Private Function GetPictureOption(ByVal w As EContainer) As String
             Const P4K As String = "4096x4096"
@@ -541,10 +426,10 @@ Namespace API.Twitter
         End Function
 #End Region
 #Region "Create media"
-        Private Shared Function MediaFromData(ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
-                                              Optional ByVal _PictureOption As String = Nothing,
-                                              Optional ByVal State As UStates = UStates.Unknown,
-                                              Optional ByVal Type As UTypes = UTypes.Undefined) As UserMedia
+        Private Function MediaFromData(ByVal _URL As String, ByVal PostID As String, ByVal PostDate As String,
+                                       Optional ByVal _PictureOption As String = Nothing,
+                                       Optional ByVal State As UStates = UStates.Unknown,
+                                       Optional ByVal Type As UTypes = UTypes.Undefined) As UserMedia
             _URL = LinkFormatterSecure(RegexReplace(_URL.Replace("\", String.Empty), LinkPattern))
             Dim m As New UserMedia(_URL) With {.PictureOption = _PictureOption, .Post = New UserPost With {.ID = PostID}, .Type = Type}
             If Not m.URL.IsEmptyString Then m.File = CStr(RegexReplace(m.URL, FilesPattern))

SCrawler/API/UserDataBind.vb

@@ -49,13 +49,10 @@ Namespace API
             _CollectionName = NewName
             If Count > 0 Then Collections.ForEach(Sub(c) c.CollectionName = NewName)
         End Sub
-        Friend Overrides Property Name As String
+        Friend Overrides ReadOnly Property Name As String
             Get
                 Return CollectionName
             End Get
-            Set(ByVal NewCollectionName As String)
-                CollectionName = NewCollectionName
-            End Set
         End Property
         Friend Overrides Property FriendlyName As String
             Get
@@ -367,7 +364,7 @@ Namespace API
 #End Region
 #Region "Open site, folder"
         Friend Overrides Sub OpenSite(Optional ByVal e As ErrorsDescriber = Nothing)
-            If Not e.Exists Then e = New ErrorsDescriber(EDP.SendInLog)
+            If Not e.Exists Then e = New ErrorsDescriber(EDP.SendToLog)
             If Count > 0 Then Collections.ForEach(Sub(c) c.OpenSite(e))
         End Sub
         Private ReadOnly RealUser As Predicate(Of IUserData) = Function(u) u.UserModel = UsageModel.Default And Not u.HOST.Key = PathPlugin.PluginKey
@@ -575,7 +572,7 @@ Namespace API
                             MainFrameObj.ImageHandler(Me, False)
                             Collections.ListClearDispose
                             Dispose(False)
-                            If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.EmptyOnly + Settings.DeleteMode, EDP.SendInLog)
+                            If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.EmptyOnly + Settings.DeleteMode, EDP.SendToLog)
                             Return 2
                         End If
                     Case 1
@@ -592,7 +589,7 @@ Namespace API
                             If Collections.All(Function(c) c.CollectionName.IsEmptyString) Then
                                 Settings.Users.Remove(Me)
                                 Collections.Clear()
-                                If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.Default + Settings.DeleteMode, EDP.SendInLog)
+                                If Not f.IsEmptyString Then f.Delete(SFO.Path, SFODelete.Default + Settings.DeleteMode, EDP.SendToLog)
                                 Downloader.UserRemove(Me)
                                 MainFrameObj.ImageHandler(Me, False)
                                 Dispose(False)

SCrawler/API/XVIDEOS/Declarations.vb

@@ -16,7 +16,7 @@ Namespace API.XVIDEOS
         Friend ReadOnly Regex_VideoID As RParams = RParams.DMS(".*?www.xvideos.com/(video\d+).*", 1)
         Friend ReadOnly Regex_M3U8_Reparse As RParams = RParams.DM("NAME=""(\d+).*?""[\r\n]*?(.+)(?=(|[\r\n]+?))", 0, RegexReturn.List)
         Friend ReadOnly Regex_M3U8_Appender As RParams = RParams.DM("(.+)(?=/.+?\.m3u8.*?)", 0)
-        Friend ReadOnly Regex_SavedVideosPlaylist As RParams = RParams.DM("<div id=""video.+?data-id=""(\d+).+?a href=""([^""]+)"".+?title=""([^""]*)""",
-                                                                          0, RegexReturn.List, EDP.ReturnValue, TitleHtmlConverter)
+        Friend ReadOnly Regex_SavedVideosPlaylist As RParams = RParams.DM("\<div id=""video.+?data-id=""(\d+).+?a href=""([^""]+)"".+?title=""([^""]*)""",
+                                                                          0, RegexReturn.List, EDP.ReturnValue)
     End Module
 End Namespace

SCrawler/API/XVIDEOS/M3U8.vb

@@ -7,11 +7,14 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Net
+Imports System.Threading
+Imports PersonalUtilities.Forms.Toolbars
 Namespace API.XVIDEOS
     Friend NotInheritable Class M3U8
         Private Sub New()
         End Sub
-        Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal f As SFile) As SFile
+        Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal f As SFile,
+                                        ByVal Token As CancellationToken, ByVal Progress As MyProgress) As SFile
             Try
                 If Not URL.IsEmptyString Then
                     Using w As New WebClient
@@ -19,13 +22,13 @@ Namespace API.XVIDEOS
                         If Not r.IsEmptyString Then
                             Dim l As List(Of String) = ListAddList(Nothing, r.StringFormatLines.StringToList(Of String)(vbNewLine).ListWithRemove(Function(v) v.Trim.StartsWith("#")),
                                                                    New ListAddParams With {.Converter = Function(Input) $"{Appender}/{Input.ToString.Trim}"})
-                            If l.ListExists Then Return Base.M3U8Base.Download(l, f)
+                            If l.ListExists Then Return Base.M3U8Base.Download(l, f,, Token, Progress)
                         End If
                     End Using
                 End If
                 Return Nothing
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[M3U8.Download({URL}, {Appender}, {f})]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[M3U8.Download({URL}, {Appender}, {f})]")
                 Throw ex
             End Try
         End Function

SCrawler/API/XVIDEOS/SiteSettings.vb

@@ -7,16 +7,15 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports SCrawler.API.Base
-Imports SCrawler.API.BaseObjects
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.XVIDEOS
     <Manifest(XvideosSiteKey), SavedPosts, SpecialForm(True), TaskGroup(SettingsCLS.TaskStackNamePornSite)>
-    Friend Class SiteSettings : Inherits SiteSettingsBase : Implements IDomainContainer
+    Friend Class SiteSettings : Inherits SiteSettingsBase
 #Region "Declarations"
-        Friend Overrides ReadOnly Property Icon As Icon Implements IDomainContainer.Icon
+        Friend Overrides ReadOnly Property Icon As Icon
             Get
                 Return My.Resources.SiteResources.XvideosIcon_48
             End Get
@@ -26,21 +25,10 @@ Namespace API.XVIDEOS
                 Return My.Resources.SiteResources.XvideosPic_32
             End Get
         End Property
-#Region "Domains"
-        Private ReadOnly Property IDomainContainer_Site As String Implements IDomainContainer.Site
-            Get
-                Return Site
-            End Get
-        End Property
-        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue Implements IDomainContainer.DomainsSettingProp
-        Friend ReadOnly Property Domains As List(Of String) Implements IDomainContainer.Domains
-        Private ReadOnly Property DomainsTemp As List(Of String) Implements IDomainContainer.DomainsTemp
-        Private Property DomainsChanged As Boolean = False Implements IDomainContainer.DomainsChanged
-        Private ReadOnly Property DomainsDefault As String = "xvideos.com|xnxx.com" Implements IDomainContainer.DomainsDefault
-#End Region
+        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue
+        Friend ReadOnly Property Domains As DomainsContainer
         <PropertyOption(ControlText:="Download UHD", ControlToolTip:="Download UHD (4K) content"), PXML>
         Friend Property DownloadUHD As PropertyValue
-        Private Property Initialized As Boolean = False Implements IDomainContainer.Initialized
         <PropertyOption(ControlText:="Playlist of saved videos",
                         ControlToolTip:="Your personal videos playlist to download as 'saved posts'. " & vbCr &
                                         "This playlist must be private (Visibility = 'Only me'). It also required cookies." & vbCr &
@@ -51,46 +39,34 @@ Namespace API.XVIDEOS
 #Region "Initializer"
         Friend Sub New()
             MyBase.New("XVIDEOS", "www.xvideos.com")
-            Responser.DeclaredError = EDP.ThrowException
-            Domains = New List(Of String)
-            DomainsTemp = New List(Of String)
-            SiteDomains = New PropertyValue(DomainsDefault, GetType(String), Sub(s) UpdateDomains())
+            Domains = New DomainsContainer(Me, "xvideos.com|xnxx.com")
+            SiteDomains = New PropertyValue(Domains.DomainsDefault, GetType(String))
+            Domains.DestinationProp = SiteDomains
             DownloadUHD = New PropertyValue(False)
             SavedVideosPlaylist = New PropertyValue(String.Empty, GetType(String))
             UrlPatternUser = "https://xvideos.com/{0}"
         End Sub
         Friend Overrides Sub EndInit()
-            Initialized = True
-            DomainContainer.EndInit(Me)
-            DomainsTemp.ListAddList(Domains)
+            Domains.PopulateInitialDomains(SiteDomains.Value)
             MyBase.EndInit()
         End Sub
 #End Region
-#Region "Edit"
-        Private Property DomainsUpdateInProgress As Boolean = False Implements IDomainContainer.DomainsUpdateInProgress
-        Private Property DomainsUpdatedBySite As Boolean = False Implements IDomainContainer.DomainsUpdatedBySite
-        Friend Sub UpdateDomains() Implements IDomainContainer.UpdateDomains
-            DomainContainer.UpdateDomains(Me)
+#Region "Domains Support"
+        Protected Overrides Sub DomainsApply()
+            Domains.Apply()
+            MyBase.DomainsApply()
         End Sub
-        Friend Overrides Sub Update()
-            DomainContainer.Update(Me)
-            Responser.SaveSettings()
-        End Sub
-        Friend Overrides Sub EndEdit()
-            DomainContainer.EndEdit(Me)
-            MyBase.EndEdit()
+        Protected Overrides Sub DomainsReset()
+            Domains.Reset()
+            MyBase.DomainsReset()
         End Sub
         Friend Overrides Sub OpenSettingsForm()
-            DomainContainer.OpenSettingsForm(Me)
+            Domains.OpenSettingsForm()
         End Sub
 #End Region
 #Region "Download"
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
-            If What = ISiteSettings.Download.SavedPosts Then
-                Return New UserData With {.IsSavedPosts = True, .User = New UserInfo With {.Name = "XVIDEOS"}}
-            Else
-                Return New UserData
-            End If
+            Return New UserData
         End Function
         Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
             If Settings.UseM3U8 Then
@@ -110,9 +86,11 @@ Namespace API.XVIDEOS
             __user &= $"/{User.Name.Replace($"{__user}_", String.Empty)}"
             Return __user
         End Function
-        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider, ByVal Channel As Boolean) As String
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
             Return String.Format(UrlPatternUser, GetUserUrlPart(User))
         End Function
+#End Region
+#Region "IsMyUser, IsMyImageVideo"
         Private Const UserRegexDefault As String = "/(profiles|[\w]*?[-]{0,1}channels)/([^/]+)(\Z|.*?)"
         Private Const URD As String = ".*?{0}{1}"
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
@@ -132,27 +110,9 @@ Namespace API.XVIDEOS
             End If
             Return Nothing
         End Function
-#End Region
-#Region "Get special data"
         Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
             If Not URL.IsEmptyString And Domains.Count > 0 Then
-                If Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions With {.UserName = URL, .Exists = True}
-            End If
-            Return Nothing
-        End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            If Not URL.IsEmptyString And Settings.UseM3U8 Then
-                Dim spf$ = String.Empty
-                Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
-                f.Name = "video"
-                f.Extension = "mp4"
-                Using resp As Responser = Responser.Copy
-                    Using user As New UserData With {.HOST = Settings(XvideosSiteKey)}
-                        DirectCast(user, UserDataBase).User.File = f
-                        Dim p As UserMedia = user.Download(URL, resp, DownloadUHD.Value, String.Empty)
-                        If p.State = UserMedia.States.Downloaded Then p.SpecialFolder = spf : Return {p}
-                    End Using
-                End Using
+                If Domains.Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions(Site, URL)
             End If
             Return Nothing
         End Function

SCrawler/API/XVIDEOS/UserData.vb

@@ -8,11 +8,11 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Documents.JSON
-Imports UStates = SCrawler.API.Base.UserMedia.States
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.XVIDEOS
     Friend Class UserData : Inherits UserDataBase
@@ -24,8 +24,8 @@ Namespace API.XVIDEOS
                 If ParamsArray.ListExists(3) Then
                     ID = ParamsArray(0)
                     URL = ParamsArray(1)
-                    If Not URL.IsEmptyString Then URL = $"https://www.xvideos.com/{URL.StringTrimStart("/")}"
-                    Title = ParamsArray(2)
+                    If Not URL.IsEmptyString Then URL = $"https://www.xvideos.com/{HtmlConverter(URL).StringTrimStart("/")}"
+                    Title = TitleHtmlConverter(ParamsArray(2))
                 End If
                 Return Me
             End Function
@@ -43,6 +43,7 @@ Namespace API.XVIDEOS
         Friend Sub New()
             SeparateVideoFolder = False
             UseInternalM3U8Function = True
+            UseClientTokens = True
         End Sub
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             If Not Settings.UseM3U8 Then MyMainLOG = $"{ToStringForLog()}: File [ffmpeg.exe] not found" : Exit Sub
@@ -55,6 +56,7 @@ Namespace API.XVIDEOS
         End Sub
         Private Sub DownloadUserVideo(ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
+            Dim isQuickies As Boolean = False
             Try
                 Dim NextPage%, d%
                 Dim limit% = If(DownloadTopCount, -1)
@@ -77,39 +79,43 @@ Namespace API.XVIDEOS
                             URL = $"https://www.xvideos.com/{user}/videos/new/{If(NextPage = 0, String.Empty, NextPage)}"
                         Else 'Quickies
                             URL = $"https://www.xvideos.com/quickies-api/profilevideos/all/none/N/{ID}/{NextPage}"
+                            isQuickies = True
                         End If
+                        If Not j Is Nothing Then j.Dispose()
                         r = Responser.GetResponse(URL,, EDP.ReturnValue)
                         If Not r.IsEmptyString Then
                             If Not EnvirSet Then UserExists = True : UserSuspended = False : EnvirSet = True
-                            j = JsonDocument.Parse(r).XmlIfNothing
-                            With j
-                                If .Contains("videos") Then
-                                    With .Item("videos")
-                                        If .Count > 0 Then
-                                            NextPage += 1
-                                            For Each jj In .Self
-                                                p = New UserMedia With {
-                                                    .Post = jj.Value("id"),
-                                                    .URL = $"https://www.xvideos.com/{jj.Value(n).StringTrimStart("/")}"
-                                                }
-                                                If Not p.Post.ID.IsEmptyString And Not jj.Value(n).IsEmptyString Then
-                                                    If Not _TempPostsList.Contains(p.Post.ID) Then
-                                                        _TempPostsList.Add(p.Post.ID)
-                                                        _TempMediaList.Add(p)
-                                                        d += 1
-                                                        If limit > 0 And d = limit Then Exit Do
-                                                    Else
-                                                        Exit Do
+                            j = JsonDocument.Parse(r)
+                            If Not j Is Nothing Then
+                                With j
+                                    If .Contains("videos") Then
+                                        With .Item("videos")
+                                            If .Count > 0 Then
+                                                NextPage += 1
+                                                For Each jj In .Self
+                                                    p = New UserMedia With {
+                                                        .Post = jj.Value("id"),
+                                                        .URL = $"https://www.xvideos.com/{jj.Value(n).StringTrimStart("/")}"
+                                                    }
+                                                    If Not p.Post.ID.IsEmptyString And Not jj.Value(n).IsEmptyString Then
+                                                        If Not _TempPostsList.Contains(p.Post.ID) Then
+                                                            _TempPostsList.Add(p.Post.ID)
+                                                            _TempMediaList.Add(p)
+                                                            d += 1
+                                                            If limit > 0 And d = limit Then Exit Do
+                                                        Else
+                                                            Exit Do
+                                                        End If
                                                     End If
-                                                End If
-                                            Next
-                                            Continue Do
-                                        End If
-                                    End With
-                                End If
-                            End With
+                                                Next
+                                                Continue Do
+                                            End If
+                                        End With
+                                    End If
+                                    .Dispose()
+                                End With
+                            End If
                         End If
-                        If Not j Is Nothing Then j.Dispose()
                         Exit Do
                     Loop While NextPage < 100
                 Next
@@ -119,18 +125,12 @@ Namespace API.XVIDEOS
                 If _TempMediaList.Count > 0 Then
                     For i% = 0 To _TempMediaList.Count - 1
                         ThrowAny(Token)
-                        _TempMediaList(i) = GetVideoData(_TempMediaList(i), Responser, MySettings.DownloadUHD.Value)
+                        _TempMediaList(i) = GetVideoData(_TempMediaList(i))
                     Next
                     _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
                 End If
-            Catch oex As OperationCanceledException
-            Catch dex As ObjectDisposedException
             Catch ex As Exception
-                If Responser.StatusCode = Net.HttpStatusCode.NotFound Then
-                    UserExists = False
-                Else
-                    ProcessException(ex, Token, $"data downloading error [{URL}]")
-                End If
+                ProcessException(ex, Token, $"data downloading error [{URL}]",, isQuickies)
             Finally
                 If _TempMediaList.ListExists Then _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
             End Try
@@ -152,8 +152,16 @@ Namespace API.XVIDEOS
                     URL = $"{MySettings.SavedVideosPlaylist.Value}{If(NextPage = 0, String.Empty, $"/{NextPage}")}"
                     r = Responser.GetResponse(URL,, EDP.ReturnValue)
                     If Responser.HasError Then
-                        If Responser.StatusCode = Net.HttpStatusCode.NotFound And NextPage > 0 Then Exit Do
-                        Throw New Exception(Responser.ErrorText, Responser.ErrorException)
+                        If Responser.StatusCode = Net.HttpStatusCode.NotFound Then
+                            If NextPage = 0 Then
+                                MyMainLOG = $"XVIDEOS saved video playlist {URL} not found."
+                                Exit Sub
+                            Else
+                                Exit Do
+                            End If
+                        Else
+                            Throw New Exception(Responser.ErrorText, Responser.ErrorException)
+                        End If
                     End If
                     NextPage += 1
                     If Not r.IsEmptyString Then
@@ -174,7 +182,7 @@ Namespace API.XVIDEOS
                 If _TempMediaList.Count > 0 Then
                     For i% = 0 To _TempMediaList.Count - 1
                         ThrowAny(Token)
-                        _TempMediaList(i) = GetVideoData(_TempMediaList(i), Responser, MySettings.DownloadUHD.Value)
+                        _TempMediaList(i) = GetVideoData(_TempMediaList(i))
                     Next
                     _TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
                 End If
@@ -182,19 +190,19 @@ Namespace API.XVIDEOS
                 ProcessException(ex, Token, $"data downloading error [{URL}]")
             End Try
         End Sub
-        Private Function GetVideoData(ByVal Media As UserMedia, ByVal resp As Responser, ByVal DownloadUHD As Boolean) As UserMedia
+        Private Function GetVideoData(ByVal Media As UserMedia) As UserMedia
             Try
                 If Not Media.URL.IsEmptyString Then
-                    Dim r$ = resp.GetResponse(Media.URL)
+                    Dim r$ = Responser.GetResponse(Media.URL)
                     If Not r.IsEmptyString Then
                         Dim NewUrl$ = RegexReplace(r, Regex_M3U8)
                         If Not NewUrl.IsEmptyString Then
                             Dim appender$ = RegexReplace(NewUrl, Regex_M3U8_Appender)
                             Dim t$ = If(Media.PictureOption.IsEmptyString, RegexReplace(r, Regex_VideoTitle), Media.PictureOption)
-                            r = resp.GetResponse(NewUrl)
+                            r = Responser.GetResponse(NewUrl)
                             If Not r.IsEmptyString Then
                                 Dim ls As List(Of Sizes) = RegexFields(Of Sizes)(r, {Regex_M3U8_Reparse}, {1, 2})
-                                If ls.ListExists And Not DownloadUHD Then ls.RemoveAll(Function(v) Not v.Value.ValueBetween(1, 1080))
+                                If ls.ListExists And Not MySettings.DownloadUHD.Value Then ls.RemoveAll(Function(v) Not v.Value.ValueBetween(1, 1080))
                                 If ls.ListExists Then
                                     ls.Sort()
                                     NewUrl = $"{appender}/{ls(0).Data.StringTrimStart("/")}"
@@ -228,31 +236,28 @@ Namespace API.XVIDEOS
                 Return Nothing
             End Try
         End Function
-        Friend Function Download(ByVal URL As String, ByVal resp As Responser, ByVal DownloadUHD As Boolean, ByVal ID As String)
-            Dim m As UserMedia = GetVideoData(New UserMedia(URL, UTypes.VideoPre) With {.Post = ID}, resp, DownloadUHD)
-            If Not m.URL.IsEmptyString Then
-                Dim f As SFile = m.File
-                f.Path = MyFile.PathNoSeparator
-                m.State = UStates.Tried
-                Try
-                    f = M3U8.Download(m.URL, m.PictureOption, f)
-                    m.File = f
-                    m.State = UStates.Downloaded
-                Catch ex As Exception
-                    m.State = UStates.Missing
-                End Try
-            End If
-            Return m
-        End Function
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
-        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
-            Return M3U8.Download(Media.URL, Media.PictureOption, DestinationFile)
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            Dim m As UserMedia = GetVideoData(New UserMedia(Data.URL, UTypes.VideoPre))
+            If Not m.URL.IsEmptyString Then _TempMediaList.Add(m)
+        End Sub
+        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile
+            Return M3U8.Download(Media.URL, Media.PictureOption, DestinationFile, Token, If(UseInternalM3U8Function_UseProgress, Progress, Nothing))
         End Function
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
                                                           Optional ByVal EObj As Object = Nothing) As Integer
-            Return 0
+            Dim isQuickies As Boolean = False
+            If Not IsNothing(EObj) AndAlso TypeOf EObj Is Boolean Then isQuickies = CBool(EObj)
+            If Responser.StatusCode = Net.HttpStatusCode.NotFound Then
+                UserExists = False
+                Return 1
+            ElseIf isQuickies And Responser.StatusCode = Net.HttpStatusCode.InternalServerError Then
+                Return 1
+            Else
+                Return 0
+            End If
         End Function
     End Class
 End Namespace

SCrawler/API/Xhamster/Declarations.vb

@@ -13,7 +13,6 @@ Namespace API.Xhamster
         Friend Const XhamsterSiteKey As String = "AndyProgram_XHamster"
         Friend ReadOnly HtmlScript As RParams = RParams.DMS("\<script id='initials-script'\>window.initials=(\{.+?\});\</script\>", 1, EDP.ReturnValue,
                                                             CType(Function(Input$) Input.StringTrim, Func(Of String, String)))
-        Friend ReadOnly DateProvider As New CustomProvider(Function(v, d, p, n, e) ADateTime.ParseUnicode(v))
         Friend ReadOnly FirstM3U8FileRegEx As RParams = RParams.DM("RESOLUTION=\d+x(\d+).*?[\r\n]+?([^#]*?\.m3u8.*)", 0, RegexReturn.List)
     End Module
 End Namespace

SCrawler/API/Xhamster/M3U8.vb

@@ -6,10 +6,12 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Threading
 Imports SCrawler.API.Base
 Imports SCrawler.API.Base.M3U8Declarations
-Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Forms.Toolbars
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Xhamster
     Friend NotInheritable Class M3U8
         Private Sub New()
@@ -72,8 +74,9 @@ Namespace API.Xhamster
                 Responser.UseGZipStream = False
             End Try
         End Function
-        Friend Shared Function Download(ByVal Media As UserMedia, ByVal Responser As Responser, ByVal UHD As Boolean) As SFile
-            Return M3U8Base.Download(ObtainUrls(Media.URL, Responser, UHD), Media.File, Responser)
+        Friend Shared Function Download(ByVal Media As UserMedia, ByVal Responser As Responser, ByVal UHD As Boolean,
+                                        ByVal Token As CancellationToken, ByVal Progress As MyProgress) As SFile
+            Return M3U8Base.Download(ObtainUrls(Media.URL, Responser, UHD), Media.File, Responser, Token, Progress)
         End Function
     End Class
 End Namespace

SCrawler/API/Xhamster/SiteSettings.vb

@@ -7,16 +7,15 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports SCrawler.API.Base
-Imports SCrawler.API.BaseObjects
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
-Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Xhamster
     <Manifest(XhamsterSiteKey), SavedPosts, SpecialForm(True), TaskGroup(SettingsCLS.TaskStackNamePornSite)>
-    Friend Class SiteSettings : Inherits SiteSettingsBase : Implements IDomainContainer
+    Friend Class SiteSettings : Inherits SiteSettingsBase
 #Region "Declarations"
-        Friend Overrides ReadOnly Property Icon As Icon Implements IDomainContainer.Icon
+        Friend Overrides ReadOnly Property Icon As Icon
             Get
                 Return My.Resources.SiteResources.XhamsterIcon_32
             End Get
@@ -26,94 +25,45 @@ Namespace API.Xhamster
                 Return My.Resources.SiteResources.XhamsterPic_32
             End Get
         End Property
-#Region "Domains"
-        Private ReadOnly Property IDomainContainer_Site As String Implements IDomainContainer.Site
-            Get
-                Return Site
-            End Get
-        End Property
-        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue Implements IDomainContainer.DomainsSettingProp
-        Friend ReadOnly Property Domains As List(Of String) Implements IDomainContainer.Domains
-        Private ReadOnly Property DomainsTemp As List(Of String) Implements IDomainContainer.DomainsTemp
-        Private Property DomainsChanged As Boolean = False Implements IDomainContainer.DomainsChanged
-        Friend ReadOnly Property DomainsUpdated As Boolean
-            Get
-                Return DomainsUpdatedBySite
-            End Get
-        End Property
-        Private ReadOnly Property DomainsDefault As String = "xhamster.com" Implements IDomainContainer.DomainsDefault
-#End Region
+        <PXML("Domains")> Private ReadOnly Property SiteDomains As PropertyValue
+        Friend ReadOnly Property Domains As DomainsContainer
         <PropertyOption(ControlText:="Download UHD", ControlToolTip:="Download UHD (4K) content"), PXML>
         Friend Property DownloadUHD As PropertyValue
-        Private Property Initialized As Boolean = False Implements IDomainContainer.Initialized
 #End Region
 #Region "Initializer"
         Friend Sub New()
             MyBase.New("XHamster", "xhamster.com")
 
-            Responser.DeclaredError = EDP.ThrowException
-
-            Domains = New List(Of String)
-            DomainsTemp = New List(Of String)
-            SiteDomains = New PropertyValue(DomainsDefault, GetType(String), Sub(s) UpdateDomains())
+            Domains = New DomainsContainer(Me, "xhamster.com")
+            SiteDomains = New PropertyValue(Domains.DomainsDefault, GetType(String))
+            Domains.DestinationProp = SiteDomains
             DownloadUHD = New PropertyValue(False)
 
-            UrlPatternUser = "https://xhamster.com/users/{0}"
-            UrlPatternChannel = "https://xhamster.com/channels/{0}"
+
+            UrlPatternUser = "https://xhamster.com/{0}/{1}"
             UserRegex = RParams.DMS($"/({UserOption}|{ChannelOption})/([^/]+)(\Z|.*)", 0, RegexReturn.ListByMatch)
             ImageVideoContains = "xhamster"
         End Sub
         Friend Overrides Sub EndInit()
-            Initialized = True
-            DomainContainer.EndInit(Me)
-            DomainsTemp.ListAddList(Domains)
+            Domains.PopulateInitialDomains(SiteDomains.Value)
             MyBase.EndInit()
         End Sub
 #End Region
-#Region "UpdateDomains"
-        Private Property DomainsUpdateInProgress As Boolean = False Implements IDomainContainer.DomainsUpdateInProgress
-        Private Property DomainsUpdatedBySite As Boolean = False Implements IDomainContainer.DomainsUpdatedBySite
-        Friend Overloads Sub UpdateDomains() Implements IDomainContainer.UpdateDomains
-            DomainContainer.UpdateDomains(Me)
+#Region "Domains Support"
+        Protected Overrides Sub DomainsApply()
+            Domains.Apply()
+            MyBase.DomainsApply()
         End Sub
-        Friend Overloads Sub UpdateDomains(ByVal NewDomains As IEnumerable(Of String), ByVal Internal As Boolean)
-            DomainContainer.UpdateDomains(Me, NewDomains, Internal)
-        End Sub
-#End Region
-#Region "Edit"
-        Friend Overrides Sub Update()
-            DomainContainer.Update(Me)
-            Responser.SaveSettings()
-            MyBase.Update()
-        End Sub
-        Friend Overrides Sub EndEdit()
-            DomainContainer.EndEdit(Me)
-            MyBase.EndEdit()
+        Protected Overrides Sub DomainsReset()
+            Domains.Reset()
+            MyBase.DomainsReset()
         End Sub
         Friend Overrides Sub OpenSettingsForm()
-            DomainContainer.OpenSettingsForm(Me)
+            Domains.OpenSettingsForm()
         End Sub
 #End Region
         Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
-            If What = ISiteSettings.Download.SavedPosts Then
-                Return New UserData With {.IsSavedPosts = True, .User = New UserInfo With {.Name = "xhamster"}}
-            Else
-                Return New UserData
-            End If
-        End Function
-        Friend Overrides Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable
-            If Available(ISiteSettings.Download.Main, True) Then
-                Using resp As Responser = Responser.Copy
-                    Dim spf$ = String.Empty
-                    Dim f As SFile = GetSpecialDataFile(Path, AskForPath, spf)
-                    Dim m As UserMedia = UserData.GetVideoInfo(URL, resp, f)
-                    If m.State = UserMedia.States.Downloaded Then
-                        m.SpecialFolder = f
-                        Return {m}
-                    End If
-                End Using
-            End If
-            Return Nothing
+            Return New UserData
         End Function
         Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
             If Settings.UseM3U8 AndAlso MyBase.Available(What, Silent) Then
@@ -126,22 +76,26 @@ Namespace API.Xhamster
                 Return False
             End If
         End Function
-        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
-            Return Media.URL_BASE
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
+            With DirectCast(User, UserData) : Return String.Format(UrlPatternUser, IIf(.IsChannel, ChannelOption, UserOption), .TrueName) : End With
         End Function
-#Region "Is my user/data"
-        Private Const ChannelOption As String = "channels"
+#Region "IsMyUser, IsMyImageVideo"
+        Friend Const ChannelOption As String = "channels"
         Private Const UserOption As String = "users"
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
-            If Not UserURL.IsEmptyString AndAlso Domains.Count > 0 AndAlso Domains.Exists(Function(d) UserURL.ToLower.Contains(d.ToLower)) Then
+            If Not UserURL.IsEmptyString AndAlso Domains.Domains.Count > 0 AndAlso Domains.Domains.Exists(Function(d) UserURL.ToLower.Contains(d.ToLower)) Then
                 Dim data As List(Of String) = RegexReplace(UserURL, UserRegex)
-                If data.ListExists(3) AndAlso Not data(2).IsEmptyString Then Return New ExchangeOptions(Site, data(2), data(1) = ChannelOption)
+                If data.ListExists(3) AndAlso Not data(2).IsEmptyString Then
+                    Dim n$ = data(2)
+                    If Not data(1).IsEmptyString AndAlso data(1) = ChannelOption Then n &= $"@{data(1)}"
+                    Return New ExchangeOptions(Site, n)
+                End If
             End If
             Return Nothing
         End Function
         Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
-            If Not URL.IsEmptyString And Domains.Count > 0 Then
-                If Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions With {.UserName = URL, .Exists = True}
+            If Not URL.IsEmptyString And Domains.Domains.Count > 0 Then
+                If Domains.Domains.Exists(Function(d) URL.Contains(d)) Then Return New ExchangeOptions(Site, URL)
             End If
             Return Nothing
         End Function

SCrawler/API/Xhamster/UserData.vb

@@ -8,6 +8,7 @@
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
 Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
@@ -15,7 +16,12 @@ Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.Xhamster
     Friend Class UserData : Inherits UserDataBase
+#Region "XML names"
+        Private Const Name_TrueName As String = "TrueName"
+#End Region
 #Region "Declarations"
+        Friend Property IsChannel As Boolean = False
+        Friend Property TrueName As String = String.Empty
         Private ReadOnly Property MySettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
@@ -26,11 +32,39 @@ Namespace API.Xhamster
         End Structure
         Private ReadOnly _TempPhotoData As List(Of UserMedia)
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            Dim setNames As Action = Sub()
+                                         If TrueName.IsEmptyString Then
+                                             Dim n$() = Name.Split("@")
+                                             If n.ListExists Then
+                                                 If n.Length = 2 Then
+                                                     TrueName = n(0)
+                                                     IsChannel = True
+                                                 ElseIf IsChannel Then
+                                                     TrueName = Name
+                                                 Else
+                                                     TrueName = n(0)
+                                                 End If
+                                             End If
+                                         End If
+                                     End Sub
+            With Container
+                If Loading Then
+                    IsChannel = .Value(Name_IsChannel).FromXML(Of Boolean)(False)
+                    TrueName = .Value(Name_TrueName)
+                    setNames.Invoke
+                Else
+                    setNames.Invoke
+                    .Add(Name_IsChannel, IsChannel.BoolToInteger)
+                    .Add(Name_TrueName, TrueName)
+                    setNames.Invoke
+                End If
+            End With
         End Sub
 #End Region
 #Region "Initializer"
         Friend Sub New()
             UseInternalM3U8Function = True
+            UseClientTokens = True
             _TempPhotoData = New List(Of UserMedia)
         End Sub
 #End Region
@@ -58,10 +92,10 @@ Namespace API.Xhamster
                     URL = $"https://xhamster.com/my/favorites/{IIf(IsVideo, "videos", "photos-and-galleries")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
                     listNode = If(IsVideo, {"favoriteVideoListComponent", "models"}, {"favoritesGalleriesAndPhotosCollection"})
                 ElseIf IsChannel Then
-                    URL = $"https://xhamster.com/channels/{Name}/newest{IIf(Page = 1, String.Empty, $"/{Page}")}"
+                    URL = $"https://xhamster.com/channels/{TrueName}/newest{IIf(Page = 1, String.Empty, $"/{Page}")}"
                     listNode = {"trendingVideoListComponent", "models"}
                 Else
-                    URL = $"https://xhamster.com/users/{Name}/{IIf(IsVideo, "videos", "photos")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
+                    URL = $"https://xhamster.com/users/{TrueName}/{IIf(IsVideo, "videos", "photos")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
                     listNode = {If(IsVideo, "userVideoCollection", "userGalleriesCollection")}
                 End If
                 ThrowAny(Token)
@@ -69,10 +103,10 @@ Namespace API.Xhamster
                 Dim r$ = Responser.GetResponse(URL)
                 If Not r.IsEmptyString Then r = RegexReplace(r, HtmlScript)
                 If Not r.IsEmptyString Then
-                    Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
-                        If j.Count > 0 Then
-                            If Not MySettings.DomainsUpdated AndAlso j.Contains("trustURLs") Then _
-                               MySettings.UpdateDomains(j("trustURLs").Select(Function(d) d(0).XmlIfNothingValue), False)
+                    Using j As EContainer = JsonDocument.Parse(r)
+                        If j.ListExists Then
+                            If Not MySettings.Domains.UpdatedBySite AndAlso j.Contains("trustURLs") Then _
+                               MySettings.Domains.Add(j("trustURLs").Select(Function(d) d(0).XmlIfNothingValue), True)
 
                             MaxPage = j.Value(mPages).FromXML(Of Integer)(-1)
 
@@ -113,7 +147,8 @@ Namespace API.Xhamster
                     End Using
                 End If
 
-                If (Not _TempMediaList.Count = cBefore Or skipped) And (IsChannel Or (MaxPage > 0 And Page < MaxPage)) Then DownloadData(Page + 1, IsVideo, Token)
+                If (Not _TempMediaList.Count = cBefore Or skipped) And
+                   (IsChannel Or (MaxPage > 0 And Page < MaxPage)) Then DownloadData(Page + 1, IsVideo, Token)
             Catch ex As Exception
                 ProcessException(ex, Token, $"data downloading error [{URL}]")
             End Try
@@ -130,7 +165,7 @@ Namespace API.Xhamster
                             m = _TempMediaList(i)
                             If Not m.URL_BASE.IsEmptyString Then
                                 m2 = Nothing
-                                If GetM3U8(m2, m.URL_BASE, Responser) Then
+                                If GetM3U8(m2, m.URL_BASE) Then
                                     m2.URL_BASE = m.URL_BASE
                                     _TempMediaList(i) = m2
                                 Else
@@ -205,7 +240,7 @@ Namespace API.Xhamster
                         If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then
                             ThrowAny(Token)
                             m2 = Nothing
-                            If GetM3U8(m2, m.URL_BASE, Responser) Then
+                            If GetM3U8(m2, m.URL_BASE) Then
                                 m2.URL_BASE = m.URL_BASE
                                 _TempMediaList.ListAddValue(m2, LNC)
                                 rList.Add(i)
@@ -224,8 +259,7 @@ Namespace API.Xhamster
         End Sub
 #End Region
 #Region "GetM3U8"
-        Private Overloads Function GetM3U8(ByRef m As UserMedia, ByVal URL As String, ByVal Responser As Responser,
-                                           Optional ByVal e As ErrorsDescriber = Nothing) As Boolean
+        Private Overloads Function GetM3U8(ByRef m As UserMedia, ByVal URL As String) As Boolean
             Try
                 If Not URL.IsEmptyString Then
                     Dim r$ = Responser.GetResponse(URL)
@@ -242,8 +276,7 @@ Namespace API.Xhamster
                 End If
                 Return False
             Catch ex As Exception
-                If Not e.Exists Then e = EDP.ReturnValue
-                Return ErrorsDescriber.Execute(e, ex, $"[{ToStringForLog()}]: API.Xhamster.GetM3U8({URL})", False)
+                Return ErrorsDescriber.Execute(EDP.ReturnValue, ex, $"[{ToStringForLog()}]: API.Xhamster.GetM3U8({URL})", False)
             End Try
         End Function
         Private Overloads Function GetM3U8(ByRef m As UserMedia, ByVal j As EContainer) As Boolean
@@ -252,44 +285,29 @@ Namespace API.Xhamster
             Return False
         End Function
 #End Region
-#Region "Standalone downloader"
-        Friend Shared Function GetVideoInfo(ByVal URL As String, ByVal Responser As Responser, ByVal Path As SFile) As UserMedia
-            Try
-                Using u As New UserData With {.Responser = Responser, .HOST = Settings(XhamsterSiteKey)}
-                    Dim m As UserMedia = Nothing
-                    If u.GetM3U8(m, URL, Responser, EDP.ThrowException) Then
-                        m.File.Path = Path.Path
-                        Dim f As SFile = u.DownloadM3U8(m.URL, m, m.File)
-                        If Not f.IsEmptyString Then
-                            m.File = f
-                            m.State = UserMedia.States.Downloaded
-                            Return m
-                        End If
-                    End If
-                End Using
-                Return Nothing
-            Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog + EDP.ReturnValue, ex, $"XHamster standalone download error: [{URL}]", New UserMedia)
-            End Try
-        End Function
+#Region "DownloadSingleObject"
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            _ContentList.Add(New UserMedia(Data.URL_BASE) With {.State = UserMedia.States.Missing})
+            ReparseMissing(Token)
+        End Sub
 #End Region
 #Region "Download data"
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
         End Sub
-        Protected Overloads Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile) As SFile
+        Protected Overrides Function DownloadM3U8(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile
             Media.File = DestinationFile
-            Return M3U8.Download(Media, Responser, MySettings.DownloadUHD.Value)
+            Return M3U8.Download(Media, Responser, MySettings.DownloadUHD.Value, Token, If(UseInternalM3U8Function_UseProgress, Progress, Nothing))
         End Function
 #End Region
 #Region "Create media"
-        Private Shared Function ExtractMedia(ByVal j As EContainer, ByVal t As UTypes, Optional ByVal UrlNode As String = "pageURL",
-                                             Optional ByVal DetectGalery As Boolean = True, Optional ByVal PostDate As Date? = Nothing) As UserMedia
+        Private Function ExtractMedia(ByVal j As EContainer, ByVal t As UTypes, Optional ByVal UrlNode As String = "pageURL",
+                                      Optional ByVal DetectGalery As Boolean = True, Optional ByVal PostDate As Date? = Nothing) As UserMedia
             If Not j Is Nothing Then
                 Dim m As New UserMedia(j.Value(UrlNode).Replace("\", String.Empty), t) With {
                     .Post = New UserPost With {
                         .ID = j.Value("id"),
-                        .Date = AConvert(Of Date)(j.Value("created"), DateProvider, Nothing)
+                        .Date = AConvert(Of Date)(j.Value("created"), UnixDate32Provider, Nothing)
                     },
                     .PictureOption = TitleHtmlConverter(j.Value("title")),
                     .Object = New ExchObj

SCrawler/API/YouTube/SiteSettings.vb

@@ -0,0 +1,116 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports SCrawler.Plugin.Attributes
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Base
+Namespace API.YouTube
+    <Manifest(YouTubeSiteKey), SpecialForm(True), SpecialForm(False), SeparatedTasks(1)>
+    Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Declarations"
+        Friend Overrides ReadOnly Property Icon As Icon
+            Get
+                Return My.Resources.SiteYouTube.YouTubeIcon_32
+            End Get
+        End Property
+        Friend Overrides ReadOnly Property Image As Image
+            Get
+                Return My.Resources.SiteYouTube.YouTubePic_96
+            End Get
+        End Property
+        <PXML, PropertyOption(ControlText:="Download user videos")>
+        Friend ReadOnly Property DownloadVideos As PropertyValue
+        <PXML, PropertyOption(ControlText:="Download user shorts")>
+        Friend ReadOnly Property DownloadShorts As PropertyValue
+        <PXML, PropertyOption(ControlText:="Download user playlists")>
+        Friend ReadOnly Property DownloadPlaylists As PropertyValue
+        <PXML, PropertyOption(ControlText:="Use cookies", ControlToolTip:="Default value for new users." & vbCr & "Use cookies when downloading data.")>
+        Friend ReadOnly Property UseCookies As PropertyValue
+#End Region
+#Region "Initializer"
+        Friend Sub New()
+            MyBase.New(YouTubeSite, "youtube.com")
+            Responser.Cookies.ChangedAllowInternalDrop = False
+            DownloadVideos = New PropertyValue(True)
+            DownloadShorts = New PropertyValue(False)
+            DownloadPlaylists = New PropertyValue(False)
+            UseCookies = New PropertyValue(False)
+        End Sub
+#End Region
+#Region "GetInstance"
+        Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider
+            Return New UserData
+        End Function
+#End Region
+#Region "Edit, Update"
+        Friend Overrides Sub Update()
+            If _SiteEditorFormOpened Then
+                With Responser.Cookies
+                    If .Changed Then
+                        .Changed = False
+                        With DirectCast(MyYouTubeSettings, YTSettings_Internal)
+                            .Cookies.Clear()
+                            .Cookies.AddRange(Responser.Cookies)
+                            .CookiesUpdated = True
+                            .PerformUpdate()
+                        End With
+                    End If
+                End With
+            End If
+            MyBase.Update()
+        End Sub
+        Friend Overrides Sub EndEdit()
+            If _SiteEditorFormOpened Then DirectCast(MyYouTubeSettings, YTSettings_Internal).ResetUpdate()
+            MyBase.EndEdit()
+        End Sub
+#End Region
+#Region "Available"
+        Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean
+            Return Settings.YtdlpFile.Exists And Settings.FfmpegFile.Exists
+        End Function
+#End Region
+#Region "MyUser, MyUrl, get urls"
+        Friend Const ChannelUserInt As Integer = 10000
+        Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
+            Dim isMusic As Boolean = False
+            Dim id$ = String.Empty
+            Dim isChannelUser As Boolean = False
+            Dim t As YouTubeMediaType = YouTubeFunctions.Info_GetUrlType(UserURL, isMusic, isChannelUser, id)
+            If Not t = YouTubeMediaType.Undefined And Not t = YouTubeMediaType.Single And Not id.IsEmptyString Then
+                Return New ExchangeOptions(Site, $"{id}@{CInt(t) + IIf(isMusic, UserMedia.Types.Audio, 0) + IIf(isChannelUser, ChannelUserInt, 0)}")
+            End If
+            Return Nothing
+        End Function
+        Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
+            If YouTubeFunctions.IsMyUrl(URL) Then Return New ExchangeOptions(Site, URL) Else Return Nothing
+        End Function
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            If Not User Is Nothing AndAlso TypeOf User Is UserData Then
+                Return $"https://{IIf(DirectCast(User, UserData).IsMusic, "music", "www")}.youtube.com/watch?v={Media.Post.ID}"
+            Else
+                Return String.Empty
+            End If
+        End Function
+        Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
+            If Not User Is Nothing AndAlso TypeOf User Is UserData Then Return DirectCast(User, UserData).GetUserUrl Else Return String.Empty
+        End Function
+#End Region
+#Region "Settings form, options"
+        Friend Overrides Sub OpenSettingsForm()
+            MyYouTubeSettings.ShowForm(False)
+        End Sub
+        Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)
+            If Options Is Nothing OrElse Not TypeOf Options Is UserExchangeOptions Then Options = New UserExchangeOptions(Me)
+            If OpenForm Then
+                Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
+            End If
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/YouTube/UserData.vb

@@ -0,0 +1,245 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports System.Threading
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Base
+Imports SCrawler.API.YouTube.Objects
+Imports PersonalUtilities.Functions.XML
+Namespace API.YouTube
+    Friend Class UserData : Inherits UserDataBase
+#Region "XML names"
+        Private Const Name_DownloadYTVideos As String = "YTDownloadVideos"
+        Private Const Name_DownloadYTShorts As String = "YTDownloadShorts"
+        Private Const Name_DownloadYTPlaylists As String = "YTDownloadPlaylists"
+        Private Const Name_YTUseCookies As String = "YTUseCookies"
+        Private Const Name_IsMusic As String = "YTIsMusic"
+        Private Const Name_IsChannelUser As String = "YTIsChannelUser"
+        Private Const Name_YTMediaType As String = "YTMediaType"
+        Private Const Name_LastDownloadDateVideos As String = "YTLastDownloadDateVideos"
+        Private Const Name_LastDownloadDateShorts As String = "YTLastDownloadDateShorts"
+        Private Const Name_LastDownloadDatePlaylist As String = "YTLastDownloadDatePlaylist"
+#End Region
+#Region "Declarations"
+        Friend Property DownloadYTVideos As Boolean = True
+        Friend Property DownloadYTShorts As Boolean = False
+        Friend Property DownloadYTPlaylists As Boolean = False
+        Friend Property YTUseCookies As Boolean = False
+        Friend Property IsMusic As Boolean = False
+        Friend Property IsChannelUser As Boolean = False
+        Friend Property YTMediaType As YouTubeMediaType = YouTubeMediaType.Undefined
+        Private LastDownloadDateVideos As Date? = Nothing
+        Private LastDownloadDateShorts As Date? = Nothing
+        Private LastDownloadDatePlaylist As Date? = Nothing
+        Friend Function GetUserUrl() As String
+            If YTMediaType = YouTubeMediaType.PlayList Then
+                Return $"https://{IIf(IsMusic, "music", "www")}.youtube.com/playlist?list={ID}"
+            Else
+                Return $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
+            End If
+        End Function
+#End Region
+#Region "Initializer, loader"
+        Friend Sub New()
+            UseInternalDownloadFileFunction = True
+            SeparateVideoFolder = False
+        End Sub
+        Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
+            With Container
+                Dim SetNames As Action = Sub()
+                                             If Not Name.IsEmptyString And ID.IsEmptyString Then
+                                                 Dim n As List(Of String) = Name.Split("@").ToList
+                                                 If n.ListExists(2) Then
+                                                     Dim intValue% = n(1)
+                                                     If intValue > 0 Then
+                                                         If intValue >= SiteSettings.ChannelUserInt Then IsChannelUser = True : intValue -= SiteSettings.ChannelUserInt
+                                                         If intValue >= UserMedia.Types.Audio Then IsMusic = True : intValue -= UserMedia.Types.Audio
+                                                         YTMediaType = intValue
+                                                         n.RemoveAt(1)
+                                                         ID = n(0)
+                                                     End If
+                                                 End If
+                                             End If
+                                         End Sub
+                If Loading Then
+                    DownloadYTVideos = .Value(Name_DownloadYTVideos).FromXML(Of Boolean)(True)
+                    DownloadYTShorts = .Value(Name_DownloadYTShorts).FromXML(Of Boolean)(False)
+                    DownloadYTPlaylists = .Value(Name_DownloadYTPlaylists).FromXML(Of Boolean)(False)
+                    IsMusic = .Value(Name_IsMusic).FromXML(Of Boolean)(False)
+                    IsChannelUser = .Value(Name_IsChannelUser).FromXML(Of Boolean)(False)
+                    YTMediaType = .Value(Name_YTMediaType).FromXML(Of Integer)(YouTubeMediaType.Undefined)
+                    LastDownloadDateVideos = AConvert(Of Date)(.Value(Name_LastDownloadDateVideos), DateTimeDefaultProvider, Nothing)
+                    LastDownloadDateShorts = AConvert(Of Date)(.Value(Name_LastDownloadDateShorts), DateTimeDefaultProvider, Nothing)
+                    LastDownloadDatePlaylist = AConvert(Of Date)(.Value(Name_LastDownloadDatePlaylist), DateTimeDefaultProvider, Nothing)
+                    SetNames.Invoke()
+                Else
+                    SetNames.Invoke()
+                    If Not ID.IsEmptyString Then .Value(Name_UserID) = ID
+                    .Add(Name_DownloadYTVideos, DownloadYTVideos.BoolToInteger)
+                    .Add(Name_DownloadYTShorts, DownloadYTShorts.BoolToInteger)
+                    .Add(Name_DownloadYTPlaylists, DownloadYTPlaylists.BoolToInteger)
+                    .Add(Name_IsMusic, IsMusic.BoolToInteger)
+                    .Add(Name_IsChannelUser, IsChannelUser.BoolToInteger)
+                    .Add(Name_YTMediaType, CInt(YTMediaType))
+                    .Add(Name_LastDownloadDateVideos, AConvert(Of String)(LastDownloadDateVideos, DateTimeDefaultProvider, String.Empty))
+                    .Add(Name_LastDownloadDateShorts, AConvert(Of String)(LastDownloadDateShorts, DateTimeDefaultProvider, String.Empty))
+                    .Add(Name_LastDownloadDatePlaylist, AConvert(Of String)(LastDownloadDatePlaylist, DateTimeDefaultProvider, String.Empty))
+                End If
+            End With
+        End Sub
+#End Region
+#Region "Exchange options"
+        Friend Overrides Function ExchangeOptionsGet() As Object
+            Return New UserExchangeOptions(Me)
+        End Function
+        Friend Overrides Sub ExchangeOptionsSet(ByVal Obj As Object)
+            If Not Obj Is Nothing AndAlso TypeOf Obj Is UserExchangeOptions Then
+                With DirectCast(Obj, UserExchangeOptions)
+                    DownloadYTVideos = .DownloadVideos
+                    DownloadYTShorts = .DownloadShorts
+                    DownloadYTPlaylists = .DownloadPlaylists
+                    YTUseCookies = .UseCookies
+                End With
+            End If
+        End Sub
+#End Region
+#Region "Download"
+        'Playlist reconfiguration implemented only for channels + music
+        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            Try
+                Dim container As IYouTubeMediaContainer = Nothing
+                Dim list As New List(Of IYouTubeMediaContainer)
+                Dim url$ = String.Empty
+                Dim maxDate As Date? = Nothing
+                Dim nDate As Func(Of Date?, Date?) = Function(ByVal dInput As Date?) As Date?
+                                                         If dInput.HasValue Then
+                                                             If dInput.Value.AddDays(3) < Now Then Return dInput.Value.AddDays(1) Else Return dInput
+                                                         Else
+                                                             Return Nothing
+                                                         End If
+                                                     End Function
+                Dim fillList As Func(Of Date?, Boolean) = Function(ByVal lDate As Date?) As Boolean
+                                                              If Not container Is Nothing AndAlso container.HasElements Then
+                                                                  Dim ce As IEnumerable(Of IYouTubeMediaContainer)
+                                                                  ce = container.Elements
+                                                                  If ce.ListExists Then ce = ce.Where(Function(e) e.ObjectType = YouTubeMediaType.Single)
+                                                                  If ce.ListExists AndAlso lDate.HasValue Then _
+                                                                     ce = ce.Where(Function(e) e.DateAdded <= lDate.Value AndAlso
+                                                                                               Not e.ID.IsEmptyString AndAlso Not _TempPostsList.Contains(e.ID))
+                                                                  If ce.ListExists Then
+                                                                      maxDate = ce.Max(Function(e) e.DateAdded)
+                                                                      list.AddRange(ce)
+                                                                      Return True
+                                                                  End If
+                                                              End If
+                                                              Return False
+                                                          End Function
+                Dim applySpecFolder As Action(Of String, Boolean) = Sub(ByVal fName As String, ByVal isPls As Boolean)
+                                                                        If If(container?.Count, 0) > 0 Then _
+                                                                           container.Elements.ForEach(Sub(ByVal el As YouTubeMediaContainerBase)
+                                                                                                          If isPls Then
+                                                                                                              el.SpecialPathSetForPlaylist(fName)
+                                                                                                          Else
+                                                                                                              el.SpecialPath = fName
+                                                                                                              el.SpecialPathDisabled = False
+                                                                                                          End If
+                                                                                                      End Sub)
+                                                                    End Sub
+                If YTMediaType = YouTubeMediaType.PlayList Then
+                    maxDate = Nothing
+                    LastDownloadDatePlaylist = nDate(LastDownloadDatePlaylist)
+                    url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/playlist?list={ID}"
+                    container = YouTubeFunctions.Parse(url, YTUseCookies, Token,, True, False,, LastDownloadDatePlaylist)
+                    applySpecFolder.Invoke(String.Empty, False)
+                    If fillList.Invoke(LastDownloadDatePlaylist) Then LastDownloadDatePlaylist = If(maxDate, Now)
+                ElseIf YTMediaType = YouTubeMediaType.Channel Then
+                    If IsMusic Or DownloadYTVideos Then
+                        maxDate = Nothing
+                        LastDownloadDateVideos = nDate(LastDownloadDateVideos)
+                        url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token,, True, False,, LastDownloadDateVideos)
+                        applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
+                        If fillList.Invoke(LastDownloadDateVideos) Then LastDownloadDateVideos = If(maxDate, Now)
+                    End If
+                    If Not IsMusic And DownloadYTShorts Then
+                        maxDate = Nothing
+                        LastDownloadDateShorts = nDate(LastDownloadDateShorts)
+                        url = $"https://www.youtube.com/{IIf(IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/shorts"
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token,, True, False,, LastDownloadDateShorts)
+                        applySpecFolder.Invoke("Shorts", False)
+                        If fillList.Invoke(LastDownloadDateShorts) Then LastDownloadDateShorts = If(maxDate, Now)
+                    End If
+                    If Not IsMusic And DownloadYTPlaylists Then
+                        maxDate = Nothing
+                        LastDownloadDatePlaylist = nDate(LastDownloadDatePlaylist)
+                        url = $"https://www.youtube.com/{IIf(IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/playlists"
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token,, True, False,, LastDownloadDatePlaylist)
+                        applySpecFolder.Invoke("Playlists", True)
+                        If fillList.Invoke(LastDownloadDatePlaylist) Then LastDownloadDatePlaylist = If(maxDate, Now)
+                    End If
+                Else
+                    Throw New InvalidOperationException($"Media type {YTMediaType} not implemented")
+                End If
+                If list.Count > 0 Then
+                    With list(0)
+                        If Settings.UserSiteNameUpdateEveryTime Or UserSiteName.IsEmptyString Then UserSiteName = .UserTitle
+                        If FriendlyName.IsEmptyString Then FriendlyName = UserSiteName
+                    End With
+                    _TempMediaList.AddRange(list.Select(Function(c) New UserMedia(c)))
+                    _TempPostsList.ListAddList(_TempMediaList.Select(Function(m) m.Post.ID), LNC)
+                    list.Clear()
+                End If
+            Catch ex As Exception
+                ProcessException(ex, Token, "data downloading error")
+            End Try
+        End Sub
+        Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
+            SeparateVideoFolder = False
+            DownloadContentDefault(Token)
+        End Sub
+        Protected Overrides Function DownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile,
+                                                  ByVal Token As CancellationToken) As SFile
+            If Not Media.Object Is Nothing AndAlso TypeOf Media.Object Is IYouTubeMediaContainer Then
+                With DirectCast(Media.Object, YouTubeMediaContainerBase)
+                    Dim f As SFile = .File
+                    f.Path = DestinationFile.Path
+                    If Not IsSingleObjectDownload And Not .FileIsPlaylistObject Then .FileIgnorePlaylist = True
+                    .File = f
+                    If IsSingleObjectDownload Then .Progress = Progress
+                    .Download(YTUseCookies, Token)
+                    If .File.Exists Then Return .File
+                End With
+            End If
+            Return Nothing
+        End Function
+        Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken)
+            _TempMediaList.Add(New UserMedia(Data))
+        End Sub
+#End Region
+#Region "DownloadingException"
+        Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
+                                                          Optional ByVal EObj As Object = Nothing) As Integer
+            Return 0
+        End Function
+#End Region
+#Region "IDisposable Support"
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            If Not disposedValue And disposing Then
+                With _ContentList.Concat(_ContentNew)
+                    If .Count > 0 Then
+                        For Each m As UserMedia In .Self
+                            If Not m.Object Is Nothing AndAlso TypeOf m.Object Is IYouTubeMediaContainer Then DirectCast(m.Object, IYouTubeMediaContainer).Dispose()
+                        Next
+                    End If
+                End With
+            End If
+            MyBase.Dispose(disposing)
+        End Sub
+#End Region
+    End Class
+End Namespace

SCrawler/API/YouTube/UserExchangeOptions.vb

@@ -0,0 +1,33 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin.Attributes
+Namespace API.YouTube
+    Friend Class UserExchangeOptions
+        <PSetting(Caption:="Download videos")>
+        Friend Property DownloadVideos As Boolean
+        <PSetting(Caption:="Download shorts")>
+        Friend Property DownloadShorts As Boolean
+        <PSetting(Caption:="Download playlists")>
+        Friend Property DownloadPlaylists As Boolean
+        <PSetting(Caption:="Use cookies", ToolTip:="Use cookies when downloading data.")>
+        Friend Property UseCookies As Boolean
+        Friend Sub New(ByVal u As UserData)
+            DownloadVideos = u.DownloadYTVideos
+            DownloadShorts = u.DownloadYTShorts
+            DownloadPlaylists = u.DownloadYTPlaylists
+            UseCookies = u.YTUseCookies
+        End Sub
+        Friend Sub New(ByVal s As SiteSettings)
+            DownloadVideos = s.DownloadVideos.Value
+            DownloadShorts = s.DownloadShorts.Value
+            DownloadPlaylists = s.DownloadPlaylists.Value
+            UseCookies = s.UseCookies.Value
+        End Sub
+    End Class
+End Namespace

SCrawler/API/YouTube/YTSettings_Internal.vb

@@ -0,0 +1,38 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML.Base
+Namespace API.YouTube
+    Friend Class YTSettings_Internal : Inherits Base.YouTubeSettings
+        Private DataUpdated As Boolean = False
+        Friend CookiesUpdated As Boolean = False
+        Friend Sub ResetUpdate()
+            XMLValuesEndEdit(Me)
+            DataUpdated = False
+            CookiesUpdated = False
+        End Sub
+        Friend Sub PerformUpdate()
+            If DataUpdated Then
+                MyBase.BeginUpdate()
+                MyBase.Apply()
+                MyBase.EndUpdate()
+            ElseIf CookiesUpdated Then
+                ApplyCookies()
+            End If
+        End Sub
+        Protected Overrides Sub Apply()
+            DataUpdated = True
+        End Sub
+        Protected Overrides Sub BeginUpdate()
+        End Sub
+        Protected Overrides Sub EndUpdate()
+        End Sub
+        Protected Overrides Sub EndEdit()
+        End Sub
+    End Class
+End Namespace

SCrawler/Channels/ChannelViewForm.vb

@@ -313,7 +313,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
             Dim StartIndex% = Settings.Users.Count
             Dim f As SFile
             Dim umo As Boolean = HOST.GetUserMediaOnly
-            Settings.Labels.Add(CannelsLabelName)
+            Settings.Labels.Add(UserData.CannelsLabelName_ChannelsForm)
             Settings.Labels.Add(LabelsKeeper.NoParsedUser)
             Dim rUsers$() = UserBanned(PendingUsers.Select(Function(u) u.ID).ToArray)
             If rUsers.ListExists Then PendingUsers.RemoveAll(Function(u) rUsers.Contains(u))
@@ -336,7 +336,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
                             End With
                             Settings.Users.Add(tmpUser)
                             With Settings.Users.Last
-                                .Labels.Add(CannelsLabelName)
+                                .Labels.Add(UserData.CannelsLabelName_ChannelsForm)
                                 .UpdateUserInformation()
                                 If Settings.FromChannelCopyImageToUser And Not f.IsEmptyString And Not .File.IsEmptyString Then _
                                    CopyFile(ListAddValue(Nothing, New ChannelsCollection.ChannelImage(cn, f)).ListAddList(Settings.Channels.GetUserFiles(.Name), c), .File)
@@ -417,7 +417,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
     Private Async Sub BTT_DOWNLOAD_Click(sender As Object, e As EventArgs) Handles BTT_DOWNLOAD.Click
         Try
             AppendPendingUsers()
-            If Not TokenSource Is Nothing OrElse Not HOST.Source.Available(Plugin.ISiteSettings.Download.Channel, False) Then Exit Sub
+            If Not TokenSource Is Nothing OrElse Not HOST.Source.Available(Plugin.ISiteSettings.Download.Main, False) Then Exit Sub
             Dim InvokeToken As Action = Sub()
                                             If TokenSource Is Nothing Then
                                                 CProgress.Maximum = 0
@@ -471,7 +471,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
         Catch aex As ArgumentException When aex.HelpLink = 1
             ErrorsDescriber.Execute(EDP.ShowAllMsg, aex)
         Catch oex As OperationCanceledException When Token.IsCancellationRequested
-            Dim ee As EDP = EDP.SendInLog
+            Dim ee As EDP = EDP.SendToLog
             If _ShowCancelNotification Then ee += EDP.ShowMainMsg
             ErrorsDescriber.Execute(ee, oex, New MMessage("Downloading operation canceled", "Status...",, MsgBoxStyle.Exclamation))
         Catch ex As Exception
@@ -803,7 +803,7 @@ Friend Class ChannelViewForm : Implements IChannelLimits
         Catch aex As ArgumentException When aex.HelpLink = 1
             ErrorsDescriber.Execute(EDP.LogMessageValue, aex)
         Catch ex As Exception
-            ErrorsDescriber.Execute(EDP.SendInLog, ex, "Post searching error")
+            ErrorsDescriber.Execute(EDP.SendToLog, ex, "Post searching error")
         End Try
         Return p
     End Function

SCrawler/Download/ActiveDownloadingProgress.vb

@@ -72,7 +72,7 @@ Namespace DownloadObjects
                                           Dim s As Size = Size
                                           Dim ss As Size = Screen.PrimaryScreen.WorkingArea.Size
                                           Dim c% = TP_MAIN.RowStyles.Count - 1
-                                          s.Height = c * RowHeight + LowestValue + (PaddingE.GetOf({TP_MAIN}).Vertical(c) / c).RoundDown - c
+                                          s.Height = c * RowHeight + LowestValue + (PaddingE.GetOf({TP_MAIN}).Vertical(c) / c).RoundDown
                                           If s.Height > ss.Height Then s.Height = ss.Height
                                           MinimumSize = Nothing
                                           Size = s

SCrawler/Download/Automation/AutoDownloader.vb

@@ -16,11 +16,6 @@ Imports PersonalUtilities.Tools.Notifications
 Namespace DownloadObjects
     Friend Class AutoDownloader : Inherits GroupParameters : Implements IIndexable, IEContainerProvider
         Friend Event PauseDisabled()
-        Private Shared ReadOnly Property CachePath As SFile
-            Get
-                Return Settings.CachePath
-            End Get
-        End Property
         Friend Enum Modes As Integer
             None = 0
             [Default] = 1
@@ -106,9 +101,9 @@ Namespace DownloadObjects
                                         uif_orig = uif
                                         If uif.Exists Then
                                             uif_compressed = uif
-                                            uif_compressed.Path = CachePath.Path
+                                            uif_compressed.Path = Settings.Cache.RootDirectory.Path
                                             uif_compressed.Name = $"360_{uif.Name}"
-                                            Using imgR As New ImageRenderer(uif, EDP.SendInLog)
+                                            Using imgR As New ImageRenderer(uif, EDP.SendToLog)
                                                 Try : imgR.FitToWidth(360).Save(uif_compressed) : Catch : End Try
                                             End Using
                                             If uif_compressed.Exists Then uif = uif_compressed
@@ -131,7 +126,7 @@ Namespace DownloadObjects
                         End If
                     End If
                 Catch ex As Exception
-                    ErrorsDescriber.Execute(EDP.SendInLog, ex, "[AutoDownloader.NotifiedUser.ShowNotification]")
+                    ErrorsDescriber.Execute(EDP.SendToLog, ex, "[AutoDownloader.NotifiedUser.ShowNotification]")
                     If Not User Is Nothing Then
                         MainFrameObj.ShowNotification(SettingsCLS.NotificationObjects.AutoDownloader,
                                                       User.ToString & vbNewLine &
@@ -151,7 +146,7 @@ Namespace DownloadObjects
                     ElseIf KeySite = _Key Then
                         User.OpenSite()
                     ElseIf Images.ContainsKey(_Key) Then
-                        Images(_Key).Open(, EDP.None)
+                        Images(_Key).Open()
                     End If
                 Else
                     Return True
@@ -453,7 +448,7 @@ Namespace DownloadObjects
                     Thread.Sleep(500)
                 End While
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[AutoDownloader.Checker]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[AutoDownloader.Checker]")
             Finally
                 _StopRequested = False
             End Try
@@ -533,7 +528,7 @@ Namespace DownloadObjects
                     End With
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[AutoDownloader.Download]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[AutoDownloader.Download]")
             Finally
                 Keys.Clear()
                 LastDownloadDate = Now

SCrawler/Download/Automation/AutoDownloaderEditorForm.vb

@@ -20,21 +20,24 @@ Namespace DownloadObjects
             MyDefs = New DefaultFormOptions(Me, Settings.Design)
             MyGroups.ListAddList(Plan.Groups, LAP.NotContainsOnly)
         End Sub
-        Private Class AutomationTimerChecker : Implements IFieldsCheckerProvider
-            Private Property ErrorMessage As String = "The timer value must be greater than 0" Implements IFieldsCheckerProvider.ErrorMessage
-            Private Property Name As String Implements IFieldsCheckerProvider.Name
-            Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
-            Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
-                                     Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+        Private Class AutomationTimerChecker : Inherits FieldsCheckerProviderBase
+            Public Overrides Property ErrorMessage As String
+                Get
+                    Return "The timer value must be greater than 0"
+                End Get
+                Set(ByVal msg As String)
+                    MyBase.ErrorMessage = msg
+                End Set
+            End Property
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
                 If CInt(AConvert(Of Integer)(Value, -10)) > 0 Then
                     Return Value
                 Else
+                    HasError = True
                     Return Nothing
                 End If
             End Function
-            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
-                Throw New NotImplementedException("[GetFormat] is not available in the context of [AutoDownloaderEditorForm]")
-            End Function
         End Class
         Private Sub AutoDownloaderEditorForm_Load(sender As Object, e As EventArgs) Handles Me.Load
             With MyDefs

SCrawler/Download/Automation/AutoDownloaderPauseButtons.vb

@@ -138,7 +138,7 @@ Namespace DownloadObjects
                                                                         BTT_PAUSE_UNLIMITED, BTT_PAUSE_DISABLE}
 
                     If UpdateBase Then UpdateBaseButton(Not p = PauseModes.Disabled)
-                    If Not VerifyAll OrElse Settings.Automation.All(Function(ByVal plan As AutoDownloader)
+                    If Not VerifyAll OrElse Settings.Automation.All(Function(ByVal plan As AutoDownloader) As Boolean
                                                                         If plan.Mode = AutoDownloader.Modes.None Then
                                                                             Return True
                                                                         Else
@@ -166,7 +166,7 @@ Namespace DownloadObjects
                     cntList.Clear()
                 End With
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[MainFrame.UpdatePauseButtons]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[MainFrame.UpdatePauseButtons]")
             Finally
                 Select Case Place
                     Case ButtonsPlace.MainFrame : TrayButtons.UpdatePauseButtons(True, True)

SCrawler/Download/Automation/Scheduler.vb

@@ -7,9 +7,9 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports SCrawler.DownloadObjects.Groups
-Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Tools
+Imports PersonalUtilities.Functions.XML
+Imports SCrawler.DownloadObjects.Groups
 Imports PauseModes = SCrawler.DownloadObjects.AutoDownloader.PauseModes
 Namespace DownloadObjects
     Friend Class Scheduler : Implements IEnumerable(Of AutoDownloader), IMyEnumerator(Of AutoDownloader), IDisposable

SCrawler/Download/Automation/SchedulerEditorForm.vb

@@ -75,7 +75,7 @@ Namespace DownloadObjects
                     _RefillInProgress = False
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[DownloadObjects.SchedulerEditorForm.Refill]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadObjects.SchedulerEditorForm.Refill]")
             End Try
         End Sub
 #Region "Add, Edit, Delete"

SCrawler/Download/DownloadProgress.vb

@@ -24,6 +24,7 @@ Namespace DownloadObjects
         Private WithEvents BTT_OPEN As Button
         Private ReadOnly PR_MAIN As ProgressBar
         Private ReadOnly LBL_INFO As Label
+        Private ReadOnly Icon As PictureBox
 #End Region
         Private ReadOnly Property Instance As API.Base.ProfileSaved
         Friend ReadOnly Property Job As TDJob
@@ -38,7 +39,21 @@ Namespace DownloadObjects
             TP_CONTROLS = New TableLayoutPanel With {.Margin = New Padding(0), .Dock = DockStyle.Fill}
             PR_MAIN = New ProgressBar With {.Dock = DockStyle.Fill}
             LBL_INFO = New Label With {.Text = String.Empty, .Dock = DockStyle.Fill}
+            Icon = New PictureBox With {
+                .SizeMode = PictureBoxSizeMode.Zoom,
+                .Dock = DockStyle.Fill,
+                .Margin = New Padding(3),
+                .Padding = New Padding(3)
+            }
             CreateButton(BTT_STOP, My.Resources.DeletePic_24)
+            Dim img As Image = Nothing
+            If Not _Job.Host Is Nothing Then
+                With Job.Host.Source
+                    If Not .Icon Is Nothing Then img = .Icon.ToBitmap
+                    If img Is Nothing AndAlso Not .Image Is Nothing Then img = .Image
+                End With
+            End If
+            If Not img Is Nothing Then Icon.Image = img : Icon.InitialImage = img
 
             If Job.Type = Download.Main Then
                 LBL_INFO.Margin = New Padding(3)
@@ -48,6 +63,7 @@ Namespace DownloadObjects
                     .RowCount = 1
                 End With
                 With TP_CONTROLS
+                    .ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 30))
                     .ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 30))
                     .ColumnStyles.Add(New ColumnStyle(SizeType.Absolute, 150))
                     .ColumnStyles.Add(New ColumnStyle(SizeType.Percent, 100))
@@ -55,9 +71,10 @@ Namespace DownloadObjects
                     .RowStyles.Add(New RowStyle(SizeType.Percent, 100))
                     .RowCount = 1
                     With .Controls
-                        .Add(BTT_STOP, 0, 0)
-                        .Add(PR_MAIN, 1, 0)
-                        .Add(LBL_INFO, 2, 0)
+                        If Not img Is Nothing Then .Add(Icon, 0, 0)
+                        .Add(BTT_STOP, 1, 0)
+                        .Add(PR_MAIN, 2, 0)
+                        .Add(LBL_INFO, 3, 0)
                     End With
                 End With
                 TP_MAIN.Controls.Add(TP_CONTROLS, 0, 0)
@@ -71,16 +88,18 @@ Namespace DownloadObjects
                         .Add(New ColumnStyle(SizeType.Absolute, 30))
                         .Add(New ColumnStyle(SizeType.Absolute, 30))
                         .Add(New ColumnStyle(SizeType.Absolute, 30))
+                        .Add(New ColumnStyle(SizeType.Absolute, 30))
                         .Add(New ColumnStyle(SizeType.Percent, 100))
                     End With
-                    .ColumnCount = 4
+                    .ColumnCount = .ColumnStyles.Count
                     .RowStyles.Add(New RowStyle(SizeType.Percent, 50))
                     .RowCount = 1
                     With .Controls
-                        .Add(BTT_START, 0, 0)
-                        .Add(BTT_STOP, 1, 0)
-                        .Add(BTT_OPEN, 2, 0)
-                        .Add(PR_MAIN, 3, 0)
+                        If Not img Is Nothing Then .Add(Icon, 0, 0)
+                        .Add(BTT_START, 1, 0)
+                        .Add(BTT_STOP, 2, 0)
+                        .Add(BTT_OPEN, 3, 0)
+                        .Add(PR_MAIN, 4, 0)
                     End With
                 End With
                 With TP_MAIN
@@ -129,11 +148,13 @@ Namespace DownloadObjects
         End Sub
 #End Region
 #Region "Start, Stop"
-        Friend Sub Start()
-            Job.Start(AddressOf DownloadData)
+        Private _IsMultiple As Boolean = False
+        Friend Sub Start(Optional ByVal Multiple As Boolean = False)
+            _IsMultiple = Multiple
+            Job.StartThread(AddressOf DownloadData)
         End Sub
         Friend Sub [Stop]()
-            Job.Stop()
+            Job.Cancel()
         End Sub
 #End Region
 #Region "SavedPosts downloading"
@@ -144,15 +165,16 @@ Namespace DownloadObjects
                 btte.Invoke(BTT_STOP, True)
                 Job.Progress.InformationTemporary = $"{Job.Host.Name} downloading started"
                 Job.Start()
-                Instance.Download(Job.Token)
+                Instance.Download(Job.Token, _IsMultiple)
                 RaiseEvent DownloadDone(SettingsCLS.NotificationObjects.SavedPosts, $"Downloading saved {Job.Host.Name} posts is completed")
             Catch ex As Exception
                 Job.Progress.InformationTemporary = $"{Job.Host.Name} downloading error"
                 ErrorsDescriber.Execute(EDP.LogMessageValue, ex, {$"{Job.Host.Name} saved posts downloading error", "Saved posts"})
             Finally
+                _IsMultiple = False
                 btte.Invoke(BTT_START, True)
                 btte.Invoke(BTT_STOP, False)
-                Job.Stopped()
+                Job.Finish()
                 If Job.Type = Download.SavedPosts Then Job.Progress.Maximum = 0 : Job.Progress.Value = 0
             End Try
         End Sub
@@ -173,6 +195,7 @@ Namespace DownloadObjects
                     If Not BTT_START Is Nothing Then BTT_START.Dispose()
                     If Not BTT_STOP Is Nothing Then BTT_STOP.Dispose()
                     If Not BTT_OPEN Is Nothing Then BTT_OPEN.Dispose()
+                    If Not Icon Is Nothing Then Icon.Dispose()
                     PR_MAIN.Dispose()
                     LBL_INFO.Dispose()
                     TP_CONTROLS.Controls.Clear()

SCrawler/Download/DownloadSavedPostsForm.vb

@@ -60,7 +60,7 @@ Friend Class DownloadSavedPostsForm
         MyView.Dispose(Settings.Design)
     End Sub
     Private Sub [Start]() Handles BTT_DOWN_ALL.Click
-        If JobsList.Count > 0 Then JobsList.ForEach(Sub(j) j.Start())
+        If JobsList.Count > 0 Then JobsList.ForEach(Sub(j) j.Start(True))
     End Sub
     Friend Sub [Stop]() Handles BTT_STOP_ALL.Click
         If JobsList.Count > 0 Then JobsList.ForEach(Sub(j) j.Stop())

SCrawler/Download/DownloadedInfoForm.vb

@@ -126,7 +126,7 @@ Namespace DownloadObjects
                     _LatestSelected = -1
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[DownloadedInfoForm.RefillList]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadedInfoForm.RefillList]")
             Finally
                 UpdateNavigationButtons(Nothing)
             End Try

SCrawler/Download/Feed/DownloadFeedForm.Designer.vb

@@ -99,6 +99,7 @@ Namespace DownloadObjects
             'TP_DATA
             '
             Me.TP_DATA.AutoScroll = True
+            Me.TP_DATA.AutoSizeMode = System.Windows.Forms.AutoSizeMode.GrowAndShrink
             Me.TP_DATA.ColumnCount = 1
             Me.TP_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
             Me.TP_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 20.0!))

SCrawler/Download/Feed/DownloadFeedForm.vb

@@ -213,7 +213,7 @@ Namespace DownloadObjects
                 End If
                 If fList.ListExists Then fList.Clear()
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[DownloadObjects.DownloadFeedForm.SessionChooser({GetLast})]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[DownloadObjects.DownloadFeedForm.SessionChooser({GetLast})]")
             End Try
         End Sub
 #End Region
@@ -242,7 +242,7 @@ Namespace DownloadObjects
                                                        If d > 0 Then
                                                            .AutoScroll = False
                                                            .AutoScroll = True
-                                                           If LatestScrollValue.HasValue Then TP_DATA.VerticalScroll.Value = LatestScrollValue.Value
+                                                           SetScrollValue(False)
                                                            .PerformLayout()
                                                            LatestScrollValueDisabled = False
                                                        End If
@@ -302,7 +302,7 @@ Namespace DownloadObjects
                     If HeightChanged Then
                         TP_DATA.AutoScroll = False
                         TP_DATA.AutoScroll = True
-                        If LatestScrollValue.HasValue Then TP_DATA.VerticalScroll.Value = LatestScrollValue.Value
+                        SetScrollValue(False)
                         TP_DATA.PerformLayout()
                         LatestScrollValueDisabled = False
                     End If
@@ -331,10 +331,7 @@ Namespace DownloadObjects
                     DirectCast(MyRange.Switcher, RangeSwitcher(Of UserMediaD)).PerformIndexChanged()
                     If Not indxChanged Then
                         LatestScrollValueDisabled = False
-                        If LatestScrollValue.HasValue Then
-                            TP_DATA.VerticalScroll.Value = LatestScrollValue.Value
-                            TP_DATA.PerformLayout()
-                        End If
+                        SetScrollValue(True)
                     End If
                 End If
                 .HandlersSuspended = False
@@ -415,7 +412,7 @@ Namespace DownloadObjects
                     RefillInProgress = False
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[DownloadObjects.DownloadFeedForm.Range.IndexChanged({Sender.CurrentIndex})]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[DownloadObjects.DownloadFeedForm.Range.IndexChanged({Sender.CurrentIndex})]")
                 RefillInProgress = False
             Finally
                 If Not RefillInProgress AndAlso Sender.CurrentIndex >= 0 Then
@@ -495,8 +492,7 @@ Namespace DownloadObjects
             If LatestScrollValue.HasValue Then ControlInvoke(TP_DATA, Sub()
                                                                           Dim b As Boolean = ScrollSuspended
                                                                           If Not b Then ScrollSuspended = True
-                                                                          TP_DATA.VerticalScroll.Value = LatestScrollValue.Value
-                                                                          TP_DATA.PerformLayout()
+                                                                          SetScrollValue(True)
                                                                           If Not b Then ScrollSuspended = False
                                                                       End Sub)
         End Sub
@@ -515,6 +511,26 @@ Namespace DownloadObjects
                                        End Sub)
             End If
         End Sub
+        Private Sub TP_DATA_StyleChanged(sender As Object, e As EventArgs) Handles TP_DATA.StyleChanged
+            ControlInvokeFast(TP_DATA, Sub()
+                                           With TP_DATA
+                                               .Padding = New Padding(0, 0, .VerticalScroll.Visible.BoolToInteger * 3, 0)
+                                               .HorizontalScroll.Visible = False
+                                               .HorizontalScroll.Enabled = False
+                                               .PerformLayout()
+                                           End With
+                                       End Sub, EDP.None)
+        End Sub
+        Private Sub SetScrollValue(ByVal Perform As Boolean)
+            With TP_DATA
+                If LatestScrollValue.HasValue Then
+                    If LatestScrollValue.Value < 0 Then LatestScrollValue = 0
+                    If LatestScrollValue.Value > .VerticalScroll.Maximum Then LatestScrollValue = .VerticalScroll.Maximum
+                    .VerticalScroll.Value = LatestScrollValue.Value
+                    If Perform Then .PerformLayout()
+                End If
+            End With
+        End Sub
 #End Region
         Private Sub ClearTable()
             ControlInvoke(TP_DATA, Sub()

SCrawler/Download/Feed/FeedMedia.vb

@@ -53,7 +53,7 @@ Namespace DownloadObjects
                 End If
             End Set
         End Property
-        Private ReadOnly Property ObjectsPaddingHeight
+        Private ReadOnly Property ObjectsPaddingHeight As Integer
             Get
                 Return TP_MAIN.RowStyles(0).Height + PaddingE.GetOf({TP_MAIN}).Vertical(2)
             End Get
@@ -205,7 +205,7 @@ Namespace DownloadObjects
             Catch tex As Threading.ThreadStateException
                 HasError = True
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"[DownloadObjects.FeedMedia({File})]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"[DownloadObjects.FeedMedia({File})]")
                 HasError = True
             End Try
         End Sub
@@ -235,7 +235,7 @@ Namespace DownloadObjects
 #End Region
 #Region "Context"
         Private Sub BTT_CONTEXT_OPEN_MEDIA_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_MEDIA.Click
-            File.Open(, EDP.None)
+            File.Open()
         End Sub
         Private Sub BTT_CONTEXT_OPEN_USER_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_USER.Click
             If Not UserKey.IsEmptyString Then

SCrawler/Download/Feed/FeedVideo.vb

@@ -54,8 +54,8 @@ Namespace DownloadObjects
                 MyVideo.MediaPlayer = MediaPlayer
                 TR_VOLUME.Value = MediaPlayer.Volume / 10
                 If Settings.UseM3U8 Then
-                    Dim f As SFile = $"{Settings.CachePath.PathWithSeparator}FeedSnapshots\{File.GetHashCode}.png"
-                    If Not f.Exists Then f = FFMPEG.TakeSnapshot(File, f, Settings.FfmpegFile, TimeSpan.FromSeconds(1))
+                    Dim f As SFile = $"{Settings.Cache.RootDirectory.PathWithSeparator}FeedSnapshots\{File.GetHashCode}.png"
+                    If Not f.Exists Then f = FFMPEG.TakeSnapshot(File, f, Settings.FfmpegFile.File, TimeSpan.FromSeconds(1))
                     If f.Exists Then
                         MyImage = New ImageRenderer(f, EDP.None)
                         Try
@@ -64,14 +64,14 @@ Namespace DownloadObjects
                                 MyVideo.BackgroundImageLayout = ImageLayout.Zoom
                             End If
                         Catch img_set_ex As Exception
-                            ErrorsDescriber.Execute(EDP.SendInLog, img_set_ex, "Error setting background image for media player." & vbCr &
+                            ErrorsDescriber.Execute(EDP.SendToLog, img_set_ex, "Error setting background image for media player." & vbCr &
                                                                                $"File: {File}{vbCr}Image: {f}")
                         End Try
                     End If
                 End If
                 UpdateButtons()
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"Media player initialization error({File})")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"Media player initialization error({File})")
                 HasError = True
             End Try
         End Sub
@@ -173,7 +173,7 @@ Namespace DownloadObjects
             Catch oex As OperationCanceledException
                 MyMainLOG = $"Cannot perform action [{ActionName}] on file [{MediaFile}]"
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, $"An error occurred while performing action [{ActionName}] on file [{MediaFile}]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, $"An error occurred while performing action [{ActionName}] on file [{MediaFile}]")
             End Try
         End Function
     End Class

SCrawler/Download/Groups/DownloadGroup.vb

@@ -171,7 +171,7 @@ Namespace DownloadObjects.Groups
                     Return Nothing
                 End If
             Catch ex As Exception
-                Return ErrorsDescriber.Execute(EDP.SendInLog, ex, "[DownloadGroup.GetUsers]")
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadGroup.GetUsers]")
             End Try
         End Function
 #End Region
@@ -187,7 +187,7 @@ Namespace DownloadObjects.Groups
                     End If
                 End If
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendInLog, ex, "[DownloadGroup.DownloadUsers]")
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadGroup.DownloadUsers]")
             End Try
         End Sub
 #End Region