2023.4.28.0

Plugins
IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function
Add IDownloadableMedia interface
Removed 'Channel' option from all functions and enums
ISiteSettings: added GetSingleMediaInstance function
ExchangeOptions: removed 'IsChannel'
UserMediaTypes: added Audio and AudioPre enums
IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums

SCrawler
Add YouTube standalone downloader
Add gallery-dl & yt-dlp support
Remove 'UserInfo' requirement from 'ProfilesSaved'
Update 'SiteSettingsBase' to use domains and Netscape cookies
UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download

Rewritten DomainsContainer
Create a universal settings form and PSettingsArttribute
Gfycat, Imgur: turn these classes into IUserData to download a single object

All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist

LPSG: fix attachments; update converters and regex
Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music
Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support
Twitter: fixed gif path bug; fixed downloading saved posts
PornHub: hide unnecessary errors; photo galleries bug
RedGifs: add 'UserAgent' option

Added icons to download progress

Rename some objects
Completely redesigned standalone downloader form and rewritten its environment
WebClient2: update to use tokens

Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser'

UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found
Remove temporary 'EncryptCookies' module

MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader)
PropertyValueHost: update for option forms compatibility
SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork
UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function
Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler
UserFinder: remove old 'Merge' const; remove channel option
UserInfo: remove channel option

SCrawler.YouTube/Objects/Channel.vb

@@ -0,0 +1,98 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Forms.Toolbars
+Namespace API.YouTube.Objects
+    Public Class Channel : Inherits YouTubeMediaContainerBase
+        Public Sub New()
+            ObjectType = Base.YouTubeMediaType.Channel
+        End Sub
+        Public Overrides Function ToString(ByVal ForMediaItem As Boolean) As String
+            Return UserTitle
+        End Function
+        Public Overrides Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean,
+                                        Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean
+            _MediaType = IIf(IsMusic, Plugin.UserMediaTypes.Audio, Plugin.UserMediaTypes.Video)
+            _ObjectType = Base.YouTubeMediaType.Channel
+            Me.IsMusic = IsMusic
+            If ParseFiles(Path, IsMusic, Token, Progress) Then
+                PlaylistID = String.Empty
+                PlaylistIndex = 0
+                PlaylistTitle = String.Empty
+                ThrowAny(Token)
+
+                'Reconfiguration
+                If IsMusic AndAlso HasElements AndAlso Elements.Exists(Function(e) Not e.PlaylistID.IsEmptyString) Then
+                    Dim elems As New List(Of IYouTubeMediaContainer)(Elements)
+                    Dim elemsNew As New List(Of IYouTubeMediaContainer)
+                    Dim playlistDic As New Dictionary(Of String, List(Of IYouTubeMediaContainer))
+                    Elements.Clear()
+                    For Each elem In elems
+                        If Not elem.PlaylistTitle.IsEmptyString Then
+                            If Not playlistDic.ContainsKey(elem.PlaylistTitle) Then playlistDic.Add(elem.PlaylistTitle, New List(Of IYouTubeMediaContainer))
+                            playlistDic(elem.PlaylistTitle).Add(elem)
+                        ElseIf elem.PlaylistID = elem.UserID Then
+                            elem.PlaylistID = String.Empty
+                            elem.PlaylistIndex = -1
+                            elem.PlaylistTitle = String.Empty
+                            elemsNew.Add(elem)
+                        Else
+                            elemsNew.Add(elem)
+                        End If
+                    Next
+                    If playlistDic.Count > 0 Then
+                        Dim i%, ii%
+                        Dim v As YouTubeMediaContainerBase
+                        For Each kv In playlistDic
+                            i = -1
+                            If elemsNew.Count > 0 Then i = elemsNew.FindIndex(Function(e) e.PlaylistID = kv.Key)
+                            If i = -1 Then
+                                elemsNew.Add(New PlayList)
+                                v = kv.Value.First
+                                With DirectCast(elemsNew.Last, YouTubeMediaContainerBase)
+                                    .ObjectType = Base.YouTubeMediaType.PlayList
+                                    .MediaType = v.MediaType
+                                    .IsMusic = v.IsMusic
+                                    .ID = v.PlaylistID
+                                    .Title = v.PlaylistTitle
+                                    .PlaylistID = .ID
+                                    .PlaylistTitle = .Title
+                                    .PlaylistIndex = -1
+                                    .UserID = v.UserID
+                                    .UserTitle = v.UserTitle
+                                End With
+                                i = elemsNew.Count - 1
+                            End If
+                            With elemsNew(i).Elements
+                                .AddRange(kv.Value)
+                                If .Count > 0 Then
+                                    For ii = 0 To .Count - 1
+                                        With DirectCast(.Item(ii), YouTubeMediaContainerBase)
+                                            .PlaylistIndex = ii + 1
+                                            .PlaylistCount = kv.Value.Count
+                                        End With
+                                    Next
+                                End If
+                            End With
+                        Next
+                        playlistDic.Clear()
+                    End If
+                    If elemsNew.Count > 0 Then Elements.AddRange(elemsNew)
+                    elems.Clear()
+                    elemsNew.Clear()
+                    File = MyYouTubeSettings.OutputPath
+                End If
+
+                Return True
+            Else
+                Return False
+            End If
+        End Function
+    End Class
+End Namespace

SCrawler.YouTube/Objects/IYouTubeMediaContainer.vb

@@ -0,0 +1,88 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
+Imports SCrawler.API.YouTube.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Functions.XML.Base
+Imports PersonalUtilities.Forms.Toolbars
+Imports UMTypes = SCrawler.Plugin.UserMediaTypes
+Imports UMStates = SCrawler.Plugin.UserMediaStates
+Namespace API.YouTube.Objects
+    Public Interface IYouTubeMediaContainer : Inherits IDownloadableMedia, IEContainerProvider, IComparable(Of IYouTubeMediaContainer)
+#Region "Events"
+        Event FileDownloaded As EventHandler
+        Event FileDownloadStarted As EventHandler
+        Event DataDownloaded As EventHandler
+#End Region
+#Region "Base data"
+        ReadOnly Property ObjectType As YouTubeMediaType
+        ReadOnly Property MediaType As UMTypes
+        ReadOnly Property MediaState As UMStates
+        Property IsMusic As Boolean
+        Property ID As String
+        Property Description As String
+        Property PlaylistID As String
+        Property PlaylistTitle As String
+        Property UserID As String
+        Property UserTitle As String
+#End Region
+#Region "Playlist support"
+        ReadOnly Property Elements As List(Of IYouTubeMediaContainer)
+        ReadOnly Property HasElements As Boolean
+        ReadOnly Property Count As Integer
+        Property PlaylistIndex As Integer
+#End Region
+#Region "Data info"
+#Region "Thumbnails"
+        ReadOnly Property Thumbnails As List(Of Thumbnail)
+        ReadOnly Property ThumbnailUrlMedia As String
+        Overloads ReadOnly Property ThumbnailFile As SFile
+#End Region
+#Region "Subtitles"
+        ReadOnly Property Subtitles As List(Of Subtitles)
+        ReadOnly Property SubtitlesSelectedIndexes As List(Of Integer)
+#End Region
+#Region "MediaObjects"
+        ReadOnly Property MediaObjects As List(Of MediaObject)
+        Property SelectedAudioIndex As Integer
+        Property SelectedVideoIndex As Integer
+#End Region
+        ReadOnly Property SizeStr As String
+        Property Height As Integer
+        Property Bitrate As Integer
+        Property DateCreated As Date
+        Property DateAdded As Date
+        Property DateDownloaded As Date
+        Property OutputVideoExtension As String
+        Property OutputAudioCodec As String
+        Property OutputSubtitlesFormat As String
+#End Region
+#Region "HasError, Exists, Checked"
+        ReadOnly Property CheckState As CheckState
+#End Region
+#Region "URL, File, Command"
+        Overloads Property File As SFile
+        ReadOnly Property Files As List(Of SFile)
+        Property UseCookies As Boolean
+        ReadOnly Property Command(ByVal WithCookies As Boolean) As String
+        Sub UpdateInfoFields()
+#End Region
+#Region "Download"
+        Overloads Property Progress As MyProgress
+#End Region
+#Region "Parse, Load"
+        Overloads Sub Load(ByVal f As SFile)
+        Overloads Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean,
+                                 Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean
+#End Region
+#Region "IDisposable"
+        ReadOnly Property IsDisposed As Boolean
+#End Region
+    End Interface
+End Namespace

SCrawler.YouTube/Objects/PlayList.vb

@@ -0,0 +1,43 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Forms.Toolbars
+Namespace API.YouTube.Objects
+    Public Class PlayList : Inherits YouTubeMediaContainerBase
+        Public Sub New()
+            _ObjectType = Base.YouTubeMediaType.PlayList
+        End Sub
+        Public Overrides Function ToString(ByVal ForMediaItem As Boolean) As String
+            Dim t$ = String.Empty
+            Dim s$ = SizeStr
+            Dim __title$ = $" - {Title}"
+            If Not s.IsEmptyString Then s = $" [{s}]"
+            If Not PlaylistTitle.IsEmptyString And Not ForMediaItem Then t = $"{PlaylistTitle} - "
+            If IsMusic Then
+                If Count <= 1 Then t &= "Single" Else t &= "Album"
+            Else
+                t &= "Playlist"
+            End If
+            If Not PlaylistTitle.IsEmptyString And Not ForMediaItem Then t &= $" - {PlaylistTitle}"
+            If PlaylistTitle = Title Then __title = String.Empty
+            If ForMediaItem Then
+                Return $"{t} ({Count}){__title}"
+            Else
+                Return $"{t} ({Count}){__title} ({AConvert(Of String)(Duration, TimeToStringProvider)}){s}"
+            End If
+        End Function
+        Public Overrides Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean,
+                                        Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean
+            _MediaType = IIf(IsMusic, Plugin.UserMediaTypes.Audio, Plugin.UserMediaTypes.Video)
+            _ObjectType = Base.YouTubeMediaType.PlayList
+            Me.IsMusic = IsMusic
+            Return ParseFiles(Path, IsMusic, Token, Progress)
+        End Function
+    End Class
+End Namespace

SCrawler.YouTube/Objects/Track.vb

@@ -0,0 +1,61 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Forms.Toolbars
+Namespace API.YouTube.Objects
+    Public Class Track : Inherits YouTubeMediaContainerBase
+        Public Sub New()
+            IsMusic = True
+            ObjectType = Base.YouTubeMediaType.Single
+        End Sub
+        Protected Overrides Sub GenerateFileName()
+            If Not FileSetManually Or _File.IsEmptyString Then
+                Dim indx$ = String.Empty
+                If PlaylistIndex > 0 Then indx = PlaylistIndex.NumToString(ANumbers.Formats.NumberGroup, PlaylistCount.ToString.Length)
+                If Not indx.IsEmptyString Then indx &= ". "
+                _File.Name = $"{indx}{Title}"
+                If Not OutputAudioCodec.IsEmptyString Then
+                    _File.Extension = OutputAudioCodec.StringToLower
+                ElseIf Not MyYouTubeSettings.DefaultAudioCodecMusic.IsEmptyString Then
+                    _File.Extension = MyYouTubeSettings.DefaultAudioCodecMusic.Value.StringToLower
+                Else
+                    _File.Extension = mp3
+                End If
+            End If
+        End Sub
+        Public Overrides Function ToString(ByVal ForMediaItem As Boolean) As String
+            Dim s$ = SizeStr
+            If Not s.IsEmptyString Then s = $" [{s}]"
+            Dim pls$ = String.Empty
+            If PlaylistIndex > 0 Then pls = $"{PlaylistIndex.NumToString(ANumbers.Formats.NumberGroup, PlaylistCount.ToString.Length)}. "
+            If ForMediaItem Then
+                Return Title
+            Else
+                Return $"{pls}{Title} ({AConvert(Of String)(Duration, TimeToStringProvider)}){s}"
+            End If
+        End Function
+        Public Overrides Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean,
+                                        Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean
+            _MediaType = Plugin.UserMediaTypes.Audio
+            _ObjectType = Base.YouTubeMediaType.Single
+            Me.IsMusic = IsMusic
+            If MyBase.Parse(Container, Path, IsMusic, Token, Progress) Then
+                Dim f As SFile = MyYouTubeSettings.OutputPath
+                If f.IsEmptyString Then f = "YouTubeDownloads\OutputFile.mp3"
+                Dim ext$ = MyYouTubeSettings.DefaultAudioCodec.Value.StringToLower
+                If ext.IsEmptyString Then ext = "mp3"
+                f.Extension = ext
+                File = f
+                Return True
+            Else
+                Return False
+            End If
+        End Function
+    End Class
+End Namespace

SCrawler.YouTube/Objects/Video.vb

@@ -0,0 +1,29 @@
+' Copyright (C) 2023  Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.YouTube.Base
+Imports PersonalUtilities.Functions.XML
+Imports PersonalUtilities.Forms.Toolbars
+Namespace API.YouTube.Objects
+    Public Class Video : Inherits YouTubeMediaContainerBase
+        Public Sub New()
+            _ObjectType = YouTubeMediaType.Single
+            _MediaType = Plugin.UserMediaTypes.Video
+        End Sub
+        Public Overrides Function ToString(ByVal ForMediaItem As Boolean) As String
+            Return Title
+        End Function
+        Public Overrides Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean,
+                                        Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean
+            _MediaType = Plugin.UserMediaTypes.Video
+            _ObjectType = YouTubeMediaType.Single
+            Me.IsMusic = IsMusic
+            Return MyBase.Parse(Container, Path, IsMusic, Token, Progress)
+        End Function
+    End Class
+End Namespace