2024.1.26.0

YT
YouTubeSettings: add property DefaultVideoFPS
VideoOptionsForm, YouTubeMediaContainerBase: add FPS reduction

SCrawler
API.Instagram: change back aspect ratio determining
API.TikTok: add the ability to use a regex to clean the title
API.YouTube: add the ability to ignore community errors

Changelog.md

@@ -1,3 +1,76 @@
+# 2024.1.26.0
+
+*2024-01-26*
+
+- Added
+  - YouTube (standalone app): **the ability to reduce video FPS**
+  - TikTok: the ability to use a regex to clean the title
+  - YouTube (SCrawler): the ability to ignore community errors
+- Fixed
+  - Instagram: stories (user) downloading with the wrong aspect ratio for some users
+  - Minor bugs
+
+# 2024.1.20.0
+
+*2024-01-20*
+
+- Added
+  - Instagram: **the ability to download reels**
+  - LPSG: handle 404 error
+
+# 2024.1.18.0
+
+*2024-01-18*
+
+- Fixed
+  - Main window: incorrect collection sorting
+  - xHamster: some user videos were not downloaded
+  - YouTube (standalone app): URL array form doesn't show scrollbars
+  - Minor bugs
+
+# 2024.1.12.1
+
+*2024-01-12*
+
+- Added
+  - YouTube (SCrawler): data downloading by dates
+  - Feed: ability to merge multiple session feeds into one
+  - Feed: remove session number from special feeds
+- Fixed
+  - **Instagram**: stories (user) downloading with the wrong aspect ratio for some users
+  - YouTube: incorrect opening of a post from the feed
+  - YouTube: wrong date to data parsing
+
+# 2024.1.12.0
+
+*2024-01-12*
+
+- Added
+  - Feed: added a prompt before clearing the current session
+  - xHamster: creators
+  - YouTube communities: add error to log
+  - Added scheduler to tray menu
+  - Other improvements
+- Fixed
+  - Feed: there is no option to create a new feed when adding checked items
+  - **Instagram**: downloading of tagged posts
+  - xHamster: profiles are not downloading
+  - Minor bugs
+
+# 2023.12.27.0
+
+*2023-12-27*
+
+- Added
+  - Notification of new log data
+  - OnlyFans: **OF-Scrapper support to download DRM protected videos**
+  - Other improvements
+- Fixed
+  - The default options are changed (`Favorite`, `Temporary`, etc.) when changing an account for a created user
+  - When changing the account for a created user, the new account does not apply to that user until SCrawler is restarted
+  - Saved posts: session file is not updated when new data is added
+  - Minor bugs
+
 # 2023.12.15.0
 
 *2023-12-15*

ProgramsComparison.md

@@ -6,11 +6,19 @@ https://github.com/yt-dlp/yt-dlp/
 
 SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
 
+# gallery-dl
+
+https://github.com/mikf/gallery-dl
+
+**Great powerful CLI tool that supports hundreds of sites.**
+
+SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
+
 # 4K Video Downloader
 
 https://www.4kdownload.com/-plbrz/video-downloader
 
-| Option | SCrawler | 4K Stogram |
+| Option | SCrawler | 4K Video Downloader |
 | ---- | ---- | ---- |
 | User managament | **Advanced** | No |
 | Automatic downloads | **Yes** | No |
@@ -121,10 +129,3 @@ https://github.com/RipMeApp/ripme
 | Other sites support | **Yes** | No |
 | Still supported | **Yes** | **No (last release date May 4, 2021)** |
 
-# gallery-dl
-
-https://github.com/mikf/gallery-dl
-
-**CLI tool**
-
-SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.

SCrawler.YouTube/Base/YouTubeSettings.vb

@@ -15,6 +15,7 @@ Imports PersonalUtilities.Forms
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.XML.Objects
+Imports PersonalUtilities.Functions.XML.Attributes
 Imports PersonalUtilities.Functions.XML.Attributes.Specialized
 Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Grid.Base
@@ -275,6 +276,71 @@ Namespace API.YouTube.Base
         <Browsable(True), GridVisible, XMLVN({"DefaultsVideo"}), Category("Defaults Video"), DisplayName("Include zero size formats"),
             Description("Include formats with zero size (or undefined size).")>
         Public ReadOnly Property DefaultVideoIncludeNullSize As XMLValue(Of Boolean)
+        <Browsable(False), XMLV("DefaultVideoFPS", {"DefaultsVideo"}, -1)>
+        Private ReadOnly Property DefaultVideoFPS_XML As XMLValue(Of Double)
+        <Browsable(True), GridVisible, Category("Defaults Video"), DisplayName("Defaults Video FPS"),
+            Description("Set default video FPS (only to reduce video FPS). Default: -1 (disabled)."),
+            TypeConverter(GetType(FieldsTypeConverter)), GridFormatProvider(GetType(FpsFormatProvider))>
+        Public Property DefaultVideoFPS As Double
+            Get
+                Return DefaultVideoFPS_XML
+            End Get
+            Set(ByVal fps As Double)
+                DefaultVideoFPS_XML.Value = fps
+            End Set
+        End Property
+        Private Function ShouldSerializeDefaultVideoFPS() As Boolean
+            Return DefaultVideoFPS <> DefaultVideoFPS_XML.Value
+        End Function
+        Private Sub ResetDefaultVideoFPS()
+            DefaultVideoFPS = -1
+        End Sub
+        Friend Class FpsFormatProvider : Implements IGridConversionProvider
+            Private Property Converter As TypeConverter Implements IGridConversionProvider.Converter
+            Private Property Context As ITypeDescriptorContext Implements IGridConversionProvider.Context
+            Private Property DataType As Type Implements IGridConversionProvider.DataType
+            Private Property Instance As Object Implements IGridConversionProvider.Instance
+            Friend Shared ReadOnly Property MyProviderDefault As ANumbers
+                Get
+                    Return New ANumbers(ANumbers.Cultures.Primitive) With {.DecimalDigits = 5, .TrimDecimalDigits = True}
+                End Get
+            End Property
+            Friend Const ErrorMessageDefault As String = "The fps value must be a number"
+            Private ReadOnly MyProvider As ANumbers = MyProviderDefault
+            Friend Function ToObject(ByVal Context As ITypeDescriptorContext, ByVal Culture As CultureInfo, ByVal Value As Object) As Object Implements IGridConversionProvider.ToObject
+                Return AConvert(Of Double)(Value, MyProvider, -1)
+            End Function
+            Friend Overloads Function ToString(ByVal Context As ITypeDescriptorContext, ByVal Culture As CultureInfo, ByVal Value As Object,
+                                               ByVal DestinationType As Type) As Object Implements IGridConversionProvider.ToString
+                If ACheck(Of Double)(Value, AModes.Var, MyProvider) Then
+                    Return Value.ToString
+                Else
+                    Return -1
+                End If
+            End Function
+            Friend Function CreateInstance(ByVal Context As ITypeDescriptorContext, ByVal NewValue As Object, ByRef RefreshGrid As Boolean) As Object Implements IGridConversionProvider.CreateInstance
+                If ACheck(Of Double)(NewValue, AModes.Var, MyProvider) Then
+                    Return NewValue
+                Else
+                    RefreshGrid = True
+                    Return -1
+                End If
+            End Function
+            Friend Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                    Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
+                Return AConvert(Value, AModes.Var, DestinationType,, True, -1, MyProvider, EDP.ReturnValue)
+            End Function
+            Friend Function IsValid(ByVal Context As ITypeDescriptorContext, ByVal Value As Object, ByVal DestinationType As Type) As Boolean Implements IGridValidator.IsValid
+                If ACheck(Of Double)(Value, AModes.Var, MyProvider) Then
+                    Return True
+                Else
+                    Throw New FormatException(ErrorMessageDefault)
+                End If
+            End Function
+            Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
+                Throw New NotImplementedException("'GetFormat' is not available in 'FpsFormatProvider'")
+            End Function
+        End Class
 #End Region
 #Region "Defaults Audio"
         <Browsable(True), GridVisible, XMLVN({"DefaultsAudio"}, "AAC"), Category("Defaults Audio"), DisplayName("Default codec"),

SCrawler.YouTube/Controls/PlaylistArrayForm.Designer.vb

@@ -100,6 +100,7 @@ Namespace API.YouTube.Controls
             Me.TXT_URLS.MaxLength = 2147483647
             Me.TXT_URLS.Multiline = True
             Me.TXT_URLS.Name = "TXT_URLS"
+            Me.TXT_URLS.ScrollBars = System.Windows.Forms.ScrollBars.Both
             Me.TXT_URLS.Size = New System.Drawing.Size(372, 261)
             Me.TXT_URLS.TabIndex = 0
             '

SCrawler.YouTube/Controls/VideoOptionsForm.Designer.vb

@@ -50,6 +50,7 @@ Namespace API.YouTube.Controls
             Dim ActionButton8 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton9 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Dim ActionButton10 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
+            Dim ActionButton11 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton()
             Me.ICON_VIDEO = New System.Windows.Forms.PictureBox()
             Me.LBL_TITLE = New System.Windows.Forms.Label()
             Me.TP_HEADER_INFO_2 = New System.Windows.Forms.TableLayoutPanel()
@@ -71,6 +72,7 @@ Namespace API.YouTube.Controls
             Me.CMB_FORMAT = New System.Windows.Forms.ComboBox()
             Me.CMB_AUDIO_CODEC = New System.Windows.Forms.ComboBox()
             Me.NUM_RES = New System.Windows.Forms.NumericUpDown()
+            Me.TXT_FPS = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TP_CONTROLS = New System.Windows.Forms.TableLayoutPanel()
             Me.TXT_SUBS_ADDIT = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TXT_EXTRA_AUDIO_FORMATS = New PersonalUtilities.Forms.Controls.TextBoxExtended()
@@ -105,6 +107,7 @@ Namespace API.YouTube.Controls
             Me.TP_MAIN.SuspendLayout()
             Me.TP_OPTIONS.SuspendLayout()
             CType(Me.NUM_RES, System.ComponentModel.ISupportInitialize).BeginInit()
+            CType(Me.TXT_FPS, System.ComponentModel.ISupportInitialize).BeginInit()
             CType(Me.TXT_SUBS_ADDIT, System.ComponentModel.ISupportInitialize).BeginInit()
             CType(Me.TXT_EXTRA_AUDIO_FORMATS, System.ComponentModel.ISupportInitialize).BeginInit()
             Me.SuspendLayout()
@@ -123,7 +126,7 @@ Namespace API.YouTube.Controls
             TP_HEADER.Name = "TP_HEADER"
             TP_HEADER.RowCount = 1
             TP_HEADER.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_HEADER.Size = New System.Drawing.Size(599, 63)
+            TP_HEADER.Size = New System.Drawing.Size(719, 63)
             TP_HEADER.TabIndex = 0
             '
             'ICON_VIDEO
@@ -152,7 +155,7 @@ Namespace API.YouTube.Controls
             TP_HEADER_INFO.RowCount = 2
             TP_HEADER_INFO.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
             TP_HEADER_INFO.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_HEADER_INFO.Size = New System.Drawing.Size(469, 63)
+            TP_HEADER_INFO.Size = New System.Drawing.Size(589, 63)
             TP_HEADER_INFO.TabIndex = 0
             '
             'LBL_TITLE
@@ -161,7 +164,7 @@ Namespace API.YouTube.Controls
             Me.LBL_TITLE.Font = New System.Drawing.Font("Arial", 9.0!, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point, CType(204, Byte))
             Me.LBL_TITLE.Location = New System.Drawing.Point(3, 0)
             Me.LBL_TITLE.Name = "LBL_TITLE"
-            Me.LBL_TITLE.Size = New System.Drawing.Size(463, 31)
+            Me.LBL_TITLE.Size = New System.Drawing.Size(583, 31)
             Me.LBL_TITLE.TabIndex = 0
             Me.LBL_TITLE.Text = "Video title"
             Me.LBL_TITLE.TextAlign = System.Drawing.ContentAlignment.MiddleLeft
@@ -183,7 +186,7 @@ Namespace API.YouTube.Controls
             Me.TP_HEADER_INFO_2.Name = "TP_HEADER_INFO_2"
             Me.TP_HEADER_INFO_2.RowCount = 1
             Me.TP_HEADER_INFO_2.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            Me.TP_HEADER_INFO_2.Size = New System.Drawing.Size(469, 32)
+            Me.TP_HEADER_INFO_2.Size = New System.Drawing.Size(589, 32)
             Me.TP_HEADER_INFO_2.TabIndex = 1
             '
             'ICON_CLOCK
@@ -230,7 +233,7 @@ Namespace API.YouTube.Controls
             Me.LBL_URL.LinkColor = System.Drawing.Color.FromArgb(CType(CType(0, Byte), Integer), CType(CType(0, Byte), Integer), CType(CType(192, Byte), Integer))
             Me.LBL_URL.Location = New System.Drawing.Point(115, 0)
             Me.LBL_URL.Name = "LBL_URL"
-            Me.LBL_URL.Size = New System.Drawing.Size(351, 32)
+            Me.LBL_URL.Size = New System.Drawing.Size(471, 32)
             Me.LBL_URL.TabIndex = 1
             Me.LBL_URL.TabStop = True
             Me.LBL_URL.Text = "https://www.youtube.com/watch?v=abcdefghijk"
@@ -250,7 +253,7 @@ Namespace API.YouTube.Controls
             TP_FOOTER.RowCount = 2
             TP_FOOTER.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
             TP_FOOTER.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 50.0!))
-            TP_FOOTER.Size = New System.Drawing.Size(589, 52)
+            TP_FOOTER.Size = New System.Drawing.Size(709, 52)
             TP_FOOTER.TabIndex = 5
             '
             'TP_DESTINATION
@@ -266,7 +269,7 @@ Namespace API.YouTube.Controls
             TP_DESTINATION.Name = "TP_DESTINATION"
             TP_DESTINATION.RowCount = 1
             TP_DESTINATION.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            TP_DESTINATION.Size = New System.Drawing.Size(589, 26)
+            TP_DESTINATION.Size = New System.Drawing.Size(709, 26)
             TP_DESTINATION.TabIndex = 0
             '
             'TXT_FILE
@@ -290,14 +293,14 @@ Namespace API.YouTube.Controls
             Me.TXT_FILE.Location = New System.Drawing.Point(1, 1)
             Me.TXT_FILE.Margin = New System.Windows.Forms.Padding(1)
             Me.TXT_FILE.Name = "TXT_FILE"
-            Me.TXT_FILE.Size = New System.Drawing.Size(507, 22)
+            Me.TXT_FILE.Size = New System.Drawing.Size(627, 22)
             Me.TXT_FILE.TabIndex = 0
             Me.TXT_FILE.TextBoxBorderStyle = System.Windows.Forms.BorderStyle.FixedSingle
             '
             'BTT_BROWSE
             '
             Me.BTT_BROWSE.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.BTT_BROWSE.Location = New System.Drawing.Point(512, 2)
+            Me.BTT_BROWSE.Location = New System.Drawing.Point(632, 2)
             Me.BTT_BROWSE.Margin = New System.Windows.Forms.Padding(3, 2, 3, 2)
             Me.BTT_BROWSE.Name = "BTT_BROWSE"
             Me.BTT_BROWSE.Size = New System.Drawing.Size(74, 22)
@@ -321,13 +324,13 @@ Namespace API.YouTube.Controls
             TP_OK_CANCEL.RowCount = 1
             TP_OK_CANCEL.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
             TP_OK_CANCEL.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 26.0!))
-            TP_OK_CANCEL.Size = New System.Drawing.Size(589, 26)
+            TP_OK_CANCEL.Size = New System.Drawing.Size(709, 26)
             TP_OK_CANCEL.TabIndex = 1
             '
             'BTT_DOWN
             '
             Me.BTT_DOWN.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.BTT_DOWN.Location = New System.Drawing.Point(432, 2)
+            Me.BTT_DOWN.Location = New System.Drawing.Point(552, 2)
             Me.BTT_DOWN.Margin = New System.Windows.Forms.Padding(3, 2, 3, 2)
             Me.BTT_DOWN.Name = "BTT_DOWN"
             Me.BTT_DOWN.Size = New System.Drawing.Size(74, 22)
@@ -339,7 +342,7 @@ Namespace API.YouTube.Controls
             '
             Me.BTT_CANCEL.DialogResult = System.Windows.Forms.DialogResult.Cancel
             Me.BTT_CANCEL.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.BTT_CANCEL.Location = New System.Drawing.Point(512, 2)
+            Me.BTT_CANCEL.Location = New System.Drawing.Point(632, 2)
             Me.BTT_CANCEL.Margin = New System.Windows.Forms.Padding(3, 2, 3, 2)
             Me.BTT_CANCEL.Name = "BTT_CANCEL"
             Me.BTT_CANCEL.Size = New System.Drawing.Size(74, 22)
@@ -354,7 +357,7 @@ Namespace API.YouTube.Controls
             LB_SEP_1.Location = New System.Drawing.Point(6, 179)
             LB_SEP_1.Margin = New System.Windows.Forms.Padding(6, 0, 6, 0)
             LB_SEP_1.Name = "LB_SEP_1"
-            LB_SEP_1.Size = New System.Drawing.Size(589, 1)
+            LB_SEP_1.Size = New System.Drawing.Size(709, 1)
             LB_SEP_1.TabIndex = 3
             '
             'LB_SEP_2
@@ -364,7 +367,7 @@ Namespace API.YouTube.Controls
             LB_SEP_2.Location = New System.Drawing.Point(6, 209)
             LB_SEP_2.Margin = New System.Windows.Forms.Padding(6, 0, 6, 0)
             LB_SEP_2.Name = "LB_SEP_2"
-            LB_SEP_2.Size = New System.Drawing.Size(589, 1)
+            LB_SEP_2.Size = New System.Drawing.Size(709, 1)
             LB_SEP_2.TabIndex = 5
             '
             'TP_WHAT
@@ -436,7 +439,7 @@ Namespace API.YouTube.Controls
             '
             LBL_SUBS_FORMAT.AutoSize = True
             LBL_SUBS_FORMAT.Dock = System.Windows.Forms.DockStyle.Fill
-            LBL_SUBS_FORMAT.Location = New System.Drawing.Point(432, 0)
+            LBL_SUBS_FORMAT.Location = New System.Drawing.Point(552, 0)
             LBL_SUBS_FORMAT.Name = "LBL_SUBS_FORMAT"
             LBL_SUBS_FORMAT.Size = New System.Drawing.Size(74, 28)
             LBL_SUBS_FORMAT.TabIndex = 2
@@ -448,7 +451,7 @@ Namespace API.YouTube.Controls
             '
             Me.LBL_AUDIO_CODEC.AutoSize = True
             Me.LBL_AUDIO_CODEC.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.LBL_AUDIO_CODEC.Location = New System.Drawing.Point(432, 0)
+            Me.LBL_AUDIO_CODEC.Location = New System.Drawing.Point(552, 0)
             Me.LBL_AUDIO_CODEC.Name = "LBL_AUDIO_CODEC"
             Me.LBL_AUDIO_CODEC.Size = New System.Drawing.Size(74, 28)
             Me.LBL_AUDIO_CODEC.TabIndex = 5
@@ -470,7 +473,7 @@ Namespace API.YouTube.Controls
             Me.TP_HEADER_BASE.RowCount = 1
             Me.TP_HEADER_BASE.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
             Me.TP_HEADER_BASE.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 64.0!))
-            Me.TP_HEADER_BASE.Size = New System.Drawing.Size(601, 65)
+            Me.TP_HEADER_BASE.Size = New System.Drawing.Size(721, 65)
             Me.TP_HEADER_BASE.TabIndex = 6
             '
             'TP_SUBS
@@ -488,7 +491,7 @@ Namespace API.YouTube.Controls
             Me.TP_SUBS.Name = "TP_SUBS"
             Me.TP_SUBS.RowCount = 1
             Me.TP_SUBS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            Me.TP_SUBS.Size = New System.Drawing.Size(589, 28)
+            Me.TP_SUBS.Size = New System.Drawing.Size(709, 28)
             Me.TP_SUBS.TabIndex = 2
             '
             'TXT_SUBS
@@ -516,7 +519,7 @@ Namespace API.YouTube.Controls
             Me.TXT_SUBS.Dock = System.Windows.Forms.DockStyle.Fill
             Me.TXT_SUBS.Location = New System.Drawing.Point(3, 3)
             Me.TXT_SUBS.Name = "TXT_SUBS"
-            Me.TXT_SUBS.Size = New System.Drawing.Size(423, 22)
+            Me.TXT_SUBS.Size = New System.Drawing.Size(543, 22)
             Me.TXT_SUBS.TabIndex = 0
             Me.TXT_SUBS.TextBoxReadOnly = True
             '
@@ -525,7 +528,7 @@ Namespace API.YouTube.Controls
             Me.CMB_SUBS_FORMAT.Dock = System.Windows.Forms.DockStyle.Fill
             Me.CMB_SUBS_FORMAT.DropDownStyle = System.Windows.Forms.ComboBoxStyle.DropDownList
             Me.CMB_SUBS_FORMAT.FormattingEnabled = True
-            Me.CMB_SUBS_FORMAT.Location = New System.Drawing.Point(512, 3)
+            Me.CMB_SUBS_FORMAT.Location = New System.Drawing.Point(632, 3)
             Me.CMB_SUBS_FORMAT.Name = "CMB_SUBS_FORMAT"
             Me.CMB_SUBS_FORMAT.Size = New System.Drawing.Size(74, 21)
             Me.CMB_SUBS_FORMAT.TabIndex = 1
@@ -557,31 +560,33 @@ Namespace API.YouTube.Controls
             Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 5.0!))
             Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 58.0!))
             Me.TP_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle())
-            Me.TP_MAIN.Size = New System.Drawing.Size(601, 271)
+            Me.TP_MAIN.Size = New System.Drawing.Size(721, 271)
             Me.TP_MAIN.TabIndex = 0
             '
             'TP_OPTIONS
             '
-            Me.TP_OPTIONS.ColumnCount = 6
+            Me.TP_OPTIONS.ColumnCount = 7
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 80.0!))
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 80.0!))
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 80.0!))
+            Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 120.0!))
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 80.0!))
             Me.TP_OPTIONS.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Absolute, 80.0!))
             Me.TP_OPTIONS.Controls.Add(LBL_FORMAT, 1, 0)
             Me.TP_OPTIONS.Controls.Add(TP_WHAT, 0, 0)
             Me.TP_OPTIONS.Controls.Add(Me.CMB_FORMAT, 2, 0)
-            Me.TP_OPTIONS.Controls.Add(Me.LBL_AUDIO_CODEC, 4, 0)
+            Me.TP_OPTIONS.Controls.Add(Me.LBL_AUDIO_CODEC, 5, 0)
-            Me.TP_OPTIONS.Controls.Add(Me.CMB_AUDIO_CODEC, 5, 0)
+            Me.TP_OPTIONS.Controls.Add(Me.CMB_AUDIO_CODEC, 6, 0)
             Me.TP_OPTIONS.Controls.Add(Me.NUM_RES, 3, 0)
+            Me.TP_OPTIONS.Controls.Add(Me.TXT_FPS, 4, 0)
             Me.TP_OPTIONS.Dock = System.Windows.Forms.DockStyle.Fill
             Me.TP_OPTIONS.Location = New System.Drawing.Point(6, 65)
             Me.TP_OPTIONS.Margin = New System.Windows.Forms.Padding(6, 0, 6, 0)
             Me.TP_OPTIONS.Name = "TP_OPTIONS"
             Me.TP_OPTIONS.RowCount = 1
             Me.TP_OPTIONS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            Me.TP_OPTIONS.Size = New System.Drawing.Size(589, 28)
+            Me.TP_OPTIONS.Size = New System.Drawing.Size(709, 28)
             Me.TP_OPTIONS.TabIndex = 1
             '
             'CMB_FORMAT
@@ -599,7 +604,7 @@ Namespace API.YouTube.Controls
             Me.CMB_AUDIO_CODEC.Dock = System.Windows.Forms.DockStyle.Fill
             Me.CMB_AUDIO_CODEC.DropDownStyle = System.Windows.Forms.ComboBoxStyle.DropDownList
             Me.CMB_AUDIO_CODEC.FormattingEnabled = True
-            Me.CMB_AUDIO_CODEC.Location = New System.Drawing.Point(512, 3)
+            Me.CMB_AUDIO_CODEC.Location = New System.Drawing.Point(632, 3)
             Me.CMB_AUDIO_CODEC.Name = "CMB_AUDIO_CODEC"
             Me.CMB_AUDIO_CODEC.Size = New System.Drawing.Size(74, 21)
             Me.CMB_AUDIO_CODEC.TabIndex = 3
@@ -616,6 +621,24 @@ Namespace API.YouTube.Controls
             Me.NUM_RES.TextAlign = System.Windows.Forms.HorizontalAlignment.Center
             Me.NUM_RES.Value = New Decimal(New Integer() {1080, 0, 0, 0})
             '
+            'TXT_FPS
+            '
+            ActionButton5.BackgroundImage = CType(resources.GetObject("ActionButton5.BackgroundImage"), System.Drawing.Image)
+            ActionButton5.Name = "Clear"
+            ActionButton5.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            Me.TXT_FPS.Buttons.Add(ActionButton5)
+            Me.TXT_FPS.CaptionText = "FPS"
+            Me.TXT_FPS.CaptionToolTipEnabled = True
+            Me.TXT_FPS.CaptionToolTipText = "You can reduce the video FPS by setting the FPS value in this field."
+            Me.TXT_FPS.CaptionWidth = 30.0R
+            Me.TXT_FPS.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.TXT_FPS.Location = New System.Drawing.Point(432, 2)
+            Me.TXT_FPS.Margin = New System.Windows.Forms.Padding(3, 2, 3, 3)
+            Me.TXT_FPS.Name = "TXT_FPS"
+            Me.TXT_FPS.Size = New System.Drawing.Size(114, 22)
+            Me.TXT_FPS.TabIndex = 6
+            Me.TXT_FPS.TextBoxWidthMinimal = 30
+            '
             'TP_CONTROLS
             '
             Me.TP_CONTROLS.ColumnCount = 1
@@ -626,29 +649,29 @@ Namespace API.YouTube.Controls
             Me.TP_CONTROLS.Name = "TP_CONTROLS"
             Me.TP_CONTROLS.RowCount = 1
             Me.TP_CONTROLS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!))
-            Me.TP_CONTROLS.Size = New System.Drawing.Size(595, 25)
+            Me.TP_CONTROLS.Size = New System.Drawing.Size(715, 25)
             Me.TP_CONTROLS.TabIndex = 0
             '
             'TXT_SUBS_ADDIT
             '
-            ActionButton5.BackgroundImage = CType(resources.GetObject("ActionButton5.BackgroundImage"), System.Drawing.Image)
-            ActionButton5.Enabled = False
-            ActionButton5.Name = "Open"
-            ActionButton5.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open
-            ActionButton5.ToolTipText = "Choose additional formats"
             ActionButton6.BackgroundImage = CType(resources.GetObject("ActionButton6.BackgroundImage"), System.Drawing.Image)
             ActionButton6.Enabled = False
-            ActionButton6.Name = "Refresh"
+            ActionButton6.Name = "Open"
-            ActionButton6.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh
+            ActionButton6.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open
-            ActionButton6.ToolTipText = "Fill in additional formats from the defaults"
+            ActionButton6.ToolTipText = "Choose additional formats"
             ActionButton7.BackgroundImage = CType(resources.GetObject("ActionButton7.BackgroundImage"), System.Drawing.Image)
             ActionButton7.Enabled = False
-            ActionButton7.Name = "Clear"
+            ActionButton7.Name = "Refresh"
-            ActionButton7.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            ActionButton7.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh
-            ActionButton7.ToolTipText = "Remove all additional formats"
+            ActionButton7.ToolTipText = "Fill in additional formats from the defaults"
-            Me.TXT_SUBS_ADDIT.Buttons.Add(ActionButton5)
+            ActionButton8.BackgroundImage = CType(resources.GetObject("ActionButton8.BackgroundImage"), System.Drawing.Image)
+            ActionButton8.Enabled = False
+            ActionButton8.Name = "Clear"
+            ActionButton8.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            ActionButton8.ToolTipText = "Remove all additional formats"
             Me.TXT_SUBS_ADDIT.Buttons.Add(ActionButton6)
             Me.TXT_SUBS_ADDIT.Buttons.Add(ActionButton7)
+            Me.TXT_SUBS_ADDIT.Buttons.Add(ActionButton8)
             Me.TXT_SUBS_ADDIT.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.CheckBox
             Me.TXT_SUBS_ADDIT.CaptionText = "Additional subtitle formats"
             Me.TXT_SUBS_ADDIT.CaptionToolTipEnabled = True
@@ -659,31 +682,31 @@ Namespace API.YouTube.Controls
             Me.TXT_SUBS_ADDIT.Location = New System.Drawing.Point(6, 124)
             Me.TXT_SUBS_ADDIT.Margin = New System.Windows.Forms.Padding(6, 3, 6, 3)
             Me.TXT_SUBS_ADDIT.Name = "TXT_SUBS_ADDIT"
-            Me.TXT_SUBS_ADDIT.Size = New System.Drawing.Size(589, 22)
+            Me.TXT_SUBS_ADDIT.Size = New System.Drawing.Size(709, 22)
             Me.TXT_SUBS_ADDIT.TabIndex = 3
             Me.TXT_SUBS_ADDIT.Tag = "s"
             Me.TXT_SUBS_ADDIT.TextBoxReadOnly = True
             '
             'TXT_EXTRA_AUDIO_FORMATS
             '
-            ActionButton8.BackgroundImage = CType(resources.GetObject("ActionButton8.BackgroundImage"), System.Drawing.Image)
-            ActionButton8.Enabled = False
-            ActionButton8.Name = "Open"
-            ActionButton8.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open
-            ActionButton8.ToolTipText = "Choose additional formats"
             ActionButton9.BackgroundImage = CType(resources.GetObject("ActionButton9.BackgroundImage"), System.Drawing.Image)
             ActionButton9.Enabled = False
-            ActionButton9.Name = "Refresh"
+            ActionButton9.Name = "Open"
-            ActionButton9.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh
+            ActionButton9.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open
-            ActionButton9.ToolTipText = "Fill in additional formats from the defaults"
+            ActionButton9.ToolTipText = "Choose additional formats"
             ActionButton10.BackgroundImage = CType(resources.GetObject("ActionButton10.BackgroundImage"), System.Drawing.Image)
             ActionButton10.Enabled = False
-            ActionButton10.Name = "Clear"
+            ActionButton10.Name = "Refresh"
-            ActionButton10.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            ActionButton10.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh
-            ActionButton10.ToolTipText = "Choose additional formats"
+            ActionButton10.ToolTipText = "Fill in additional formats from the defaults"
-            Me.TXT_EXTRA_AUDIO_FORMATS.Buttons.Add(ActionButton8)
+            ActionButton11.BackgroundImage = CType(resources.GetObject("ActionButton11.BackgroundImage"), System.Drawing.Image)
+            ActionButton11.Enabled = False
+            ActionButton11.Name = "Clear"
+            ActionButton11.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear
+            ActionButton11.ToolTipText = "Choose additional formats"
             Me.TXT_EXTRA_AUDIO_FORMATS.Buttons.Add(ActionButton9)
             Me.TXT_EXTRA_AUDIO_FORMATS.Buttons.Add(ActionButton10)
+            Me.TXT_EXTRA_AUDIO_FORMATS.Buttons.Add(ActionButton11)
             Me.TXT_EXTRA_AUDIO_FORMATS.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.CheckBox
             Me.TXT_EXTRA_AUDIO_FORMATS.CaptionText = "Additional audio formats"
             Me.TXT_EXTRA_AUDIO_FORMATS.CaptionToolTipEnabled = True
@@ -693,7 +716,7 @@ Namespace API.YouTube.Controls
             Me.TXT_EXTRA_AUDIO_FORMATS.Location = New System.Drawing.Point(6, 152)
             Me.TXT_EXTRA_AUDIO_FORMATS.Margin = New System.Windows.Forms.Padding(6, 3, 6, 3)
             Me.TXT_EXTRA_AUDIO_FORMATS.Name = "TXT_EXTRA_AUDIO_FORMATS"
-            Me.TXT_EXTRA_AUDIO_FORMATS.Size = New System.Drawing.Size(589, 22)
+            Me.TXT_EXTRA_AUDIO_FORMATS.Size = New System.Drawing.Size(709, 22)
             Me.TXT_EXTRA_AUDIO_FORMATS.TabIndex = 4
             Me.TXT_EXTRA_AUDIO_FORMATS.Tag = "a"
             Me.TXT_EXTRA_AUDIO_FORMATS.TextBoxReadOnly = True
@@ -704,13 +727,14 @@ Namespace API.YouTube.Controls
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
             Me.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font
             Me.CancelButton = Me.BTT_CANCEL
-            Me.ClientSize = New System.Drawing.Size(601, 271)
+            Me.ClientSize = New System.Drawing.Size(721, 271)
             Me.Controls.Add(Me.TP_MAIN)
             Me.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedSingle
             Me.Icon = Global.SCrawler.My.Resources.SiteYouTube.YouTubeIcon_32
             Me.KeyPreview = True
             Me.MaximizeBox = False
             Me.MinimizeBox = False
+            Me.MinimumSize = New System.Drawing.Size(737, 310)
             Me.Name = "VideoOptionsForm"
             Me.ShowInTaskbar = False
             Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
@@ -736,6 +760,7 @@ Namespace API.YouTube.Controls
             Me.TP_OPTIONS.ResumeLayout(False)
             Me.TP_OPTIONS.PerformLayout()
             CType(Me.NUM_RES, System.ComponentModel.ISupportInitialize).EndInit()
+            CType(Me.TXT_FPS, System.ComponentModel.ISupportInitialize).EndInit()
             CType(Me.TXT_SUBS_ADDIT, System.ComponentModel.ISupportInitialize).EndInit()
             CType(Me.TXT_EXTRA_AUDIO_FORMATS, System.ComponentModel.ISupportInitialize).EndInit()
             Me.ResumeLayout(False)
@@ -765,5 +790,6 @@ Namespace API.YouTube.Controls
         Private WithEvents BTT_DOWN As Button
         Private WithEvents BTT_CANCEL As Button
         Private WithEvents TP_HEADER_INFO_2 As TableLayoutPanel
+        Private WithEvents TXT_FPS As PersonalUtilities.Forms.Controls.TextBoxExtended
     End Class
 End Namespace

SCrawler.YouTube/Controls/VideoOptionsForm.resx

@@ -289,6 +289,14 @@
 </value>
   </data>
   <data name="ActionButton5.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+    <value>
+        iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
+        xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go
+        tbbB43rK5xSAQq1VYFtmeQBoqZTSreVZvgTknM8yyyjA/qodsDF9gspD2Bj6B+DH+NqzhQQAG+POMnSX
+        AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
+</value>
+  </data>
+  <data name="ActionButton6.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
         wwAADsMBx2+oZAAAAR5JREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbGujg3ZATZPKYdC6FQhPwlAMkg3dP
@@ -299,7 +307,7 @@
         cMaRN0UdBBkAAAAASUVORK5CYII=
 </value>
   </data>
-  <data name="ActionButton6.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="ActionButton7.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6
         JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACOElE
@@ -315,7 +323,7 @@
         VnR1MIwhwMTCyqEQ37qEmZVDFF0OE/9nAACtFF4Ey6OP+wAAAABJRU5ErkJggg==
 </value>
   </data>
-  <data name="ActionButton7.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="ActionButton8.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
         xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go
@@ -323,7 +331,7 @@
         AFuc5QFgn6ClHh5iOQVAKNixyucB8NY0vG9JOzzyhrdq5IRgAAAAAElFTkSuQmCC
 </value>
   </data>
-  <data name="ActionButton8.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="ActionButton9.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
         wwAADsMBx2+oZAAAAR5JREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbGujg3ZATZPKYdC6FQhPwlAMkg3dP
@@ -334,7 +342,7 @@
         cMaRN0UdBBkAAAAASUVORK5CYII=
 </value>
   </data>
-  <data name="ActionButton9.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="ActionButton10.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6
         JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACOElE
@@ -350,7 +358,7 @@
         VnR1MIwhwMTCyqEQ37qEmZVDFF0OE/9nAACtFF4Ey6OP+wAAAABJRU5ErkJggg==
 </value>
   </data>
-  <data name="ActionButton10.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="ActionButton11.BackgroundImage" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO
         xAAADsQBlSsOGwAAAIZJREFUOE+1j10KwCAMgz2b755xl/IsvnaL2K20UfbDAmEako+ZROSTafjE12Go

SCrawler.YouTube/Controls/VideoOptionsForm.vb

@@ -22,6 +22,7 @@ Namespace API.YouTube.Controls
     Friend Class VideoOptionsForm : Implements IDesignXMLContainer
 #Region "Declarations"
         Private MyView As FormView
+        Private ReadOnly MyFieldsChecker As FieldsChecker
         Friend Property DesignXML As EContainer Implements IDesignXMLContainer.DesignXML
         Private Property DesignXMLNodes As String() Implements IDesignXMLContainer.DesignXMLNodes
         Private Property DesignXMLNodeName As String Implements IDesignXMLContainer.DesignXMLNodeName
@@ -30,6 +31,25 @@ Namespace API.YouTube.Controls
         Friend Property MyContainer As YouTubeMediaContainerBase
         Private Initialization As Boolean = True
         Private ReadOnly IsSavedObject As Boolean
+        Private Class FpsFieldChecker : Inherits FieldsCheckerProviderBase
+            Private ReadOnly MyProvider As ANumbers = YouTubeSettings.FpsFormatProvider.MyProviderDefault
+            Public Overrides Property ErrorMessage As String
+                Get
+                    Return IIf(HasError, YouTubeSettings.FpsFormatProvider.ErrorMessageDefault, String.Empty)
+                End Get
+                Set : End Set
+            End Property
+            Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
+                                              Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
+                If ACheck(Of Double)(Value, AModes.Var, MyProvider) Then
+                    Return Value
+                Else
+                    HasError = True
+                    TypeError = True
+                    Return Nothing
+                End If
+            End Function
+        End Class
 #End Region
 #Region "Initializers"
         Friend Sub New(ByVal Container As YouTubeMediaContainerBase, Optional ByVal IsSavedObject As Boolean = False)
@@ -37,6 +57,7 @@ Namespace API.YouTube.Controls
             MyContainer = Container
             CNT_PROCESSOR = New TableControlsProcessor(TP_CONTROLS)
             Me.IsSavedObject = IsSavedObject
+            MyFieldsChecker = New FieldsChecker
         End Sub
 #End Region
 #Region "Form handlers"
@@ -132,6 +153,9 @@ Namespace API.YouTube.Controls
                     End If
                     setDef(CMB_SUBS_FORMAT, __optionValue)
 
+                    If MyYouTubeSettings.DefaultVideoFPS > 0 Then TXT_FPS.Text = MyYouTubeSettings.DefaultVideoFPS
+                    MyFieldsChecker.AddControl(Of Double)(TXT_FPS, TXT_FPS.CaptionText, True, New FpsFieldChecker)
+                    MyFieldsChecker.EndLoaderOperations()
                     TP_SUBS.Enabled = .Subtitles.Count > 0
                     TXT_SUBS_ADDIT.Enabled = .Subtitles.Count > 0
                     RefillTextBoxes()
@@ -275,6 +299,7 @@ Namespace API.YouTube.Controls
 #Region "OK, Cancel"
         Private Sub BTT_DOWN_Click(sender As Object, e As EventArgs) Handles BTT_DOWN.Click
             Try
+                If Not MyFieldsChecker.AllParamsOK Then Exit Sub
                 Dim f As SFile
                 If MyContainer.HasElements Then
                     f = TXT_FILE.Text.CSFileP
@@ -284,6 +309,7 @@ Namespace API.YouTube.Controls
                 If f.IsEmptyString Then Throw New ArgumentNullException("File", "The output file cannot be null")
                 With MyContainer
                     .OutputVideoExtension = CMB_FORMAT.Text.StringToLower
+                    .OutputVideoFPS = AConvert(Of Double)(TXT_FPS.Text, YouTubeSettings.FpsFormatProvider.MyProviderDefault, -1)
                     .OutputAudioCodec = CMB_AUDIO_CODEC.Text.StringToLower
                     .OutputSubtitlesFormat = CMB_SUBS_FORMAT.Text.StringToLower
 

SCrawler.YouTube/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2023.12.14.0")>
+<Assembly: AssemblyVersion("2024.1.26.0")>
-<Assembly: AssemblyFileVersion("2023.12.14.0")>
+<Assembly: AssemblyFileVersion("2024.1.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb

@@ -217,6 +217,16 @@ Namespace API.YouTube.Objects
                 If HasElements Then Elements.ForEach(Sub(e) e.OutputVideoExtension = _OutputVideoExtension)
             End Set
         End Property
+        <XMLEC("OutputVideoFPS")> Protected _OutputVideoFPS As Double = -1
+        Friend Property OutputVideoFPS As Double
+            Get
+                Return _OutputVideoFPS
+            End Get
+            Set(ByVal fps As Double)
+                _OutputVideoFPS = fps
+                If HasElements Then Elements.ForEach(Sub(elem) DirectCast(elem, YouTubeMediaContainerBase).OutputVideoFPS = fps)
+            End Set
+        End Property
 #End Region
 #Region "Audio"
         <XMLEC> Friend Property SelectedAudioIndex As Integer = -1 Implements IYouTubeMediaContainer.SelectedAudioIndex
@@ -1026,6 +1036,16 @@ Namespace API.YouTube.Objects
                                     If fAacAudio.Exists And Not aacRequested Then fAacAudio.Delete()
                                     If fAc3Audio.Exists And Not ac3Requested And SelectedVideoIndex >= 0 Then fAc3Audio.Delete()
                                 End If
+
+                                If SelectedVideoIndex >= 0 AndAlso OutputVideoFPS > 0 AndAlso SelectedVideo.Bitrate > OutputVideoFPS Then
+                                    f = File
+                                    f.Name &= "tmp00"
+                                    .Execute($"ffmpeg -i ""{File}"" -filter:v fps={OutputVideoFPS.ToString.Replace(",", ".")} -c:a copy ""{f}""")
+                                    If f.Exists Then
+                                        File.Delete()
+                                        SFile.Rename(f, File,, EDP.LogMessageValue)
+                                    End If
+                                End If
                             End If
                         End If
                     End With

SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2023.12.14.0")>
+<Assembly: AssemblyVersion("2024.1.26.0")>
-<Assembly: AssemblyFileVersion("2023.12.14.0")>
+<Assembly: AssemblyFileVersion("2024.1.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/API/Base/ProfileSaved.vb

@@ -47,7 +47,7 @@ Namespace API.Base
                     Progress.InformationTemporary = $"{HOST.Name} ({c - s}/{c}) Images: {_TotalImages}; Videos: {_TotalVideos}"
                 End If
             End If
-            If _FeedDataExists Then Downloader.Files.Sort()
+            If _FeedDataExists Then Downloader.Files.Sort() : Downloader.FilesSave()
         End Sub
         Private Overloads Sub Download(ByVal Host As SettingsHost, ByVal Number As Integer, ByVal Count As Integer,
                                        ByVal Token As CancellationToken, ByVal Multiple As Boolean)

SCrawler/API/Base/UserDataBase.vb

@@ -205,7 +205,7 @@ Namespace API.Base
                 If Not h Is Nothing Then _HostKey = h.Key
             End Set
         End Property
-        Private Sub ResetHost()
+        Friend Sub ResetHost()
             _HostObtained = False
         End Sub
         Friend Property HostStatic As Boolean = False Implements IUserData.HostStatic
@@ -1403,7 +1403,7 @@ BlockNullPicture:
                         End If
                         ff.Name &= "_thumb"
                         ff.Extension = "jpg"
-                        f = Web.FFMPEG.TakeSnapshot(f, ff, Settings.FfmpegFile, TimeSpan.FromSeconds(1),,, EDP.LogMessageValue)
+                        f = Web.FFMPEG.TakeSnapshot(f, ff, Settings.FfmpegFile, TimeSpan.FromSeconds(1),,, EDP.SendToLog + EDP.ReturnValue)
                         If f.Exists Then DirectCast(Data, IDownloadableMedia).ThumbnailFile = f
                     End If
                 Else

SCrawler/API/Instagram/Declarations.vb

@@ -15,6 +15,9 @@ Namespace API.Instagram
         Friend Const InstagramSite As String = "Instagram"
         Friend Const InstagramSiteKey As String = "AndyProgram_Instagram"
         Friend ReadOnly FilesPattern As RParams = RParams.DMS(".+?([^/\?]+?\.[\w\d]{3,4})(?=(\?|\Z))", 1, EDP.ReturnValue)
+        Friend ReadOnly ObtainMedia_SizeFuncPic_RegexP As RParams = RParams.DMS("_p(\d+)x(\d+)", 1, EDP.ReturnValue)
+        Friend ReadOnly ObtainMedia_SizeFuncPic_RegexS As RParams = RParams.DMS("_s(\d+)x(\d+)", 1, EDP.ReturnValue)
+        Friend Const PageTokenRegexPatternDefault As String = "\[\],{""token"":""(.*?)""},\d+\]"
         Friend Sub UpdateResponser(ByVal Source As IResponse, ByRef Destination As Responser)
             Const r_wwwClaimName$ = "x-ig-set-www-claim"
             Const r_tokenName$ = SiteSettings.Header_CSRF_TOKEN_COOKIE

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -11,6 +11,8 @@ Namespace API.Instagram
     Friend Class EditorExchangeOptions
         <PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
         Friend Property GetTimeline As Boolean
+        <PSetting(Caption:="Get reels", ToolTip:="Download user reels")>
+        Friend Property GetReels As Boolean
         <PSetting(Caption:="Get stories", ToolTip:="Download user stories (pinned)")>
         Friend Property GetStories As Boolean
         <PSetting(Caption:="Get stories: user", ToolTip:="Download user stories")>
@@ -20,6 +22,7 @@ Namespace API.Instagram
         Friend Sub New(ByVal u As UserData)
             With u
                 GetTimeline = .GetTimeline
+                GetReels = .GetReels
                 GetStories = .GetStories
                 GetStoriesUser = .GetStoriesUser
                 GetTagged = .GetTaggedData
@@ -28,6 +31,7 @@ Namespace API.Instagram
         Friend Sub New(ByVal s As SiteSettings)
             With s
                 GetTimeline = CBool(.GetTimeline.Value)
+                GetReels = CBool(.GetReels.Value)
                 GetStories = CBool(.GetStories.Value)
                 GetStoriesUser = CBool(.GetStoriesUser.Value)
                 GetTagged = CBool(.GetTagged.Value)

SCrawler/API/Instagram/SiteSettings.vb

@@ -63,8 +63,6 @@ Namespace API.Instagram
         Friend Const Header_Browser As String = "Sec-Ch-Ua"
         Friend Const Header_BrowserExt As String = "Sec-Ch-Ua-Full-Version-List"
         Friend Const Header_Platform As String = "Sec-Ch-Ua-Platform-Version"
-        <PropertyOption(ControlText:="Hash", ControlToolTip:="Instagram session hash for tagged posts", IsAuth:=True), PXML("InstaHash"), ControlNumber(0), PClonable(Clone:=False)>
-        Friend ReadOnly Property HashTagged As PropertyValue
         <PropertyOption(ControlText:="x-csrftoken", IsAuth:=True, AllowNull:=False), ControlNumber(2), PClonable(Clone:=False)>
         Friend ReadOnly Property HH_CSRF_TOKEN As PropertyValue
         <PropertyOption(ControlText:="x-ig-app-id", IsAuth:=True, AllowNull:=False), ControlNumber(3), PClonable(Clone:=False)>
@@ -123,11 +121,13 @@ Namespace API.Instagram
         Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
         <PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users"), PXML, ControlNumber(23), PClonable>
         Friend ReadOnly Property GetTimeline As PropertyValue
-        <PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users"), PXML, ControlNumber(24), PClonable>
+        <PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users"), PXML, ControlNumber(24), PClonable>
+        Friend ReadOnly Property GetReels As PropertyValue
+        <PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users"), PXML, ControlNumber(25), PClonable>
         Friend ReadOnly Property GetStories As PropertyValue
-        <PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users"), PXML, ControlNumber(25), PClonable>
+        <PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users"), PXML, ControlNumber(26), PClonable>
         Friend ReadOnly Property GetStoriesUser As PropertyValue
-        <PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users"), PXML, ControlNumber(26), PClonable>
+        <PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users"), PXML, ControlNumber(27), PClonable>
         Friend ReadOnly Property GetTagged As PropertyValue
         <PropertyOption(ControlText:="Tagged notify limit",
                         ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
@@ -139,11 +139,13 @@ Namespace API.Instagram
 #Region "Download ready"
         <PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline"), PXML, ControlNumber(10), PClonable>
         Friend ReadOnly Property DownloadTimeline As PropertyValue
-        <PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories"), PXML, ControlNumber(11), PClonable>
+        <PropertyOption(ControlText:="Download reels", ControlToolTip:="Download reels"), PXML, ControlNumber(11), PClonable>
+        Friend ReadOnly Property DownloadReels As PropertyValue
+        <PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories"), PXML, ControlNumber(12), PClonable>
         Friend ReadOnly Property DownloadStories As PropertyValue
-        <PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)"), PXML, ControlNumber(12), PClonable>
+        <PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)"), PXML, ControlNumber(13), PClonable>
         Friend ReadOnly Property DownloadStoriesUser As PropertyValue
-        <PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts"), PXML, ControlNumber(13), PClonable>
+        <PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts"), PXML, ControlNumber(14), PClonable>
         Friend ReadOnly Property DownloadTagged As PropertyValue
 #End Region
 #Region "429 bypass"
@@ -236,7 +238,6 @@ Namespace API.Instagram
                 .CookiesExtractedAutoSave = False
             End With
 
-            HashTagged = New PropertyValue(String.Empty, GetType(String))
             HH_CSRF_TOKEN = New PropertyValue(token, GetType(String), Sub(v) ChangeResponserFields(NameOf(HH_CSRF_TOKEN), v))
             HH_IG_APP_ID = New PropertyValue(app_id, GetType(String), Sub(v) ChangeResponserFields(NameOf(HH_IG_APP_ID), v))
             HH_ASBD_ID = New PropertyValue(asbd, GetType(String), Sub(v) ChangeResponserFields(NameOf(HH_ASBD_ID), v))
@@ -247,9 +248,10 @@ Namespace API.Instagram
             HH_USER_AGENT = New PropertyValue(useragent, GetType(String), Sub(v) ChangeResponserFields(NameOf(HH_USER_AGENT), v))
 
             DownloadTimeline = New PropertyValue(True)
+            DownloadReels = New PropertyValue(True)
             DownloadStories = New PropertyValue(True)
             DownloadStoriesUser = New PropertyValue(True)
-            DownloadTagged = New PropertyValue(False)
+            DownloadTagged = New PropertyValue(True)
 
             RequestsWaitTimer = New PropertyValue(1000)
             RequestsWaitTimerProvider = New TimersChecker(100)
@@ -259,6 +261,7 @@ Namespace API.Instagram
             SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
 
             GetTimeline = New PropertyValue(True)
+            GetReels = New PropertyValue(False)
             GetStories = New PropertyValue(False)
             GetStoriesUser = New PropertyValue(False)
             GetTagged = New PropertyValue(False)

SCrawler/API/Instagram/UserData.vb

@@ -15,6 +15,7 @@ Imports PersonalUtilities.Functions.XML.Base
 Imports PersonalUtilities.Functions.Messaging
 Imports PersonalUtilities.Functions.RegularExpressions
 Imports PersonalUtilities.Tools.Web.Clients
+Imports PersonalUtilities.Tools.Web.Clients.Base
 Imports PersonalUtilities.Tools.Web.Documents.JSON
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Imports UStates = SCrawler.API.Base.UserMedia.States
@@ -24,6 +25,7 @@ Namespace API.Instagram
         Private Const Name_LastCursor As String = "LastCursor"
         Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
         Private Const Name_GetTimeline As String = "GetTimeline"
+        Private Const Name_GetReels As String = "GetReels"
         Private Const Name_GetStories As String = "GetStories"
         Private Const Name_GetStoriesUser As String = "GetStoriesUser"
         Private Const Name_GetTagged As String = "GetTaggedData"
@@ -66,6 +68,7 @@ Namespace API.Instagram
                 Return New EContainer("Post", ID, {New EAttribute(Name_Section, CInt(Section)), New EAttribute(Name_Code, Code)})
             End Function
         End Structure
+        Friend Const Header_FB_LSD As String = "x-fb-lsd"
         Private ReadOnly Property MySiteSettings As SiteSettings
             Get
                 Return DirectCast(HOST.Source, SiteSettings)
@@ -76,6 +79,7 @@ Namespace API.Instagram
         Private LastCursor As String = String.Empty
         Private FirstLoadingDone As Boolean = False
         Friend Property GetTimeline As Boolean = True
+        Friend Property GetReels As Boolean = False
         Friend Property GetStories As Boolean
         Friend Property GetStoriesUser As Boolean
         Friend Property GetTaggedData As Boolean
@@ -94,6 +98,7 @@ Namespace API.Instagram
                     LastCursor = .Value(Name_LastCursor)
                     FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
                     GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
+                    GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value)
                     GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
                     GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value)
                     GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
@@ -103,6 +108,7 @@ Namespace API.Instagram
                     .Add(Name_LastCursor, LastCursor)
                     .Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
                     .Add(Name_GetTimeline, GetTimeline.BoolToInteger)
+                    .Add(Name_GetReels, GetReels.BoolToInteger)
                     .Add(Name_GetStories, GetStories.BoolToInteger)
                     .Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
                     .Add(Name_GetTagged, GetTaggedData.BoolToInteger)
@@ -120,6 +126,7 @@ Namespace API.Instagram
             If Not Obj Is Nothing AndAlso TypeOf Obj Is EditorExchangeOptions Then
                 With DirectCast(Obj, EditorExchangeOptions)
                     GetTimeline = .GetTimeline
+                    GetReels = .GetReels
                     GetStories = .GetStories
                     GetStoriesUser = .GetStoriesUser
                     GetTaggedData = .GetTagged
@@ -253,9 +260,22 @@ Namespace API.Instagram
                 End If
                 If FirstLoadingDone Then LastCursor = String.Empty
                 If Not IsSavedPosts AndAlso MySiteSettings.BaseAuthExists() Then
+                    If CBool(MySiteSettings.DownloadReels.Value) And GetReels Then
+                        s = Sections.Reels
+                        DefaultParser_ElemNode = {"node", "media"}
+                        DownloadData(String.Empty, s, Token)
+                        DefaultParser_ElemNode = Nothing
+                        DownloadReels_SetEnvir = False
+                        ProgressPre.Done()
+                    End If
                     If CBool(MySiteSettings.DownloadStories.Value) And GetStories Then s = Sections.Stories : DownloadData(String.Empty, s, Token) : ProgressPre.Done()
                     If CBool(MySiteSettings.DownloadStoriesUser.Value) And GetStoriesUser Then s = Sections.UserStories : DownloadData(String.Empty, s, Token) : ProgressPre.Done()
-                    If CBool(MySiteSettings.DownloadTagged.Value) And ACheck(MySiteSettings.HashTagged.Value) And GetTaggedData Then s = Sections.Tagged : DownloadData(String.Empty, s, Token) : ProgressPre.Done()
+                    If CBool(MySiteSettings.DownloadTagged.Value) And GetTaggedData Then
+                        s = Sections.Tagged
+                        DownloadData(String.Empty, s, Token)
+                        ProgressPre.Done()
+                        If PostsToReparse.Count > 0 Then DownloadPosts(Token, True)
+                    End If
                 End If
                 If WaitNotificationMode = WNM.SkipTemp Or WaitNotificationMode = WNM.SkipCurrent Then WaitNotificationMode = WNM.Notify
             Catch eex As ExitException
@@ -263,6 +283,8 @@ Namespace API.Instagram
                 errorFound = True
                 Throw ex
             Finally
+                DefaultParser_ElemNode = Nothing
+                DownloadReels_SetEnvir = False
                 E560Thrown = False
                 UpdateResponser()
                 ValidateExtension()
@@ -296,7 +318,7 @@ Namespace API.Instagram
         Protected Overridable Sub Responser_ResponseReceived(ByVal Sender As Object, ByVal e As EventArguments.WebDataResponse)
             Declarations.UpdateResponser(e, Responser)
         End Sub
-        Protected Enum Sections : Timeline : Tagged : Stories : UserStories : SavedPosts : End Enum
+        Protected Enum Sections : Timeline : Reels : Tagged : Stories : UserStories : SavedPosts : End Enum
         Protected Const StoriesFolder As String = "Stories"
         Private Const TaggedFolder As String = "Tagged"
 #Region "429 bypass"
@@ -436,6 +458,7 @@ Namespace API.Instagram
                     ReconfigureAwaiter()
 
                     Try
+                        Dim r$ = String.Empty
                         Dim n As EContainer, nn As EContainer
                         Dim HasNextPage As Boolean = False
                         Dim EndCursor$ = String.Empty
@@ -456,16 +479,17 @@ Namespace API.Instagram
                                 URL = $"https://www.instagram.com/api/v1/feed/user/{NameTrue}/username/?count=50" &
                                         If(Cursor.IsEmptyString, String.Empty, $"&max_id={Cursor}")
                                 ENode = Nothing
+                            Case Sections.Reels
+                                r = DownloadReels(Cursor, Token)
+                                ENode = {"data", "xdt_api__v1__clips__user__connection_v2"}
                             Case Sections.SavedPosts
                                 SavedPostsDownload(String.Empty, Token)
                                 Exit Sub
                             Case Sections.Tagged
-                                Dim h$ = AConvert(Of String)(MySiteSettings.HashTagged.Value, String.Empty)
-                                If h.IsEmptyString Then Throw New ExitException
                                 Dim vars$ = "{""id"":" & ID & ",""first"":50,""after"":""" & Cursor & """}"
                                 vars = SymbolsConverter.ASCII.EncodeSymbolsOnly(vars)
-                                URL = $"https://www.instagram.com/graphql/query/?query_hash={h}&variables={vars}"
+                                URL = $"https://www.instagram.com/graphql/query/?doc_id=17946422347485809&variables={vars}"
-                                ENode = {"data", "user", 0}
+                                ENode = {"data", "user", "edge_user_to_photos_of_you"}
                                 SpecFolder = TaggedFolder
                             Case Sections.Stories
                                 If Not StoriesRequested Then
@@ -493,7 +517,7 @@ Namespace API.Instagram
                         End Select
 
                         'Get response
-                        Dim r$ = Responser.GetResponse(URL,, EDP.ThrowException)
+                        If Not Section = Sections.Reels Then r = Responser.GetResponse(URL,, EDP.ThrowException)
                         MySiteSettings.TooManyRequests(False)
                         RequestsCount += 1
                         ThrowAny(Token)
@@ -515,6 +539,20 @@ Namespace API.Instagram
                                                     HasNextPage = False
                                                 End If
                                             End With
+                                        Case Sections.Reels
+                                            With n
+                                                If .Contains("page_info") Then
+                                                    With .Item("page_info")
+                                                        HasNextPage = .Value("has_next_page").FromXML(Of Boolean)(False)
+                                                        EndCursor = .Value("end_cursor")
+                                                    End With
+                                                Else
+                                                    HasNextPage = False
+                                                End If
+                                                If If(.Item("edges")?.Count, 0) > 0 Then
+                                                    If Not DefaultParser(.Item("edges"), Section, Token, "Reels*") Then Throw New ExitException
+                                                End If
+                                            End With
                                         Case Sections.Tagged
                                             With n
                                                 If .Contains("page_info") Then
@@ -574,18 +612,19 @@ Namespace API.Instagram
                     End Try
                 Loop
             Catch eex2 As ExitException
-                If (Section = Sections.Timeline Or Section = Sections.Tagged) And Not Cursor.IsEmptyString Then Throw eex2
+                If Not Section = Sections.Reels And (Section = Sections.Timeline Or Section = Sections.Tagged) And Not Cursor.IsEmptyString Then Throw eex2
             Catch oex2 As OperationCanceledException When Token.IsCancellationRequested Or oex2.HelpLink = InstAborted
                 If oex2.HelpLink = InstAborted Then HasError = True
             Catch DoEx As Exception
                 ProcessException(DoEx, Token, $"data downloading error [{URL}]",, Section)
             End Try
         End Sub
-        Private Sub DownloadPosts(ByVal Token As CancellationToken)
+        Private Sub DownloadPosts(ByVal Token As CancellationToken, Optional ByVal IsTagged As Boolean = False)
             Dim URL$ = String.Empty
             Dim dValue% = 1
             Dim _Index% = 0
             Dim before%
+            Dim specFolder$ = IIf(IsTagged, "Tagged", String.Empty)
             If PostsToReparse.Count > 0 Then ProgressPre.ChangeMax(PostsToReparse.Count)
             Try
                 Do While dValue = 1
@@ -616,7 +655,7 @@ Namespace API.Instagram
                                             With j("items")
                                                 For Each jj In .Self
                                                     before = _TempMediaList.Count
-                                                    ObtainMedia(jj, PostsToReparse(i).ID)
+                                                    ObtainMedia(jj, PostsToReparse(i).ID, specFolder)
                                                     If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
                                                     If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
                                                 Next
@@ -664,6 +703,7 @@ Namespace API.Instagram
         End Sub
         Protected DefaultParser_ElemNode() As Object = Nothing
         Protected DefaultParser_IgnorePass As Boolean = False
+        Private ReadOnly DefaultParser_PostUrlCreator_Default As Func(Of PostKV, String) = Function(post) $"https://www.instagram.com/p/{post.Code}/"
         Protected DefaultParser_PostUrlCreator As Func(Of PostKV, String) = Function(post) $"https://www.instagram.com/p/{post.Code}/"
         Protected Function DefaultParser(ByVal Items As IEnumerable(Of EContainer), ByVal Section As Sections, ByVal Token As CancellationToken,
                                          Optional ByVal SpecFolder As String = Nothing, Optional ByVal State As UStates = UStates.Unknown,
@@ -713,6 +753,106 @@ Namespace API.Instagram
             End If
         End Function
 #End Region
+#Region "Get reels"
+        Private _GetReels_LSD As String = String.Empty
+        Private _GetReels_dtsg As String = String.Empty
+        Private ReadOnly Property DownloadReels_Tokens_Valid As Boolean
+            Get
+                Return Not _GetReels_LSD.IsEmptyString And Not _GetReels_dtsg.IsEmptyString
+            End Get
+        End Property
+        Private WriteOnly Property DownloadReels_SetEnvir As Boolean
+            Set(ByVal init As Boolean)
+                If init Then
+                    ObtainMedia_SetReelsFunc()
+                    DefaultParser_PostUrlCreator = Function(post) $"{MySiteSettings.GetUserUrl(Me).TrimEnd("/")}/reel/{post.Code}"
+                Else
+                    ObtainMedia_SizeFuncPic = Nothing
+                    ObtainMedia_SizeFuncVid = Nothing
+                    DefaultParser_PostUrlCreator = DefaultParser_PostUrlCreator_Default
+                End If
+            End Set
+        End Property
+        Private Class Responser2 : Inherits Responser
+            Friend Sub New(ByVal Source As Responser)
+                MyBase.New
+                Copy(Source)
+                ErrorProcessor = New ResponserErrorProcessor(Source)
+            End Sub
+        End Class
+        ''' <returns>Response</returns>
+        Private Function DownloadReels(ByVal Cursor As String, ByVal Token As CancellationToken) As String
+            Const requestPattern$ = "https://www.instagram.com/api/graphql?fb_dtsg={0}&fb_api_req_friendly_name=PolarisProfileReelsTabContentQuery&lsd={1}&doc_id=7191572580905225&variables={2}"
+
+            DownloadReels_SetEnvir = True
+
+            If Cursor.IsEmptyString And Not DownloadReels_Tokens_Valid Then GetPageTokens()
+            If Cursor.IsEmptyString And Not DownloadReels_Tokens_Valid Then Throw New ExitException
+
+            Using resp As New Responser2(Responser)
+                Try
+                    resp.Method = "POST"
+                    AddHandler resp.ResponseReceived, AddressOf Responser_ResponseReceived
+                    resp.Headers.Add(Header_FB_LSD, _GetReels_LSD)
+
+                    Dim vars$ = """data"":{""include_feed_video"":true,""page_size"":50,""target_user_id"":""" & ID & """}"
+                    If Not Cursor.IsEmptyString Then vars = $"""after"":""{Cursor}"",""before"":null,{vars},""first"":4,""last"":null"
+                    vars = "{" & vars & "}"
+
+                    Dim url$ = String.Format(requestPattern, _GetReels_dtsg, _GetReels_LSD, SymbolsConverter.ASCII.EncodeSymbolsOnly(vars))
+
+                    Return resp.GetResponse(url,, EDP.ThrowException)
+                Finally
+                    With resp
+                        Responser.Cookies.Update(.Cookies)
+                        With .Headers
+                            If .Contains(SiteSettings.Header_IG_WWW_CLAIM) Then Responser.Headers.Add(SiteSettings.Header_IG_WWW_CLAIM, .Value(SiteSettings.Header_IG_WWW_CLAIM))
+                            If .Contains(SiteSettings.Header_CSRF_TOKEN) Then Responser.Headers.Add(SiteSettings.Header_CSRF_TOKEN, .Value(SiteSettings.Header_CSRF_TOKEN))
+                        End With
+                    End With
+                End Try
+            End Using
+        End Function
+        Private Function GetPageTokens() As Boolean
+            _GetReels_LSD = String.Empty
+            _GetReels_dtsg = String.Empty
+            Try
+                Dim r$ = Responser.GetResponse(MySiteSettings.GetUserUrl(Me),, EDP.ThrowException)
+                If Not r.IsEmptyString Then
+                    Dim rr As RParams = RParams.DM(PageTokenRegexPatternDefault, 0, RegexReturn.List, EDP.ReturnValue)
+                    Dim tokens As List(Of String) = RegexReplace(r, rr)
+                    Dim tt$, ttVal$
+                    If tokens.ListExists Then
+                        With rr
+                            .Match = Nothing
+                            .MatchSub = 1
+                            .WhatGet = RegexReturn.Value
+                        End With
+                        For Each tt In tokens
+                            If Not _GetReels_LSD.IsEmptyString And Not _GetReels_dtsg.IsEmptyString Then
+                                Exit For
+                            Else
+                                ttVal = RegexReplace(tt, rr)
+                                If Not ttVal.IsEmptyString Then
+                                    If ttVal.Contains(":") Then
+                                        If _GetReels_dtsg.IsEmptyString Then _GetReels_dtsg = ttVal
+                                    Else
+                                        If _GetReels_LSD.IsEmptyString Then _GetReels_LSD = ttVal
+                                    End If
+                                End If
+                            End If
+                        Next
+                    End If
+                End If
+            Catch ex As Exception
+                Dim notFound$ = String.Empty
+                If _GetReels_dtsg.IsEmptyString Then notFound.StringAppend(Header_FB_LSD)
+                If _GetReels_LSD.IsEmptyString Then notFound.StringAppend("lsd")
+                LogError(ex, $"failed to update some{IIf(notFound.IsEmptyString, String.Empty, $" ({notFound})")} credentials", EDP.SendToLog)
+            End Try
+            Return DownloadReels_Tokens_Valid
+        End Function
+#End Region
 #Region "Code ID converters"
         Protected Function CodeToID(ByVal Code As String) As String
             Const CodeSymbols$ = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"
@@ -737,15 +877,38 @@ Namespace API.Instagram
         Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
         Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
         Protected ObtainMedia_AllowAbstract As Boolean = False
+        Protected Sub ObtainMedia_SetReelsFunc()
+            ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
+                                          If ss.Value("url").IsEmptyString Then
+                                              Return New Sizes("----", "")
+                                          ElseIf Not ss.Value("width").IsEmptyString Or Not ss.Value("width").IsEmptyString Then
+                                              Return New Sizes(CInt(AConvert(Of Integer)(ss.Value("width"), 0)) +
+                                                               CInt(AConvert(Of Integer)(ss.Value("height"), 0)), ss.Value("url"))
+                                          Else
+                                              Dim rval$ = RegexReplace(ss.Value("url"), ObtainMedia_SizeFuncPic_RegexP)
+                                              If Not rval.IsEmptyString Then Return New Sizes(rval, ss.Value("url"))
+                                              rval = RegexReplace(ss.Value("url"), ObtainMedia_SizeFuncPic_RegexS)
+                                              If Not rval.IsEmptyString Then Return New Sizes(AConvert(Of Integer)(rval, 1) * -1, ss.Value("url"))
+                                              Return New Sizes(10000, ss.Value("url"))
+                                          End If
+                                      End Function
+            ObtainMedia_SizeFuncVid = Function(ss) If(ss.Value("url").IsEmptyString, New Sizes("----", ""), New Sizes(10000, ss.Value("url")))
+        End Sub
         Protected Sub ObtainMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal SpecialFolder As String = Nothing,
                                   Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
                                   Optional ByVal PostOriginUrl As String = Nothing,
                                   Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0)
             Try
+                Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
+                                                                  Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
+                                                                  Dim h% = AConvert(Of Integer)(_ss.Value("height"), 0)
+                                                                  'Return w + h
+                                                                  Return Math.Max(w, h)
+                                                              End Function
                 Dim wrongData As Predicate(Of Sizes) = Function(_ss) _ss.HasError Or _ss.Data.IsEmptyString
                 Dim img As Predicate(Of EContainer) = Function(_img) Not _img.Name.IsEmptyString AndAlso _img.Name.StartsWith("image_versions") AndAlso _img.Count > 0
                 Dim vid As Predicate(Of EContainer) = Function(_vid) Not _vid.Name.IsEmptyString AndAlso _vid.Name.StartsWith("video_versions") AndAlso _vid.Count > 0
-                Dim ss As Func(Of EContainer, Sizes) = Function(_ss) New Sizes(_ss.Value("width"), _ss.Value("url"))
+                Dim ss As Func(Of EContainer, Sizes) = Function(_ss) New Sizes(maxSize(_ss), _ss.Value("url"))
                 Dim ssVid As Func(Of EContainer, Sizes) = ss
                 Dim ssPic As Func(Of EContainer, Sizes) = ss
                 Dim mDate As Func(Of EContainer, String) = Function(ByVal elem As EContainer) As String
@@ -1041,11 +1204,12 @@ Namespace API.Instagram
                 Dim s As Sections = DirectCast(Section, Sections)
                 Select Case s
                     Case Sections.Timeline : MySiteSettings.DownloadTimeline.Value = False
+                    Case Sections.Reels : MySiteSettings.DownloadReels.Value = False
+                    Case Sections.Tagged : MySiteSettings.DownloadTagged.Value = False
                     Case Sections.Stories, Sections.UserStories
                         MySiteSettings.DownloadTimeline.Value = False
                         MySiteSettings.DownloadStories.Value = False
                         MySiteSettings.DownloadStoriesUser.Value = False
-                    Case Else : MySiteSettings.DownloadTagged.Value = False
                 End Select
                 MyMainLOG = $"[{s}] downloading is disabled until you update your credentials".ToUpper
             End If

SCrawler/API/LPSG/UserData.vb

@@ -106,9 +106,12 @@ Namespace API.LPSG
         End Sub
         Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False,
                                                           Optional ByVal EObj As Object = Nothing) As Integer
-            If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
+            If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then '503
                 MyMainLOG = $"{ToStringForLog()}: LPSG not available"
                 Return 1
+            ElseIf Responser.StatusCode = Net.HttpStatusCode.NotFound Then '404
+                UserExists = False
+                Return 1
             Else
                 Return 0
             End If

SCrawler/API/OnlyFans/OFResources.Designer.vb

@@ -0,0 +1,77 @@
+'------------------------------------------------------------------------------
+' <auto-generated>
+'     This code was generated by a tool.
+'     Runtime Version:4.0.30319.42000
+'
+'     Changes to this file may cause incorrect behavior and will be lost if
+'     the code is regenerated.
+' </auto-generated>
+'------------------------------------------------------------------------------
+
+Option Strict On
+Option Explicit On
+
+Imports System
+
+Namespace My.Resources
+    
+    'This class was auto-generated by the StronglyTypedResourceBuilder
+    'class via a tool like ResGen or Visual Studio.
+    'To add or remove a member, edit your .ResX file then rerun ResGen
+    'with the /str option, or rebuild your VS project.
+    '''<summary>
+    '''  A strongly-typed resource class, for looking up localized strings, etc.
+    '''</summary>
+    <Global.System.CodeDom.Compiler.GeneratedCodeAttribute("System.Resources.Tools.StronglyTypedResourceBuilder", "16.0.0.0"),  _
+     Global.System.Diagnostics.DebuggerNonUserCodeAttribute(),  _
+     Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute()>  _
+    Friend Class OFResources
+        
+        Private Shared resourceMan As Global.System.Resources.ResourceManager
+        
+        Private Shared resourceCulture As Global.System.Globalization.CultureInfo
+        
+        <Global.System.Diagnostics.CodeAnalysis.SuppressMessageAttribute("Microsoft.Performance", "CA1811:AvoidUncalledPrivateCode")>  _
+        Friend Sub New()
+            MyBase.New
+        End Sub
+        
+        '''<summary>
+        '''  Returns the cached ResourceManager instance used by this class.
+        '''</summary>
+        <Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)>  _
+        Friend Shared ReadOnly Property ResourceManager() As Global.System.Resources.ResourceManager
+            Get
+                If Object.ReferenceEquals(resourceMan, Nothing) Then
+                    Dim temp As Global.System.Resources.ResourceManager = New Global.System.Resources.ResourceManager("SCrawler.OFResources", GetType(OFResources).Assembly)
+                    resourceMan = temp
+                End If
+                Return resourceMan
+            End Get
+        End Property
+        
+        '''<summary>
+        '''  Overrides the current thread's CurrentUICulture property for all
+        '''  resource lookups using this strongly typed resource class.
+        '''</summary>
+        <Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)>  _
+        Friend Shared Property Culture() As Global.System.Globalization.CultureInfo
+            Get
+                Return resourceCulture
+            End Get
+            Set
+                resourceCulture = value
+            End Set
+        End Property
+        
+        '''<summary>
+        '''  Looks up a localized resource of type System.Byte[].
+        '''</summary>
+        Friend Shared ReadOnly Property OFScraperConfigPattern() As Byte()
+            Get
+                Dim obj As Object = ResourceManager.GetObject("OFScraperConfigPattern", resourceCulture)
+                Return CType(obj,Byte())
+            End Get
+        End Property
+    End Class
+End Namespace

SCrawler/API/OnlyFans/OFResources.resx

@@ -0,0 +1,124 @@
+<?xml version="1.0" encoding="utf-8"?>
+<root>
+  <!-- 
+    Microsoft ResX Schema 
+    
+    Version 2.0
+    
+    The primary goals of this format is to allow a simple XML format 
+    that is mostly human readable. The generation and parsing of the 
+    various data types are done through the TypeConverter classes 
+    associated with the data types.
+    
+    Example:
+    
+    ... ado.net/XML headers & schema ...
+    <resheader name="resmimetype">text/microsoft-resx</resheader>
+    <resheader name="version">2.0</resheader>
+    <resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
+    <resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
+    <data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
+    <data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
+    <data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
+        <value>[base64 mime encoded serialized .NET Framework object]</value>
+    </data>
+    <data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+        <value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
+        <comment>This is a comment</comment>
+    </data>
+                
+    There are any number of "resheader" rows that contain simple 
+    name/value pairs.
+    
+    Each data row contains a name, and value. The row also contains a 
+    type or mimetype. Type corresponds to a .NET class that support 
+    text/value conversion through the TypeConverter architecture. 
+    Classes that don't support this are serialized and stored with the 
+    mimetype set.
+    
+    The mimetype is used for serialized objects, and tells the 
+    ResXResourceReader how to depersist the object. This is currently not 
+    extensible. For a given mimetype the value must be set accordingly:
+    
+    Note - application/x-microsoft.net.object.binary.base64 is the format 
+    that the ResXResourceWriter will generate, however the reader can 
+    read any of the formats listed below.
+    
+    mimetype: application/x-microsoft.net.object.binary.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
+            : and then encoded with base64 encoding.
+    
+    mimetype: application/x-microsoft.net.object.soap.base64
+    value   : The object must be serialized with 
+            : System.Runtime.Serialization.Formatters.Soap.SoapFormatter
+            : and then encoded with base64 encoding.
+
+    mimetype: application/x-microsoft.net.object.bytearray.base64
+    value   : The object must be serialized into a byte array 
+            : using a System.ComponentModel.TypeConverter
+            : and then encoded with base64 encoding.
+    -->
+  <xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
+    <xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
+    <xsd:element name="root" msdata:IsDataSet="true">
+      <xsd:complexType>
+        <xsd:choice maxOccurs="unbounded">
+          <xsd:element name="metadata">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" />
+              </xsd:sequence>
+              <xsd:attribute name="name" use="required" type="xsd:string" />
+              <xsd:attribute name="type" type="xsd:string" />
+              <xsd:attribute name="mimetype" type="xsd:string" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="assembly">
+            <xsd:complexType>
+              <xsd:attribute name="alias" type="xsd:string" />
+              <xsd:attribute name="name" type="xsd:string" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="data">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+                <xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
+              <xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
+              <xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
+              <xsd:attribute ref="xml:space" />
+            </xsd:complexType>
+          </xsd:element>
+          <xsd:element name="resheader">
+            <xsd:complexType>
+              <xsd:sequence>
+                <xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
+              </xsd:sequence>
+              <xsd:attribute name="name" type="xsd:string" use="required" />
+            </xsd:complexType>
+          </xsd:element>
+        </xsd:choice>
+      </xsd:complexType>
+    </xsd:element>
+  </xsd:schema>
+  <resheader name="resmimetype">
+    <value>text/microsoft-resx</value>
+  </resheader>
+  <resheader name="version">
+    <value>2.0</value>
+  </resheader>
+  <resheader name="reader">
+    <value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <resheader name="writer">
+    <value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </resheader>
+  <assembly alias="System.Windows.Forms" name="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
+  <data name="OFScraperConfigPattern" type="System.Resources.ResXFileRef, System.Windows.Forms">
+    <value>OFScraperConfigPattern.json;System.Byte[], mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
+  </data>
+</root>

SCrawler/API/OnlyFans/OFScraperConfigPattern.json

@@ -0,0 +1,61 @@
+{
+    "config": {
+        "main_profile": "main_profile",
+        "metadata": "{configpath}/{profile}/.data/{model_username}_{model_id}",
+        "discord": "",
+        "file_options": {
+            "save_location": "",
+            "dir_format": "",
+            "file_format": "{filename}.{ext}",
+            "textlength": 0,
+            "space-replacer": " ",
+            "date": "YYYY-MM-DD"
+        },
+        "download_options": {
+            "file_size_limit": 0,
+            "file_size_min": 0,
+            "filter": [
+                "Images",
+                "Audios",
+                "Videos"
+            ],
+            "auto_resume": false
+        },
+        "binary_options": {
+            "mp4decrypt": "",
+            "ffmpeg": ""
+        },
+        "cdm_options": {
+            "private-key": null,
+            "client-id": null,
+            "key-mode-default": "cdrm",
+            "keydb_api": ""
+        },
+        "performance_options": {
+            "download-sems": 6,
+            "maxfile-sem": 0,
+            "threads": 5
+        },
+        "advanced_options": {
+            "code-execution": false,
+            "dynamic-mode-default": "deviint",
+            "backend": "aio",
+            "downloadbars": false,
+            "cache-mode": "sqlite",
+            "appendlog": true,
+            "custom": null,
+            "sanitize_text": false,
+            "avatar": true
+        },
+        "responsetype": {
+            "timeline": "Posts",
+            "message": "Messages",
+            "archived": "Archived",
+            "paid": "Messages",
+            "stories": "Stories",
+            "highlights": "Stories",
+            "profile": "Profile",
+            "pinned": "Posts"
+        }
+    }
+}

SCrawler/API/OnlyFans/SiteSettings.vb

@@ -26,8 +26,8 @@ Namespace API.OnlyFans
 #Region "Headers"
         Private Const HeaderBrowser As String = "sec-ch-ua"
         Private Const HeaderUserID As String = "User-Id"
-        Private Const HeaderXBC As String = "X-Bc"
+        Friend Const HeaderXBC As String = "X-Bc"
-        Private Const HeaderAppToken As String = "App-Token"
+        Friend Const HeaderAppToken As String = "App-Token"
         <PropertyOption(ControlText:=HeaderUserID, AllowNull:=False), PClonable(Clone:=False)>
         Friend ReadOnly Property HH_USER_ID As PropertyValue
         <PropertyOption(ControlText:=HeaderXBC, AllowNull:=False), PClonable(Clone:=False)>
@@ -37,7 +37,7 @@ Namespace API.OnlyFans
         <PropertyOption(ControlText:=HeaderBrowser, ControlToolTip:="Can be null", AllowNull:=True), PClonable>
         Private ReadOnly Property HH_BROWSER As PropertyValue
         <PropertyOption(AllowNull:=False), PClonable>
-        Private ReadOnly Property UserAgent As PropertyValue
+        Friend ReadOnly Property UserAgent As PropertyValue
         Private Sub UpdateHeader(ByVal PropertyName As String, ByVal Value As String)
             Dim hName$ = String.Empty
             Dim isUserAgent As Boolean = False
@@ -78,6 +78,42 @@ Namespace API.OnlyFans
                                         "Change this value only if you know what you are doing."), PXML, PClonable>
         Friend ReadOnly Property DynamicRules As PropertyValue
 #End Region
+#Region "OFScraper"
+        <PClonable, PXML("OFScraperPath")> Private ReadOnly Property OFScraperPath_XML As PropertyValue
+        <PropertyOption(ControlText:="OF-Scraper path", ControlToolTip:="The path to the 'ofscraper.exe'")>
+        Friend ReadOnly Property OFScraperPath As PropertyValue
+            Get
+                If Not DefaultInstance Is Nothing Then
+                    Return DirectCast(DefaultInstance, SiteSettings).OFScraperPath_XML
+                Else
+                    Return OFScraperPath_XML
+                End If
+            End Get
+        End Property
+        <PClonable, PXML("OFScraperMP4decrypt")> Private ReadOnly Property OFScraperMP4decrypt_XML As PropertyValue
+        <PropertyOption(ControlText:="mp4decrypt path", ControlToolTip:="The path to the 'mp4decrypt.exe'")>
+        Friend ReadOnly Property OFScraperMP4decrypt As PropertyValue
+            Get
+                If Not DefaultInstance Is Nothing Then
+                    Return DirectCast(DefaultInstance, SiteSettings).OFScraperMP4decrypt_XML
+                Else
+                    Return OFScraperMP4decrypt_XML
+                End If
+            End Get
+        End Property
+        Friend Const KeyModeDefault_Default As String = "cdrm"
+        <PClonable, PXML("KeyModeDefault")> Private ReadOnly Property KeyModeDefault_XML As PropertyValue
+        <PropertyOption(ControlText:="key-mode-default")>
+        Friend ReadOnly Property KeyModeDefault As PropertyValue
+            Get
+                If Not DefaultInstance Is Nothing Then
+                    Return DirectCast(DefaultInstance, SiteSettings).KeyModeDefault_XML
+                Else
+                    Return KeyModeDefault_XML
+                End If
+            End Get
+        End Property
+#End Region
 #End Region
 #Region "Initializer"
         Friend Sub New(ByVal AccName As String, ByVal Temp As Boolean)
@@ -112,11 +148,27 @@ Namespace API.OnlyFans
             DownloadChatMedia = New PropertyValue(True)
 
             LastDateUpdated_XML = New PropertyValue(Now.AddYears(-1), GetType(Date))
-            UseOldAuthRules = New PropertyValue(False)
+            'URGENT: OF [UseOldAuthRules = True]
+            UseOldAuthRules = New PropertyValue(True)
             DynamicRulesUpdateInterval = New PropertyValue(60 * 24)
             DynamicRulesUpdateIntervalProvider = New FieldsCheckerProviderSimple(Function(v) IIf(AConvert(Of Integer)(v, 0) > 0, v, Nothing),
                                                                                  "The value of [{0}] field must be greater than 0")
             DynamicRules = New PropertyValue(String.Empty, GetType(String))
+            OFScraperPath_XML = New PropertyValue(String.Empty, GetType(String))
+            If ACheck(OFScraperPath_XML.Value) Then
+                Dim f As SFile = OFScraperPath_XML.Value
+                If Not f.Exists AndAlso f.Exists(SFO.Path, False) Then
+                    With SFile.GetFiles(f, "*.exe",, EDP.ReturnValue)
+                        If .ListExists Then
+                            f = .FirstOrDefault(Function(ff) ff.Name.StringToLower.StartsWith("ofscraper"))
+                            If f.Exists Then OFScraperPath_XML.Value = f.ToString
+                        End If
+                    End With
+                End If
+            End If
+            OFScraperMP4decrypt_XML = New PropertyValue(String.Empty, GetType(String))
+            KeyModeDefault_XML = New PropertyValue(KeyModeDefault_Default)
+
             UserRegex = RParams.DMS("onlyfans.com/([\w\._]+)", 1, EDP.ReturnValue)
             UrlPatternUser = "https://onlyfans.com/{0}"
             ImageVideoContains = "onlyfans.com"
@@ -151,6 +203,7 @@ Namespace API.OnlyFans
         End Sub
 #End Region
 #Region "GetUserUrl, GetUserPostUrl, UserOptions"
+        Friend Const UserPostPattern As String = "https://onlyfans.com/{0}/{1}"
         Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String
             Return String.Format(UrlPatternUser, If(User.ID.IsEmptyString, User.Name, $"u{User.ID}"))
         End Function
@@ -168,7 +221,7 @@ Namespace API.OnlyFans
                 If p.IsEmptyString Then
                     Return GetUserUrl(User)
                 Else
-                    Return String.Format("https://onlyfans.com/{0}/{1}", p, If(User.ID.IsEmptyString, User.Name, $"u{User.ID}"))
+                    Return String.Format(UserPostPattern, p, If(User.ID.IsEmptyString, User.Name, $"u{User.ID}"))
                 End If
             Else
                 Return String.Empty

SCrawler/API/OnlyFans/UserData.vb

@@ -7,10 +7,12 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
+Imports System.Text.RegularExpressions
 Imports SCrawler.API.Base
 Imports SCrawler.API.YouTube.Objects
 Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Clients.EventArguments
 Imports PersonalUtilities.Tools.Web.Cookies
@@ -65,11 +67,20 @@ Namespace API.OnlyFans
 #Region "Initializer"
         Friend Sub New()
             HighlightsList = New List(Of String)
+            UseInternalDownloadFileFunction = True
         End Sub
 #End Region
 #Region "Download functions"
+        Private _OFScraperExists As Boolean = False
+        Private OFSCache As CacheKeeper = Nothing
+        Private _AbsMediaIndex As Integer = 0
+        Private Sub ValidateOFScraper()
+            _OFScraperExists = ACheck(MySettings.OFScraperPath.Value) AndAlso CStr(MySettings.OFScraperPath.Value).CSFile.Exists
+        End Sub
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             If Not MySettings.SessionAborted Then
+                ValidateOFScraper()
+                _AbsMediaIndex = 0
                 If Not CCookie Is Nothing Then CCookie.Dispose()
                 CCookie = Responser.Cookies.Copy
                 Responser.Cookies.Clear()
@@ -307,8 +318,8 @@ Namespace API.OnlyFans
 #End Region
         Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
                                         Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
-                                        Optional ByVal SpecFolder As String = Nothing) As List(Of UserMedia)
+                                        Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing) As List(Of UserMedia)
-            Dim postUrl$, ext$
+            Dim postUrl$, postUrlBase$, ext$
             Dim t As UTypes
             Dim mList As New List(Of UserMedia)
             Result = False
@@ -320,16 +331,27 @@ Namespace API.OnlyFans
                         Else
                             postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
                         End If
+                        postUrlBase = String.Empty
                         Select Case m.Value("type")
                             Case "photo" : t = UTypes.Picture : ext = "jpg"
-                            Case "video" : t = UTypes.Video : ext = "mp4"
+                            Case "video"
+                                t = UTypes.Video
+                                ext = "mp4"
+                                If postUrl.IsEmptyString And Not IsHL Then
+                                    t = UTypes.VideoPre
+                                    _AbsMediaIndex += 1
+                                    If Not PostUserID.IsEmptyString And IsSingleObjectDownload Then _
+                                       postUrlBase = String.Format(SiteSettings.UserPostPattern, PostID, $"u{PostUserID}")
+                                End If
                             Case Else : t = UTypes.Undefined : ext = String.Empty
                         End Select
-                        If Not t = UTypes.Undefined And Not postUrl.IsEmptyString Then
+                        If Not t = UTypes.Undefined And (Not postUrl.IsEmptyString Or t = UTypes.VideoPre) Then
-                            Dim media As New UserMedia(postUrl, t) With {
+                            Dim media As New UserMedia(postUrl.IfNullOrEmpty(IIf(t = UTypes.VideoPre, $"{t}{_AbsMediaIndex}", String.Empty)), t) With {
                                 .Post = New UserPost(PostID, AConvert(Of Date)(PostDate, DateProvider, Nothing)),
                                 .SpecialFolder = SpecFolder
                             }
+                            If postUrlBase.IsEmptyString And Not IsSingleObjectDownload Then postUrlBase = GetPostUrl(Me, media)
+                            If Not postUrlBase.IsEmptyString Then media.URL_BASE = postUrlBase
                             media.File.Extension = ext
                             Result = True
                             mList.Add(media)
@@ -387,7 +409,7 @@ Namespace API.OnlyFans
                                                                                 End Function
                     Dim mList As List(Of UserMedia)
                     Dim mediaResult As Boolean
-                    Dim r$, path$, postDate$
+                    Dim r$, path$, postDate$, postUserID$
                     Dim j As EContainer
                     ProgressPre.ChangeMax(_ContentList.Count)
                     For i% = 0 To _ContentList.Count - 1
@@ -404,8 +426,9 @@ Namespace API.OnlyFans
                                         j = JsonDocument.Parse(r)
                                         If Not j Is Nothing Then
                                             postDate = j.Value("postedAt")
+                                            postUserID = j.Value({"author"}, "id")
                                             mediaResult = False
-                                            mList = TryCreateMedia(j, m.Post.ID, postDate, mediaResult)
+                                            mList = TryCreateMedia(j, m.Post.ID, postDate, mediaResult,,, postUserID)
                                             If mediaResult Then
                                                 _TempMediaList.ListAddList(mList.ListForEachCopy(stateRefill, True), LNC)
                                                 rList.Add(i)
@@ -531,10 +554,145 @@ Namespace API.OnlyFans
             Return result
         End Function
 #End Region
+#Region "OFScraper support"
+        Private Function OFS_DownloadFile(ByVal URL As String, ByVal Token As CancellationToken) As List(Of SFile)
+            Try
+                Const requestPattern$ = """{0}"" manual --config ""{1}"" --url {2}"
+                Dim conf As SFile = OFS_CreateConfig()
+                If conf.Exists Then
+                    Dim command$ = String.Format(requestPattern, MySettings.OFScraperPath.Value, conf, URL)
+                    '#If DEBUG Then
+                    'Debug.WriteLine(command)
+                    '#End If
+                    Using b As New TokenBatch(Token) : b.Execute(command) : End Using
+                    Return SFile.GetFiles(conf, "*.mp4", IO.SearchOption.AllDirectories, EDP.ReturnValue)
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "OnlyFans.UserData.OFS_DownloadFile", Nothing)
+            End Try
+        End Function
+        Private Function OFS_CreateConfig() As SFile
+            Try
+                Const confMainPattern$ = "{0}"": ""([^""]*)"""
+                If OFSCache Is Nothing Then OFSCache = If(IsSingleObjectDownload, Settings.Cache.NewInstance, CreateCache())
+                Dim currentCache As CacheKeeper = OFSCache.NewInstance
+                currentCache.Validate()
+                Dim cacheRoot As SFile = currentCache.NewPath
+                cacheRoot.Exists(SFO.Path, True, EDP.ThrowException)
+                Dim f As SFile = $"{SettingsFolderName}\OFScraperConfigPattern.json"
+                Dim configText$
+                If Not f.Exists Then
+                    configText = Text.Encoding.UTF8.GetString(My.Resources.OFResources.OFScraperConfigPattern)
+                    TextSaver.SaveTextToFile(configText, f, True)
+                End If
+                If f.Exists Then
+                    Dim replaceValue$ = String.Empty
+                    Dim rp As RParams = RParams.DMS(String.Empty, 1, RegexReturn.Replace, RegexOptions.IgnoreCase,
+                                                    CType(Function(input) replaceValue, Func(Of String, String)), String.Empty, EDP.ReturnValue)
+                    Dim ff As SFile
+                    configText = f.GetText
+                    Dim updateConf As Action(Of String, String) = Sub(ByVal patternValue As String, ByVal __replaceValue As String)
+                                                                      rp.Pattern = String.Format(confMainPattern, patternValue)
+                                                                      rp.Nothing = configText
+                                                                      replaceValue = __replaceValue
+                                                                      configText = RegexReplace(configText, rp)
+                                                                  End Sub
+                    If Not configText.IsEmptyString Then
+                        updateConf("save_location", cacheRoot.PathNoSeparator.Replace("\", "/"))
+                        If ACheck(MySettings.OFScraperMP4decrypt.Value) Then
+                            ff = CStr(MySettings.OFScraperMP4decrypt.Value)
+                            If ff.Exists Then updateConf("mp4decrypt", ff.ToString.Replace("\", "/"))
+                        End If
+                        If Settings.FfmpegFile.Exists Then updateConf("ffmpeg", Settings.FfmpegFile.File.ToString.Replace("\", "/"))
+                        updateConf("key-mode-default", CStr(MySettings.KeyModeDefault.Value).IfNullOrEmpty(SiteSettings.KeyModeDefault_Default))
+                        f = currentCache
+                        f.Name = "config"
+                        f.Extension = "json"
+                        If TextSaver.SaveTextToFile(configText, f, True).Exists AndAlso OFS_CreateAuth(currentCache) Then Return f
+                    End If
+                End If
+                Return Nothing
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "OnlyFans.UserData.OFS_CreateConfig", Nothing)
+            End Try
+        End Function
+        Private Function OFS_CreateAuth(ByVal DestinationPath As SFile) As Boolean
+            Const authText$ = """user_agent"": ""{0}"",""app-token"": ""{1}"",""x-bc"": ""{2}"",""auth_id"": ""{3}"",""sess"": ""{4}"",""auth_uid_"": ""{3}"",""cookie"": ""{5}"""
+            Try
+                Dim sess$ = If(If(CCookie, Responser.Cookies).FirstOrDefault(Function(c) c.Name.StringToLower = "sess")?.Value, String.Empty)
+                Dim outText$ = "{""auth"":{" &
+                               String.Format(authText,
+                                             MySettings.UserAgent.Value,
+                                             Responser.Headers.Value(SiteSettings.HeaderAppToken),
+                                             Responser.Headers.Value(SiteSettings.HeaderXBC),
+                                             MySettings.HH_USER_ID.Value,
+                                             sess,
+                                             If(CCookie, Responser.Cookies).ToString()) &
+                                             "}}"
+                If DestinationPath.Exists(SFO.Path, False) Then
+                    Dim f As SFile = $"{DestinationPath.PathWithSeparator}main_profile\auth.json"
+                    Return TextSaver.SaveTextToFile(outText, f, True).Exists
+                End If
+                Return False
+            Catch ex As Exception
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "OnlyFans.UserData.OFS_CreateAuth", False)
+            End Try
+        End Function
+#End Region
 #Region "DownloadContent"
+        Private OFSPostFiles As Dictionary(Of String, List(Of SFile)) = Nothing
         Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken)
             DownloadContentDefault(Token)
+            OFSCache.DisposeIfReady
+            OFSPostFiles.ListClearDispose
         End Sub
+        Protected Overrides Function ValidateDownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByRef Interrupt As Boolean) As Boolean
+            Return Media.Type = UTypes.VideoPre
+        End Function
+        Protected Overrides Function DownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile,
+                                                  ByVal Token As CancellationToken) As SFile
+            ValidateOFScraper()
+            If _OFScraperExists Then
+                If OFSPostFiles Is Nothing Then OFSPostFiles = New Dictionary(Of String, List(Of SFile))
+                If IsSingleObjectDownload Then
+                    URL = Media.URL_BASE
+                Else
+                    URL = GetPostUrl(Me, Media)
+                End If
+                If Not URL.IsEmptyString Then
+                    Dim f As SFile = Nothing
+                    If OFSPostFiles.Count > 0 AndAlso OFSPostFiles.ContainsKey(Media.Post.ID) AndAlso OFSPostFiles(Media.Post.ID).Count > 0 Then
+                        f = OFSPostFiles(Media.Post.ID)(0)
+                        OFSPostFiles(Media.Post.ID).RemoveAt(0)
+                    Else
+                        Dim files As List(Of SFile) = OFS_DownloadFile(URL, Token)
+                        If files.ListExists Then
+                            Dim ff As SFile
+                            For i% = files.Count - 1 To 0 Step -1
+                                ff = files(i)
+                                DestinationFile.Name = ff.Name
+                                DestinationFile.Extension = ff.Extension
+                                If SFile.Move(ff, DestinationFile,,,, EDP.ThrowException) Then
+                                    files(i) = DestinationFile
+                                Else
+                                    files.RemoveAt(i)
+                                End If
+                            Next
+                            If files.Count > 0 Then
+                                f = files(0)
+                                files.RemoveAt(0)
+                                If files.Count > 0 Then OFSPostFiles.Add(Media.Post.ID, files)
+                            End If
+                        End If
+                    End If
+                    Return f
+                End If
+                Return Nothing
+            Else
+                Throw New InvalidProgramException("OF-Scraper not found")
+            End If
+        End Function
 #End Region
 #Region "DownloadingException"
         Private _DownloadingException_AuthFileUpdate As Boolean = False
@@ -546,7 +704,7 @@ Namespace API.OnlyFans
                     Return 2
                 Else
                     MySettings.SessionAborted = True
-                    MyMainLOG = $"{ToStringForLog()}: OnlyFans credentials expired"
+                    MyMainLOG = $"{ToStringForLog()} [{CInt(Responser.StatusCode)}]: OnlyFans credentials expired"
                     Return 1
                 End If
             ElseIf Responser.StatusCode = Net.HttpStatusCode.NotFound Then '404
@@ -558,7 +716,7 @@ Namespace API.OnlyFans
                 Return 1
             ElseIf Responser.StatusCode = Net.HttpStatusCode.Unauthorized Then '401
                 MySettings.SessionAborted = True
-                MyMainLOG = $"{ToStringForLog()}: OnlyFans credentials expired"
+                MyMainLOG = $"{ToStringForLog()} [{CInt(Responser.StatusCode)}]: OnlyFans credentials expired"
                 Return 1
             Else
                 Return 0
@@ -567,7 +725,13 @@ Namespace API.OnlyFans
 #End Region
 #Region "IDisposable Support"
         Protected Overrides Sub Dispose(ByVal disposing As Boolean)
-            If Not disposedValue And disposing Then CCookie.DisposeIfReady(False) : CCookie = Nothing : HighlightsList.Clear()
+            If Not disposedValue And disposing Then
+                CCookie.DisposeIfReady(False)
+                CCookie = Nothing
+                HighlightsList.Clear()
+                OFSCache.DisposeIfReady
+                OFSPostFiles.ListClearDispose
+            End If
             MyBase.Dispose(disposing)
         End Sub
 #End Region

SCrawler/API/ThreadsNet/UserData.vb

@@ -18,14 +18,11 @@ Imports IGS = SCrawler.API.Instagram.SiteSettings
 Namespace API.ThreadsNet
     Friend Class UserData : Inherits Instagram.UserData
 #Region "Declarations"
-        Friend Const Header_FB_LSD As String = "x-fb-lsd"
         Private ReadOnly Property MySettings As SiteSettings
             Get
                 Return HOST.Source
             End Get
         End Property
-        Private ReadOnly ObtainMedia_SizeFuncPic_RegexP As RParams = RParams.DMS("_p(\d+)x(\d+)", 1, EDP.ReturnValue)
-        Private ReadOnly ObtainMedia_SizeFuncPic_RegexS As RParams = RParams.DMS("_s(\d+)x(\d+)", 1, EDP.ReturnValue)
         Private ReadOnly DefaultParser_ElemNode_Default() As Object = {"node", "thread_items", 0, "post"}
         Private OPT_LSD As String = String.Empty
         Private OPT_FB_DTSG As String = String.Empty
@@ -48,20 +45,7 @@ Namespace API.ThreadsNet
 #End Region
 #Region "Initializer"
         Friend Sub New()
-            ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
+            ObtainMedia_SetReelsFunc()
-                                          If ss.Value("url").IsEmptyString Then
-                                              Return New Sizes("----", "")
-                                          ElseIf Not ss.Value("width").IsEmptyString Then
-                                              Return New Sizes(ss.Value("height").IfNullOrEmpty(ss.Value("width")), ss.Value("url"))
-                                          Else
-                                              Dim rval$ = RegexReplace(ss.Value("url"), ObtainMedia_SizeFuncPic_RegexP)
-                                              If Not rval.IsEmptyString Then Return New Sizes(rval, ss.Value("url"))
-                                              rval = RegexReplace(ss.Value("url"), ObtainMedia_SizeFuncPic_RegexS)
-                                              If Not rval.IsEmptyString Then Return New Sizes(AConvert(Of Integer)(rval, 1) * -1, ss.Value("url"))
-                                              Return New Sizes(10000, ss.Value("url"))
-                                          End If
-                                      End Function
-            ObtainMedia_SizeFuncVid = Function(ss) If(ss.Value("url").IsEmptyString, New Sizes("----", ""), New Sizes(10000, ss.Value("url")))
             ObtainMedia_AllowAbstract = True
             DefaultParser_ElemNode = DefaultParser_ElemNode_Default
             DefaultParser_PostUrlCreator = Function(post) $"https://www.threads.net/@{NameTrue}/post/{post.Code}"
@@ -174,7 +158,7 @@ Namespace API.ThreadsNet
                 Dim rr As RParams
                 Dim tt$, ttVal$
                 If Not r.IsEmptyString Then
-                    rr = RParams.DM("\[\],{""token"":""(.*?)""},\d+\]", 0, RegexReturn.List, EDP.ReturnValue)
+                    rr = RParams.DM(Instagram.PageTokenRegexPatternDefault, 0, RegexReturn.List, EDP.ReturnValue)
                     Dim tokens As List(Of String) = RegexReplace(r, rr)
                     If tokens.ListExists Then
                         With rr

SCrawler/API/TikTok/SiteSettings.vb

@@ -22,6 +22,10 @@ Namespace API.TikTok
         Friend ReadOnly Property TitleUseNativeSTD As PropertyValue
         <PropertyOption(ControlText:="Add video ID to video title"), PXML, PClonable>
         Friend ReadOnly Property TitleAddVideoID As PropertyValue
+        <PropertyOption(ControlText:="Use regex to clean video title"), PXML, PClonable>
+        Friend ReadOnly Property TitleUseRegexForTitle As PropertyValue
+        <PropertyOption(ControlText:="Title regex", ControlToolTip:="Regex to clean video title"), PXML, PClonable>
+        Friend ReadOnly Property TitleUseRegexForTitle_Value As PropertyValue
         <PropertyOption(ControlText:="Use video date as file date",
                         ControlToolTip:="Set the file date to the date the video was added (website) (if available)."), PXML, PClonable>
         Friend ReadOnly Property UseParsedVideoDate As PropertyValue
@@ -31,6 +35,8 @@ Namespace API.TikTok
             TitleUseNative = New PropertyValue(True)
             TitleUseNativeSTD = New PropertyValue(False)
             TitleAddVideoID = New PropertyValue(True)
+            TitleUseRegexForTitle = New PropertyValue(False)
+            TitleUseRegexForTitle_Value = New PropertyValue(String.Empty, GetType(String))
             UseParsedVideoDate = New PropertyValue(True)
             UseNetscapeCookies = True
             UrlPatternUser = "https://www.tiktok.com/@{0}/"

SCrawler/API/TikTok/UserData.vb

@@ -20,6 +20,9 @@ Namespace API.TikTok
         Private Const Name_TitleUseNative As String = "TitleUseNative"
         Private Const Name_TitleAddVideoID As String = "TitleAddVideoID"
         Private Const Name_LastDownloadDate As String = "LastDownloadDate"
+        Private Const Name_TitleUseRegexForTitle As String = "TitleUseRegexForTitle"
+        Private Const Name_TitleUseRegexForTitle_Value As String = "TitleUseRegexForTitle_Value"
+        Private Const Name_TitleUseGlobalRegexOptions As String = "TitleUseGlobalRegexOptions"
 #End Region
 #Region "Declarations"
         Private ReadOnly Property MySettings As SiteSettings
@@ -27,26 +30,37 @@ Namespace API.TikTok
                 Return HOST.Source
             End Get
         End Property
+        Private UserCache As CacheKeeper = Nothing
         Private ReadOnly Property RootCacheTikTok As ICacheKeeper
             Get
-                With Settings.Cache
+                If Not UserCache Is Nothing AndAlso Not UserCache.Disposed Then
-                    Dim f As SFile = $"{Settings.Cache.RootDirectory.PathWithSeparator}TikTokCache\"
+                    With DirectCast(UserCache.NewInstance(Of BatchFileExchanger), BatchFileExchanger)
-                    If .ContainsFolder(f) Then
+                        .Validate()
-                        Return .GetInstance(f)
+                        Return .Self
-                    Else
+                    End With
-                        f.Exists(SFO.Path, True)
+                Else
-                        With .NewInstance(Of BatchFileExchanger)(f)
+                    With Settings.Cache
-                            .DeleteCacheOnDispose = False
+                        Dim f As SFile = $"{Settings.Cache.RootDirectory.PathWithSeparator}TikTokCache\"
-                            .DeleteRootOnDispose = False
+                        If .ContainsFolder(f) Then
-                            Return .Self
+                            Return .GetInstance(f)
-                        End With
+                        Else
-                    End If
+                            f.Exists(SFO.Path, True)
-                End With
+                            With .NewInstance(Of BatchFileExchanger)(f)
+                                .DeleteCacheOnDispose = False
+                                .DeleteRootOnDispose = False
+                                Return .Self
+                            End With
+                        End If
+                    End With
+                End If
             End Get
         End Property
         Friend Property RemoveTagsFromTitle As Boolean = False
         Friend Property TitleUseNative As Boolean = True
         Friend Property TitleAddVideoID As Boolean = True
+        Friend Property TitleUseRegexForTitle As Boolean = False
+        Friend Property TitleUseRegexForTitle_Value As String = String.Empty
+        Friend Property TitleUseGlobalRegexOptions As Boolean = True
         Private Property LastDownloadDate As Date? = Nothing
         Private _TrueName As String = String.Empty
         Friend Property TrueName As String
@@ -68,6 +82,9 @@ Namespace API.TikTok
                     RemoveTagsFromTitle = .RemoveTagsFromTitle
                     TitleUseNative = .TitleUseNative
                     TitleAddVideoID = .TitleAddVideoID
+                    TitleUseRegexForTitle = .TitleUseRegexForTitle
+                    TitleUseRegexForTitle_Value = .TitleUseRegexForTitle_Value
+                    TitleUseGlobalRegexOptions = .TitleUseGlobalRegexOptions
                 End With
             End If
         End Sub
@@ -82,12 +99,18 @@ Namespace API.TikTok
                     LastDownloadDate = AConvert(Of Date)(.Value(Name_LastDownloadDate), ADateTime.Formats.BaseDateTime, Nothing)
                     If Not LastDownloadDate.HasValue Then LastDownloadDate = LastUpdated
                     _TrueName = .Value(Name_TrueName)
+                    TitleUseRegexForTitle = .Value(Name_TitleUseRegexForTitle).FromXML(Of Boolean)(False)
+                    TitleUseRegexForTitle_Value = .Value(Name_TitleUseRegexForTitle_Value)
+                    TitleUseGlobalRegexOptions = .Value(Name_TitleUseGlobalRegexOptions).FromXML(Of Boolean)(True)
                 Else
                     .Add(Name_RemoveTagsFromTitle, RemoveTagsFromTitle.BoolToInteger)
                     .Add(Name_TitleUseNative, TitleUseNative.BoolToInteger)
                     .Add(Name_TitleAddVideoID, TitleAddVideoID.BoolToInteger)
                     .Add(Name_LastDownloadDate, AConvert(Of String)(LastDownloadDate, AModes.XML, ADateTime.Formats.BaseDateTime, String.Empty))
                     .Add(Name_TrueName, _TrueName)
+                    .Add(Name_TitleUseRegexForTitle, TitleUseRegexForTitle.BoolToInteger)
+                    .Add(Name_TitleUseRegexForTitle_Value, TitleUseRegexForTitle_Value)
+                    .Add(Name_TitleUseGlobalRegexOptions, TitleUseGlobalRegexOptions.BoolToInteger)
                 End If
             End With
         End Sub
@@ -99,110 +122,145 @@ Namespace API.TikTok
         End Sub
 #End Region
 #Region "Download functions"
+        Private Function GetTitleRegex() As RParams
+            Dim titleRegex As RParams = Nothing
+            If TitleUseGlobalRegexOptions Then
+                If CBool(MySettings.TitleUseRegexForTitle.Value) AndAlso Not CStr(MySettings.TitleUseRegexForTitle_Value.Value).IsEmptyString Then _
+                   titleRegex = RParams.DM(MySettings.TitleUseRegexForTitle_Value.Value, 0, RegexReturn.List, EDP.ReturnValue)
+            ElseIf TitleUseRegexForTitle And Not TitleUseRegexForTitle_Value.IsEmptyString Then
+                titleRegex = RParams.DM(TitleUseRegexForTitle_Value, 0, RegexReturn.List, EDP.ReturnValue)
+            End If
+            If Not titleRegex Is Nothing Then
+                titleRegex.NothingExists = True
+                titleRegex.Nothing = New List(Of String)
+                titleRegex.Converter = Function(input) input.StringTrim
+            End If
+            Return titleRegex
+        End Function
+        Private Function ChangeTitleRegex(ByVal Title As String, ByVal Regex As RParams) As String
+            Try
+                If Not Regex Is Nothing Then
+                    With DirectCast(RegexReplace(Title, Regex), List(Of String))
+                        If .ListExists Then
+                            Dim newTitle$ = .ListToString(String.Empty, EDP.ReturnValue).StringTrim
+                            If Not newTitle.IsEmptyString Then Return newTitle
+                        End If
+                    End With
+                End If
+            Catch ex As Exception
+            End Try
+            Return Title
+        End Function
+        Friend Overrides Sub DownloadData(ByVal Token As CancellationToken)
+            MyBase.DownloadData(Token)
+            UserCache.DisposeIfReady(False)
+            UserCache = Nothing
+        End Sub
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             Dim URL$ = $"https://www.tiktok.com/@{TrueName}"
-            Using cache As CacheKeeper = CreateCache()
+            UserCache = CreateCache()
-                Try
+            Try
-                    Dim postID$, title$, postUrl$, newName$
+                Dim postID$, title$, postUrl$, newName$
-                    Dim postDate As Date?
+                Dim postDate As Date?
-                    Dim dateAfterC As Date? = Nothing
+                Dim dateAfterC As Date? = Nothing
-                    Dim dateBefore As Date? = DownloadDateTo
+                Dim dateBefore As Date? = DownloadDateTo
-                    Dim dateAfter As Date? = DownloadDateFrom
+                Dim dateAfter As Date? = DownloadDateFrom
-                    Dim baseDataObtained As Boolean = False
+                Dim baseDataObtained As Boolean = False
+                Dim titleRegex As RParams = GetTitleRegex()
 
-                    If _ContentList.Count > 0 Then
+                If _ContentList.Count > 0 Then
-                        With (From d In _ContentList Where d.Post.Date.HasValue Select d.Post.Date.Value)
+                    With (From d In _ContentList Where d.Post.Date.HasValue Select d.Post.Date.Value)
-                            If .ListExists Then dateAfterC = .Min
+                        If .ListExists Then dateAfterC = .Min
-                        End With
-                    End If
-
-                    With {CStr(AConvert(Of String)(dateAfter, SimpleDateConverter, String.Empty)).FromXML(Of Integer)(-1),
-                          CStr(AConvert(Of String)(dateAfterC, SimpleDateConverter, String.Empty)).FromXML(Of Integer)(-1)}.ListWithRemove(Function(d) d = -1)
-                        If .ListExists Then dateAfter = AConvert(Of Date)(CStr(.Min), SimpleDateConverter, Nothing)
                     End With
+                End If
 
-                    If LastDownloadDate.HasValue Then
+                With {CStr(AConvert(Of String)(dateAfter, SimpleDateConverter, String.Empty)).FromXML(Of Integer)(-1),
-                        If Not DownloadDateTo.HasValue And Not DownloadDateFrom.HasValue Then
+                      CStr(AConvert(Of String)(dateAfterC, SimpleDateConverter, String.Empty)).FromXML(Of Integer)(-1)}.ListWithRemove(Function(d) d = -1)
-                            If LastDownloadDate.Value.AddDays(1) <= Now Then
+                    If .ListExists Then dateAfter = AConvert(Of Date)(CStr(.Min), SimpleDateConverter, Nothing)
-                                dateAfter = LastDownloadDate.Value
+                End With
-                            Else
+
-                                dateAfter = LastDownloadDate.Value.AddDays(-1)
+                If LastDownloadDate.HasValue Then
-                            End If
+                    If Not DownloadDateTo.HasValue And Not DownloadDateFrom.HasValue Then
-                            dateBefore = Nothing
+                        If LastDownloadDate.Value.AddDays(1) <= Now Then
-                        ElseIf dateAfter.HasValue And Not DownloadDateFrom.HasValue Then
+                            dateAfter = LastDownloadDate.Value
-                            If (LastDownloadDate.Value - dateAfter.Value).TotalDays > 1 Then dateAfter = dateAfter.Value.AddDays(1)
+                        Else
+                            dateAfter = LastDownloadDate.Value.AddDays(-1)
                         End If
+                        dateBefore = Nothing
+                    ElseIf dateAfter.HasValue And Not DownloadDateFrom.HasValue Then
+                        If (LastDownloadDate.Value - dateAfter.Value).TotalDays > 1 Then dateAfter = dateAfter.Value.AddDays(1)
                     End If
+                End If
 
-                    Using b As New YTDLP.YTDLPBatch(Token) With {.TempPostsList = _TempPostsList}
+                Using b As New YTDLP.YTDLPBatch(Token) With {.TempPostsList = _TempPostsList}
-                        b.Commands.Clear()
+                    b.Commands.Clear()
-                        b.ChangeDirectory(cache)
+                    b.ChangeDirectory(UserCache)
-                        b.Encoding = BatchExecutor.UnicodeEncoding
+                    b.Encoding = BatchExecutor.UnicodeEncoding
-                        b.Execute(CreateYTCommand(cache.RootDirectory, URL, False, dateBefore, dateAfter))
+                    b.Execute(CreateYTCommand(UserCache.RootDirectory, URL, False, dateBefore, dateAfter))
-                    End Using
+                End Using
 
-                    ThrowAny(Token)
+                ThrowAny(Token)
 
-                    Dim files As List(Of SFile) = SFile.GetFiles(cache, "*.json",, EDP.ReturnValue)
+                Dim files As List(Of SFile) = SFile.GetFiles(UserCache, "*.json",, EDP.ReturnValue)
-                    If files.ListExists Then
+                If files.ListExists Then
-                        Dim j As EContainer
+                    Dim j As EContainer
-                        For Each file As SFile In files
+                    For Each file As SFile In files
-                            j = JsonDocument.Parse(file.GetText, EDP.ReturnValue)
+                        j = JsonDocument.Parse(file.GetText, EDP.ReturnValue)
-                            If j.ListExists Then
+                        If j.ListExists Then
-                                If j.Value("_type").StringToLower = "video" Then
+                            If j.Value("_type").StringToLower = "video" Then
-                                    If Not baseDataObtained Then
+                                If Not baseDataObtained Then
-                                        baseDataObtained = True
+                                    baseDataObtained = True
-                                        If ID.IsEmptyString Then
+                                    If ID.IsEmptyString Then
-                                            ID = j.Value("uploader_id")
+                                        ID = j.Value("uploader_id")
-                                            If Not ID.IsEmptyString Then _ForceSaveUserInfo = True
+                                        If Not ID.IsEmptyString Then _ForceSaveUserInfo = True
-                                        End If
-                                        newName = j.Value("uploader")
-                                        If Not newName.IsEmptyString Then
-                                            If Not _TrueName = newName Then _ForceSaveUserInfo = True
-                                            _TrueName = newName
-                                        End If
-                                        newName = j.Value("creator")
-                                        If Not newName.IsEmptyString Then UserSiteName = newName
                                     End If
-                                    postID = j.Value("id")
+                                    newName = j.Value("uploader")
-                                    If Not _TempPostsList.Contains(postID) Then
+                                    If Not newName.IsEmptyString Then
-                                        _TempPostsList.Add(postID)
+                                        If Not _TrueName = newName Then _ForceSaveUserInfo = True
-                                    Else
+                                        _TrueName = newName
-                                        Exit Sub
                                     End If
-                                    title = j.Value("title").StringRemoveWinForbiddenSymbols
+                                    newName = j.Value("creator")
-                                    If title.IsEmptyString Or Not TitleUseNative Then
+                                    If Not newName.IsEmptyString Then UserSiteName = newName
-                                        title = postID
-                                    Else
-                                        If RemoveTagsFromTitle Then title = RegexReplace(title, RegexTagsReplacer)
-                                        title = title.StringTrim
-                                        If title.IsEmptyString Then
-                                            title = postID
-                                        ElseIf TitleAddVideoID Then
-                                            title &= $" ({postID})"
-                                        End If
-                                    End If
-                                    postDate = AConvert(Of Date)(j.Value("timestamp"), UnixDate32Provider, Nothing)
-                                    If Not postDate.HasValue Then postDate = AConvert(Of Date)(j.Value("upload_date"), SimpleDateConverter, Nothing)
-                                    Select Case CheckDatesLimit(postDate, SimpleDateConverter)
-                                        Case DateResult.Skip : Continue For
-                                        Case DateResult.Exit : Exit Sub
-                                    End Select
-
-                                    postUrl = j.Value("webpage_url")
-                                    If postUrl.IsEmptyString Then postUrl = $"https://www.tiktok.com/@{Name}/video/{postID}"
-                                    _TempMediaList.Add(New UserMedia(postUrl, UserMedia.Types.Video) With {
-                                                       .File = $"{title}.mp4", .Post = New UserPost(postID, postDate)})
                                 End If
-                                j.Dispose()
+                                postID = j.Value("id")
+                                If Not _TempPostsList.Contains(postID) Then
+                                    _TempPostsList.Add(postID)
+                                Else
+                                    Exit Sub
+                                End If
+                                title = j.Value("title").StringRemoveWinForbiddenSymbols
+                                If title.IsEmptyString Or Not TitleUseNative Then
+                                    title = postID
+                                Else
+                                    If RemoveTagsFromTitle Then title = RegexReplace(title, RegexTagsReplacer)
+                                    title = title.StringTrim
+                                    If title.IsEmptyString Then
+                                        title = postID
+                                    ElseIf TitleAddVideoID Then
+                                        title &= $" ({postID})"
+                                    End If
+                                    title = ChangeTitleRegex(title, titleRegex)
+                                End If
+                                postDate = AConvert(Of Date)(j.Value("timestamp"), UnixDate32Provider, Nothing)
+                                If Not postDate.HasValue Then postDate = AConvert(Of Date)(j.Value("upload_date"), SimpleDateConverter, Nothing)
+                                Select Case CheckDatesLimit(postDate, SimpleDateConverter)
+                                    Case DateResult.Skip : Continue For
+                                    Case DateResult.Exit : Exit Sub
+                                End Select
+
+                                postUrl = j.Value("webpage_url")
+                                If postUrl.IsEmptyString Then postUrl = $"https://www.tiktok.com/@{Name}/video/{postID}"
+                                _TempMediaList.Add(New UserMedia(postUrl, UserMedia.Types.Video) With {
+                                                   .File = $"{title}.mp4", .Post = New UserPost(postID, postDate)})
                             End If
-                        Next
+                            j.Dispose()
-                    End If
+                        End If
-                    If _TempMediaList.Count > 0 Then LastDownloadDate = Now
+                    Next
-                Catch ex As Exception
+                End If
-                    ProcessException(ex, Token, $"data downloading error [{URL}]")
+                If _TempMediaList.Count > 0 Then LastDownloadDate = Now
-                End Try
+            Catch ex As Exception
-            End Using
+                ProcessException(ex, Token, $"data downloading error [{URL}]")
+            End Try
         End Sub
 #End Region
 #Region "ReparseMissing"
@@ -292,6 +350,7 @@ Namespace API.TikTok
                 f = f.StringTrim
                 If Not f.IsEmptyString Then
                     If CBool(MySettings.TitleAddVideoID.Value) Then f &= $" ({m.File.Name})"
+                    f = ChangeTitleRegex(f, GetTitleRegex)
                     m.File.Name = f
                 End If
             End If
@@ -303,6 +362,15 @@ Namespace API.TikTok
                                                           Optional ByVal EObj As Object = Nothing) As Integer
             Return 0
         End Function
+#End Region
+#Region "IDisposable Support"
+        Protected Overrides Sub Dispose(ByVal disposing As Boolean)
+            If Not disposedValue And disposing Then
+                UserCache.DisposeIfReady(False)
+                UserCache = Nothing
+            End If
+            MyBase.Dispose(disposing)
+        End Sub
 #End Region
     End Class
 End Namespace

SCrawler/API/TikTok/UserExchangeOptions.vb

@@ -15,18 +15,29 @@ Namespace API.TikTok
         Friend Property TitleUseNative As Boolean
         <PSetting(NameOf(SiteSettings.TitleAddVideoID), NameOf(MySettings))>
         Friend Property TitleAddVideoID As Boolean
+        <PSetting(NameOf(SiteSettings.TitleUseRegexForTitle), NameOf(MySettings))>
+        Friend Property TitleUseRegexForTitle As Boolean
+        <PSetting(NameOf(SiteSettings.TitleUseRegexForTitle_Value), NameOf(MySettings))>
+        Friend Property TitleUseRegexForTitle_Value As String
+        <PSetting(Caption:="Use global regex", ToolTip:="Use the global regex from the site settings to clean the video title")>
+        Friend Property TitleUseGlobalRegexOptions As Boolean = True
         Private ReadOnly MySettings As SiteSettings
         Friend Sub New(ByVal u As UserData)
             MySettings = u.HOST.Source
             RemoveTagsFromTitle = u.RemoveTagsFromTitle
             TitleUseNative = u.TitleUseNative
             TitleAddVideoID = u.TitleAddVideoID
+            TitleUseRegexForTitle = u.TitleUseRegexForTitle
+            TitleUseRegexForTitle_Value = u.TitleUseRegexForTitle_Value
+            TitleUseGlobalRegexOptions = u.TitleUseGlobalRegexOptions
         End Sub
         Friend Sub New(ByVal s As SiteSettings)
             MySettings = s
             RemoveTagsFromTitle = s.RemoveTagsFromTitle.Value
             TitleUseNative = s.TitleUseNative.Value
             TitleAddVideoID = s.TitleAddVideoID.Value
+            TitleUseRegexForTitle = s.TitleUseRegexForTitle.Value
+            TitleUseRegexForTitle_Value = s.TitleUseRegexForTitle_Value.Value
         End Sub
     End Class
 End Namespace

SCrawler/API/Twitter/UserData.vb

@@ -364,7 +364,7 @@ Namespace API.Twitter
                 DownloadModelForceApply = False
                 FirstDownloadComplete = True
             Catch jsonNull_ex As ArgumentNullException When jsonArgs.State = WebDocumentEventArgs.States.Error
-                Throw New Plugin.ExitException($"{ToStringForLog()}: No deserialized data found")
+                Throw New Plugin.ExitException("No deserialized data found")
             Catch limit_ex As TwitterLimitException
                 Throw limit_ex
             Catch ex As Exception

SCrawler/API/UserDataBind.vb

@@ -741,6 +741,15 @@ Namespace API
             Return GetEnumerator()
         End Function
 #End Region
+#Region "IComparable Support"
+        Friend Overrides Function CompareTo(ByVal Other As UserDataBase) As Integer
+            If TypeOf Other Is UserDataBind Then
+                Return CollectionName.CompareTo(Other.CollectionName)
+            Else
+                Return -1
+            End If
+        End Function
+#End Region
 #Region "IEquatable support"
         Friend Overrides Function Equals(ByVal Other As UserDataBase) As Boolean
             If Other.IsCollection Then

SCrawler/API/Xhamster/M3U8.vb

@@ -84,10 +84,15 @@ Namespace API.Xhamster
         End Function
         Private Shared Function ObtainUrls(ByVal URL As String, ByVal Responser As Responser, ByVal UHD As Boolean) As List(Of M3U8URL)
             Try
+                Const sk$ = "/key="
                 Dim file$ = ParseFirstM3U8(URL, Responser, UHD)
                 If Not file.IsEmptyString Then
                     Responser.UseGZipStream = False
                     Dim appender$ = URL.Replace(URL.Split("/").LastOrDefault, String.Empty)
+                    If file.StartsWith(sk) Then
+                        Dim position% = InStr(URL, sk)
+                        If position > 0 Then appender = URL.Remove(position - 1)
+                    End If
                     URL = M3U8Base.CreateUrl(appender, file)
                     Dim l As List(Of M3U8URL) = ParseSecondM3U8(URL, Responser, appender)
                     If l.ListExists Then Return l

SCrawler/API/Xhamster/SiteSettings.vb

@@ -46,7 +46,7 @@ Namespace API.Xhamster
 
             _SubscriptionsAllowed = True
             UrlPatternUser = "https://xhamster.com/{0}/{1}"
-            UserRegex = RParams.DMS($"/({UserOption}|{ChannelOption})/([^/]+)(\Z|.*)", 0, RegexReturn.ListByMatch)
+            UserRegex = RParams.DMS($"/({UserOption}|{ChannelOption}|{P_Creators})/([^/]+)(\Z|.*)", 0, RegexReturn.ListByMatch)
             ImageVideoContains = "xhamster"
         End Sub
         Friend Overrides Sub EndInit()
@@ -96,8 +96,9 @@ Namespace API.Xhamster
         Friend Const P_Search As String = "search"
         Friend Const P_Tags As String = "tags"
         Friend Const P_Categories As String = "categories"
-        Friend Const P_Pornstars = "pornstars"
+        Friend Const P_Pornstars As String = "pornstars"
-        Private ReadOnly NonUsersRegex As RParams = RParams.DM("https?://[^/]+/((gay)/|(shemale)/|)(pornstars|tags|categories|search)/([^/\?]+)[/\?]?(.*)", 0,
+        Friend Const P_Creators As String = "creators"
+        Private ReadOnly NonUsersRegex As RParams = RParams.DM("https?://[^/]+/((gay)/|(shemale)/|)(pornstars|creators|tags|categories|search)/([^/\?]+)[/\?]?(.*)", 0,
                                                                RegexReturn.ListByMatch, EDP.ReturnValue)
         Private ReadOnly PageRemover_1 As RParams = RParams.DM("[\?&]?[Pp]age=\d+", 0, RegexReturn.Replace, EDP.ReturnValue,
                                                                CType(Function(input) String.Empty, Func(Of String, String)))
@@ -106,12 +107,23 @@ Namespace API.Xhamster
         Friend Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
             If Not UserURL.IsEmptyString AndAlso Domains.Domains.Count > 0 AndAlso Domains.Domains.Exists(Function(d) UserURL.ToLower.Contains(d.ToLower)) Then
                 Dim n$, opt$
+                Dim tryNext As Boolean = False
                 Dim data As List(Of String) = RegexReplace(UserURL, UserRegex)
                 If data.ListExists(3) AndAlso Not data(2).IsEmptyString Then
                     n = data(2)
-                    If Not data(1).IsEmptyString AndAlso data(1) = ChannelOption Then n &= $"@{data(1)}"
+                    If Not data(1).IsEmptyString Then
-                    Return New ExchangeOptions(Site, n)
+                        If data(1) = ChannelOption Then
+                            n &= $"@{data(1)}"
+                        ElseIf data(1) = P_Creators Then
+                            tryNext = True
+                        End If
+                    End If
+                    If Not tryNext Then Return New ExchangeOptions(Site, n)
                 Else
+                    tryNext = True
+                End If
+
+                If tryNext Then
                     data = RegexReplace(UserURL, NonUsersRegex)
                     If data.ListExists(7) AndAlso Not data(5).IsEmptyString Then
                         n = data(5).StringRemoveWinForbiddenSymbols
@@ -122,6 +134,7 @@ Namespace API.Xhamster
                                 Case P_Tags : mode = SiteModes.Tags
                                 Case P_Categories : mode = SiteModes.Categories
                                 Case P_Pornstars : mode = SiteModes.Pornstars
+                                Case P_Creators : mode = SiteModes.User
                                 Case Else : Return Nothing
                             End Select
                             n = $"{CInt(mode)}@{n}"

SCrawler/API/Xhamster/UserData.vb

@@ -19,6 +19,7 @@ Namespace API.Xhamster
     Friend Class UserData : Inherits UserDataBase
 #Region "XML names"
         Private Const Name_Gender As String = "Gender"
+        Private Const Name_IsCreator As String = "IsCreator"
 #End Region
 #Region "Declarations"
         Friend Overrides ReadOnly Property FeedIsUser As Boolean
@@ -27,6 +28,7 @@ Namespace API.Xhamster
             End Get
         End Property
         Friend Property IsChannel As Boolean = False
+        Friend Property IsCreator As Boolean = False
         Friend Property TrueName As String = String.Empty
         Friend Property Gender As String = String.Empty
         Friend Property SiteMode As SiteModes = SiteModes.User
@@ -77,7 +79,8 @@ Namespace API.Xhamster
                         If n.Length = 2 And If(Force, eObj.Options, Options).IsEmptyString Then
                             If Force Then Return False
                             TrueName = n(0)
-                            IsChannel = True
+                            IsChannel = n(1) = SiteSettings.ChannelOption
+                            IsCreator = n(1) = SiteSettings.P_Creators
                         ElseIf IsChannel Then
                             If Force Then Return False
                             TrueName = Name
@@ -89,6 +92,7 @@ Namespace API.Xhamster
                             If n2.ListExists Then
                                 IsChannel = False
                                 __Mode = CInt(n2(0))
+                                IsCreator = __Mode = SiteModes.User
                                 __Gender = n2(1)
                                 __Arguments = n2(3)
                                 __TrueName = n2.ListTake(3, 100, EDP.ReturnValue).ListToString(String.Empty)
@@ -139,6 +143,7 @@ Namespace API.Xhamster
             With Container
                 If Loading Then
                     IsChannel = .Value(Name_IsChannel).FromXML(Of Boolean)(False)
+                    IsCreator = .Value(Name_IsCreator).FromXML(Of Boolean)(False)
                     TrueName = .Value(Name_TrueName)
                     Gender = .Value(Name_Gender)
                     SiteMode = .Value(Name_SiteMode).FromXML(Of Integer)(SiteModes.User)
@@ -151,6 +156,7 @@ Namespace API.Xhamster
                         .Value(Name_FriendlyName) = FriendlyName
                     End If
                     .Add(Name_IsChannel, IsChannel.BoolToInteger)
+                    .Add(Name_IsCreator, IsCreator.BoolToInteger)
                     .Add(Name_TrueName, TrueName)
                     .Add(Name_Gender, Gender)
                     .Add(Name_SiteMode, CInt(SiteMode))
@@ -178,7 +184,7 @@ Namespace API.Xhamster
 #End Region
 #Region "Download functions"
         Friend Function GetNonUserUrl(ByVal Page As Integer) As String
-            If SiteMode = SiteModes.User Then
+            If SiteMode = SiteModes.User And Not IsCreator Then
                 Return String.Empty
             Else
                 Dim url$ = "https://xhamster.com/"
@@ -188,6 +194,7 @@ Namespace API.Xhamster
                     Case SiteModes.Categories : url &= SiteSettings.P_Categories
                     Case SiteModes.Search : url &= SiteSettings.P_Search
                     Case SiteModes.Pornstars : url &= SiteSettings.P_Pornstars
+                    Case SiteModes.User : url &= SiteSettings.P_Creators
                     Case Else : Return String.Empty
                 End Select
                 url &= $"/{TrueName}"
@@ -224,15 +231,20 @@ Namespace API.Xhamster
         Private ReadOnly SessionPosts As List(Of String)
         Private _PageVideosRepeat As Integer = 0
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
-            _TempPhotoData.Clear()
+            Try
-            SearchPostsCount = 0
+                _TempPhotoData.Clear()
-            _PageVideosRepeat = 0
+                SearchPostsCount = 0
-            SessionPosts.Clear()
+                _PageVideosRepeat = 0
-            If DownloadVideos Then DownloadData(1, True, Token)
+                SessionPosts.Clear()
-            If Not IsChannel And DownloadImages And Not IsSubscription Then
+                Responser.CookiesAsHeader = True
-                DownloadData(1, False, Token)
+                If DownloadVideos Then DownloadData(1, True, Token)
-                ReparsePhoto(Token)
+                If Not IsChannel And Not IsCreator And DownloadImages And Not IsSubscription Then
-            End If
+                    DownloadData(1, False, Token)
+                    ReparsePhoto(Token)
+                End If
+            Finally
+                Responser.CookiesAsHeader = False
+            End Try
         End Sub
         Private Overloads Sub DownloadData(ByVal Page As Integer, ByVal IsVideo As Boolean, ByVal Token As CancellationToken)
             Dim URL$ = String.Empty
@@ -260,7 +272,7 @@ Namespace API.Xhamster
                 ElseIf SiteMode = SiteModes.Search Then
                     URL = GetNonUserUrl(Page)
                     containerNodes.Add({"searchResult", "models"})
-                ElseIf SiteMode = SiteModes.Tags Or SiteMode = SiteModes.Categories Or SiteMode = SiteModes.Pornstars Then
+                ElseIf IsCreator Or SiteMode = SiteModes.Tags Or SiteMode = SiteModes.Categories Or SiteMode = SiteModes.Pornstars Then
                     URL = GetNonUserUrl(Page)
                     If SiteMode = SiteModes.Pornstars Then
                         containerNodes.Add({"trendingVideoListComponent", "models"})
@@ -269,9 +281,11 @@ Namespace API.Xhamster
                         containerNodes.Add({"pagesCategoryComponent", "trendingVideoListProps", "models"})
                         containerNodes.Add({"trendingVideoListComponent", "models"})
                     End If
+                    containerNodes.Add({"trendingVideoSectionComponent", "videoModels"})
                 Else
                     URL = $"https://xhamster.com/users/{TrueName}/{IIf(IsVideo, "videos", "photos")}{IIf(Page = 1, String.Empty, $"/{Page}")}"
                     containerNodes.Add({If(IsVideo, "userVideoCollection", "userGalleriesCollection")})
+                    containerNodes.Add(If(IsVideo, {"videoListComponent", "models"}, {"userGalleriesCollection"}))
                 End If
                 ThrowAny(Token)
 

SCrawler/API/YouTube/SiteSettings.vb

@@ -24,6 +24,8 @@ Namespace API.YouTube
         Friend ReadOnly Property DownloadCommunityImages As PropertyValue
         <PXML, PropertyOption(ControlText:="Download user community: videos"), PClonable>
         Friend ReadOnly Property DownloadCommunityVideos As PropertyValue
+        <PXML, PropertyOption(ControlText:="Ignore community errors", ControlToolTip:="If true, community errors will not be added to the log."), PClonable>
+        Friend ReadOnly Property IgnoreCommunityErrors As PropertyValue
         <PXML, PropertyOption(ControlText:="Use cookies", ControlToolTip:="Default value for new users." & vbCr & "Use cookies when downloading data."), PClonable>
         Friend ReadOnly Property UseCookies As PropertyValue
 #End Region
@@ -36,6 +38,7 @@ Namespace API.YouTube
             DownloadPlaylists = New PropertyValue(False)
             DownloadCommunityImages = New PropertyValue(False)
             DownloadCommunityVideos = New PropertyValue(False)
+            IgnoreCommunityErrors = New PropertyValue(False)
             UseCookies = New PropertyValue(False)
             _SubscriptionsAllowed = True
             UseNetscapeCookies = True
@@ -90,7 +93,11 @@ Namespace API.YouTube
         End Function
         Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
             If Not User Is Nothing AndAlso TypeOf User Is UserData Then
-                Return $"https://{IIf(DirectCast(User, UserData).IsMusic, "music", "www")}.youtube.com/watch?v={Media.Post.ID}"
+                If DirectCast(User, UserData).IsMusic Or Media.URL_BASE.IsEmptyString Then
+                    Return $"https://{IIf(DirectCast(User, UserData).IsMusic, "music", "www")}.youtube.com/watch?v={Media.Post.ID}"
+                Else
+                    Return Media.URL_BASE
+                End If
             Else
                 Return String.Empty
             End If

SCrawler/API/YouTube/UserData.vb

@@ -137,6 +137,26 @@ Namespace API.YouTube
                 Dim list As New List(Of IYouTubeMediaContainer)
                 Dim url$ = String.Empty
                 Dim maxDate As Date? = Nothing
+                Dim __minDate As Date? = DownloadDateFrom
+                Dim __maxDate As Date? = DownloadDateTo
+                Dim __getMinDate As Func(Of Date?, Date?) = Function(ByVal dInput As Date?) As Date?
+                                                                If dInput.HasValue Then
+                                                                    If __minDate.HasValue Then
+                                                                        Return {__minDate.Value, dInput.Value}.Min
+                                                                    Else
+                                                                        Return dInput
+                                                                    End If
+                                                                ElseIf __minDate.HasValue Then
+                                                                    Return __minDate
+                                                                Else
+                                                                    Return Nothing
+                                                                End If
+                                                            End Function
+                Dim shortsUrlStandardize As Action(Of IYouTubeMediaContainer, Integer) = Sub(ByVal c As IYouTubeMediaContainer, ByVal ii As Integer)
+                                                                                             Dim sUrl$ = $"https://www.youtube.com/shorts/{c.ID}"
+                                                                                             'c.URL = sUrl
+                                                                                             c.URL_BASE = sUrl
+                                                                                         End Sub
                 Dim nDate As Func(Of Date?, Date?) = Function(ByVal dInput As Date?) As Date?
                                                          If dInput.HasValue Then
                                                              If dInput.Value.AddDays(3) < Now Then Return dInput.Value.AddDays(1) Else Return dInput
@@ -144,22 +164,23 @@ Namespace API.YouTube
                                                              Return Nothing
                                                          End If
                                                      End Function
-                Dim fillList As Func(Of Date?, Boolean) = Function(ByVal lDate As Date?) As Boolean
+                Dim fillList As Func(Of Date?, Boolean, Boolean) = Function(ByVal lDate As Date?, ByVal ___isShorts As Boolean) As Boolean
-                                                              If Not container Is Nothing AndAlso container.HasElements Then
+                                                                       If Not container Is Nothing AndAlso container.HasElements Then
-                                                                  Dim ce As IEnumerable(Of IYouTubeMediaContainer)
+                                                                           Dim ce As IEnumerable(Of IYouTubeMediaContainer)
-                                                                  ce = container.Elements
+                                                                           ce = container.Elements
-                                                                  If ce.ListExists Then ce = ce.Where(Function(e) e.ObjectType = YouTubeMediaType.Single)
+                                                                           If ce.ListExists Then ce = ce.Where(Function(e) e.ObjectType = YouTubeMediaType.Single)
-                                                                  If ce.ListExists AndAlso lDate.HasValue Then _
+                                                                           If ce.ListExists AndAlso lDate.HasValue Then _
-                                                                     ce = ce.Where(Function(e) e.DateAdded <= lDate.Value AndAlso
+                                                                              ce = ce.Where(Function(e) e.DateAdded >= lDate.Value AndAlso
-                                                                                               Not e.ID.IsEmptyString AndAlso Not _TempPostsList.Contains(e.ID))
+                                                                                                        Not e.ID.IsEmptyString AndAlso Not _TempPostsList.Contains(e.ID))
-                                                                  If ce.ListExists Then
+                                                                           If ce.ListExists Then
-                                                                      maxDate = ce.Max(Function(e) e.DateAdded)
+                                                                               maxDate = ce.Max(Function(e) e.DateAdded)
-                                                                      list.AddRange(ce)
+                                                                               If ___isShorts Then ce.ListForEach(shortsUrlStandardize, EDP.None)
-                                                                      Return True
+                                                                               list.AddRange(ce)
-                                                                  End If
+                                                                               Return True
-                                                              End If
+                                                                           End If
-                                                              Return False
+                                                                       End If
-                                                          End Function
+                                                                       Return False
+                                                                   End Function
                 Dim applySpecFolder As Action(Of String, Boolean) = Sub(ByVal fName As String, ByVal isPls As Boolean)
                                                                         If If(container?.Count, 0) > 0 Then _
                                                                            container.Elements.ForEach(Sub(ByVal el As YouTubeMediaContainerBase)
@@ -175,33 +196,33 @@ Namespace API.YouTube
                     maxDate = Nothing
                     LastDownloadDatePlaylist = nDate(LastDownloadDatePlaylist)
                     url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/playlist?list={ID}"
-                    container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr,, LastDownloadDatePlaylist,, True)
+                    container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDatePlaylist), __maxDate,, True)
                     applySpecFolder.Invoke(String.Empty, False)
-                    If fillList.Invoke(LastDownloadDatePlaylist) Then LastDownloadDatePlaylist = If(maxDate, Now)
+                    If fillList.Invoke(LastDownloadDatePlaylist, False) Then LastDownloadDatePlaylist = If(maxDate, Now)
                 ElseIf YTMediaType = YouTubeMediaType.Channel Then
                     If IsMusic Or DownloadYTVideos Then
                         maxDate = Nothing
                         LastDownloadDateVideos = nDate(LastDownloadDateVideos)
                         url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
-                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr,, LastDownloadDateVideos,, True)
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
                         applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
-                        If fillList.Invoke(LastDownloadDateVideos) Then LastDownloadDateVideos = If(maxDate, Now)
+                        If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)
                     End If
                     If Not IsMusic And DownloadYTShorts Then
                         maxDate = Nothing
                         LastDownloadDateShorts = nDate(LastDownloadDateShorts)
                         url = $"https://www.youtube.com/{IIf(IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/shorts"
-                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr,, LastDownloadDateShorts,, True)
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateShorts), __maxDate,, True)
                         applySpecFolder.Invoke("Shorts", False)
-                        If fillList.Invoke(LastDownloadDateShorts) Then LastDownloadDateShorts = If(maxDate, Now)
+                        If fillList.Invoke(LastDownloadDateShorts, True) Then LastDownloadDateShorts = If(maxDate, Now)
                     End If
                     If Not IsMusic And DownloadYTPlaylists Then
                         maxDate = Nothing
                         LastDownloadDatePlaylist = nDate(LastDownloadDatePlaylist)
                         url = $"https://www.youtube.com/{IIf(IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/playlists"
-                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr,, LastDownloadDatePlaylist,, True)
+                        container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDatePlaylist), __maxDate,, True)
                         applySpecFolder.Invoke("Playlists", True)
-                        If fillList.Invoke(LastDownloadDatePlaylist) Then LastDownloadDatePlaylist = If(maxDate, Now)
+                        If fillList.Invoke(LastDownloadDatePlaylist, False) Then LastDownloadDatePlaylist = If(maxDate, Now)
                     End If
                     If Not IsMusic And (DownloadYTCommunityImages Or DownloadYTCommunityVideos) Then DownloadCommunity(String.Empty, Token)
                 Else
@@ -225,6 +246,7 @@ Namespace API.YouTube
         End Sub
         Private Sub DownloadCommunity(ByVal Cursor As String, ByVal Token As CancellationToken, Optional ByVal Round As Integer = 0)
             Dim URL$ = String.Empty
+            Const errMsg$ = "community data downloading error"
             Try
                 Const postIdTemp$ = "Community_{0}"
                 Const specFolder$ = "Community"
@@ -311,6 +333,10 @@ Namespace API.YouTube
                                                 Next
                                             End If
                                         End With
+                                    ElseIf Not CBool(DirectCast(HOST.Source, SiteSettings).IgnoreCommunityErrors.Value) Then
+                                        With j({"error"})
+                                            If .ListExists Then MyMainLOG = $"{ToStringForLog()} {errMsg} [{ .Value("code")}]: { .Value("message")}"
+                                        End With
                                     End If
                                 End With
                             End If
@@ -327,7 +353,7 @@ Namespace API.YouTube
 
                 If Not nextToken.IsEmptyString Then DownloadCommunity(nextToken, Token)
             Catch ex As Exception
-                ProcessException(ex, Token, "community data downloading error")
+                ProcessException(ex, Token, errMsg)
             End Try
         End Sub
         Private Sub GetChannelID()

SCrawler/Download/Feed/DownloadFeedForm.Designer.vb

@@ -30,6 +30,7 @@ Namespace DownloadObjects
             Dim MENU_LOAD_SEP_2 As System.Windows.Forms.ToolStripSeparator
             Dim MENU_LOAD_SEP_3 As System.Windows.Forms.ToolStripSeparator
             Dim MENU_LOAD_SEP_4 As System.Windows.Forms.ToolStripSeparator
+            Dim MENU_LOAD_SEP_5 As System.Windows.Forms.ToolStripSeparator
             Me.OPT_DEFAULT = New System.Windows.Forms.ToolStripMenuItem()
             Me.OPT_SUBSCRIPTIONS = New System.Windows.Forms.ToolStripMenuItem()
             Me.ToolbarTOP = New System.Windows.Forms.ToolStrip()
@@ -43,18 +44,19 @@ Namespace DownloadObjects
             Me.BTT_FEED_REMOVE_FAV = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_ADD_SPEC = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_REMOVE_SPEC = New System.Windows.Forms.ToolStripMenuItem()
-            Me.SEP_0 = New System.Windows.Forms.ToolStripSeparator()
-            Me.MENU_DOWN = New System.Windows.Forms.ToolStripDropDownButton()
-            Me.BTT_DOWN_ALL = New System.Windows.Forms.ToolStripMenuItem()
-            Me.BTT_DOWN_SELECTED = New System.Windows.Forms.ToolStripMenuItem()
-            Me.BTT_REFRESH = New System.Windows.Forms.ToolStripButton()
-            Me.BTT_CLEAR = New System.Windows.Forms.ToolStripButton()
-            Me.TP_DATA = New System.Windows.Forms.TableLayoutPanel()
             Me.BTT_FEED_CLEAR_FAV = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_CLEAR_SPEC = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_DELETE_SPEC = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_DELETE_DAILY_LIST = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_DELETE_DAILY_DATE = New System.Windows.Forms.ToolStripMenuItem()
+            Me.BTT_MERGE_SESSIONS = New System.Windows.Forms.ToolStripMenuItem()
+            Me.BTT_CLEAR_DAILY = New System.Windows.Forms.ToolStripMenuItem()
+            Me.SEP_0 = New System.Windows.Forms.ToolStripSeparator()
+            Me.MENU_DOWN = New System.Windows.Forms.ToolStripDropDownButton()
+            Me.BTT_DOWN_ALL = New System.Windows.Forms.ToolStripMenuItem()
+            Me.BTT_DOWN_SELECTED = New System.Windows.Forms.ToolStripMenuItem()
+            Me.BTT_REFRESH = New System.Windows.Forms.ToolStripButton()
+            Me.TP_DATA = New System.Windows.Forms.TableLayoutPanel()
             SEP_1 = New System.Windows.Forms.ToolStripSeparator()
             SEP_2 = New System.Windows.Forms.ToolStripSeparator()
             MENU_VIEW = New System.Windows.Forms.ToolStripDropDownButton()
@@ -62,6 +64,7 @@ Namespace DownloadObjects
             MENU_LOAD_SEP_2 = New System.Windows.Forms.ToolStripSeparator()
             MENU_LOAD_SEP_3 = New System.Windows.Forms.ToolStripSeparator()
             MENU_LOAD_SEP_4 = New System.Windows.Forms.ToolStripSeparator()
+            MENU_LOAD_SEP_5 = New System.Windows.Forms.ToolStripSeparator()
             Me.ToolbarTOP.SuspendLayout()
             Me.SuspendLayout()
             '
@@ -112,10 +115,20 @@ Namespace DownloadObjects
             MENU_LOAD_SEP_3.Name = "MENU_LOAD_SEP_3"
             MENU_LOAD_SEP_3.Size = New System.Drawing.Size(264, 6)
             '
+            'MENU_LOAD_SEP_4
+            '
+            MENU_LOAD_SEP_4.Name = "MENU_LOAD_SEP_4"
+            MENU_LOAD_SEP_4.Size = New System.Drawing.Size(264, 6)
+            '
+            'MENU_LOAD_SEP_5
+            '
+            MENU_LOAD_SEP_5.Name = "MENU_LOAD_SEP_5"
+            MENU_LOAD_SEP_5.Size = New System.Drawing.Size(264, 6)
+            '
             'ToolbarTOP
             '
             Me.ToolbarTOP.GripStyle = System.Windows.Forms.ToolStripGripStyle.Hidden
-            Me.ToolbarTOP.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.MENU_LOAD_SESSION, Me.SEP_0, MENU_VIEW, SEP_1, Me.MENU_DOWN, Me.BTT_REFRESH, Me.BTT_CLEAR, SEP_2})
+            Me.ToolbarTOP.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.MENU_LOAD_SESSION, Me.SEP_0, MENU_VIEW, SEP_1, Me.MENU_DOWN, Me.BTT_REFRESH, SEP_2})
             Me.ToolbarTOP.Location = New System.Drawing.Point(0, 0)
             Me.ToolbarTOP.Name = "ToolbarTOP"
             Me.ToolbarTOP.Size = New System.Drawing.Size(484, 25)
@@ -124,7 +137,7 @@ Namespace DownloadObjects
             'MENU_LOAD_SESSION
             '
             Me.MENU_LOAD_SESSION.DisplayStyle = System.Windows.Forms.ToolStripItemDisplayStyle.Image
-            Me.MENU_LOAD_SESSION.DropDownItems.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_LOAD_SESSION_CURRENT, Me.BTT_LOAD_SESSION_LAST, Me.BTT_LOAD_SESSION_CHOOSE, MENU_LOAD_SEP_1, Me.BTT_LOAD_FAV, Me.BTT_LOAD_SPEC, MENU_LOAD_SEP_2, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_REMOVE_FAV, MENU_LOAD_SEP_3, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_REMOVE_SPEC, MENU_LOAD_SEP_4, Me.BTT_FEED_CLEAR_FAV, Me.BTT_FEED_CLEAR_SPEC, Me.BTT_FEED_DELETE_SPEC, Me.BTT_FEED_DELETE_DAILY_LIST, Me.BTT_FEED_DELETE_DAILY_DATE})
+            Me.MENU_LOAD_SESSION.DropDownItems.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_LOAD_SESSION_CURRENT, Me.BTT_LOAD_SESSION_LAST, Me.BTT_LOAD_SESSION_CHOOSE, MENU_LOAD_SEP_1, Me.BTT_LOAD_FAV, Me.BTT_LOAD_SPEC, MENU_LOAD_SEP_2, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_REMOVE_FAV, MENU_LOAD_SEP_3, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_REMOVE_SPEC, MENU_LOAD_SEP_4, Me.BTT_FEED_CLEAR_FAV, Me.BTT_FEED_CLEAR_SPEC, Me.BTT_FEED_DELETE_SPEC, Me.BTT_FEED_DELETE_DAILY_LIST, Me.BTT_FEED_DELETE_DAILY_DATE, MENU_LOAD_SEP_5, Me.BTT_MERGE_SESSIONS, Me.BTT_CLEAR_DAILY})
             Me.MENU_LOAD_SESSION.Image = Global.SCrawler.My.Resources.Resources.ArrowDownPic_Blue_24
             Me.MENU_LOAD_SESSION.ImageTransparentColor = System.Drawing.Color.Magenta
             Me.MENU_LOAD_SESSION.Name = "MENU_LOAD_SESSION"
@@ -194,6 +207,59 @@ Namespace DownloadObjects
             Me.BTT_FEED_REMOVE_SPEC.Size = New System.Drawing.Size(267, 22)
             Me.BTT_FEED_REMOVE_SPEC.Text = "Remove checked from special feed..."
             '
+            'BTT_FEED_CLEAR_FAV
+            '
+            Me.BTT_FEED_CLEAR_FAV.Image = Global.SCrawler.My.Resources.Resources.BrushToolPic_16
+            Me.BTT_FEED_CLEAR_FAV.Name = "BTT_FEED_CLEAR_FAV"
+            Me.BTT_FEED_CLEAR_FAV.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_FEED_CLEAR_FAV.Text = "Clear Favorite"
+            '
+            'BTT_FEED_CLEAR_SPEC
+            '
+            Me.BTT_FEED_CLEAR_SPEC.Image = Global.SCrawler.My.Resources.Resources.BrushToolPic_16
+            Me.BTT_FEED_CLEAR_SPEC.Name = "BTT_FEED_CLEAR_SPEC"
+            Me.BTT_FEED_CLEAR_SPEC.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_FEED_CLEAR_SPEC.Text = "Clear special feed..."
+            '
+            'BTT_FEED_DELETE_SPEC
+            '
+            Me.BTT_FEED_DELETE_SPEC.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
+            Me.BTT_FEED_DELETE_SPEC.Name = "BTT_FEED_DELETE_SPEC"
+            Me.BTT_FEED_DELETE_SPEC.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_FEED_DELETE_SPEC.Text = "Delete special feed..."
+            '
+            'BTT_FEED_DELETE_DAILY_LIST
+            '
+            Me.BTT_FEED_DELETE_DAILY_LIST.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
+            Me.BTT_FEED_DELETE_DAILY_LIST.Name = "BTT_FEED_DELETE_DAILY_LIST"
+            Me.BTT_FEED_DELETE_DAILY_LIST.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_FEED_DELETE_DAILY_LIST.Text = "Delete daily feed (by list)"
+            '
+            'BTT_FEED_DELETE_DAILY_DATE
+            '
+            Me.BTT_FEED_DELETE_DAILY_DATE.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
+            Me.BTT_FEED_DELETE_DAILY_DATE.Name = "BTT_FEED_DELETE_DAILY_DATE"
+            Me.BTT_FEED_DELETE_DAILY_DATE.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_FEED_DELETE_DAILY_DATE.Text = "Delete daily feed (by date)"
+            '
+            'BTT_MERGE_SESSIONS
+            '
+            Me.BTT_MERGE_SESSIONS.AutoToolTip = True
+            Me.BTT_MERGE_SESSIONS.Image = Global.SCrawler.My.Resources.Resources.DBPic_32
+            Me.BTT_MERGE_SESSIONS.Name = "BTT_MERGE_SESSIONS"
+            Me.BTT_MERGE_SESSIONS.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_MERGE_SESSIONS.Text = "Merge sessions"
+            Me.BTT_MERGE_SESSIONS.ToolTipText = "Merge multiple session feeds into one"
+            '
+            'BTT_CLEAR_DAILY
+            '
+            Me.BTT_CLEAR_DAILY.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
+            Me.BTT_CLEAR_DAILY.ImageTransparentColor = System.Drawing.Color.Magenta
+            Me.BTT_CLEAR_DAILY.Name = "BTT_CLEAR_DAILY"
+            Me.BTT_CLEAR_DAILY.Size = New System.Drawing.Size(267, 22)
+            Me.BTT_CLEAR_DAILY.Text = "Clear session"
+            Me.BTT_CLEAR_DAILY.ToolTipText = "Clear data list (session)"
+            '
             'SEP_0
             '
             Me.SEP_0.Name = "SEP_0"
@@ -235,15 +301,6 @@ Namespace DownloadObjects
             Me.BTT_REFRESH.Text = "Refresh"
             Me.BTT_REFRESH.ToolTipText = "Refresh data list"
             '
-            'BTT_CLEAR
-            '
-            Me.BTT_CLEAR.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
-            Me.BTT_CLEAR.ImageTransparentColor = System.Drawing.Color.Magenta
-            Me.BTT_CLEAR.Name = "BTT_CLEAR"
-            Me.BTT_CLEAR.Size = New System.Drawing.Size(54, 22)
-            Me.BTT_CLEAR.Text = "Clear"
-            Me.BTT_CLEAR.ToolTipText = "Clear data list"
-            '
             'TP_DATA
             '
             Me.TP_DATA.AutoScroll = True
@@ -269,46 +326,6 @@ Namespace DownloadObjects
             Me.TP_DATA.Size = New System.Drawing.Size(484, 436)
             Me.TP_DATA.TabIndex = 1
             '
-            'MENU_LOAD_SEP_4
-            '
-            MENU_LOAD_SEP_4.Name = "MENU_LOAD_SEP_4"
-            MENU_LOAD_SEP_4.Size = New System.Drawing.Size(264, 6)
-            '
-            'BTT_FEED_CLEAR_FAV
-            '
-            Me.BTT_FEED_CLEAR_FAV.Image = Global.SCrawler.My.Resources.Resources.BrushToolPic_16
-            Me.BTT_FEED_CLEAR_FAV.Name = "BTT_FEED_CLEAR_FAV"
-            Me.BTT_FEED_CLEAR_FAV.Size = New System.Drawing.Size(267, 22)
-            Me.BTT_FEED_CLEAR_FAV.Text = "Clear Favorite"
-            '
-            'BTT_FEED_CLEAR_SPEC
-            '
-            Me.BTT_FEED_CLEAR_SPEC.Image = Global.SCrawler.My.Resources.Resources.BrushToolPic_16
-            Me.BTT_FEED_CLEAR_SPEC.Name = "BTT_FEED_CLEAR_SPEC"
-            Me.BTT_FEED_CLEAR_SPEC.Size = New System.Drawing.Size(267, 22)
-            Me.BTT_FEED_CLEAR_SPEC.Text = "Clear special feed..."
-            '
-            'BTT_FEED_DELETE_SPEC
-            '
-            Me.BTT_FEED_DELETE_SPEC.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
-            Me.BTT_FEED_DELETE_SPEC.Name = "BTT_FEED_DELETE_SPEC"
-            Me.BTT_FEED_DELETE_SPEC.Size = New System.Drawing.Size(267, 22)
-            Me.BTT_FEED_DELETE_SPEC.Text = "Delete special feed..."
-            '
-            'BTT_FEED_DELETE_DAILY_LIST
-            '
-            Me.BTT_FEED_DELETE_DAILY_LIST.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
-            Me.BTT_FEED_DELETE_DAILY_LIST.Name = "BTT_FEED_DELETE_DAILY_LIST"
-            Me.BTT_FEED_DELETE_DAILY_LIST.Size = New System.Drawing.Size(267, 22)
-            Me.BTT_FEED_DELETE_DAILY_LIST.Text = "Delete daily feed (by list)"
-            '
-            'BTT_FEED_DELETE_DAILY_DATE
-            '
-            Me.BTT_FEED_DELETE_DAILY_DATE.Image = Global.SCrawler.My.Resources.Resources.DeletePic_24
-            Me.BTT_FEED_DELETE_DAILY_DATE.Name = "BTT_FEED_DELETE_DAILY_DATE"
-            Me.BTT_FEED_DELETE_DAILY_DATE.Size = New System.Drawing.Size(267, 22)
-            Me.BTT_FEED_DELETE_DAILY_DATE.Text = "Delete daily feed (by date)"
-            '
             'DownloadFeedForm
             '
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -330,7 +347,7 @@ Namespace DownloadObjects
 
         End Sub
         Private WithEvents BTT_REFRESH As ToolStripButton
-        Private WithEvents BTT_CLEAR As ToolStripButton
+        Private WithEvents BTT_CLEAR_DAILY As ToolStripMenuItem
         Private WithEvents MENU_LOAD_SESSION As ToolStripDropDownButton
         Private WithEvents BTT_LOAD_SESSION_LAST As ToolStripMenuItem
         Private WithEvents BTT_LOAD_SESSION_CHOOSE As ToolStripMenuItem
@@ -354,5 +371,6 @@ Namespace DownloadObjects
         Private WithEvents BTT_FEED_DELETE_SPEC As ToolStripMenuItem
         Private WithEvents BTT_FEED_DELETE_DAILY_LIST As ToolStripMenuItem
         Private WithEvents BTT_FEED_DELETE_DAILY_DATE As ToolStripMenuItem
+        Private WithEvents BTT_MERGE_SESSIONS As ToolStripMenuItem
     End Class
 End Namespace

SCrawler/Download/Feed/DownloadFeedForm.resx

@@ -144,10 +144,13 @@
   <metadata name="MENU_LOAD_SEP_3.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
-  <metadata name="ToolbarTOP.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
-    <value>17, 17</value>
-  </metadata>
   <metadata name="MENU_LOAD_SEP_4.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
     <value>False</value>
   </metadata>
+  <metadata name="MENU_LOAD_SEP_5.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
+    <value>False</value>
+  </metadata>
+  <metadata name="ToolbarTOP.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
+    <value>17, 17</value>
+  </metadata>
 </root>

SCrawler/Download/Feed/DownloadFeedForm.vb

@@ -37,6 +37,7 @@ Namespace DownloadObjects
                 Return OPT_SUBSCRIPTIONS.Checked
             End Get
         End Property
+        Private IsSession As Boolean = True
 #End Region
 #Region "Initializer"
         Friend Sub New()
@@ -105,7 +106,7 @@ Namespace DownloadObjects
         Private Sub DownloadFeedForm_Disposed(sender As Object, e As EventArgs) Handles Me.Disposed
             ClearTable()
             MyRange.Dispose()
-            BTT_CLEAR.Dispose()
+            BTT_CLEAR_DAILY.Dispose()
             DataList.Clear()
         End Sub
         Private Sub DownloadFeedForm_KeyDown(sender As Object, e As KeyEventArgs) Handles Me.KeyDown
@@ -171,6 +172,7 @@ Namespace DownloadObjects
             End Try
         End Sub
         Private Sub Feed_SPEC_LOAD(ByVal Source As ToolStripMenuItem, ByVal e As EventArgs)
+            IsSession = False
             Dim f As FeedSpecial = Source.Tag
             If Not f Is Nothing AndAlso Not f.Disposed Then
                 DataList.Clear()
@@ -291,12 +293,15 @@ Namespace DownloadObjects
 #Region "Feed"
 #Region "Load"
         Private Sub BTT_LOAD_SESSION_CURRENT_Click(sender As Object, e As EventArgs) Handles BTT_LOAD_SESSION_CURRENT.Click
+            IsSession = True
             RefillList()
         End Sub
         Private Sub BTT_LOAD_SESSION_LAST_Click(sender As Object, e As EventArgs) Handles BTT_LOAD_SESSION_LAST.Click
+            IsSession = True
             SessionChooser(True)
         End Sub
         Private Sub BTT_LOAD_SESSION_CHOOSE_Click(sender As Object, e As EventArgs) Handles BTT_LOAD_SESSION_CHOOSE.Click
+            IsSession = True
             SessionChooser(False)
         End Sub
         Private Sub SessionChooser(ByVal GetLast As Boolean, Optional ByVal GetFilesOnly As Boolean = False,
@@ -318,7 +323,7 @@ Namespace DownloadObjects
                                                              Return False
                                                          End If
                                                      End Function
-                If Not GetLast Then __getFiles.Invoke
+                __getFiles.Invoke
                 If Not GetLast And GetFilesOnly And Not fList.ListExists Then
                     MsgBoxE({"No session files found", "Get session files"}, vbExclamation)
                 ElseIf Not GetLast AndAlso fList.ListExists Then
@@ -352,18 +357,24 @@ Namespace DownloadObjects
                             MsgBoxE(m)
                         End If
                     End Using
-                ElseIf Downloader.FilesSessionActual(False).Exists OrElse __getFiles.Invoke Then
+                ElseIf Downloader.FilesSessionActual(False).Exists OrElse fList.ListExists Then
-                    If Downloader.FilesSessionActual(False).Exists Then
+                    If GetLast Then
-                        f = Downloader.FilesSessionActual(False)
+                        If fList.ListExists Then
+                            f = fList(IIf(fList.Count > 1 And Downloader.FilesSessionActual(False).Exists, 1, 0))
+                        Else
+                            f = Downloader.FilesSessionActual(False)
+                        End If
                     Else
-                        f = fList(0)
+                        f = Downloader.FilesSessionActual(False)
+                    End If
+                    If f.Exists Then
+                        x = New XmlFile(f,, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                        x.LoadData()
+                        If x.Count > 0 Then DataList.Clear() : DataList.ListAddList(x, lcr)
+                        x.Dispose()
+                        CleanDataList()
+                        RefillList(False)
                     End If
-                    x = New XmlFile(f,, False) With {.AllowSameNames = True, .XmlReadOnly = True}
-                    x.LoadData()
-                    If x.Count > 0 Then DataList.Clear() : DataList.ListAddList(x, lcr)
-                    x.Dispose()
-                    CleanDataList()
-                    RefillList(False)
                 Else
                     m.Text = "Saved sessions not found"
                     MsgBoxE(m)
@@ -376,6 +387,7 @@ Namespace DownloadObjects
 #End Region
 #Region "Load fav, spec"
         Private Sub BTT_LOAD_FAV_Click(sender As Object, e As EventArgs) Handles BTT_LOAD_FAV.Click
+            IsSession = False
             DataList.Clear()
             With Settings.Feeds.Favorite
                 .RemoveNotExist(FileNotExist)
@@ -383,6 +395,7 @@ Namespace DownloadObjects
             End With
         End Sub
         Private Sub BTT_LOAD_SPEC_Click(sender As Object, e As EventArgs) Handles BTT_LOAD_SPEC.Click
+            IsSession = False
             With FeedSpecialCollection.ChooseFeeds(False)
                 If .ListExists Then
                     DataList.Clear()
@@ -408,7 +421,7 @@ Namespace DownloadObjects
         Private Sub BTT_FEED_ADD_SPEC_Click(sender As Object, e As EventArgs) Handles BTT_FEED_ADD_SPEC.Click
             Dim c As IEnumerable(Of UserMediaD) = GetCheckedMedia()
             If c.ListExists Then
-                With FeedSpecialCollection.ChooseFeeds(False)
+                With FeedSpecialCollection.ChooseFeeds(True)
                     If .ListExists Then .ForEach(Sub(f) f.Add(c))
                 End With
             Else
@@ -512,6 +525,77 @@ Namespace DownloadObjects
             End Try
         End Sub
 #End Region
+#Region "Clear session"
+        Private Sub BTT_CLEAR_DAILY_Click(sender As Object, e As EventArgs) Handles BTT_CLEAR_DAILY.Click
+            If MsgBoxE({"Are you sure you want to clear this session data?", "Clear session"}, vbExclamation,,, {"Process", "Cancel"}) = 0 Then
+                Downloader.Files.Clear()
+                ClearTable()
+                RefillList()
+            End If
+        End Sub
+#End Region
+#Region "Merge feeds"
+        Private Sub MergeFeeds() Handles BTT_MERGE_SESSIONS.Click
+            Try
+                Const msgTitle$ = "Merge feeds"
+                Dim files As New List(Of SFile)
+                Dim abs% = 0, prev% = 0, curr%, i%
+                Dim x As XmlFile
+                Dim f As SFile
+                Dim um As UserMediaD
+                Dim data As New List(Of UserMediaD)
+                Dim tmpData As New List(Of UserMediaD)
+                Dim lrc As New ListAddParams(LAP.NotContainsOnly + LAP.IgnoreICopier)
+                SessionChooser(False, True, files)
+                If files.ListExists(2) Then
+                    files.Sort()
+                    For Each f In files
+                        x = New XmlFile(f,, False) With {.AllowSameNames = True, .XmlReadOnly = True}
+                        x.LoadData(EDP.None)
+                        If x.Count > 0 Then tmpData.ListAddList(x, lrc)
+                        If tmpData.Count > 0 Then tmpData.Reverse() : data.AddRange(tmpData) : tmpData.Clear()
+                        x.Dispose()
+                    Next
+                    If data.Count > 0 Then
+                        For i = 0 To data.Count - 1
+                            um = data(i)
+                            curr = um.Session
+                            If i = 0 Then
+                                abs = curr
+                            Else
+                                If curr < abs And prev <> curr Then
+                                    abs += 1
+                                ElseIf curr >= abs Then
+                                    abs = curr
+                                End If
+                            End If
+                            prev = curr
+                            um.Session = abs
+                            data(i) = um
+                        Next
+                        data.Reverse()
+                        x = New XmlFile With {.Name = TDownloader.Name_SessionXML, .AllowSameNames = True}
+                        x.AddRange(data)
+                        x.Save(files(0))
+                        x.Dispose()
+                        For i = 1 To files.Count - 1 : files(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, EDP.ReturnValue) : Next
+                        MsgBoxE({$"Session data was combined into '{files(0).Name}'.{vbCr}{vbCr}" &
+                                 files.ListToStringE(vbCr, New CustomProvider(Function(ff As SFile) ff.Name),,, EDP.ReturnValue), msgTitle})
+                        files.Clear()
+                        data.Clear()
+                    Else
+                        MsgBoxE({"There is no session data in the selected files", msgTitle}, vbExclamation)
+                    End If
+                ElseIf files.ListExists(1) Then
+                    MsgBoxE({"You must select two or more files to merge feeds", msgTitle}, vbExclamation)
+                Else
+                    MsgBoxE({"You haven't selected any feeds", msgTitle}, vbExclamation)
+                End If
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadFeedForm.MergeFeeds]")
+            End Try
+        End Sub
+#End Region
 #End Region
 #Region "View modes"
         Private Sub OPT_Click(ByVal Sender As ToolStripMenuItem, ByVal e As EventArgs) Handles OPT_DEFAULT.Click, OPT_SUBSCRIPTIONS.Click
@@ -538,11 +622,7 @@ Namespace DownloadObjects
             BTT_REFRESH.ControlChangeColor(ToolbarTOP, Added, False)
         End Sub
         Private Sub BTT_REFRESH_Click(sender As Object, e As EventArgs) Handles BTT_REFRESH.Click
-            RefillList()
+            IsSession = True
-        End Sub
-        Private Sub BTT_CLEAR_Click(sender As Object, e As EventArgs) Handles BTT_CLEAR.Click
-            Downloader.Files.Clear()
-            ClearTable()
             RefillList()
         End Sub
 #End Region
@@ -753,7 +833,7 @@ Namespace DownloadObjects
                         Dim p As New TPCELL(DataRows, DataColumns)
                         Dim fmList As New List(Of FeedMedia)
                         d.ForEach(Sub(ByVal de As UserMediaD)
-                                      fmList.Add(New FeedMedia(de, w, h))
+                                      fmList.Add(New FeedMedia(de, w, h, IsSession))
                                       With fmList.Last
                                           AddHandler .MediaDeleted, AddressOf FeedMedia_MediaDeleted
                                           AddHandler .MediaDownload, AddressOf FeedMedia_Download

SCrawler/Download/Feed/FeedMedia.vb

@@ -159,7 +159,7 @@ Namespace DownloadObjects
         Public Sub New()
             InitializeComponent()
         End Sub
-        Friend Sub New(ByVal Media As UserMediaD, ByVal Width As Integer, ByVal Height As Integer)
+        Friend Sub New(ByVal Media As UserMediaD, ByVal Width As Integer, ByVal Height As Integer, ByVal IsSession As Boolean)
             Try
                 InitializeComponent()
                 Me.Media = Media
@@ -278,7 +278,7 @@ Namespace DownloadObjects
                         End With
                     End If
 
-                    If Settings.FeedAddSessionToCaption Then info = $"[{Media.Session}] {info}"
+                    If Settings.FeedAddSessionToCaption And IsSession Then info = $"[{Media.Session}] {info}"
                     If Settings.FeedAddDateToCaption Then info &= $" ({Media.Date.ToStringDate(ADateTime.Formats.BaseDateTime)})"
                     LBL_INFO.Text = info
                     If Not Media.User Is Nothing AndAlso Not Media.User.HOST Is Nothing Then

SCrawler/Download/TDownloader.vb

@@ -26,6 +26,7 @@ Namespace DownloadObjects
 #End Region
 #Region "Declarations"
 #Region "Files"
+        Friend Const Name_SessionXML As String = "Session"
         Friend Structure UserMediaD : Implements IComparable(Of UserMediaD), IEquatable(Of UserMediaD), IEContainerProvider
 #Region "XML Names"
             Private Const Name_Data As String = "Data"
@@ -40,7 +41,7 @@ Namespace DownloadObjects
             Friend ReadOnly Data As UserMedia
             Friend ReadOnly UserInfo As UserInfo
             Friend ReadOnly [Date] As Date
-            Friend ReadOnly Session As Integer
+            Friend Session As Integer
             Friend IsSavedPosts As Boolean
             Friend Sub New(ByVal Data As UserMedia, ByVal User As IUserData, ByVal Session As Integer)
                 Me.Data = Data
@@ -122,19 +123,29 @@ Namespace DownloadObjects
                 Return _FilesSessionActual
             End Get
         End Property
+        Private _FilesSaving As Boolean = False
         Friend Sub FilesSave()
+            While _FilesSaving Or _FilesUpdating : Thread.Sleep(100) : End While
+            Dim i% = 0
+            While Not FilesSaveImpl() And i < 10 : i += 1 : End While
+        End Sub
+        Private Function FilesSaveImpl() As Boolean
+            _FilesSaving = True
             Try
                 If Settings.FeedStoreSessionsData And Files.Count > 0 Then
                     ClearSessions()
-                    Using x As New XmlFile With {.Name = "Session", .AllowSameNames = True}
+                    Using x As New XmlFile With {.Name = Name_SessionXML, .AllowSameNames = True}
                         x.AddRange(Files)
                         x.Save(FilesSessionActual)
                     End Using
                 End If
+                Return True
             Catch ex As Exception
-                ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadObjects.TDownloader.FilesSave]")
+                Return ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadObjects.TDownloader.FilesSave]", False)
+            Finally
+                _FilesSaving = False
             End Try
-        End Sub
+        End Function
         Private _FilesUpdating As Boolean = False
         Friend Sub FilesUpdatePendingUsers()
             _FilesUpdating = True

SCrawler/Editors/GlobalSettingsForm.Designer.vb

@@ -157,6 +157,7 @@ Namespace Editors
             Me.TXT_FEED_CENTER_IMAGE = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.COLORS_FEED = New SCrawler.Editors.ColorPicker()
             Me.CH_FEED_SHOW_FRIENDLY = New System.Windows.Forms.CheckBox()
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM = New System.Windows.Forms.CheckBox()
             Me.TXT_YTDLP = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TXT_FFMPEG = New PersonalUtilities.Forms.Controls.TextBoxExtended()
             Me.TXT_CURL = New PersonalUtilities.Forms.Controls.TextBoxExtended()
@@ -177,7 +178,7 @@ Namespace Editors
             Me.TAB_MAIN = New System.Windows.Forms.TabControl()
             Me.TAB_ENVIR = New System.Windows.Forms.TabPage()
             Me.CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM = New System.Windows.Forms.CheckBox()
+            Me.CH_NOTIFY_LOG = New System.Windows.Forms.CheckBox()
             TP_BASIS = New System.Windows.Forms.TableLayoutPanel()
             TP_IMAGES = New System.Windows.Forms.TableLayoutPanel()
             TP_FILE_NAME = New System.Windows.Forms.TableLayoutPanel()
@@ -1670,6 +1671,17 @@ Namespace Editors
             Me.CH_FEED_SHOW_FRIENDLY.Text = "Show friendly names instead of usernames"
             Me.CH_FEED_SHOW_FRIENDLY.UseVisualStyleBackColor = True
             '
+            'CH_FEED_SHOW_SPEC_MEDIAITEM
+            '
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.AutoSize = True
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Location = New System.Drawing.Point(4, 247)
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Name = "CH_FEED_SHOW_SPEC_MEDIAITEM"
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Size = New System.Drawing.Size(568, 19)
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.TabIndex = 9
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Text = "Show special feeds in media items"
+            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.UseVisualStyleBackColor = True
+            '
             'TAB_NOTIFY
             '
             TAB_NOTIFY.Controls.Add(TP_NOTIFY_MAIN)
@@ -1692,10 +1704,12 @@ Namespace Editors
             TP_NOTIFY_MAIN.Controls.Add(Me.CH_NOTIFY_SAVED_POSTS, 0, 5)
             TP_NOTIFY_MAIN.Controls.Add(Me.CH_STD, 0, 6)
             TP_NOTIFY_MAIN.Controls.Add(Me.CH_STD_EVERY, 0, 7)
+            TP_NOTIFY_MAIN.Controls.Add(Me.CH_NOTIFY_LOG, 0, 8)
             TP_NOTIFY_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
             TP_NOTIFY_MAIN.Location = New System.Drawing.Point(0, 0)
             TP_NOTIFY_MAIN.Name = "TP_NOTIFY_MAIN"
-            TP_NOTIFY_MAIN.RowCount = 9
+            TP_NOTIFY_MAIN.RowCount = 10
+            TP_NOTIFY_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_NOTIFY_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_NOTIFY_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
             TP_NOTIFY_MAIN.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!))
@@ -2140,16 +2154,17 @@ Namespace Editors
             Me.CONTAINER_MAIN.TabIndex = 0
             Me.CONTAINER_MAIN.TopToolStripPanelVisible = False
             '
-            'CH_FEED_SHOW_SPEC_MEDIAITEM
+            'CH_NOTIFY_LOG
             '
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.AutoSize = True
+            Me.CH_NOTIFY_LOG.AutoSize = True
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Dock = System.Windows.Forms.DockStyle.Fill
+            Me.CH_NOTIFY_LOG.Dock = System.Windows.Forms.DockStyle.Fill
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Location = New System.Drawing.Point(4, 247)
+            Me.CH_NOTIFY_LOG.Location = New System.Drawing.Point(4, 212)
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Name = "CH_FEED_SHOW_SPEC_MEDIAITEM"
+            Me.CH_NOTIFY_LOG.Name = "CH_NOTIFY_LOG"
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Size = New System.Drawing.Size(568, 19)
+            Me.CH_NOTIFY_LOG.Size = New System.Drawing.Size(568, 19)
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.TabIndex = 9
+            Me.CH_NOTIFY_LOG.TabIndex = 8
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.Text = "Show special feeds in media items"
+            Me.CH_NOTIFY_LOG.Text = "The log contains new data"
-            Me.CH_FEED_SHOW_SPEC_MEDIAITEM.UseVisualStyleBackColor = True
+            TT_MAIN.SetToolTip(Me.CH_NOTIFY_LOG, "Show a notification when the new data is added to the log.")
+            Me.CH_NOTIFY_LOG.UseVisualStyleBackColor = True
             '
             'GlobalSettingsForm
             '
@@ -2341,5 +2356,6 @@ Namespace Editors
         Private WithEvents CH_STD_YT_CREATE_URL As CheckBox
         Private WithEvents CH_USE_DEF_ACC As CheckBox
         Private WithEvents CH_FEED_SHOW_SPEC_MEDIAITEM As CheckBox
+        Private WithEvents CH_NOTIFY_LOG As CheckBox
     End Class
 End Namespace

SCrawler/Editors/GlobalSettingsForm.vb

@@ -74,6 +74,7 @@ Namespace Editors
                         CH_NOTIFY_SAVED_POSTS.Checked = .ShowNotificationsDownSavedPosts
                         CH_STD.Checked = .ShowNotificationsSTDownloader
                         CH_STD_EVERY.Checked = .ShowNotificationsSTDownloaderEveryDownload
+                        CH_NOTIFY_LOG.Checked = .ShowNotificationsLOG
                         'Defaults
                         CH_SEPARATE_VIDEO_FOLDER.Checked = .SeparateVideoFolder.Value
                         CH_DEF_TEMP.Checked = .DefaultTemporary
@@ -248,6 +249,7 @@ Namespace Editors
                     .ShowNotificationsDownSavedPosts.Value = CH_NOTIFY_SAVED_POSTS.Checked
                     .ShowNotificationsSTDownloader.Value = CH_STD.Checked
                     .ShowNotificationsSTDownloaderEveryDownload.Value = CH_STD_EVERY.Checked
+                    .ShowNotificationsLOG.Value = CH_NOTIFY_LOG.Checked
                     'Defaults
                     .SeparateVideoFolder.Value = CH_SEPARATE_VIDEO_FOLDER.Checked
                     .DefaultTemporary.Value = CH_DEF_TEMP.Checked

SCrawler/Editors/SiteEditorForm.vb

@@ -213,7 +213,7 @@ Namespace Editors
                             Dim loAdded As Boolean = False
                             Dim pArr() As Boolean
                             If .PropList.Exists(Function(p) If(p.Options?.IsAuth, False)) Then pArr = {True, False} Else pArr = {False}
-                            .PropList.Sort()
+                            If .PropList.Exists(Function(p) p.ControlNumber >= 0) Then .PropList.Sort()
                             For Each pAuth As Boolean In pArr
                                 For Each prop As PropertyValueHost In .PropList
                                     If Not prop.Options Is Nothing Then

SCrawler/Editors/UserCreatorForm.vb

@@ -430,6 +430,7 @@ Namespace Editors
                                 If Not UserInstance Is Nothing Then
                                     With DirectCast(UserInstance, UserDataBase)
                                         .User = User
+                                        .ResetHost()
                                         Dim setFriendly As Boolean = True
                                         If FriendlyNameIsSiteName Then
                                             If Not FriendlyNameChanged Then
@@ -590,7 +591,7 @@ CloseForm:
         End Sub
         Private _AccountsRefilling As Boolean = False
         Private Sub CMB_ACCOUNT_ActionSelectedItemChanged(ByVal Sender As Object, ByVal e As EventArgs, ByVal Item As ListViewItem) Handles CMB_ACCOUNT.ActionSelectedItemChanged
-            If Not _AccountsRefilling Then SetParamsBySite(False)
+            If Not _AccountsRefilling And UserInstance Is Nothing Then SetParamsBySite(False)
         End Sub
         Private Sub CH_TEMP_CheckedChanged(sender As Object, e As EventArgs) Handles CH_TEMP.CheckedChanged
             If CH_TEMP.Checked Then CH_FAV.Checked = False : CH_READY_FOR_DOWN.Checked = False

SCrawler/MainFrame.Designer.vb

@@ -139,6 +139,7 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         Me.BTT_TRAY_FEED_SHOW = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_CHANNELS = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_DOWNLOADER = New System.Windows.Forms.ToolStripMenuItem()
+        Me.BTT_TRAY_SCHEDULER = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_SHOW_HIDE = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_CLOSE = New System.Windows.Forms.ToolStripMenuItem()
         Me.BTT_TRAY_CLOSE_NO_SCRIPT = New System.Windows.Forms.ToolStripMenuItem()
@@ -987,9 +988,9 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         '
         'TRAY_CONTEXT
         '
-        Me.TRAY_CONTEXT.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_TRAY_PAUSE_AUTOMATION, Me.BTT_TRAY_SILENT_MODE, Me.BTT_TRAY_FEED_SHOW, Me.BTT_TRAY_CHANNELS, Me.BTT_TRAY_DOWNLOADER, TRAY_SEP_1, Me.BTT_TRAY_SHOW_HIDE, TRAY_SEP_2, Me.BTT_TRAY_CLOSE, Me.BTT_TRAY_CLOSE_NO_SCRIPT})
+        Me.TRAY_CONTEXT.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_TRAY_PAUSE_AUTOMATION, Me.BTT_TRAY_SILENT_MODE, Me.BTT_TRAY_FEED_SHOW, Me.BTT_TRAY_CHANNELS, Me.BTT_TRAY_DOWNLOADER, Me.BTT_TRAY_SCHEDULER, TRAY_SEP_1, Me.BTT_TRAY_SHOW_HIDE, TRAY_SEP_2, Me.BTT_TRAY_CLOSE, Me.BTT_TRAY_CLOSE_NO_SCRIPT})
         Me.TRAY_CONTEXT.Name = "TRAY_CONTEXT"
-        Me.TRAY_CONTEXT.Size = New System.Drawing.Size(171, 192)
+        Me.TRAY_CONTEXT.Size = New System.Drawing.Size(171, 214)
         '
         'BTT_TRAY_PAUSE_AUTOMATION
         '
@@ -1032,6 +1033,13 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         Me.BTT_TRAY_DOWNLOADER.Size = New System.Drawing.Size(170, 22)
         Me.BTT_TRAY_DOWNLOADER.Text = "Downloader"
         '
+        'BTT_TRAY_SCHEDULER
+        '
+        Me.BTT_TRAY_SCHEDULER.Image = Global.SCrawler.My.Resources.Resources.ScriptPic_32
+        Me.BTT_TRAY_SCHEDULER.Name = "BTT_TRAY_SCHEDULER"
+        Me.BTT_TRAY_SCHEDULER.Size = New System.Drawing.Size(170, 22)
+        Me.BTT_TRAY_SCHEDULER.Text = "Scheduler"
+        '
         'BTT_TRAY_SHOW_HIDE
         '
         Me.BTT_TRAY_SHOW_HIDE.Image = Global.SCrawler.My.Resources.Resources.ApplicationPic_16
@@ -1181,4 +1189,5 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
     Private WithEvents MENU_INFO_SHOW_QUEUE As ToolStripMenuItem
     Private WithEvents BTT_DOWN_SPEC As ToolStripKeyMenuItem
     Private WithEvents BTT_SHOW_FILTER_ADV As ToolStripMenuItem
+    Private WithEvents BTT_TRAY_SCHEDULER As ToolStripMenuItem
 End Class

SCrawler/MainFrame.vb

@@ -603,7 +603,7 @@ CloseResume:
         ControlInvokeFast(Toolbar_TOP, BTT_DOWN_AUTOMATION_PAUSE, Sub() BTT_DOWN_AUTOMATION_PAUSE.Visible = b)
         ControlInvokeFast(Me, Sub() BTT_TRAY_PAUSE_AUTOMATION.Visible = b)
     End Sub
-    Private Async Sub BTT_DOWN_AUTOMATION_Click(sender As Object, e As EventArgs) Handles BTT_DOWN_AUTOMATION.Click
+    Private Async Sub BTT_DOWN_AUTOMATION_Click(sender As Object, e As EventArgs) Handles BTT_DOWN_AUTOMATION.Click, BTT_TRAY_SCHEDULER.Click
         Try
             Using f As New SchedulerEditorForm : f.ShowDialog() : End Using
             Await Settings.Automation.Start(False)
@@ -842,7 +842,7 @@ CloseResume:
 #End Region
 #End Region
     Private Sub BTT_LOG_Click(sender As Object, e As EventArgs) Handles BTT_LOG.Click
-        MyMainLOG_ShowForm(Settings.Design,,,, Sub() MainFrameObj.UpdateLogButton())
+        MainFrameObj.ShowLog()
     End Sub
     Private Sub BTT_VERSION_INFO_Click(sender As Object, e As EventArgs) Handles BTT_VERSION_INFO.Click
         CheckVersion(True)

SCrawler/MainFrameObjects.vb

@@ -19,6 +19,11 @@ Friend Class MainFrameObjects : Implements INotificator
         MF = f
         Notificator = New NotificationsManager
         PauseButtons = New DownloadObjects.AutoDownloaderPauseButtons(DownloadObjects.AutoDownloaderPauseButtons.ButtonsPlace.MainFrame)
+        ProgramLogInitialize()
+        With ProgramLog
+            AddHandler .TextAdded, AddressOf ProgramLog_TextAdded
+            AddHandler .TextCleared, AddressOf ProgramLog_TextCleared
+        End With
     End Sub
 #Region "Users"
     Friend Sub FocusUser(ByVal Key As String, Optional ByVal ActivateForm As Boolean = False)
@@ -65,6 +70,12 @@ Friend Class MainFrameObjects : Implements INotificator
     Friend Function GetUserListProvider(ByVal WithCollections As Boolean) As IFormatProvider
         Return MF.GetUserListProvider(WithCollections)
     End Function
+    Friend Sub ShowLog()
+        MyMainLOG_ShowForm(Settings.Design,,,, Sub()
+                                                   UpdateLogButton()
+                                                   LogFormClosed()
+                                               End Sub)
+    End Sub
 #End Region
 #Region "Notifications"
     Private Sub INotificator_ShowNotification(ByVal Text As String, ByVal Image As SFile) Implements INotificator.ShowNotification
@@ -88,6 +99,7 @@ Friend Class MainFrameObjects : Implements INotificator
                     Case $"{NotificationInternalKey}_{NotifyObj.Channels}" : MF.MyChannels.FormShowS()
                     Case $"{NotificationInternalKey}_{NotifyObj.SavedPosts}" : MF.MySavedPosts.FormShowS()
                     Case $"{NotificationInternalKey}_{NotifyObj.STDownloader}" : VideoDownloader.FormShowS()
+                    Case $"{NotificationInternalKey}_{NotifyObj.LOG}" : ControlInvokeFast(MF, AddressOf ShowLog, EDP.LogMessageValue)
                     Case Else : Focus(True)
                 End Select
             ElseIf Settings.Automation Is Nothing OrElse Not Settings.Automation.NotificationClicked(Key, found, activateForm) Then
@@ -100,4 +112,16 @@ Friend Class MainFrameObjects : Implements INotificator
         End If
     End Sub
 #End Region
+#Region "LOG events support"
+    Private _LogNotificationsEnabled As Boolean = True
+    Private Sub ProgramLog_TextAdded(ByVal Sender As Object, ByVal e As EventArgs)
+        If _LogNotificationsEnabled Then _LogNotificationsEnabled = False : ShowNotification(NotifyObj.LOG, "There is new data in the log")
+    End Sub
+    Private Sub ProgramLog_TextCleared(ByVal Sender As Object, ByVal e As EventArgs)
+        _LogNotificationsEnabled = True
+    End Sub
+    Friend Sub LogFormClosed()
+        _LogNotificationsEnabled = True
+    End Sub
+#End Region
 End Class

SCrawler/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2023.12.15.0")>
+<Assembly: AssemblyVersion("2024.1.26.0")>
-<Assembly: AssemblyFileVersion("2023.12.15.0")>
+<Assembly: AssemblyFileVersion("2024.1.26.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/PluginsEnvironment/Hosts/PropertyValueHost.vb

@@ -23,6 +23,7 @@ Namespace Plugin.Hosts
         Private ReadOnly Keeper As SettingsHost
         Protected Source As Object 'ReadOnly
         Protected Member As MemberInfo
+        Private ReadOnly MemberRef As MemberInfo
         Friend ReadOnly Options As PropertyOption
         Friend Overridable ReadOnly Property Name As String
         Protected _Type As Type
@@ -41,7 +42,7 @@ Namespace Plugin.Hosts
 #End Region
 #Region "Control"
         Friend Property Control As Control
-        Protected ControlNumber As Integer = -1
+        Friend Property ControlNumber As Integer = -1
         Friend ReadOnly Property ControlHeight As Integer
             Get
                 If Not Control Is Nothing Then
@@ -245,13 +246,12 @@ Namespace Plugin.Hosts
             Me.Keeper = Keeper
             Source = PropertySource
             Name = Member.Name
+            MemberRef = Member
 
             ControlNumber = If(Member.GetCustomAttribute(Of ControlNumber)()?.PropertyNumber, -1)
 
             If DirectCast(Member, PropertyInfo).PropertyType Is GetType(PropertyValue) Then
-                ExternalValue = DirectCast(DirectCast(Member, PropertyInfo).GetValue(Source), PropertyValue)
+                UpdateMember()
-                _Value = ExternalValue.Value
-                AddHandler ExternalValue.ValueChanged, AddressOf ExternalValueChanged
                 Options = Member.GetCustomAttribute(Of PropertyOption)()
                 IsTaskCounter = Not Member.GetCustomAttribute(Of TaskCounter)() Is Nothing
                 _XmlName = If(Member.GetCustomAttribute(Of PXML)()?.ElementName, String.Empty)
@@ -274,9 +274,22 @@ Namespace Plugin.Hosts
                 Next
             End If
         End Sub
+        Friend Sub UpdateMember()
+            If Not ExternalValue Is Nothing Then
+                Try : RemoveHandler ExternalValue.ValueChanged, AddressOf ExternalValueChanged : Catch : End Try
+            End If
+            _ExternalValue = DirectCast(DirectCast(MemberRef, PropertyInfo).GetValue(Source), PropertyValue)
+            _Value = ExternalValue.Value
+            AddHandler ExternalValue.ValueChanged, AddressOf ExternalValueChanged
+        End Sub
 #End Region
 #Region "Value"
-        Protected ReadOnly Property ExternalValue As PropertyValue
+        Private _ExternalValue As PropertyValue = Nothing
+        Private ReadOnly Property ExternalValue As PropertyValue
+            Get
+                Return _ExternalValue
+            End Get
+        End Property
         Friend ReadOnly Property XValue As IXMLValue
         Protected _Value As Object
         Friend Overloads Property Value As Object Implements IPropertyValue.Value

SCrawler/PluginsEnvironment/Hosts/SettingsHost.vb

@@ -143,6 +143,9 @@ Namespace Plugin.Hosts
             End Set
         End Property
         Friend ReadOnly Property [Default] As Boolean
+        Friend Sub DefaultInstanceChanged()
+            If PropList.Count > 0 Then PropList.ForEach(Sub(p) p.UpdateMember())
+        End Sub
         Friend ReadOnly Property IsSeparatedTasks As Boolean = False
         Friend ReadOnly Property IsSavedPostsCompatible As Boolean = False
         Private ReadOnly _TaskCountDefined As Integer? = Nothing

SCrawler/PluginsEnvironment/Hosts/SettingsHostCollection.vb

@@ -122,7 +122,7 @@ Namespace Plugin.Hosts
             AddHandler Host.OkClick, AddressOf Hosts_OkClick
             AddHandler Host.Deleted, AddressOf Hosts_Deleted
             AddHandler Host.CloneClick, AddressOf Hosts_CloneClick
-            If Host.Index > 0 Then Host.Source.DefaultInstance = [Default].Source
+            If Host.Index > 0 Then Host.Source.DefaultInstance = [Default].Source : Host.DefaultInstanceChanged()
         End Sub
         Private Sub Hosts_OkClick(ByVal Obj As SettingsHost)
             If Obj.Index = -1 Then

SCrawler/SCrawler.vbproj

@@ -194,6 +194,11 @@
     <Compile Include="API\Mastodon\SiteSettings.vb" />
     <Compile Include="API\Mastodon\UserData.vb" />
     <Compile Include="API\OnlyFans\Declarations.vb" />
+    <Compile Include="API\OnlyFans\OFResources.Designer.vb">
+      <AutoGen>True</AutoGen>
+      <DesignTime>True</DesignTime>
+      <DependentUpon>OFResources.resx</DependentUpon>
+    </Compile>
     <Compile Include="API\OnlyFans\SiteSettings.vb" />
     <Compile Include="API\OnlyFans\UserData.vb" />
     <Compile Include="API\OnlyFans\UserExchangeOptions.vb" />
@@ -490,6 +495,11 @@
     <EmbeddedResource Include="API\BaseObjects\InternalSettingsForm.resx">
       <DependentUpon>InternalSettingsForm.vb</DependentUpon>
     </EmbeddedResource>
+    <EmbeddedResource Include="API\OnlyFans\OFResources.resx">
+      <CustomToolNamespace>My.Resources</CustomToolNamespace>
+      <Generator>ResXFileCodeGenerator</Generator>
+      <LastGenOutput>OFResources.Designer.vb</LastGenOutput>
+    </EmbeddedResource>
     <EmbeddedResource Include="API\Reddit\RedditViewSettingsForm.resx">
       <DependentUpon>RedditViewSettingsForm.vb</DependentUpon>
     </EmbeddedResource>
@@ -588,6 +598,7 @@
   </ItemGroup>
   <ItemGroup>
     <None Include=".editorconfig" />
+    <None Include="API\OnlyFans\OFScraperConfigPattern.json" />
     <None Include="Content\Pictures\ApplicationPic_16.png" />
     <None Include="Content\Pictures\BookmarkBlack_16.png" />
     <None Include="Content\Pictures\DBPic_32.png" />

SCrawler/SettingsCLS.vb

@@ -27,6 +27,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
     Friend Const CookieEncryptKey As String = "SCrawlerCookiesEncryptKeyword"
     Private Const EnvironmentPath As String = "Environment\"
     Friend Const CollectionsFolderName As String = "Collections"
+    Private Const PermanentCacheSnapshotsPath As String = "_CacheSnapshots\"
     Friend Const DefaultCmdEncoding As Integer = BatchExecutor.UnicodeEncoding
     Friend ReadOnly Design As XmlFile
     Private ReadOnly MyXML As XmlFile
@@ -113,7 +114,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
     Friend ReadOnly Property CacheSnapshots(ByVal Permanent As Boolean) As CacheKeeper
         Get
             If Permanent Then
-                If _CacheSnapshots Is Nothing Then _CacheSnapshots = New CacheKeeper("_CacheSnapshots\") With {.DeleteCacheOnDispose = False, .DeleteRootOnDispose = False}
+                If _CacheSnapshots Is Nothing Then _CacheSnapshots = New CacheKeeper(PermanentCacheSnapshotsPath) With {.DeleteCacheOnDispose = False, .DeleteRootOnDispose = False}
                 Return _CacheSnapshots
             Else
                 Dim dir As SFile = $"{Cache.RootDirectory.PathWithSeparator}Snapshots\"
@@ -336,6 +337,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
         ShowNotificationsDownSavedPosts = New XMLValue(Of Boolean)("SavedPosts", True, MyXML, n)
         ShowNotificationsSTDownloader = New XMLValue(Of Boolean)("STDownloader", True, MyXML, n)
         ShowNotificationsSTDownloaderEveryDownload = New XMLValue(Of Boolean)("STDownloaderEveryDownload", True, MyXML, n)
+        ShowNotificationsLOG = New XMLValue(Of Boolean)("LOG", True, MyXML, n)
 
         ProgramText = New XMLValue(Of String)("ProgramText",, MyXML)
         ProgramDescription = New XMLValue(Of String)("ProgramDescription",, MyXML)
@@ -672,6 +674,14 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
     Friend Sub DeleteCachePath()
         Reddit.ChannelsCollection.ChannelsPathCache.Delete(SFO.Path, SFODelete.None, EDP.None)
     End Sub
+    Private Sub DeleteCachePathPermanent()
+        Try
+            Dim f As New SFile(PermanentCacheSnapshotsPath)
+            If f.Exists(SFO.Path, False) AndAlso Not SFile.GetFiles(f,, IO.SearchOption.AllDirectories, EDP.ReturnValue).ListExists Then _
+               f.Delete(SFO.Path, SFODelete.DeletePermanently, EDP.None)
+        Catch
+        End Try
+    End Sub
     Friend Overloads Function UserExists(ByVal UserSite As String, ByVal UserID As String) As Boolean
         Dim UserFinderBase As Predicate(Of IUserData) = Function(user) user.Site = UserSite And user.Name = UserID
         Dim UserFinder As Predicate(Of IUserData) = Function(ByVal user As IUserData) As Boolean
@@ -944,6 +954,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
         Channels
         SavedPosts
         STDownloader
+        LOG
     End Enum
     Friend ReadOnly Property ProcessNotification(ByVal Sender As NotificationObjects) As Boolean
         Get
@@ -955,6 +966,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
                     Case NotificationObjects.Channels : Return ShowNotificationsDownChannels
                     Case NotificationObjects.SavedPosts : Return ShowNotificationsDownSavedPosts
                     Case NotificationObjects.STDownloader : Return ShowNotificationsSTDownloader
+                    Case NotificationObjects.LOG : Return ShowNotificationsLOG
                     Case Else : Return True
                 End Select
             Else
@@ -970,6 +982,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
     Friend ReadOnly Property ShowNotificationsDownSavedPosts As XMLValue(Of Boolean)
     Friend ReadOnly Property ShowNotificationsSTDownloader As XMLValue(Of Boolean)
     Friend ReadOnly Property ShowNotificationsSTDownloaderEveryDownload As XMLValue(Of Boolean)
+    Friend ReadOnly Property ShowNotificationsLOG As XMLValue(Of Boolean)
 #End Region
 #Region "Other program properties"
     Friend ReadOnly Property ProgramText As XMLValue(Of String)
@@ -1007,6 +1020,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable
                 End If
                 If Not Automation Is Nothing Then Automation.Dispose()
                 Cache.Dispose()
+                DeleteCachePathPermanent()
                 Plugins.Clear()
                 LastCollections.Clear()
                 Users.ListClearDispose