2026.2.14.0

TokenBatch: add 'MyWorkingDirectory' property
API.Instagram: update 'ID' extraction; reduce the number of downloaded stories (GQL) to 5
API.Twitter: get a new username based on the user ID
API.XHamster: videos aren't downloading
Feed: fix a bug when removing from favorites

Changelog.md

@@ -2,11 +2,58 @@
 - [ffmpeg](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ffmpeg)
   - x64 version - [release](https://github.com/GyanD/codexffmpeg/releases/tag/5.1.2); [zip](https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build.zip); **version `5.1.2-full_build-www.gyan.dev`**
   - x86 version - [release](https://github.com/yt-dlp/FFmpeg-Builds/releases/tag/autobuild-2022-11-30-12-57); [zip](https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2022-11-30-12-57/ffmpeg-N-109274-gd7a5f068c2-win32-gpl.zip); **version `N-109457-geeb280f351-20221226`**
-- [Gallery-dl](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl) - **1.30.10**
-- [YT-DLP](https://github.com/AAndyProgram/SCrawler/wiki/Settings#yt-dlp) - **2025.11.12**
+- [Gallery-dl](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl) - **1.31.6**
+- [YT-DLP](https://github.com/AAndyProgram/SCrawler/wiki/Settings#yt-dlp) - **2026.02.04.233607**
 - [Deno](https://github.com/AAndyProgram/SCrawler/wiki/Settings#deno) - latest *(`2.0.0` or higher)*
 - [OF-Scraper](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) - **3.12.9** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/3.12.9))
 
+# 2026
+
+## 2026.2.14.0
+
+*2026-02-14*
+
+- Added
+  - Sites:
+    - Twitter: get a new username based on the user ID
+  - Minor improvements
+- Updated
+  - gallery-dl up to version **1.31.6**
+  - yt-dlp up to version **2026.02.04.233607**
+- Fixed
+  - Sites:
+    - **Instagram: some profiles aren't downloading**
+    - xHamster: videos aren't downloading
+  - Minor bugs
+
+## 2026.1.24.0
+
+*2026-01-24*
+
+- Updated
+  - gallery-dl up to version **1.31.4**
+- Fixed
+  - Minor bugs
+
+## 2026.1.17.0
+
+*2026-01-17*
+
+- Added
+  - Sites:
+    - OnlyFans: handling error `502`
+    - Threads: user name and description extraction
+    - TikTok: **downloading `Stories` and `Reposts`**
+  - Download groups: excluded groups
+- Updated
+  - yt-dlp up to version **2025.12.08**
+  - gallery-dl up to version **1.31.3**
+- Fixed
+  - Sites:
+    - PornHub: videos aren't downloading
+    - TikTok: new videos aren't downloading
+    - xHamster: new users aren't added in some cases
+
 # 2025
 
 ## 2025.11.25.0
@@ -1765,7 +1812,7 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
   - Wrong some Reddit videos parsing
   - Wrong some Reddit images parsing
 
-# 1.0.0.0
+## 1.0.0.0
 
 *2021-12-07*
 

FAQ.md

@@ -33,6 +33,7 @@ I strongly recommend you to **regularly** create backup copies of the settings f
 ## General questions
 - **PROFILES**
     - I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
+    - :exclamation: **Try to avoid Chinese/Japanese symbols in the paths.**
     - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). **Don't forget to attach the LOG.**
     - [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
     - How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.

SCrawler/API/Base/TokenBatch.vb

@@ -13,12 +13,17 @@ Namespace API.Base
         Friend Property TempPostsList As List(Of String)
         Protected ReadOnly Token As CancellationToken
         Friend Property DebugMode As Boolean = False
+        Friend Overridable Property MyWorkingDirectory As SFile = Nothing
         Friend Sub New(ByVal _Token As CancellationToken, Optional ByVal _MainProcessName As String = Nothing, Optional ByVal WorkingDir As SFile = Nothing)
             MyBase.New(True)
             Token = _Token
             MainProcessName = _MainProcessName
             If Not WorkingDir.IsEmptyString Then ChangeDirectory(WorkingDir)
         End Sub
+        Public Overrides Sub ChangeDirectory(ByVal Directory As SFile)
+            MyBase.ChangeDirectory(Directory)
+            If Not Directory.IsEmptyString Then MyWorkingDirectory = Directory
+        End Sub
         Protected Overrides Function Internal_Execute(ByVal Commands As IEnumerable(Of String), ByVal e As ErrorsDescriber) As Boolean
             If Not Encoding.HasValue Then Encoding = UnicodeEncoding
             Return MyBase.Internal_Execute(Commands, e)

SCrawler/API/Base/UserDataBase.vb

@@ -1439,6 +1439,16 @@ BlockNullPicture:
             Cache.Validate()
             Return Cache
         End Function
+#Region "GDL File Names"
+        Protected GDLFileNameProvider As ANumbers = Nothing
+        Protected Sub GDLResetFileNameProvider(Optional ByVal GroupSize As Integer? = Nothing)
+            GDLFileNameProvider = New ANumbers With {.FormatOptions = ANumbers.Options.FormatNumberGroup + ANumbers.Options.Groups}
+            GDLFileNameProvider.GroupSize = If(GroupSize, 3)
+        End Sub
+        Protected Function GDLRenameFile(ByVal Input As SFile, ByVal i As Integer) As SFile
+            Return SFile.Rename(Input, $"{Input.PathWithSeparator}{i.NumToString(GDLFileNameProvider)}.{Input.Extension}",, EDP.ThrowException)
+        End Function
+#End Region
 #Region "DownloadSingleObject"
         Protected IsSingleObjectDownload As Boolean = False
         Friend Overridable Sub DownloadSingleObject(ByVal Data As YouTube.Objects.IYouTubeMediaContainer, ByVal Token As CancellationToken) Implements IUserData.DownloadSingleObject
@@ -2453,6 +2463,7 @@ stxt:
                     _TempPostsList.Clear()
                     _MD5List.Clear()
                     TokenPersonal = Nothing
+                    GDLFileNameProvider = Nothing
                     If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
                     If Not Responser Is Nothing Then Responser.Dispose()
                     If Not BTT_CONTEXT_DOWN Is Nothing Then BTT_CONTEXT_DOWN.Dispose()

SCrawler/API/Instagram/Declarations.vb

@@ -20,6 +20,7 @@ Namespace API.Instagram
         Friend Const PageTokenRegexPatternDefault As String = "\[\],{""token"":""(.*?)""},\d+\]"
         Friend ReadOnly Regex_UserToken_dtsg As RParams = RParams.DMS("DTSGInitialData["":,.\[\]]*?{\s*.token.:\s*""([^""]+)", 1, EDP.ReturnValue)
         Friend ReadOnly Regex_UserToken_lsd As RParams = RParams.DMS("LSD["":,.\[\]]*?{\s*.token.:\s*""([^""]+)", 1, EDP.ReturnValue)
+        Friend ReadOnly Regex_ProfileID As RParams = RParams.DMS("profilePage_(\d+)", 1, EDP.ReturnValue)
         Friend Sub UpdateResponser(ByVal Source As IResponse, ByRef Destination As Responser, ByVal UpdateWwwClaim As Boolean)
             Const r_wwwClaimName$ = "x-ig-set-www-claim"
             Const r_tokenName$ = SiteSettings.Header_CSRF_TOKEN_COOKIE

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -36,6 +36,8 @@ Namespace API.Instagram
         <PSetting(Caption:="Place the extracted image into the video folder")>
         Friend Property PutImageVideoFolder As Boolean
         Friend Overrides Property UserName As String
+        <PSetting(Address:=SettingAddress.User, Caption:="Verified profile", ToolTip:="This profile has a verified mark")>
+        Friend Property IsVerifiedProfile As Boolean = False
         <PSetting(Address:=SettingAddress.User, Caption:="Force update UserName", ToolTip:="Try to force update UserName if it is not found on the site")>
         Friend Property ForceUpdateUserName As Boolean = False
         <PSetting(Address:=SettingAddress.User, Caption:="Force update user information")>
@@ -57,6 +59,8 @@ Namespace API.Instagram
 
                 PutImageVideoFolder = .PutImageVideoFolder
 
+                IsVerifiedProfile = .IsVerifiedProfile
+
                 ForceUpdateUserName = .ForceUpdateUserName
                 ForceUpdateUserInfo = .ForceUpdateUserInfo
             End With

SCrawler/API/Instagram/SiteSettings.vb

@@ -148,6 +148,8 @@ Namespace API.Instagram
 #End Region
         <PropertyOption(ControlText:="Use GraphQL to download", IsAuth:=True), PXML, PClonable>
         Friend ReadOnly Property USE_GQL As PropertyValue
+        <PropertyOption(ControlText:="Use GraphQL to download user data", IsAuth:=True), PXML, PClonable, HiddenControl>
+        Friend ReadOnly Property USE_GQL_UserData As PropertyValue
 #End Region
 #Region "Download data"
         <PropertyOption(ControlText:="Download timeline", Category:=CAT_DOWN), PXML, PClonable>
@@ -165,6 +167,14 @@ Namespace API.Instagram
         <PropertyOption(ControlText:="Download tagged posts", Category:=CAT_DOWN), PXML, PClonable>
         Friend ReadOnly Property DownloadTagged As PropertyValue
         <PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue
+        <PropertyOption(ControlText:="Number of posts (verified)", ControlToolTip:="The number of posts received per request if the profile has a verified mark", Category:=CAT_DOWN), PXML, PClonable, HiddenControl>
+        Friend ReadOnly Property PostNumberVerified As PropertyValue
+        <Provider(NameOf(PostNumberVerified), FieldsChecker:=True)>
+        Private ReadOnly Property PostNumberVerifiedProvider As IFormatProvider
+        <PropertyOption(ControlText:="Number of posts (unverified)", ControlToolTip:="The number of posts received per request if the profile doesn't have a verified mark", Category:=CAT_DOWN), PXML, PClonable, HiddenControl>
+        Friend ReadOnly Property PostNumberVerifiedNot As PropertyValue
+        <Provider(NameOf(PostNumberVerifiedNot), FieldsChecker:=True)>
+        Private ReadOnly Property PostNumberVerifiedNotProvider As IFormatProvider
 #End Region
 #Region "Timers"
         Friend Const TimersUrgentTip As String = vbCr & "It is highly recommended not to change the default value."
@@ -485,6 +495,7 @@ Namespace API.Instagram
             HH_IG_WWW_CLAIM_USE_DEFAULT_ALGO = New PropertyValue(True)
             TokenUpdateIntervalProvider = New TokenRefreshIntervalProvider
             USE_GQL = New PropertyValue(False)
+            USE_GQL_UserData = New PropertyValue(True)
 
             DownloadTimeline = New PropertyValue(True)
             DownloadTimeline_Def = New PropertyValue(DownloadTimeline.Value, GetType(Boolean))
@@ -496,6 +507,10 @@ Namespace API.Instagram
             DownloadStoriesUser_Def = New PropertyValue(DownloadStoriesUser.Value, GetType(Boolean))
             DownloadTagged = New PropertyValue(False)
             DownloadTagged_Def = New PropertyValue(DownloadTagged.Value, GetType(Boolean))
+            PostNumberVerified = New PropertyValue(50)
+            PostNumberVerifiedProvider = New TimersChecker(12)
+            PostNumberVerifiedNot = New PropertyValue(12)
+            PostNumberVerifiedNotProvider = New TimersChecker(12)
 
             RequestsWaitTimer_Any = New PropertyValue(1000)
             RequestsWaitTimer_AnyProvider = New TimersChecker(0)
@@ -545,18 +560,17 @@ Namespace API.Instagram
             UserRegex = RParams.DMS(String.Format(UserRegexDefaultPattern, "instagram.com/"), 1)
             ImageVideoContains = "instagram.com"
         End Sub
-        Private Const SettingsVersionCurrent As Integer = 2
+        Private Const SettingsVersionCurrent As Integer = 3
         Friend Overrides Sub EndInit()
             Try : MyLastRequests.Add(LastDownloadDate.Value, LastRequestsCount.Value) : Catch : End Try
             If Not CBool(HH_IG_WWW_CLAIM_USE.Value) Then Responser.Headers.Remove(Header_IG_WWW_CLAIM)
             If CInt(SettingsVersion.Value) < SettingsVersionCurrent Then
                 SettingsVersion.Value = SettingsVersionCurrent
-                HH_IG_WWW_CLAIM_UPDATE_INTERVAL.Value = 120
-                HH_IG_WWW_CLAIM_ALWAYS_ZERO.Value = False
-                HH_IG_WWW_CLAIM_RESET_EACH_SESSION.Value = True
-                HH_IG_WWW_CLAIM_RESET_EACH_TARGET.Value = True
-                HH_IG_WWW_CLAIM_USE.Value = True
-                HH_IG_WWW_CLAIM_USE_DEFAULT_ALGO.Value = True
+                HH_IG_WWW_CLAIM_RESET_EACH_TARGET.Value = False
+                RequestsWaitTimer_Any.Value = 5000
+                TaggedNotifyLimit.Value = 50
+                DownDetectorValue.Value = 30
+                DownDetectorValueAddToLog.Value = True
             End If
             MyBase.EndInit()
         End Sub

SCrawler/API/Instagram/UserData.GQL.vb

@@ -6,12 +6,13 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports System.Security.Cryptography
 Imports System.Threading
-Imports SCrawler.API.Base
-Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Tools.Web.Clients
 Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports SCrawler.API.Base
 Namespace API.Instagram
     Partial Friend Class UserData
 #Region "Tokens"
@@ -43,9 +44,9 @@ Namespace API.Instagram
         Private Const GQL_UserStories_DocId As String = "25231722019806941"
         Private Const GQL_UserStories_FbFriendlyName As String = "PolarisStoriesV3ReelPageStandaloneQuery"
 
-        Private Const GQL_Timeline_DocId As String = "7268577773270422"
+        Private Const GQL_Timeline_DocId As String = "7268577773270422" '"34579740524958711" '"7268577773270422"
         Private Const GQL_Timeline_FbFriendlyName As String = "PolarisProfilePostsQuery"
-        Private Const GQL_Timeline_DocId_Second As String = "7286316061475375"
+        Private Const GQL_Timeline_DocId_Second As String = "7286316061475375" '"33944389991841132" '"7286316061475375"
         Private Const GQL_Timeline_FbFriendlyName_Second As String = "PolarisProfilePostsTabContentQuery_connection"
 
         Private Const GQL_Reels_DocId As String = "7191572580905225"
@@ -64,33 +65,42 @@ Namespace API.Instagram
             Responser.Headers.Add(GQL_HEADER_FB_FRINDLY_NAME, HeaderValue)
             Responser.Headers.Add(GQL_HEADER_FB_LSD, Token_lsd)
         End Sub
-        <Obsolete("Use 'GET' function: 'GetUserData'", False)>
-        Private Sub GetUserDataGQL(ByVal Token As CancellationToken)
+        '<Obsolete("Use 'GET' function: 'GetUserData'", False)>
+        Private Function GetUserDataGQL(ByVal Token As CancellationToken) As String
             Dim vars$ = String.Format(GQL_URL_PATTERN_VARS, GQL_UserData_DocId, Token_lsd, Token_dtsg_Var, GQL_UserData_FbFriendlyName,
                                       SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & $"""id"":""{ID}"",""relay_header"":false,""render_surface"":""PROFILE""" & "}"))
             UpdateRequestNumber()
             ChangeResponserMode(True)
             UpdateHeadersGQL(GQL_UserData_FbFriendlyName)
             Dim r$ = Responser.GetResponse(GQL_URL, vars)
-            If Not r.IsEmptyString Then
-                Using j As EContainer = JsonDocument.Parse(r)
-                    If j.ListExists Then
-                        With j({"data", "user"})
-                            If .ListExists Then
-                                UserSiteName = .Value("full_name").IfNullOrEmpty(UserSiteName)
-                                Dim f As New SFile With {.Path = DownloadContentDefault_GetRootDir(), .Name = "ProfilePicture", .Extension = "jpg"}
-                                Dim pic$ = .Value({"hd_profile_pic_url_info"}, "url").IfNullOrEmpty(.Value("profile_pic_url"))
-                                If Not pic.IsEmptyString Then GetWebFile(pic, f, EDP.ReturnValue)
-                                UserDescriptionUpdate(.Value("biography"))
-                            End If
-                        End With
-                    End If
-                End Using
-            End If
-        End Sub
+            Return r
+            'If Not r.IsEmptyString Then
+            '    Using j As EContainer = JsonDocument.Parse(r)
+            '        If j.ListExists Then
+            '            With j({"data", "user"})
+            '                If .ListExists Then
+            '                    UserSiteName = .Value("full_name").IfNullOrEmpty(UserSiteName)
+            '                    IsVerifiedProfile = .Value("is_verified").FromXML(Of Boolean)(False)
+            '                    IsVerifiedProfile_Checked = True
+            '                    Dim descr$ = .Value("biography")
+            '                    If If(.Item("bio_links")?.Count, 0) > 0 Then descr.StringAppend(.Item("bio_links").Select(Function(bl) bl.Value("url")).ListToString(vbNewLine), vbNewLine)
+            '                    Dim eUrl$ = .Value("external_url")
+            '                    If Not eUrl.IsEmptyString AndAlso (descr.IsEmptyString OrElse Not descr.Contains(eUrl)) Then descr.StringAppendLine(eUrl)
+            '                    UserDescriptionUpdate(descr)
+
+            '                    Dim f As New SFile With {.Path = DownloadContentDefault_GetRootDir(), .Name = "ProfilePicture", .Extension = "jpg"}
+            '                    Dim pic$ = .Value({"hd_profile_pic_url_info"}, "url").IfNullOrEmpty(.Value("profile_pic_url"))
+            '                    If Not pic.IsEmptyString Then GetWebFile(pic, f, EDP.ReturnValue)
+            '                End If
+            '            End With
+            '        End If
+            '    End Using
+            'End If
+        End Function
         Private Function GetTimelineGQL(ByVal Cursor As String, ByVal Token As CancellationToken) As String
             Const none_cursor$ = "none"
-            Dim nextCursor$ = String.Empty, hasNextPage$ = String.Empty
+            Dim nextCursor$ = String.Empty
+            Dim hasNextPage As Boolean = False
             Dim vars$
 
             ThrowAny(Token)
@@ -98,14 +108,18 @@ Namespace API.Instagram
             ChangeResponserMode(True)
 
             If Cursor.IsEmptyString Then
-                vars = "{""data"":{""count"":50,""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""username"":""" &
+                vars = "{""data"":{""count"":" & PostNumberPerRequest & ",""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""username"":""" &
                         NameTrue & """,""__relay_internal__pv__PolarisShareMenurelayprovider"":false}"
+                'vars = "{""data"":{""count"":" & PostNumberPerRequest & ",""include_reel_media_seen_timestamp"":true,""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""username"":""" &
+                '        NameTrue & """,""__relay_internal__pv__PolarisShareMenurelayprovider"":false}"
                 vars = String.Format(GQL_URL_PATTERN_VARS, GQL_Timeline_DocId, Token_lsd, Token_dtsg_Var, GQL_Timeline_FbFriendlyName,
                                      SymbolsConverter.ASCII.EncodeSymbolsOnly(vars))
                 UpdateHeadersGQL(GQL_Timeline_FbFriendlyName)
             Else
-                vars = "{""after"":""" & Cursor & """,""before"":null,""data"":{""count"":50,""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""first"":50,""last"":null,""username"":""" &
+                vars = "{""after"":""" & Cursor & """,""before"":null,""data"":{""count"":" & PostNumberPerRequest & ",""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""first"":" & PostNumberPerRequest & ",""last"":null,""username"":""" &
                         NameTrue & """,""__relay_internal__pv__PolarisShareMenurelayprovider"":false}"
+                'vars = "{""after"":""" & Cursor & """,""before"":null,""data"":{""count"":" & PostNumberPerRequest & ",""include_reel_media_seen_timestamp"":true,""include_relationship_info"":true,""latest_besties_reel_media"":true,""latest_reel_media"":true},""first"":" & PostNumberPerRequest & ",""last"":null,""username"":""" &
+                '        NameTrue & """}"
                 vars = String.Format(GQL_URL_PATTERN_VARS, GQL_Timeline_DocId_Second, Token_lsd, Token_dtsg_Var, GQL_Timeline_FbFriendlyName_Second,
                                      SymbolsConverter.ASCII.EncodeSymbolsOnly(vars))
                 UpdateHeadersGQL(GQL_Timeline_FbFriendlyName_Second)
@@ -140,7 +154,8 @@ Namespace API.Instagram
         End Function
         Private Function GetHighlightsGQL_List() As List(Of String)
 
-            Dim nextCursor$ = String.Empty, hasNextPage$ = String.Empty
+            Dim nextCursor$ = String.Empty
+            Dim hasNextPage As Boolean = False
             Dim i% = -1
             Dim hList As New List(Of String)
             Dim tmpList As New List(Of String)
@@ -178,7 +193,9 @@ Namespace API.Instagram
             Dim tmpList As New List(Of String)
             Dim i% = -1
             If StoriesList.ListExists Then
-                tmpList.AddRange(StoriesList.Take(10))
+                'TODO: 5 Instagram stories
+                'tmpList.AddRange(StoriesList.Take(10))
+                tmpList.AddRange(StoriesList.Take(5))
                 StoriesList.RemoveRange(0, tmpList.Count)
 
                 Dim vars$ = String.Format(GQL_URL_PATTERN_VARS, GQL_Highlights_DocId_Second, Token_lsd, Token_dtsg_Var, GQL_Highlights_FbFriendlyName_Second,
@@ -238,11 +255,9 @@ Namespace API.Instagram
         Private Function GetReelsGQL(ByVal Cursor As String) As String
             GetReelsGQL_SetEnvir = True
 
-            Dim errData$ = String.Empty
-            If Cursor.IsEmptyString And Not ValidateBaseTokens() Then GetPageTokens()
-            If Cursor.IsEmptyString And Not ValidateBaseTokens(errData) Then ValidateBaseTokens_Error(errData)
+            UpdateTokens(Cursor.IsEmptyString)
 
-            Dim vars$ = """data"":{""include_feed_video"":true,""page_size"":50,""target_user_id"":""" & ID & """}"
+            Dim vars$ = """data"":{""include_feed_video"":true,""page_size"":" & PostNumberPerRequest & ",""target_user_id"":""" & ID & """}"
             If Not Cursor.IsEmptyString Then vars = $"""after"":""{Cursor}"",""before"":null,{vars},""first"":4,""last"":null"
             vars = String.Format(GQL_URL_PATTERN_VARS, GQL_Reels_DocId, Token_lsd, Token_dtsg_Var, GQL_Reels_FbFriendlyName,
                                  SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & vars & "}"))
@@ -258,10 +273,10 @@ Namespace API.Instagram
             Dim vars$
             If Cursor.IsEmptyString Then
                 vars = String.Format(GQL_URL_PATTERN_VARS, GQL_Tagged_DocId, Token_lsd, Token_dtsg_Var, GQL_Tagged_FbFriendlyName,
-                                     SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & $"""count"":50,""user_id"":""{ID}""" & "}"))
+                                     SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & $"""count"":{PostNumberPerRequest},""user_id"":""{ID}""" & "}"))
             Else
                 vars = String.Format(GQL_URL_PATTERN_VARS, GQL_Tagged_DocId, Token_lsd, Token_dtsg_Var, GQL_Tagged_FbFriendlyName,
-                                     SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & $"""after"":""{Cursor}"",""before"":null,""count"":50,""first"":50,""last"":null,""user_id"":""{ID}""" & "}"))
+                                     SymbolsConverter.ASCII.EncodeSymbolsOnly("{" & $"""after"":""{Cursor}"",""before"":null,""count"":{PostNumberPerRequest},""first"":{PostNumberPerRequest},""last"":null,""user_id"":""{ID}""" & "}"))
             End If
             UpdateRequestNumber()
             ChangeResponserMode(True)
@@ -270,6 +285,13 @@ Namespace API.Instagram
         End Function
 #End Region
 #Region "ValidateBaseTokens"
+        Private Sub UpdateTokens(ByVal process As Boolean)
+            If process Then
+                Dim TokensErrData$ = String.Empty
+                If Not ValidateBaseTokens() Then GetPageTokens()
+                If Not ValidateBaseTokens(TokensErrData) Then ValidateBaseTokens_Error(TokensErrData)
+            End If
+        End Sub
         Protected Overridable Overloads Function ValidateBaseTokens() As Boolean
             Return ValidateBaseTokens(Nothing)
         End Function
@@ -307,6 +329,10 @@ Namespace API.Instagram
             Try
                 If Not r.IsEmptyString Then
                     ResetBaseTokens()
+                    If ID.IsEmptyString Then
+                        Dim __id$ = RegexReplace(r, Regex_ProfileID)
+                        If CLng(AConvert(Of Long)(__id, 0, EDP.ReturnValue)) <> 0 Then ID = __id
+                    End If
                     Select Case Attempt
                         Case 0
                             Dim rr As RParams = RParams.DM(PageTokenRegexPatternDefault, 0, RegexReturn.List, EDP.ReturnValue)

SCrawler/API/Instagram/UserData.vb

@@ -39,6 +39,8 @@ Namespace API.Instagram
         Private Const Name_TaggedChecked As String = "TaggedChecked"
         Private Const Name_ForceUpdateUserName As String = "ForceUpdateUserName"
         Private Const Name_ForceUpdateUserInfo As String = "ForceUpdateUserInfo"
+        Private Const Name_IsVerifiedProfile As String = "IsVerifiedProfile"
+        Private Const Name_IsVerifiedProfile_Checked As String = "IsVerifiedProfile_Checked"
 #End Region
 #Region "Declarations"
         Friend Structure PostKV : Implements IEContainerProvider
@@ -115,6 +117,13 @@ Namespace API.Instagram
         Private UserNameRequested As Boolean = False
         Friend Property ForceUpdateUserName As Boolean = False
         Friend Property ForceUpdateUserInfo As Boolean = False
+        Friend Property IsVerifiedProfile As Boolean = False
+        Friend Property IsVerifiedProfile_Checked As Boolean = False
+        Private ReadOnly Property PostNumberPerRequest As Integer
+            Get
+                With MySiteSettings : Return If(IsVerifiedProfile, .PostNumberVerified, .PostNumberVerifiedNot).Value : End With
+            End Get
+        End Property
 #End Region
 #Region "Loader"
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
@@ -136,6 +145,8 @@ Namespace API.Instagram
                     TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
                     ForceUpdateUserName = .Value(Name_ForceUpdateUserName).FromXML(Of Boolean)(False)
                     ForceUpdateUserInfo = .Value(Name_ForceUpdateUserInfo).FromXML(Of Boolean)(False)
+                    IsVerifiedProfile = .Value(Name_IsVerifiedProfile).FromXML(Of Boolean)(False)
+                    IsVerifiedProfile_Checked = .Value(Name_IsVerifiedProfile_Checked).FromXML(Of Boolean)(False)
                 Else
                     .Add(Name_LastCursor, LastCursor)
                     .Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
@@ -153,6 +164,8 @@ Namespace API.Instagram
                     .Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
                     .Add(Name_ForceUpdateUserName, ForceUpdateUserName.BoolToInteger)
                     .Add(Name_ForceUpdateUserInfo, ForceUpdateUserInfo.BoolToInteger)
+                    .Add(Name_IsVerifiedProfile, IsVerifiedProfile.BoolToInteger)
+                    .Add(Name_IsVerifiedProfile_Checked, IsVerifiedProfile_Checked.BoolToInteger)
                 End If
             End With
         End Sub
@@ -179,6 +192,9 @@ Namespace API.Instagram
 
                     PutImageVideoFolder = .PutImageVideoFolder
 
+                    IsVerifiedProfile = .IsVerifiedProfile
+                    If IsVerifiedProfile Then IsVerifiedProfile_Checked = True
+
                     ForceUpdateUserName = .ForceUpdateUserName
                     ForceUpdateUserInfo = .ForceUpdateUserInfo
                 End With
@@ -525,7 +541,7 @@ Namespace API.Instagram
         Protected Overrides Sub Responser_ResponseReceived(ByVal Sender As Object, ByVal e As EventArguments.WebDataResponse)
             Declarations.UpdateResponser(e, Responser, WwwClaimUpdate)
         End Sub
-        Friend Enum Sections : Timeline : Reels : Tagged : Stories : UserStories : SavedPosts : End Enum
+        Friend Enum Sections : Timeline : Reels : Tagged : Stories : UserStories : SavedPosts : Reposts : Likes : End Enum
         Protected Const StoriesFolder As String = "Stories"
         Private Const TaggedFolder As String = "Tagged"
 #Region "429 bypass"
@@ -663,6 +679,7 @@ Namespace API.Instagram
             Dim StoriesList As List(Of String) = Nothing
             Dim StoriesRequested As Boolean = False
             Dim dValue% = 1
+            Dim __idIsEmpty As Boolean = ID.IsEmptyString
             LastCursor = Cursor
             Try
                 Do While dValue = 1
@@ -676,7 +693,6 @@ Namespace API.Instagram
                         Dim HasNextPage As Boolean = False
                         Dim EndCursor$ = String.Empty
                         Dim PostID$ = String.Empty, PostDate$ = String.Empty, SpecFolder$ = String.Empty
-                        Dim TokensErrData$ = String.Empty
                         Dim PostIDKV As PostKV
                         Dim ENode() As Object = Nothing
                         Dim processGetResponse As Boolean = True
@@ -684,14 +700,11 @@ Namespace API.Instagram
 
                         'Check environment
                         If Not IsSavedPosts Then
-                            If ID.IsEmptyString Then GetUserData()
+                            If _UseGQL And Cursor.IsEmptyString And Not Section = Sections.SavedPosts Then UpdateTokens(True)
+                            If ID.IsEmptyString Or __idIsEmpty Or Not IsVerifiedProfile_Checked Then GetUserData(Token)
                             If ID.IsEmptyString Then UserExists = False : _ForceSaveUserInfoOnException = True : Throw New Plugin.ExitException("can't get user ID")
-                            If _UseGQL And Cursor.IsEmptyString And Not Section = Sections.SavedPosts Then
-                                If Not ValidateBaseTokens() Then GetPageTokens()
-                                If Not ValidateBaseTokens(TokensErrData) Then ValidateBaseTokens_Error(TokensErrData)
-                            End If
                             If ForceUpdateUserName Then GetUserNameById()
-                            If ForceUpdateUserInfo Then GetUserData()
+                            If ForceUpdateUserInfo Then GetUserData(Token)
                         End If
 
                         'Create query
@@ -703,7 +716,7 @@ Namespace API.Instagram
                                     MySiteSettings.TooManyRequests(False)
                                     GoTo NextPageBlock
                                 Else
-                                    URL = $"https://www.instagram.com/api/v1/feed/user/{NameTrue}/username/?count=50" &
+                                    URL = $"https://www.instagram.com/api/v1/feed/user/{NameTrue}/username/?count={PostNumberPerRequest}" &
                                            If(Cursor.IsEmptyString, String.Empty, $"&max_id={Cursor}")
                                     ENode = Nothing
                                 End If
@@ -726,7 +739,7 @@ Namespace API.Instagram
                                     ENode = {"data", "xdt_api__v1__usertags__user_id__feed_connection"}
                                     processGetResponse = False
                                 Else
-                                    Dim vars$ = "{""id"":" & ID & ",""first"":50,""after"":""" & Cursor & """}"
+                                    Dim vars$ = "{""id"":" & ID & $",""first"":{PostNumberPerRequest},""after"":""" & Cursor & """}"
                                     vars = SymbolsConverter.ASCII.EncodeSymbolsOnly(vars)
                                     URL = $"https://www.instagram.com/graphql/query/?doc_id=17946422347485809&variables={vars}"
                                     ENode = {"data", "user", "edge_user_to_photos_of_you"}
@@ -1241,25 +1254,37 @@ NextPageBlock:
         End Sub
 #End Region
 #Region "GetUserId, GetUserName"
-        Private Sub GetUserData()
+        Private Sub GetUserData(ByVal Token As CancellationToken)
             Dim __idFound As Boolean = False
             If ForceUpdateUserInfo Then ForceUpdateUserInfo = False : _ForceSaveUserInfo = True
             Try
+                Dim r$
+                Dim ____dataGql As Boolean = _UseGQL Or CBool(MySiteSettings.USE_GQL_UserData.Value)
+                If ____dataGql Then
+                    UpdateTokens(True)
+                    r = GetUserDataGQL(Token)
+                    If Not _UseGQL Then ChangeResponserMode(_UseGQL)
+                Else
                     ChangeResponserMode(False)
                     UpdateRequestNumber()
-                Dim r$ = Responser.GetResponse($"https://i.instagram.com/api/v1/users/web_profile_info/?username={NameTrue}")
+                    r = Responser.GetResponse($"https://i.instagram.com/api/v1/users/web_profile_info/?username={NameTrue}")
+                End If
+
                 If Not r.IsEmptyString Then
                     Using j As EContainer = JsonDocument.Parse(r)
                         If Not j Is Nothing AndAlso j.Contains({"data", "user"}) Then
                             With j({"data", "user"})
-                                ID = .Value("id")
+                                If Not ____dataGql Or ID.IsEmptyString Then ID = .Value("id")
                                 __idFound = True
                                 UserSiteNameUpdate(.Value("full_name"))
+                                IsVerifiedProfile = .Value("is_verified").FromXML(Of Boolean)(False)
+                                IsVerifiedProfile_Checked = True
                                 Dim descr$ = .Value("biography")
                                 If If(.Item("bio_links")?.Count, 0) > 0 Then descr.StringAppend(.Item("bio_links").Select(Function(bl) bl.Value("url")).ListToString(vbNewLine), vbNewLine)
                                 Dim eUrl$ = .Value("external_url")
                                 If Not eUrl.IsEmptyString AndAlso (descr.IsEmptyString OrElse Not descr.Contains(eUrl)) Then descr.StringAppendLine(eUrl)
                                 UserDescriptionUpdate(descr)
+
                                 Dim f As New SFile With {.Path = DownloadContentDefault_GetRootDir(), .Name = "ProfilePicture", .Extension = "jpg"}
                                 f = SFile.IndexReindex(f)
                                 If Not f.Exists Then
@@ -1434,7 +1459,7 @@ NextPageBlock:
                 MyMainLOG = $"Number of requests before error 429: {RequestsCount}"
                 Return 1
             ElseIf Responser.StatusCode = 560 Or Responser.StatusCode = HttpStatusCode.InternalServerError Then '560, 500
-                If Responser.StatusCode = 560 And s = Sections.Stories And MySiteSettings.IgnoreStoriesDownloadingErrors Then
+                If Responser.StatusCode = 560 And s = Sections.Stories And MySiteSettings.IgnoreStoriesDownloadingErrors.Value Then
                     MyMainLOG = $"{ToStringForLog()}: Stories downloading skipped (560)"
                     Return ErrHandlingValueStories
                 Else

SCrawler/API/OnlyFans/DynamicRulesEnv.vb

@@ -219,12 +219,12 @@ Namespace API.OnlyFans
             DynamicRulesXml.Extension = "xml"
             ReplacePattern_RepoToRaw = New RParams("(.*github.com/([^/]+)/([^/]+)/blob/(.+))", Nothing, 0,
                                                    RegexReturn.ReplaceChangeListMatch, EDP.ReturnValue) With {
-                                                   .PatternReplacement = "https://raw.githubusercontent.com/{2}/{3}/{4}"}
+                                                   .PatternReplacement = "https://raw.githubusercontent.com/{2}/{3}/refs/heads/{4}"}
             ReplacePattern_JsonInfo = ReplacePattern_RepoToRaw.Copy
             ReplacePattern_JsonInfo.PatternReplacement = "https://github.com/{2}/{3}/latest-commit/{4}"
             ReplacePattern_RawToRepo = ReplacePattern_RepoToRaw.Copy
-            ReplacePattern_RawToRepo.Pattern = "(.*raw.githubusercontent.com/([^/]+)/([^/]+)/([^/]+)/(.+))"
-            ReplacePattern_RawToRepo.PatternReplacement = "https://github.com/{2}/{3}/blob/{4}/{5}"
+            ReplacePattern_RawToRepo.Pattern = "(.*raw.githubusercontent.com/([^/]+)/([^/]+)(/refs/heads)?/([^/]+)/(.+))"
+            ReplacePattern_RawToRepo.PatternReplacement = "https://github.com/{2}/{3}/blob/{5}/{6}"
             ConfigRulesExtract = RParams.DMS("DYNAMIC_RULE"":(\{.+?\}[\r\n]+)", 1, RegexOptions.Singleline, EDP.ReturnValue)
             OFLOG = New TextSaver($"LOGs\OF_{Now:yyyyMMdd_HHmmss}.txt") With {.LogMode = True, .AutoSave = True, .AutoClear = True}
             AddHandler OFLOG.TextSaved, AddressOf OFLOG_TextSaved

SCrawler/API/OnlyFans/SiteSettings.vb

@@ -99,6 +99,19 @@ Namespace API.OnlyFans
             End Get
         End Property
 #End Region
+#Region "Other"
+        <PClonable, PXML("OpenPostsUsingID")> Private ReadOnly Property OpenPostsUsingID_XML As PropertyValue
+        <PropertyOption(ControlText:="Open posts using ID", ControlToolTip:="Open posts using the user ID instead of the user name"), HiddenControl>
+        Private ReadOnly Property OpenPostsUsingID As PropertyValue
+            Get
+                If Not DefaultInstance Is Nothing Then
+                    Return DirectCast(DefaultInstance, SiteSettings).OpenPostsUsingID_XML
+                Else
+                    Return OpenPostsUsingID_XML
+                End If
+            End Get
+        End Property
+#End Region
 #Region "OFScraper"
         <PClonable, PXML("OFScraperPath")> Private ReadOnly Property OFScraperPath_XML As PropertyValue
         <PropertyOption(ControlText:="OF-Scraper path", ControlToolTip:="The path to the 'ofscraper.exe'", Category:=CAT_OFS)>
@@ -285,6 +298,8 @@ Namespace API.OnlyFans
 
             UpdateRules401_XML = New PropertyValue(False)
 
+            OpenPostsUsingID_XML = New PropertyValue(True)
+
             UserRegex = RParams.DMS(String.Format(UserRegexDefaultPattern, "onlyfans.com/"), 1, EDP.ReturnValue)
             UrlPatternUser = "https://onlyfans.com/{0}"
             ImageVideoContains = "onlyfans.com"
@@ -363,8 +378,9 @@ Namespace API.OnlyFans
                 End If
                 If p.IsEmptyString Then
                     Return GetUserUrl(User)
+                ElseIf CBool(OpenPostsUsingID.Value) Then
+                    Return String.Format(UserPostPattern, p, If(User.ID.IsEmptyString, User.NameTrue, $"u{User.ID}"))
                 Else
-                    'Return String.Format(UserPostPattern, p, If(User.ID.IsEmptyString, User.Name, $"u{User.ID}"))
                     Return String.Format(UserPostPattern, p, User.NameTrue)
                 End If
             Else

SCrawler/API/OnlyFans/UserData.vb

@@ -88,6 +88,7 @@ Namespace API.OnlyFans
         Private _DownloadedPostsSession As Integer = 0
         Private FunctionErr As Integer = FunctionErrDef
         Private Const FunctionErrDef As Integer = -100
+        Private _TimelineDownloading As Boolean = False
         Private Sub ValidateOFScraper()
             _OFScraperExists = ACheck(MySettings.OFScraperPath.Value) AndAlso CStr(MySettings.OFScraperPath.Value).CSFile.Exists
         End Sub
@@ -110,7 +111,9 @@ Namespace API.OnlyFans
                         If ID.IsEmptyString Then Throw New ArgumentNullException("ID", "Unable to get user ID")
                     End If
 
+                    _TimelineDownloading = True
                     If MediaDownloadTimeline Then DownloadTimeline(IIf(IsSavedPosts, 0, String.Empty), Token)
+                    _TimelineDownloading = False
                     If Not IsSavedPosts Then
                         If MediaDownloadStories And FunctionErr = FunctionErrDef Then DownloadStories(Token)
                         If MediaDownloadHighlights And FunctionErr = FunctionErrDef Then DownloadHighlights(Token)
@@ -827,6 +830,8 @@ Namespace API.OnlyFans
                 Return 3
             ElseIf Responser.StatusCode = Net.HttpStatusCode.InternalServerError Then '500
                 Return 3
+            ElseIf Not _TimelineDownloading And Responser.StatusCode = Net.HttpStatusCode.BadGateway Then '502
+                Return 3
             Else
                 Return 0
             End If

SCrawler/API/PornHub/UserData.vb

@@ -385,7 +385,7 @@ Namespace API.PornHub
                                 If PersonType = PersonTypeCannel Then
                                     l = l.ListTake(4, l.Count)
                                 Else
-                                    l.RemoveAll(Function(uv) uv.UserRef.IsEmptyString OrElse Not uv.UserRef = usrRef)
+                                    l.RemoveAll(Function(uv) Not uv.UserRef.IsEmptyString AndAlso Not uv.UserRef = usrRef)
                                 End If
                             ElseIf Type = VideoTypes.Favorite Then
                                 l.RemoveAll(Function(uv) uv.Type = VideoTypes.Private)

SCrawler/API/ThreadsNet/Declarations.vb

@@ -0,0 +1,17 @@
+' Copyright (C) Andy https://github.com/AAndyProgram
+' This program is free software: you can redistribute it and/or modify
+' it under the terms of the GNU General Public License as published by
+' the Free Software Foundation, either version 3 of the License, or
+' (at your option) any later version.
+'
+' This program is distributed in the hope that it will be useful,
+' but WITHOUT ANY WARRANTY
+Imports SCrawler.API.Base
+Imports PersonalUtilities.Functions.RegularExpressions
+Namespace API.ThreadsNet
+    Friend Module Declarations
+        Friend ReadOnly RegexUserID As RParams = RParams.DMS("""props"":\{[^\{\}]*?""user_id"":""(\d+)""", 1, EDP.ReturnValue)
+        Friend ReadOnly RegexUserName As RParams = RParams.DMS("\<meta property=""og:title"" content=""([^\>]+)""\s*/\>", 1, TitleHtmlConverter, EDP.ReturnValue)
+        Friend ReadOnly RegexUserDescr As RParams = RParams.DMS("\<meta property=""og:description"" content=""([^\>]+)""\s*/\>", 1, HtmlConverter, EDP.ReturnValue)
+    End Module
+End Namespace

SCrawler/API/ThreadsNet/UserData.vb

@@ -388,8 +388,10 @@ Namespace API.ThreadsNet
                 Dim newID$
                 Dim idStr$ = String.Empty
                 If Not r.IsEmptyString Then
+                    UserSiteNameUpdate(RegexReplace(r, RegexUserName))
+                    UserDescriptionUpdate(RegexReplace(r, RegexUserDescr))
                     ParseTokens(r, 0)
-                    newID = RegexReplace(r, RParams.DMS("""props"":\{[^\{\}]*?""user_id"":""(\d+)""", 1, EDP.ReturnValue))
+                    newID = RegexReplace(r, RegexUserID)
                     If ID.IsEmptyString OrElse ID = newID Then
                         _IdChanged = ID.IsEmptyString
                         ID = newID

SCrawler/API/TikTok/Declarations.vb

@@ -11,6 +11,7 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.TikTok
     Friend Module Declarations
         Friend ReadOnly SimpleDateConverter As New ADateTime("yyyyMMdd")
+        Friend ReadOnly SimpleDateConverterWithTime As New ADateTime("yyyyMMdd_HHmmss")
         Friend ReadOnly RegexTagsReplacer As RParams = RParams.DM("#\w+\s?", -1, RegexReturn.Replace,
                                                                   CType(Function(input$) String.Empty, Func(Of String, String)), EDP.ReturnValue)
         Friend ReadOnly RegexPhotoJson As RParams = RParams.DMS("UNIVERSAL_DATA_FOR_REHYDRATION__"" type=""application/json""\>([^\<]+)\<", 1,

SCrawler/API/TikTok/SiteSettings.vb

@@ -10,11 +10,13 @@ Imports SCrawler.API.Base
 Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports DN = SCrawler.API.Base.DeclaredNames
 Namespace API.TikTok
     <Manifest("AndyProgram_TikTok"), SpecialForm(False), SeparatedTasks(1)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
 #Region "Categories"
         Private Const CAT_DOWN As String = "Download"
+        Private Const CAT_UserDefs_Title As String = DN.CAT_UserDefs & " (Title)"
 #End Region
 #Region "Download"
         <PropertyOption(ControlText:="Download videos", Category:=CAT_DOWN), PXML, PClonable>
@@ -22,21 +24,34 @@ Namespace API.TikTok
         <PropertyOption(ControlText:="Download photos", Category:=CAT_DOWN), PXML, PClonable>
         Friend ReadOnly Property DownloadTTPhotos As PropertyValue
 #End Region
-        <PropertyOption(ControlText:="Remove tags from title"), PXML, PClonable>
+#Region "User defaults"
+#Region "Sections"
+        <PropertyOption(ControlText:="Get Timeline", Category:=DN.CAT_UserDefs), PXML, PClonable>
+        Friend ReadOnly Property GetTimeline As PropertyValue
+        <PropertyOption(ControlText:="Get User Stories", Category:=DN.CAT_UserDefs), PXML, PClonable>
+        Friend ReadOnly Property GetStoriesUser As PropertyValue
+        <PropertyOption(ControlText:="Get Reposts", Category:=DN.CAT_UserDefs), PXML, PClonable>
+        Friend ReadOnly Property GetReposts As PropertyValue
+#End Region
+#Region "Title"
+        <PropertyOption(ControlText:="Remove tags from title", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property RemoveTagsFromTitle As PropertyValue
-        <PropertyOption(ControlText:="Use native title", ControlToolTip:="Use a user-created video title for the filename instead of the video ID."), PXML, PClonable>
+        <PropertyOption(ControlText:="Use native title", ControlToolTip:="Use a user-created video title for the filename instead of the video ID.",
+                        Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleUseNative As PropertyValue
         <PropertyOption(ControlText:="Use native title (standalone downloader)",
-                        ControlToolTip:="Use a user-created video title for the filename instead of the video ID."), PXML, PClonable>
+                        ControlToolTip:="Use a user-created video title for the filename instead of the video ID.", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleUseNativeSTD As PropertyValue
-        <PropertyOption(ControlText:="Add video ID to video title"), PXML, PClonable>
+        <PropertyOption(ControlText:="Add video ID to video title", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleAddVideoID As PropertyValue
-        <PropertyOption(ControlText:="Add video ID to video title (standalone downloader)"), PXML, PClonable>
+        <PropertyOption(ControlText:="Add video ID to video title (standalone downloader)", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleAddVideoIDSTD As PropertyValue
-        <PropertyOption(ControlText:="Use regex to clean video title"), PXML, PClonable>
+        <PropertyOption(ControlText:="Use regex to clean video title", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleUseRegexForTitle As PropertyValue
-        <PropertyOption(ControlText:="Title regex", ControlToolTip:="Regex to clean video title"), PXML, PClonable>
+        <PropertyOption(ControlText:="Title regex", ControlToolTip:="Regex to clean video title", Category:=CAT_UserDefs_Title), PXML, PClonable>
         Friend ReadOnly Property TitleUseRegexForTitle_Value As PropertyValue
+#End Region
+#End Region
         <PropertyOption(ControlText:="Use video date as file date",
                         ControlToolTip:="Set the file date to the date the video was added (website) (if available)."), PXML, PClonable>
         Friend ReadOnly Property UseParsedVideoDate As PropertyValue
@@ -46,6 +61,10 @@ Namespace API.TikTok
         Friend Sub New(ByVal AccName As String, ByVal Temp As Boolean)
             MyBase.New("TikTok", "www.tiktok.com", AccName, Temp, My.Resources.SiteResources.TikTokIcon_32, My.Resources.SiteResources.TikTokPic_192)
 
+            GetTimeline = New PropertyValue(True)
+            GetStoriesUser = New PropertyValue(False)
+            GetReposts = New PropertyValue(False)
+
             DownloadTTVideos = New PropertyValue(True)
             DownloadTTPhotos = New PropertyValue(True)
 
@@ -76,5 +95,10 @@ Namespace API.TikTok
                 Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using
             End If
         End Sub
+        Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String
+            Dim url$ = MyBase.GetUserPostUrl(User, Media)
+            If Not url.IsEmptyString AndAlso url.EndsWith(UserData.GDL_POSTFIX) Then url = url.Replace(UserData.GDL_POSTFIX, String.Empty)
+            Return url
+        End Function
     End Class
 End Namespace

SCrawler/API/TikTok/UserData.vb

@@ -7,16 +7,21 @@
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
 Imports System.Threading
-Imports SCrawler.API.Base
-Imports SCrawler.API.YouTube.Objects
-Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Functions.RegularExpressions
+Imports PersonalUtilities.Functions.XML
 Imports PersonalUtilities.Tools
 Imports PersonalUtilities.Tools.Web.Documents.JSON
+Imports SCrawler.API.Base
+Imports SCrawler.API.YouTube.Objects
+Imports SCrawler.Plugin.Attributes
+Imports Sections = SCrawler.API.Instagram.UserData.Sections
 Imports UTypes = SCrawler.API.Base.UserMedia.Types
 Namespace API.TikTok
     Friend Class UserData : Inherits UserDataBase
 #Region "XML names"
+        Private Const Name_GetTimeline As String = "GetTimeline"
+        Private Const Name_GetStoriesUser As String = "GetStoriesUser"
+        Private Const Name_GetReposts As String = "GetReposts"
         Private Const Name_RemoveTagsFromTitle As String = "RemoveTagsFromTitle"
         Private Const Name_TitleUseNative As String = "TitleUseNative"
         Private Const Name_TitleAddVideoID As String = "TitleAddVideoID"
@@ -27,6 +32,7 @@ Namespace API.TikTok
         Private Const Name_PhotosDownloaded As String = "PhotosDownloaded"
 #End Region
 #Region "Declarations"
+        Friend Const GDL_POSTFIX As String = "--GDL"
         Private ReadOnly Property MySettings As SiteSettings
             Get
                 Return HOST.Source
@@ -57,6 +63,9 @@ Namespace API.TikTok
                 End If
             End Get
         End Property
+        Friend Property GetTimeline As Boolean = True
+        Friend Property GetStoriesUser As Boolean = False
+        Friend Property GetReposts As Boolean = False
         Friend Property RemoveTagsFromTitle As Boolean = False
         Friend Property TitleUseNative As Boolean = True
         Friend Property TitleAddVideoID As Boolean = True
@@ -74,6 +83,9 @@ Namespace API.TikTok
             If Not Obj Is Nothing AndAlso TypeOf Obj Is UserExchangeOptions Then
                 With DirectCast(Obj, UserExchangeOptions)
                     .ApplyBase(Me)
+                    GetTimeline = .GetTimeline
+                    GetStoriesUser = .GetStoriesUser
+                    GetReposts = .GetReposts
                     RemoveTagsFromTitle = .RemoveTagsFromTitle
                     TitleUseNative = .TitleUseNative
                     TitleAddVideoID = .TitleAddVideoID
@@ -88,6 +100,9 @@ Namespace API.TikTok
         Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
             With Container
                 If Loading Then
+                    GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(True)
+                    GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(False)
+                    GetReposts = .Value(Name_GetReposts).FromXML(Of Boolean)(False)
                     RemoveTagsFromTitle = .Value(Name_RemoveTagsFromTitle).FromXML(Of Boolean)(False)
                     TitleUseNative = .Value(Name_TitleUseNative).FromXML(Of Boolean)(True)
                     TitleAddVideoID = .Value(Name_TitleAddVideoID).FromXML(Of Boolean)(True)
@@ -98,6 +113,9 @@ Namespace API.TikTok
                     TitleUseGlobalRegexOptions = .Value(Name_TitleUseGlobalRegexOptions).FromXML(Of Boolean)(True)
                     PhotosDownloaded = .Value(Name_PhotosDownloaded).FromXML(Of Boolean)(False)
                 Else
+                    .Add(Name_GetTimeline, GetTimeline.BoolToInteger)
+                    .Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
+                    .Add(Name_GetReposts, GetReposts.BoolToInteger)
                     .Add(Name_RemoveTagsFromTitle, RemoveTagsFromTitle.BoolToInteger)
                     .Add(Name_TitleUseNative, TitleUseNative.BoolToInteger)
                     .Add(Name_TitleAddVideoID, TitleAddVideoID.BoolToInteger)
@@ -166,17 +184,25 @@ Namespace API.TikTok
         Private Function GetPhotoNode() As Object()
             Return {"imageURL", "urlList", 0, 0}
         End Function
+        Private Sub ValidateCache()
+            If If(UserCache?.Disposed, True) Then UserCache = CreateCache()
+        End Sub
         Friend Overrides Sub DownloadData(ByVal Token As CancellationToken)
             MyBase.DownloadData(Token)
             UserCache.DisposeIfReady(False)
             UserCache = Nothing
         End Sub
-        Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+        Protected Overloads Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
+            ValidateCache()
+            If GetTimeline Then DownloadDataF(Sections.Timeline, Token)
+            If GetStoriesUser Then DownloadDataF(Sections.UserStories, Token)
+            If GetReposts Then DownloadDataF(Sections.Reposts, Token)
+        End Sub
+        Protected Overloads Sub DownloadDataF(ByVal Section As Sections, ByVal Token As CancellationToken)
             Dim URL$ = $"https://www.tiktok.com/@{NameTrue}"
-            UserCache = CreateCache()
             Try
                 Const photoPrefix$ = "photo_"
-                Dim postID$, title$, postUrl$, newName$, t$, postID2$, imgUrl$, pText$
+                Dim postID$, title$, postUrl$, newName$, t$, tOrig$, postID2$, imgUrl$, pText$
                 Dim postDate As Date?
                 Dim dateAfterC As Date? = Nothing
                 Dim dateBefore As Date? = DownloadDateTo
@@ -185,12 +211,24 @@ Namespace API.TikTok
                 Dim titleRegex As RParams = GetTitleRegex()
                 Dim vPath As SFile = Nothing, pPath As SFile = Nothing
                 Dim file As SFile
-                Dim j As EContainer, photo As EContainer
+                Dim j As EContainer = Nothing, photo As EContainer, item As EContainer
                 Dim photoNode As Object() = GetPhotoNode()
                 Dim c%, cc%, i%
                 Dim errDef As New ErrorsDescriber(EDP.ReturnValue)
                 Dim infoParsed As Boolean = False
 
+                Dim gdlTmpIDs As New Dictionary(Of String, Integer)
+                Dim gdlCmd$ = String.Empty
+                Dim gdlIsNativeJson As Boolean
+
+                Dim __specFolder$ = String.Empty
+                Dim __specFolder_Cr As Func(Of String, String) = Function(_sp$) String.Empty.StringAppend(__specFolder).StringAppend(_sp, "\") &
+                                                                                IIf(__specFolder.IsEmptyString, String.Empty, "*")
+                Select Case Section
+                    Case Sections.UserStories : URL &= "/stories" : __specFolder = "Stories (user)" : gdlCmd = "-o videos -o photos"
+                    Case Sections.Reposts : URL &= "/reposts" : __specFolder = "Reposts"
+                End Select
+
                 If _ContentList.Count > 0 Then
                     With (From d In _ContentList Where d.Post.Date.HasValue Select d.Post.Date.Value)
                         If .ListExists Then dateAfterC = .Min
@@ -215,7 +253,7 @@ Namespace API.TikTok
                     End If
                 End If
 
-                If DownloadVideos And Settings.YtdlpFile.Exists And CBool(MySettings.DownloadTTVideos.Value) Then
+                If Section = Sections.Timeline And DownloadVideos And Settings.YtdlpFile.Exists And CBool(MySettings.DownloadTTVideos.Value) Then
                     With UserCache.NewInstance : .Validate() : vPath = .RootDirectory : End With
                     Using b As New YTDLP.YTDLPBatch(Token,, vPath) With {.TempPostsList = _TempPostsList}
                         b.Execute(CreateYTCommand(vPath, URL, False, dateBefore, dateAfter))
@@ -233,7 +271,7 @@ Namespace API.TikTok
                             Else
                                 .TempPostsList = New List(Of String)
                             End If
-                            .Execute(CreateGDLCommand(URL))
+                            .Execute(CreateGDLCommand(URL, gdlCmd))
                             If Not PhotosDownloaded Then _ForceSaveUserInfo = True : _ForceSaveUserInfoOnException = True
                             PhotosDownloaded = True
                         End With
@@ -243,6 +281,7 @@ Namespace API.TikTok
                 ThrowAny(Token)
 
                 Dim files As List(Of SFile)
+                'YTDLP
                 If Not vPath.IsEmptyString AndAlso vPath.Exists(SFO.Path, False) Then
                     files = SFile.GetFiles(vPath, "*.json",, errDef)
                     If files.ListExists Then
@@ -250,7 +289,7 @@ Namespace API.TikTok
                             j = JsonDocument.Parse(file.GetText, errDef)
                             If j.ListExists Then
                                 If j.Value("_type").StringToLower = "video" Then
-                                    If Not baseDataObtained Then
+                                    If Not baseDataObtained And Section = Sections.Timeline Then
                                         baseDataObtained = True
                                         If ID.IsEmptyString Then ID = j.Value("uploader_id")
                                         newName = j.Value("uploader")
@@ -262,7 +301,8 @@ Namespace API.TikTok
                                     If Not _TempPostsList.Contains(postID) Then
                                         _TempPostsList.ListAddValue(postID, LNC)
                                     Else
-                                        Exit For 'Exit Sub
+                                        'Exit For 'Exit Sub
+                                        Continue For
                                     End If
                                     title = GetNewFileName(j.Value("title").StringRemoveWinForbiddenSymbols,
                                                            TitleUseNative, RemoveTagsFromTitle, TitleAddVideoID, postID, titleRegex)
@@ -279,6 +319,7 @@ Namespace API.TikTok
                                     If postUrl.IsEmptyString Then postUrl = $"https://www.tiktok.com/@{Name}/video/{postID}"
                                     _TempMediaList.Add(New UserMedia(postUrl, UTypes.Video) With {
                                                        .File = $"{title}.mp4",
+                                                       .SpecialFolder = __specFolder_Cr(String.Empty),
                                                        .Post = New UserPost(postID, postDate),
                                                        .PostText = pText,
                                                        .PostTextFileSpecialFolder = DownloadTextSpecialFolder,
@@ -291,25 +332,122 @@ Namespace API.TikTok
                     End If
                 End If
 
+                j.DisposeIfReady
+
+                'GDL
                 If Not pPath.IsEmptyString AndAlso pPath.Exists(SFO.Path, False) Then
                     files = SFile.GetFiles(pPath, "*.txt",, errDef)
                     If files.ListExists Then
+
+                        If Not Section = Sections.Timeline Then
+                            GDLResetFileNameProvider(Math.Max(files.Count.ToString.Length, 2))
+                            For i = 0 To files.Count - 1 : files(i) = GDLRenameFile(files(i), i) : Next
+                        End If
+
                         For Each file In files
                             t = file.GetText(errDef)
-                            If Not t.IsEmptyString Then t = RegexReplace(t, RegexPhotoJson)
+                            tOrig = t
+                            gdlIsNativeJson = False
+                            If Not t.IsEmptyString And Not Section = Sections.UserStories Then
+                                t = RegexReplace(t, RegexPhotoJson)
+                                If t.IsEmptyString Then t = tOrig : gdlIsNativeJson = True
+                            End If
                             If Not t.IsEmptyString Then
                                 j = JsonDocument.Parse(t, errDef)
                                 If j.ListExists Then
-                                    With j.ItemF({0, "webapp.video-detail", "itemInfo", "itemStruct"})
+                                    If Section = Sections.UserStories Then
+                                        With j("itemList")
                                             If .ListExists Then
+                                                For Each item In .Self
+                                                    With item
+                                                        postID = .Value("id")
+                                                        postDate = AConvert(Of Date)(.Value("createTime"), UnixDate32Provider, Nothing)
+                                                        If Not _TempPostsList.Contains(postID) Then
+                                                            _TempPostsList.Add(postID)
+                                                            postUrl = $"https://www.tiktok.com/@{Name}/video/{postID}{GDL_POSTFIX}"
+                                                            If postDate.HasValue Then
+                                                                title = CStr(AConvert(Of String)(postDate.Value, SimpleDateConverterWithTime, String.Empty)).StringAppend(postID, " ")
+                                                            Else
+                                                                title = postID
+                                                            End If
+                                                            _TempMediaList.Add(New UserMedia(postUrl, UTypes.Video) With {
+                                                                               .URL_BASE = postUrl,
+                                                                               .SpecialFolder = __specFolder_Cr(String.Empty),
+                                                                               .File = $"{title}.mp4",
+                                                                               .Post = New UserPost(postID, postDate)
+                                                                               })
+                                                            With .Item("video")
+                                                                If .ListExists AndAlso Not .Value("cover").IsEmptyString Then _
+                                                                   _TempMediaList.Add(New UserMedia(.Value("cover"), UTypes.Picture) With {
+                                                                                      .URL_BASE = postUrl,
+                                                                                      .SpecialFolder = __specFolder_Cr("Photo"),
+                                                                                      .File = $"{title}.jpg"
+                                                                                      })
+                                                            End With
+                                                        Else
+                                                            Continue For
+                                                        End If
+                                                    End With
+                                                Next
+                                            End If
+                                        End With
+                                    ElseIf Section = Sections.Reposts And gdlIsNativeJson Then
+                                        With j("itemList")
+                                            If .ListExists Then
+                                                For Each item In .Self
+                                                    With item
                                                         postID = .Value("id")
                                                         postID2 = $"{photoPrefix}{postID}"
-                                            If Not _TempPostsList.Contains(postID2) Then _TempPostsList.ListAddValue(postID2, LNC) Else Exit For 'Exit Sub
-                                            postDate = AConvert(Of Date)(.Value("createTime"), UnixDate32Provider, Nothing)
+                                                        If Not _TempPostsList.Contains(postID) And Not _TempPostsList.Contains(postID2) Then
+                                                            title = GetNewFileName(.Value("title").StringRemoveWinForbiddenSymbols,
+                                                                                   TitleUseNative, RemoveTagsFromTitle, TitleAddVideoID, postID, titleRegex)
+                                                            pText = .Value("title")
+                                                            If Not .Value("desc").IsEmptyString Then
+                                                                pText &= vbCr & vbCr & .Value("desc")
+                                                                If title.IsEmptyString Then title = GetNewFileName(.Value("desc").StringRemoveWinForbiddenSymbols,
+                                                                                                                   TitleUseNative, RemoveTagsFromTitle, TitleAddVideoID, postID, titleRegex)
+                                                            End If
+                                                            postDate = AConvert(Of Date)(j.Value("createTime"), UnixDate32Provider, Nothing)
+                                                            If postDate.HasValue Then
                                                                 Select Case CheckDatesLimit(postDate, SimpleDateConverter)
                                                                     Case DateResult.Skip : Continue For
                                                                     Case DateResult.Exit : Exit For 'Exit Sub
                                                                 End Select
+                                                            End If
+
+                                                            postUrl = .Value({"author"}, "uniqueId")
+                                                            If Not postUrl.IsEmptyString Then
+                                                                postUrl = $"https://www.tiktok.com/@{postUrl}/video/{postID}"
+                                                                _TempMediaList.Add(New UserMedia(postUrl, UTypes.Video) With {
+                                                                                   .File = $"{title}.mp4",
+                                                                                   .SpecialFolder = __specFolder_Cr(String.Empty),
+                                                                                   .Post = New UserPost(postID, postDate),
+                                                                                   .PostText = pText,
+                                                                                   .PostTextFileSpecialFolder = DownloadTextSpecialFolder,
+                                                                                   .PostTextFile = $"{ .File.Name}.txt"
+                                                                                   })
+                                                                If Not gdlTmpIDs.ContainsKey(postID) Then gdlTmpIDs.Add(postID, _TempMediaList.Count - 1)
+                                                            End If
+                                                        Else
+                                                            Continue For
+                                                        End If
+                                                    End With
+                                                Next
+                                            End If
+                                        End With
+                                    Else
+                                        With j.ItemF({0, "webapp.video-detail", "itemInfo", "itemStruct"})
+                                            If .ListExists Then
+                                                postID = .Value("id")
+                                                postID2 = $"{photoPrefix}{postID}"
+                                                'If Not _TempPostsList.Contains(postID2) Then _TempPostsList.ListAddValue(postID2, LNC) Else Exit For 'Exit Sub
+                                                postDate = AConvert(Of Date)(.Value("createTime"), UnixDate32Provider, Nothing)
+                                                If Not Section = Sections.UserStories Then
+                                                    Select Case CheckDatesLimit(postDate, SimpleDateConverter)
+                                                        Case DateResult.Skip : Continue For
+                                                        Case DateResult.Exit : Exit For 'Exit Sub
+                                                    End Select
+                                                End If
 
                                                 If Not infoParsed Then
                                                     With .Item("author")
@@ -340,6 +478,15 @@ Namespace API.TikTok
                                                 postUrl = $"https://www.tiktok.com/@{Name}/photo/{postID}"
                                                 With .Item({"imagePost", "images"})
                                                     If .ListExists Then
+                                                        If Not _TempPostsList.Contains(postID2) Then
+                                                            _TempPostsList.ListAddValue(postID2, LNC)
+                                                            If gdlTmpIDs.ContainsKey(postID) Then
+                                                                _TempMediaList.RemoveAt(gdlTmpIDs(postID))
+                                                                gdlTmpIDs.Remove(postID)
+                                                            End If
+                                                        Else
+                                                            Continue For 'Exit Sub
+                                                        End If
                                                         i = 0
                                                         c = .Count
                                                         cc = Math.Max(c.ToString.Length, 3)
@@ -349,7 +496,7 @@ Namespace API.TikTok
                                                             If Not imgUrl.IsEmptyString Then _
                                                                _TempMediaList.Add(New UserMedia(imgUrl, UTypes.Picture) With {
                                                                                   .URL_BASE = postUrl,
-                                                                              .SpecialFolder = "Photo",
+                                                                                  .SpecialFolder = __specFolder_Cr("Photo"),
                                                                                   .File = $"{title}{IIf(c > 1, $"_{i.NumToString(ANumbers.Formats.NumberGroup, cc)}", String.Empty)}.jpg",
                                                                                   .Post = New UserPost(postID, postDate),
                                                                                   .PostText = pText,
@@ -361,6 +508,7 @@ Namespace API.TikTok
                                                 End With
                                             End If
                                         End With
+                                    End If
                                     j.Dispose()
                                 End If
                             End If
@@ -368,6 +516,9 @@ Namespace API.TikTok
                     End If
                 End If
 
+                j.DisposeIfReady
+                _TempPostsList.ListAddList(gdlTmpIDs.Keys)
+                gdlTmpIDs.Clear()
                 If _TempMediaList.Count > 0 Then LastDownloadDate = Now
             Catch ex As Exception
                 ProcessException(ex, Token, $"data downloading error [{URL}]")
@@ -452,8 +603,17 @@ Namespace API.TikTok
         End Function
 #End Region
 #Region "GDL Support"
-        Private Function CreateGDLCommand(ByVal URL As String) As String
-            Return $"""{Settings.GalleryDLFile}"" --verbose --no-download --no-skip --write-pages {URL}"
+        Private Function CreateGDLCommand(ByVal URL As String, Optional ByVal SectionCommand As String = Nothing,
+                                          Optional ByVal IsDownload As Boolean = False, Optional ByVal Output As SFile = Nothing) As String
+            Dim command$ = $"""{Settings.GalleryDLFile}"" "
+            If Not IsDownload Then
+                command &= "--verbose --no-download --no-skip --write-pages "
+            Else
+                command &= $"--dest ""{Output.PathNoSeparator}"" "
+            End If
+            If Not CBool(If(IsSingleObjectDownload, MySettings.UseParsedVideoDateSTD, MySettings.UseParsedVideoDate).Value) Then command &= "--no-mtime "
+            command &= $"{SectionCommand} {URL}"
+            Return command
         End Function
 #End Region
 #Region "DownloadContent, DownloadFile"
@@ -465,7 +625,16 @@ Namespace API.TikTok
         End Function
         Protected Overrides Function DownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile
             Using b As New TokenBatch(Token) With {.FileExchanger = RootCacheTikTok}
+                If URL.EndsWith(GDL_POSTFIX) Then
+                    ValidateCache()
+                    Dim tmpPath As SFile
+                    With UserCache.NewInstance : .Validate() : tmpPath = .RootDirectory : End With
+                    b.Execute(CreateGDLCommand(URL.Replace(GDL_POSTFIX, String.Empty),, True, tmpPath))
+                    tmpPath = SFile.GetFiles(tmpPath, "*.mp4", IO.SearchOption.AllDirectories, EDP.ReturnValue).FirstOrDefault
+                    If Not tmpPath.IsEmptyString Then SFile.Move(tmpPath, DestinationFile)
+                Else
                     b.Execute(CreateYTCommand(DestinationFile, URL, True))
+                End If
             End Using
             If DestinationFile.Exists Then Return DestinationFile Else Return Nothing
         End Function

SCrawler/API/TikTok/UserExchangeOptions.vb

@@ -6,9 +6,16 @@
 '
 ' This program is distributed in the hope that it will be useful,
 ' but WITHOUT ANY WARRANTY
+Imports SCrawler.Plugin
 Imports SCrawler.Plugin.Attributes
 Namespace API.TikTok
     Friend Class UserExchangeOptions : Inherits Base.EditorExchangeOptionsBase
+        <PSetting(NameOf(SiteSettings.GetTimeline), NameOf(MySettings))>
+        Friend Property GetTimeline As Boolean
+        <PSetting(NameOf(SiteSettings.GetStoriesUser), NameOf(MySettings))>
+        Friend Property GetStoriesUser As Boolean
+        <PSetting(NameOf(SiteSettings.GetReposts), NameOf(MySettings))>
+        Friend Property GetReposts As Boolean
         <PSetting(NameOf(SiteSettings.RemoveTagsFromTitle), NameOf(MySettings))>
         Friend Property RemoveTagsFromTitle As Boolean
         <PSetting(NameOf(SiteSettings.TitleUseNative), NameOf(MySettings))>
@@ -27,6 +34,9 @@ Namespace API.TikTok
             MyBase.New(u)
             _ApplyBase_Name = False
             MySettings = u.HOST.Source
+            GetTimeline = u.GetTimeline
+            GetStoriesUser = u.GetStoriesUser
+            GetReposts = u.GetReposts
             RemoveTagsFromTitle = u.RemoveTagsFromTitle
             TitleUseNative = u.TitleUseNative
             TitleAddVideoID = u.TitleAddVideoID
@@ -38,6 +48,9 @@ Namespace API.TikTok
             MyBase.New(s)
             _ApplyBase_Name = False
             MySettings = s
+            GetTimeline = s.GetTimeline.Value
+            GetStoriesUser = s.GetStoriesUser.Value
+            GetReposts = s.GetReposts.Value
             RemoveTagsFromTitle = s.RemoveTagsFromTitle.Value
             TitleUseNative = s.TitleUseNative.Value
             TitleAddVideoID = s.TitleAddVideoID.Value

SCrawler/API/Twitter/UserData.vb

@@ -112,14 +112,6 @@ Namespace API.Twitter
                 Return HOST.Source
             End Get
         End Property
-        Private FileNameProvider As ANumbers = Nothing
-        Private Sub ResetFileNameProvider(Optional ByVal GroupSize As Integer? = Nothing)
-            FileNameProvider = New ANumbers With {.FormatOptions = ANumbers.Options.FormatNumberGroup + ANumbers.Options.Groups}
-            FileNameProvider.GroupSize = If(GroupSize, 3)
-        End Sub
-        Private Function RenameGdlFile(ByVal Input As SFile, ByVal i As Integer) As SFile
-            Return SFile.Rename(Input, $"{Input.PathWithSeparator}{i.NumToString(FileNameProvider)}.{Input.Extension}",, EDP.ThrowException)
-        End Function
         Friend Function GetUserUrl() As String
             Return $"https://x.com{IIf(IsCommunity, SiteSettings.CommunitiesUser, String.Empty)}/{NameTrue}"
         End Function
@@ -305,10 +297,17 @@ Namespace API.Twitter
         Private Const DEBUG_PROFILE As Boolean = False
         Private Const DEBUG_LEAVE_CACHE As Boolean = False
         Private JsonNullErr As Boolean = False
+        Private ____UserExists As Boolean = True
+        Private NotUserExistsAttempts As Integer = 0
+        Friend Overrides Sub DownloadData(ByVal Token As CancellationToken)
+            ____UserExists = UserExists
+            MyBase.DownloadData(Token)
+        End Sub
         Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
             Try
                 GDL_REQUESTS_COUNT = 0
                 JsonNullErr = False
+                NotUserExistsAttempts = 0
                 If MySettings.LIMIT_ABORT Then
                     Throw New TwitterLimitException(Me)
                 Else
@@ -325,6 +324,14 @@ Namespace API.Twitter
                         End If
                         LikesPosts.Clear()
                         If _ContentList.Count > 0 Then _DataNames.ListAddList(_ContentList.Select(Function(c) c.File.File), LAP.ClearBeforeAdd, LAP.NotContainsOnly)
+                        If Not ____UserExists Then
+                            For i% = 0 To 1
+                                NotUserExistsAttempts += 1
+                                DownloadData_Timeline(Token)
+                                If UserExists Then ____UserExists = True : Exit For
+                            Next
+                        End If
+                        If Not UserExists Then Exit Sub
                         DownloadData_Timeline(Token)
                         If _TempMediaList.Count = 0 And LikesPosts.Count = 0 And JsonNullErr Then Throw New Plugin.ExitException("No deserialized data found")
                         LoadSavePostsKV(False)
@@ -374,6 +381,7 @@ Namespace API.Twitter
                 Dim j As EContainer, rootNode As EContainer, optionalNode As EContainer, workingNode As EContainer, tmpNode As EContainer, nn As EContainer = Nothing
                 Dim multiMode As Boolean = IsMultiMode
                 Dim currentModel As DownloadModels = DownloadModels.Undefined
+                Dim onlyUpdateUser As Boolean = Not ____UserExists
 
                 Dim __parseContainer As Func(Of EContainer, Boolean) =
                     Function(ByVal ee As EContainer) As Boolean
@@ -479,13 +487,14 @@ Namespace API.Twitter
                             ThrowAny(Token)
                             Dim timelineFiles As List(Of SFile) = SFile.GetFiles(dir, "*.txt",, EDP.ReturnValue)
                             If timelineFiles.ListExists Then
-                                ResetFileNameProvider(Math.Max(timelineFiles.Count.ToString.Length, 2))
+                                GDLResetFileNameProvider(Math.Max(timelineFiles.Count.ToString.Length, 2))
                                 'rename files
                                 If Not DEBUG_PROFILE Then
-                                    For i = 0 To timelineFiles.Count - 1 : timelineFiles(i) = RenameGdlFile(timelineFiles(i), i) : Next
+                                    For i = 0 To timelineFiles.Count - 1 : timelineFiles(i) = GDLRenameFile(timelineFiles(i), i) : Next
                                 End If
                                 'parse files
                                 For i = 0 To timelineFiles.Count - 1
+                                    If userInfoParsed And onlyUpdateUser Then Exit Sub
                                     j = JsonDocument.Parse(timelineFiles(i).GetText, jsonArgs)
                                     If jsonArgs.State = WebDocumentEventArgs.States.Error Then
                                         jsonArgs.Reset(Token)
@@ -538,6 +547,14 @@ Namespace API.Twitter
                                                             Dim tScreenName$ = .Value({"core"}, "screen_name")
                                                             With .Item({"legacy"})
                                                                 If .ListExists Then
+                                                                    If onlyUpdateUser Then
+                                                                        If Not NameTrue = tScreenName Or 1 = 1 Then
+                                                                            Dim uStr$ = $"username changed from '{NameTrue}' to '{tScreenName}'"
+                                                                            LogError(Nothing, uStr)
+                                                                            UserDescriptionUpdate(uStr, True, True, True)
+                                                                        End If
+                                                                        NameTrue = tScreenName
+                                                                    End If
                                                                     If .Value("screen_name").IfNullOrEmpty(tScreenName).StringToLower = NameTrue.ToLower Then
                                                                         UserSiteNameUpdate(.Value("name"))
                                                                         UserDescriptionUpdate(.Value("description"))
@@ -681,14 +698,14 @@ nextpIndx:
                 Dim f As SFile = GetDataFromGalleryDL("https://x.com/i/bookmarks", Settings.Cache, True, Token)
                 Dim files As List(Of SFile) = SFile.GetFiles(f, "*.txt")
                 If files.ListExists Then
-                    ResetFileNameProvider(Math.Max(files.Count.ToString.Length, 3))
+                    GDLResetFileNameProvider(Math.Max(files.Count.ToString.Length, 3))
                     Dim id$
                     Dim nodes As List(Of String()) = GetContainerSubnodes()
                     Dim node$()
                     Dim j As EContainer, jj As EContainer
                     Dim jErr As New ErrorsDescriber(EDP.ReturnValue)
                     For i% = 0 To files.Count - 1
-                        f = RenameGdlFile(files(i), i)
+                        f = GDLRenameFile(files(i), i)
                         j = JsonDocument.Parse(f.GetText, jErr)
                         If Not j Is Nothing Then
                             With j.ItemF({"data", 0, "timeline", "instructions", 0, "entries"})
@@ -854,6 +871,23 @@ nextpIndx:
         Private Class TwitterGDL : Inherits GDL.GDLBatch
             Private ReadOnly KillOnLimit As Boolean
             Friend LimitReached As Boolean = False
+            Private _GetOnlyUserInfo As Boolean = False
+            Friend Overrides Property MyWorkingDirectory As SFile
+                Get
+                    Return If(MyBase.MyWorkingDirectory.IsEmptyString, If(FileExchanger?.RootDirectory, MyBase.MyWorkingDirectory), MyBase.MyWorkingDirectory)
+                End Get
+                Set(ByVal dir As SFile)
+                    MyBase.MyWorkingDirectory = dir
+                End Set
+            End Property
+            Friend Property GetOnlyUserInfo As Boolean
+                Get
+                    Return _GetOnlyUserInfo And Not MyWorkingDirectory.IsEmptyString
+                End Get
+                Set(ByVal __GetOnlyUserInfo As Boolean)
+                    _GetOnlyUserInfo = __GetOnlyUserInfo
+                End Set
+            End Property
             Friend Sub New(ByVal Dir As SFile, ByVal _Token As CancellationToken, ByVal _KillOnLimit As Boolean)
                 MyBase.New(_Token,, Dir)
                 KillOnLimit = _KillOnLimit
@@ -863,11 +897,15 @@ nextpIndx:
             End Function
             Private Function IdExists(ByVal Value As String) As Boolean
                 Try
+                    If GetOnlyUserInfo Then
+                        Return CheckForData()
+                    Else
                         Value = Value.StringTrim
                         If Not Value.IsEmptyString AndAlso (Value.StartsWith("*") Or Value.StartsWith(".\gallery-dl\")) Then
                             Dim id$ = Value.Split("\").Last.Split(".").First.Split("_").First
                             If Not id.IsEmptyString Then Return TempPostsList.Contains(id)
                         End If
+                    End If
                 Catch ex As Exception
                 End Try
                 Return False
@@ -875,8 +913,14 @@ nextpIndx:
             Protected Overrides Async Sub ErrorDataReceiver(ByVal Sender As Object, ByVal e As DataReceivedEventArgs)
                 Await Task.Run(Sub() CheckForLimit(e.Data))
             End Sub
+            Private Function CheckForData()
+                If GetOnlyUserInfo Then
+                    If SFile.GetFiles(MyWorkingDirectory, "*.txt",, EDP.ReturnValue).Count > 2 Then Return True
+                End If
+                Return False
+            End Function
             Private Sub CheckForLimit(ByVal Value As String)
-                If Token.IsCancellationRequested Or (KillOnLimit AndAlso Not ProcessKilled AndAlso
+                If CheckForData() Or Token.IsCancellationRequested Or (KillOnLimit AndAlso Not ProcessKilled AndAlso
                    Not Value.IsEmptyString AndAlso (Value.ToLower.Contains("for rate limit reset") OrElse
                                                     Not CStr(RegexReplace(Value, GdlLimitRegEx)).IsEmptyString)) Then
                     LimitReached = True
@@ -1025,11 +1069,12 @@ nextpIndx:
                     .AutoClear = True,
                     .AutoReset = True,
                     .CommandPermanent = $"chcp {BatchExecutor.UnicodeEncoding}",
-                    .FileExchanger = confCache
+                    .FileExchanger = confCache,
+                    .GetOnlyUserInfo = NotUserExistsAttempts > 0
                 }
                     tgdl.FileExchanger.DeleteCacheOnDispose = False
                     tgdl.FileExchanger.DeleteRootOnDispose = False
-                    For i As Byte = 0 To IIf(IsCommunity, 0, 3)
+                    For i As Byte = 0 To IIf(IsCommunity Or NotUserExistsAttempts > 0, 0, 3)
                         dir = rootDir.NewPath
                         dir.Exists(SFO.Path, True, EDP.ThrowException)
                         outList.Add(dir)
@@ -1040,6 +1085,20 @@ nextpIndx:
                         Else
                             command &= GdlGetIdFilterString()
                         End If
+                        If NotUserExistsAttempts > 0 Then
+                            Select Case NotUserExistsAttempts
+                                Case 1 : command &= $"{urlPrePattern}{NameTrue}/media" : currentModel = DownloadModels.Media : process = True
+                                Case 2
+                                    If ID.IsEmptyString Then
+                                        process = False
+                                    Else
+                                        command &= $"https://twitter.com/intent/user?user_id={ID}"
+                                        currentModel = DownloadModels.Media
+                                        process = True
+                                    End If
+                                Case Else : process = False
+                            End Select
+                        Else
                             Select Case i
                                 Case 0 : command &= $"{urlPrePattern}{NameTrue}/media" : currentModel = DownloadModels.Media : process = dm.Contains(currentModel) Or IsCommunity
                                 Case 1 : command &= $"{urlPrePattern}{NameTrue}" : currentModel = DownloadModels.Profile : process = dm.Contains(currentModel)
@@ -1047,6 +1106,7 @@ nextpIndx:
                                 Case 3 : command &= $"{urlPrePattern}{NameTrue}/likes" : currentModel = DownloadModels.Likes : process = dm.Contains(currentModel)
                                 Case Else : process = False
                             End Select
+                        End If
                         '#If DEBUG Then
                         'Debug.WriteLine(command)
                         '#End If
@@ -1140,7 +1200,7 @@ nextpIndx:
                     Dim files As List(Of SFile)
                     Dim lim%
                     Dim specFolder$ = IIf(_ReparseLikes, "Likes", String.Empty)
-                    ResetFileNameProvider()
+                    GDLResetFileNameProvider()
                     cache = If(IsSingleObjectDownload, Settings.Cache, CreateCache())
                     If _ReparseLikes Then lim = LikesPosts.Count Else lim = _ContentList.Count
                     ProgressPre.ChangeMax(lim)
@@ -1166,7 +1226,7 @@ nextpIndx:
                                     files = SFile.GetFiles(f, "*.txt")
                                     If files.ListExists Then
                                         For ii = 0 To files.Count - 1
-                                            f = RenameGdlFile(files(ii), ii)
+                                            f = GDLRenameFile(files(ii), ii)
                                             j = JsonDocument.Parse(f.GetText)
                                             If Not j Is Nothing Then
                                                 With j.ItemF({"data", 0, "instructions", 0, "entries"})

SCrawler/API/Xhamster/SiteSettings.vb

@@ -14,6 +14,9 @@ Imports PersonalUtilities.Functions.RegularExpressions
 Namespace API.Xhamster
     <Manifest(XhamsterSiteKey), SavedPosts, SpecialForm(True), SpecialForm(False), TaskGroup(SettingsCLS.TaskStackNamePornSite)>
     Friend Class SiteSettings : Inherits SiteSettingsBase
+#Region "Consts"
+        Friend Const GetMomentsCaption As String = "Get moments (short videos)"
+#End Region
 #Region "Declarations"
         Private Const CAT_YTDLP As String = "yt-dlp support"
         <PXML("Domains"), PClonable> Private ReadOnly Property SiteDomains As PropertyValue
@@ -45,6 +48,8 @@ Namespace API.Xhamster
         End Property
         <PropertyOption(ControlText:="Disable internal algorithm", ControlToolTip:="If checked, the internal algorithm will be forcibly disabled and replaced with yt-dlp", Category:=CAT_YTDLP), PXML, PClonable, HiddenControl>
         Friend ReadOnly Property UseYTDLPForceDisableInternal As PropertyValue
+        <PropertyOption(ControlText:=GetMomentsCaption, Category:=DeclaredNames.CAT_UserDefs), PXML, PClonable>
+        Friend ReadOnly Property GetMoments As PropertyValue
         <DoNotUse> Friend Overrides Property DownloadText As PropertyValue
         <DoNotUse> Friend Overrides Property DownloadTextPosts As PropertyValue
         <DoNotUse> Friend Overrides Property DownloadTextSpecialFolder As PropertyValue
@@ -61,10 +66,11 @@ Namespace API.Xhamster
             UseYTDLPJSON = New PropertyValue(True)
             UseYTDLPDownload = New PropertyValue(True)
             UseYTDLPForceDisableInternal = New PropertyValue(False)
+            GetMoments = New PropertyValue(True)
 
             _SubscriptionsAllowed = True
             UrlPatternUser = "https://xhamster.com/{0}/{1}"
-            UserRegex = RParams.DMS($"/({UserOption}|{ChannelOption}|{P_Creators})/([^/]+)(\Z|.*)", 0, RegexReturn.ListByMatch)
+            UserRegex = RParams.DMS($"/({UserOption}|{UserOption2}|{ChannelOption}|{P_Creators})/([^/]+)(\Z|.*)", 0, RegexReturn.ListByMatch)
             ImageVideoContains = "xhamster"
             UserOptionsType = GetType(UserExchangeOptions)
             UseNetscapeCookies = True
@@ -113,6 +119,7 @@ Namespace API.Xhamster
 #Region "IsMyUser, IsMyImageVideo"
         Friend Const ChannelOption As String = "channels"
         Friend Const UserOption As String = "users/profiles"
+        Private Const UserOption2 As String = "users"
         Friend Const P_Search As String = "search"
         Friend Const P_Tags As String = "tags"
         Friend Const P_Categories As String = "categories"

SCrawler/API/Xhamster/UserData.vb

@@ -740,10 +740,18 @@ Namespace API.Xhamster
 #Region "yt-dlp support"
         Private Function YTDLPGetInfo(ByVal URL As String, ByVal n As Integer) As SFile
             Try
-                If MyCache Is Nothing Then MyCache = CreateCache() : MyCache.Validate()
-                Dim path As SFile = MyCache.NewPath
+                Dim cc As CacheKeeper
+                If IsSingleObjectDownload Then
+                    cc = Settings.Cache
+                Else
+                    If MyCache Is Nothing Then MyCache = CreateCache()
+                    cc = MyCache
+                End If
+                cc.Validate()
+                Dim path As SFile = cc.NewPath
                 Dim c$ = If(MySettings.CookiesNetscapeFile.Exists, $" --no-cookies-from-browser --cookies ""{MySettings.CookiesNetscapeFile}""", String.Empty)
-                Dim cmd$ = $"{Settings.YtdlpFile} --write-info-json --skip-download{c} {URL} -o ""{path.PathWithSeparator}file"""
+                Dim cmd$ = $"""{Settings.YtdlpFile}"" --write-info-json --skip-download{c} {URL} -o ""{path.PathWithSeparator}file"""
+                path.Exists()
                 Using ytdlp As New YTDLP.YTDLPBatch(TokenPersonal,, path) : ytdlp.Encoding = Settings.CMDEncoding : ytdlp.Execute(cmd) : End Using
                 Return SFile.GetFiles(path, "*.json",, EDP.ReturnValue).FirstOrDefault
             Catch ex As Exception
@@ -787,7 +795,7 @@ Namespace API.Xhamster
         Private Function YTDLPDownload(ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile
             DestinationFile.Extension = "mp4"
             Dim c$ = If(MySettings.CookiesNetscapeFile.Exists, $" --no-cookies-from-browser --cookies ""{MySettings.CookiesNetscapeFile}""", String.Empty)
-            Dim cmd$ = $"{Settings.YtdlpFile} --format {DirectCast(Media.Object, XMMediaInfo).FormatID}{c} {Media.URL_BASE} -o ""{DestinationFile}"""
+            Dim cmd$ = $"""{Settings.YtdlpFile}"" --format {DirectCast(Media.Object, XMMediaInfo).FormatID}{c} {Media.URL_BASE} -o ""{DestinationFile}"""
             Using ytdlp As New YTDLP.YTDLPBatch(TokenPersonal,, DestinationFile) : ytdlp.Encoding = Settings.CMDEncoding : ytdlp.Execute(cmd) : End Using
             Return DestinationFile
         End Function

SCrawler/API/Xhamster/UserExchangeOptions.vb

@@ -10,7 +10,7 @@ Imports SCrawler.API.Base
 Imports SCrawler.Plugin.Attributes
 Namespace API.Xhamster
     Friend NotInheritable Class UserExchangeOptions : Inherits API.Base.EditorExchangeOptionsBase_P
-        <PSetting(Address:=SettingAddress.User, Caption:="Get moments")>
+        <PSetting(Address:=SettingAddress.User, Caption:=SiteSettings.GetMomentsCaption)>
         Friend Property GetMoments As Boolean = False
         Friend Sub New()
             MyBase.New
@@ -19,6 +19,10 @@ Namespace API.Xhamster
             MyBase.New(DirectCast(u, UserData))
             GetMoments = DirectCast(u, UserData).GetMoments
         End Sub
+        Friend Sub New(ByVal s As SiteSettings)
+            MyBase.New(s)
+            GetMoments = s.GetMoments.Value
+        End Sub
         Friend Overrides Sub Apply(ByRef u As IPSite)
             MyBase.Apply(u)
             DirectCast(u, UserData).GetMoments = GetMoments

SCrawler/Download/Automation/AutoDownloader.vb

@@ -410,7 +410,6 @@ Namespace DownloadObjects
             With newObj
                 .Name = String.Empty
                 .Enabled = Enabled
-                .Groups.ListAddList(Groups, LAP.ClearBeforeAdd)
                 .IsManual = IsManual
                 .Timer = Timer
                 .StartupDelay = StartupDelay
@@ -690,7 +689,6 @@ Namespace DownloadObjects
             If Not disposedValue And disposing Then
                 [Stop]()
                 UserKeys.ListClearDispose()
-                Groups.Clear()
             End If
             MyBase.Dispose(disposing)
         End Sub

SCrawler/Download/Feed/DownloadFeedForm.vb

@@ -99,15 +99,15 @@ Namespace DownloadObjects
         End Sub
         Private Sub FeedRemoveCheckedMedia(ByVal MediaList As IEnumerable(Of UserMediaD), Optional ByVal OverriddenNames As List(Of String) = Nothing,
                                            Optional ByVal RemoveChecked As Boolean = True, Optional ByVal ExcludingNames As IEnumerable(Of String) = Nothing,
-                                           Optional ByVal RemoveFromDataListOnly As Boolean = False)
+                                           Optional ByVal IsAddAndRemove As Boolean = False)
             Try
-                If FeedMode = FeedModes.Special Then
-                    If LoadedFeedNames.Count > 0 Then
+                If FeedMode = FeedModes.Saved Then Exit Sub
+
                 Dim dataRemoved As Boolean = False
-                        If OverriddenNames.ListExists And Not LoadedFeedNames.ListContains(OverriddenNames) Then Exit Sub
-                        If Not RemoveFromDataListOnly Then
+                If FeedMode = FeedModes.Special And OverriddenNames.ListExists And Not LoadedFeedNames.ListContains(OverriddenNames) Then Exit Sub
                 Dim eNames As IEnumerable(Of String) = If(ExcludingNames, New String() {})
                 With If(OverriddenNames, LoadedFeedNames)
+                    If FeedMode = FeedModes.Special And .ListExists Then
                         .ForEach(Sub(ByVal feedName As String)
                                      If Not eNames.Contains(feedName) Then
                                          Dim indx% = Settings.Feeds.IndexOf(feedName)
@@ -116,25 +116,33 @@ Namespace DownloadObjects
                                          End If
                                      End If
                                  End Sub)
-                            End With
+                    ElseIf FeedMode = FeedModes.Current And Not OverriddenNames.ListExists And IsAddAndRemove Then
+                        dataRemoved = Downloader.Files.ListDisposeRemove(MediaList) > 0
+                        'Downloader.FilesSave()
+                    Else
+                        Exit Sub
                     End If
-                        If RemoveFromDataListOnly Then
-                            RefillSpecialFeedsData()
-                        ElseIf dataRemoved Then
-                            DataList.ListDisposeRemove(MediaList)
-                            If RemoveChecked Then
+                End With
+
+                If dataRemoved Then DataList.ListDisposeRemove(MediaList)
+
+                Select Case FeedMode
+                    Case FeedModes.Special
+                        If RemoveChecked And IsAddAndRemove Then
                             If RemoveCheckedMedia(False) Then RefillAfterDelete()
                         Else
                             RefillSpecialFeedsData()
                         End If
-                        End If
-                    End If
-                ElseIf FeedMode = FeedModes.Current Then
-                    If OverriddenNames Is Nothing AndAlso Downloader.Files.ListDisposeRemove(MediaList) > 0 AndAlso RemoveCheckedMedia(False) Then
-                        DataList.ListDisposeRemove(MediaList)
+                    Case FeedModes.Current
+                        If dataRemoved Then
+                            If RemoveChecked Then
+                                RemoveCheckedMedia(False)
                                 RefillAfterDelete()
+                            Else
+                                RefillList()
                             End If
                         End If
+                End Select
             Catch ex As Exception
                 ErrorsDescriber.Execute(EDP.SendToLog, ex, "[DownloadFeedForm.FeedRemoveCheckedMedia]")
             End Try
@@ -333,7 +341,7 @@ Namespace DownloadObjects
                 Dim c As IEnumerable(Of UserMediaD) = GetCheckedMedia()
                 If c.ListExists Then
                     f.Add(c)
-                    FeedRemoveCheckedMedia(c,,, {f.Name})
+                    FeedRemoveCheckedMedia(c,,, {f.Name}, True)
                 End If
             End If
         End Sub
@@ -352,7 +360,7 @@ Namespace DownloadObjects
                 Dim m As IEnumerable(Of UserMediaD) = GetCheckedMedia()
                 If m.ListExists Then
                     f.Remove(m)
-                    FeedRemoveCheckedMedia(m, {f.Name}.ToList)
+                    FeedRemoveCheckedMedia(m, {f.Name}.ToList,,, False)
                 End If
             End If
         End Sub
@@ -1000,14 +1008,14 @@ Namespace DownloadObjects
             Dim m As IEnumerable(Of UserMediaD) = GetCheckedMedia()
             If m.ListExists Then
                 Settings.Feeds.Favorite.Add(m)
-                If sender Is BTT_FEED_ADD_FAV_REMOVE Then FeedRemoveCheckedMedia(m,,, {FeedSpecial.FavoriteName})
+                If sender Is BTT_FEED_ADD_FAV_REMOVE Then FeedRemoveCheckedMedia(m,,, {FeedSpecial.FavoriteName}, True)
             End If
         End Sub
         Private Sub BTT_FEED_REMOVE_FAV_Click(sender As Object, e As EventArgs) Handles BTT_FEED_REMOVE_FAV.Click
             Dim m As IEnumerable(Of UserMediaD) = GetCheckedMedia()
             If m.ListExists Then
                 Settings.Feeds.Favorite.Remove(m)
-                If FeedMode = FeedModes.Special Then FeedRemoveCheckedMedia(m, {FeedSpecial.FavoriteName}.ToList)
+                If FeedMode = FeedModes.Special Then FeedRemoveCheckedMedia(m, {FeedSpecial.FavoriteName}.ToList,,, False)
             End If
         End Sub
         Private Sub BTT_FEED_ADD_SPEC_Click(sender As Object, e As EventArgs) Handles BTT_FEED_ADD_SPEC.Click, BTT_FEED_ADD_SPEC_REMOVE.Click
@@ -1020,7 +1028,7 @@ Namespace DownloadObjects
                                                      f.Add(c)
                                                  End Sub)
                 End With
-                If sender Is BTT_FEED_ADD_SPEC_REMOVE Then FeedRemoveCheckedMedia(c,,, names)
+                If sender Is BTT_FEED_ADD_SPEC_REMOVE Then FeedRemoveCheckedMedia(c,,, names, True)
                 names.Clear()
             Else
                 MsgBoxE({"You haven't selected media to add to your feed(s)", "Add to feed(s)"}, vbExclamation)
@@ -1036,7 +1044,7 @@ Namespace DownloadObjects
                                                      f.Remove(c)
                                                  End Sub)
                 End With
-                If FeedMode = FeedModes.Special Then FeedRemoveCheckedMedia(c, names)
+                If FeedMode = FeedModes.Special Then FeedRemoveCheckedMedia(c, names,,, False)
             Else
                 MsgBoxE({"You haven't selected media to remove from your feed(s)", "Remove from feed(s)"}, vbExclamation)
             End If
@@ -1408,8 +1416,10 @@ Namespace DownloadObjects
                 ErrorsDescriber.Execute(EDP.LogMessageValue, ex, "Download subscription media")
             End Try
         End Sub
-        Private Sub FeedMedia_FeedAddWithRemove(ByVal Sender As FeedMedia, ByVal Feeds As IEnumerable(Of String), ByVal Media As UserMediaD, ByVal RemoveOperation As Boolean)
-            FeedRemoveCheckedMedia({Media},, False, Feeds, RemoveOperation)
+        Private Sub FeedMedia_FeedRemoveCheckedMedia(ByVal Sender As FeedMedia, ByVal Media As UserMediaD, ByVal Names As IEnumerable(Of String),
+                                                     ByVal IsOverriddenNames As Boolean, ByVal IsAddAndRemove As Boolean)
+            FeedRemoveCheckedMedia({Media}, CObj(If(IsOverriddenNames, Names.ListIfNothing, Nothing)), False,
+                                   CObj(If(IsOverriddenNames, Nothing, Names)), IsAddAndRemove)
         End Sub
 #End Region
 #Region "Delete / Remove"
@@ -1616,7 +1626,7 @@ Namespace DownloadObjects
                                                   AddHandler .MediaDownload, AddressOf FeedMedia_Download
                                                   AddHandler .MediaMove, AddressOf FeedMedia_MediaMove
                                                   AddHandler .MediaCopy, AddressOf FeedMedia_MediaCopy
-                                                  AddHandler .FeedAddWithRemove, AddressOf FeedMedia_FeedAddWithRemove
+                                                  AddHandler .FeedRemoveCheckedMedia, AddressOf FeedMedia_FeedRemoveCheckedMedia
                                               End With
                                               If de.Data.Type = UTypes.Text OrElse de.Data.PostTextFile.IsEmptyString Then Exit For
                                           Next

SCrawler/Download/Feed/FeedMedia.vb

@@ -20,7 +20,8 @@ Namespace DownloadObjects
         Friend Event MediaDeleted(ByVal Sender As Object)
         Friend Event MediaDeletedText(ByVal Sender As Object)
         Friend Event MediaDownload As EventHandler
-        Friend Event FeedAddWithRemove(ByVal Sender As FeedMedia, ByVal Feeds As IEnumerable(Of String), ByVal Media As UserMediaD, ByVal RemoveOperation As Boolean)
+        Friend Event FeedRemoveCheckedMedia(ByVal Sender As FeedMedia, ByVal Media As UserMediaD, ByVal Names As IEnumerable(Of String),
+                                            ByVal IsOverriddenNames As Boolean, ByVal IsAddAndRemove As Boolean)
         Friend Event MediaMove As MediaMoveCopyEventHandler
         Friend Event MediaCopy As MediaMoveCopyEventHandler
 #End Region
@@ -435,11 +436,11 @@ Namespace DownloadObjects
         End Function
         Private Sub Feed_SPEC_ADD_REMOVE(ByVal Source As ToolStripMenuItem, ByVal e As EventArgs)
             Dim f As FeedSpecial = Feed_SPEC_ADD_Impl(Source)
-            If Not f Is Nothing Then RaiseEvent FeedAddWithRemove(Me, {f.Name}, Media, False)
+            If Not f Is Nothing Then RaiseEvent FeedRemoveCheckedMedia(Me, Media, {f.Name}, False, True)
         End Sub
         Private Sub Feed_SPEC_REMOVE(ByVal Source As ToolStripMenuItem, ByVal e As EventArgs)
             Dim f As FeedSpecial = Source.Tag
-            If Not f Is Nothing AndAlso Not f.Disposed Then f.Remove(Media) : RaiseEvent FeedAddWithRemove(Me, {f.Name}, Media, True)
+            If Not f Is Nothing AndAlso Not f.Disposed Then f.Remove(Media) : RaiseEvent FeedRemoveCheckedMedia(Me, Media, {f.Name}, True, False)
         End Sub
 #End Region
 #Region "Dispose"
@@ -598,14 +599,14 @@ Namespace DownloadObjects
             With Settings.Feeds.Favorite
                 If Not .Contains(Media) Then .Add(Media)
                 BTT_FEED_ADD_FAV.ControlChangeColor(True, False)
-                If sender Is BTT_FEED_ADD_FAV_REMOVE Then RaiseEvent FeedAddWithRemove(Me, {FeedSpecial.FavoriteName}, Media, False)
+                If sender Is BTT_FEED_ADD_FAV_REMOVE Then RaiseEvent FeedRemoveCheckedMedia(Me, Media, {FeedSpecial.FavoriteName}, False, True)
             End With
         End Sub
         Private Sub BTT_FEED_ADD_SPEC_Click(sender As Object, e As EventArgs) Handles BTT_FEED_ADD_SPEC.Click, BTT_FEED_ADD_SPEC_REMOVE.Click
             With FeedSpecialCollection.ChooseFeeds(True)
                 If .ListExists Then
                     .ForEach(Sub(f) f.Add(Media))
-                    If sender Is BTT_FEED_ADD_SPEC_REMOVE Then RaiseEvent FeedAddWithRemove(Me, .Select(Function(f) f.Name), Media, False)
+                    If sender Is BTT_FEED_ADD_SPEC_REMOVE Then RaiseEvent FeedRemoveCheckedMedia(Me, Media, .Select(Function(f) f.Name), False, True)
                 End If
             End With
         End Sub
@@ -613,14 +614,14 @@ Namespace DownloadObjects
             With Settings.Feeds.Favorite
                 If .Contains(Media) Then .Remove(Media)
                 BTT_FEED_ADD_FAV.ControlChangeColor(True)
-                RaiseEvent FeedAddWithRemove(Me, {FeedSpecial.FavoriteName}, Media, True)
+                RaiseEvent FeedRemoveCheckedMedia(Me, Media, {FeedSpecial.FavoriteName}, True, False)
             End With
         End Sub
         Private Sub BTT_FEED_REMOVE_SPEC_Click(sender As Object, e As EventArgs) Handles BTT_FEED_REMOVE_SPEC.Click
             With FeedSpecialCollection.ChooseFeeds(False)
                 If .ListExists Then
                     .ForEach(Sub(f) f.Remove(Media))
-                    RaiseEvent FeedAddWithRemove(Me, .Select(Function(f) f.Name), Media, True)
+                    RaiseEvent FeedRemoveCheckedMedia(Me, Media, .Select(Function(f) f.Name), True, False)
                 End If
             End With
         End Sub

SCrawler/Download/Groups/DownloadGroup.vb

@@ -365,18 +365,30 @@ Namespace DownloadObjects.Groups
                             (.Sites.Count = 0 OrElse .Sites.Contains(user.Site)) AndAlso
                             (.SitesExcluded.Count = 0 OrElse Not .SitesExcluded.Contains(user.Site))
                         Dim users As New List(Of IUserData)
+                        Dim l As New ListAddParams(LAP.IgnoreICopier)
                         If Not .GroupsOnly Or (.GroupsOnly And .Groups.Count = 0) Then
                             users.ListAddList(Settings.GetUsers(Function(user) CheckLabels.Invoke(user) AndAlso CheckSites.Invoke(user) AndAlso
                                                                                CheckParams.Invoke(user) AndAlso CheckSubscription.Invoke(user) AndAlso
-                                                                               CheckDays.Invoke(user) AndAlso CheckDateRange.Invoke(user)), LAP.IgnoreICopier)
+                                                                               CheckDays.Invoke(user) AndAlso CheckDateRange.Invoke(user)), l)
                         End If
-                        If .Groups.Count > 0 And Settings.Groups.Count > 0 Then
+                        If Settings.Groups.Count > 0 Then
                             Dim i%
-                            For Each groupName$ In .Groups
+                            Dim groupName$
+                            l.NotContainsOnly = True
+                            If .Groups.Count > 0 Then
+                                For Each groupName In .Groups
                                     i = Settings.Groups.IndexOf(groupName)
-                                If i >= 0 Then users.ListAddList(Settings.Groups(i).GetUsers, LAP.NotContainsOnly, LAP.IgnoreICopier)
+                                    If i >= 0 Then users.ListAddList(Settings.Groups(i).GetUsers, l)
                                 Next
                             End If
+                            l.DisableDispose = True
+                            If .GroupsExcluded.Count > 0 Then
+                                For Each groupName In .GroupsExcluded
+                                    i = Settings.Groups.IndexOf(groupName)
+                                    If i >= 0 Then users.ListDisposeRemove(Settings.Groups(i).GetUsers, l)
+                                Next
+                            End If
+                        End If
 
                         If .UsersCount <> 0 And users.ListExists Then
                             users = users.ListTake(If(.UsersCount > 0, -1, -2), Math.Abs(.UsersCount))

SCrawler/Download/Groups/GroupDefaults.vb

@@ -58,6 +58,7 @@ Namespace DownloadObjects.Groups
         Private ReadOnly Sites As List(Of String)
         Private ReadOnly SitesExcluded As List(Of String)
         Private ReadOnly Groups As List(Of String)
+        Private ReadOnly GroupsExcluded As List(Of String)
         Private ReadOnly TT_MAIN As ToolTip
         Friend ReadOnly Property GroupsOnly As Boolean
             Get
@@ -72,6 +73,7 @@ Namespace DownloadObjects.Groups
             Sites = New List(Of String)
             SitesExcluded = New List(Of String)
             Groups = New List(Of String)
+            GroupsExcluded = New List(Of String)
             TT_MAIN = New ToolTip
 
             InitTextBox(TXT_LABELS, "Labels", {New ActionButton(ADB.Edit) With {.ToolTipText = "Edit selected labels"},
@@ -82,7 +84,8 @@ Namespace DownloadObjects.Groups
                                              New ActionButton(ADB.Delete) With {.ToolTipText = "Edit excluded sites"}, ADB.Clear})
             TXT_SITES.TextBoxReadOnly = True
 
-            InitTextBox(TXT_GROUPS, "Groups", {New ActionButton(ADB.Edit) With {.ToolTipText = "Edit selected groups"}, ADB.Clear}, CaptionModes.CheckBox)
+            InitTextBox(TXT_GROUPS, "Groups", {New ActionButton(ADB.Edit) With {.ToolTipText = "Edit selected groups"},
+                                               New ActionButton(ADB.Delete) With {.ToolTipText = "Edit excluded groups"}, ADB.Clear}, CaptionModes.CheckBox)
             With TXT_GROUPS
                 .TextBoxReadOnly = True
                 .CaptionCheckAlign = ContentAlignment.MiddleLeft
@@ -301,6 +304,7 @@ Namespace DownloadObjects.Groups
             Sites.Clear()
             SitesExcluded.Clear()
             Groups.Clear()
+            GroupsExcluded.Clear()
             CH_REGULAR.Dispose()
             CH_TEMPORARY.Dispose()
             CH_FAV.Dispose()
@@ -363,7 +367,7 @@ Namespace DownloadObjects.Groups
                             End If
                         End Using
                     End With
-                Case ADB.Clear : Labels.Clear() : LabelsExcluded.Clear() : TXT_LABELS.Clear() : UpdateLabelsText()
+                Case ADB.Clear : Labels.Clear() : LabelsExcluded.Clear() : UpdateLabelsText()
             End Select
         End Sub
         Private Sub TXT_SITES_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As ActionButtonEventArgs) Handles TXT_SITES.ActionOnButtonClick
@@ -379,36 +383,38 @@ Namespace DownloadObjects.Groups
                             End If
                         End Using
                     End With
-                Case ADB.Clear : Sites.Clear() : SitesExcluded.Clear() : TXT_SITES.Clear() : UpdateSitesText()
+                Case ADB.Clear : Sites.Clear() : SitesExcluded.Clear() : UpdateSitesText()
             End Select
         End Sub
         Private Sub TXT_GROUPS_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As ActionButtonEventArgs) Handles TXT_GROUPS.ActionOnButtonClick
             Select Case Sender.DefaultButton
-                Case ADB.Edit
-                    Using f As New LabelsForm(Groups, (From g As DownloadGroup In Settings.Groups Where Not g.IsViewFilter Select g.Name)) With {
-                        .Text = "Groups (F3 to edit)",
+                Case ADB.Edit, ADB.Delete
+                    With If(Sender.DefaultButton = ADB.Edit, Groups, GroupsExcluded)
+                        Using f As New LabelsForm(.Self, (From g As DownloadGroup In Settings.Groups Where Not g.IsViewFilter Select g.Name)) With {
+                            .Text = $"Groups {IIf(Sender.DefaultButton = ADB.Delete, "excluded ", String.Empty)}(F3 to edit)",
                             .Icon = My.Resources.GroupByIcon_16,
                             .IsGroups = True
                         }
                             f.ShowDialog()
-                        If f.DialogResult = DialogResult.OK Then Groups.ListAddList(f.LabelsList, LAP.ClearBeforeAdd) : UpdateGroupsText()
+                            If f.DialogResult = DialogResult.OK Then .ListAddList(f.LabelsList, LAP.ClearBeforeAdd) : UpdateGroupsText()
                         End Using
-                Case ADB.Clear : Groups.Clear() : TXT_GROUPS.Clear() : UpdateGroupsText()
+                    End With
+                Case ADB.Clear : Groups.Clear() : GroupsExcluded.Clear() : UpdateGroupsText()
             End Select
         End Sub
         Private Sub UpdateLabelsText()
-            TXT_LABELS.Clear()
-            If Not _JustExcludeOptions Then TXT_LABELS.Text = Labels.ListToString
-            If LabelsExcluded.Count > 0 Then TXT_LABELS.Text.StringAppend($"EXCLUDED: {LabelsExcluded.ListToString}", "; ")
+            __UpdateTextImpl(TXT_LABELS, Labels, LabelsExcluded)
         End Sub
         Private Sub UpdateSitesText()
-            TXT_SITES.Clear()
-            If Not _JustExcludeOptions Then TXT_SITES.Text = Sites.ListToString
-            If SitesExcluded.Count > 0 Then TXT_SITES.Text.StringAppend($"EXCLUDED: {SitesExcluded.ListToString}", "; ")
+            __UpdateTextImpl(TXT_SITES, Sites, SitesExcluded)
         End Sub
         Private Sub UpdateGroupsText()
-            TXT_GROUPS.Clear()
-            TXT_GROUPS.Text = Groups.ListToString
+            __UpdateTextImpl(TXT_GROUPS, Groups, GroupsExcluded)
+        End Sub
+        Private Sub __UpdateTextImpl(ByRef txt As TextBoxExtended, ByVal filter As List(Of String), ByVal excluded As List(Of String))
+            txt.Clear()
+            txt.Text = filter.ListToString
+            If excluded.Count > 0 Then txt.Text.StringAppend($"EXCLUDED: {excluded.ListToString}", "; ")
         End Sub
 #End Region
 #Region "Get/set"
@@ -455,6 +461,7 @@ Namespace DownloadObjects.Groups
                     .SitesExcluded.ListAddList(SitesExcluded)
                     .Groups.Clear()
                     .Groups.ListAddList(Groups)
+                    .GroupsExcluded.ListAddList(GroupsExcluded)
                     .GroupsOnly = GroupsOnly
                 End With
             End If
@@ -505,6 +512,7 @@ Namespace DownloadObjects.Groups
                     UpdateSitesText()
 
                     Groups.ListAddList(.Groups)
+                    GroupsExcluded.ListAddList(.GroupsExcluded)
                     TXT_GROUPS.Checked = .GroupsOnly
                     UpdateGroupsText()
                 End With
@@ -513,14 +521,12 @@ Namespace DownloadObjects.Groups
 #End Region
 #Region "Enabled"
         Private _Enabled As Boolean = True
-        Private _JustExcludeOptions As Boolean = False
         Friend Overloads Property Enabled() As Boolean
             Get
                 Return _Enabled
             End Get
             Set(ByVal e As Boolean)
                 _Enabled = e
-                _JustExcludeOptions = False
                 TP_1.Enabled = e
                 TP_2.Enabled = e
                 TP_3.Enabled = e

SCrawler/Download/Groups/GroupParameters.vb

@@ -17,6 +17,7 @@ Namespace DownloadObjects.Groups
         ReadOnly Property Sites As List(Of String)
         ReadOnly Property SitesExcluded As List(Of String)
         ReadOnly Property Groups As List(Of String)
+        ReadOnly Property GroupsExcluded As List(Of String)
         Property GroupsOnly As Boolean
         Property Regular As Boolean
         Property Temporary As Boolean
@@ -59,6 +60,7 @@ Namespace DownloadObjects.Groups
         Protected Const Name_Sites As String = "Sites"
         Protected Const Name_Sites_Excluded As String = "SitesExcluded"
         Protected Const Name_Groups As String = "Groups"
+        Protected Const Name_GroupsExcluded As String = "GroupsExcluded"
         Protected Const Name_GroupsOnly As String = "GroupsOnly"
         Protected Const Name_DaysNumber As String = "DaysNumber"
         Protected Const Name_DaysIsDownloaded As String = "DaysIsDownloaded"
@@ -79,6 +81,7 @@ Namespace DownloadObjects.Groups
         Friend ReadOnly Property Sites As List(Of String) Implements IGroup.Sites
         Friend ReadOnly Property SitesExcluded As List(Of String) Implements IGroup.SitesExcluded
         Friend ReadOnly Property Groups As List(Of String) Implements IGroup.Groups
+        Friend ReadOnly Property GroupsExcluded As List(Of String) Implements IGroup.GroupsExcluded
         Friend Property GroupsOnly As Boolean = False Implements IGroup.GroupsOnly
         Friend Property Regular As Boolean = True Implements IGroup.Regular
         Friend Property Temporary As Boolean = True Implements IGroup.Temporary
@@ -105,6 +108,7 @@ Namespace DownloadObjects.Groups
             Sites = New List(Of String)
             SitesExcluded = New List(Of String)
             Groups = New List(Of String)
+            GroupsExcluded = New List(Of String)
         End Sub
 #End Region
 #Region "Base functions"
@@ -129,6 +133,7 @@ Namespace DownloadObjects.Groups
                 Sites.ListAddList(.Sites, LAP.ClearBeforeAdd)
                 SitesExcluded.ListAddList(.SitesExcluded, LAP.ClearBeforeAdd)
                 Groups.ListAddList(.Groups, LAP.ClearBeforeAdd)
+                GroupsExcluded.ListAddList(.GroupsExcluded, LAP.ClearBeforeAdd)
                 GroupsOnly = .GroupsOnly
                 Regular = .Regular
                 Temporary = .Temporary
@@ -163,6 +168,7 @@ Namespace DownloadObjects.Groups
             If Not e.Value(Name_Sites).IsEmptyString Then Sites.ListAddList(e.Value(Name_Sites).Split("|"), l)
             If Not e.Value(Name_Sites_Excluded).IsEmptyString Then SitesExcluded.ListAddList(e.Value(Name_Sites_Excluded).Split("|"), l)
             If Not e.Value(Name_Groups).IsEmptyString Then Groups.ListAddList(e.Value(Name_Groups).Split("|"), l)
+            If Not e.Value(Name_GroupsExcluded).IsEmptyString Then GroupsExcluded.ListAddList(e.Value(Name_GroupsExcluded).Split("|"), l)
             GroupsOnly = e.Value(Name_GroupsOnly).FromXML(Of Boolean)(False)
 
             Regular = e.Value(Name_Regular).FromXML(Of Boolean)(True)
@@ -202,6 +208,7 @@ Namespace DownloadObjects.Groups
                         New EContainer(Name_Sites, Sites.ListToString("|")),
                         New EContainer(Name_Sites_Excluded, SitesExcluded.ListToString("|")),
                         New EContainer(Name_Groups, Groups.ListToString("|")),
+                        New EContainer(Name_GroupsExcluded, GroupsExcluded.ListToString("|")),
                         New EContainer(Name_GroupsOnly, GroupsOnly.BoolToInteger),
                         New EContainer(Name_Regular, Regular.BoolToInteger),
                         New EContainer(Name_Temporary, Temporary.BoolToInteger),
@@ -233,6 +240,7 @@ Namespace DownloadObjects.Groups
                     Sites.Clear()
                     SitesExcluded.Clear()
                     Groups.Clear()
+                    GroupsExcluded.Clear()
                 End If
                 disposedValue = True
             End If

SCrawler/MainFrame.Designer.vb

@@ -368,7 +368,7 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
         Me.BTT_FEED.Name = "BTT_FEED"
         Me.BTT_FEED.Size = New System.Drawing.Size(52, 22)
         Me.BTT_FEED.Text = "Feed"
-        Me.BTT_FEED.ToolTipText = "Feed of recently downloaded data (Ctrl+F)"
+        Me.BTT_FEED.ToolTipText = "Feed of recently downloaded data (Alt+F)"
         '
         'BTT_CHANNELS
         '

SCrawler/MainFrame.resx

@@ -184,36 +184,37 @@
   <data name="MENU_VIEW.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
-        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAABkSURBVDhPY6AKyO86WFDQfeg/iIYKkQZAmkNbnvyXta76
-        DxViYGFi+Y8PQ5VBAMhmkGYgJs8FAw9GA5EKILFiWUFixfL/IBoqRBoAafYsOvpf0jiTvEAE2QzSLGmU
-        MeQCkYEBAD3tUdo+/cEPAAAAAElFTkSuQmCC
+        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAABkSURBVDhPY2CgBsjvOlhQ0H3oP4hGlyMKgDSHtjz5L2td
+        9R8mxsLE8h8fRjEAZDNIs6x1FXkuGHgwGohUAIkVywoSK5b/B9HockQBkGbPoqP/JY0zyQtEkM0gzZJG
+        GeS5YEABAD3tUdqXHMg6AAAAAElFTkSuQmCC
 </value>
   </data>
   <data name="BTT_LOG.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
-        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAFmSURBVFhH1dc/K4VhHMbxJ5EFEQbFiERKCotIrMJIiYEi
-        pbwCZcOqJC9AikUWiqRkJYtSRDbESMT3V07dna7zHHru+9T51me+Ts//E+V7LRjFFAZRiZzUhDVc4/vX
-        B47Rh6D14Aqp4XQ36ECQ2nALNezaQjG8Vo5DqMF0bxiA1+bwCTWoLMFbNTiDGsrkABXw0jDsKldDmdyj
-        HokrwCrUSBz7wXbRJs4eLkdQI9m0I3ENeIAaiGN3QjMSZ4fxv+ffnKIKibOnmhqI84V5eMleOHY41VAm
-        9k7wdgtW4wRqSHlCP7y2AjWmbMB7Y7DzqgZdz2iF9zrxCDXq2oU9uLz31+tgAcHahhp1DSFY9pGhRl29
-        CFYXxrMoQ7BmsZfFPkoRpHWow+56hX26BWkRatR1gRIEaQLvUMMpOyhCkBpxBzWcMoOgLUMNm0vUIWj2
-        ebaJF7jj5+hGTiqE/f+bxDRGUIt8LIp+AC/GHt3tQnwvAAAAAElFTkSuQmCC
+        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAGTSURBVFhH1ZfPK21RGIaf5GaCCAPF0JVISWEiElNhSIkB
+        RUr5C5QZpkryB0gxkQlFUjKlO1HKjcwQQyJatdXp7TtnH3t/Z+CpZ7be7z3tH2uvA7+cFmAUmAIGgSpd
+        UCiagDXgCviMfAOOgT5d7E0P8C+jWL0GOjTkRRtwY5SqW0CJhtNSARwaZZYvwIAOSMsc8G6UZXNJB6Sh
+        FjgzSnJ5AFTqoKQMR0+5luTyFmjQQUkoAlaNgjjDDw4PbWrC5nJkFORjuw5Lwl/gzhgeZ3gTmnVYEsJl
+        /On9D54C1TosCWFX0+FxfgDzOigp4YMTLqeW5DJ8E9xewRrgxCjJ5gPQr0PSsmIUZXNDwx6MRfdVy9RH
+        oFXDHnQC90ahuhttXO7k+xwsaNCTbaNQHdKQJ+GQoYVqr4Y86QLGYyzXkCezwF6M+0CZBr1YNy65+hwd
+        3QrColGoXgClGvRiAng1SjPdAf5o0ItG4L9RmumMhrxZNkq/vQTqNeBNOJ5tAk9Sfg506+JCURz9/5sE
+        poERoE4X/Rq+AC/GHt09Rk0KAAAAAElFTkSuQmCC
 </value>
   </data>
   <data name="BTT_BUG_REPORT.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
-        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAIDSURBVDhPpZLrS5NhGMb3j4SWh0oRQVExD4gonkDpg4hG
-        YKxG6WBogkMZKgPNCEVJFBGdGETEvgwyO9DJE5syZw3PIlPEE9pgBCLZ5XvdMB8Ew8gXbl54nuf63dd9
-        0OGSnwCahxbPRNPAPMw9Xpg6ZmF46kZZ0xSKzJPIrhpDWsVnpBhGkKx3nAX8Pv7z1zg8OoY/cITdn4fw
-        bf/C0kYAN3Ma/w3gWfZL5kzTKBxjWyK2DftwI9tyMYCZKXbNHaD91bLYJrDXsYbrWfUKwJrPE9M2M1Oc
-        VzOOpHI7Jr376Hi9ogHqFIANO0/MmmmbmSmm9a8ze+I4MrNWAdjtoJgWcx+PSzg166yZZ8xM8XvXDix9
-        c4jIqFYAjoriBV9AhEPv1mH/sonogha0afbZMMZz+yreTGyhpusHwtNNCsA5U1zS4BLxzJIfg299qO32
-        Ir7UJtZfftyATqeT+8o2D8JSjQrAJblrncYL7ZJ2+bfaFnC/1S1NjL3diRat7qrO7wLRP3HjWsojBeCo
-        mDEo5mNjuweFGvjWg2EBhCbpkW78htSHHwRyNdmgAFzPEee2iFkzayy2OLXzT4gr6UdUnlXrullsxxQ+
-        kx0g8BTA3aZlButjSTyjODq/WcQcW/B/Je4OQhLvKQDnzN1mp0nnkvAhR8VuMzNrpm1mpjgkoVwB/v8D
-        TgDQASA1MVpwzwAAAABJRU5ErkJggg==
+        YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAIFSURBVDhPpZLtS1NhGMbPPxJmmlYSgqHiKzGU1EDxg4iK
+        YKyG2WBogqMYJQOtCEVRFBGdTBCJfRnkS4VaaWNT5sqx1BUxRXxDHYxAJLvkusEeBaPAB+5z4Jzn+t3X
+        /aLhnEfjo8m+dCoa+7/C3O2Hqe0zDC+8KG+cRZHZhdzaaWTVTCLDMIY0vfM04Nfh77/G/sEhwpEDbO3t
+        I7TxE8urEVy99fT/AL5gWDLrTB/hnF4XsW0khCu5ln8DmJliT2AXrcNBsU1gj/MH4nMeKwBrPktM28xM
+        cX79DFKrHHD5d9D26hvicx4pABt2lpg10zYzU0zr7+e3xXGcrkEB2O2TNec9nJFwB3alZn5jZorfeDZh
+        6Q3g8s06BeCoKF4MRURoH1+BY2oNCbeb0TIclIYxOhzf8frTOuo7FxCbbVIAzpni0iceEc8vhzEwGkJD
+        lx83ymxifejdKjRNk/8PWnyIyTQqAJek0jqHwfEVscu31baIu8+90sTE4nY025dQ2/5FIPpnXlzKuK8A
+        HBUzHot52djqQ6HZhfR7IwK4mKpHtvEDMqvfCiQ6zaAAXM8x94aIWTNrLLG4kVUzgaTSPlzLtyJOZxbb
+        1wtfyg4Q+AfA3aZlButjSfxGcUJBk4g5tuP3haQKRKXcUQDOmbvNTpPOJeFFjordZmbWTNvMTHFUcpUC
+        nOccAdABIDXXE1nzAAAAAElFTkSuQmCC
 </value>
   </data>
   <metadata name="Toolbar_BOTTOM.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">

SCrawler/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2025.11.25.0")>
-<Assembly: AssemblyFileVersion("2025.11.25.0")>
+<Assembly: AssemblyVersion("2026.2.14.0")>
+<Assembly: AssemblyFileVersion("2026.2.14.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/SCrawler.vbproj

@@ -255,6 +255,7 @@
     <Compile Include="API\ThisVid\SiteSettings.vb" />
     <Compile Include="API\ThisVid\UserData.vb" />
     <Compile Include="API\ThisVid\UserExchangeOptions.vb" />
+    <Compile Include="API\ThreadsNet\Declarations.vb" />
     <Compile Include="API\ThreadsNet\SiteSettings.vb" />
     <Compile Include="API\ThreadsNet\UserData.vb" />
     <Compile Include="API\TikTok\Declarations.vb" />