diff --git a/Changelog.md b/Changelog.md index c1e85a5..dee7007 100644 --- a/Changelog.md +++ b/Changelog.md @@ -2,13 +2,46 @@ - [ffmpeg](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ffmpeg) - x64 version - [release](https://github.com/GyanD/codexffmpeg/releases/tag/5.1.2); [zip](https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build.zip); **version `5.1.2-full_build-www.gyan.dev`** - x86 version - [release](https://github.com/yt-dlp/FFmpeg-Builds/releases/tag/autobuild-2022-11-30-12-57); [zip](https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2022-11-30-12-57/ffmpeg-N-109274-gd7a5f068c2-win32-gpl.zip); **version `N-109457-geeb280f351-20221226`** -- [Gallery-dl](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl) - **1.32.9** -- [YT-DLP](https://github.com/AAndyProgram/SCrawler/wiki/Settings#yt-dlp) - **2026.07.04** +- [Gallery-dl](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl) - **1.32.13** +- [YT-DLP](https://github.com/AAndyProgram/SCrawler/wiki/Settings#yt-dlp) - **2026.08.19** - [Deno](https://github.com/AAndyProgram/SCrawler/wiki/Settings#deno) - latest *(`2.0.0` or higher)* - [OF-Scraper](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) - **3.12.9** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/3.12.9)) # 2026 +## 2026.9.21.0 + +*2026-09-21* + +**ATTENTION!** + +To download **OnlyFans** DRM-protected content, you need to download and run this [script](https://github.com/AAndyProgram/SCrawler/blob/main/Tools/Install-OFScraper.ps1) and follow the instructions. **Without this script, downloading DRM-protected content will be impossible!** The script was written by @cjb900. @cjb900, you are the best!🩵 Thank you so much for your invaluable help!🩵 + +**Reddit** `REST` API has been restored. To use it, you need to remove your `OAuth` credentials from SCrawler. **Don't forget to add cookies!** + +- Added + - Sites: + - BlueSky: + - MD5 comparison support + - handling errors `502` and `504` + - Instagram: **completely moved to `GraphQL`** + - Reddit: **restore `REST` API support** *(requires cookies)* + - Settings: new option `Add 'missing' users to log` *(Settings - [Downloading](https://github.com/AAndyProgram/SCrawler/wiki/Settings#downloading))* + - Minor improvements +- Removed + - Mastodon support + - Imgur support +- Updated + - gallery-dl up to version **1.32.13** + - yt-dlp up to version **2026.08.19** +- Fixed + - Sites: + - OnlyFans: **DRM-protected content is not downloading** (special thanks to @cjb900 🩵) + - PornHub: **photos and videos are not downloading** + - Reddit: some carousels are not downloading + - YouTube downloader: unable to download videos that have audio and video combined into one container + - Minor bugs + ## 2026.8.7.0 *2026-08-07* diff --git a/FAQ.md b/FAQ.md index 39632f3..51688bf 100644 --- a/FAQ.md +++ b/FAQ.md @@ -62,14 +62,14 @@ I strongly recommend you to **regularly** create backup copies of the settings f *How to use: find the site you need in the list and read the answer.* -- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!** Don't use OAuth token to download saved posts (use cookies only). +- Reddit: add cookies and/or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without credentials using cookies. Subreddits require OAuth! If nothing downloads, use OAuth!** Don't use OAuth token to download saved posts (use cookies only). - **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields** - **Instagram [TIPS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-tips)** - **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat. -- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download. +- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download. Photos are downloaded using gallery-dl. - Porn sites: **COOKIES**! - ThisVid: https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid-faq -- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) & **DRM keys** to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq) +- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & **DRM keys** to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq) - **JustForFans**: **THE VIDEO ISN'T DOWNLOADING AT THE MOMENT** ([Issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870)) ## Other questions diff --git a/README.md b/README.md index e63eae8..95d7257 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,4 @@ -# 🏳️‍🌈 Happy LGBT Pride Month 🎉 - + # 🏳️‍🌈 Social networks crawler 🏳️‍🌈 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest) @@ -41,14 +40,11 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo - Bluesky images and videos, text; - OnlyFans images and videos, text, saved (bookmarked) posts, stories; - JustForFans images and videos, saved (bookmarked) posts; - - Mastodon images and videos, saved (bookmarked) posts; - Instagram images and videos, text, tagged posts, stories, saved posts; - Threads images and videos, text, saved posts; - Facebook images and videos, stories, saved posts; - TikTok images and videos; - Pinterest boards, users, saved posts; - - Imgur images, galleries and videos; - - Gfycat videos; - PornHub images, videos, save (liked) posts, search queries, search categories; - XHamster images, videos, saved posts, search queries, search categories, search tags; - XVIDEOS videos, saved posts, search queries, search categories; @@ -85,12 +81,9 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo - **Threads** - **Facebook** - JustForFans *(partial support) ([video issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))*[^1] -- Mastodon *(out of support)* - TikTok - RedGifs - Pinterest -- Imgur *(out of support)* -- Gfycat - LPSG - **PornHub** - **XHamster** @@ -135,7 +128,6 @@ First, the program downloads the full profile. After the program downloads only - [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter) - [Bluesky](https://github.com/AAndyProgram/SCrawler/wiki/Settings#bluesky) - [OnlyFans](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans) - - [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#mastodon) - [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) - [Threads](https://github.com/AAndyProgram/SCrawler/wiki/Settings#threads) - [Facebook](https://github.com/AAndyProgram/SCrawler/wiki/Settings#facebook) diff --git a/SCrawler.YouTube/Controls/VideoOptionsForm.vb b/SCrawler.YouTube/Controls/VideoOptionsForm.vb index e29485c..c3a9fc5 100644 --- a/SCrawler.YouTube/Controls/VideoOptionsForm.vb +++ b/SCrawler.YouTube/Controls/VideoOptionsForm.vb @@ -137,6 +137,19 @@ Namespace API.YouTube.Controls End If CMB_FORMAT.Enabled = OPT_VIDEO.Checked + With .MediaObjects + Dim disableOpts As Boolean = False + If .Count = 0 Or .Count = 1 Then + disableOpts = True + ElseIf .Exists(Function(mm) mm.Type = UMTypes.Video Or mm.Type = YouTubeMediaContainerBase.TypeAudioVideo) And + .Exists(Function(mm) mm.Type = UMTypes.Audio) Then + disableOpts = False + Else + disableOpts = True + End If + If disableOpts Then OPT_AUDIO.Enabled = False : OPT_VIDEO.Enabled = False + End With + arr = AvailableVideoFormats CMB_FORMAT.Items.AddRange(arr) If InheritsFromContainer Then @@ -232,7 +245,7 @@ Namespace API.YouTube.Controls data = .Elements.Select(Function(ee) New MediaItem(ee, True) With {.Dock = DockStyle.Fill, .Checked = ee.Checked, .UseCookies = UseCookies}) Else data = (From m As MediaObject In .Self.MediaObjects - Where m.Type = __contentType + Where m.Type = __contentType Or (__contentType = UMTypes.Video And m.Type = YouTubeMediaContainerBase.TypeAudioVideo) Select New VideoOption(m, audio) With {.Dock = DockStyle.Fill, .Checked = m.Index = MyContainer.SelectedVideoIndex}) End If diff --git a/SCrawler.YouTube/Declarations.vb b/SCrawler.YouTube/Declarations.vb index ac3e20e..b4d3591 100644 --- a/SCrawler.YouTube/Declarations.vb +++ b/SCrawler.YouTube/Declarations.vb @@ -57,6 +57,7 @@ Namespace API.YouTube Friend ReadOnly TimeToStringProviderH As IFormatProvider = New TimeToStringConverter(True) Friend ReadOnly TitleHtmlConverter As Func(Of String, String) = Function(Input) Input.StringRemoveWinForbiddenSymbols().StringTrim() Friend ReadOnly ProgressProvider As IMyProgressNumberProvider = MyProgressNumberProvider.Percentage + Friend ReadOnly DownloadProgressPattern As RParams = RParams.DMS("\[download\]\s*([\d\.,]+)", 1, EDP.ReturnValue) Public ReadOnly TrueUrlRegEx As RParams = RParams.DM(Base.YouTubeFunctions.TrueUrlPattern, 0, EDP.ReturnValue) Friend ReadOnly MusicUrlApply As RParams = RParams.DMS("https://([w\.]*)youtube.com.+", 1, RegexReturn.Replace, EDP.ReturnValue, CType(Function(input$) "music.", Func(Of String, String)), String.Empty) @@ -78,6 +79,15 @@ Namespace API.YouTube Return f End If End Function + Public Function CreateYtdlpProgressHandler(ByVal Progress As MyProgress, ByVal Token As Threading.CancellationToken, ByVal Batch As BatchExecutor) As DataReceivedEventHandler + Return Sub(ByVal Sender As Object, ByVal e As DataReceivedEventArgs) + If Not e.Data.IsEmptyString Then + Dim v# = AConvert(Of Double)(RegexReplace(e.Data, DownloadProgressPattern), NumberProvider, -1) + If v >= 0 Then Progress.Value = v : Progress.Perform(0) + If Token.IsCancellationRequested Then Batch.Kill() + End If + End Sub + End Function Private Class TimeToStringConverter : Implements ICustomProvider Private ReadOnly _Provider As New ADateTime("mm\:ss") With {.TimeParseMode = ADateTime.TimeModes.TimeSpan} Private ReadOnly _ProviderWithHours As New ADateTime("h\:mm\:ss") With {.TimeParseMode = ADateTime.TimeModes.TimeSpan} diff --git a/SCrawler.YouTube/My Project/AssemblyInfo.vb b/SCrawler.YouTube/My Project/AssemblyInfo.vb index 7444ce0..d5791bc 100644 --- a/SCrawler.YouTube/My Project/AssemblyInfo.vb +++ b/SCrawler.YouTube/My Project/AssemblyInfo.vb @@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices ' by using the '*' as shown below: ' - - + + diff --git a/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb b/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb index fe54719..226de9a 100644 --- a/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb +++ b/SCrawler.YouTube/Objects/YouTubeMediaContainerBase.vb @@ -951,7 +951,6 @@ Namespace API.YouTube.Objects Return String.Format(m3u8DataRow, CInt(.Duration.TotalSeconds), fName, __f) End With End Function - Private ReadOnly DownloadProgressPattern As RParams = RParams.DMS("\[download\]\s*([\d\.,]+)", 1, EDP.ReturnValue) Public Property Progress As MyProgress Implements IYouTubeMediaContainer.Progress Private Property IDownloadableMedia_Progress As Object Implements IDownloadableMedia.Progress Get @@ -1191,13 +1190,7 @@ Namespace API.YouTube.Objects If MediaState = UMStates.Downloaded Or Not Checked Then Exit Sub RaiseEvent FileDownloadStarted(Me, Nothing) Using batch As New BatchExecutor(True) With {.Encoding = 65001} - Dim h As DataReceivedEventHandler = Sub(ByVal Sender As Object, ByVal e As DataReceivedEventArgs) - If Not e.Data.IsEmptyString Then - Dim v# = AConvert(Of Double)(RegexReplace(e.Data, DownloadProgressPattern), NumberProvider, -1) - If v >= 0 Then Progress.Value = v : Progress.Perform(0) - If Token.IsCancellationRequested Then batch.Kill() - End If - End Sub + Dim h As DataReceivedEventHandler = CreateYtdlpProgressHandler(Progress, Token, batch) With batch Dim prExists As Boolean = Not Progress Is Nothing If prExists Then @@ -1698,6 +1691,7 @@ Namespace API.YouTube.Objects #End Region #Region "Parse" Friend Const DRC As String = "drc" + Friend Const TypeAudioVideo As UMTypes = UMTypes.Audio + UMTypes.Video Public Overridable Function Parse(ByVal Container As EContainer, ByVal Path As SFile, ByVal IsMusic As Boolean, Optional ByVal Token As CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing) As Boolean Implements IYouTubeMediaContainer.Parse Try @@ -1850,7 +1844,6 @@ Namespace API.YouTube.Objects Return _Exists End Function Protected Sub ParseFormats(ByVal e As EContainer) - Const av As UMTypes = UMTypes.Audio + UMTypes.Video If If(e({"formats"})?.Count, 0) > 0 Then Dim obj As MediaObject Dim nValue# @@ -1887,7 +1880,7 @@ Namespace API.YouTube.Objects If validCodecValue(ee.Value("vcodec")) Then obj.Type = UMTypes.Video obj.Codec = sValue.Split(".").First - If validCodecValue(ee.Value("acodec")) Then obj.Type = av + If validCodecValue(ee.Value("acodec")) Then obj.Type = TypeAudioVideo ElseIf validCodecValue(ee.Value("acodec")) Then obj.Type = UMTypes.Audio obj.Codec = sValue.Split(".").First diff --git a/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb b/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb index 7e74c66..420b8b3 100644 --- a/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb +++ b/SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb @@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices ' by using the '*' as shown below: ' - - + + diff --git a/SCrawler/.editorconfig b/SCrawler/.editorconfig index 72ec1d2..c38ee62 100644 --- a/SCrawler/.editorconfig +++ b/SCrawler/.editorconfig @@ -53,10 +53,21 @@ dotnet_naming_style.pascal_case_style.capitalization = pascal_case # Use PascalCase for constant fields dotnet_naming_rule.constant_fields_should_be_pascal_case.severity = suggestion dotnet_naming_rule.constant_fields_should_be_pascal_case.symbols = constant_fields -dotnet_naming_rule.constant_fields_should_be_pascal_case.style = pascal_case_style +dotnet_naming_rule.constant_fields_should_be_pascal_case.style = pascal_case_style dotnet_naming_symbols.constant_fields.applicable_kinds = field dotnet_naming_symbols.constant_fields.applicable_accessibilities = * dotnet_naming_symbols.constant_fields.required_modifiers = const +dotnet_style_operator_placement_when_wrapping = beginning_of_line +tab_width = 4 +end_of_line = crlf +dotnet_style_prefer_simplified_boolean_expressions = true:suggestion +dotnet_style_prefer_compound_assignment = true:suggestion +dotnet_style_prefer_simplified_interpolation = true:suggestion +dotnet_style_prefer_collection_expression = when_types_loosely_match:suggestion +dotnet_style_namespace_match_folder = true:suggestion +dotnet_style_allow_multiple_blank_lines_experimental = true:silent +dotnet_style_allow_statement_immediately_after_block_experimental = true:silent +dotnet_code_quality_unused_parameters = all:suggestion ############################### # C# Coding Conventions # ############################### diff --git a/SCrawler/API/Base/UserDataBase.vb b/SCrawler/API/Base/UserDataBase.vb index 112b236..cd2900f 100644 --- a/SCrawler/API/Base/UserDataBase.vb +++ b/SCrawler/API/Base/UserDataBase.vb @@ -183,8 +183,6 @@ Namespace API.Base #Region "Additional names" Protected Const Name_SiteMode As String = "SiteMode" Protected Const Name_TrueName As String = "TrueName" - 'TODELETE Name_TrueName2 - Protected Const Name_TrueName2 As String = "NameTrue" Protected Const Name_Arguments As String = "Arguments" #End Region #End Region @@ -977,9 +975,6 @@ BlockNullPicture: Using x As New XmlFile(MyFileSettings) With {.XmlReadOnly = True} If User.Name.IsEmptyString Then User.Name = x.Value(Name_UserName) _NameTrue = x.Value(Name_TrueName) -#Disable Warning BC40008 - If _NameTrue.IsEmptyString AndAlso x.Contains(Name_TrueName2) Then _NameTrue = x.Value(Name_TrueName2) -#Enable Warning UserExists = x.Value(Name_UserExists).FromXML(Of Boolean)(True) UserSuspended = x.Value(Name_UserSuspended).FromXML(Of Boolean)(False) ID = x.Value(Name_UserID) @@ -1224,15 +1219,15 @@ BlockNullPicture: _DownloadInProgress = True _DescriptionChecked = False _DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime - _ForceSaveUserData = False - _ForceSaveUserInfo = False _EnvirUserExists = UserExists _EnvirUserSuspended = UserSuspended - _EnvirCreatedByChannel = CreatedByChannel - _EnvirChanged = False - _EnvirInvokeUserUpdated = False UserExists = True UserSuspended = False + _EnvirCreatedByChannel = CreatedByChannel + _ForceSaveUserData = False + _ForceSaveUserInfo = False + _EnvirChanged = False + _EnvirInvokeUserUpdated = False DownloadedPictures(False) = 0 DownloadedVideos(False) = 0 _PictureExists = Settings.ViewModeIsPicture AndAlso Not GetPicture(Of Image)(False) Is Nothing @@ -1240,8 +1235,8 @@ BlockNullPicture: Private Sub EnvirChanged(ByVal NewValue As Object, Optional ByVal Caller As String = Nothing) If _DownloadInProgress Then Select Case Caller - Case NameOf(UserExists) : If Not _EnvirUserExists = CBool(NewValue) Then _EnvirChanged = True : _EnvirInvokeUserUpdated = True - Case NameOf(UserSuspended) : If Not _EnvirUserSuspended = CBool(NewValue) Then _EnvirChanged = True : _EnvirInvokeUserUpdated = True + Case NameOf(UserExists) : If Not _EnvirUserExists = CBool(NewValue) Then _EnvirChanged = True : _EnvirInvokeUserUpdated = True : _ForceSaveUserInfo = True + Case NameOf(UserSuspended) : If Not _EnvirUserSuspended = CBool(NewValue) Then _EnvirChanged = True : _EnvirInvokeUserUpdated = True : _ForceSaveUserInfo = True Case NameOf(NameTrue) : _EnvirChanged = True : _ForceSaveUserInfo = True Case NameOf(ID) : _EnvirChanged = True : _ForceSaveUserInfo = True Case Else : _EnvirChanged = True @@ -1440,8 +1435,10 @@ BlockNullPicture: #Region "GDL File Names" Protected GDLFileNameProvider As ANumbers = Nothing Protected Sub GDLResetFileNameProvider(Optional ByVal GroupSize As Integer? = Nothing) - GDLFileNameProvider = New ANumbers With {.FormatOptions = ANumbers.Options.FormatNumberGroup + ANumbers.Options.Groups} - GDLFileNameProvider.GroupSize = If(GroupSize, 3) + GDLFileNameProvider = New ANumbers With { + .FormatOptions = ANumbers.Options.FormatNumberGroup + ANumbers.Options.Groups, + .GroupSize = If(GroupSize, 3) + } End Sub Protected Function GDLRenameFile(ByVal Input As SFile, ByVal i As Integer) As SFile Return SFile.Rename(Input, $"{Input.PathWithSeparator}{i.NumToString(GDLFileNameProvider)}.{Input.Extension}",, EDP.ThrowException) @@ -1732,7 +1729,7 @@ BlockNullPicture: If _ContentNew.Count > 0 Then If UseMD5Comparison Then LoadMD5() MyFile.Exists(SFO.Path) - Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog + Dim MissingErrorsAdd As Boolean = Settings.AddMissingToLog And Settings.AddMissingErrorsToLog Dim MyDir$ = DownloadContentDefault_GetRootDir() Dim vsf As Boolean = SeparateVideoFolderF Dim __isVideo As Boolean @@ -2349,7 +2346,7 @@ stxt: ErrorsDescriber.Execute(If(e.Exists, e, New ErrorsDescriber(EDP.SendToLog)), ex, $"{ToStringForLog()}: {Message}") End Sub Protected Sub ErrorDownloading(ByVal f As SFile, ByVal URL As String) - If Not f.Exists Then MyMainLOG = $"Error downloading from [{URL}] to [{f}]" + If Not f.Exists Then LogError(Nothing, $"error downloading{vbCr}{URL}{vbCr}{f}") End Sub ''' Protected Sub ThrowIfDisposed() diff --git a/SCrawler/API/Base/YTDLP.vb b/SCrawler/API/Base/YTDLP.vb index 36bae23..8af5f08 100644 --- a/SCrawler/API/Base/YTDLP.vb +++ b/SCrawler/API/Base/YTDLP.vb @@ -6,10 +6,20 @@ ' ' This program is distributed in the hope that it will be useful, ' but WITHOUT ANY WARRANTY +Imports PersonalUtilities.Tools +Imports PersonalUtilities.Forms.Toolbars +Imports SCrawler.API.YouTube Namespace API.Base.YTDLP Friend Class YTDLPBatch : Inherits GDL.GDLBatch Friend Sub New(ByVal _Token As Threading.CancellationToken, Optional ByVal __MainProcessName As String = Nothing, Optional ByVal WorkingDir As SFile = Nothing) MyBase.New(_Token, __MainProcessName.IfNullOrEmpty(Settings.YtdlpFile.File.Name), WorkingDir.IfNullOrEmpty(Settings.YtdlpFile.File)) End Sub + Friend Sub SetProgress(ByVal Progress As MyProgress) + If Not Progress Is Nothing Then AddHandler OutputDataReceived, CreateYtdlpProgressHandler(Progress, Token, Me) + End Sub + Friend Sub CreateFileExchanger(ByVal cache As CacheKeeper, Optional ByVal root As SFile = Nothing) + FileExchanger = cache.NewInstance(Of BatchFileExchanger)(root, EDP.ReturnValue) + If Not FileExchanger Is Nothing Then FileExchanger.DeleteCacheOnDispose = True + End Sub End Class End Namespace \ No newline at end of file diff --git a/SCrawler/API/Bluesky/EditorExchangeOptions.vb b/SCrawler/API/Bluesky/EditorExchangeOptions.vb index be298d2..d6f6c84 100644 --- a/SCrawler/API/Bluesky/EditorExchangeOptions.vb +++ b/SCrawler/API/Bluesky/EditorExchangeOptions.vb @@ -9,12 +9,18 @@ Imports SCrawler.API.Base Imports SCrawler.Plugin.Attributes Namespace API.Bluesky - Friend Class EditorExchangeOptions : Inherits Base.EditorExchangeOptionsBase + Friend Class EditorExchangeOptions : Inherits EditorExchangeOptionsBase Friend Overrides Property SiteKey As String = BlueskySiteKey Friend Overridable Property DownloadModelMedia As Boolean = False Friend Overridable Property DownloadModelProfile As Boolean = False + + Friend Property UseMD5Comparison As Boolean = False + + Friend Property RemoveExistingDuplicates As Boolean = False Private ReadOnly Property MySettings As Object Friend Sub New(ByVal s As SiteSettings) MyBase.New(s) @@ -26,6 +32,8 @@ Namespace API.Bluesky MyBase.New(u) DownloadModelMedia = u.DownloadModelMedia DownloadModelProfile = u.DownloadModelProfile + UseMD5Comparison = u.UseMD5Comparison + RemoveExistingDuplicates = u.RemoveExistingDuplicates MySettings = u.HOST.Source End Sub Friend Overrides Sub Apply(ByRef u As UserDataBase) @@ -40,6 +48,8 @@ Namespace API.Bluesky With DirectCast(u, UserData) .DownloadModelMedia = DownloadModelMedia .DownloadModelProfile = DownloadModelProfile + .UseMD5Comparison = UseMD5Comparison + .RemoveExistingDuplicates = RemoveExistingDuplicates End With End Sub End Class diff --git a/SCrawler/API/Bluesky/UserData.vb b/SCrawler/API/Bluesky/UserData.vb index 7dd0d61..dc9525f 100644 --- a/SCrawler/API/Bluesky/UserData.vb +++ b/SCrawler/API/Bluesky/UserData.vb @@ -7,10 +7,12 @@ ' This program is distributed in the hope that it will be useful, ' but WITHOUT ANY WARRANTY Imports System.Threading +Imports SCrawler.Plugin Imports SCrawler.API.Base Imports SCrawler.API.YouTube.Objects Imports PersonalUtilities.Functions.XML Imports PersonalUtilities.Functions.RegularExpressions +Imports PersonalUtilities.Tools.Web.Clients Imports PersonalUtilities.Tools.Web.Documents.JSON Imports UTypes = SCrawler.API.Base.UserMedia.Types Imports UStates = SCrawler.API.Base.UserMedia.States @@ -56,6 +58,8 @@ Namespace API.Bluesky DownloadModelProfile = Not ParseUserMediaOnly End If ForceParseProfileInfo = .Value(Name_ForceParseProfileInfo).FromXML(Of Boolean)(False) + UseMD5Comparison = .Value(Name_UseMD5Comparison).FromXML(Of Boolean)(False) + RemoveExistingDuplicates = .Value(Name_RemoveExistingDuplicates).FromXML(Of Boolean)(False) Else If ID.IsEmptyString Then UpdateUserOptions() @@ -64,6 +68,8 @@ Namespace API.Bluesky .Add(Name_DownloadModelMedia, DownloadModelMedia.BoolToInteger) .Add(Name_DownloadModelProfile, DownloadModelProfile.BoolToInteger) .Add(Name_ForceParseProfileInfo, ForceParseProfileInfo.BoolToInteger) + .Add(Name_UseMD5Comparison, UseMD5Comparison.BoolToInteger) + .Add(Name_RemoveExistingDuplicates, RemoveExistingDuplicates.BoolToInteger) End If End With End Sub @@ -98,6 +104,7 @@ Namespace API.Bluesky #Region "Download" Private _PostCount As Integer = 0 Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken) + _CurlProfileTried = False _TmpPosts2.Clear() Try If Not DownloadModelMedia And Not DownloadModelProfile Then @@ -169,7 +176,11 @@ Namespace API.Bluesky End If End If Catch ex As Exception - ProcessException(ex, Token, $"DownloadData({URL})") + Select Case ProcessException(ex, Token, $"DownloadData({URL})") + Case 1 : DownloadData(Cursor, Token) + Case 2 : Throw ex + Case Else : Throw New ExitException($"StatusCode = {CInt(Responser.StatusCode)}", ex) With {.Silent = True} + End Select End Try End Sub #End Region @@ -279,21 +290,40 @@ Namespace API.Bluesky End Function #End Region #Region "GetProfileInfo" - Private Sub GetProfileInfo(ByVal Token As CancellationToken) + Private _CurlProfileTried As Boolean = False + Private Sub GetProfileInfo(ByVal Token As CancellationToken, Optional ByVal UseCurl As Boolean = False) Try If ForceParseProfileInfo Then ForceParseProfileInfo = False : _ForceSaveUserInfo = True - If UpdateToken() Then + If UseCurl OrElse UpdateToken() Then + If UseCurl Then + With Responser + .Mode = Responser.Modes.Curl + .CurlSslNoRevoke = True + .CurlInsecure = True + End With + _CurlProfileTried = True + End If Dim r$ = Responser.GetResponse($"https://bsky.social/xrpc/app.bsky.actor.getProfile?actor={ID.IfNullOrEmpty(NameTrue)}") TokenUpdateCountReset() If Not r.IsEmptyString Then Using j As EContainer = JsonDocument.Parse(r) If j.ListExists Then - ID = j.Value("did") - UserSiteNameUpdate(j.Value("displayName")) - UserDescriptionUpdate(j.Value("description")) - NameTrue = j.Value("handle") - SimpleDownloadAvatar(j.Value("avatar")) - SimpleDownloadAvatar(j.Value("banner")) + If UseCurl Then + If j.Value("error").StringToLower = "accountdeactivated" OrElse + (j.Value("error").StringToLower = "invalidrequest" And j.Value("message").StringToLower = "profile not found") Then + UserExists = False + Throw New ExitException(r) With {.Silent = True} + Else + Throw New Exception(r) + End If + Else + ID = j.Value("did") + UserSiteNameUpdate(j.Value("displayName")) + UserDescriptionUpdate(j.Value("description")) + NameTrue = j.Value("handle") + SimpleDownloadAvatar(j.Value("avatar")) + SimpleDownloadAvatar(j.Value("banner")) + End If End If End Using End If @@ -302,6 +332,8 @@ Namespace API.Bluesky End If Catch ex As Exception ProcessException(ex, Token, "GetProfileInfo") + Finally + If UseCurl Then Responser.Mode = Responser.Modes.Default End Try End Sub #End Region @@ -409,11 +441,17 @@ Namespace API.Bluesky Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False, Optional ByVal EObj As Object = Nothing) As Integer If Responser.StatusCode = Net.HttpStatusCode.BadRequest Then '400 - If _TokenUpdateCount = 0 AndAlso UpdateToken(True) Then + If Not _CurlProfileTried Then + GetProfileInfo(Nothing, True) + Return 1 + ElseIf _TokenUpdateCount = 0 AndAlso UpdateToken(True) Then + If _CurlProfileTried Then GetProfileInfo(Nothing, True) : _CurlProfileTried = False Return 1 Else - Return 0 + Return 2 End If + ElseIf Responser.StatusCode = Net.HttpStatusCode.BadGateway Or Responser.StatusCode = Net.HttpStatusCode.GatewayTimeout Then '502, 504 + Return 3 Else Return 0 End If diff --git a/SCrawler/API/Imgur/Envir.vb b/SCrawler/API/Imgur/Envir.vb deleted file mode 100644 index 9d47c2b..0000000 --- a/SCrawler/API/Imgur/Envir.vb +++ /dev/null @@ -1,139 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Imports System.Net -Imports System.Threading -Imports SCrawler.API.Base -Imports SCrawler.API.Imgur.Declarations -Imports SCrawler.API.YouTube.Objects -Imports SCrawler.Plugin.Hosts -Imports PersonalUtilities.Functions.XML -Imports PersonalUtilities.Functions.RegularExpressions -Imports PersonalUtilities.Tools.Web.Documents.JSON -Namespace API.Imgur - Namespace Declarations - Friend Module Imgur_Declarations - Friend ReadOnly PostRegex As RParams = RParams.DMS("/([^/]+?)(|#.*?|\.[\w]{0,4})(|\?.*?)\Z", 1) - End Module - End Namespace - Friend NotInheritable Class Envir : Inherits UserDataBase - Friend Const SiteKey As String = "AndyProgram_Imgur" - Friend Const SiteName As String = "Imgur" - Friend Sub New() - End Sub -#Region "UserDataBase Support" - Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean) - End Sub - Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken) - End Sub - Protected Overrides Sub DownloadContent(ByVal Token As CancellationToken) - SeparateVideoFolder = False - DownloadContentDefault(Token) - End Sub - Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False, - Optional ByVal EObj As Object = Nothing) As Integer - Return 0 - End Function -#End Region - Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken) - Dim videos As IEnumerable(Of UserMedia) = GetVideoInfo(Data.URL, EDP.SendToLog) - If videos.ListExists Then _TempMediaList.AddRange(videos) - End Sub - Friend Shared Function GetGallery(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As List(Of String) - Try - If Not Settings.ImgurClientID.IsEmptyString And Not URL.IsEmptyString Then - Dim __url$ = RegexReplace(URL, PostRegex) - If Not __url.IsEmptyString Then - __url = $"https://api.imgur.com/post/v1/albums/{__url}?client_id={Settings.ImgurClientID.Value}&include=media" - Using w As New WebClient - Dim r$ = w.DownloadString(__url) - If Not r.IsEmptyString Then - Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing - If j.Contains("media") Then - Dim UrlsList As New List(Of String) - Dim tmpUrl$ - For Each m As EContainer In j("media") - tmpUrl = m.Value("url") - If Not tmpUrl.IsEmptyString Then UrlsList.ListAddValue(tmpUrl, Base.LNC) - Next - Return UrlsList - End If - End Using - End If - End Using - End If - End If - Return Nothing - Catch ex As Exception - Return DownloadingException_Internal(ex, $"[API.Imgur.Envir.GetGallery({URL})]", Nothing, e) - End Try - End Function - Friend Shared Function GetImage(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As String - Try - If Not Settings.ImgurClientID.IsEmptyString And Not URL.IsEmptyString Then - Dim __url$ = RegexReplace(URL, PostRegex) - If Not __url.IsEmptyString Then - __url = $"https://api.imgur.com/3/image/{__url}?client_id={Settings.ImgurClientID.Value}&include=media" - Using w As New WebClient - Dim r$ = w.DownloadString(__url) - If Not r.IsEmptyString Then Return JsonDocument.Parse(r).XmlIfNothing.Value({"data"}, "link") - End Using - End If - End If - Return String.Empty - Catch ex As Exception - Return DownloadingException_Internal(ex, $"[API.Imgur.Envir.GetImage({URL})]", String.Empty, e) - End Try - End Function - Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia) - Try - If Not URL.IsEmptyString AndAlso URL.ToLower.Contains("imgur") AndAlso Not Settings.ImgurClientID.IsEmptyString Then - Dim imgList As List(Of String) = GetGallery(URL, EDP.ReturnValue) - If imgList.ListExists Then - Return imgList.Select(Function(u) New UserMedia(u)) - Else - Dim img$ = GetImage(URL, EDP.ReturnValue) - If Not img.IsEmptyString Then Return {New UserMedia(img)} - End If - End If - Return Nothing - Catch ex As Exception - If Not e.Exists Then e = EDP.LogMessageValue - Return ErrorsDescriber.Execute(e, ex, $"[API.Imgur.Envir.GetVideoInfo({URL})]: fetch media error") - End Try - End Function - Friend Shared Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As SFile) As IYouTubeMediaContainer - If Not URL.IsEmptyString AndAlso URL.Contains("imgur.com") Then - Return New DownloadableMediaHost(URL, OutputFile) With { - .Instance = New Envir, - .Site = SiteName, - .SiteKey = SiteKey, - .SiteIcon = Nothing - } - Else - Return Nothing - End If - End Function - Private Shared Function DownloadingException_Internal(ByVal ex As Exception, ByVal Message As String, - ByVal NullArg As Object, ByVal e As ErrorsDescriber) As Object - If TypeOf ex Is WebException Then - Dim obj As HttpWebResponse = TryCast(DirectCast(ex, WebException).Response, HttpWebResponse) - If Not obj Is Nothing Then - If obj.StatusCode = HttpStatusCode.NotFound Then - Return NullArg - ElseIf obj.StatusCode = HttpStatusCode.Unauthorized Then - MyMainLOG = "Imgur credentials expired" - Return NullArg - End If - End If - End If - If Not e.Exists Then e = New ErrorsDescriber(EDP.ReturnValue + EDP.SendToLog) - Return ErrorsDescriber.Execute(e, ex, Message, NullArg) - End Function - End Class -End Namespace \ No newline at end of file diff --git a/SCrawler/API/Instagram/SiteSettings.vb b/SCrawler/API/Instagram/SiteSettings.vb index 3113063..a6cb32d 100644 --- a/SCrawler/API/Instagram/SiteSettings.vb +++ b/SCrawler/API/Instagram/SiteSettings.vb @@ -146,7 +146,7 @@ Namespace API.Instagram Private ReadOnly Property TokenUpdateIntervalProvider As IFormatProvider #End Region - + Friend ReadOnly Property USE_GQL As PropertyValue Friend ReadOnly Property USE_GQL_UserData As PropertyValue @@ -492,11 +492,11 @@ Namespace API.Instagram HH_IG_WWW_CLAIM_UPDATE_INTERVAL = New PropertyValue(120) HH_IG_WWW_CLAIM_ALWAYS_ZERO = New PropertyValue(False) HH_IG_WWW_CLAIM_RESET_EACH_SESSION = New PropertyValue(True) - HH_IG_WWW_CLAIM_RESET_EACH_TARGET = New PropertyValue(True) + HH_IG_WWW_CLAIM_RESET_EACH_TARGET = New PropertyValue(False) HH_IG_WWW_CLAIM_USE = New PropertyValue(True) HH_IG_WWW_CLAIM_USE_DEFAULT_ALGO = New PropertyValue(True) TokenUpdateIntervalProvider = New TokenRefreshIntervalProvider - USE_GQL = New PropertyValue(False) + USE_GQL = New PropertyValue(True) USE_GQL_UserData = New PropertyValue(True) USE_GQL_Highlights = New PropertyValue(True) @@ -515,7 +515,7 @@ Namespace API.Instagram PostNumberVerifiedNot = New PropertyValue(12) PostNumberVerifiedNotProvider = New TimersChecker(12) - RequestsWaitTimer_Any = New PropertyValue(1000) + RequestsWaitTimer_Any = New PropertyValue(5000) RequestsWaitTimer_AnyProvider = New TimersChecker(0) RequestsWaitTimer = New PropertyValue(1000) RequestsWaitTimerProvider = New TimersChecker(100) @@ -544,10 +544,10 @@ Namespace API.Instagram SkipErrors_AddToLog_Silent = New PropertyValue(String.Empty, GetType(String)) IgnoreStoriesDownloadingErrors = New PropertyValue(False) - DownDetectorValue = New PropertyValue(20) + DownDetectorValue = New PropertyValue(30) DownDetectorValueProvider = New TimersChecker(-1) - DownDetectorValueAddToLog = New PropertyValue(False) - TaggedNotifyLimit = New PropertyValue(200) + DownDetectorValueAddToLog = New PropertyValue(True) + TaggedNotifyLimit = New PropertyValue(50) TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker DownloadingErrorDate = New PropertyValue(Now.AddYears(-10), GetType(Date)) @@ -563,17 +563,13 @@ Namespace API.Instagram UserRegex = RParams.DMS(String.Format(UserRegexDefaultPattern, "instagram.com/"), 1) ImageVideoContains = "instagram.com" End Sub - Private Const SettingsVersionCurrent As Integer = 3 + Private Const SettingsVersionCurrent As Integer = 4 Friend Overrides Sub EndInit() Try : MyLastRequests.Add(LastDownloadDate.Value, LastRequestsCount.Value) : Catch : End Try If Not CBool(HH_IG_WWW_CLAIM_USE.Value) Then Responser.Headers.Remove(Header_IG_WWW_CLAIM) If CInt(SettingsVersion.Value) < SettingsVersionCurrent Then SettingsVersion.Value = SettingsVersionCurrent - HH_IG_WWW_CLAIM_RESET_EACH_TARGET.Value = False - RequestsWaitTimer_Any.Value = 5000 - TaggedNotifyLimit.Value = 50 - DownDetectorValue.Value = 30 - DownDetectorValueAddToLog.Value = True + USE_GQL.Value = True End If MyBase.EndInit() End Sub diff --git a/SCrawler/API/Mastodon/Credentials.vb b/SCrawler/API/Mastodon/Credentials.vb deleted file mode 100644 index cd40555..0000000 --- a/SCrawler/API/Mastodon/Credentials.vb +++ /dev/null @@ -1,48 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Imports PersonalUtilities.Functions.XML -Imports PersonalUtilities.Functions.XML.Base -Namespace API.Mastodon - Friend Structure Credentials : Implements IEContainerProvider - Private Const Name_Bearer As String = "Bearer" - Private Const Name_Csrf As String = "Csrf" - Friend Domain As String - Friend Bearer As String - Friend Csrf As String - Friend ReadOnly Property Exists As Boolean - Get - Return Not Domain.IsEmptyString And Not Bearer.IsEmptyString And Not Csrf.IsEmptyString - End Get - End Property - Private Sub New(ByVal e As EContainer) - Domain = e.Value - Bearer = e.Attribute(Name_Bearer) - Csrf = e.Attribute(Name_Csrf) - End Sub - Public Shared Widening Operator CType(ByVal e As EContainer) As Credentials - Return New Credentials(e) - End Operator - Public Shared Widening Operator CType(ByVal Domain As String) As Credentials - Return New Credentials With {.Domain = Domain} - End Operator - Private Function ToEContainer(Optional ByVal e As ErrorsDescriber = Nothing) As EContainer Implements IEContainerProvider.ToEContainer - Return New EContainer("Item", Domain, {New EAttribute(Name_Bearer, Bearer), New EAttribute(Name_Csrf, Csrf)}) - End Function - Public Overrides Function Equals(ByVal Obj As Object) As Boolean - If Not IsNothing(Obj) Then - If TypeOf Obj Is Credentials Then - Return Domain = DirectCast(Obj, Credentials).Domain - Else - Return Domain = CStr(Obj) - End If - End If - Return False - End Function - End Structure -End Namespace \ No newline at end of file diff --git a/SCrawler/API/Mastodon/Declarations.vb b/SCrawler/API/Mastodon/Declarations.vb deleted file mode 100644 index 3c97354..0000000 --- a/SCrawler/API/Mastodon/Declarations.vb +++ /dev/null @@ -1,14 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Namespace API.Mastodon - Friend Module Declarations - Friend Const MastodonSiteKey As String = "AndyProgram_Mastodon" - Friend ReadOnly DateProvider As New ADateTime("O") - End Module -End Namespace \ No newline at end of file diff --git a/SCrawler/API/Mastodon/EditorExchangeOptions.vb b/SCrawler/API/Mastodon/EditorExchangeOptions.vb deleted file mode 100644 index 2398d33..0000000 --- a/SCrawler/API/Mastodon/EditorExchangeOptions.vb +++ /dev/null @@ -1,27 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Imports SCrawler.Plugin.Attributes -Namespace API.Mastodon - Friend Class EditorExchangeOptions : Inherits Twitter.EditorExchangeOptions - Friend Overrides Property MediaModelAllowNonUserTweets As Boolean - Friend Overrides Property DownloadModelMedia As Boolean - Friend Overrides Property DownloadModelProfile As Boolean - Friend Overrides Property DownloadModelSearch As Boolean - Friend Overrides Property DownloadModelForceApply As Boolean - Friend Overrides Property DownloadModelLikes As Boolean - Friend Overrides Property DownloadBroadcasts As Boolean - Friend Overrides Property UserName As String - Friend Sub New(ByVal s As SiteSettings) - MyBase.New(s) - End Sub - Friend Sub New(ByVal u As UserData) - MyBase.New(u) - End Sub - End Class -End Namespace \ No newline at end of file diff --git a/SCrawler/API/Mastodon/SiteSettings.vb b/SCrawler/API/Mastodon/SiteSettings.vb deleted file mode 100644 index 7251281..0000000 --- a/SCrawler/API/Mastodon/SiteSettings.vb +++ /dev/null @@ -1,253 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Imports SCrawler.API.Base -Imports SCrawler.Plugin -Imports SCrawler.Plugin.Attributes -Imports PersonalUtilities.Functions.XML -Imports PersonalUtilities.Functions.RegularExpressions -Imports PersonalUtilities.Tools.Web.Clients -Imports PersonalUtilities.Tools.Web.Documents.JSON -Imports DN = SCrawler.API.Base.DeclaredNames -Namespace API.Mastodon - - Friend Class SiteSettings : Inherits SiteSettingsBase -#Region "Declarations" -#Region "Domains" - Private ReadOnly Property SiteDomains As PropertyValue - Private Shadows ReadOnly Property DefaultInstance As SiteSettings - Get - Return MyBase.DefaultInstance - End Get - End Property - Private ReadOnly _Domains As DomainsContainer - Friend ReadOnly Property Domains As DomainsContainer - Get - Return If(DefaultInstance?.Domains, _Domains) - End Get - End Property - Private ReadOnly Property Base_DomainsLastUpdateDate As PropertyValue - Private ReadOnly Property DomainsLastUpdateDate As PropertyValue - Get - Return If(DefaultInstance?.DomainsLastUpdateDate, Base_DomainsLastUpdateDate) - End Get - End Property -#End Region -#Region "Auth" - - Friend ReadOnly Property MyDomain As PropertyValue - - Friend ReadOnly Property Auth As PropertyValue - - Friend ReadOnly Property Token As PropertyValue - Private Sub ChangeResponserFields(ByVal PropName As String, ByVal Value As Object) - If Not PropName.IsEmptyString Then - Dim f$ = String.Empty - Select Case PropName - Case NameOf(Auth) : f = DN.Header_Authorization - Case NameOf(Token) : f = DN.Header_CSRFToken - End Select - If Not f.IsEmptyString Then - Responser.Headers.Remove(f) - If Not CStr(Value).IsEmptyString Then Responser.Headers.Add(f, CStr(Value)) - Responser.SaveSettings() - End If - End If - End Sub -#End Region -#Region "Other properties" - - Friend ReadOnly Property GifsDownload As PropertyValue - - Friend ReadOnly Property GifsSpecialFolder As PropertyValue - - Friend ReadOnly Property GifsPrefix As PropertyValue - - Private ReadOnly Property GifStringChecker As IFormatProvider - - Friend ReadOnly Property UseMD5Comparison As PropertyValue - - Friend ReadOnly Property UserRelatedToMyDomain As PropertyValue - Friend Overrides Property DownloadText As PropertyValue - Friend Overrides Property DownloadTextPosts As PropertyValue - Friend Overrides Property DownloadTextSpecialFolder As PropertyValue -#End Region -#End Region -#Region "Initializer" - Friend Sub New(ByVal AccName As String, ByVal Temp As Boolean) - MyBase.New("Mastodon", "mastodon.social", AccName, Temp, My.Resources.SiteResources.MastodonIcon_48, My.Resources.SiteResources.MastodonPic_48) - - _Domains = New DomainsContainer(Me, "mastodon.social") - SiteDomains = New PropertyValue(Domains.DomainsDefault, GetType(String)) - Domains.DestinationProp = SiteDomains - Base_DomainsLastUpdateDate = New PropertyValue(Now.AddYears(-1)) - - Auth = New PropertyValue(Responser.Headers.Value(DN.Header_Authorization), GetType(String), Sub(v) ChangeResponserFields(NameOf(Auth), v)) - Token = New PropertyValue(Responser.Headers.Value(DN.Header_CSRFToken), GetType(String), Sub(v) ChangeResponserFields(NameOf(Token), v)) - - GifsDownload = New PropertyValue(True) - GifsSpecialFolder = New PropertyValue(String.Empty, GetType(String)) - GifsPrefix = New PropertyValue("GIF_") - GifStringChecker = New API.Twitter.SiteSettings.GifStringProvider - UseMD5Comparison = New PropertyValue(False) - MyDomain = New PropertyValue(String.Empty, GetType(String)) - UserRelatedToMyDomain = New PropertyValue(False) - - UserRegex = RParams.DMS("", 0, RegexReturn.ListByMatch, EDP.ReturnValue) - End Sub - Friend Overrides Sub EndInit() - Domains.PopulateInitialDomains(SiteDomains.Value) - If CDate(DomainsLastUpdateDate.Value).AddDays(7) < Now Then UpdateServersList() - MyBase.EndInit() - End Sub -#End Region -#Region "GetInstance" - Friend Overrides Function GetInstance(ByVal What As ISiteSettings.Download) As IPluginContentProvider - Return New UserData - End Function -#End Region -#Region "Domains Support" - Protected Overrides Sub DomainsApply() - Domains.Apply() - MyBase.DomainsApply() - End Sub - Protected Overrides Sub DomainsReset() - Domains.Reset() - MyBase.DomainsReset() - End Sub - Friend Overrides Sub OpenSettingsForm() - Domains.OpenSettingsForm() - End Sub -#End Region -#Region "Update" - Friend Overrides Sub Update() - If _SiteEditorFormOpened Then - Dim tf$ = GifsSpecialFolder.Value - If Not tf.IsEmptyString Then tf = tf.StringTrim("\") : GifsSpecialFolder.Value = tf - Dim md$ = AConvert(Of String)(MyDomain.Value, String.Empty) - If Not md.IsEmptyString AndAlso Not Domains.Domains.Contains(md) AndAlso Not Domains.DomainsTemp.Contains(md) Then - If Domains.Changed Then - Domains.DomainsTemp.Add(md) - Else - Domains.Domains.Add(md) - Domains.Save() - End If - End If - End If - MyBase.Update() - End Sub -#End Region -#Region "UserOptions" - Friend Overrides Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean) - If Options Is Nothing OrElse (Not TypeOf Options Is EditorExchangeOptions OrElse - Not DirectCast(Options, EditorExchangeOptions).SiteKey = MastodonSiteKey) Then _ - Options = New EditorExchangeOptions(Me) With {.SiteKey = MastodonSiteKey} - If OpenForm Then - Using f As New InternalSettingsForm(Options, Me, False) : f.ShowDialog() : End Using - End If - End Sub -#End Region -#Region "Download" - Friend Overrides Function BaseAuthExists() As Boolean - Return ACheck(Token.Value) And ACheck(Auth.Value) And ACheck(MyDomain.Value) - End Function - Friend Overrides Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean - If What = ISiteSettings.Download.SavedPosts Or What = ISiteSettings.Download.SingleObject Then - If Not ACheck(MyDomain.Value) Then MyMainLOG = "Mastodon account domain not set" : Return False - Else - If CDate(DomainsLastUpdateDate.Value).AddDays(7) < Now Then UpdateServersList() - End If - Return MyBase.Available(What, Silent) - End Function -#End Region -#Region "GetUserUrl, GetUserPostUrl" - Friend Overrides Function GetUserUrl(ByVal User As IPluginContentProvider) As String - If Not ACheck(MyDomain.Value) Then Return String.Empty - With DirectCast(User, UserData) - If UserRelatedToMyDomain.Value Then - If MyDomain.Value = .UserDomain Then - Return $"https://{ .UserDomain}/@{ .NameTrue}" - Else - Return $"https://{MyDomain.Value}/@{ .NameTrue}@{ .UserDomain}" - End If - Else - Return $"https://{ .UserDomain}/@{ .NameTrue}" - End If - End With - End Function - Friend Overrides Function GetUserPostUrl(ByVal User As UserDataBase, ByVal Media As UserMedia) As String - If Not ACheck(MyDomain.Value) Then Return String.Empty - Return $"{GetUserUrl(User)}/{Media.Post.ID}" - End Function -#End Region -#Region "IsMyUser, IsMyImageVideo" - Private Const UserRegexDefault As String = "https?://{0}/@([^/@]+)@?([^/]*)" - Friend Overloads Overrides Function IsMyUser(ByVal UserURL As String) As ExchangeOptions - If Domains.Count > 0 Then - Dim e As ExchangeOptions - If ACheck(MyDomain.Value) Then - e = IsMyUser(UserURL, MyDomain.Value) - If Not e.SiteName.IsEmptyString Then Return e - End If - For Each domain$ In Domains - e = IsMyUser(UserURL, domain) - If Not e.SiteName.IsEmptyString Then Return e - Next - End If - Return Nothing - End Function - Private Overloads Function IsMyUser(ByVal UserURL As String, ByVal Domain As String) As ExchangeOptions - UserRegex.Pattern = String.Format(UserRegexDefault, Domain) - Dim l As List(Of String) = RegexReplace(UserURL, UserRegex) - If l.ListExists(2) Then Return New ExchangeOptions(Site, $"{l(2).IfNullOrEmpty(Domain)}@{l(1)}") Else Return Nothing - End Function - Friend Overrides Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions - If Not URL.IsEmptyString And Domains.Count > 0 Then - Dim urlDomain$ = RegexReplace(URL, RParams.DM("[^/]+", 1, EDP.ReturnValue, String.Empty)) - If Not urlDomain.IsEmptyString Then - urlDomain = urlDomain.StringToLower - If Domains.Domains.Exists(Function(d) urlDomain = d.StringToLower) Then Return New ExchangeOptions(Site, URL) With {.Exists = True} - End If - End If - Return Nothing - End Function -#End Region -#Region "UpdateServersList" - Private Sub UpdateServersList() - Try - Using resp As New Responser With { - .ProcessExceptionDecision = Function(rr, obj, e) If(rr.StatusCode = Net.HttpStatusCode.ServiceUnavailable, - EDP.ReturnValue, EDP.ThrowException)} - Dim r$ = resp.GetResponse("https://api.joinmastodon.org/servers?language=&category=®ion=&ownership=®istrations=") - If Not r.IsEmptyString Then - Dim j As EContainer = JsonDocument.Parse(r, EDP.ReturnValue) - If If(j?.Count, 0) > 0 Then - Domains.Domains.ListAddList(j.Select(Function(e) e.Value("domain")), LAP.NotContainsOnly, EDP.ReturnValue) - Domains.Domains.Sort() - Domains.Save() - j.Dispose() - End If - DomainsLastUpdateDate.Value = Now - End If - End Using - Catch ex As Exception - ErrorsDescriber.Execute(EDP.SendToLog, ex, "[API.Mastodon.SiteSettings.UpdateServersList]") - End Try - End Sub -#End Region -#Region "IDisposable Support" - Protected Overrides Sub Dispose(ByVal disposing As Boolean) - If Not disposedValue And disposing Then _Domains.Dispose() - MyBase.Dispose(disposing) - End Sub -#End Region - End Class -End Namespace \ No newline at end of file diff --git a/SCrawler/API/Mastodon/UserData.vb b/SCrawler/API/Mastodon/UserData.vb deleted file mode 100644 index 7e43dfe..0000000 --- a/SCrawler/API/Mastodon/UserData.vb +++ /dev/null @@ -1,319 +0,0 @@ -' Copyright (C) 2023 Andy https://github.com/AAndyProgram -' This program is free software: you can redistribute it and/or modify -' it under the terms of the GNU General Public License as published by -' the Free Software Foundation, either version 3 of the License, or -' (at your option) any later version. -' -' This program is distributed in the hope that it will be useful, -' but WITHOUT ANY WARRANTY -Imports System.Threading -Imports SCrawler.API.Base -Imports SCrawler.API.YouTube.Objects -Imports PersonalUtilities.Functions.XML -Imports PersonalUtilities.Functions.RegularExpressions -Imports PersonalUtilities.Tools.Web.Documents.JSON -Imports UTypes = SCrawler.API.Base.UserMedia.Types -Imports UStates = SCrawler.API.Base.UserMedia.States -Namespace API.Mastodon - Friend Class UserData : Inherits Twitter.UserData -#Region "XML names" - Private Const Name_UserDomain As String = "UserDomain" -#End Region -#Region "Declarations" - Private _UserDomain As String = String.Empty - Friend Property UserDomain As String - Get - Return _UserDomain.IfNullOrEmpty(MySettings.MyDomain.Value) - End Get - Set(ByVal d As String) - _UserDomain = d - End Set - End Property - Private ReadOnly Property MySettings As SiteSettings - Get - Return HOST.Source - End Get - End Property - Private MyCredentials As Credentials - Private Sub ResetCredentials() - MyCredentials = New Credentials With {.Domain = MySettings.MyDomain.Value, .Bearer = MySettings.Auth.Value, .Csrf = MySettings.Token.Value} - With MyCredentials - Responser.Headers.Add(DeclaredNames.Header_Authorization, .Bearer) - Responser.Headers.Add(DeclaredNames.Header_CSRFToken, .Csrf) - End With - End Sub -#End Region -#Region "LoadUserInformation" - Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean) - MyBase.LoadUserInformation_OptionalFields(Container, Loading) - Dim obtainNames As Action = Sub() - If _UserDomain.IsEmptyString And Not Name.IsEmptyString Then - Dim l$() = Name.Split("@") - If l.ListExists(2) Then - _UserDomain = l(0) - NameTrue = l(1) - Else - _UserDomain = MySettings.MyDomain.Value - NameTrue = Name - End If - If FriendlyName.IsEmptyString Then FriendlyName = NameTrue - End If - End Sub - If Loading Then - _UserDomain = Container.Value(Name_UserDomain) - obtainNames.Invoke - Else - obtainNames.Invoke - Container.Add(Name_UserDomain, _UserDomain) - Container.Add(Name_TrueName, NameTrue(True)) - Container.Value(Name_FriendlyName) = FriendlyName - End If - End Sub -#End Region -#Region "Download functions" - Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken) - ResetCredentials() - DownloadData(String.Empty, Token) - End Sub - Private Overloads Sub DownloadData(ByVal POST As String, ByVal Token As CancellationToken) - Dim URL$ = String.Empty - Try - Dim PostID$ = String.Empty - Dim PostDate$ - Dim s As EContainer, ss As EContainer - Dim NewPostDetected As Boolean = False - Dim ExistsDetected As Boolean = False - - If IsSavedPosts Then - URL = $"https://{MySettings.MyDomain.Value}/api/v1/bookmarks" - If Not POST.IsEmptyString Then URL &= $"?max_id={POST}" - Else - If POST.IsEmptyString And ID.IsEmptyString Then - ObtainUserID() - If ID.IsEmptyString Then Throw New ArgumentNullException("ID", "Unable to get user ID") With {.HelpLink = 1} - End If - URL = $"https://{MyCredentials.Domain}/api/v1/accounts/{ID}/statuses?" - If ParseUserMediaOnly Then URL &= "only_media=true&" - URL &= "limit=40" - If Not POST.IsEmptyString Then URL &= $"&max_id={POST}" - End If - - ThrowAny(Token) - Dim r$ = Responser.GetResponse(URL) - If Not r.IsEmptyString Then - Using j As EContainer = JsonDocument.Parse(r) - If If(j?.Count, 0) > 0 Then - ProgressPre.ChangeMax(j.Count) - For Each jj As EContainer In j - ProgressPre.Perform() - With jj - If Not IsSavedPosts And POST.IsEmptyString And Not .Item("account") Is Nothing Then - With .Item("account") - If .Value("id") = ID Then - UserSiteNameUpdate(.Value("display_name")) - UserDescriptionUpdate(.Value("note")) - Dim __getImage As Action(Of String) = Sub(ByVal img As String) - If Not img.IsEmptyString Then - Dim __imgFile As SFile = img - If Not __imgFile.Name.IsEmptyString Then - If __imgFile.Extension.IsEmptyString Then __imgFile.Extension = "jpg" - __imgFile.Path = MyFile.CutPath.Path - If Not __imgFile.Exists Then GetWebFile(img, __imgFile, EDP.None) - If __imgFile.Exists Then IconBannerDownloaded = True - End If - End If - End Sub - If DownloadIconBanner Then - __getImage.Invoke(.Value("header").IfNullOrEmpty(.Value("header_static"))) - __getImage.Invoke(.Value("avatar").IfNullOrEmpty(.Value("avatar_static"))) - End If - End If - End With - End If - - PostID = .Value("id") - PostDate = .Value("created_at") - - If Not IsSavedPosts And Not PostDate.IsEmptyString Then - Select Case CheckDatesLimit(PostDate, DateProvider) - Case DateResult.Skip : Continue For - Case DateResult.Exit : Exit Sub - End Select - End If - - If Not _TempPostsList.Contains(PostID) Then - NewPostDetected = True - _TempPostsList.Add(PostID) - Else - ExistsDetected = True - Continue For - End If - - If IsSavedPosts OrElse (Not ParseUserMediaOnly OrElse - (If(.Item("reblog")?.Count, 0) = 0 OrElse .Value({"reblog", "account"}, "id") = ID)) Then - If If(.Item("media_attachments")?.Count, 0) > 0 Then - s = .Item("media_attachments") - Else - s = .Item({"reblog"}, "media_attachments") - End If - If s.ListExists Then - For Each ss In s : ObtainMedia(ss, PostID, PostDate) : Next - End If - End If - End With - Next - End If - End Using - End If - - If POST.IsEmptyString And ExistsDetected Then Exit Sub - If Not PostID.IsEmptyString And NewPostDetected Then DownloadData(PostID, Token) - Catch ex As Exception - ProcessException(ex, Token, $"data downloading error{IIf(IsSavedPosts, " (Saved Posts)", String.Empty)} [{URL}]") - End Try - End Sub - Private Sub ObtainMedia(ByVal e As EContainer, ByVal PostID As String, ByVal PostDate As String, Optional ByVal BaseUrl As String = Nothing, - Optional ByVal SourceMedia As UserMedia = Nothing) - Dim t As UTypes = UTypes.Undefined - Select Case e.Value("type") - Case "video" : t = UTypes.Video - Case "image" : t = UTypes.Picture - Case "gifv" : t = UTypes.GIF - End Select - If Not t = UTypes.Undefined Then - Dim m As New UserMedia(e.Value("url"), t) With { - .Post = New UserPost(PostID, AConvert(Of Date)(PostDate, DateProvider, Nothing, EDP.ReturnValue)), - .URL_BASE = BaseUrl.IfNullOrEmpty(MySettings.GetUserPostUrl(Me, m)) - } - If Not t = UTypes.GIF Or GifsDownload Then - If t = UTypes.GIF Then - If Not GifsSpecialFolder.IsEmptyString Then m.SpecialFolder = GifsSpecialFolder - If Not GifsPrefix.IsEmptyString Then m.File.Name = $"{GifsPrefix}{m.File.Name}" - End If - If Not m.URL.IsEmptyString Then - If SourceMedia.State = UStates.Missing Then - m.State = UStates.Missing - m.Attempts = SourceMedia.Attempts - End If - _TempMediaList.ListAddValue(m, LNC) - End If - End If - End If - End Sub - Private Sub ObtainUserID() - Try - If ID.IsEmptyString Then - Dim url$ = $"https://{MyCredentials.Domain}/api/v1/accounts/lookup?acct=" - If Not UserDomain.IsEmptyString Then - If UserDomain = MyCredentials.Domain Then - url &= $"@{NameTrue}" - Else - url &= $"@{NameTrue}@{UserDomain}" - End If - Else - url &= $"@{NameTrue}" - End If - Dim r$ = Responser.GetResponse(url) - If Not r.IsEmptyString Then - Using j As EContainer = JsonDocument.Parse(r) - If Not j Is Nothing Then ID = j.Value("id") - End Using - End If - End If - Catch ex As Exception - ErrorsDescriber.Execute(EDP.SendToLog, ex, $"API.Mastodon.UserData.ObtainUserID({ToStringForLog()})") - End Try - End Sub - Private Function GetSinglePostPattern(ByVal Domain As String) As String - Return $"https://{Domain}/api/v1/statuses/" & "{0}" - End Function - Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken) - Dim rList As New List(Of Integer) - Dim URL$ = String.Empty - Try - If ContentMissingExists Then - Dim SinglePostUrl$ = GetSinglePostPattern(MyCredentials.Domain) - Dim m As UserMedia - Dim r$, PostDate$ - Dim j As EContainer - ProgressPre.ChangeMax(_ContentList.Count) - For i% = 0 To _ContentList.Count - 1 - ProgressPre.Perform() - If _ContentList(i).State = UStates.Missing Then - m = _ContentList(i) - If Not m.Post.ID.IsEmptyString Then - ThrowAny(Token) - URL = String.Format(SinglePostUrl, m.Post.ID) - r = Responser.GetResponse(URL,, EDP.ReturnValue) - If Not r.IsEmptyString Then - j = JsonDocument.Parse(r) - If Not j Is Nothing Then - PostDate = String.Empty - If j.Contains("created_at") Then PostDate = j("created_at").Value Else PostDate = String.Empty - ObtainMedia(j, m.Post.ID, PostDate, m.URL_BASE, m) - rList.Add(i) - j.Dispose() - End If - End If - End If - End If - Next - End If - Catch ex As Exception - ProcessException(ex, Token, $"ReparseMissing error [{URL}]") - Finally - If rList.Count > 0 Then - For i% = rList.Count - 1 To 0 Step -1 : _ContentList.RemoveAt(rList(i)) : Next - rList.Clear() - End If - End Try - End Sub -#End Region -#Region "DownloadSingleObject" - Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken) - Dim PostID$ = RegexReplace(Data.URL, RParams.DM("(?<=/)\d+", 0, EDP.ReturnValue)) - If Not PostID.IsEmptyString Then - ResetCredentials() - Dim pattern$ - If Not ACheck(MySettings.MyDomain.Value) Then - Throw New ArgumentNullException("Mastodon domain", "Mastodon domain not set") - Else - pattern = GetSinglePostPattern(MySettings.MyDomain.Value) - End If - Dim r$ = Responser.GetResponse(String.Format(pattern, PostID),, EDP.ReturnValue) - If Not r.IsEmptyString Then - Using j As EContainer = JsonDocument.Parse(r) - If j.ListExists AndAlso j.Contains("media_attachments") Then - For Each jj As EContainer In j("media_attachments") : ObtainMedia(jj, PostID, String.Empty, Data.URL) : Next - End If - End Using - End If - End If - End Sub -#End Region -#Region "Exception" - Protected Overrides Function DownloadingException(ByVal ex As Exception, ByVal Message As String, Optional ByVal FromPE As Boolean = False, - Optional ByVal EObj As Object = Nothing) As Integer - If TypeOf ex Is ArgumentNullException AndAlso Not ex.HelpLink.IsEmptyString And ex.HelpLink = 1 Then - Return 0 - Else - If Responser.Status = Net.WebExceptionStatus.NameResolutionFailure Then - MyMainLOG = $"User domain ({UserDomain}) not found: {ToStringForLog()}" - Return 1 - ElseIf Responser.StatusCode = Net.HttpStatusCode.NotFound Or Responser.StatusCode = Net.HttpStatusCode.Forbidden Then - UserExists = False - Return 1 - ElseIf Responser.StatusCode = Net.HttpStatusCode.Unauthorized Then - MyMainLOG = $"{ToStringForLog()}: account credentials have expired" - Return 2 - ElseIf Responser.StatusCode = Net.HttpStatusCode.Gone Then - UserSuspended = True - Return 1 - Else - Return 0 - End If - End If - End Function -#End Region - End Class -End Namespace \ No newline at end of file diff --git a/SCrawler/API/OnlyFans/UserData.vb b/SCrawler/API/OnlyFans/UserData.vb index c91b4a7..4e93dd4 100644 --- a/SCrawler/API/OnlyFans/UserData.vb +++ b/SCrawler/API/OnlyFans/UserData.vb @@ -31,6 +31,7 @@ Namespace API.OnlyFans Friend Property CCookie As CookieKeeper = Nothing Private Const HeaderSign As String = "Sign" Private Const HeaderTime As String = "Time" + Private Const LabelDRM As String = "DRM" Private ReadOnly HighlightsList As List(Of String) Friend Property MediaDownloadTimeline As Boolean = True Friend Property MediaDownloadStories As Boolean = True @@ -41,6 +42,11 @@ Namespace API.OnlyFans Return HOST.Source End Get End Property + Friend Overrides ReadOnly Property SpecialLabels As IEnumerable(Of String) + Get + Return {LabelDRM} + End Get + End Property #End Region #Region "Load" Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean) @@ -626,6 +632,7 @@ Namespace API.OnlyFans #Region "OFScraper support" Private Function OFS_DownloadFile(ByVal URL As String, ByVal Token As CancellationToken) As List(Of SFile) Try + If Labels.Count = 0 OrElse Not Labels.Contains(LabelDRM) Then Labels.Add(LabelDRM) : _ForceSaveUserInfo = True Const requestPattern$ = """{0}"" manual --config ""{1}"" --url {2}" Dim conf As SFile = OFS_CreateConfig() If conf.Exists Then @@ -790,6 +797,7 @@ Namespace API.OnlyFans End If End If End If + If Not f.Exists Then f = Nothing Return f End If Return Nothing diff --git a/SCrawler/API/PornHub/Declarations.vb b/SCrawler/API/PornHub/Declarations.vb index 8c92bdf..a838329 100644 --- a/SCrawler/API/PornHub/Declarations.vb +++ b/SCrawler/API/PornHub/Declarations.vb @@ -47,6 +47,7 @@ Namespace API.PornHub CType(Function(Input$) If(Input.IsEmptyString, String.Empty, $"https://www.pornhub.com{Input.Trim}"), Func(Of String, String))) Friend ReadOnly Regex_Photo_PornHub_SinglePhoto As RParams = RParams.DM("data-image=""([^""]+)""\s*src=""([^""]+)""", 0, RegexReturn.ListByMatch, EDP.ReturnValue) Friend ReadOnly Regex_Photo_PornHub_SinglePhoto2 As RParams = RParams.DMS("image:src"" content=""([^""]+)""", 1, EDP.ReturnValue) + Friend ReadOnly Regex_Photo_PornHub_SinglePhoto3 As RParams = RParams.DMS("div id=""photoImageSection"".*?a href=""[^""\d]+\d+""\>[\r\n\s\t]*\ Friend Class SiteSettings : Inherits SiteSettingsBase +#Region "Categories" + Private Const CAT_YTDLP As String = "yt-dlp support" +#End Region #Region "Declarations" +#Region "Auth" + + Friend Property CookiesUse As PropertyValue +#End Region Friend Property DownloadUHD As PropertyValue @@ -34,12 +41,19 @@ Namespace API.PornHub Friend Overrides Property DownloadText As PropertyValue Friend Overrides Property DownloadTextPosts As PropertyValue Friend Overrides Property DownloadTextSpecialFolder As PropertyValue +#Region "YT-DLP" + + Friend Property YTDLP_Use As PropertyValue + + Friend Property YTDLP_UseCookies As PropertyValue +#End Region #End Region #Region "Initializer" Friend Sub New(ByVal AccName As String, ByVal Temp As Boolean) MyBase.New("PornHub", "pornhub.com", AccName, Temp, My.Resources.SiteResources.PornHubIcon_16, My.Resources.SiteResources.PornHubPic_16) With Responser : .CurlSslNoRevoke = True : .CurlInsecure = True : End With + CookiesUse = New PropertyValue(False) DownloadUHD = New PropertyValue(False) DownloadUploaded = New PropertyValue(True) DownloadTagged = New PropertyValue(False) @@ -49,10 +63,14 @@ Namespace API.PornHub DownloadGifs = New PropertyValue(CInt(CheckState.Indeterminate), GetType(Integer)) SavedPostsUserName = New PropertyValue(String.Empty, GetType(String)) + YTDLP_Use = New PropertyValue(True) + YTDLP_UseCookies = New PropertyValue(False) + _SubscriptionsAllowed = True UrlPatternUser = "https://www.pornhub.com/{0}/{1}" UserRegex = RParams.DMS("pornhub.com/(model|user[s]?|pornstar|channel[s]?)/([^/]+).*?", 0, RegexReturn.ListByMatch) ImageVideoContains = "pornhub.com" + UseNetscapeCookies = True End Sub #End Region #Region "GetInstance" diff --git a/SCrawler/API/PornHub/UserData.vb b/SCrawler/API/PornHub/UserData.vb index b6c2eb6..62f700e 100644 --- a/SCrawler/API/PornHub/UserData.vb +++ b/SCrawler/API/PornHub/UserData.vb @@ -6,13 +6,16 @@ ' ' This program is distributed in the hope that it will be useful, ' but WITHOUT ANY WARRANTY +Imports System.IO Imports System.Security.Policy Imports System.Threading Imports PersonalUtilities.Functions.RegularExpressions Imports PersonalUtilities.Functions.XML +Imports PersonalUtilities.Tools Imports PersonalUtilities.Tools.Web.Clients Imports PersonalUtilities.Tools.Web.Documents.JSON Imports SCrawler.API.Base +Imports SCrawler.API.Base.YTDLP Imports SCrawler.API.YouTube.Objects Imports UTypes = SCrawler.API.Base.UserMedia.Types Namespace API.PornHub @@ -53,11 +56,12 @@ Namespace API.PornHub Friend Title As String Friend UserRef As String Friend Type As VideoTypes - Friend Function ToUserMedia(Optional ByVal SpecialFolder As String = Nothing) As UserMedia - Return New UserMedia(URL, UTypes.VideoPre) With { - .File = If(Title.IsEmptyString, .File, New SFile($"{Title}.mp4")), + Friend Function ToUserMedia(Optional ByVal SpecialFolder As String = Nothing, Optional ByVal UseYTDLP As Boolean = False) As UserMedia + Return New UserMedia(URL, If(UseYTDLP, UTypes.Video, UTypes.VideoPre)) With { + .File = If(Title.IsEmptyString, .File, New SFile($"{TitleHtmlConverter(Title)}.mp4")), .Post = ID, - .SpecialFolder = SpecialFolder + .SpecialFolder = SpecialFolder, + .Type = If(UseYTDLP And Not Title.IsEmptyString, UTypes.Video, UTypes.VideoPre) } End Function Private Function CreateFromArray(ByVal ParamsArray() As String) As Object Implements IRegExCreator.CreateFromArray @@ -178,6 +182,7 @@ Namespace API.PornHub Return DirectCast(HOST.Source, SiteSettings) End Get End Property + Private MyCache As CacheKeeper = Nothing #End Region #Region "Initializer" Friend Sub New() @@ -264,8 +269,10 @@ Namespace API.PornHub Try PlaylistToken = String.Empty Responser.ResetStatus() + If Not CBool(MySettings.CookiesUse.Value) Then Responser.Cookies.Clear() _PageVideosRepeat = 0 SessionPosts.Clear() + UseInternalDownloadFileFunction = MySettings.YTDLP_Use.Value If IsSavedPosts Then PersonType = PersonTypeUser @@ -304,7 +311,7 @@ Namespace API.PornHub End If End If - If _TempMediaList.Count > 0 Then + If Not CBool(MySettings.YTDLP_Use.Value) And _TempMediaList.Count > 0 Then _TempMediaList.RemoveAll(Function(m) Not m.Type = UTypes.m3u8 And Not m.Type = UTypes.VideoPre) If limit > 0 And _TempMediaList.Count > limit Then _TempMediaList.ListAddList(_TempMediaList.ListTake(-1, limit), LAP.ClearBeforeAdd) End If @@ -341,6 +348,7 @@ Namespace API.PornHub ProgressPre.ChangeMax(1) Try Dim specFolder$ = String.Empty + Dim useYtdlp As Boolean = MySettings.YTDLP_Use.Value Dim tryNextPage As Boolean = False Dim limit% = If(DownloadTopCount, -1) Dim cBefore% = _TempMediaList.Count @@ -408,7 +416,7 @@ Namespace API.PornHub _TempPostsList.Add(uv.ID) newPostsFound = True Return False - ElseIf SessionPosts.Count > 0 AndAlso SessionPosts.Contains(uv.id) Then + ElseIf SessionPosts.Count > 0 AndAlso SessionPosts.Contains(uv.ID) Then prevPostsFound = True Return True Else @@ -420,7 +428,7 @@ Namespace API.PornHub 'Debug.WriteLineIf(l.Count > 0, l.Select(Function(ll) ll.Title).ListToString(vbNewLine)) If prevPostsFound And Not pageRepeatSet And Not newPostsFound Then pageRepeatSet = True : _PageVideosRepeat += 1 If prevPostsFound And newPostsFound And pageRepeatSet Then _PageVideosRepeat -= 1 - If l.Count > 0 Then _TempMediaList.ListAddList(l.Select(Function(uv) uv.ToUserMedia(specFolder))) + If l.Count > 0 Then _TempMediaList.ListAddList(l.Select(Function(uv) uv.ToUserMedia(specFolder, useYtdlp))) SessionPosts.ListAddList(newLastPageIDs, LNC) newLastPageIDs.Clear() @@ -560,6 +568,7 @@ Namespace API.PornHub If .ListExists(3) Then url = .Item(2).IfNullOrEmpty(.Item(1)).StringTrim End With If url.IsEmptyString Then url = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto2) + If url.IsEmptyString Then url = RegexReplace(r, Regex_Photo_PornHub_SinglePhoto3) Return url End Function Private Overloads Function DownloadUserPhotos_PornHub(ByVal Page As Integer, ByVal AlbumID As String, ByVal AlbumName As String, @@ -677,6 +686,7 @@ Namespace API.PornHub Optional ByRef Data As IYouTubeMediaContainer = Nothing) Const ERR_NEW_URL$ = "ERR_NEW_URL" Dim URL$ = String.Empty + Dim useYtdlp As Boolean = MySettings.YTDLP_Use.Value Try If _TempMediaList.Count > 0 AndAlso _TempMediaList.Exists(Function(tm) tm.Type = UTypes.VideoPre) Then Dim m As UserMedia @@ -693,13 +703,18 @@ Namespace API.PornHub For isCurl = 1 To 0 Step -1 If CBool(isCurl) Then r = Responser.Curl(URL) Else r = Responser.GetResponse(URL) If Not r.IsEmptyString Then - NewUrl = CreateVideoURL(r) - If NewUrl.IsEmptyString Then + NewUrl = String.Empty + If Not useYtdlp Then NewUrl = CreateVideoURL(r) + If Not useYtdlp And NewUrl.IsEmptyString Then Throw New Exception With {.HelpLink = ERR_NEW_URL} Else - m.URL = NewUrl - m.Type = UTypes.m3u8 - If CreateFileName Then + If useYtdlp Then + m.Type = UTypes.Video + Else + m.URL = NewUrl + m.Type = UTypes.m3u8 + End If + If CreateFileName Or useYtdlp Then tmpName = GetVideoTitle(r, URL) If Not tmpName.IsEmptyString Then If Not Data Is Nothing Then Data.Title = tmpName @@ -799,6 +814,7 @@ Namespace API.PornHub #Region "ReparseMissing" Protected Overrides Sub ReparseMissing(ByVal Token As CancellationToken) Dim rList As New List(Of Integer) + If CBool(MySettings.YTDLP_Use.Value) Then UseInternalDownloadFileFunction = True Try If ContentMissingExists Then Dim m As UserMedia @@ -810,13 +826,20 @@ Namespace API.PornHub m = _ContentList(i) If m.State = UserMedia.States.Missing AndAlso Not m.URL_BASE.IsEmptyString Then ThrowAny(Token) - r = Responser.Curl(m.URL_BASE,, eCurl) - If Not r.IsEmptyString Then - Dim NewUrl$ = CreateVideoURL(r) - If Not NewUrl.IsEmptyString Then - m.URL = NewUrl - _TempMediaList.ListAddValue(m, LNC) - rList.Add(i) + If CBool(MySettings.YTDLP_Use.Value) Then + m.URL = m.URL_BASE + m.Type = UTypes.Video + _TempMediaList.ListAddValue(m, LNC) + rList.Add(i) + Else + r = Responser.Curl(m.URL_BASE,, eCurl) + If Not r.IsEmptyString Then + Dim NewUrl$ = CreateVideoURL(r) + If Not NewUrl.IsEmptyString Then + m.URL = NewUrl + _TempMediaList.ListAddValue(m, LNC) + rList.Add(i) + End If End If End If End If @@ -857,6 +880,34 @@ Namespace API.PornHub Return Nothing End Try End Function + Protected Overrides Function ValidateDownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByRef Interrupt As Boolean) As Boolean + Return Media.Type = UTypes.Video + End Function + Protected Overrides Function DownloadFile(ByVal URL As String, ByVal Media As UserMedia, ByVal DestinationFile As SFile, ByVal Token As CancellationToken) As SFile + Dim cc As CacheKeeper + If IsSingleObjectDownload Then + cc = Settings.Cache + Else + If MyCache Is Nothing Then MyCache = CreateCache() + cc = MyCache + End If + cc.Validate() + Dim cmd$ = YTDLP_CreateCommand(URL, cc) + Dim f As SFile = Nothing + If Not cmd.IsEmptyString Then + Dim path As SFile = cc.NewPath + path.Exists() + cmd &= $" -o ""{DestinationFile}""" + Using b As New YTDLPBatch(Token) + b.Encoding = Settings.CMDEncoding + b.CreateFileExchanger(cc) + If IsSingleObjectDownload Then b.SetProgress(Progress) + b.Execute(cmd) + End Using + If DestinationFile.Exists Then f = DestinationFile + End If + Return f + End Function #End Region #Region "CreateVideoURL" Private Function CreateVideoURL(ByVal r As String) As String @@ -950,8 +1001,43 @@ Namespace API.PornHub End Try End Function #End Region +#Region "YT-DLP support" + Private Function YTDLP_CreateCommand(ByVal URL As String, ByRef Cache As CacheKeeper) As String + Dim path As SFile = Cache.NewPath + Dim c$ = If(CBool(MySettings.YTDLP_UseCookies.Value) And MySettings.CookiesNetscapeFile.Exists, + $" --no-cookies-from-browser --cookies ""{MySettings.CookiesNetscapeFile}""", String.Empty) + Dim cmd$ = $"""{Settings.YtdlpFile}"" --write-info-json --skip-download{c} {URL} -o ""{path.PathWithSeparator}file""" + path.Exists() + Using ytdlp As New YTDLPBatch(TokenPersonal,, path) : ytdlp.Encoding = Settings.CMDEncoding : ytdlp.Execute(cmd) : End Using + cmd = String.Empty + Dim f As SFile = SFile.GetFiles(path, "*.json",, EDP.ReturnValue).FirstOrDefault + If f.Exists Then + Using j As EContainer = JsonDocument.Parse(f.GetText(EDP.ReturnValue), EDP.ReturnValue) + If j.ListExists Then + With j("formats") + If .ListExists Then + Dim formats As List(Of Sizes) = Nothing + Try : formats = .Select(Function(ee) New Sizes(CInt(ee.Value("height")), ee.Value("format_id"))).ListIfNothing : Catch : End Try + If formats.ListExists Then + formats.RemoveAll(Function(ff) ff.Value <= 0 Or (Not CBool(MySettings.DownloadUHD.Value) AndAlso ff.Value > 1080) Or ff.Data.IsEmptyString) + If formats.Count > 0 Then + formats.Sort() + cmd = $"-f {formats(0).Data} " + formats.Clear() + End If + End If + End If + End With + End If + End Using + End If + Return $"""{Settings.YtdlpFile}"" {cmd}{URL}" + End Function +#End Region #Region "DownloadSingleObject" Protected Overrides Sub DownloadSingleObject_GetPosts(ByVal Data As IYouTubeMediaContainer, ByVal Token As CancellationToken) + If CBool(MySettings.YTDLP_Use.Value) Then UseInternalDownloadFileFunction = True + If Not CBool(MySettings.CookiesUse.Value) Then Responser.Cookies.Clear() _TempMediaList.Add(New UserMedia(Data.URL, UTypes.VideoPre)) ReparseVideo(Token, True, Data) End Sub @@ -973,7 +1059,7 @@ Namespace API.PornHub #End Region #Region "IDisposable Support" Protected Overrides Sub Dispose(ByVal disposing As Boolean) - If Not disposedValue And disposing Then SessionPosts.Clear() + If Not disposedValue And disposing Then SessionPosts.Clear() : MyCache.DisposeIfReady MyBase.Dispose(disposing) End Sub #End Region diff --git a/SCrawler/API/Reddit/SiteSettings.vb b/SCrawler/API/Reddit/SiteSettings.vb index 905da5e..0162cb6 100644 --- a/SCrawler/API/Reddit/SiteSettings.vb +++ b/SCrawler/API/Reddit/SiteSettings.vb @@ -23,24 +23,14 @@ Namespace API.Reddit Friend Class SiteSettings : Inherits SiteSettingsBase : Implements DownDetector.IDownDetector #Region "Declarations" #Region "Authorization" - Private Const ApiClientID_Default As String = "dYctRA-SIJxyykHe27lGZg" - Private Const ApiClientSecret_Default As String = "_5D6KzplRPDga-es1YlpzDIe9hiFlg" Friend ReadOnly Property AuthUserName As PropertyValue Friend ReadOnly Property AuthPassword As PropertyValue Friend ReadOnly Property ApiClientID As PropertyValue - Private Function ApiClientID_SetDefault() As Boolean - ApiClientID.Value = ApiClientID_Default - Return True - End Function Friend ReadOnly Property ApiClientSecret As PropertyValue - Private Function ApiClientSecret_SetDefault() As Boolean - ApiClientSecret.Value = ApiClientSecret_Default - Return True - End Function 0 Then Dim t As EContainer For Each n As EContainer In node - t = n.ItemF({"s", "u"}) - If Not t Is Nothing AndAlso Not t.Value.IsEmptyString Then - _TempMediaList.ListAddValue(MediaFromData(UTypes.Picture, t.Value, PostID, PostDate, _UserID,, PostText), LNC) - added = True - If FirstOnly Then Exit For - End If + isGif = False + Select Case n.Value("e") + Case "AnimatedImage" : isGif = True : t = Nothing + Case Else : t = n.ItemF({"s", "u"}) + End Select + For i = 0 To IIf(isGif, 1, 0) + If isGif Then t = If(i = 0, n.ItemF({"s", "gif"}), n.ItemF({"s", "mp4"})) + If Not t Is Nothing AndAlso Not t.Value.IsEmptyString Then + m = MediaFromData(IIf(i = 0, IIf(isGif, UTypes.GIF, UTypes.Picture), UTypes.GIF), t.Value, PostID, PostDate, _UserID,, PostText) + If isGif And i = 1 Then + m.File = CreateFileFromUrl(t.Value) + m.File.Extension = "mp4" + m.URL = t.Value + End If + _TempMediaList.ListAddValue(m, LNC) + added = True + If FirstOnly Then Exit For + End If + Next Next End If Return added diff --git a/SCrawler/API/ThisVid/UserData.vb b/SCrawler/API/ThisVid/UserData.vb index 33c7daa..733d1c4 100644 --- a/SCrawler/API/ThisVid/UserData.vb +++ b/SCrawler/API/ThisVid/UserData.vb @@ -179,7 +179,7 @@ Namespace API.ThisVid If Not IsSavedPosts Then Dim r$ = Responser.GetResponse($"https://thisvid.com/members/{ID}/") If Not r.IsEmptyString Then - Dim rr As New RParams("", Nothing, 1, EDP.ReturnValue) + Dim rr As RParams = RParams.DMS("", 1, EDP.ReturnValue) Dim __getValue As Func(Of String, Boolean, String) = Function(ByVal member As String, ByVal appendMember As Boolean) As String rr.Pattern = String.Format(ProfileDataPattern, member) Dim v$ = CStr(RegexReplace(r, rr)).StringTrim @@ -190,13 +190,14 @@ Namespace API.ThisVid If Not UserSiteName.IsEmptyString And FriendlyName.IsEmptyString Then FriendlyName = UserSiteName : _ForceSaveUserData = True Dim descr$ = String.Empty descr.StringAppendLine(__getValue("Birth date", True)) - descr.StringAppendLine(__getValue("Country", True)) - descr.StringAppendLine(__getValue("City", True)) + descr.StringAppendLine(__getValue("Location", True)) + 'descr.StringAppendLine(__getValue("City", True)) descr.StringAppendLine(__getValue("Gender", True)) descr.StringAppendLine(__getValue("Orientation", True)) descr.StringAppendLine(__getValue("Relationship status", True)) descr.StringAppendLine(__getValue("Favourite category", True)) descr.StringAppendLine(__getValue("My interests", True)) + descr.StringAppendLine(__getValue("Favorite niche", True)) rr.Pattern = DescriptionPattern descr.StringAppendLine(CStr(RegexReplace(r, rr)).StringTrim) UserDescriptionUpdate(descr) diff --git a/SCrawler/API/Twitter/EditorExchangeOptions.vb b/SCrawler/API/Twitter/EditorExchangeOptions.vb index f59763b..7d50225 100644 --- a/SCrawler/API/Twitter/EditorExchangeOptions.vb +++ b/SCrawler/API/Twitter/EditorExchangeOptions.vb @@ -65,14 +65,6 @@ Namespace API.Twitter MediaModelAllowNonUserTweets = s.MediaModelAllowNonUserTweets.Value MySettings = s End Sub - Friend Sub New(ByVal s As Mastodon.SiteSettings) - MyBase.New(s) - GifsDownload = s.GifsDownload.Value - GifsSpecialFolder = s.GifsSpecialFolder.Value - GifsPrefix = s.GifsPrefix.Value - UseMD5Comparison = s.UseMD5Comparison.Value - MySettings = s - End Sub Friend Sub New(ByVal u As UserData) MyBase.New(u) GifsDownload = u.GifsDownload @@ -82,16 +74,14 @@ Namespace API.Twitter RemoveExistingDuplicates = u.RemoveExistingDuplicates MediaModelAllowNonUserTweets = u.MediaModelAllowNonUserTweets LargeProfile = u.LargeProfile - If Not TypeOf u Is Mastodon.UserData Then - DownloadModelForceApply = u.DownloadModelForceApply - DownloadBroadcasts = u.DownloadBroadcasts - Dim dm As DModels() = EnumExtract(Of DModels)(u.DownloadModel) - If dm.ListExists Then - DownloadModelMedia = dm.Contains(DModels.Media) - DownloadModelProfile = dm.Contains(DModels.Profile) - DownloadModelSearch = dm.Contains(DModels.Search) - DownloadModelLikes = dm.Contains(DModels.Likes) - End If + DownloadModelForceApply = u.DownloadModelForceApply + DownloadBroadcasts = u.DownloadBroadcasts + Dim dm As DModels() = EnumExtract(Of DModels)(u.DownloadModel) + If dm.ListExists Then + DownloadModelMedia = dm.Contains(DModels.Media) + DownloadModelProfile = dm.Contains(DModels.Profile) + DownloadModelSearch = dm.Contains(DModels.Search) + DownloadModelLikes = dm.Contains(DModels.Likes) End If MySettings = u.HOST.Source End Sub diff --git a/SCrawler/API/Twitter/UserData.vb b/SCrawler/API/Twitter/UserData.vb index 1b7241d..fcf3508 100644 --- a/SCrawler/API/Twitter/UserData.vb +++ b/SCrawler/API/Twitter/UserData.vb @@ -695,7 +695,7 @@ nextpIndx: End Sub Private Sub DownloadData_SavedPosts(ByVal Token As CancellationToken) Try - Dim f As SFile = GetDataFromGalleryDL("https://x.com/i/bookmarks", Settings.Cache, True, Token) + Dim f As SFile = GetDataFromGalleryDL("https://x.com/i/history", Settings.Cache, True, Token) Dim files As List(Of SFile) = SFile.GetFiles(f, "*.txt") If files.ListExists Then GDLResetFileNameProvider(Math.Max(files.Count.ToString.Length, 3)) @@ -1108,7 +1108,7 @@ nextpIndx: If ID.IsEmptyString Then process = False Else - command &= $"https://twitter.com/intent/user?user_id={ID}" + command &= $"https://x.com/intent/user?user_id={ID}" currentModel = DownloadModels.Media process = True End If diff --git a/SCrawler/API/Xhamster/UserData.vb b/SCrawler/API/Xhamster/UserData.vb index 1ce8b41..e51497d 100644 --- a/SCrawler/API/Xhamster/UserData.vb +++ b/SCrawler/API/Xhamster/UserData.vb @@ -182,6 +182,14 @@ Namespace API.Xhamster MyCache.DisposeIfReady(False) MyCache = Nothing End Sub + Private Function GetCache() As CacheKeeper + If IsSingleObjectDownload Then + Return Settings.Cache + Else + If MyCache Is Nothing Then MyCache = CreateCache() + Return MyCache + End If + End Function #End Region #Region "Initializer" Friend Sub New() @@ -744,15 +752,11 @@ Namespace API.Xhamster End Function #End Region #Region "yt-dlp support" +#Disable Warning IDE0060 Private Function YTDLPGetInfo(ByVal URL As String, ByVal n As Integer) As SFile +#Enable Warning Try - Dim cc As CacheKeeper - If IsSingleObjectDownload Then - cc = Settings.Cache - Else - If MyCache Is Nothing Then MyCache = CreateCache() - cc = MyCache - End If + Dim cc As CacheKeeper = GetCache() cc.Validate() Dim path As SFile = cc.NewPath Dim c$ = If(MySettings.CookiesNetscapeFile.Exists, $" --no-cookies-from-browser --cookies ""{MySettings.CookiesNetscapeFile}""", String.Empty) @@ -802,7 +806,12 @@ Namespace API.Xhamster DestinationFile.Extension = "mp4" Dim c$ = If(MySettings.CookiesNetscapeFile.Exists, $" --no-cookies-from-browser --cookies ""{MySettings.CookiesNetscapeFile}""", String.Empty) Dim cmd$ = $"""{Settings.YtdlpFile}"" --format {DirectCast(Media.Object, XMMediaInfo).FormatID}{c} {Media.URL_BASE} -o ""{DestinationFile}""" - Using ytdlp As New YTDLP.YTDLPBatch(TokenPersonal,, DestinationFile) : ytdlp.Encoding = Settings.CMDEncoding : ytdlp.Execute(cmd) : End Using + Using ytdlp As New YTDLP.YTDLPBatch(Token,, DestinationFile) + ytdlp.Encoding = Settings.CMDEncoding + ytdlp.CreateFileExchanger(GetCache) + If IsSingleObjectDownload Then ytdlp.SetProgress(Progress) + ytdlp.Execute(cmd) + End Using Return DestinationFile End Function Protected Overrides Function DownloadContentDefault_ConvertWebp(ByVal m As UserMedia, ByVal Process As Boolean) As SFile diff --git a/SCrawler/Content/Icons/SiteIcons/MastodonIcon_48.ico b/SCrawler/Content/Icons/SiteIcons/MastodonIcon_48.ico deleted file mode 100644 index b09a98b..0000000 Binary files a/SCrawler/Content/Icons/SiteIcons/MastodonIcon_48.ico and /dev/null differ diff --git a/SCrawler/Content/Pictures/SitePictures/MastodonPic_48.png b/SCrawler/Content/Pictures/SitePictures/MastodonPic_48.png deleted file mode 100644 index 259676c..0000000 Binary files a/SCrawler/Content/Pictures/SitePictures/MastodonPic_48.png and /dev/null differ diff --git a/SCrawler/Download/Automation/AutoDownloader.vb b/SCrawler/Download/Automation/AutoDownloader.vb index 455b390..d7c9bce 100644 --- a/SCrawler/Download/Automation/AutoDownloader.vb +++ b/SCrawler/Download/Automation/AutoDownloader.vb @@ -183,8 +183,6 @@ Namespace DownloadObjects End Class #End Region #Region "XML Names" - 'TODELETE: AutoDownloader.Modes - Private Const Name_Mode As String = "Mode" Private Const Name_Enabled As String = "Enabled" Private Const Name_IsManual As String = "IsManual" Private Const Name_Timer As String = "Timer" @@ -374,15 +372,7 @@ Namespace DownloadObjects Me.New Initialization = True Import(x) -#Disable Warning BC40008 - If x.Contains(Name_Mode) Then - Dim g% = x.Value(Name_Mode).FromXML(Of Integer)(0) - If g = 4 Then GroupsOnly = True - Enabled = g - Else - Enabled = x.Value(Name_Enabled).FromXML(Of Boolean)(False) - End If -#Enable Warning + Enabled = x.Value(Name_Enabled).FromXML(Of Boolean)(False) If Name.IsEmptyString Then Name = "Default" IsManual = x.Value(Name_IsManual).FromXML(Of Boolean)(False) diff --git a/SCrawler/Download/Feed/DownloadFeedForm.vb b/SCrawler/Download/Feed/DownloadFeedForm.vb index 57e995c..1f4a80f 100644 --- a/SCrawler/Download/Feed/DownloadFeedForm.vb +++ b/SCrawler/Download/Feed/DownloadFeedForm.vb @@ -385,6 +385,7 @@ Namespace DownloadObjects End Function Private Sub Feed_FeedAdded(ByVal Source As FeedSpecialCollection, ByVal Feed As FeedSpecial) AddNewFeedItem(BTT_LOAD_SPEC, Feed, My.Resources.RSSPic_512, AddressOf Feed_SPEC_LOAD, True) + AddNewFeedItem(BTT_LOAD_T, Feed, My.Resources.RSSPic_512, AddressOf Feed_SPEC_LOAD, True) AddNewFeedItem(BTT_FEED_ADD_SPEC, Feed, My.Resources.RSSPic_512, AddressOf Feed_SPEC_ADD, True) AddNewFeedItem(BTT_FEED_ADD_T, Feed, My.Resources.RSSPic_512, AddressOf Feed_SPEC_ADD, True) AddNewFeedItem(BTT_FEED_ADD_SPEC_REMOVE, Feed, My.Resources.RSSPic_512, AddressOf Feed_SPEC_ADD_REMOVE, True) @@ -394,6 +395,7 @@ Namespace DownloadObjects End Sub Private Overloads Sub Feed_FeedRemoved(ByVal Source As FeedSpecialCollection, ByVal Feed As FeedSpecial) Feed_FeedRemoved(BTT_LOAD_SPEC, Feed) + Feed_FeedRemoved(BTT_LOAD_T, Feed) Feed_FeedRemoved(BTT_FEED_ADD_SPEC, Feed) Feed_FeedRemoved(BTT_FEED_ADD_T, Feed) Feed_FeedRemoved(BTT_FEED_REMOVE_SPEC, Feed) diff --git a/SCrawler/Download/MissingPostsForm.vb b/SCrawler/Download/MissingPostsForm.vb index 5d84ee0..636fe17 100644 --- a/SCrawler/Download/MissingPostsForm.vb +++ b/SCrawler/Download/MissingPostsForm.vb @@ -66,7 +66,12 @@ Namespace DownloadObjects MUsers.Clear() End Sub Private Sub MissingPostsForm_KeyDown(sender As Object, e As KeyEventArgs) Handles Me.KeyDown - If e.KeyCode = Keys.F1 Then ShowPostInformation() : e.Handled = True + If e.KeyCode = Keys.F1 Then + ShowPostInformation() + e.Handled = True + ElseIf e.KeyCode = Keys.Escape Then + Close() + End If End Sub #End Region #Region "RefillList" diff --git a/SCrawler/Download/STDownloader/VideoDownloaderForm.vb b/SCrawler/Download/STDownloader/VideoDownloaderForm.vb index b29efb7..6dc59e7 100644 --- a/SCrawler/Download/STDownloader/VideoDownloaderForm.vb +++ b/SCrawler/Download/STDownloader/VideoDownloaderForm.vb @@ -202,8 +202,6 @@ Namespace DownloadObjects.STDownloader If Not URL.IsEmptyString Then If URL.Contains("gfycat") Then Return API.Gfycat.Envir.GetSingleMediaInstance(URL, OutputFile) - ElseIf URL.Contains("imgur.com") Then - Return API.Imgur.Envir.GetSingleMediaInstance(URL, OutputFile) Else For i% = 0 To Settings.Plugins.Count - 1 With Settings.Plugins(i).Settings diff --git a/SCrawler/Download/TDownloader.vb b/SCrawler/Download/TDownloader.vb index ac2151e..ebc3519 100644 --- a/SCrawler/Download/TDownloader.vb +++ b/SCrawler/Download/TDownloader.vb @@ -311,6 +311,7 @@ Namespace DownloadObjects End Sub #End Region Friend ReadOnly Property Downloaded As List(Of IUserData) + Private ReadOnly MissingPostsUsers As List(Of String) Private ReadOnly NProv As IFormatProvider #End Region #Region "Working, Count" @@ -479,6 +480,7 @@ Namespace DownloadObjects Friend Sub New() Files = New List(Of UserMediaD) Downloaded = New List(Of IUserData) + MissingPostsUsers = New List(Of String) NProv = New ANumbers With {.FormatOptions = ANumbers.Options.GroupIntegral} Pool = New List(Of Job) End Sub @@ -562,6 +564,7 @@ Namespace DownloadObjects MainProgress.Visible = True If Not AutoDownloaderWorking AndAlso InfoForm.ReadyToOpen Then InfoForm.Show() : MainFrameObj.Focus() MissingPostsDetected = False + MissingPostsUsers.Clear() Session += 1 CheckerThread = New Thread(New ThreadStart(AddressOf JobsChecker)) CheckerThread.SetApartmentState(ApartmentState.MTA) @@ -593,8 +596,15 @@ Namespace DownloadObjects If Pool.Count > 0 Then Pool.ForEach(Sub(p) If Not p.Progress Is Nothing Then p.Progress.Maximum = 0) ExecuteCommand(Settings.DownloadsCompleteCommand) UpdateJobsLabel() - If MissingPostsDetected And Settings.AddMissingToLog Then _ - MyMainLOG = "Some posts didn't download. You can see them in the 'Missing posts' form." + If MissingPostsDetected And Settings.AddMissingToLog Then + MyMainLOG = "Some posts didn't download. You can see them in the 'Missing posts' form." + If Settings.AddMissingToLog And Settings.AddMissingPostUsersToLog And MissingPostsUsers.Count > 0 Then + MissingPostsUsers.Sort() + With MissingPostsUsers.Distinct + If .ListExists Then MyMainLOG = "List of users whose posts are missing" : .ListForEach(Sub(mu, mi) MyMainLOG = mu, EDP.None) + End With + End If + End If Files.Sort() FilesChanged = Not fBefore = Files.Count RaiseEvent Downloading(False) @@ -715,7 +725,9 @@ Namespace DownloadObjects If i >= 0 Then If KeysSkipped.Count = 0 OrElse Not KeysSkipped.Contains(k) Then With _Job.Items(i) - If DirectCast(.Self, UserDataBase).ContentMissingExists Then MissingPostsDetected = True + With DirectCast(.Self, UserDataBase) + If .ContentMissingExists Then MissingPostsDetected = True : MissingPostsUsers.Add(.ToStringForLog) + End With RaiseEvent UserDownloadStateChanged(.Self, False) host = _Job.UserHost(.Self) host.AfterDownload(.Self, Download.Main) @@ -754,31 +766,36 @@ Namespace DownloadObjects End Sub #End Region #Region "Add" - Private Sub AddItem(ByVal Item As IUserData, ByVal _UpdateJobsLabel As Boolean, ByVal _IncludedInTheFeed As Boolean) - ReconfPool() - If Item.IsCollection Then - DirectCast(Item, API.UserDataBind).DownloadData(Nothing, _IncludedInTheFeed) - Else - If Not Contains(Item) Then - If Pool.Count > 0 Then - For i% = 0 To Pool.Count - 1 - If Pool(i).Add(Item, _IncludedInTheFeed) Then Exit For - Next + Private Function AddItem(ByVal Item As IUserData, ByVal _UpdateJobsLabel As Boolean, ByVal _IncludedInTheFeed As Boolean) As Boolean + Dim added As Boolean = False + If Not Item Is Nothing Then + ReconfPool() + If Item.IsCollection Then + DirectCast(Item, API.UserDataBind).DownloadData(Nothing, _IncludedInTheFeed) + Else + If Not Item.Disposed AndAlso Not Contains(Item) AndAlso + Not Item.HOST.Key.IsEmptyString AndAlso Not Item.HOST.Key = API.PathPlugin.PluginKey Then + If Pool.Count > 0 Then + For i% = 0 To Pool.Count - 1 + If Pool(i).Add(Item, _IncludedInTheFeed) Then added = True : Exit For + Next + End If + If _UpdateJobsLabel Then UpdateJobsLabel() End If - If _UpdateJobsLabel Then UpdateJobsLabel() End If End If - End Sub + Return added + End Function Friend Sub Add(ByVal Item As IUserData, ByVal _IncludedInTheFeed As Boolean) - AddItem(Item, True, _IncludedInTheFeed) - Start() + If AddItem(Item, True, _IncludedInTheFeed) Then Start() End Sub Friend Sub AddRange(ByVal _Items As IEnumerable(Of IUserData), ByVal _IncludedInTheFeed As Boolean) If _Items.ListExists Then - For i% = 0 To _Items.Count - 1 : AddItem(_Items(i), False, _IncludedInTheFeed) : Next + Dim added% = 0 + For i% = 0 To _Items.Count - 1 : added += AddItem(_Items(i), False, _IncludedInTheFeed).BoolToInteger : Next UpdateJobsLabel() + If added <> 0 Then Start() End If - Start() End Sub #End Region #Region "Contains, Remove" @@ -804,6 +821,7 @@ Namespace DownloadObjects Pool.ListClearDispose Files.Clear() Downloaded.Clear() + MissingPostsUsers.Clear() End If disposedValue = True End If diff --git a/SCrawler/Editors/GlobalSettingsForm.Designer.vb b/SCrawler/Editors/GlobalSettingsForm.Designer.vb index 8feff4b..7b76396 100644 --- a/SCrawler/Editors/GlobalSettingsForm.Designer.vb +++ b/SCrawler/Editors/GlobalSettingsForm.Designer.vb @@ -31,7 +31,6 @@ Namespace Editors Dim ActionButton3 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton4 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton5 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton6 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TP_FILE_NAME As System.Windows.Forms.TableLayoutPanel Dim TP_FILE_PATTERNS As System.Windows.Forms.TableLayoutPanel Dim LBL_DATE_POS As System.Windows.Forms.Label @@ -44,28 +43,30 @@ Namespace Editors Dim TP_CHANNELS As System.Windows.Forms.TableLayoutPanel Dim TAB_BEHAVIOR As System.Windows.Forms.TabPage Dim TP_BEHAVIOR As System.Windows.Forms.TableLayoutPanel + Dim ActionButton6 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton7 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton8 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TP_OPEN_INFO As System.Windows.Forms.TableLayoutPanel Dim TP_OPEN_PROGRESS As System.Windows.Forms.TableLayoutPanel Dim TP_BEHAVIOR_F6 As System.Windows.Forms.TableLayoutPanel Dim TAB_DOWN As System.Windows.Forms.TabPage Dim TP_DOWNLOADING As System.Windows.Forms.TableLayoutPanel + Dim ActionButton8 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton9 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton10 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TP_MISSING_DATA As System.Windows.Forms.TableLayoutPanel Dim TAB_FEED As System.Windows.Forms.TabPage Dim TP_FEED As System.Windows.Forms.TableLayoutPanel Dim TP_FEED_IMG_COUNT As System.Windows.Forms.TableLayoutPanel Dim TP_FEED_SES As System.Windows.Forms.TableLayoutPanel + Dim ActionButton10 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton11 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton12 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TP_FEED_SPEC_SEARCH As System.Windows.Forms.TableLayoutPanel Dim TP_FEED_1 As System.Windows.Forms.TableLayoutPanel Dim TP_FEED_2 As System.Windows.Forms.TableLayoutPanel + Dim TP_FEED_3 As System.Windows.Forms.TableLayoutPanel Dim TAB_NOTIFY As System.Windows.Forms.TabPage Dim TP_NOTIFY_MAIN As System.Windows.Forms.TableLayoutPanel Dim TP_ENVIR As System.Windows.Forms.TableLayoutPanel + Dim ActionButton12 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton13 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton14 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton15 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() @@ -79,26 +80,24 @@ Namespace Editors Dim ActionButton23 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton24 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton25 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton26 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TAB_STD As System.Windows.Forms.TabPage Dim TP_STD As System.Windows.Forms.TableLayoutPanel - Dim ActionButton27 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() + Dim ActionButton26 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ListColumn1 As PersonalUtilities.Forms.Controls.Base.ListColumn = New PersonalUtilities.Forms.Controls.Base.ListColumn() Dim ListColumn2 As PersonalUtilities.Forms.Controls.Base.ListColumn = New PersonalUtilities.Forms.Controls.Base.ListColumn() Dim TAB_DESIGN As System.Windows.Forms.TabPage Dim TP_DESIGN As System.Windows.Forms.TableLayoutPanel + Dim ActionButton27 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton28 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton29 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton30 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton31 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TP_HEADERS_DEF As System.Windows.Forms.TableLayoutPanel + Dim ActionButton31 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton32 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton33 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton34 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim ActionButton35 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() - Dim ActionButton36 As PersonalUtilities.Forms.Controls.Base.ActionButton = New PersonalUtilities.Forms.Controls.Base.ActionButton() Dim TAB_HEADERS As System.Windows.Forms.TabPage - Dim TP_FEED_3 As System.Windows.Forms.TableLayoutPanel Me.TXT_GLOBAL_PATH = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.TXT_IMAGE_LARGE = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.TXT_IMAGE_SMALL = New PersonalUtilities.Forms.Controls.TextBoxExtended() @@ -106,7 +105,6 @@ Namespace Editors Me.TXT_MAX_JOBS_USERS = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.TXT_MAX_JOBS_CHANNELS = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.CH_CHECK_VER_START = New System.Windows.Forms.CheckBox() - Me.TXT_IMGUR_CLIENT_ID = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.OPT_FILE_NAME_REPLACE = New System.Windows.Forms.RadioButton() Me.OPT_FILE_NAME_ADD_DATE = New System.Windows.Forms.RadioButton() Me.CH_FILE_NAME_CHANGE = New System.Windows.Forms.CheckBox() @@ -155,6 +153,8 @@ Namespace Editors Me.CH_FEED_SPEC_SEARCH_DEEP = New System.Windows.Forms.CheckBox() Me.CH_FEED_OPEN_CTRLF = New System.Windows.Forms.CheckBox() Me.CH_FEED_TEXT = New System.Windows.Forms.CheckBox() + Me.CH_FEED_TEXT_ALWAYS_MOVE = New System.Windows.Forms.CheckBox() + Me.CH_ADD_MISSING_USERS_TO_LOG = New System.Windows.Forms.CheckBox() Me.TXT_CHANNELS_ROWS = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.TXT_CHANNELS_COLUMNS = New PersonalUtilities.Forms.Controls.TextBoxExtended() Me.CH_DOWN_IMAGES_NATIVE = New System.Windows.Forms.CheckBox() @@ -214,7 +214,6 @@ Namespace Editors Me.TAB_MAIN = New System.Windows.Forms.TabControl() Me.TAB_ENVIR = New System.Windows.Forms.TabPage() Me.CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer() - Me.CH_FEED_TEXT_ALWAYS_MOVE = New System.Windows.Forms.CheckBox() TP_BASIS = New System.Windows.Forms.TableLayoutPanel() TP_IMAGES = New System.Windows.Forms.TableLayoutPanel() TP_FILE_NAME = New System.Windows.Forms.TableLayoutPanel() @@ -242,6 +241,7 @@ Namespace Editors TP_FEED_SPEC_SEARCH = New System.Windows.Forms.TableLayoutPanel() TP_FEED_1 = New System.Windows.Forms.TableLayoutPanel() TP_FEED_2 = New System.Windows.Forms.TableLayoutPanel() + TP_FEED_3 = New System.Windows.Forms.TableLayoutPanel() TAB_NOTIFY = New System.Windows.Forms.TabPage() TP_NOTIFY_MAIN = New System.Windows.Forms.TableLayoutPanel() TP_ENVIR = New System.Windows.Forms.TableLayoutPanel() @@ -251,7 +251,6 @@ Namespace Editors TP_DESIGN = New System.Windows.Forms.TableLayoutPanel() TP_HEADERS_DEF = New System.Windows.Forms.TableLayoutPanel() TAB_HEADERS = New System.Windows.Forms.TabPage() - TP_FEED_3 = New System.Windows.Forms.TableLayoutPanel() TP_BASIS.SuspendLayout() CType(Me.TXT_GLOBAL_PATH, System.ComponentModel.ISupportInitialize).BeginInit() TP_IMAGES.SuspendLayout() @@ -260,7 +259,6 @@ Namespace Editors CType(Me.TXT_COLLECTIONS_PATH, System.ComponentModel.ISupportInitialize).BeginInit() CType(Me.TXT_MAX_JOBS_USERS, System.ComponentModel.ISupportInitialize).BeginInit() CType(Me.TXT_MAX_JOBS_CHANNELS, System.ComponentModel.ISupportInitialize).BeginInit() - CType(Me.TXT_IMGUR_CLIENT_ID, System.ComponentModel.ISupportInitialize).BeginInit() TP_FILE_NAME.SuspendLayout() TP_FILE_PATTERNS.SuspendLayout() TP_CHANNELS_IMGS.SuspendLayout() @@ -298,6 +296,7 @@ Namespace Editors TP_FEED_SPEC_SEARCH.SuspendLayout() TP_FEED_1.SuspendLayout() TP_FEED_2.SuspendLayout() + TP_FEED_3.SuspendLayout() TAB_NOTIFY.SuspendLayout() TP_NOTIFY_MAIN.SuspendLayout() TP_ENVIR.SuspendLayout() @@ -326,7 +325,6 @@ Namespace Editors Me.TAB_ENVIR.SuspendLayout() Me.CONTAINER_MAIN.ContentPanel.SuspendLayout() Me.CONTAINER_MAIN.SuspendLayout() - TP_FEED_3.SuspendLayout() Me.SuspendLayout() ' 'TP_BASIS @@ -340,18 +338,16 @@ Namespace Editors TP_BASIS.Controls.Add(Me.TXT_MAX_JOBS_USERS, 0, 3) TP_BASIS.Controls.Add(Me.TXT_MAX_JOBS_CHANNELS, 0, 4) TP_BASIS.Controls.Add(Me.CH_CHECK_VER_START, 0, 5) - TP_BASIS.Controls.Add(Me.TXT_IMGUR_CLIENT_ID, 0, 6) TP_BASIS.Dock = System.Windows.Forms.DockStyle.Fill TP_BASIS.Location = New System.Drawing.Point(3, 3) TP_BASIS.Name = "TP_BASIS" - TP_BASIS.RowCount = 8 + TP_BASIS.RowCount = 7 TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!)) - TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 28.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!)) TP_BASIS.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 20.0!)) TP_BASIS.Size = New System.Drawing.Size(615, 393) @@ -486,18 +482,6 @@ Namespace Editors Me.CH_CHECK_VER_START.Text = "Check new version at start" Me.CH_CHECK_VER_START.UseVisualStyleBackColor = True ' - 'TXT_IMGUR_CLIENT_ID - ' - ActionButton6.BackgroundImage = CType(resources.GetObject("ActionButton6.BackgroundImage"), System.Drawing.Image) - ActionButton6.Name = "Clear" - Me.TXT_IMGUR_CLIENT_ID.Buttons.Add(ActionButton6) - Me.TXT_IMGUR_CLIENT_ID.CaptionText = "Imgur Client ID" - Me.TXT_IMGUR_CLIENT_ID.Dock = System.Windows.Forms.DockStyle.Fill - Me.TXT_IMGUR_CLIENT_ID.Location = New System.Drawing.Point(4, 175) - Me.TXT_IMGUR_CLIENT_ID.Name = "TXT_IMGUR_CLIENT_ID" - Me.TXT_IMGUR_CLIENT_ID.Size = New System.Drawing.Size(607, 22) - Me.TXT_IMGUR_CLIENT_ID.TabIndex = 7 - ' 'TP_FILE_NAME ' TP_FILE_NAME.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single] @@ -521,9 +505,9 @@ Namespace Editors ' Me.OPT_FILE_NAME_REPLACE.AutoSize = True Me.OPT_FILE_NAME_REPLACE.Dock = System.Windows.Forms.DockStyle.Fill - Me.OPT_FILE_NAME_REPLACE.Location = New System.Drawing.Point(210, 4) + Me.OPT_FILE_NAME_REPLACE.Location = New System.Drawing.Point(209, 4) Me.OPT_FILE_NAME_REPLACE.Name = "OPT_FILE_NAME_REPLACE" - Me.OPT_FILE_NAME_REPLACE.Size = New System.Drawing.Size(199, 22) + Me.OPT_FILE_NAME_REPLACE.Size = New System.Drawing.Size(198, 22) Me.OPT_FILE_NAME_REPLACE.TabIndex = 1 Me.OPT_FILE_NAME_REPLACE.TabStop = True Me.OPT_FILE_NAME_REPLACE.Text = "Replace file name by date" @@ -533,9 +517,9 @@ Namespace Editors ' Me.OPT_FILE_NAME_ADD_DATE.AutoSize = True Me.OPT_FILE_NAME_ADD_DATE.Dock = System.Windows.Forms.DockStyle.Fill - Me.OPT_FILE_NAME_ADD_DATE.Location = New System.Drawing.Point(416, 4) + Me.OPT_FILE_NAME_ADD_DATE.Location = New System.Drawing.Point(414, 4) Me.OPT_FILE_NAME_ADD_DATE.Name = "OPT_FILE_NAME_ADD_DATE" - Me.OPT_FILE_NAME_ADD_DATE.Size = New System.Drawing.Size(199, 22) + Me.OPT_FILE_NAME_ADD_DATE.Size = New System.Drawing.Size(201, 22) Me.OPT_FILE_NAME_ADD_DATE.TabIndex = 2 Me.OPT_FILE_NAME_ADD_DATE.TabStop = True Me.OPT_FILE_NAME_ADD_DATE.Text = "Add date/time to file name" @@ -547,7 +531,7 @@ Namespace Editors Me.CH_FILE_NAME_CHANGE.Dock = System.Windows.Forms.DockStyle.Fill Me.CH_FILE_NAME_CHANGE.Location = New System.Drawing.Point(4, 4) Me.CH_FILE_NAME_CHANGE.Name = "CH_FILE_NAME_CHANGE" - Me.CH_FILE_NAME_CHANGE.Size = New System.Drawing.Size(199, 22) + Me.CH_FILE_NAME_CHANGE.Size = New System.Drawing.Size(198, 22) Me.CH_FILE_NAME_CHANGE.TabIndex = 0 Me.CH_FILE_NAME_CHANGE.Text = "Change file names" Me.CH_FILE_NAME_CHANGE.UseVisualStyleBackColor = True @@ -773,7 +757,7 @@ Namespace Editors Me.CH_ADD_MISSING_TO_LOG.Dock = System.Windows.Forms.DockStyle.Fill Me.CH_ADD_MISSING_TO_LOG.Location = New System.Drawing.Point(4, 4) Me.CH_ADD_MISSING_TO_LOG.Name = "CH_ADD_MISSING_TO_LOG" - Me.CH_ADD_MISSING_TO_LOG.Size = New System.Drawing.Size(302, 17) + Me.CH_ADD_MISSING_TO_LOG.Size = New System.Drawing.Size(198, 17) Me.CH_ADD_MISSING_TO_LOG.TabIndex = 0 Me.CH_ADD_MISSING_TO_LOG.Text = "Add 'missing' information to log" TT_MAIN.SetToolTip(Me.CH_ADD_MISSING_TO_LOG, resources.GetString("CH_ADD_MISSING_TO_LOG.ToolTip")) @@ -783,9 +767,9 @@ Namespace Editors ' Me.CH_ADD_MISSING_ERROS_TO_LOG.AutoSize = True Me.CH_ADD_MISSING_ERROS_TO_LOG.Dock = System.Windows.Forms.DockStyle.Fill - Me.CH_ADD_MISSING_ERROS_TO_LOG.Location = New System.Drawing.Point(313, 4) + Me.CH_ADD_MISSING_ERROS_TO_LOG.Location = New System.Drawing.Point(209, 4) Me.CH_ADD_MISSING_ERROS_TO_LOG.Name = "CH_ADD_MISSING_ERROS_TO_LOG" - Me.CH_ADD_MISSING_ERROS_TO_LOG.Size = New System.Drawing.Size(302, 17) + Me.CH_ADD_MISSING_ERROS_TO_LOG.Size = New System.Drawing.Size(198, 17) Me.CH_ADD_MISSING_ERROS_TO_LOG.TabIndex = 1 Me.CH_ADD_MISSING_ERROS_TO_LOG.Text = "Add 'missing' errors to log" TT_MAIN.SetToolTip(Me.CH_ADD_MISSING_ERROS_TO_LOG, resources.GetString("CH_ADD_MISSING_ERROS_TO_LOG.ToolTip")) @@ -1071,9 +1055,9 @@ Namespace Editors ' Me.CH_DOWN_ALL_F6_NOTIFY.AutoSize = True Me.CH_DOWN_ALL_F6_NOTIFY.Dock = System.Windows.Forms.DockStyle.Fill - Me.CH_DOWN_ALL_F6_NOTIFY.Location = New System.Drawing.Point(210, 4) + Me.CH_DOWN_ALL_F6_NOTIFY.Location = New System.Drawing.Point(209, 4) Me.CH_DOWN_ALL_F6_NOTIFY.Name = "CH_DOWN_ALL_F6_NOTIFY" - Me.CH_DOWN_ALL_F6_NOTIFY.Size = New System.Drawing.Size(199, 17) + Me.CH_DOWN_ALL_F6_NOTIFY.Size = New System.Drawing.Size(198, 17) Me.CH_DOWN_ALL_F6_NOTIFY.TabIndex = 1 Me.CH_DOWN_ALL_F6_NOTIFY.Text = "'F6' confirmation" TT_MAIN.SetToolTip(Me.CH_DOWN_ALL_F6_NOTIFY, "Request confirmation to download all users when pressing 'F6'") @@ -1083,9 +1067,9 @@ Namespace Editors ' Me.CH_DOWN_ALL_NOTIFY.AutoSize = True Me.CH_DOWN_ALL_NOTIFY.Dock = System.Windows.Forms.DockStyle.Fill - Me.CH_DOWN_ALL_NOTIFY.Location = New System.Drawing.Point(416, 4) + Me.CH_DOWN_ALL_NOTIFY.Location = New System.Drawing.Point(414, 4) Me.CH_DOWN_ALL_NOTIFY.Name = "CH_DOWN_ALL_NOTIFY" - Me.CH_DOWN_ALL_NOTIFY.Size = New System.Drawing.Size(199, 17) + Me.CH_DOWN_ALL_NOTIFY.Size = New System.Drawing.Size(201, 17) Me.CH_DOWN_ALL_NOTIFY.TabIndex = 2 Me.CH_DOWN_ALL_NOTIFY.Text = "Confirmation of downloading all" TT_MAIN.SetToolTip(Me.CH_DOWN_ALL_NOTIFY, "Request confirmation to download all users anyway") @@ -1141,6 +1125,30 @@ Namespace Editors TT_MAIN.SetToolTip(Me.CH_FEED_TEXT, "If checked, downloaded text posts will be displayed in the feed (as an image)") Me.CH_FEED_TEXT.UseVisualStyleBackColor = True ' + 'CH_FEED_TEXT_ALWAYS_MOVE + ' + Me.CH_FEED_TEXT_ALWAYS_MOVE.AutoSize = True + Me.CH_FEED_TEXT_ALWAYS_MOVE.Dock = System.Windows.Forms.DockStyle.Fill + Me.CH_FEED_TEXT_ALWAYS_MOVE.Location = New System.Drawing.Point(312, 3) + Me.CH_FEED_TEXT_ALWAYS_MOVE.Name = "CH_FEED_TEXT_ALWAYS_MOVE" + Me.CH_FEED_TEXT_ALWAYS_MOVE.Size = New System.Drawing.Size(304, 19) + Me.CH_FEED_TEXT_ALWAYS_MOVE.TabIndex = 1 + Me.CH_FEED_TEXT_ALWAYS_MOVE.Text = "Auto-move linked text" + TT_MAIN.SetToolTip(Me.CH_FEED_TEXT_ALWAYS_MOVE, "Always move the linked text file along with the original file") + Me.CH_FEED_TEXT_ALWAYS_MOVE.UseVisualStyleBackColor = True + ' + 'CH_ADD_MISSING_USERS_TO_LOG + ' + Me.CH_ADD_MISSING_USERS_TO_LOG.AutoSize = True + Me.CH_ADD_MISSING_USERS_TO_LOG.Dock = System.Windows.Forms.DockStyle.Fill + Me.CH_ADD_MISSING_USERS_TO_LOG.Location = New System.Drawing.Point(414, 4) + Me.CH_ADD_MISSING_USERS_TO_LOG.Name = "CH_ADD_MISSING_USERS_TO_LOG" + Me.CH_ADD_MISSING_USERS_TO_LOG.Size = New System.Drawing.Size(201, 17) + Me.CH_ADD_MISSING_USERS_TO_LOG.TabIndex = 2 + Me.CH_ADD_MISSING_USERS_TO_LOG.Text = "Add 'missing' users to log" + TT_MAIN.SetToolTip(Me.CH_ADD_MISSING_USERS_TO_LOG, resources.GetString("CH_ADD_MISSING_USERS_TO_LOG.ToolTip")) + Me.CH_ADD_MISSING_USERS_TO_LOG.UseVisualStyleBackColor = True + ' 'TP_CHANNELS_IMGS ' TP_CHANNELS_IMGS.ColumnCount = 2 @@ -1357,11 +1365,11 @@ Namespace Editors 'TXT_FOLDER_CMD ' Me.TXT_FOLDER_CMD.AutoShowClearButton = True - ActionButton7.BackgroundImage = CType(resources.GetObject("ActionButton7.BackgroundImage"), System.Drawing.Image) - ActionButton7.Enabled = False - ActionButton7.Name = "Clear" - ActionButton7.Visible = False - Me.TXT_FOLDER_CMD.Buttons.Add(ActionButton7) + ActionButton6.BackgroundImage = CType(resources.GetObject("ActionButton6.BackgroundImage"), System.Drawing.Image) + ActionButton6.Enabled = False + ActionButton6.Name = "Clear" + ActionButton6.Visible = False + Me.TXT_FOLDER_CMD.Buttons.Add(ActionButton6) Me.TXT_FOLDER_CMD.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.CheckBox Me.TXT_FOLDER_CMD.CaptionText = "Folder cmd" Me.TXT_FOLDER_CMD.CaptionToolTipEnabled = True @@ -1400,11 +1408,11 @@ Namespace Editors 'TXT_CLOSE_SCRIPT ' Me.TXT_CLOSE_SCRIPT.AutoShowClearButton = True - ActionButton8.BackgroundImage = CType(resources.GetObject("ActionButton8.BackgroundImage"), System.Drawing.Image) - ActionButton8.Enabled = False - ActionButton8.Name = "Clear" - ActionButton8.Visible = False - Me.TXT_CLOSE_SCRIPT.Buttons.Add(ActionButton8) + ActionButton7.BackgroundImage = CType(resources.GetObject("ActionButton7.BackgroundImage"), System.Drawing.Image) + ActionButton7.Enabled = False + ActionButton7.Name = "Clear" + ActionButton7.Visible = False + Me.TXT_CLOSE_SCRIPT.Buttons.Add(ActionButton7) Me.TXT_CLOSE_SCRIPT.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.CheckBox Me.TXT_CLOSE_SCRIPT.CaptionText = "Close cmd" Me.TXT_CLOSE_SCRIPT.CaptionToolTipEnabled = True @@ -1512,7 +1520,7 @@ Namespace Editors Me.CH_DOWN_ALL_F6_USE.Dock = System.Windows.Forms.DockStyle.Fill Me.CH_DOWN_ALL_F6_USE.Location = New System.Drawing.Point(4, 4) Me.CH_DOWN_ALL_F6_USE.Name = "CH_DOWN_ALL_F6_USE" - Me.CH_DOWN_ALL_F6_USE.Size = New System.Drawing.Size(199, 17) + Me.CH_DOWN_ALL_F6_USE.Size = New System.Drawing.Size(198, 17) Me.CH_DOWN_ALL_F6_USE.TabIndex = 0 Me.CH_DOWN_ALL_F6_USE.Text = "Use 'F6' to download all users" Me.CH_DOWN_ALL_F6_USE.UseVisualStyleBackColor = True @@ -1588,12 +1596,12 @@ Namespace Editors ' 'TXT_SCRIPT ' + ActionButton8.BackgroundImage = CType(resources.GetObject("ActionButton8.BackgroundImage"), System.Drawing.Image) + ActionButton8.Name = "Open" ActionButton9.BackgroundImage = CType(resources.GetObject("ActionButton9.BackgroundImage"), System.Drawing.Image) - ActionButton9.Name = "Open" - ActionButton10.BackgroundImage = CType(resources.GetObject("ActionButton10.BackgroundImage"), System.Drawing.Image) - ActionButton10.Name = "Clear" + ActionButton9.Name = "Clear" + Me.TXT_SCRIPT.Buttons.Add(ActionButton8) Me.TXT_SCRIPT.Buttons.Add(ActionButton9) - Me.TXT_SCRIPT.Buttons.Add(ActionButton10) Me.TXT_SCRIPT.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.CheckBox Me.TXT_SCRIPT.CaptionText = "Script" Me.TXT_SCRIPT.CaptionToolTipEnabled = True @@ -1627,18 +1635,19 @@ Namespace Editors 'TP_MISSING_DATA ' TP_MISSING_DATA.CellBorderStyle = System.Windows.Forms.TableLayoutPanelCellBorderStyle.[Single] - TP_MISSING_DATA.ColumnCount = 2 - TP_MISSING_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) - TP_MISSING_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) + TP_MISSING_DATA.ColumnCount = 3 + TP_MISSING_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 33.33333!)) + TP_MISSING_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 33.33333!)) + TP_MISSING_DATA.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 33.33333!)) TP_MISSING_DATA.Controls.Add(Me.CH_ADD_MISSING_TO_LOG, 0, 0) TP_MISSING_DATA.Controls.Add(Me.CH_ADD_MISSING_ERROS_TO_LOG, 1, 0) + TP_MISSING_DATA.Controls.Add(Me.CH_ADD_MISSING_USERS_TO_LOG, 2, 0) TP_MISSING_DATA.Dock = System.Windows.Forms.DockStyle.Fill TP_MISSING_DATA.Location = New System.Drawing.Point(1, 199) TP_MISSING_DATA.Margin = New System.Windows.Forms.Padding(0) TP_MISSING_DATA.Name = "TP_MISSING_DATA" TP_MISSING_DATA.RowCount = 1 TP_MISSING_DATA.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!)) - TP_MISSING_DATA.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 24.0!)) TP_MISSING_DATA.Size = New System.Drawing.Size(619, 25) TP_MISSING_DATA.TabIndex = 7 ' @@ -1888,14 +1897,14 @@ Namespace Editors ' 'NUM_FEED_SES_CURR_LOAD_LAST ' + ActionButton10.BackgroundImage = CType(resources.GetObject("ActionButton10.BackgroundImage"), System.Drawing.Image) + ActionButton10.Name = "Refresh" + ActionButton10.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh ActionButton11.BackgroundImage = CType(resources.GetObject("ActionButton11.BackgroundImage"), System.Drawing.Image) - ActionButton11.Name = "Refresh" - ActionButton11.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton12.BackgroundImage = CType(resources.GetObject("ActionButton12.BackgroundImage"), System.Drawing.Image) - ActionButton12.Name = "Clear" - ActionButton12.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton11.Name = "Clear" + ActionButton11.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.NUM_FEED_SES_CURR_LOAD_LAST.Buttons.Add(ActionButton10) Me.NUM_FEED_SES_CURR_LOAD_LAST.Buttons.Add(ActionButton11) - Me.NUM_FEED_SES_CURR_LOAD_LAST.Buttons.Add(ActionButton12) Me.NUM_FEED_SES_CURR_LOAD_LAST.CaptionText = "Load last session" Me.NUM_FEED_SES_CURR_LOAD_LAST.CaptionToolTipEnabled = True Me.NUM_FEED_SES_CURR_LOAD_LAST.CaptionToolTipText = resources.GetString("NUM_FEED_SES_CURR_LOAD_LAST.CaptionToolTipText") @@ -2018,6 +2027,23 @@ Namespace Editors Me.CH_FEED_ADD_DATE.Text = "Add the date to the post title" Me.CH_FEED_ADD_DATE.UseVisualStyleBackColor = True ' + 'TP_FEED_3 + ' + TP_FEED_3.ColumnCount = 2 + TP_FEED_3.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) + TP_FEED_3.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) + TP_FEED_3.Controls.Add(Me.CH_FEED_TEXT, 0, 0) + TP_FEED_3.Controls.Add(Me.CH_FEED_TEXT_ALWAYS_MOVE, 1, 0) + TP_FEED_3.Dock = System.Windows.Forms.DockStyle.Fill + TP_FEED_3.Location = New System.Drawing.Point(1, 163) + TP_FEED_3.Margin = New System.Windows.Forms.Padding(0) + TP_FEED_3.Name = "TP_FEED_3" + TP_FEED_3.RowCount = 1 + TP_FEED_3.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!)) + TP_FEED_3.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 25.0!)) + TP_FEED_3.Size = New System.Drawing.Size(619, 25) + TP_FEED_3.TabIndex = 6 + ' 'TAB_NOTIFY ' TAB_NOTIFY.Controls.Add(TP_NOTIFY_MAIN) @@ -2083,19 +2109,19 @@ Namespace Editors ' 'TXT_YTDLP ' + ActionButton12.BackgroundImage = CType(resources.GetObject("ActionButton12.BackgroundImage"), System.Drawing.Image) + ActionButton12.Name = "Open" + ActionButton12.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open ActionButton13.BackgroundImage = CType(resources.GetObject("ActionButton13.BackgroundImage"), System.Drawing.Image) - ActionButton13.Name = "Open" - ActionButton13.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open + ActionButton13.Name = "Refresh" + ActionButton13.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh + ActionButton13.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" ActionButton14.BackgroundImage = CType(resources.GetObject("ActionButton14.BackgroundImage"), System.Drawing.Image) - ActionButton14.Name = "Refresh" - ActionButton14.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton14.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" - ActionButton15.BackgroundImage = CType(resources.GetObject("ActionButton15.BackgroundImage"), System.Drawing.Image) - ActionButton15.Name = "Clear" - ActionButton15.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton14.Name = "Clear" + ActionButton14.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_YTDLP.Buttons.Add(ActionButton12) Me.TXT_YTDLP.Buttons.Add(ActionButton13) Me.TXT_YTDLP.Buttons.Add(ActionButton14) - Me.TXT_YTDLP.Buttons.Add(ActionButton15) Me.TXT_YTDLP.CaptionText = "yt-dlp" Me.TXT_YTDLP.CaptionToolTipEnabled = True Me.TXT_YTDLP.CaptionToolTipText = "Path to yt-dlp.exe file" @@ -2109,19 +2135,19 @@ Namespace Editors ' 'TXT_FFMPEG ' + ActionButton15.BackgroundImage = CType(resources.GetObject("ActionButton15.BackgroundImage"), System.Drawing.Image) + ActionButton15.Name = "Open" + ActionButton15.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open ActionButton16.BackgroundImage = CType(resources.GetObject("ActionButton16.BackgroundImage"), System.Drawing.Image) - ActionButton16.Name = "Open" - ActionButton16.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open + ActionButton16.Name = "Refresh" + ActionButton16.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh + ActionButton16.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" ActionButton17.BackgroundImage = CType(resources.GetObject("ActionButton17.BackgroundImage"), System.Drawing.Image) - ActionButton17.Name = "Refresh" - ActionButton17.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton17.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" - ActionButton18.BackgroundImage = CType(resources.GetObject("ActionButton18.BackgroundImage"), System.Drawing.Image) - ActionButton18.Name = "Clear" - ActionButton18.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton17.Name = "Clear" + ActionButton17.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_FFMPEG.Buttons.Add(ActionButton15) Me.TXT_FFMPEG.Buttons.Add(ActionButton16) Me.TXT_FFMPEG.Buttons.Add(ActionButton17) - Me.TXT_FFMPEG.Buttons.Add(ActionButton18) Me.TXT_FFMPEG.CaptionText = "ffmpeg" Me.TXT_FFMPEG.CaptionToolTipEnabled = True Me.TXT_FFMPEG.CaptionToolTipText = "Path to ffmpeg.exe file" @@ -2135,19 +2161,19 @@ Namespace Editors ' 'TXT_CURL ' + ActionButton18.BackgroundImage = CType(resources.GetObject("ActionButton18.BackgroundImage"), System.Drawing.Image) + ActionButton18.Name = "Open" + ActionButton18.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open ActionButton19.BackgroundImage = CType(resources.GetObject("ActionButton19.BackgroundImage"), System.Drawing.Image) - ActionButton19.Name = "Open" - ActionButton19.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open + ActionButton19.Name = "Refresh" + ActionButton19.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh + ActionButton19.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" ActionButton20.BackgroundImage = CType(resources.GetObject("ActionButton20.BackgroundImage"), System.Drawing.Image) - ActionButton20.Name = "Refresh" - ActionButton20.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton20.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" - ActionButton21.BackgroundImage = CType(resources.GetObject("ActionButton21.BackgroundImage"), System.Drawing.Image) - ActionButton21.Name = "Clear" - ActionButton21.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton20.Name = "Clear" + ActionButton20.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_CURL.Buttons.Add(ActionButton18) Me.TXT_CURL.Buttons.Add(ActionButton19) Me.TXT_CURL.Buttons.Add(ActionButton20) - Me.TXT_CURL.Buttons.Add(ActionButton21) Me.TXT_CURL.CaptionText = "cURL" Me.TXT_CURL.CaptionToolTipEnabled = True Me.TXT_CURL.CaptionToolTipText = "Path to curl.exe file" @@ -2161,19 +2187,19 @@ Namespace Editors ' 'TXT_GALLERYDL ' + ActionButton21.BackgroundImage = CType(resources.GetObject("ActionButton21.BackgroundImage"), System.Drawing.Image) + ActionButton21.Name = "Open" + ActionButton21.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open ActionButton22.BackgroundImage = CType(resources.GetObject("ActionButton22.BackgroundImage"), System.Drawing.Image) - ActionButton22.Name = "Open" - ActionButton22.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open + ActionButton22.Name = "Refresh" + ActionButton22.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh + ActionButton22.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" ActionButton23.BackgroundImage = CType(resources.GetObject("ActionButton23.BackgroundImage"), System.Drawing.Image) - ActionButton23.Name = "Refresh" - ActionButton23.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton23.ToolTipText = "Try to find this program automatically (in SCrawler and/or system environment)" - ActionButton24.BackgroundImage = CType(resources.GetObject("ActionButton24.BackgroundImage"), System.Drawing.Image) - ActionButton24.Name = "Clear" - ActionButton24.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton23.Name = "Clear" + ActionButton23.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_GALLERYDL.Buttons.Add(ActionButton21) Me.TXT_GALLERYDL.Buttons.Add(ActionButton22) Me.TXT_GALLERYDL.Buttons.Add(ActionButton23) - Me.TXT_GALLERYDL.Buttons.Add(ActionButton24) Me.TXT_GALLERYDL.CaptionText = "gallery-dl" Me.TXT_GALLERYDL.CaptionToolTipText = "Path to gallery-dl.exe file" Me.TXT_GALLERYDL.CaptionWidth = 80.0R @@ -2186,14 +2212,14 @@ Namespace Editors ' 'TXT_CMD_ENCODING ' + ActionButton24.BackgroundImage = CType(resources.GetObject("ActionButton24.BackgroundImage"), System.Drawing.Image) + ActionButton24.Name = "Refresh" + ActionButton24.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh ActionButton25.BackgroundImage = CType(resources.GetObject("ActionButton25.BackgroundImage"), System.Drawing.Image) - ActionButton25.Name = "Refresh" - ActionButton25.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Refresh - ActionButton26.BackgroundImage = CType(resources.GetObject("ActionButton26.BackgroundImage"), System.Drawing.Image) - ActionButton26.Name = "Clear" - ActionButton26.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton25.Name = "Clear" + ActionButton25.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_CMD_ENCODING.Buttons.Add(ActionButton24) Me.TXT_CMD_ENCODING.Buttons.Add(ActionButton25) - Me.TXT_CMD_ENCODING.Buttons.Add(ActionButton26) Me.TXT_CMD_ENCODING.CaptionText = "CMD Encoding" Me.TXT_CMD_ENCODING.CaptionToolTipEnabled = True Me.TXT_CMD_ENCODING.CaptionToolTipText = "Command line encoding" @@ -2296,10 +2322,10 @@ Namespace Editors ' 'CMB_STD_OPEN_DBL ' - ActionButton27.BackgroundImage = CType(resources.GetObject("ActionButton27.BackgroundImage"), System.Drawing.Image) - ActionButton27.Name = "ArrowDown" - ActionButton27.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.ArrowDown - Me.CMB_STD_OPEN_DBL.Buttons.Add(ActionButton27) + ActionButton26.BackgroundImage = CType(resources.GetObject("ActionButton26.BackgroundImage"), System.Drawing.Image) + ActionButton26.Name = "ArrowDown" + ActionButton26.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.ArrowDown + Me.CMB_STD_OPEN_DBL.Buttons.Add(ActionButton26) Me.CMB_STD_OPEN_DBL.CaptionMode = PersonalUtilities.Forms.Controls.Base.ICaptionControl.Modes.Label Me.CMB_STD_OPEN_DBL.CaptionText = "DoubleClick opens" Me.CMB_STD_OPEN_DBL.CaptionToolTipEnabled = True @@ -2381,10 +2407,10 @@ Namespace Editors ' 'TXT_PRG_TITLE ' - ActionButton28.BackgroundImage = CType(resources.GetObject("ActionButton28.BackgroundImage"), System.Drawing.Image) - ActionButton28.Name = "Clear" - ActionButton28.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_PRG_TITLE.Buttons.Add(ActionButton28) + ActionButton27.BackgroundImage = CType(resources.GetObject("ActionButton27.BackgroundImage"), System.Drawing.Image) + ActionButton27.Name = "Clear" + ActionButton27.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_PRG_TITLE.Buttons.Add(ActionButton27) Me.TXT_PRG_TITLE.CaptionText = "Program title" Me.TXT_PRG_TITLE.CaptionToolTipEnabled = True Me.TXT_PRG_TITLE.CaptionToolTipText = "Change the title of the main window if you need to" @@ -2396,10 +2422,10 @@ Namespace Editors ' 'TXT_PRG_DESCR ' - ActionButton29.BackgroundImage = CType(resources.GetObject("ActionButton29.BackgroundImage"), System.Drawing.Image) - ActionButton29.Name = "Clear" - ActionButton29.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_PRG_DESCR.Buttons.Add(ActionButton29) + ActionButton28.BackgroundImage = CType(resources.GetObject("ActionButton28.BackgroundImage"), System.Drawing.Image) + ActionButton28.Name = "Clear" + ActionButton28.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_PRG_DESCR.Buttons.Add(ActionButton28) Me.TXT_PRG_DESCR.CaptionText = "Program description" Me.TXT_PRG_DESCR.CaptionToolTipEnabled = True Me.TXT_PRG_DESCR.CaptionToolTipText = "Add some additional info to the program info if you need" @@ -2411,14 +2437,14 @@ Namespace Editors ' 'TXT_USER_LIST_IMAGE ' + ActionButton29.BackgroundImage = CType(resources.GetObject("ActionButton29.BackgroundImage"), System.Drawing.Image) + ActionButton29.Name = "Open" + ActionButton29.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open ActionButton30.BackgroundImage = CType(resources.GetObject("ActionButton30.BackgroundImage"), System.Drawing.Image) - ActionButton30.Name = "Open" - ActionButton30.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Open - ActionButton31.BackgroundImage = CType(resources.GetObject("ActionButton31.BackgroundImage"), System.Drawing.Image) - ActionButton31.Name = "Clear" - ActionButton31.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + ActionButton30.Name = "Clear" + ActionButton30.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_USER_LIST_IMAGE.Buttons.Add(ActionButton29) Me.TXT_USER_LIST_IMAGE.Buttons.Add(ActionButton30) - Me.TXT_USER_LIST_IMAGE.Buttons.Add(ActionButton31) Me.TXT_USER_LIST_IMAGE.CaptionText = "Userlist image" Me.TXT_USER_LIST_IMAGE.CaptionToolTipEnabled = True Me.TXT_USER_LIST_IMAGE.CaptionToolTipText = "Background image for user list" @@ -2492,10 +2518,10 @@ Namespace Editors ' 'TXT_H_DEF_UserAgent ' - ActionButton32.BackgroundImage = CType(resources.GetObject("ActionButton32.BackgroundImage"), System.Drawing.Image) - ActionButton32.Name = "Clear" - ActionButton32.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_H_DEF_UserAgent.Buttons.Add(ActionButton32) + ActionButton31.BackgroundImage = CType(resources.GetObject("ActionButton31.BackgroundImage"), System.Drawing.Image) + ActionButton31.Name = "Clear" + ActionButton31.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_H_DEF_UserAgent.Buttons.Add(ActionButton31) Me.TXT_H_DEF_UserAgent.CaptionText = "UserAgent" Me.TXT_H_DEF_UserAgent.CaptionWidth = 140.0R Me.TXT_H_DEF_UserAgent.Dock = System.Windows.Forms.DockStyle.Fill @@ -2506,10 +2532,10 @@ Namespace Editors ' 'TXT_H_DEF_sec_ch_ua ' - ActionButton33.BackgroundImage = CType(resources.GetObject("ActionButton33.BackgroundImage"), System.Drawing.Image) - ActionButton33.Name = "Clear" - ActionButton33.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_H_DEF_sec_ch_ua.Buttons.Add(ActionButton33) + ActionButton32.BackgroundImage = CType(resources.GetObject("ActionButton32.BackgroundImage"), System.Drawing.Image) + ActionButton32.Name = "Clear" + ActionButton32.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_H_DEF_sec_ch_ua.Buttons.Add(ActionButton32) Me.TXT_H_DEF_sec_ch_ua.CaptionText = "sec-ch-ua" Me.TXT_H_DEF_sec_ch_ua.CaptionWidth = 140.0R Me.TXT_H_DEF_sec_ch_ua.Dock = System.Windows.Forms.DockStyle.Fill @@ -2520,10 +2546,10 @@ Namespace Editors ' 'TXT_H_DEF_sec_ch_ua_full_version_list ' - ActionButton34.BackgroundImage = CType(resources.GetObject("ActionButton34.BackgroundImage"), System.Drawing.Image) - ActionButton34.Name = "Clear" - ActionButton34.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_H_DEF_sec_ch_ua_full_version_list.Buttons.Add(ActionButton34) + ActionButton33.BackgroundImage = CType(resources.GetObject("ActionButton33.BackgroundImage"), System.Drawing.Image) + ActionButton33.Name = "Clear" + ActionButton33.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_H_DEF_sec_ch_ua_full_version_list.Buttons.Add(ActionButton33) Me.TXT_H_DEF_sec_ch_ua_full_version_list.CaptionText = "sec-ch-ua-full-version-list" Me.TXT_H_DEF_sec_ch_ua_full_version_list.CaptionWidth = 140.0R Me.TXT_H_DEF_sec_ch_ua_full_version_list.Dock = System.Windows.Forms.DockStyle.Fill @@ -2534,10 +2560,10 @@ Namespace Editors ' 'TXT_H_DEF_sec_ch_ua_platform ' - ActionButton35.BackgroundImage = CType(resources.GetObject("ActionButton35.BackgroundImage"), System.Drawing.Image) - ActionButton35.Name = "Clear" - ActionButton35.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_H_DEF_sec_ch_ua_platform.Buttons.Add(ActionButton35) + ActionButton34.BackgroundImage = CType(resources.GetObject("ActionButton34.BackgroundImage"), System.Drawing.Image) + ActionButton34.Name = "Clear" + ActionButton34.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_H_DEF_sec_ch_ua_platform.Buttons.Add(ActionButton34) Me.TXT_H_DEF_sec_ch_ua_platform.CaptionText = "sec-ch-ua-platform" Me.TXT_H_DEF_sec_ch_ua_platform.CaptionWidth = 140.0R Me.TXT_H_DEF_sec_ch_ua_platform.Dock = System.Windows.Forms.DockStyle.Fill @@ -2548,10 +2574,10 @@ Namespace Editors ' 'TXT_H_DEF_sec_ch_ua_platform_version ' - ActionButton36.BackgroundImage = CType(resources.GetObject("ActionButton36.BackgroundImage"), System.Drawing.Image) - ActionButton36.Name = "Clear" - ActionButton36.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear - Me.TXT_H_DEF_sec_ch_ua_platform_version.Buttons.Add(ActionButton36) + ActionButton35.BackgroundImage = CType(resources.GetObject("ActionButton35.BackgroundImage"), System.Drawing.Image) + ActionButton35.Name = "Clear" + ActionButton35.Tag = PersonalUtilities.Forms.Controls.Base.ActionButton.DefaultButtons.Clear + Me.TXT_H_DEF_sec_ch_ua_platform_version.Buttons.Add(ActionButton35) Me.TXT_H_DEF_sec_ch_ua_platform_version.CaptionText = "sec-ch-ua-platform-version" Me.TXT_H_DEF_sec_ch_ua_platform_version.CaptionWidth = 140.0R Me.TXT_H_DEF_sec_ch_ua_platform_version.Dock = System.Windows.Forms.DockStyle.Fill @@ -2614,35 +2640,6 @@ Namespace Editors Me.CONTAINER_MAIN.TabIndex = 0 Me.CONTAINER_MAIN.TopToolStripPanelVisible = False ' - 'TP_FEED_3 - ' - TP_FEED_3.ColumnCount = 2 - TP_FEED_3.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) - TP_FEED_3.ColumnStyles.Add(New System.Windows.Forms.ColumnStyle(System.Windows.Forms.SizeType.Percent, 50.0!)) - TP_FEED_3.Controls.Add(Me.CH_FEED_TEXT, 0, 0) - TP_FEED_3.Controls.Add(Me.CH_FEED_TEXT_ALWAYS_MOVE, 1, 0) - TP_FEED_3.Dock = System.Windows.Forms.DockStyle.Fill - TP_FEED_3.Location = New System.Drawing.Point(1, 163) - TP_FEED_3.Margin = New System.Windows.Forms.Padding(0) - TP_FEED_3.Name = "TP_FEED_3" - TP_FEED_3.RowCount = 1 - TP_FEED_3.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Percent, 100.0!)) - TP_FEED_3.RowStyles.Add(New System.Windows.Forms.RowStyle(System.Windows.Forms.SizeType.Absolute, 20.0!)) - TP_FEED_3.Size = New System.Drawing.Size(619, 25) - TP_FEED_3.TabIndex = 6 - ' - 'CH_FEED_TEXT_ALWAYS_MOVE - ' - Me.CH_FEED_TEXT_ALWAYS_MOVE.AutoSize = True - Me.CH_FEED_TEXT_ALWAYS_MOVE.Dock = System.Windows.Forms.DockStyle.Fill - Me.CH_FEED_TEXT_ALWAYS_MOVE.Location = New System.Drawing.Point(312, 3) - Me.CH_FEED_TEXT_ALWAYS_MOVE.Name = "CH_FEED_TEXT_ALWAYS_MOVE" - Me.CH_FEED_TEXT_ALWAYS_MOVE.Size = New System.Drawing.Size(304, 19) - Me.CH_FEED_TEXT_ALWAYS_MOVE.TabIndex = 1 - Me.CH_FEED_TEXT_ALWAYS_MOVE.Text = "Auto-move linked text" - TT_MAIN.SetToolTip(Me.CH_FEED_TEXT_ALWAYS_MOVE, "Always move the linked text file along with the original file") - Me.CH_FEED_TEXT_ALWAYS_MOVE.UseVisualStyleBackColor = True - ' 'GlobalSettingsForm ' Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!) @@ -2669,7 +2666,6 @@ Namespace Editors CType(Me.TXT_COLLECTIONS_PATH, System.ComponentModel.ISupportInitialize).EndInit() CType(Me.TXT_MAX_JOBS_USERS, System.ComponentModel.ISupportInitialize).EndInit() CType(Me.TXT_MAX_JOBS_CHANNELS, System.ComponentModel.ISupportInitialize).EndInit() - CType(Me.TXT_IMGUR_CLIENT_ID, System.ComponentModel.ISupportInitialize).EndInit() TP_FILE_NAME.ResumeLayout(False) TP_FILE_NAME.PerformLayout() TP_FILE_PATTERNS.ResumeLayout(False) @@ -2721,6 +2717,8 @@ Namespace Editors TP_FEED_1.PerformLayout() TP_FEED_2.ResumeLayout(False) TP_FEED_2.PerformLayout() + TP_FEED_3.ResumeLayout(False) + TP_FEED_3.PerformLayout() TAB_NOTIFY.ResumeLayout(False) TP_NOTIFY_MAIN.ResumeLayout(False) TP_NOTIFY_MAIN.PerformLayout() @@ -2752,8 +2750,6 @@ Namespace Editors Me.CONTAINER_MAIN.ContentPanel.ResumeLayout(False) Me.CONTAINER_MAIN.ResumeLayout(False) Me.CONTAINER_MAIN.PerformLayout() - TP_FEED_3.ResumeLayout(False) - TP_FEED_3.PerformLayout() Me.ResumeLayout(False) End Sub @@ -2774,7 +2770,6 @@ Namespace Editors Private WithEvents CH_DOWN_IMAGES As CheckBox Private WithEvents CH_DEF_TEMP As CheckBox Private WithEvents CH_CHANNELS_USERS_TEMP As CheckBox - Private WithEvents TXT_IMGUR_CLIENT_ID As PersonalUtilities.Forms.Controls.TextBoxExtended Private WithEvents OPT_FILE_NAME_REPLACE As RadioButton Private WithEvents OPT_FILE_NAME_ADD_DATE As RadioButton Private WithEvents CH_FILE_NAME_CHANGE As CheckBox @@ -2873,5 +2868,6 @@ Namespace Editors Private WithEvents CH_FEED_OPEN_CTRLF As CheckBox Private WithEvents CH_FEED_TEXT As CheckBox Private WithEvents CH_FEED_TEXT_ALWAYS_MOVE As CheckBox + Private WithEvents CH_ADD_MISSING_USERS_TO_LOG As CheckBox End Class End Namespace \ No newline at end of file diff --git a/SCrawler/Editors/GlobalSettingsForm.resx b/SCrawler/Editors/GlobalSettingsForm.resx index c53f391..63e762f 100644 --- a/SCrawler/Editors/GlobalSettingsForm.resx +++ b/SCrawler/Editors/GlobalSettingsForm.resx @@ -124,7 +124,7 @@ iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -135,7 +135,7 @@ iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -146,7 +146,7 @@ iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -154,7 +154,7 @@ iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -170,7 +170,7 @@ iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -181,14 +181,6 @@ dhGSC9h4dUoWntZf9e6/0qr3/5V3/Puvuu//f6Xt///Lrvr8X3ryzf8SfsWbmNBiAsV/bJIqpkpLX/xU 2/n3v+qcmx9VZl77oth97L1U2pQTgtq2WUwMDGzI6jEMAGFR/7wGzS3f//PIagawcvIqsLJzKTAyMLCi q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -219,6 +211,11 @@ You can find more detailed information about the missing posts in the form that Add error information about missing posts (that were not downloaded) to the log. That means that the error text about the not downloaded post will be added to the log. +You can find more detailed information about the missing posts in the form that opens by right-clicking on the 'Info' button in the main window. + + + Add information to the log about users who have missing posts. +This means that the list of user (who have missing posts) will be added to the log. You can find more detailed information about the missing posts in the form that opens by right-clicking on the 'Info' button in the main window. @@ -245,18 +242,18 @@ You can find more detailed information about the missing posts in the form that False - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -276,10 +273,10 @@ You can find more detailed information about the missing posts in the form that False - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -287,10 +284,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -310,10 +307,10 @@ You can find more detailed information about the missing posts in the form that False - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -326,10 +323,10 @@ You can find more detailed information about the missing posts in the form that q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -361,10 +358,10 @@ You can find more detailed information about the missing posts in the form that False - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -372,10 +369,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -386,20 +383,20 @@ You can find more detailed information about the missing posts in the form that dhGSC9h4dUoWntZf9e6/0qr3/5V3/Puvuu//f6Xt///Lrvr8X3ryzf8SfsWbmNBiAsV/bJIqpkpLX/xU 2/n3v+qcmx9VZl77oth97L1U2pQTgtq2WUwMDGzI6jEMAGFR/7wGzS3f//PIagawcvIqsLJzKTAyMLCi q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= + + + + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 + gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -407,10 +404,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -421,20 +418,20 @@ You can find more detailed information about the missing posts in the form that dhGSC9h4dUoWntZf9e6/0qr3/5V3/Puvuu//f6Xt///Lrvr8X3ryzf8SfsWbmNBiAsV/bJIqpkpLX/xU 2/n3v+qcmx9VZl77oth97L1U2pQTgtq2WUwMDGzI6jEMAGFR/7wGzS3f//PIagawcvIqsLJzKTAyMLCi q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= + + + + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 + gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -442,10 +439,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -456,20 +453,20 @@ You can find more detailed information about the missing posts in the form that dhGSC9h4dUoWntZf9e6/0qr3/5V3/Puvuu//f6Xt///Lrvr8X3ryzf8SfsWbmNBiAsV/bJIqpkpLX/xU 2/n3v+qcmx9VZl77oth97L1U2pQTgtq2WUwMDGzI6jEMAGFR/7wGzS3f//PIagawcvIqsLJzKTAyMLCi q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= + + + + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 + gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -477,10 +474,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -493,18 +490,18 @@ You can find more detailed information about the missing posts in the form that q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGOfPtRkwAAACBjSFJNAAB6 - JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsTAAALEwEAmpwYAAACM0lE + JQAAgIMAAPn/AACA6QAAdTAAAOpgAAA6mAAAF2+SX8VGAAAACXBIWXMAAAsPAAALDwGS+QOlAAACM0lE QVQ4T2P4//8/AyHMzicsqh6c34guDsIYAshYQFadhZmZhU1c18Yp8NCP/7LWvjHoajA0gbCwpqW7YeGs FXYTD15xnHnihuuiiw98D//6777kynNuAREpnAYwMjIyqQQX9zkvffDfZce3//b7//23PPD/v/HWr//1 Vr74b7jm3X/VpI7FTIyMjFgNUAku6nRd//6/0ZTT31QbNlxXbthwQa3/6CO1ZS/+K8y+91968u3/qr1X @@ -517,10 +514,10 @@ You can find more detailed information about the missing posts in the form that q8NpABMLK4dCfOsSZlYOUXQ5bBgArRReBMoH61gAAAAASUVORK5CYII= - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -531,7 +528,7 @@ You can find more detailed information about the missing posts in the form that False - + iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAYAAAD0eNT6AAAABGdBTUEAALGPC/xhBQAAE0xJREFUeF7t 3X2MZWddB/BZSkspUFsoXZidOc/vnOfsTmGgARaaBgK2VkFBsIAEgkAQsX8QNJFEq39oMMEoSDREElOB @@ -625,10 +622,18 @@ You can find more detailed information about the missing posts in the form that False + + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 + gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -636,15 +641,7 @@ You can find more detailed information about the missing posts in the form that iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - wwAADsMBx2+oZAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP + vwAADr8BOAVTJAAAARlJREFUOE+VkjFqwzAUhn2D9iShRyi+QhYbWujg3dATZPKYZC6BQhvw1AMkg3dP XQyl7WIyJIEW5CbS0/jKE5GwpCghgg9s6/8/y5Kj6DA45zcAwAAAezB6rjNnB4XX244NHt8wGs7wblop yRGxwZQBYKIfbn477EvqusY4jj2MgMpPiwav7l9UyYXmdrs9duzP4ApUmd72sfrxVsD33JQISyClvFUX w9nJssvJFei9CJUtgQ7394Du3YKLJaCbLMuwqips21ZNuDve/35X8J7nuRcMsVwsbYEQYlSWpRcMMR5P @@ -652,10 +649,10 @@ You can find more detailed information about the missing posts in the form that khP7AAAAAElFTkSuQmCC - + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -663,10 +660,18 @@ You can find more detailed information about the missing posts in the form that False + + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 + gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC + + iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -674,7 +679,7 @@ You can find more detailed information about the missing posts in the form that iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -682,7 +687,7 @@ You can find more detailed information about the missing posts in the form that iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC @@ -690,15 +695,7 @@ You can find more detailed information about the missing posts in the form that iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 - DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 - gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC - - - - - iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAO - xAAADsQBlSsOGwAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 + wAAADsABataJCQAAAHpJREFUOE+1kVEKgDAMQ3e2/e+MXqpn6W/HxM7SpkIVB4HxSMKiTUTaFwVQ1X25 DjMfSxskHBYsAxHJkjUjHgrUNMY4peaMPxb03rcZMVhgn2oDKAwn+L0aROH/Cny4NAGFSx8x+10ZDwV2 gt+LOCxQsw1nPBS8VQBVTTzyhrdZSUm7AAAAAElFTkSuQmCC diff --git a/SCrawler/Editors/GlobalSettingsForm.vb b/SCrawler/Editors/GlobalSettingsForm.vb index 20465da..0aad6dc 100644 --- a/SCrawler/Editors/GlobalSettingsForm.vb +++ b/SCrawler/Editors/GlobalSettingsForm.vb @@ -38,7 +38,6 @@ Namespace Editors TXT_MAX_JOBS_USERS.Value = .MaxUsersJobsCount.Value TXT_MAX_JOBS_CHANNELS.Value = .ChannelsMaxJobsCount.Value CH_CHECK_VER_START.Checked = .CheckUpdatesAtStart - TXT_IMGUR_CLIENT_ID.Text = .ImgurClientID 'Design TXT_PRG_TITLE.Text = .ProgramText TXT_PRG_DESCR.Text = .ProgramDescription @@ -124,6 +123,7 @@ Namespace Editors TXT_DOWN_COMPLETE_SCRIPT.Checked = .DownloadsCompleteCommand.Attribute CH_ADD_MISSING_TO_LOG.Checked = .AddMissingToLog CH_ADD_MISSING_ERROS_TO_LOG.Checked = .AddMissingErrorsToLog + CH_ADD_MISSING_USERS_TO_LOG.Checked = .AddMissingPostUsersToLog CH_DOWN_REPARSE_MISSING.Checked = .ReparseMissingInTheRoutine CH_USE_DEF_ACC.Checked = .UseDefaultAccountIfMissing TXT_AUTO_BRUSH_MIN.Value = .AutomationBrushUndownloadedPlansMinutes.Value @@ -256,7 +256,6 @@ Namespace Editors .MaxUsersJobsCount.Value = CInt(TXT_MAX_JOBS_USERS.Value) .ChannelsMaxJobsCount.Value = TXT_MAX_JOBS_CHANNELS.Value .CheckUpdatesAtStart.Value = CH_CHECK_VER_START.Checked - .ImgurClientID.Value = TXT_IMGUR_CLIENT_ID.Text 'Design .ProgramText.Value = TXT_PRG_TITLE.Text .ProgramDescription.Value = TXT_PRG_DESCR.Text @@ -343,6 +342,7 @@ Namespace Editors .DownloadsCompleteCommand.Attribute.Value = TXT_DOWN_COMPLETE_SCRIPT.Checked .AddMissingToLog.Value = CH_ADD_MISSING_TO_LOG.Checked .AddMissingErrorsToLog.Value = CH_ADD_MISSING_ERROS_TO_LOG.Checked + .AddMissingPostUsersToLog.Value = CH_ADD_MISSING_USERS_TO_LOG.Checked .ReparseMissingInTheRoutine.Value = CH_DOWN_REPARSE_MISSING.Checked .UseDefaultAccountIfMissing.Value = CH_USE_DEF_ACC.Checked .AutomationBrushUndownloadedPlansMinutes.Value = TXT_AUTO_BRUSH_MIN.Value diff --git a/SCrawler/My Project/AssemblyInfo.vb b/SCrawler/My Project/AssemblyInfo.vb index 0e13b40..73ec001 100644 --- a/SCrawler/My Project/AssemblyInfo.vb +++ b/SCrawler/My Project/AssemblyInfo.vb @@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices ' by using the '*' as shown below: ' - - + + diff --git a/SCrawler/PluginsEnvironment/Hosts/DownloadableMediaHost.vb b/SCrawler/PluginsEnvironment/Hosts/DownloadableMediaHost.vb index 5e5cc12..262335a 100644 --- a/SCrawler/PluginsEnvironment/Hosts/DownloadableMediaHost.vb +++ b/SCrawler/PluginsEnvironment/Hosts/DownloadableMediaHost.vb @@ -45,7 +45,9 @@ Namespace Plugin.Hosts End Property Public Overrides Sub Delete(ByVal RemoveFiles As Boolean) MyBase.Delete(RemoveFiles) - If Not RemoveFiles And Not Settings.STDownloader_SnapshotsKeepWithFiles And Settings.STDownloader_SnapShotsCachePermamnent And Not ThumbAlong Then _ + If Not RemoveFiles And Not Settings.STDownloader_SnapshotsKeepWithFiles And + Settings.STDownloader_SnapShotsCachePermamnent And Not ThumbAlong And + (Files.Count = 0 OrElse Not Files.Contains(ThumbnailFile)) Then _ ThumbnailFile.Delete(SFO.File, SFODelete.DeleteToRecycleBin, EDP.None) End Sub Friend Sub New(ByVal URL As String, ByVal OutputFile As SFile) @@ -61,8 +63,6 @@ Namespace Plugin.Hosts If Not SiteKey.IsEmptyString Then If SiteKey = API.Gfycat.Envir.SiteKey Then Instance = New API.Gfycat.Envir - ElseIf SiteKey = API.Imgur.Envir.SiteKey Then - Instance = New API.Imgur.Envir Else plugin = Settings(SiteKey) If plugin Is Nothing Then diff --git a/SCrawler/PluginsEnvironment/Hosts/PluginHost.vb b/SCrawler/PluginsEnvironment/Hosts/PluginHost.vb index 4f1fee8..dca7e7f 100644 --- a/SCrawler/PluginsEnvironment/Hosts/PluginHost.vb +++ b/SCrawler/PluginsEnvironment/Hosts/PluginHost.vb @@ -89,7 +89,6 @@ Namespace Plugin.Hosts Return {New PluginHost(GetType(API.Reddit.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), New PluginHost(GetType(API.Twitter.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), New PluginHost(GetType(API.Bluesky.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), - New PluginHost(GetType(API.Mastodon.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), New PluginHost(GetType(API.Instagram.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), New PluginHost(GetType(API.ThreadsNet.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), New PluginHost(GetType(API.Facebook.SiteSettings), _XML, GlobalPath, _Temp, _Imgs, _Vids), diff --git a/SCrawler/SCrawler.vbproj b/SCrawler/SCrawler.vbproj index ade2942..dec2e8b 100644 --- a/SCrawler/SCrawler.vbproj +++ b/SCrawler/SCrawler.vbproj @@ -211,11 +211,6 @@ - - - - - @@ -425,7 +420,6 @@ - @@ -672,6 +666,7 @@ My.Resources ResXFileCodeGenerator SiteResources.Designer.vb + Designer UserSearchForm.vb @@ -788,12 +783,6 @@ false - - - - - - diff --git a/SCrawler/SettingsCLS.vb b/SCrawler/SettingsCLS.vb index c119f71..923eced 100644 --- a/SCrawler/SettingsCLS.vb +++ b/SCrawler/SettingsCLS.vb @@ -266,7 +266,6 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable MaxSmallImageHeight = New XMLValue(Of Integer)("MaxSmallImageHeight", 15, MyXML, n) CollectionsPath = New XMLValue(Of String)("CollectionsPath", CollectionsFolderName, MyXML, n) MaxUsersJobsCount = New XMLValue(Of Integer)("MaxJobsCount", DefaultMaxDownloadingTasks, MyXML, n) - ImgurClientID = New XMLValue(Of String)("ImgurClientID", String.Empty, MyXML, {Name_Node_Sites}) 'Basis: new version n = {"ProgramVersion"} @@ -363,6 +362,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable UpdateUserIconBannerEveryTime = New XMLValue(Of Boolean)("UpdateUserIconBannerEveryTime", True, MyXML, n) AddMissingToLog = New XMLValue(Of Boolean)("AddMissingToLog", True, MyXML, n) AddMissingErrorsToLog = New XMLValue(Of Boolean)("AddMissingErrorsToLog", False, MyXML, n) + AddMissingPostUsersToLog = New XMLValue(Of Boolean)("AddMissingPostUsersToLog", False, MyXML, n) DownloadsCompleteCommand = New XMLValueAttribute(Of String, Boolean)("DownloadsCompleteCommand", "Use",,, MyXML, n) ReparseMissingInTheRoutine = New XMLValue(Of Boolean)("ReparseMissingInTheRoutine", False, MyXML, n) UseDefaultAccountIfMissing = New XMLValue(Of Boolean)("UseDefaultAccountIfMissing", True, MyXML, n) @@ -887,7 +887,6 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable Return HEADER_UserAgent End Get End Property - Friend ReadOnly Property ImgurClientID As XMLValue(Of String) #End Region #Region "Basis: new version" Friend ReadOnly Property CheckUpdatesAtStart As XMLValue(Of Boolean) @@ -1072,6 +1071,7 @@ Friend Class SettingsCLS : Implements IDownloaderSettings, IDisposable Friend ReadOnly Property UpdateUserIconBannerEveryTime As XMLValue(Of Boolean) Friend ReadOnly Property AddMissingToLog As XMLValue(Of Boolean) Friend ReadOnly Property AddMissingErrorsToLog As XMLValue(Of Boolean) + Friend ReadOnly Property AddMissingPostUsersToLog As XMLValue(Of Boolean) Friend ReadOnly Property DownloadsCompleteCommand As XMLValueAttribute(Of String, Boolean) Friend ReadOnly Property ReparseMissingInTheRoutine As XMLValue(Of Boolean) Friend ReadOnly Property UseDefaultAccountIfMissing As XMLValue(Of Boolean) diff --git a/SCrawler/SiteResources.Designer.vb b/SCrawler/SiteResources.Designer.vb index 95fc80f..c09066b 100644 --- a/SCrawler/SiteResources.Designer.vb +++ b/SCrawler/SiteResources.Designer.vb @@ -22,7 +22,7 @@ Namespace My.Resources ''' ''' A strongly-typed resource class, for looking up localized strings, etc. ''' - _ Friend Class SiteResources @@ -164,26 +164,6 @@ Namespace My.Resources End Get End Property - ''' - ''' Looks up a localized resource of type System.Drawing.Icon similar to (Icon). - ''' - Friend Shared ReadOnly Property MastodonIcon_48() As System.Drawing.Icon - Get - Dim obj As Object = ResourceManager.GetObject("MastodonIcon_48", resourceCulture) - Return CType(obj,System.Drawing.Icon) - End Get - End Property - - ''' - ''' Looks up a localized resource of type System.Drawing.Bitmap. - ''' - Friend Shared ReadOnly Property MastodonPic_48() As System.Drawing.Bitmap - Get - Dim obj As Object = ResourceManager.GetObject("MastodonPic_48", resourceCulture) - Return CType(obj,System.Drawing.Bitmap) - End Get - End Property - ''' ''' Looks up a localized resource of type System.Drawing.Icon similar to (Icon). ''' diff --git a/SCrawler/SiteResources.resx b/SCrawler/SiteResources.resx index 8dc9110..92a66fb 100644 --- a/SCrawler/SiteResources.resx +++ b/SCrawler/SiteResources.resx @@ -148,12 +148,6 @@ Content\Pictures\SitePictures\LPSGPic_32.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - Content\Icons\SiteIcons\MastodonIcon_48.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - - - Content\Pictures\SitePictures\MastodonPic_48.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a - Content\Icons\SiteIcons\OnlyFansIcon_32.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a diff --git a/Tools/Install-OFScraper.ps1 b/Tools/Install-OFScraper.ps1 new file mode 100644 index 0000000..a7ef6e1 --- /dev/null +++ b/Tools/Install-OFScraper.ps1 @@ -0,0 +1,782 @@ +# Self-contained SCrawler OF-Scraper setup (this file only; no extra .py/.bat). +# +# Installs ofscraper 3.12.9 via pip, pins aiolimiter 1.1.0, then applies DRM +# false-success patch 20260827_drm_minimal_v2 (embedded alt_download.py). +# +# Needs Python 3.11.x or 3.12.x. Older than 3.11.x and 3.13+ both fail with +# ofscraper 3.12.9. If a usable Python/pip is missing, this script points you +# to Python 3.11.6. After install, empty DRM segments / 0-byte audio+video are +# not marked downloaded. +# +# Usage: +# .\Install-OFScraper.ps1 +# .\Install-OFScraper.ps1 -Force +# .\Install-OFScraper.ps1 -Quiet +# .\Install-OFScraper.ps1 -SkipDrmPatch +# .\Install-OFScraper.ps1 -PythonPath "C:\Path\to\python.exe" +# +# The script was written by @cjb900. @cjb900, you are the best! + +param( + [string]$PythonPath = "", + [switch]$Force, + [switch]$Quiet, + [switch]$SkipDrmPatch +) + +$ErrorActionPreference = "Stop" + +$OfScraperVersion = "3.12.9" +$AioLimiterVersion = "1.1.0" +$DrmPatchId = "20260827_drm_minimal_v2" +$DrmPatchRelPath = "actions\actions\download\managers\alt_download.py" +$DrmPatchSentinel = "DRM download produced 0 bytes" +$PythonMin = [version]"3.11.0" +$PythonMaxExclusive = [version]"3.13.0" +$PythonRecommended = "3.11.6" +$PythonReleaseUrl = "https://www.python.org/downloads/release/python-3116/" +$PythonInstaller64Url = "https://www.python.org/ftp/python/3.11.6/python-3.11.6-amd64.exe" + +if ($ExecutionContext.SessionState.LanguageMode -eq "ConstrainedLanguage") { + Write-Host "Error: PowerShell execution policy is restricting script execution." -ForegroundColor Red + Write-Host "Run: powershell -ExecutionPolicy Bypass -File `"$($MyInvocation.MyCommand.Path)`"" -ForegroundColor Cyan + if ($Host.Name -eq "ConsoleHost" -and -not $Quiet) { + Write-Host "Press any key to exit..." -ForegroundColor Gray + $null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") + } + exit 1 +} + +function Write-ColorOutput { + param( + [string]$Message, + [string]$Color = "White" + ) + if (-not $Quiet) { + Write-Host $Message -ForegroundColor $Color + } +} + +function Test-WindowsAppsStub { + param([string]$ExePath) + return ($ExePath -match '\\WindowsApps\\') +} + +function Get-PythonVersion { + param([string]$ExePath) + try { + $output = & $ExePath -c "import sys; print('%d.%d.%d' % sys.version_info[:3])" 2>$null + if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($output)) { + return $null + } + return [version]($output.Trim()) + } + catch { + return $null + } +} + +function Test-SupportedPython { + param([version]$Version) + return ($Version -ge $PythonMin -and $Version -lt $PythonMaxExclusive) +} + +function Get-PythonAgeLabel { + param([version]$Version) + if ($Version -lt $PythonMin) { return "too old" } + if ($Version -ge $PythonMaxExclusive) { return "too new" } + return "supported" +} + +function Add-PythonCandidate { + param( + [System.Collections.Generic.List[object]]$List, + [string]$ExePath + ) + if ([string]::IsNullOrWhiteSpace($ExePath)) { return } + try { $ExePath = [System.IO.Path]::GetFullPath($ExePath) } catch { return } + if (-not (Test-Path -LiteralPath $ExePath)) { return } + if (Test-WindowsAppsStub -ExePath $ExePath) { return } + foreach ($existing in $List) { + if ($existing.Path -eq $ExePath) { return } + } + $version = Get-PythonVersion -ExePath $ExePath + if (-not $version) { return } + $List.Add([pscustomobject]@{ + Path = $ExePath + Version = $version + Supported = (Test-SupportedPython -Version $version) + Age = (Get-PythonAgeLabel -Version $version) + }) +} + +function Find-PythonInstalls { + $found = New-Object "System.Collections.Generic.List[object]" + + if ($PythonPath) { + Add-PythonCandidate -List $found -ExePath $PythonPath + return $found + } + + $pyLauncher = Get-Command py -ErrorAction SilentlyContinue + if ($pyLauncher) { + try { + $pyList = & py -0p 2>$null + foreach ($line in @($pyList)) { + if ($line -match '([A-Za-z]:\\.+\\python\.exe)') { + Add-PythonCandidate -List $found -ExePath $Matches[1].Trim() + } + } + } + catch { } + + foreach ($tag in @("3.14", "3.13", "3.12", "3.11", "3.10", "3.9", "3.8")) { + try { + $resolved = & py "-$tag" -c "import sys; print(sys.executable)" 2>$null + if ($LASTEXITCODE -eq 0 -and -not [string]::IsNullOrWhiteSpace($resolved)) { + Add-PythonCandidate -List $found -ExePath $resolved.Trim() + } + } + catch { } + } + } + + foreach ($cmdName in @("python", "python3")) { + $cmd = Get-Command $cmdName -ErrorAction SilentlyContinue + if ($cmd -and $cmd.Source) { + Add-PythonCandidate -List $found -ExePath $cmd.Source + } + } + + $searchRoots = @( + "$env:LOCALAPPDATA\Programs\Python", + "$env:ProgramFiles", + "${env:ProgramFiles(x86)}" + ) + foreach ($root in $searchRoots) { + if (-not (Test-Path -LiteralPath $root)) { continue } + Get-ChildItem -Path $root -Directory -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^Python3\d+$' } | + ForEach-Object { + $exe = Join-Path $_.FullName "python.exe" + Add-PythonCandidate -List $found -ExePath $exe + } + if ($root -eq "$env:LOCALAPPDATA\Programs\Python") { + Get-ChildItem -Path $root -Filter python.exe -Recurse -ErrorAction SilentlyContinue | + Where-Object { $_.Directory.Name -notmatch '^(Scripts|Lib|include)$' } | + ForEach-Object { Add-PythonCandidate -List $found -ExePath $_.FullName } + } + } + + return $found +} + +function Test-Pip { + param([string]$ExePath) + try { + $null = & $ExePath -m pip --version 2>$null + return ($LASTEXITCODE -eq 0) + } + catch { + return $false + } +} + +function Install-Pip { + param([string]$ExePath) + Write-ColorOutput "pip is missing. Trying python -m ensurepip..." "Yellow" + try { + & $ExePath -m ensurepip --upgrade + if (Test-Pip -ExePath $ExePath) { return $true } + } + catch { } + return $false +} + +function Get-PipPackageVersion { + param( + [string]$ExePath, + [string]$PackageName + ) + try { + $output = & $ExePath -m pip show $PackageName 2>$null + foreach ($line in @($output)) { + if ($line -match '^Version:\s*(.+)$') { + return $Matches[1].Trim() + } + } + } + catch { } + return $null +} + +function Invoke-PipInstall { + param( + [string]$ExePath, + [string[]]$PipArgs + ) + $common = @( + "-m", "pip", "install", + "--disable-pip-version-check", + "--no-warn-script-location" + ) + $PipArgs + + & $ExePath @common + if ($LASTEXITCODE -eq 0) { return $true } + + Write-ColorOutput "Retrying with --user (install into the current user site-packages)..." "Yellow" + $userArgs = @( + "-m", "pip", "install", + "--user", + "--disable-pip-version-check", + "--no-warn-script-location" + ) + $PipArgs + & $ExePath @userArgs + return ($LASTEXITCODE -eq 0) +} + +function Get-OfScraperExe { + param([string]$ExePath) + try { + $resolved = & $ExePath -c @" +import os, sys +from pathlib import Path +candidates = [] +scripts = Path(sys.executable).resolve().parent / 'Scripts' +candidates.append(scripts / 'ofscraper.exe') +try: + import sysconfig + scripts2 = Path(sysconfig.get_path('scripts')) + candidates.append(scripts2 / 'ofscraper.exe') +except Exception: + pass +base = Path(os.environ.get('APPDATA', '')) / 'Python' +if base.exists(): + candidates.extend(base.glob('Python3*/Scripts/ofscraper.exe')) +for c in candidates: + if c.is_file(): + print(c) + break +"@ 2>$null + if ($LASTEXITCODE -eq 0 -and -not [string]::IsNullOrWhiteSpace($resolved)) { + $path = $resolved.Trim() + if (Test-Path -LiteralPath $path) { return $path } + } + } + catch { } + return $null +} + +function Show-PythonMissing { + param( + [string]$Reason, + [object[]]$Unsupported = @() + ) + Write-ColorOutput $Reason "Red" + Write-ColorOutput "" "White" + Write-ColorOutput "OF-Scraper $OfScraperVersion only works with Python 3.11.x or 3.12.x." "Yellow" + Write-ColorOutput " Too old: 3.10.x and earlier (missing language/package features)" "Gray" + Write-ColorOutput " Too new: 3.13.x and later (incompatible with this ofscraper release)" "Gray" + Write-ColorOutput "" "White" + + $tooOld = @($Unsupported | Where-Object { $_.Age -eq "too old" } | Sort-Object Version) + $tooNew = @($Unsupported | Where-Object { $_.Age -eq "too new" } | Sort-Object Version) + + if ($tooOld.Count -gt 0) { + Write-ColorOutput "Found Python older than 3.11.x (cannot be used):" "Yellow" + foreach ($item in $tooOld) { + Write-ColorOutput (" {0} too old ({1})" -f $item.Version, $item.Path) "Gray" + } + Write-ColorOutput "" "White" + } + if ($tooNew.Count -gt 0) { + Write-ColorOutput "Found Python 3.13 or newer (cannot be used):" "Yellow" + foreach ($item in $tooNew) { + Write-ColorOutput (" {0} too new ({1})" -f $item.Version, $item.Path) "Gray" + } + Write-ColorOutput "" "White" + } + if ($Unsupported.Count -eq 0) { + Write-ColorOutput "No Python install was detected on this machine." "Yellow" + Write-ColorOutput "" "White" + } + + Write-ColorOutput "Install CPython $PythonRecommended (64-bit), then re-run this script." "Yellow" + Write-ColorOutput "In the installer, enable 'Add python.exe to PATH'." "Yellow" + Write-ColorOutput "Leave any 3.10-or-older and 3.13+ installs in place; this script will pick 3.11/3.12." "Yellow" + Write-ColorOutput "" "White" + Write-ColorOutput "Release page: $PythonReleaseUrl" "Cyan" + Write-ColorOutput "64-bit installer: $PythonInstaller64Url" "Cyan" + Write-ColorOutput "" "White" + + if ($Quiet) { return } + + $answer = Read-Host "Open the Python $PythonRecommended download page now? (y/n)" + if ($answer -match '^[Yy]') { + Start-Process $PythonReleaseUrl + } +} + +function Get-OfScraperPackageDir { + param([string]$ExePath) + try { + $output = & $ExePath -c "import ofscraper, pathlib; print(pathlib.Path(next(iter(ofscraper.__path__))))" 2>$null + if ($LASTEXITCODE -eq 0 -and -not [string]::IsNullOrWhiteSpace($output)) { + $dir = $output.Trim() + if (Test-Path -LiteralPath $dir) { return $dir } + } + } + catch { } + return $null +} + +function Test-DrmPatchApplied { + param([string]$AltDownloadPath) + if (-not (Test-Path -LiteralPath $AltDownloadPath)) { return $false } + try { + $text = [System.IO.File]::ReadAllText($AltDownloadPath) + return $text.Contains($DrmPatchSentinel) + } + catch { + return $false + } +} + +function Get-DrmPatchBase64 { + # Patched alt_download.py for ofscraper 3.12.9 (patch 20260827_drm_minimal_v2) + return @' +ciIiIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiBfX19fX19fICBfX19fX19fICAgICAg +ICAgX19fX19fXyAgX19fX19fXyAgX19fX19fXyAgX19fX19fXyAgX19fX19fXyAgX19fX19fXyAgX19fX19fXyAKKCAgX19fICApKCAgX19fXyBcICAgICAg +ICggIF9fX18gXCggIF9fX18gXCggIF9fX18gKSggIF9fXyAgKSggIF9fX18gKSggIF9fX18gXCggIF9fX18gKQp8ICggICApIHx8ICggICAgXC8gICAgICAg +fCAoICAgIFwvfCAoICAgIFwvfCAoICAgICl8fCAoICAgKSB8fCAoICAgICl8fCAoICAgIFwvfCAoICAgICl8CnwgfCAgIHwgfHwgKF9fICAgICBfX19fXyB8 +IChfX19fXyB8IHwgICAgICB8IChfX19fKXx8IChfX18pIHx8IChfX19fKXx8IChfXyAgICB8IChfX19fKXwKfCB8ICAgfCB8fCAgX18pICAgKF9fX19fKShf +X19fXyAgKXwgfCAgICAgIHwgICAgIF9fKXwgIF9fXyAgfHwgIF9fX19fKXwgIF9fKSAgIHwgICAgIF9fKQp8IHwgICB8IHx8ICggICAgICAgICAgICAgICAg +ICAgKSB8fCB8ICAgICAgfCAoXCAoICAgfCAoICAgKSB8fCAoICAgICAgfCAoICAgICAgfCAoXCAoICAgCnwgKF9fXykgfHwgKSAgICAgICAgICAgICAvXF9f +X18pIHx8IChfX19fL1x8ICkgXCBcX198ICkgICAoIHx8ICkgICAgICB8IChfX19fL1x8ICkgXCBcX18KKF9fX19fX18pfC8gICAgICAgICAgICAgIFxfX19f +X19fKShfX19fX19fL3wvICAgXF9fL3wvICAgICBcfHwvICAgICAgIChfX19fX19fL3wvICAgXF9fLwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg +ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAoiIiIKCmltcG9ydCBhc3luY2lvCmltcG9ydCBwYXRobGli +CmltcG9ydCByZQppbXBvcnQgdHJhY2ViYWNrCmZyb20gZnVuY3Rvb2xzIGltcG9ydCBwYXJ0aWFsCmZyb20gdXJsbGliLnBhcnNlIGltcG9ydCB1cmxzcGxp +dCwgdXJsdW5zcGxpdAoKCmltcG9ydCBhaW9maWxlcwppbXBvcnQgYXJyb3cKaW1wb3J0IHBzdXRpbApmcm9tIGh1bWFuZnJpZW5kbHkgaW1wb3J0IGZvcm1h +dF9zaXplCgppbXBvcnQgb2ZzY3JhcGVyLmNsYXNzZXMucGxhY2Vob2xkZXIgYXMgcGxhY2Vob2xkZXIKaW1wb3J0IG9mc2NyYXBlci5hY3Rpb25zLnV0aWxz +Lmdsb2JhbHMgYXMgY29tbW9uX2dsb2JhbHMKaW1wb3J0IG9mc2NyYXBlci51dGlscy5jb25zdGFudHMgYXMgY29uc3RhbnRzCmZyb20gb2ZzY3JhcGVyLmNs +YXNzZXMuZG93bmxvYWRfcmV0cmllcyBpbXBvcnQgZG93bmxvYWRfcmV0cnkKCmZyb20gb2ZzY3JhcGVyLmFjdGlvbnMudXRpbHMucGFyYW1zIGltcG9ydCBn +ZXRfYWx0X3BhcmFtcwpmcm9tIG9mc2NyYXBlci5hY3Rpb25zLnV0aWxzLmxvZyBpbXBvcnQgZ2V0X21lZGlhbG9nCmZyb20gb2ZzY3JhcGVyLmFjdGlvbnMu +dXRpbHMubG9nIGltcG9ydCAoCiAgICBnZXRfdXJsX2xvZywKICAgIHBhdGhfdG9fZmlsZV9sb2dnZXIsCiAgICB0ZW1wX2ZpbGVfbG9nZ2VyLAopCmZyb20g +b2ZzY3JhcGVyLmFjdGlvbnMuYWN0aW9ucy5kb3dubG9hZC51dGlscy5jaHVuayBpbXBvcnQgKAogICAgZ2V0X2lkZWFsX2NodW5rX3NpemUsCikKZnJvbSBv +ZnNjcmFwZXIuYWN0aW9ucy51dGlscy5yZXRyaWVzIGltcG9ydCBnZXRfZG93bmxvYWRfcmV0cmllcwpmcm9tIG9mc2NyYXBlci5hY3Rpb25zLnV0aWxzLnNl +bmQuY2h1bmsgaW1wb3J0IHNlbmRfY2h1bmtfbXNnCmZyb20gb2ZzY3JhcGVyLmNsYXNzZXMuc2Vzc2lvbm1hbmFnZXIuc2Vzc2lvbm1hbmFnZXIgaW1wb3J0 +ICgKICAgIEZPUkNFRF9ORVcsCiAgICBTSUdOLAopCmltcG9ydCBvZnNjcmFwZXIudXRpbHMuYXV0aC5yZXF1ZXN0IGFzIGF1dGhfcmVxdWVzdHMKZnJvbSBv +ZnNjcmFwZXIuYWN0aW9ucy5hY3Rpb25zLmRvd25sb2FkLm1hbmFnZXJzLmRvd25sb2FkbWFuYWdlciBpbXBvcnQgRG93bmxvYWRNYW5hZ2VyCmltcG9ydCBv +ZnNjcmFwZXIuYWN0aW9ucy51dGlscy5wYXRocy5wYXRocyBhcyBjb21tb25fcGF0aHMKaW1wb3J0IG9mc2NyYXBlci5hY3Rpb25zLnV0aWxzLmxvZyBhcyBj +b21tb25fbG9ncwpmcm9tIG9mc2NyYXBlci5kYi5vcGVyYXRpb25zXy5tZWRpYSBpbXBvcnQgZG93bmxvYWRfbWVkaWFfdXBkYXRlCmltcG9ydCBvZnNjcmFw +ZXIuYWN0aW9ucy51dGlscy5nZW5lcmFsIGFzIGNvbW1vbgppbXBvcnQgb2ZzY3JhcGVyLnV0aWxzLmRhdGVzIGFzIGRhdGVzCmZyb20gb2ZzY3JhcGVyLnV0 +aWxzLnN5c3RlbS5zdWJwcm9jZXNzIGltcG9ydCBydW4KaW1wb3J0IG9mc2NyYXBlci51dGlscy5zZXR0aW5ncyBhcyBzZXR0aW5ncwppbXBvcnQgb2ZzY3Jh +cGVyLnV0aWxzLnN5c3RlbS5zeXN0ZW0gYXMgc3lzdGVtCmltcG9ydCBvZnNjcmFwZXIuYWN0aW9ucy5hY3Rpb25zLmRvd25sb2FkLnV0aWxzLmtleWhlbHBl +cnMgYXMga2V5aGVscGVycwppbXBvcnQgb2ZzY3JhcGVyLnV0aWxzLmNhY2hlIGFzIGNhY2hlCmltcG9ydCBvZnNjcmFwZXIudXRpbHMubGl2ZS51cGRhdGVy +IGFzIHByb2dyZXNzX3VwZGF0ZXIKZnJvbSBvZnNjcmFwZXIuYWN0aW9ucy51dGlscy5zZW5kLm1lc3NhZ2UgaW1wb3J0IHNlbmRfbXNnCgoKY2xhc3MgQWx0 +RG93bmxvYWRNYW5hZ2VyKERvd25sb2FkTWFuYWdlcik6CiAgICBkZWYgX19pbml0X18oc2VsZiwgbXVsdGk9RmFsc2UpOgogICAgICAgIHN1cGVyKCkuX19p +bml0X18obXVsdGk9bXVsdGkpCgogICAgYXN5bmMgZGVmIGFsdF9kb3dubG9hZChzZWxmLCBjLCBlbGUsIHVzZXJuYW1lLCBtb2RlbF9pZCk6CiAgICAgICAg +Y29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gRG93bmxvYWRpbmcgd2l0aCBwcm90ZWN0ZWQgbWVk +aWEgZG93bmxvYWRlciIKICAgICAgICApCiAgICAgICAgYXN5bmMgZm9yIF8gaW4gZG93bmxvYWRfcmV0cnkoKToKICAgICAgICAgICAgd2l0aCBfOgogICAg +ICAgICAgICAgICAgdHJ5OgogICAgICAgICAgICAgICAgICAgIHNoYXJlZFBsYWNlaG9sZGVyT2JqID0gYXdhaXQgcGxhY2Vob2xkZXIuUGxhY2Vob2xkZXJz +KAogICAgICAgICAgICAgICAgICAgICAgICBlbGUsICJtcDQiCiAgICAgICAgICAgICAgICAgICAgKS5pbml0KCkKICAgICAgICAgICAgICAgICAgICBjb21t +b25fZ2xvYmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgICAgICAgICAgICAgIGYie2dldF9tZWRpYWxvZyhlbGUpfSBkb3dubG9hZCB1cmw6ICB7Z2V0X3Vy +bF9sb2coZWxlKX0iCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgZXhjZXB0IEV4Y2VwdGlvbiBhcyBlOgogICAgICAgICAgICAgICAg +ICAgIHJhaXNlIGUKCiAgICAgICAgYXVkaW8gPSBhd2FpdCBlbGUubXBkX2F1ZGlvCiAgICAgICAgdmlkZW8gPSBhd2FpdCBlbGUubXBkX3ZpZGVvCiAgICAg +ICAgcGF0aF90b19maWxlX2xvZ2dlcihzaGFyZWRQbGFjZWhvbGRlck9iaiwgZWxlKQoKICAgICAgICBhdWRpbyA9IGF3YWl0IHNlbGYuX2FsdF9kb3dubG9h +ZF9kb3dubG9hZGVyKGF1ZGlvLCBjLCBlbGUpCiAgICAgICAgdmlkZW8gPSBhd2FpdCBzZWxmLl9hbHRfZG93bmxvYWRfZG93bmxvYWRlcih2aWRlbywgYywg +ZWxlKQoKICAgICAgICBwb3N0X3Jlc3VsdCA9IGF3YWl0IHNlbGYuX21lZGlhX2l0ZW1fcG9zdF9wcm9jZXNzX2FsdCgKICAgICAgICAgICAgYXVkaW8sIHZp +ZGVvLCBlbGUsIHVzZXJuYW1lLCBtb2RlbF9pZAogICAgICAgICkKICAgICAgICBpZiBwb3N0X3Jlc3VsdDoKICAgICAgICAgICAgcmV0dXJuIHBvc3RfcmVz +dWx0CiAgICAgICAgYXdhaXQgc2VsZi5fbWVkaWFfaXRlbV9rZXlzX2FsdChjLCBhdWRpbywgdmlkZW8sIGVsZSkKCiAgICAgICAgcmV0dXJuIGF3YWl0IHNl +bGYuX2hhbmRsZV9yZXN1bHRfYWx0KAogICAgICAgICAgICBzaGFyZWRQbGFjZWhvbGRlck9iaiwgZWxlLCBhdWRpbywgdmlkZW8sIHVzZXJuYW1lLCBtb2Rl +bF9pZAogICAgICAgICkKCiAgICBhc3luYyBkZWYgX2FsdF9kb3dubG9hZF9kb3dubG9hZGVyKHNlbGYsIGl0ZW0sIGMsIGVsZSk6CiAgICAgICAgc2VsZi5f +ZG93bmxvYWRzcGFjZShtZWRpYXR5cGU9ZWxlLm1lZGlhdHlwZSkKICAgICAgICBwbGFjZWhvbGRlck9iaiA9IGF3YWl0IHBsYWNlaG9sZGVyLnRlbXBGaWxl +UGxhY2Vob2xkZXIoCiAgICAgICAgICAgIGVsZSwgZiJ7aXRlbVsnbmFtZSddfS5wYXJ0IgogICAgICAgICkuaW5pdCgpCiAgICAgICAgaXRlbVsicGF0aCJd +ID0gcGxhY2Vob2xkZXJPYmoudGVtcGZpbGVwYXRoCiAgICAgICAgaXRlbVsidG90YWwiXSA9IE5vbmUKICAgICAgICAKICAgICAgICBhc3luYyBmb3IgXyBp +biBkb3dubG9hZF9yZXRyeSgpOgogICAgICAgICAgICB3aXRoIF86CiAgICAgICAgICAgICAgICB0cnk6CiAgICAgICAgICAgICAgICAgICAgX2F0dGVtcHQg +PSBzZWxmLl9hbHRfYXR0ZW1wdF9nZXQoaXRlbSkKICAgICAgICAgICAgICAgICAgICBfYXR0ZW1wdC5zZXQoX2F0dGVtcHQuZ2V0KDApICsgMSkKICAgICAg +ICAgICAgICAgICAgICBpZiBfYXR0ZW1wdC5nZXQoKSA+IDE6CiAgICAgICAgICAgICAgICAgICAgICAgIHBhdGhsaWIuUGF0aChwbGFjZWhvbGRlck9iai50 +ZW1wZmlsZXBhdGgpLnVubGluaygKICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1pc3Npbmdfb2s9VHJ1ZQogICAgICAgICAgICAgICAgICAgICAgICAp +CiAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGF3YWl0IHNlbGYuX2dldF9kYXRhKGVsZSwgaXRlbSkKICAgICAgICAgICAgICAgICAgICBzdGF0dXMgPSBG +YWxzZQogICAgICAgICAgICAgICAgICAgIGlmIGRhdGE6CiAgICAgICAgICAgICAgICAgICAgICAgIGl0ZW0sIHN0YXR1cyA9IGF3YWl0IHNlbGYuX3Jlc3Vt +ZV9kYXRhX2hhbmRsZXJfYWx0KAogICAgICAgICAgICAgICAgICAgICAgICAgICAgZGF0YSwgaXRlbSwgZWxlLCBwbGFjZWhvbGRlck9iagogICAgICAgICAg +ICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgICAgIGVsc2U6CiAgICAgICAgICAgICAgICAgICAgICAgIGl0ZW0sIHN0YXR1cyA9IGF3YWl0IHNl +bGYuX2ZyZXNoX2RhdGFfaGFuZGxlcl9hbHQoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBpdGVtLCBlbGUsIHBsYWNlaG9sZGVyT2JqCiAgICAgICAg +ICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgICAgICBpZiBub3Qgc3RhdHVzOgogICAgICAgICAgICAgICAgICAgICAgICB0cnk6CiAgICAgICAg +ICAgICAgICAgICAgICAgICAgICBpdGVtID0gYXdhaXQgc2VsZi5fYWx0X2Rvd25sb2FkX3NlbmRyZXEoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg +ICAgaXRlbSwgYywgZWxlLCBwbGFjZWhvbGRlck9iagogICAgICAgICAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgICAgICBleGNl +cHQgRXhjZXB0aW9uIGFzIEU6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICByYWlzZSBFCiAgICAgICAgICAgICAgICAgICAgcmV0dXJuIGl0ZW0KICAg +ICAgICAgICAgICAgIGV4Y2VwdCBPU0Vycm9yIGFzIEU6CiAgICAgICAgICAgICAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAg +ICAgICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gW2F0dGVtcHQge19hdHRlbXB0LmdldCgpfS97Z2V0X2Rvd25sb2FkX3JldHJpZXMoKX1dIE51 +bWJlciBvZiBPcGVuIEZpbGVzIC0+IHsgbGVuKHBzdXRpbC5Qcm9jZXNzKCkub3Blbl9maWxlcygpKX0iCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAg +ICAgICAgICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy5kZWJ1ZygKICAgICAgICAgICAgICAgICAgICAgICAgZiIge2dldF9tZWRpYWxvZyhlbGUpfSBbYXR0 +ZW1wdCB7X2F0dGVtcHQuZ2V0KCl9L3tnZXRfZG93bmxvYWRfcmV0cmllcygpfV0gT3BlbiBGaWxlcyAtPiB7bGlzdChtYXAobGFtYmRhIHg6KHgucGF0aCx4 +LmZkKSxwc3V0aWwuUHJvY2VzcygpLm9wZW5fZmlsZXMoKSkpfSIKICAgICAgICAgICAgICAgICAgICApCiAgICAgICAgICAgICAgICAgICAgcmFpc2UgRQog +ICAgICAgICAgICAgICAgZXhjZXB0IEV4Y2VwdGlvbiBhcyBFOgogICAgICAgICAgICAgICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy50cmFjZWJhY2tfKAog +ICAgICAgICAgICAgICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gW2F0dGVtcHQge19hdHRlbXB0LmdldCgpfS97Z2V0X2Rvd25sb2FkX3JldHJp +ZXMoKX1dIHt0cmFjZWJhY2suZm9ybWF0X2V4YygpfSIKICAgICAgICAgICAgICAgICAgICApCiAgICAgICAgICAgICAgICAgICAgY29tbW9uX2dsb2JhbHMu +bG9nLnRyYWNlYmFja18oCiAgICAgICAgICAgICAgICAgICAgICAgIGYie2dldF9tZWRpYWxvZyhlbGUpfSBbYXR0ZW1wdCB7X2F0dGVtcHQuZ2V0KCl9L3tn +ZXRfZG93bmxvYWRfcmV0cmllcygpfV0ge0V9IgogICAgICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgICAgICByYWlzZSBFCgogICAgYXN5bmMg +ZGVmIF9hbHRfZG93bmxvYWRfc2VuZHJlcShzZWxmLCBpdGVtLCBjLCBlbGUsIHBsYWNlaG9sZGVyT2JqKToKICAgICAgICB0cnk6CiAgICAgICAgICAgIF9h +dHRlbXB0ID0gc2VsZi5fYWx0X2F0dGVtcHRfZ2V0KGl0ZW0pCiAgICAgICAgICAgIHVybCA9IF9tcGRfc2VnbWVudF91cmwoZWxlLm1wZCwgaXRlbVsib3Jp +Z25hbWUiXSkKICAgICAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICAgICAgZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IEF0dGVt +cHRpbmcgdG8gZG93bmxvYWQgbWVkaWEge2l0ZW1bJ29yaWduYW1lJ119IHdpdGgge3VybH0iCiAgICAgICAgICAgICkKICAgICAgICAgICAgY29tbW9uX2ds +b2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICAgICAgZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IFthdHRlbXB0IHtfYXR0ZW1wdC5nZXQoKX0ve2dldF9kb3du +bG9hZF9yZXRyaWVzKCl9XSBkb3dubG9hZCB0ZW1wIHBhdGgge3BsYWNlaG9sZGVyT2JqLnRlbXBmaWxlcGF0aH0iCiAgICAgICAgICAgICkKICAgICAgICAg +ICAgcmV0dXJuIGF3YWl0IHNlbGYuX3NlbmRfcmVxX2lubmVyKGMsIGVsZSwgaXRlbSwgcGxhY2Vob2xkZXJPYmopCiAgICAgICAgZXhjZXB0IE9TRXJyb3Ig +YXMgRToKICAgICAgICAgICAgcmFpc2UgRQogICAgICAgIGV4Y2VwdCBFeGNlcHRpb24gYXMgRToKICAgICAgICAgICAgcmFpc2UgRQoKICAgIGFzeW5jIGRl +ZiBfc2VuZF9yZXFfaW5uZXIoc2VsZiwgYywgZWxlLCBpdGVtLCBwbGFjZWhvbGRlck9iaik6CiAgICAgICAgdG90YWwgPSBOb25lCiAgICAgICAgdHJ5OgoK +ICAgICAgICAgICAgcmVzdW1lX3NpemUgPSBzZWxmLl9nZXRfcmVzdW1lX3NpemUocGxhY2Vob2xkZXJPYmosIG1lZGlhdHlwZT1lbGUubWVkaWF0eXBlKQog +ICAgICAgICAgICBoZWFkZXJzID0gc2VsZi5fZ2V0X3Jlc3VtZV9oZWFkZXIocmVzdW1lX3NpemUsIGl0ZW1bInRvdGFsIl0pCiAgICAgICAgICAgICMgcmVz +ZXQgdG90YWwKICAgICAgICAgICAgdG90YWwgPSBOb25lCiAgICAgICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy5kZWJ1ZyhmIntnZXRfbWVkaWFsb2coZWxl +KX0gcmVzdW1lIGhlYWRlciB7aGVhZGVyc30iKQogICAgICAgICAgICBwYXJhbXMgPSBnZXRfYWx0X3BhcmFtcyhlbGUpCiAgICAgICAgICAgIHVybCA9IF9t +cGRfc2VnbWVudF91cmwoZWxlLm1wZCwgaXRlbVsib3JpZ25hbWUiXSkKICAgICAgICAgICAgaGVhZGVycyA9IHsiQ29va2llIjogZiJ7ZWxlLmhsc19oZWFk +ZXJ9e2F1dGhfcmVxdWVzdHMuZ2V0X2Nvb2tpZXNfc3RyKCl9In0KICAgICAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICAg +ICAgZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IFthdHRlbXB0IHtzZWxmLl9hbHRfYXR0ZW1wdF9nZXQoaXRlbSkuZ2V0KCl9L3tnZXRfZG93bmxvYWRfcmV0cmll +cygpfV0gRG93bmxvYWRpbmcgbWVkaWEgd2l0aCB1cmwgIHt1cmx9IgogICAgICAgICAgICApCiAgICAgICAgICAgIGFzeW5jIHdpdGggYy5yZXF1ZXN0c19h +c3luYygKICAgICAgICAgICAgICAgIHVybD11cmwsCiAgICAgICAgICAgICAgICBoZWFkZXJzPWhlYWRlcnMsCiAgICAgICAgICAgICAgICBwYXJhbXM9cGFy +YW1zLAogICAgICAgICAgICAgICAgIyBhY3Rpb249W0ZPUkNFRF9ORVcsU0lHTl0gaWYgY29uc3RhbnRzLmdldGF0dHIoIkFMVF9GT1JDRV9LRVkiKSBlbHNl +IE5vbmUKICAgICAgICAgICAgKSBhcyBsOgogICAgICAgICAgICAgICAgY29udGVudF90eXBlID0gKGwuaGVhZGVycy5nZXQoImNvbnRlbnQtdHlwZSIpIG9y +ICIiKS5sb3dlcigpCiAgICAgICAgICAgICAgICBjbF9yYXcgPSBsLmhlYWRlcnMuZ2V0KCJjb250ZW50LWxlbmd0aCIpCiAgICAgICAgICAgICAgICB0cnk6 +CiAgICAgICAgICAgICAgICAgICAgdG90YWwgPSBpbnQoY2xfcmF3KSBpZiBjbF9yYXcgaXMgbm90IE5vbmUgZWxzZSBOb25lCiAgICAgICAgICAgICAgICBl +eGNlcHQgKFR5cGVFcnJvciwgVmFsdWVFcnJvcik6CiAgICAgICAgICAgICAgICAgICAgdG90YWwgPSBOb25lCiAgICAgICAgICAgICAgICBpdGVtWyJ0b3Rh +bCJdID0gdG90YWwKCiAgICAgICAgICAgICAgICBpZiBhbnkoCiAgICAgICAgICAgICAgICAgICAgeCBpbiBjb250ZW50X3R5cGUKICAgICAgICAgICAgICAg +ICAgICBmb3IgeCBpbiAoImRhc2greG1sIiwgIm1wZWd1cmwiLCAiYXBwbGljYXRpb24veG1sIiwgInRleHQveG1sIikKICAgICAgICAgICAgICAgICk6CiAg +ICAgICAgICAgICAgICAgICAgcmFpc2UgRXhjZXB0aW9uKAogICAgICAgICAgICAgICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gRFJNIHNlZ21l +bnQgcmV0dXJuZWQgcGxheWxpc3QgIgogICAgICAgICAgICAgICAgICAgICAgICBmImNvbnRlbnQtdHlwZT17Y29udGVudF90eXBlIXJ9IHVybD17dXJsfSIK +ICAgICAgICAgICAgICAgICAgICApCiAgICAgICAgICAgICAgICBpZiBub3QgdG90YWwgb3IgdG90YWwgPD0gMDoKICAgICAgICAgICAgICAgICAgICByYWlz +ZSBFeGNlcHRpb24oCiAgICAgICAgICAgICAgICAgICAgICAgIGYie2dldF9tZWRpYWxvZyhlbGUpfSBEUk0gc2VnbWVudCBlbXB0eSBzaXplICIKICAgICAg +ICAgICAgICAgICAgICAgICAgZiIoY29udGVudC10eXBlPXtjb250ZW50X3R5cGUgb3IgJ3Vua25vd24nfSwgdXJsPXt1cmx9KSIKICAgICAgICAgICAgICAg +ICAgICApCgogICAgICAgICAgICAgICAgZGF0YSA9IHsKICAgICAgICAgICAgICAgICAgICAiY29udGVudC10b3RhbCI6IHRvdGFsLAogICAgICAgICAgICAg +ICAgICAgICJjb250ZW50LXR5cGUiOiBsLmhlYWRlcnMuZ2V0KCJjb250ZW50LXR5cGUiKSwKICAgICAgICAgICAgICAgIH0KCiAgICAgICAgICAgICAgICBj +b21tb25fZ2xvYmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgICAgICAgICAgZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IGRhdGEgZnJvbSByZXF1ZXN0IHtkYXRh +fSIKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy5kZWJ1ZygKICAgICAgICAgICAgICAgICAgICBmIntnZXRf +bWVkaWFsb2coZWxlKX0gdG90YWwgZnJvbSByZXF1ZXN0IHtmb3JtYXRfc2l6ZShkYXRhLmdldCgnY29udGVudC10b3RhbCcpKSBpZiBkYXRhLmdldCgnY29u +dGVudC10b3RhbCcpIGVsc2UgJ3Vua25vd24nfSIKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIGF3YWl0IHNlbGYuX3RvdGFsX2NoYW5nZV9o +ZWxwZXIoTm9uZSwgdG90YWwpCiAgICAgICAgICAgICAgICBhd2FpdCBzZWxmLl9zZXRfZGF0YShlbGUsIGl0ZW0sIGRhdGEpCgogICAgICAgICAgICAgICAg +dGVtcF9maWxlX2xvZ2dlcihwbGFjZWhvbGRlck9iaiwgZWxlKQogICAgICAgICAgICAgICAgaWYgYXdhaXQgc2VsZi5fY2hlY2tfZm9yY2VkX3NraXAoZWxl +LCB0b3RhbCkgPT0gMDoKICAgICAgICAgICAgICAgICAgICBpdGVtWyJ0b3RhbCJdID0gMAogICAgICAgICAgICAgICAgICAgIHRvdGFsID0gaXRlbVsidG90 +YWwiXQogICAgICAgICAgICAgICAgICAgIGF3YWl0IHNlbGYuX3RvdGFsX2NoYW5nZV9oZWxwZXIodG90YWwsIDApCiAgICAgICAgICAgICAgICAgICAgcmV0 +dXJuIGl0ZW0KICAgICAgICAgICAgICAgIGVsaWYgdG90YWwgIT0gcmVzdW1lX3NpemU6CiAgICAgICAgICAgICAgICAgICAgYXdhaXQgc2VsZi5fZG93bmxv +YWRfZmlsZW9iamVjdF93cml0ZXIoCiAgICAgICAgICAgICAgICAgICAgICAgIHRvdGFsLCBsLCBlbGUsIHBsYWNlaG9sZGVyT2JqLCBpdGVtCiAgICAgICAg +ICAgICAgICAgICAgKQoKICAgICAgICAgICAgYXdhaXQgc2VsZi5fc2l6ZV9jaGVja2VyKHBsYWNlaG9sZGVyT2JqLnRlbXBmaWxlcGF0aCwgZWxlLCB0b3Rh +bCkKICAgICAgICAgICAgcmV0dXJuIGl0ZW0KICAgICAgICBleGNlcHQgRXhjZXB0aW9uIGFzIEU6CiAgICAgICAgICAgIGF3YWl0IHNlbGYuX3RvdGFsX2No +YW5nZV9oZWxwZXIodG90YWwsIDApIGlmIHRvdGFsIGVsc2UgTm9uZQogICAgICAgICAgICByYWlzZSBFCgogICAgYXN5bmMgZGVmIF9kb3dubG9hZF9maWxl +b2JqZWN0X3dyaXRlcihzZWxmLCB0b3RhbCwgbCwgZWxlLCBwbGFjZWhvbGRlck9iaiwgaXRlbSk6CiAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVn +KAogICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gW2F0dGVtcHQge3NlbGYuX2FsdF9hdHRlbXB0X2dldChpdGVtKS5nZXQoKX0ve2dldF9kb3du +bG9hZF9yZXRyaWVzKCl9XSB3cml0aW5nIG1lZGlhIHRvIGRpc2siCiAgICAgICAgKQogICAgICAgIGlmIHRvdGFsID4gY29uc3RhbnRzLmdldGF0dHIoIk1B +WF9SRUFEX1NJWkUiKToKICAgICAgICAgICAgYXdhaXQgc2VsZi5fZG93bmxvYWRfZmlsZW9iamVjdF93cml0ZXJfc3RyZWFtZXIoZWxlLHRvdGFsLCBsLCBw +bGFjZWhvbGRlck9iaikKICAgICAgICBlbHNlOgogICAgICAgICAgICBhd2FpdCBzZWxmLl9kb3dubG9hZF9maWxlb2JqZWN0X3dyaXRlcl9yZWFkZXIoZWxl +LHRvdGFsLCBsLCBwbGFjZWhvbGRlck9iaikKICAgICAgICBjb21tb25fZ2xvYmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgIGYie2dldF9tZWRpYWxvZyhl +bGUpfSBbYXR0ZW1wdCB7c2VsZi5fYWx0X2F0dGVtcHRfZ2V0KGl0ZW0pLmdldCgpfS97Z2V0X2Rvd25sb2FkX3JldHJpZXMoKX1dIGZpbmlzaGVkIHdyaXRp +bmcgbWVkaWEgdG8gZGlzayIKICAgICAgICApCgogICAgYXN5bmMgZGVmIF9kb3dubG9hZF9maWxlb2JqZWN0X3dyaXRlcl9yZWFkZXIoc2VsZiwgZWxlLCB0 +b3RhbCwgcmVzLCBwbGFjZWhvbGRlck9iaik6CgogICAgICAgIHRhc2sxID0gYXdhaXQgc2VsZi5fYWRkX2Rvd25sb2FkX2pvYl90YXNrKAogICAgICAgICAg +ICBlbGUsIHRvdGFsPXRvdGFsLCBwbGFjZWhvbGRlck9iaj1wbGFjZWhvbGRlck9iagogICAgICAgICkKICAgICAgICBmaWxlb2JqZWN0ID0gYXdhaXQgYWlv +ZmlsZXMub3BlbihwbGFjZWhvbGRlck9iai50ZW1wZmlsZXBhdGgsICJhYiIpLl9fYWVudGVyX18oKQogICAgICAgIHRyeToKICAgICAgICAgICAgYXdhaXQg +ZmlsZW9iamVjdC53cml0ZShhd2FpdCByZXMucmVhZF8oKSkKICAgICAgICBleGNlcHQgRXhjZXB0aW9uIGFzIEU6CiAgICAgICAgICAgIHJhaXNlIEUKICAg +ICAgICBmaW5hbGx5OgogICAgICAgICAgICAjIENsb3NlIGZpbGUgaWYgbmVlZGVkCiAgICAgICAgICAgIHRyeToKICAgICAgICAgICAgICAgIGF3YWl0IGZp +bGVvYmplY3QuY2xvc2UoKQogICAgICAgICAgICBleGNlcHQgRXhjZXB0aW9uIGFzIEU6CiAgICAgICAgICAgICAgICByYWlzZSBFCiAgICAgICAgICAgIHRy +eToKICAgICAgICAgICAgICAgIGF3YWl0IHNlbGYuX3JlbW92ZV9kb3dubG9hZF9qb2JfdGFzayh0YXNrMSwgZWxlKQogICAgICAgICAgICBleGNlcHQgRXhj +ZXB0aW9uIGFzIEU6CiAgICAgICAgICAgICAgICByYWlzZSBFCgogICAgYXN5bmMgZGVmIF9kb3dubG9hZF9maWxlb2JqZWN0X3dyaXRlcl9zdHJlYW1lcigK +ICAgICAgICBzZWxmLCBlbGUsIHRvdGFsLCByZXMsIHBsYWNlaG9sZGVyT2JqCiAgICApOgoKICAgICAgICB0YXNrMSA9IGF3YWl0IHNlbGYuX2FkZF9kb3du +bG9hZF9qb2JfdGFzayhlbGUsIHRvdGFsLCBwbGFjZWhvbGRlck9iaikKICAgICAgICBmaWxlb2JqZWN0ID0gYXdhaXQgYWlvZmlsZXMub3BlbihwbGFjZWhv +bGRlck9iai50ZW1wZmlsZXBhdGgsICJhYiIpLl9fYWVudGVyX18oKQogICAgICAgIGNodW5rX3NpemUgPSBnZXRfaWRlYWxfY2h1bmtfc2l6ZSh0b3RhbCwg +cGxhY2Vob2xkZXJPYmoudGVtcGZpbGVwYXRoKQogICAgICAgIHRyeToKICAgICAgICAgICAgYXN5bmMgZm9yIGNodW5rIGluIHJlcy5pdGVyX2NodW5rZWQo +Y2h1bmtfc2l6ZSk6CiAgICAgICAgICAgICAgICBhd2FpdCBmaWxlb2JqZWN0LndyaXRlKGNodW5rKQogICAgICAgICAgICAgICAgc2VuZF9jaHVua19tc2co +ZWxlLCB0b3RhbCwgcGxhY2Vob2xkZXJPYmopCiAgICAgICAgZXhjZXB0IEV4Y2VwdGlvbiBhcyBFOgogICAgICAgICAgICByYWlzZSBFCiAgICAgICAgZmlu +YWxseToKICAgICAgICAgICAgIyBDbG9zZSBmaWxlIGlmIG5lZWRlZAogICAgICAgICAgICB0cnk6CiAgICAgICAgICAgICAgICBhd2FpdCBmaWxlb2JqZWN0 +LmNsb3NlKCkKICAgICAgICAgICAgZXhjZXB0IEV4Y2VwdGlvbiBhcyBFOgogICAgICAgICAgICAgICAgcmFpc2UgRQoKICAgICAgICAgICAgdHJ5OgogICAg +ICAgICAgICAgICAgYXdhaXQgc2VsZi5fcmVtb3ZlX2Rvd25sb2FkX2pvYl90YXNrKHRhc2sxLCBlbGUpCiAgICAgICAgICAgIGV4Y2VwdCBFeGNlcHRpb24g +YXMgRToKICAgICAgICAgICAgICAgIHJhaXNlIEUKCiAgICBhc3luYyBkZWYgX2hhbmRsZV9yZXN1bHRfYWx0KAogICAgICAgIHNlbGYsIHNoYXJlZFBsYWNl +aG9sZGVyT2JqLCBlbGUsIGF1ZGlvLCB2aWRlbywgdXNlcm5hbWUsIG1vZGVsX2lkCiAgICApOgogICAgICAgIHRlbXBQbGFjZWhvbGRlciA9IGF3YWl0IHBs +YWNlaG9sZGVyLnRlbXBGaWxlUGxhY2Vob2xkZXIoCiAgICAgICAgICAgIGVsZSwgZiJ0ZW1wX3tlbGUuaWQgb3IgYXdhaXQgZWxlLmZpbmFsX2ZpbGVuYW1l +fS5tcDQiCiAgICAgICAgKS5pbml0KCkKICAgICAgICB0ZW1wX3BhdGggPSB0ZW1wUGxhY2Vob2xkZXIudGVtcGZpbGVwYXRoCiAgICAgICAgdGVtcF9wYXRo +LnVubGluayhtaXNzaW5nX29rPVRydWUpCiAgICAgICAgdCA9IHJ1bigKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgc2V0dGluZ3MuZ2V0X2ZmbXBl +ZygpLAogICAgICAgICAgICAgICAgIi1pIiwKICAgICAgICAgICAgICAgIHN0cih2aWRlb1sicGF0aCJdKSwKICAgICAgICAgICAgICAgICItaSIsCiAgICAg +ICAgICAgICAgICBzdHIoYXVkaW9bInBhdGgiXSksCiAgICAgICAgICAgICAgICAiLWMiLAogICAgICAgICAgICAgICAgImNvcHkiLAogICAgICAgICAgICAg +ICAgIi1tb3ZmbGFncyIsCiAgICAgICAgICAgICAgICAidXNlX21ldGFkYXRhX3RhZ3MiLAogICAgICAgICAgICAgICAgc3RyKHRlbXBfcGF0aCksCiAgICAg +ICAgICAgIF0sCiAgICAgICAgKQogICAgICAgIGlmIHQuc3RkZXJyLmRlY29kZSgpLmZpbmQoIk91dHB1dCIpID09IC0xOgogICAgICAgICAgICBjb21tb25f +Z2xvYmFscy5sb2cuZGVidWcoZiJ7Y29tbW9uX2xvZ3MuZ2V0X21lZGlhbG9nKGVsZSl9IGZmbXBlZyBmYWlsZWQiKQogICAgICAgICAgICBjb21tb25fZ2xv +YmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgICAgICBmIntjb21tb25fbG9ncy5nZXRfbWVkaWFsb2coZWxlKX0gZmZtcGVnIHt0LnN0ZGVyci5kZWNvZGUo +KX0iCiAgICAgICAgICAgICkKICAgICAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICAgICAgZiJ7Y29tbW9uX2xvZ3MuZ2V0 +X21lZGlhbG9nKGVsZSl9IGZmbXBlZyB7dC5zdGRvdXQuZGVjb2RlKCl9IgogICAgICAgICAgICApCgogICAgICAgIHZpZGVvWyJwYXRoIl0udW5saW5rKG1p +c3Npbmdfb2s9VHJ1ZSkKICAgICAgICBhdWRpb1sicGF0aCJdLnVubGluayhtaXNzaW5nX29rPVRydWUpCiAgICAgICAKCiAgICAgICAgY29tbW9uX2dsb2Jh +bHMubG9nLmRlYnVnKAogICAgICAgICAgICBmIk1vdmluZyBpbnRlcm1lZGlhdGUgcGF0aCB7dGVtcF9wYXRofSB0byB7c2hhcmVkUGxhY2Vob2xkZXJPYmou +dHJ1bmljYXRlZF9maWxlcGF0aH0iCiAgICAgICAgKQogICAgICAgIGNvbW1vbl9wYXRocy5tb3ZlSGVscGVyKAogICAgICAgICAgICB0ZW1wX3BhdGgsIHNo +YXJlZFBsYWNlaG9sZGVyT2JqLnRydW5pY2F0ZWRfZmlsZXBhdGgsIGVsZQogICAgICAgICkKICAgICAgICAoCiAgICAgICAgICAgIGNvbW1vbl9wYXRocy5h +ZGRHbG9iYWxEaXIoc2hhcmVkUGxhY2Vob2xkZXJPYmouZmlsZWRpcikKICAgICAgICAgICAgaWYgc3lzdGVtLmdldF9wYXJlbnRfcHJvY2VzcygpCiAgICAg +ICAgICAgIGVsc2UgY29tbW9uX3BhdGhzLmFkZExvY2FsRGlyKHNoYXJlZFBsYWNlaG9sZGVyT2JqLmZpbGVkaXIpCiAgICAgICAgKQogICAgICAgIGlmIGVs +ZS5wb3N0ZGF0ZToKICAgICAgICAgICAgbmV3RGF0ZSA9IGRhdGVzLmNvbnZlcnRfbG9jYWxfdGltZShlbGUucG9zdGRhdGUpCiAgICAgICAgICAgIGNvbW1v +bl9nbG9iYWxzLmxvZy5kZWJ1ZygKICAgICAgICAgICAgICAgIGYie2NvbW1vbl9sb2dzLmdldF9tZWRpYWxvZyhlbGUpfSBBdHRlbXB0IHRvIHNldCBEYXRl +IHRvIHthcnJvdy5nZXQobmV3RGF0ZSkuZm9ybWF0KCdZWVlZLU1NLUREIEhIOm1tJyl9IgogICAgICAgICAgICApCiAgICAgICAgICAgIGNvbW1vbl9wYXRo +cy5zZXRfdGltZShzaGFyZWRQbGFjZWhvbGRlck9iai50cnVuaWNhdGVkX2ZpbGVwYXRoLCBuZXdEYXRlKQogICAgICAgICAgICBjb21tb25fZ2xvYmFscy5s +b2cuZGVidWcoCiAgICAgICAgICAgICAgICBmIntjb21tb25fbG9ncy5nZXRfbWVkaWFsb2coZWxlKX0gRGF0ZSBzZXQgdG8ge2Fycm93LmdldChzaGFyZWRQ +bGFjZWhvbGRlck9iai50cnVuaWNhdGVkX2ZpbGVwYXRoLnN0YXQoKS5zdF9tdGltZSkuZm9ybWF0KCdZWVlZLU1NLUREIEhIOm1tJyl9IgogICAgICAgICAg +ICApCiAgICAgICAgaWYgZWxlLmlkOgogICAgICAgICAgICBhd2FpdCBkb3dubG9hZF9tZWRpYV91cGRhdGUoCiAgICAgICAgICAgICAgICBlbGUsCiAgICAg +ICAgICAgICAgICBmaWxlcGF0aD1zaGFyZWRQbGFjZWhvbGRlck9iai50cnVuaWNhdGVkX2ZpbGVwYXRoLAogICAgICAgICAgICAgICAgbW9kZWxfaWQ9bW9k +ZWxfaWQsCiAgICAgICAgICAgICAgICB1c2VybmFtZT11c2VybmFtZSwKICAgICAgICAgICAgICAgIGRvd25sb2FkZWQ9VHJ1ZSwKICAgICAgICAgICAgICAg +IGhhc2hkYXRhPWF3YWl0IGNvbW1vbi5nZXRfaGFzaCgKICAgICAgICAgICAgICAgICAgICBzaGFyZWRQbGFjZWhvbGRlck9iaiwgbWVkaWF0eXBlPWVsZS5t +ZWRpYXR5cGUKICAgICAgICAgICAgICAgICksCiAgICAgICAgICAgICAgICBzaXplPXNoYXJlZFBsYWNlaG9sZGVyT2JqLnNpemUsCiAgICAgICAgICAgICkK +ICAgICAgICBjb21tb24uYWRkX2FkZGl0aW9uYWxfZGF0YShzaGFyZWRQbGFjZWhvbGRlck9iaiwgZWxlKQogICAgICAgIHJldHVybiBlbGUubWVkaWF0eXBl +LCB2aWRlb1sidG90YWwiXSArIGF1ZGlvWyJ0b3RhbCJdCgogICAgYXN5bmMgZGVmIF9yZXN1bWVfZGF0YV9oYW5kbGVyX2FsdChzZWxmLCBkYXRhLCBpdGVt +LCBlbGUsIHBsYWNlaG9sZGVyT2JqKToKICAgICAgICBjb21tb25fZ2xvYmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgIGYie2dldF9tZWRpYWxvZyhlbGUp +fSBbYXR0ZW1wdCB7Y29tbW9uX2dsb2JhbHMuYXR0ZW1wdC5nZXQoKX0ve2dldF9kb3dubG9hZF9yZXRyaWVzKCl9XSB1c2luZyBkYXRhIGZvciBwb3NzaWJs +ZSBkb3dubG9hZCByZXN1bXB0aW9uIgogICAgICAgICkKICAgICAgICBjb21tb25fZ2xvYmFscy5sb2cuZGVidWcoZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IERh +dGEgZnJvbSBjYWNoZXtkYXRhfSIpCiAgICAgICAgY29tbW9uX2dsb2JhbHMubG9nLmRlYnVnKAogICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0g +VG90YWwgc2l6ZSBmcm9tIGNhY2hlIHtmb3JtYXRfc2l6ZShkYXRhLmdldCgnY29udGVudC10b3RhbCcpKSBpZiBkYXRhLmdldCgnY29udGVudC10b3RhbCcp +IGVsc2UgJ3Vua25vd24nfSIKICAgICAgICApCiAgICAgICAgdG90YWwgPSBpbnQoZGF0YS5nZXQoImNvbnRlbnQtdG90YWwiKSkgaWYgZGF0YS5nZXQoImNv +bnRlbnQtdG90YWwiKSBlbHNlIE5vbmUKICAgICAgICBpdGVtWyJ0b3RhbCJdID0gdG90YWwKICAgICAgICByZXN1bWVfc2l6ZSA9IHNlbGYuX2dldF9yZXN1 +bWVfc2l6ZShwbGFjZWhvbGRlck9iaiwgbWVkaWF0eXBlPWVsZS5tZWRpYXR5cGUpCiAgICAgICAgcmVzdW1lX3NpemUgPSBzZWxmLl9yZXN1bWVfY2xlYW5l +cigKICAgICAgICAgICAgcmVzdW1lX3NpemUsIHRvdGFsLCBwbGFjZWhvbGRlck9iai50ZW1wZmlsZXBhdGgKICAgICAgICApCgogICAgICAgIGNvbW1vbl9n +bG9iYWxzLmxvZy5kZWJ1ZygKICAgICAgICAgICAgZiJ7Z2V0X21lZGlhbG9nKGVsZSl9IHJlc3VtZV9zaXplOiB7cmVzdW1lX3NpemV9ICBhbmQgdG90YWw6 +IHt0b3RhbCB9IgogICAgICAgICkKCiAgICAgICAgaWYgdG90YWwgaXMgTm9uZSBvciBpbnQodG90YWwpIDw9IDA6CiAgICAgICAgICAgIHRyeToKICAgICAg +ICAgICAgICAgIGF3YWl0IGFzeW5jaW8uZ2V0X2V2ZW50X2xvb3AoKS5ydW5faW5fZXhlY3V0b3IoCiAgICAgICAgICAgICAgICAgICAgY29tbW9uX2dsb2Jh +bHMudGhyZWFkLAogICAgICAgICAgICAgICAgICAgIHBhcnRpYWwoCiAgICAgICAgICAgICAgICAgICAgICAgIGNhY2hlLnNldCwKICAgICAgICAgICAgICAg +ICAgICAgICAgZiJ7aXRlbVsnbmFtZSddfV97ZWxlLmlkfV97ZWxlLnVzZXJuYW1lfV9oZWFkZXJzIiwKICAgICAgICAgICAgICAgICAgICAgICAgTm9uZSwK +ICAgICAgICAgICAgICAgICAgICApLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICBleGNlcHQgRXhjZXB0aW9uOgogICAgICAgICAgICAgICAgcGFz +cwogICAgICAgICAgICByZXR1cm4gaXRlbSwgRmFsc2UKCiAgICAgICAgaWYgYXdhaXQgc2VsZi5fY2hlY2tfZm9yY2VkX3NraXAoZWxlLCB0b3RhbCkgPT0g +MDoKICAgICAgICAgICAgaXRlbVsidG90YWwiXSA9IDAKICAgICAgICAgICAgcmV0dXJuIGl0ZW0sIFRydWUKICAgICAgICBlbGlmIHRvdGFsID09IHJlc3Vt +ZV9zaXplOgogICAgICAgICAgICBjb21tb25fZ2xvYmFscy5sb2cuZGVidWcoCiAgICAgICAgICAgICAgICBmIntnZXRfbWVkaWFsb2coZWxlKX0gdG90YWw9 +PXJlc3VtZV9zaXplIHNraXBwaW5nIGRvd25sb2FkIgogICAgICAgICAgICApCiAgICAgICAgICAgIHRlbXBfZmlsZV9sb2dnZXIocGxhY2Vob2xkZXJPYmos +IGVsZSkKICAgICAgICAgICAgaWYgc2VsZi5fYWx0X2F0dGVtcHRfZ2V0KGl0ZW0pLmdldCgpID09IDA6CiAgICAgICAgICAgICAgICBwYXNzCiAgICAgICAg +ICAgIGF3YWl0IHNlbGYuX3RvdGFsX2NoYW5nZV9oZWxwZXIoTm9uZSwgdG90YWwpCiAgICAgICAgICAgIHJldHVybiBpdGVtLCBUcnVlCiAgICAgICAgZWxp +ZiB0b3RhbCAhPSByZXN1bWVfc2l6ZToKICAgICAgICAgICAgcmV0dXJuIGl0ZW0sIEZhbHNlCgogICAgYXN5bmMgZGVmIF9mcmVzaF9kYXRhX2hhbmRsZXJf +YWx0KHNlbGYsIGl0ZW0sIGVsZSwgcGxhY2Vob2xkZXJPYmopOgogICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy5kZWJ1ZygKICAgICAgICAgICAgZiJ7Z2V0 +X21lZGlhbG9nKGVsZSl9IFthdHRlbXB0IHtjb21tb25fZ2xvYmFscy5hdHRlbXB0LmdldCgpfS97Z2V0X2Rvd25sb2FkX3JldHJpZXMoKX1dIGZyZXNoIGRv +d25sb2FkIGZvciBtZWRpYSIKICAgICAgICApCiAgICAgICAgcmVzdW1lX3NpemUgPSBzZWxmLl9nZXRfcmVzdW1lX3NpemUocGxhY2Vob2xkZXJPYmosIG1l +ZGlhdHlwZT1lbGUubWVkaWF0eXBlKQogICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy5kZWJ1ZyhmIntnZXRfbWVkaWFsb2coZWxlKX0gcmVzdW1lX3NpemU6 +IHtyZXN1bWVfc2l6ZX0iKQogICAgICAgIHJldHVybiBpdGVtLCBGYWxzZQoKICAgIGRlZiBfYWx0X2F0dGVtcHRfZ2V0KHNlbGYsIGl0ZW0pOgogICAgICAg +IGlmIGl0ZW1bInR5cGUiXSA9PSAidmlkZW8iOgogICAgICAgICAgICByZXR1cm4gY29tbW9uX2dsb2JhbHMuYXR0ZW1wdAogICAgICAgIGlmIGl0ZW1bInR5 +cGUiXSA9PSAiYXVkaW8iOgogICAgICAgICAgICByZXR1cm4gY29tbW9uX2dsb2JhbHMuYXR0ZW1wdDIKCiAgICBhc3luYyBkZWYgX2dldF9kYXRhKHNlbGYs +IGVsZSwgaXRlbSk6CiAgICAgICAgZGF0YSA9IGF3YWl0IGFzeW5jaW8uZ2V0X2V2ZW50X2xvb3AoKS5ydW5faW5fZXhlY3V0b3IoCiAgICAgICAgICAgIGNv +bW1vbl9nbG9iYWxzLnRocmVhZCwKICAgICAgICAgICAgcGFydGlhbChjYWNoZS5nZXQsIGYie2l0ZW1bJ25hbWUnXX1fe2VsZS5pZH1fe2VsZS51c2VybmFt +ZX1faGVhZGVycyIpLAogICAgICAgICkKICAgICAgICByZXR1cm4gZGF0YQoKICAgIGFzeW5jIGRlZiBfc2V0X2RhdGEoc2VsZiwgZWxlLCBpdGVtLCBkYXRh +KToKICAgICAgICBkYXRhID0gYXdhaXQgYXN5bmNpby5nZXRfZXZlbnRfbG9vcCgpLnJ1bl9pbl9leGVjdXRvcigKICAgICAgICAgICAgY29tbW9uX2dsb2Jh +bHMudGhyZWFkLAogICAgICAgICAgICBwYXJ0aWFsKGNhY2hlLnNldCwgZiJ7aXRlbVsnbmFtZSddfV97ZWxlLmlkfV97ZWxlLnVzZXJuYW1lfV9oZWFkZXJz +IiwgZGF0YSksCiAgICAgICAgKQogICAgICAgIHJldHVybiBkYXRhCgogICAgZGVmIF9nZXRfaXRlbV90b3RhbChzZWxmLCBpdGVtKToKICAgICAgICByZXR1 +cm4gaXRlbVsicGF0aCJdLmFic29sdXRlKCkuc3RhdCgpLnN0X3NpemUKCiAgICBhc3luYyBkZWYgX21lZGlhX2l0ZW1fcG9zdF9wcm9jZXNzX2FsdChzZWxm +LCBhdWRpbywgdmlkZW8sIGVsZSwgdXNlcm5hbWUsIG1vZGVsX2lkKToKICAgICAgICBhdWRpb190b3RhbCA9IGF1ZGlvWyJ0b3RhbCJdIGlmIGF1ZGlvIGVs +c2UgMAogICAgICAgIHZpZGVvX3RvdGFsID0gdmlkZW9bInRvdGFsIl0gaWYgdmlkZW8gZWxzZSAwCgogICAgICAgIGlmIChhdWRpb190b3RhbCArIHZpZGVv +X3RvdGFsKSA9PSAwOgogICAgICAgICAgICBpZiBlbGUubWVkaWF0eXBlLmNhcGl0YWxpemUoKSA9PSAiRm9yY2VkX3NraXBwZWQiOgogICAgICAgICAgICAg +ICAgcmV0dXJuIGVsZS5tZWRpYXR5cGUsIDAKICAgICAgICAgICAgcmFpc2UgRXhjZXB0aW9uKAogICAgICAgICAgICAgICAgZiJ7Z2V0X21lZGlhbG9nKGVs +ZSl9IERSTSBkb3dubG9hZCBwcm9kdWNlZCAwIGJ5dGVzICIKICAgICAgICAgICAgICAgICIoYXVkaW8rdmlkZW8pLiBOb3QgbWFya2luZyBhcyBkb3dubG9h +ZGVkLiIKICAgICAgICAgICAgKQogICAgICAgIGZvciBtIGluIFthdWRpbywgdmlkZW9dOgogICAgICAgICAgICBtWyJ0b3RhbCJdID0gc2VsZi5fZ2V0X2l0 +ZW1fdG90YWwobSkKCiAgICAgICAgZm9yIG0gaW4gW2F1ZGlvLCB2aWRlb106CiAgICAgICAgICAgIGlmIG5vdCBpc2luc3RhbmNlKG0sIGRpY3QpOgogICAg +ICAgICAgICAgICAgcmV0dXJuIG0KICAgICAgICAgICAgYXdhaXQgc2VsZi5fc2l6ZV9jaGVja2VyKG1bInBhdGgiXSwgZWxlLCBtWyJ0b3RhbCJdKQoKICAg +IGFzeW5jIGRlZiBfbWVkaWFfaXRlbV9rZXlzX2FsdChzZWxmLCBjLCBhdWRpbywgdmlkZW8sIGVsZSk6CiAgICAgICAgYXN5bmMgZm9yIF8gaW4gZG93bmxv +YWRfcmV0cnkoKToKICAgICAgICAgICAgd2l0aCBfOgogICAgICAgICAgICAgICAgdHJ5OgogICAgICAgICAgICAgICAgICAgIGZvciBpdGVtIGluIFthdWRp +bywgdmlkZW9dOgogICAgICAgICAgICAgICAgICAgICAgICBpdGVtID0gYXdhaXQga2V5aGVscGVycy51bl9lbmNyeXB0KGl0ZW0sIGMsIGVsZSkKICAgICAg +ICAgICAgICAgIGV4Y2VwdCBFeGNlcHRpb24gYXMgRToKICAgICAgICAgICAgICAgICAgICBjb21tb25fZ2xvYmFscy5sb2cudHJhY2ViYWNrXyhFKQogICAg +ICAgICAgICAgICAgICAgIGNvbW1vbl9nbG9iYWxzLmxvZy50cmFjZWJhY2tfKHRyYWNlYmFjay5mb3JtYXRfZXhjKCkpCiAgICAgICAgICAgICAgICAgICAg +cmFpc2UgRQoKCgogICAgYXN5bmMgZGVmIF9hZGRfZG93bmxvYWRfam9iX3Rhc2soc2VsZiwgZWxlLCB0b3RhbD1Ob25lLCBwbGFjZWhvbGRlck9iaj1Ob25l +KToKICAgICAgICBwYXRoc3RyID0gc3RyKHBsYWNlaG9sZGVyT2JqLnRlbXBmaWxlcGF0aCkKICAgICAgICB0YXNrMSA9IE5vbmUKICAgICAgICBpZiBub3Qg +c2VsZi5fbXVsdGk6CiAgICAgICAgICAgIHRhc2sxID0gcHJvZ3Jlc3NfdXBkYXRlci5hZGRfZG93bmxvYWRfam9iX3Rhc2soCiAgICAgICAgICAgICAgICBm +InsocGF0aHN0cls6Y29uc3RhbnRzLmdldGF0dHIoJ1BBVEhfU1RSX01BWCcpXSArICcuLi4uJykgaWYgbGVuKHBhdGhzdHIpID4gY29uc3RhbnRzLmdldGF0 +dHIoJ1BBVEhfU1RSX01BWCcpIGVsc2UgcGF0aHN0cn1cbiIsCiAgICAgICAgICAgICAgICB0b3RhbD10b3RhbCwKICAgICAgICAgICAgKQogICAgICAgIGVs +c2U6CiAgICAgICAgICAgIGF3YWl0IHNlbmRfbXNnKAogICAgICAgICAgICAgICAgcGFydGlhbCgKICAgICAgICAgICAgICAgICAgICBwcm9ncmVzc191cGRh +dGVyLmFkZF9kb3dubG9hZF9qb2JfbXVsdGlfdGFzaywKICAgICAgICAgICAgICAgICAgICBmInsocGF0aHN0cls6Y29uc3RhbnRzLmdldGF0dHIoJ1BBVEhf +U1RSX01BWCcpXSArICcuLi4uJykgaWYgbGVuKHBhdGhzdHIpID4gY29uc3RhbnRzLmdldGF0dHIoJ1BBVEhfU1RSX01BWCcpIGVsc2UgcGF0aHN0cn1cbiIs +CiAgICAgICAgICAgICAgICAgICAgZWxlLmlkLAogICAgICAgICAgICAgICAgICAgIHRvdGFsPXRvdGFsLAogICAgICAgICAgICAgICAgICAgIGZpbGU9cGxh +Y2Vob2xkZXJPYmoudGVtcGZpbGVwYXRoLAogICAgICAgICAgICAgICAgKQogICAgICAgICAgICApCiAgICAgICAgcmV0dXJuIHRhc2sxCgpkZWYgX21wZF9z +ZWdtZW50X3VybChtcGRfdXJsOiBzdHIsIHNlZ21lbnRfbmFtZTogc3RyKSAtPiBzdHI6CiAgICBwYXJ0cyA9IHVybHNwbGl0KChtcGRfdXJsIG9yICIiKS5z +dHJpcCgpKQogICAgbmFtZSA9IChzZWdtZW50X25hbWUgb3IgIiIpLnN0cmlwKCkubHN0cmlwKCIvIikKICAgIHBhdGggPSBwYXJ0cy5wYXRoIG9yICIiCiAg +ICBiYXNlX3BhdGggPSByZS5zdWIociJbXi9dK1wubXBkJCIsICIiLCBwYXRoLCBmbGFncz1yZS5JR05PUkVDQVNFKQogICAgaWYgYmFzZV9wYXRoID09IHBh +dGg6CiAgICAgICAgYmFzZV9wYXRoID0gKHBhdGgucnNwbGl0KCIvIiwgMSlbMF0gKyAiLyIpIGlmICIvIiBpbiBwYXRoIGVsc2UgIi8iCiAgICBlbGlmIG5v +dCBiYXNlX3BhdGguZW5kc3dpdGgoIi8iKToKICAgICAgICBiYXNlX3BhdGggKz0gIi8iCiAgICByZXR1cm4gdXJsdW5zcGxpdCgocGFydHMuc2NoZW1lLCBw +YXJ0cy5uZXRsb2MsIGJhc2VfcGF0aCArIG5hbWUsIHBhcnRzLnF1ZXJ5LCAiIikpCgo= +'@ +} + +function Get-DrmPatchBytes { + $b64 = Get-DrmPatchBase64 + $b64 = ($b64 -replace '\s', '') + return [Convert]::FromBase64String($b64) +} + +function Install-DrmPatch { + param( + [string]$ExePath, + [bool]$Reinstalled + ) + $script:DrmPatchStatus = "not applied" + if ($SkipDrmPatch) { + Write-ColorOutput "Skipping DRM patch (-SkipDrmPatch)." "Yellow" + $script:DrmPatchStatus = "skipped" + return $true + } + + $pkg = Get-OfScraperPackageDir -ExePath $ExePath + if (-not $pkg) { + Write-ColorOutput "Could not locate the ofscraper package directory; DRM patch was not applied." "Red" + return $false + } + + $target = Join-Path $pkg $DrmPatchRelPath + if (-not (Test-Path -LiteralPath $target)) { + Write-ColorOutput "Expected DRM file is missing: $target" "Red" + return $false + } + + if (-not $Reinstalled -and (Test-DrmPatchApplied -AltDownloadPath $target)) { + Write-ColorOutput "DRM false-success patch is already present." "Green" + $script:DrmPatchStatus = $DrmPatchId + return $true + } + + Write-ColorOutput "" "White" + Write-ColorOutput "Applying DRM false-success patch ($DrmPatchId)..." "Yellow" + Write-ColorOutput "Empty DRM segments and 0-byte audio+video will no longer be marked downloaded." "Gray" + + $stamp = Get-Date -Format "yyyyMMdd_HHmmss" + $backupRoot = Join-Path $env:TEMP "ofscraper_patch_backup_${DrmPatchId}_$stamp" + $backupFile = Join-Path $backupRoot $DrmPatchRelPath + New-Item -ItemType Directory -Force -Path (Split-Path -Parent $backupFile) | Out-Null + Copy-Item -LiteralPath $target -Destination $backupFile -Force + + try { + $bytes = Get-DrmPatchBytes + [System.IO.File]::WriteAllBytes($target, $bytes) + } + catch { + Write-ColorOutput "DRM patch failed: $($_.Exception.Message)" "Red" + Write-ColorOutput "Restore from backup: $backupFile" "Yellow" + return $false + } + + if (-not (Test-DrmPatchApplied -AltDownloadPath $target)) { + Write-ColorOutput "DRM patch wrote a file, but the expected fix marker is missing." "Red" + Write-ColorOutput "Restore from backup: $backupFile" "Yellow" + return $false + } + + Write-ColorOutput "DRM patch applied." "Green" + Write-ColorOutput "Backup: $backupFile" "Gray" + Write-ColorOutput "If an earlier run marked protected media as downloaded with 0 bytes, rescrape that media." "Gray" + $script:DrmPatchStatus = $DrmPatchId + return $true +} + +function Pause-IfInteractive { + if ($Host.Name -eq "ConsoleHost" -and -not $Quiet) { + Write-Host "" + Write-Host "Press any key to exit..." -ForegroundColor Gray + $null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown") + } +} + +Write-ColorOutput "" "White" +Write-ColorOutput "=== SCrawler OF-Scraper installer ===" "Cyan" +Write-ColorOutput "Target: ofscraper==$OfScraperVersion + aiolimiter==$AioLimiterVersion" "Gray" +Write-ColorOutput "Python: 3.11.x or 3.12.x only (not 3.10 or older, not 3.13+)" "Gray" +Write-ColorOutput "DRM patch: $DrmPatchId (alt_download.py false-success fix)" "Gray" +Write-ColorOutput "" "White" + +$installs = @(Find-PythonInstalls) +$supported = @($installs | Where-Object { $_.Supported } | Sort-Object @{ Expression = { $_.Version.Minor } }, @{ Expression = { $_.Version.Build } } -Descending) +# Prefer 3.11 over 3.12: 3.11 is the version SCrawler/OF-Scraper docs recommend (3.11.6). +$preferred = @($supported | Where-Object { $_.Version.Minor -eq 11 } | Sort-Object Version -Descending) +if ($preferred.Count -gt 0) { + $python = $preferred[0] +} elseif ($supported.Count -gt 0) { + $python = ($supported | Sort-Object Version -Descending)[0] +} else { + $python = $null +} + +if (-not $python) { + $reason = "A supported Python was not found. Need 3.11.x or 3.12.x (not older than 3.11, not 3.13+)." + if ($PythonPath) { + $reason = "The Python executable at '$PythonPath' is missing, or is older than 3.11.x, or is 3.13+." + } + Show-PythonMissing -Reason $reason -Unsupported @($installs | Where-Object { -not $_.Supported }) + Pause-IfInteractive + exit 1 +} + +Write-ColorOutput ("Using Python {0}" -f $python.Version) "Green" +Write-ColorOutput $python.Path "Gray" + +if (-not (Test-Pip -ExePath $python.Path)) { + if (-not (Install-Pip -ExePath $python.Path)) { + Show-PythonMissing -Reason "pip is not available for this Python install and ensurepip failed." + Pause-IfInteractive + exit 1 + } + Write-ColorOutput "pip is now available." "Green" +} + +$currentOfs = Get-PipPackageVersion -ExePath $python.Path -PackageName "ofscraper" +$currentAio = Get-PipPackageVersion -ExePath $python.Path -PackageName "aiolimiter" + +if ($currentOfs) { + Write-ColorOutput "Installed ofscraper: $currentOfs" "White" +} else { + Write-ColorOutput "ofscraper is not installed." "Yellow" +} +if ($currentAio) { + Write-ColorOutput "Installed aiolimiter: $currentAio" "White" +} + +$needOfs = $Force -or ($currentOfs -ne $OfScraperVersion) + +if ($needOfs) { + Write-ColorOutput "" "White" + Write-ColorOutput "Installing ofscraper==$OfScraperVersion ..." "Yellow" + if (-not (Invoke-PipInstall -ExePath $python.Path -PipArgs @("ofscraper==$OfScraperVersion"))) { + Write-ColorOutput "Failed to install ofscraper==$OfScraperVersion" "Red" + Pause-IfInteractive + exit 1 + } +} else { + Write-ColorOutput "ofscraper $OfScraperVersion is already installed." "Green" +} + +Write-ColorOutput "" "White" +Write-ColorOutput "Pinning aiolimiter==$AioLimiterVersion (newer versions break OF-Scraper $OfScraperVersion)..." "Yellow" +if (-not (Invoke-PipInstall -ExePath $python.Path -PipArgs @("--force-reinstall", "--no-deps", "aiolimiter==$AioLimiterVersion"))) { + Write-ColorOutput "Failed to install aiolimiter==$AioLimiterVersion" "Red" + Pause-IfInteractive + exit 1 +} + +$finalOfs = Get-PipPackageVersion -ExePath $python.Path -PackageName "ofscraper" +$finalAio = Get-PipPackageVersion -ExePath $python.Path -PackageName "aiolimiter" + +if ($finalOfs -ne $OfScraperVersion) { + Write-ColorOutput "ofscraper version after install is '$finalOfs', expected $OfScraperVersion." "Red" + Pause-IfInteractive + exit 1 +} +if ($finalAio -ne $AioLimiterVersion) { + Write-ColorOutput "aiolimiter version after install is '$finalAio', expected $AioLimiterVersion." "Red" + Pause-IfInteractive + exit 1 +} + +$drmOk = Install-DrmPatch -ExePath $python.Path -Reinstalled $needOfs +if (-not $drmOk -and -not $SkipDrmPatch) { + Write-ColorOutput "Install finished, but the DRM patch did not apply." "Yellow" +} + +$ofsExe = Get-OfScraperExe -ExePath $python.Path + +Write-ColorOutput "" "White" +Write-ColorOutput "Install complete." "Green" +Write-ColorOutput (" ofscraper {0}" -f $finalOfs) "White" +Write-ColorOutput (" aiolimiter {0}" -f $finalAio) "White" +Write-ColorOutput (" DRM patch {0}" -f $DrmPatchStatus) "White" +Write-ColorOutput (" Python {0}" -f $python.Version) "White" + +if ($ofsExe) { + Write-ColorOutput "" "White" + Write-ColorOutput "Paste this into SCrawler: Settings -> OnlyFans -> OF-Scraper path" "Cyan" + Write-ColorOutput $ofsExe "Green" +} else { + Write-ColorOutput "" "White" + Write-ColorOutput "ofscraper installed, but ofscraper.exe was not found on disk." "Yellow" + Write-ColorOutput "Look under the Python Scripts folder next to:" "Yellow" + Write-ColorOutput $python.Path "Gray" +} + +Pause-IfInteractive +exit 0