mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-15 16:22:17 +00:00
2023.8.6.0
Plugins.Attributes: add 'DependentFields' attribute Plugins.IPluginContentProvider: add 'Options' and 'IsSubscription' properties Plugins.ISiteSettings: add 'SubscriptionsAllowed' property Plugins.ExchangeOptions: add 'Options' field Plugins.Attributes.PropertyUpdater: replace 'Dependencies' with 'Arguments' YT: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties; add the ability to store download locations; add 'DownloadLocation' and 'DownloadLocationsCollection' objects YT.IDownloaderSettings: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties YT.Downloader: fixed bug with re-saving elements when loading a video list; fixed bug when files were not deleted when clicking on the delete button; fixed a bug that caused the video to redownload; download job removes elements at wrong indexes; added skipping of downloaded elements in the job; fixed a bug, pending option did not change after download complete YT.YouTubeMediaContainerBase: add '_MediaStateOnLoad' field and 'NeedToSave' function; update the 'Save' function to prevent saving a file when a download is complete and the file has already been saved; update code for new yt-dlp version Fixed cache deletion errors Add user queue Add global locations API.Base.SiteSettingsBase: implement 'SubscriptionsAllowed' property; remove request headers with null values on save; add '_AllowUserAgentUpdate' parameter API.Base.Structures: add 'SiteModes' enum API.Base.UserDataBase: add 'Erase' button; implement 'Options' and 'IsSubscription' properties; add 'SpecialLabels' property; update 'LVIKey'; update 'FitToAddParams' function; add 'EraseData' function; user colors; Not UserExists notification, UserQueue support API.Base: add 'DeclaredNames' API.Instagram: remove default values for headers; disable updating UserAgent from global; check for a new username for non-existent users API.Mastodon: bypass new inherited twitter options; update names and headers API.OnlyFans: make 'HH_BROWSER' property nullable; remove 'HH_BROWSER' from required; fix username bug (dots); handling of 504 and 429 errors; add 'DownloadHighlights' and 'DownloadChatMedia' options; add 'UserExchangeOptions'; fixed incorrect error handler API.PathPlugin: fixed incorrect detection of path existence API.Pinterest: add 'SpecialLabels' API.PornHub: add new video regex; remove old regex; added 'DownloadUploaded', 'DownloadTagged', 'DownloadPrivate' and 'DownloadFavorite' properties to 'SiteSettings', 'UserData' and 'UserExchangeOptions'; update regex to define user; added downloading search queries; update 'GetUserUrl' function; hide unnecessary 'RegexFieldsTextBecameNullException' errors; add subscriptions API.Reddit: add 'SpecialLabels'; add bearer token and its refresh interval; add OAuth; add additional options API.RedGifs: add 'DependentFields' for 'Token' API.ThisVid: add 'DownloadFavourite' option; add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; updating cookies issue API.TikTok: rewrite algorithms API.Twitter: add 'UseAppropriateModel', 'UseNewEndPointSearch', 'UseNewEndPointProfiles', 'AbortOnLimit', 'DownloadAlreadyParsed', 'MediaModelAllowNonUserTweets' properties; remove old commented code; remove 'TwitterPic_400' and replace with 'TwitterIcon_32.ToBitmap'; add 'DownloadModelForceApply' user option; update environment to GDL 1.25.8; fixed gifs downloading; fix typo in 'ReparseMissing'; update names API.UserDataBind: prevent adding site-specific labels when adding to a collection API.Xhamster: add downloading search queries, tags, categories; add 'SpecialLabels'; add additional nodes for channels; add subscriptions API.XVIDEOS: add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; changed users creation method; add subscriptions API.YouTube: add subscriptions AutoDownloader: add new group subscription options; update predicates; fixed excluded labels and sites in default mode; update notifications; add an additional skip options, add 'Force start' option DownloadedInfoForm: add subscriptions; fixed size/location bug; hide unnecessary error (refill) Feed: add subscriptions; update filters; add 'Ctrl+G' shortcut FeedMedia: add subscriptions; fixed 'webm' bug; add title for subscription media; add site icon to post; user colors; always using 'FriendlyName' instead of 'UserName' if it exists DownloadGroup, GroupDefaults, GroupParameters: add subscription and 'UsersCount' options MissingPostsForm: add 'BTT_DELETE_ALL' VideoDownloaderForm, DownloaderUrlForm, DownloaderUrlsArrForm: add download locations support VideoDownloaderForm: add subscriptions support GlobalSettingsForm: add new properties UserCreatorForm: add subscriptions; add 'Options' support (of 'ExchangeOptions'); user colors ListImagesLoader: add subscription colors; user colors MainFrame: add subscriptions; add filters by subscription and user; update predicates NuGet: update 'LibVLCSharp', 'LibVLCSharp.WinForms', 'VideoLAN.LibVLC.Windows' DownloadableMediaHost: update 'Save' function PropertyValueHost: fix 'CaptionWidth' bug; add 'Dependents' SettingsHost: add 'Dependents' UserDataHost: add 'Options' and 'IsSubscription' properties SettingsCLS: implement new 'IDownloaderSettings' properties; add 'CacheSnapshots'; add 'DownloadLocations'; add new properties UserInfo, UserFinder: add subscriptions UserSearchForm: fixed search by name bug
This commit is contained in:
Andy
@@ -22,7 +22,7 @@ Imports UTypes = SCrawler.API.Base.UserMedia.Types
|
||||
Imports CView = SCrawler.API.Reddit.IRedditView.View
|
||||
Imports CPeriod = SCrawler.API.Reddit.IRedditView.Period
|
||||
Namespace API.Reddit
|
||||
Friend Class UserData : Inherits UserDataBase : Implements IChannelData, IRedditView
|
||||
Friend Class UserData : Inherits UserDataBase : Implements IChannelLimits, IRedditView
|
||||
#Region "XML names"
|
||||
Private Const Name_TrueName As String = "TrueName"
|
||||
#End Region
|
||||
@@ -46,6 +46,11 @@ Namespace API.Reddit
|
||||
End Property
|
||||
Friend Property IsChannel As Boolean = False
|
||||
Friend Property TrueName As String = String.Empty
|
||||
Friend Overrides ReadOnly Property SpecialLabels As IEnumerable(Of String)
|
||||
Get
|
||||
Return {CannelsLabelName, CannelsLabelName_ChannelsForm, UserLabelName}
|
||||
End Get
|
||||
End Property
|
||||
#End Region
|
||||
#Region "Channels Support"
|
||||
#Region "IChannelLimits Support"
|
||||
@@ -70,9 +75,9 @@ Namespace API.Reddit
|
||||
#End Region
|
||||
Friend Property ChannelInfo As Channel
|
||||
Private ReadOnly ChannelPostsNames As List(Of String)
|
||||
Friend Property SkipExistsUsers As Boolean = False Implements IChannelData.SkipExistsUsers
|
||||
Friend Property SkipExistsUsers As Boolean = False
|
||||
Private ReadOnly _ExistsUsersNames As List(Of String)
|
||||
Friend Property SaveToCache As Boolean = False Implements IChannelData.SaveToCache
|
||||
Friend Property SaveToCache As Boolean = False
|
||||
Friend Function GetNewChannelPosts() As IEnumerable(Of UserPost)
|
||||
If _ContentNew.Count > 0 Then Return (From c As UserMedia In _ContentNew
|
||||
Where Not c.Post.CachedFile.IsEmptyString And c.State = UStates.Downloaded
|
||||
@@ -127,7 +132,7 @@ Namespace API.Reddit
|
||||
End Sub
|
||||
#End Region
|
||||
#Region "Load and Update user info"
|
||||
Private Sub UpdateNames()
|
||||
Private Function UpdateNames() As Boolean
|
||||
If TrueName.IsEmptyString Then
|
||||
Dim n$() = Name.Split("@")
|
||||
If n.ListExists Then
|
||||
@@ -145,9 +150,11 @@ Namespace API.Reddit
|
||||
Settings.Labels.Add(l)
|
||||
Labels.ListAddValue(l, LNC)
|
||||
Labels.Sort()
|
||||
Return True
|
||||
End If
|
||||
End If
|
||||
End Sub
|
||||
Return False
|
||||
End Function
|
||||
Protected Overrides Sub LoadUserInformation_OptionalFields(ByRef Container As XmlFile, ByVal Loading As Boolean)
|
||||
With Container
|
||||
If Loading Then
|
||||
@@ -157,7 +164,7 @@ Namespace API.Reddit
|
||||
TrueName = .Value(Name_TrueName)
|
||||
UpdateNames()
|
||||
Else
|
||||
UpdateNames()
|
||||
If UpdateNames() Then .Value(Name_LabelsName) = LabelsString
|
||||
.Add(Name_ViewMode, CInt(ViewMode))
|
||||
.Add(Name_ViewPeriod, CInt(ViewPeriod))
|
||||
.Add(Name_IsChannel, IsChannel.BoolToInteger)
|
||||
@@ -198,6 +205,15 @@ Namespace API.Reddit
|
||||
End If
|
||||
End Sub
|
||||
Protected Overrides Sub DownloadDataF(ByVal Token As CancellationToken)
|
||||
With MySiteSettings
|
||||
If IsSavedPosts Then
|
||||
If Not CBool(.UseTokenForSavedPosts.Value) Then Responser.Headers.Remove(DeclaredNames.Header_Authorization)
|
||||
Else
|
||||
If Not CBool(.UseCookiesForTimelines.Value) Then Responser.Cookies.Clear()
|
||||
If Not CBool(.UseTokenForTimelines.Value) Then Responser.Headers.Remove(DeclaredNames.Header_Authorization)
|
||||
End If
|
||||
End With
|
||||
|
||||
_TotalPostsDownloaded = 0
|
||||
If IsSavedPosts Then
|
||||
Responser.DecodersError = EDP.ReturnValue
|
||||
@@ -302,7 +318,7 @@ Namespace API.Reddit
|
||||
End If
|
||||
End Using
|
||||
If POST.IsEmptyString And ExistsDetected Then Exit Sub
|
||||
If Not PostID.IsEmptyString And NewPostDetected Then DownloadDataUser(PostID, Token)
|
||||
If Not _PostID().IsEmptyString And NewPostDetected Then DownloadDataUser(_PostID(), Token)
|
||||
End If
|
||||
_completed = True
|
||||
Catch ex As Exception
|
||||
@@ -979,8 +995,13 @@ Namespace API.Reddit
|
||||
UserSuspended = True
|
||||
ElseIf .StatusCode = HttpStatusCode.BadGateway Or .StatusCode = HttpStatusCode.ServiceUnavailable Then
|
||||
MyMainLOG = $"[{CInt(Responser.StatusCode)}] Reddit is currently unavailable ({ToString()})"
|
||||
Throw New Plugin.ExitException With {.Silent = True}
|
||||
ElseIf .StatusCode = HttpStatusCode.GatewayTimeout Then
|
||||
Return 1
|
||||
ElseIf .StatusCode = HttpStatusCode.Unauthorized Then
|
||||
MyMainLOG = $"[{CInt(Responser.StatusCode)}] Reddit credentials expired ({ToString()})"
|
||||
MySiteSettings.SessionInterrupted = True
|
||||
Throw New Plugin.ExitException With {.Silent = True}
|
||||
ElseIf .StatusCode = HttpStatusCode.InternalServerError Then
|
||||
If Not IsNothing(EObj) AndAlso IsNumeric(EObj) AndAlso CInt(EObj) = HttpStatusCode.InternalServerError Then Return 1
|
||||
Return HttpStatusCode.InternalServerError
|
||||
|
||||
Reference in New Issue
Block a user