geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-15 08:12:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

2023.4.28.0

Browse Source 
Plugins
IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function
Add IDownloadableMedia interface
Removed 'Channel' option from all functions and enums
ISiteSettings: added GetSingleMediaInstance function
ExchangeOptions: removed 'IsChannel'
UserMediaTypes: added Audio and AudioPre enums
IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums

SCrawler
Add YouTube standalone downloader
Add gallery-dl & yt-dlp support
Remove 'UserInfo' requirement from 'ProfilesSaved'
Update 'SiteSettingsBase' to use domains and Netscape cookies
UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download

Rewritten DomainsContainer
Create a universal settings form and PSettingsArttribute
Gfycat, Imgur: turn these classes into IUserData to download a single object

All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist

LPSG: fix attachments; update converters and regex
Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music
Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support
Twitter: fixed gif path bug; fixed downloading saved posts
PornHub: hide unnecessary errors; photo galleries bug
RedGifs: add 'UserAgent' option

Added icons to download progress

Rename some objects
Completely redesigned standalone downloader form and rewritten its environment
WebClient2: update to use tokens

Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser'

UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found
Remove temporary 'EncryptCookies' module

MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader)
PropertyValueHost: update for option forms compatibility
SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork
UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function
Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler
UserFinder: remove old 'Merge' const; remove channel option
UserInfo: remove channel option
This commit is contained in:
Andy
2023-04-28 10:13:46 +03:00
parent db9e2cfb88
commit b2a9b22478
270 changed files with 18120 additions and 3332 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
SCrawler/Editors/SiteEditorForm.vb
View File

@@ -19,23 +19,20 @@ Namespace Editors
Private ReadOnly LBL_OTHER As Label
Private WithEvents MyDefs As DefaultFormOptions
Private WithEvents SpecialButton As Button
Private Property Cookies As CookieKeeper
Private CookiesChanged As Boolean = False
#Region "Providers"
Private Class SavedPostsChecker : Implements IFieldsCheckerProvider
Private Property ErrorMessage As String Implements IFieldsCheckerProvider.ErrorMessage
Private Property Name As String Implements IFieldsCheckerProvider.Name
Private Property TypeError As Boolean Implements IFieldsCheckerProvider.TypeError
Private Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object Implements ICustomProvider.Convert
Private Class SavedPostsChecker : Inherits FieldsCheckerProviderBase
Public Overrides Function Convert(ByVal Value As Object, ByVal DestinationType As Type, ByVal Provider As IFormatProvider,
Optional ByVal NothingArg As Object = Nothing, Optional ByVal e As ErrorsDescriber = Nothing) As Object
If ACheck(Value) AndAlso CStr(Value).Contains("/") Then
ErrorMessage = $"Path [{Name}] contains forbidden character ""/"""
HasError = True
Return Nothing
Else
Return Value
End If
End Function
Private Function GetFormat(ByVal FormatType As Type) As Object Implements IFormatProvider.GetFormat
Throw New NotImplementedException("[GetFormat] is not available in the context of [SavedPostsChecker]")
End Function
End Class
#End Region
Private ReadOnly Property Host As SettingsHost
@@ -43,6 +40,7 @@ Namespace Editors
InitializeComponent()
MyDefs = New DefaultFormOptions(Me, Settings.Design)
Host = h
If Not Host.Responser Is Nothing Then Cookies = Host.Responser.Cookies.Copy
LBL_AUTH = New Label With {.Text = "Authorization", .TextAlign = ContentAlignment.MiddleCenter, .Dock = DockStyle.Fill}
LBL_OTHER = New Label With {.Text = "Other Parameters", .TextAlign = ContentAlignment.MiddleCenter, .Dock = DockStyle.Fill}
Host.Source.BeginEdit()
@@ -155,6 +153,7 @@ Namespace Editors
LBL_AUTH.Dispose()
LBL_OTHER.Dispose()
Host.Source.EndEdit()
If Not Cookies Is Nothing Then Cookies.Dispose()
End Sub
Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
If MyDefs.MyFieldsChecker.AllParamsOK Then
@@ -185,6 +184,12 @@ Namespace Editors
.SavedPostsPath = TXT_PATH_SAVED_POSTS.Text
.DownloadSiteData.Value = CH_DOWNLOAD_SITE_DATA.Checked
.GetUserMediaOnly.Value = CH_GET_USER_MEDIA_ONLY.Checked
If CookiesChanged And Not Cookies Is Nothing And Not .Responser Is Nothing Then
With .Responser.Cookies
.Clear()
.AddRange(Cookies,, EDP.LogMessageValue)
End With
End If
If .PropList.Count > 0 Then .PropList.ForEach(Sub(p) If Not p.Options Is Nothing Then p.UpdateValueByControl())
@@ -211,27 +216,29 @@ Namespace Editors
Private Sub TXT_COOKIES_ActionOnButtonClick(ByVal Sender As ActionButton, ByVal e As EventArgs) Handles TXT_COOKIES.ActionOnButtonClick
Select Case Sender.DefaultButton
Case ADB.Edit
If Not Host.Responser Is Nothing Then
If Not Cookies Is Nothing Then
Using f As New CookieListForm With {.DesignXML = Settings.Design, .ShowGrid = False}
f.SetCollection(Host.Responser.Cookies)
f.SetCollection(Cookies)
f.ShowDialog()
If f.DialogResult = DialogResult.OK Then
f.GetCollection(Host.Responser)
CookiesChanged = True
f.GetCollection(Cookies)
MyDefs.MyOkCancel.EnableOK = True
End If
End Using
SetCookieText()
End If
Case ADB.Clear
If Not Host.Responser Is Nothing Then
Host.Responser.Cookies.Clear()
If Not Cookies Is Nothing Then
CookiesChanged = True
Cookies.Clear()
MyDefs.MyOkCancel.EnableOK = True
SetCookieText()
End If
End Select
End Sub
Private Sub SetCookieText()
If Not Host.Responser Is Nothing Then TXT_COOKIES.Text = $"{Host.Responser.Cookies.Count} cookies"
If Not Cookies Is Nothing Then TXT_COOKIES.Text = $"{Cookies.Count} cookies"
End Sub
Private Sub SpecialButton_Click(sender As Object, e As EventArgs) Handles SpecialButton.Click
MyDefs.Detector()