geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2024.9.2.0

Browse Source 
Instagram: add options to enable/disable image extraction from video
OnlyFans: update to the changed API
YouTube: videos are parsed from the 'featured', not from the 'videos' page
Feed: add prompt before moving entire feed/session
MainFrame: add 'Alt+U' and 'Ctrl+U' to open the user search form
UserImage: user image creation update
This commit is contained in:
Andy
2024-09-02 18:22:11 +03:00
parent 2055461829
commit 00a06d3e9a
17 changed files with 371 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
Changelog.md
View File

@@ -1,3 +1,19 @@
# 2024.9.2.0
*2024-09-02*
- Added
- Instagram: options to enable/disable image extraction from video
- Feed: **prompt before moving entire feed/session**
- Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
- Minor improvements
- Updated
- gallery-dl up to version **1.27.3**
- Fixed
- **OnlyFans**: data is not downloading
- YouTube (SCrawler): incorrect parsing of video page
- Minor bugs
# 2024.8.10.0 # 2024.8.10.0
*2024-08-10* *2024-08-10*

31
FAQ.md
View File

@@ -28,22 +28,24 @@ I strongly recommend you to **regularly** create backup copies of the settings f
**Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!** **Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
*How to use: find your problem in the list and read the answer.*
## General questions ## General questions
- **PROFILES** - **PROFILES**
- I added a profile but **nothing downloaded**: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. - I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
- User downloading failed: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u> - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
- [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user) - [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
- How to **add profile** to download: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button. - How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)** - How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)** - **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
- [How to report a problem](#how-to-report-a-problem) - [How to report a problem](#how-to-report-a-problem)
- I want you to **add the site** to SCrawler: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler. - I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
- What language is SCrawler written in: vb.net - What language is SCrawler written in :arrow_forward: vb.net
- I don't know vb.net and I can't write a plugin. You can write a plugin in `C#` - I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
- I have a suggestion, will it be added: maybe if it interested me. - I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`): **there is no such functionality and there are no such plans**. - How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)). - **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing**: restore user settings from [backup](#backup). - My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
- Installation, update and configuration - Installation, update and configuration
- How to install: https://github.com/AAndyProgram/SCrawler#installation - How to install: https://github.com/AAndyProgram/SCrawler#installation
- How to update: https://github.com/AAndyProgram/SCrawler#updating - How to update: https://github.com/AAndyProgram/SCrawler#updating
@@ -52,12 +54,15 @@ I strongly recommend you to **regularly** create backup copies of the settings f
- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source) - [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
- [Video how to configure](#video-how-to-configure) - [Video how to configure](#video-how-to-configure)
- **Antivirus** - **Antivirus**
- **Antivirus detects SCrawler as a virus**: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.** - **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
- **Antivirus detects gallery-dl as a virus**: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself. - **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
## Sites questions ## Sites questions
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials)
- **META** (Instagram, Threads, Facebook): you need **cookies** and fill in **all fields** *How to use: find the site you need in the list and read the answer.*
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat. - **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download. - TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
- Porn sites: **COOKIES**! - Porn sites: **COOKIES**!

BIN
ProgramScreenshots/SettingsSiteInstagram.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 46 KiB

272
SCrawler/API/Base/UserDataBase.vb
View File

@@ -410,9 +410,7 @@ Namespace API.Base
End Function End Function
Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture
Try Try
If f.Exists Then If f.Exists Then UserImage.NewUserPicture(f, MyFile)
Using p As New UserImage(f, MyFile) : p.Save() : End Using
End If
Catch Catch
End Try End Try
End Sub End Sub
@@ -451,11 +449,7 @@ BlockPictureScan:
New ErrorsDescriber(EDP.ReturnValue) With { New ErrorsDescriber(EDP.ReturnValue) With {
.ReturnValue = New List(Of SFile), .ReturnValue = New List(Of SFile),
.ReturnValueExists = True}).FirstOrDefault .ReturnValueExists = True}).FirstOrDefault
If NewPicFile.Exists Then If NewPicFile.Exists Then p = UserImage.NewUserPicture(NewPicFile, MyFile,, True) : GoTo BlockReturn
p = New UserImage(NewPicFile, MyFile)
p.Save()
GoTo BlockReturn
End If
BlockDeletePictureFolder: BlockDeletePictureFolder:
On Error GoTo BlockReturn On Error GoTo BlockReturn
If DelPath Then If DelPath Then
@@ -654,6 +648,7 @@ BlockNullPicture:
End Sub End Sub
Protected ReadOnly _TempMediaList As List(Of UserMedia) Protected ReadOnly _TempMediaList As List(Of UserMedia)
Protected ReadOnly _TempPostsList As List(Of String) Protected ReadOnly _TempPostsList As List(Of String)
Private ReadOnly _MD5List As List(Of String)
Friend Function GetLastImageAddress() As SFile Friend Function GetLastImageAddress() As SFile
If _ContentList.Count > 0 Then If _ContentList.Count > 0 Then
Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File
@@ -679,6 +674,7 @@ BlockNullPicture:
Protected MyFileSettings As SFile Protected MyFileSettings As SFile
Protected MyFileData As SFile Protected MyFileData As SFile
Protected MyFilePosts As SFile Protected MyFilePosts As SFile
Private MyMD5File As SFile
Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists
Friend Overridable Property DataMerging As Boolean Friend Overridable Property DataMerging As Boolean
Get Get
@@ -856,6 +852,7 @@ BlockNullPicture:
LatestData = New List(Of UserMedia) LatestData = New List(Of UserMedia)
_TempMediaList = New List(Of UserMedia) _TempMediaList = New List(Of UserMedia)
_TempPostsList = New List(Of String) _TempPostsList = New List(Of String)
_MD5List = New List(Of String)
Labels = New List(Of String) Labels = New List(Of String)
UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler) UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler) UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler)
@@ -1037,6 +1034,8 @@ BlockNullPicture:
If _ContentList.Count > 0 Then x.AddRange(_ContentList) If _ContentList.Count > 0 Then x.AddRange(_ContentList)
x.Save(MyFileData) x.Save(MyFileData)
End Using End Using
If Not MyMD5File.IsEmptyString And _MD5List.Count > 0 Then _
TextSaver.SaveTextToFile(_MD5List.ListToString(Environment.NewLine), MyMD5File, True,, EDP.None)
Catch ex As Exception Catch ex As Exception
LogError(ex, "history saving error") LogError(ex, "history saving error")
End Try End Try
@@ -1131,6 +1130,7 @@ BlockNullPicture:
TokenPersonal = Nothing TokenPersonal = Nothing
ProgressPre.Reset() ProgressPre.Reset()
UpdateDataFiles() UpdateDataFiles()
_MD5Loaded = False
_DownloadInProgress = True _DownloadInProgress = True
_DescriptionChecked = False _DescriptionChecked = False
_DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime _DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
@@ -1212,7 +1212,7 @@ BlockNullPicture:
ProgressPre.Done() ProgressPre.Done()
ThrowAny(Token) ThrowAny(Token)
If UseMD5Comparison And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token) If RemoveExistingDuplicates And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then
If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd) If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd)
@@ -1315,6 +1315,11 @@ BlockNullPicture:
MyFilePosts = MyFileSettings MyFilePosts = MyFileSettings
MyFilePosts.Name &= "_Posts" MyFilePosts.Name &= "_Posts"
MyFilePosts.Extension = "txt" MyFilePosts.Extension = "txt"
If Not IsSavedPosts Then
MyMD5File = MyFileSettings
MyMD5File.Name &= "_MD5"
MyMD5File.Extension = "txt"
End If
Else Else
Throw New ArgumentNullException("User.File", "User file not detected") Throw New ArgumentNullException("User.File", "User file not detected")
End If End If
@@ -1438,81 +1443,94 @@ BlockNullPicture:
End Sub End Sub
#End Region #End Region
#Region "MD5 support" #Region "MD5 support"
Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR" Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
Friend Property UseMD5Comparison As Boolean = False Friend Property UseMD5Comparison As Boolean = False
Protected Property StartMD5Checked As Boolean = False Protected Property StartMD5Checked As Boolean = False
Friend Property RemoveExistingDuplicates As Boolean = False Friend Property RemoveExistingDuplicates As Boolean = False
Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken) Private ReadOnly ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
Private _MD5Loaded As Boolean = False
Private Sub LoadMD5()
Try
If Not _MD5Loaded Then
_MD5Loaded = True
_MD5List.Clear()
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(c) c.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
If MyMD5File.Exists Then _MD5List.ListAddList(MyMD5File.GetLines, LAP.NotContainsOnly, EDP.ThrowException)
End If
Catch ex As Exception
ErrorsDescriber.Execute(EDP.SendToLog, ex, "LoadMD5")
End Try
End Sub
Private Function ValidateMD5_GetMD5(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
Try
Dim ImgFormat As Imaging.ImageFormat = Nothing
Dim hash$ = String.Empty
Dim __isGif As Boolean = False
If __data.Type = UTypes.GIF Then
ImgFormat = Imaging.ImageFormat.Gif
__isGif = True
ElseIf Not __data.File.IsEmptyString Then
ImgFormat = GetImageFormat(__data.File)
End If
If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl And Not __isGif Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
ElseIf IsUrl And __isGif Then
hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
If hash.IsEmptyString And Not __isGif Then
If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
End If
Return hash
Catch
Return String.Empty
End Try
End Function
Private Sub ValidateMD5(ByVal Token As CancellationToken)
Try Try
Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then If RemoveExistingDuplicates Then
RemoveExistingDuplicates = False
_ForceSaveUserInfo = True
LoadMD5()
Dim i% Dim i%
Dim itemsCount% = 0 Dim itemsCount% = 0
Dim limit% = If(DownloadTopCount, 0) Dim limit% = If(DownloadTopCount, 0)
Dim data As UserMedia = Nothing Dim data As UserMedia = Nothing
Dim hashList As New Dictionary(Of String, SFile)
Dim f As SFile Dim f As SFile
Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
Try
Dim ImgFormat As Imaging.ImageFormat = Nothing
Dim hash$ = String.Empty
Dim __isGif As Boolean = False
If __data.Type = UTypes.GIF Then
ImgFormat = Imaging.ImageFormat.Gif
__isGif = True
ElseIf Not __data.File.IsEmptyString Then
ImgFormat = GetImageFormat(__data.File)
End If
If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl And Not __isGif Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
ElseIf IsUrl And __isGif Then
hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
If hash.IsEmptyString And Not __isGif Then
If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
End If
Return hash
Catch
Return String.Empty
End Try
End Function
If Not StartMD5Checked Then If Not StartMD5Checked Then
StartMD5Checked = True StartMD5Checked = True
If _ContentList.Exists(missingMD5) Then Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing Dim eIndx%
Dim eIndx% Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
If RemoveExistingDuplicates Then If existingFiles.Count > 0 Then
RemoveExistingDuplicates = False Dim h$
_ForceSaveUserInfo = True ProgressPre.ChangeMax(existingFiles.Count)
If existingFiles.Count > 0 Then For i = existingFiles.Count - 1 To 0 Step -1
Dim h$ ProgressPre.Perform()
ProgressPre.ChangeMax(existingFiles.Count) h = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(i)}, False)
For i = existingFiles.Count - 1 To 0 Step -1 If Not h.IsEmptyString Then
ProgressPre.Perform() If _MD5List.Contains(h) Then
h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False) MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate)"
If Not h.IsEmptyString Then existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
If hashList.ContainsKey(h) Then existingFiles.RemoveAt(i)
MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])" Else
existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5) _MD5List.Add(h)
existingFiles.RemoveAt(i) End If
Else
hashList.Add(h, existingFiles(i))
End If
End If
Next
End If End If
End If Next
End If
If _ContentList.Count > 0 AndAlso _ContentList.Exists(missingMD5) Then
ProgressPre.ChangeMax(_ContentList.Count) ProgressPre.ChangeMax(_ContentList.Count)
For i = 0 To _ContentList.Count - 1 For i = 0 To _ContentList.Count - 1
data = _ContentList(i) data = _ContentList(i)
@@ -1522,61 +1540,34 @@ BlockNullPicture:
ThrowAny(Token) ThrowAny(Token)
eIndx = existingFiles.FindIndex(eFinder) eIndx = existingFiles.FindIndex(eFinder)
If eIndx >= 0 Then If eIndx >= 0 Then
data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False) data.MD5 = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
End If End If
End If End If
existingFiles.RemoveAll(eFinder) existingFiles.RemoveAll(eFinder)
End If End If
Next Next
If existingFiles.Count > 0 Then
ProgressPre.ChangeMax(existingFiles.Count)
For i = 0 To existingFiles.Count - 1
f = existingFiles(i)
ProgressPre.Perform()
data = New UserMedia(f.File) With {
.State = UStates.Downloaded,
.Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
.File = f
}
ThrowAny(Token)
data.MD5 = __getMD5(data, False)
If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
Next
existingFiles.Clear()
End If
End If End If
End If
If _ContentList.Count > 0 Then If existingFiles.Count > 0 Then
With _ContentList.Select(Function(d) d.MD5) ProgressPre.ChangeMax(existingFiles.Count)
If .ListExists Then .ToList.ForEach(Sub(md5value) _ For i = 0 To existingFiles.Count - 1
If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile)) f = existingFiles(i)
End With ProgressPre.Perform()
End If data = New UserMedia(f.File) With {
.State = UStates.Downloaded,
ProgressPre.ChangeMax(_TempMediaList.Count) .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
For i = _TempMediaList.Count - 1 To 0 Step -1 .File = f
ProgressPre.Perform() }
If limit > 0 And itemsCount >= limit Then
_TempMediaList.RemoveAt(i)
Else
data = _TempMediaList(i)
If missingMD5(data) Then
ThrowAny(Token) ThrowAny(Token)
data.MD5 = __getMD5(data, True) data.MD5 = ValidateMD5_GetMD5(data, False)
If Not data.MD5.IsEmptyString Then If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
If hashList.ContainsKey(data.MD5) Then Next
_TempMediaList.RemoveAt(i) existingFiles.Clear()
Else
hashList.Add(data.MD5, New SFile)
_TempMediaList(i) = data
itemsCount += 1
End If
End If
End If
End If End If
Next End If
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(d) d.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
End If End If
Catch iex As ArgumentOutOfRangeException When Disposed Catch iex As ArgumentOutOfRangeException When Disposed
Catch ex As Exception Catch ex As Exception
@@ -1614,6 +1605,7 @@ BlockNullPicture:
Source.Progress.Done() Source.Progress.Done()
End Sub End Sub
End Class End Class
Protected Const VideoFolderName As String = "Video"
Protected Sub DownloadContentDefault(ByVal Token As CancellationToken) Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
Try Try
Dim i% Dim i%
@@ -1622,6 +1614,7 @@ BlockNullPicture:
If _ContentNew.Count > 0 Then If _ContentNew.Count > 0 Then
_ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString) _ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
If _ContentNew.Count > 0 Then If _ContentNew.Count > 0 Then
If UseMD5Comparison Then LoadMD5()
MyFile.Exists(SFO.Path) MyFile.Exists(SFO.Path)
Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
Dim MyDir$ = DownloadContentDefault_GetRootDir() Dim MyDir$ = DownloadContentDefault_GetRootDir()
@@ -1630,6 +1623,7 @@ BlockNullPicture:
Dim __interrupt As Boolean Dim __interrupt As Boolean
Dim f As SFile Dim f As SFile
Dim v As UserMedia Dim v As UserMedia
Dim __fileDeleted As Boolean
Dim fileNumProvider As SFileNumbers = SFileNumbers.Default Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String) Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String)
Try Try
@@ -1641,9 +1635,21 @@ BlockNullPicture:
ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex) ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex)
End Try End Try
End Sub End Sub
Dim updateDownCount As Action = Sub()
Dim __n% = IIf(__fileDeleted, -1, 1)
If __isVideo Then
v.Type = UTypes.Video
DownloadedVideos(False) += __n
ElseIf v.Type = UTypes.GIF Then
DownloadedPictures(False) += __n
Else
v.Type = UTypes.Picture
DownloadedPictures(False) += __n
End If
End Sub
Using w As New OptionalWebClient(Me) Using w As New OptionalWebClient(Me)
If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path) If vsf Then CSFileP($"{MyDir}\{VideoFolderName}\").Exists(SFO.Path)
Progress.Maximum += _ContentNew.Count Progress.Maximum += _ContentNew.Count
If IsSingleObjectDownload Then If IsSingleObjectDownload Then
If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
@@ -1671,6 +1677,8 @@ BlockNullPicture:
If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
__fileDeleted = False
If Not f.IsEmptyString And Not v.URL.IsEmptyString Then If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
Try Try
__isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8 __isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
@@ -1691,7 +1699,7 @@ BlockNullPicture:
End If End If
If __isVideo And vsf Then If __isVideo And vsf Then
If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then
f.Path = $"{f.PathWithSeparator}Video" f.Path = $"{f.PathWithSeparator}{VideoFolderName}"
If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path) If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
End If End If
End If End If
@@ -1715,19 +1723,26 @@ BlockNullPicture:
End If End If
End If End If
If __isVideo Then updateDownCount()
v.Type = UTypes.Video
DownloadedVideos(False) += 1
ElseIf v.Type = UTypes.GIF Then
DownloadedPictures(False) += 1
Else
v.Type = UTypes.Picture
DownloadedPictures(False) += 1
End If
v.File = ChangeFileNameByProvider(f, v) v.File = ChangeFileNameByProvider(f, v)
v.State = UStates.Downloaded v.State = UStates.Downloaded
DownloadContentDefault_PostProcessing(v, f, Token) DownloadContentDefault_PostProcessing(v, f, Token)
If UseMD5Comparison And (v.Type = UTypes.GIF Or v.Type = UTypes.Picture) Then
If v.File.Exists Then
v.MD5 = ValidateMD5_GetMD5(v, False)
If Not v.MD5.IsEmptyString Then
If _MD5List.Contains(v.MD5) Then
__fileDeleted = v.File.Delete(SFO.File, SFODelete.DeletePermanently, EDP.ReturnValue)
If __fileDeleted Then dCount -= 1 : updateDownCount()
Else
_MD5List.Add(v.MD5)
End If
End If
Else
dCount -= 1
End If
End If
dCount += 1 dCount += 1
Catch woex As OperationCanceledException When Token.IsCancellationRequested Catch woex As OperationCanceledException When Token.IsCancellationRequested
__deleteFile.Invoke(f, v.URL_BASE) __deleteFile.Invoke(f, v.URL_BASE)
@@ -1745,7 +1760,7 @@ BlockNullPicture:
Else Else
v.State = UStates.Skipped v.State = UStates.Skipped
End If End If
_ContentNew(i) = v If Not __fileDeleted Then _ContentNew(i) = v
If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then
Progress.Perform(_ContentNew.Count - dTotal) Progress.Perform(_ContentNew.Count - dTotal)
Exit Sub Exit Sub
@@ -2240,6 +2255,7 @@ BlockNullPicture:
LatestData.Clear() LatestData.Clear()
_TempMediaList.Clear() _TempMediaList.Clear()
_TempPostsList.Clear() _TempPostsList.Clear()
_MD5List.Clear()
TokenPersonal = Nothing TokenPersonal = Nothing
If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose() If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
If Not Responser Is Nothing Then Responser.Dispose() If Not Responser Is Nothing Then Responser.Dispose()

32
SCrawler/API/Instagram/EditorExchangeOptions.vb
View File

@@ -9,6 +9,7 @@
Imports SCrawler.Plugin.Attributes Imports SCrawler.Plugin.Attributes
Namespace API.Instagram Namespace API.Instagram
Friend Class EditorExchangeOptions Friend Class EditorExchangeOptions
#Region "Download"
<PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")> <PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
Friend Property GetTimeline As Boolean Friend Property GetTimeline As Boolean
<PSetting(Caption:="Get reels", ToolTip:="Download user reels")> <PSetting(Caption:="Get reels", ToolTip:="Download user reels")>
@@ -19,6 +20,21 @@ Namespace API.Instagram
Friend Property GetStoriesUser As Boolean Friend Property GetStoriesUser As Boolean
<PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")> <PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")>
Friend Property GetTagged As Boolean Friend Property GetTagged As Boolean
#End Region
#Region "Extract image"
<PSetting(Caption:="Extract image from video: timeline")>
Friend Property GetTimeline_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: reels")>
Friend Property GetReels_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: stories")>
Friend Property GetStories_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: stories: user")>
Friend Property GetStoriesUser_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: tagged posts")>
Friend Property GetTagged_VideoPic As Boolean
#End Region
<PSetting(Caption:="Place the extracted image into the video folder")>
Friend Property PutImageVideoFolder As Boolean
Friend Sub New(ByVal u As UserData) Friend Sub New(ByVal u As UserData)
With u With u
GetTimeline = .GetTimeline GetTimeline = .GetTimeline
@@ -26,6 +42,14 @@ Namespace API.Instagram
GetStories = .GetStories GetStories = .GetStories
GetStoriesUser = .GetStoriesUser GetStoriesUser = .GetStoriesUser
GetTagged = .GetTaggedData GetTagged = .GetTaggedData
GetTimeline_VideoPic = .GetTimeline_VideoPic
GetReels_VideoPic = .GetReels_VideoPic
GetStories_VideoPic = .GetStories_VideoPic
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
GetTagged_VideoPic = .GetTaggedData_VideoPic
PutImageVideoFolder = .PutImageVideoFolder
End With End With
End Sub End Sub
Friend Sub New(ByVal s As SiteSettings) Friend Sub New(ByVal s As SiteSettings)
@@ -35,6 +59,14 @@ Namespace API.Instagram
GetStories = CBool(.GetStories.Value) GetStories = CBool(.GetStories.Value)
GetStoriesUser = CBool(.GetStoriesUser.Value) GetStoriesUser = CBool(.GetStoriesUser.Value)
GetTagged = CBool(.GetTagged.Value) GetTagged = CBool(.GetTagged.Value)
GetTimeline_VideoPic = CBool(.GetTimeline_VideoPic.Value)
GetReels_VideoPic = CBool(.GetReels_VideoPic.Value)
GetStories_VideoPic = CBool(.GetStories_VideoPic.Value)
GetStoriesUser_VideoPic = CBool(.GetStoriesUser_VideoPic.Value)
GetTagged_VideoPic = CBool(.GetTagged_VideoPic.Value)
PutImageVideoFolder = CBool(.PutImageVideoFolder.Value)
End With End With
End Sub End Sub
End Class End Class

34
SCrawler/API/Instagram/SiteSettings.vb
View File

@@ -57,6 +57,7 @@ Namespace API.Instagram
#End Region #End Region
#Region "Categories" #Region "Categories"
Private Const CAT_DOWN As String = "Download data" Private Const CAT_DOWN As String = "Download data"
Private Const CAT_UserDefs_VIDEO As String = DN.CAT_UserDefs & ": extract image from video"
#End Region #End Region
#Region "Authorization properties" #Region "Authorization properties"
Friend Const Header_IG_APP_ID As String = "x-ig-app-id" Friend Const Header_IG_APP_ID As String = "x-ig-app-id"
@@ -187,14 +188,28 @@ Namespace API.Instagram
Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
<PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(23), PClonable> <PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(23), PClonable>
Friend ReadOnly Property GetTimeline As PropertyValue Friend ReadOnly Property GetTimeline As PropertyValue
<PropertyOption(ControlText:="From timeline", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(23), PClonable>
Friend ReadOnly Property GetTimeline_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(24), PClonable> <PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(24), PClonable>
Friend ReadOnly Property GetReels As PropertyValue Friend ReadOnly Property GetReels As PropertyValue
<PropertyOption(ControlText:="From reels", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(24), PClonable>
Friend ReadOnly Property GetReels_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(25), PClonable> <PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(25), PClonable>
Friend ReadOnly Property GetStories As PropertyValue Friend ReadOnly Property GetStories As PropertyValue
<PropertyOption(ControlText:="From stories", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(25), PClonable>
Friend ReadOnly Property GetStories_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(26), PClonable> <PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(26), PClonable>
Friend ReadOnly Property GetStoriesUser As PropertyValue Friend ReadOnly Property GetStoriesUser As PropertyValue
<PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable> <PropertyOption(ControlText:="From stories: user", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(26), PClonable>
Friend ReadOnly Property GetStoriesUser_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get tagged posts", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
Friend ReadOnly Property GetTagged As PropertyValue Friend ReadOnly Property GetTagged As PropertyValue
<PropertyOption(ControlText:="From tagged posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(27), PClonable>
Friend ReadOnly Property GetTagged_VideoPic As PropertyValue
<PropertyOption(ControlText:="From saved posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(28), PClonable>
Friend ReadOnly Property GetSavedPosts_VideoPic As PropertyValue
<PropertyOption(ControlText:="Place the extracted image into the video folder", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(29), PClonable>
Friend ReadOnly Property PutImageVideoFolder As PropertyValue
<PropertyOption(ControlText:="Tagged notify limit", <PropertyOption(ControlText:="Tagged notify limit",
ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr & ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
"-1 to disable"), PXML, ControlNumber(27), PClonable> "-1 to disable"), PXML, ControlNumber(27), PClonable>
@@ -203,19 +218,19 @@ Namespace API.Instagram
Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
#End Region #End Region
#Region "Download ready" #Region "Download ready"
<PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable> <PropertyOption(ControlText:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
Friend ReadOnly Property DownloadTimeline As PropertyValue Friend ReadOnly Property DownloadTimeline As PropertyValue
<PXML> Private ReadOnly Property DownloadTimeline_Def As PropertyValue <PXML> Private ReadOnly Property DownloadTimeline_Def As PropertyValue
<PropertyOption(ControlText:="Download reels", ControlToolTip:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable> <PropertyOption(ControlText:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
Friend ReadOnly Property DownloadReels As PropertyValue Friend ReadOnly Property DownloadReels As PropertyValue
<PXML> Private ReadOnly Property DownloadReels_Def As PropertyValue <PXML> Private ReadOnly Property DownloadReels_Def As PropertyValue
<PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable> <PropertyOption(ControlText:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
Friend ReadOnly Property DownloadStories As PropertyValue Friend ReadOnly Property DownloadStories As PropertyValue
<PXML> Private ReadOnly Property DownloadStories_Def As PropertyValue <PXML> Private ReadOnly Property DownloadStories_Def As PropertyValue
<PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable> <PropertyOption(ControlText:="Download stories: user", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
Friend ReadOnly Property DownloadStoriesUser As PropertyValue Friend ReadOnly Property DownloadStoriesUser As PropertyValue
<PXML> Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue <PXML> Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue
<PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable> <PropertyOption(ControlText:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
Friend ReadOnly Property DownloadTagged As PropertyValue Friend ReadOnly Property DownloadTagged As PropertyValue
<PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue <PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue
#End Region #End Region
@@ -425,10 +440,17 @@ Namespace API.Instagram
SleepTimerOnPostsLimitProvider = New TimersChecker(10000) SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
GetTimeline = New PropertyValue(True) GetTimeline = New PropertyValue(True)
GetTimeline_VideoPic = New PropertyValue(True)
GetReels = New PropertyValue(False) GetReels = New PropertyValue(False)
GetReels_VideoPic = New PropertyValue(True)
GetStories = New PropertyValue(False) GetStories = New PropertyValue(False)
GetStories_VideoPic = New PropertyValue(True)
GetStoriesUser = New PropertyValue(False) GetStoriesUser = New PropertyValue(False)
GetStoriesUser_VideoPic = New PropertyValue(True)
GetTagged = New PropertyValue(False) GetTagged = New PropertyValue(False)
GetTagged_VideoPic = New PropertyValue(True)
GetSavedPosts_VideoPic = New PropertyValue(True)
PutImageVideoFolder = New PropertyValue(False)
TaggedNotifyLimit = New PropertyValue(200) TaggedNotifyLimit = New PropertyValue(200)
TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker

4
SCrawler/API/Instagram/UserData.GQL.vb
View File

@@ -194,7 +194,7 @@ Namespace API.Instagram
With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"}) With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"})
If .ListExists Then If .ListExists Then
ProgressPre.ChangeMax(.Count) ProgressPre.ChangeMax(.Count)
For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token) : Next For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token, Sections.Stories) : Next
End If End If
End With End With
End If End If
@@ -217,7 +217,7 @@ Namespace API.Instagram
Using j As EContainer = JsonDocument.Parse(r) Using j As EContainer = JsonDocument.Parse(r)
If j.ListExists Then If j.ListExists Then
Dim i% = -1 Dim i% = -1
GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token) GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token, Sections.UserStories)
End If End If
End Using End Using
End If End If

77
SCrawler/API/Instagram/UserData.vb
View File

@@ -26,10 +26,16 @@ Namespace API.Instagram
Private Const Name_LastCursor As String = "LastCursor" Private Const Name_LastCursor As String = "LastCursor"
Private Const Name_FirstLoadingDone As String = "FirstLoadingDone" Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
Private Const Name_GetTimeline As String = "GetTimeline" Private Const Name_GetTimeline As String = "GetTimeline"
Private Const Name_GetTimeline_VideoPic As String = "GetTimeline_VideoPic"
Private Const Name_GetReels As String = "GetReels" Private Const Name_GetReels As String = "GetReels"
Private Const Name_GetReels_VideoPic As String = "GetReels_VideoPic"
Private Const Name_GetStories As String = "GetStories" Private Const Name_GetStories As String = "GetStories"
Private Const Name_GetStories_VideoPic As String = "GetStories_VideoPic"
Private Const Name_GetStoriesUser As String = "GetStoriesUser" Private Const Name_GetStoriesUser As String = "GetStoriesUser"
Private Const Name_GetStoriesUser_VideoPic As String = "GetStoriesUser_VideoPic"
Private Const Name_GetTagged As String = "GetTaggedData" Private Const Name_GetTagged As String = "GetTaggedData"
Private Const Name_GetTagged_VideoPic As String = "GetTaggedData_VideoPic"
Private Const Name_PutImageVideoFolder As String = "PutImageVideoFolder"
Private Const Name_TaggedChecked As String = "TaggedChecked" Private Const Name_TaggedChecked As String = "TaggedChecked"
Private Const Name_NameTrue As String = "NameTrue" Private Const Name_NameTrue As String = "NameTrue"
#End Region #End Region
@@ -79,10 +85,32 @@ Namespace API.Instagram
Private LastCursor As String = String.Empty Private LastCursor As String = String.Empty
Private FirstLoadingDone As Boolean = False Private FirstLoadingDone As Boolean = False
Friend Property GetTimeline As Boolean = True Friend Property GetTimeline As Boolean = True
Friend Property GetTimeline_VideoPic As Boolean = True
Friend Property GetReels As Boolean = False Friend Property GetReels As Boolean = False
Friend Property GetReels_VideoPic As Boolean = True
Friend Property GetStories As Boolean Friend Property GetStories As Boolean
Friend Property GetStories_VideoPic As Boolean = True
Friend Property GetStoriesUser As Boolean Friend Property GetStoriesUser As Boolean
Friend Property GetStoriesUser_VideoPic As Boolean = True
Friend Property GetTaggedData As Boolean Friend Property GetTaggedData As Boolean
Friend Property GetTaggedData_VideoPic As Boolean = True
Friend Property PutImageVideoFolder As Boolean = False
Private Function ExtractImageFrom(ByVal Section As Sections) As Boolean
Select Case Section
Case Sections.Timeline : Return GetTimeline_VideoPic
Case Sections.Reels : Return GetReels_VideoPic
Case Sections.Tagged : Return GetTaggedData_VideoPic
Case Sections.Stories : Return GetStories_VideoPic
Case Sections.UserStories : Return GetStoriesUser_VideoPic
Case Sections.SavedPosts
Try
If Not HOST Is Nothing AndAlso HOST.Key = InstagramSiteKey Then Return MySiteSettings.GetSavedPosts_VideoPic.Value
Catch
End Try
Return True
Case Else : Return True
End Select
End Function
Protected _NameTrue As String = String.Empty Protected _NameTrue As String = String.Empty
Friend ReadOnly Property NameTrue As String Friend ReadOnly Property NameTrue As String
Get Get
@@ -98,20 +126,32 @@ Namespace API.Instagram
LastCursor = .Value(Name_LastCursor) LastCursor = .Value(Name_LastCursor)
FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False) FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value)) GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value) GetTimeline_VideoPic = .Value(Name_GetTimeline_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline_VideoPic.Value))
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels.Value))
GetReels_VideoPic = .Value(Name_GetReels_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels_VideoPic.Value))
GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value)) GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value) GetStories_VideoPic = .Value(Name_GetStories_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories_VideoPic.Value))
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser.Value))
GetStoriesUser_VideoPic = .Value(Name_GetStoriesUser_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser_VideoPic.Value))
PutImageVideoFolder = .Value(Name_PutImageVideoFolder).FromXML(Of Boolean)(CBool(MySiteSettings.PutImageVideoFolder.Value))
GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value)) GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
GetTaggedData_VideoPic = .Value(Name_GetTagged_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged_VideoPic.Value))
TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False) TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
_NameTrue = .Value(Name_NameTrue) _NameTrue = .Value(Name_NameTrue)
Else Else
.Add(Name_LastCursor, LastCursor) .Add(Name_LastCursor, LastCursor)
.Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger) .Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
.Add(Name_GetTimeline, GetTimeline.BoolToInteger) .Add(Name_GetTimeline, GetTimeline.BoolToInteger)
.Add(Name_GetTimeline_VideoPic, GetTimeline_VideoPic.BoolToInteger)
.Add(Name_GetReels, GetReels.BoolToInteger) .Add(Name_GetReels, GetReels.BoolToInteger)
.Add(Name_GetReels_VideoPic, GetReels_VideoPic.BoolToInteger)
.Add(Name_GetStories, GetStories.BoolToInteger) .Add(Name_GetStories, GetStories.BoolToInteger)
.Add(Name_GetStories_VideoPic, GetStories_VideoPic.BoolToInteger)
.Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger) .Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
.Add(Name_GetStoriesUser_VideoPic, GetStoriesUser_VideoPic.BoolToInteger)
.Add(Name_GetTagged, GetTaggedData.BoolToInteger) .Add(Name_GetTagged, GetTaggedData.BoolToInteger)
.Add(Name_GetTagged_VideoPic, GetTaggedData_VideoPic.BoolToInteger)
.Add(Name_PutImageVideoFolder, PutImageVideoFolder.BoolToInteger)
.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger) .Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
.Add(Name_NameTrue, _NameTrue) .Add(Name_NameTrue, _NameTrue)
End If End If
@@ -130,6 +170,14 @@ Namespace API.Instagram
GetStories = .GetStories GetStories = .GetStories
GetStoriesUser = .GetStoriesUser GetStoriesUser = .GetStoriesUser
GetTaggedData = .GetTagged GetTaggedData = .GetTagged
GetTimeline_VideoPic = .GetTimeline_VideoPic
GetReels_VideoPic = .GetReels_VideoPic
GetStories_VideoPic = .GetStories_VideoPic
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
GetTaggedData_VideoPic = .GetTagged_VideoPic
PutImageVideoFolder = .PutImageVideoFolder
End With End With
End If End If
End Sub End Sub
@@ -809,7 +857,7 @@ NextPageBlock:
With j("items") With j("items")
For Each jj In .Self For Each jj In .Self
before = _TempMediaList.Count before = _TempMediaList.Count
ObtainMedia(jj, PostsToReparse(i).ID, specFolder) ObtainMedia(jj, PostsToReparse(i).ID, specFolder,,,,,,, IIf(IsTagged, Sections.Tagged, Sections.Timeline))
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1 If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
Next Next
@@ -911,7 +959,7 @@ NextPageBlock:
End Select End Select
End If End If
before = _TempMediaList.Count before = _TempMediaList.Count
ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts) ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts,, Section)
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1 If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False
End If End If
@@ -950,6 +998,7 @@ NextPageBlock:
Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_AllowAbstract As Boolean = False Protected ObtainMedia_AllowAbstract As Boolean = False
Private Const ObtainMedia_NoSection As Integer = -10
Protected Sub ObtainMedia_SetReelsFunc() Protected Sub ObtainMedia_SetReelsFunc()
ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
If ss.Value("url").IsEmptyString Then If ss.Value("url").IsEmptyString Then
@@ -971,7 +1020,8 @@ NextPageBlock:
Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1, Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
Optional ByVal PostOriginUrl As String = Nothing, Optional ByVal PostOriginUrl As String = Nothing,
Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0, Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0,
Optional ByVal TryExtractImage As Boolean = False) Optional ByVal TryExtractImage As Boolean = False,
Optional ByVal Section As Sections = ObtainMedia_NoSection)
Try Try
Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0) Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
@@ -1018,6 +1068,12 @@ NextPageBlock:
If TryExtractImage Then If TryExtractImage Then
t = 1 t = 1
abstractDecision = True abstractDecision = True
If Not SpecialFolder.IsEmptyString AndAlso PutImageVideoFolder Then
Dim endsAbs As Boolean = SpecialFolder.EndsWith("*")
If endsAbs Then SpecialFolder = SpecialFolder.TrimEnd("*")
If Not SpecialFolder.IsEmptyString Then SpecialFolder = $"{SpecialFolder.TrimEnd("\")}\{VideoFolderName}{IIf(Not endsAbs, "*", String.Empty)}"
If endsAbs Then SpecialFolder &= "*"
End If
ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then
If n.Contains(vid) Then If n.Contains(vid) Then
t = 2 t = 2
@@ -1064,7 +1120,8 @@ NextPageBlock:
End If End If
End With End With
End If End If
If Not TryExtractImage Then ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True) If Not TryExtractImage And Not Section = ObtainMedia_NoSection And ExtractImageFrom(Section) Then _
ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True, Section)
Case 8 'gallery Case 8 'gallery
DateObj = mDate(n) DateObj = mDate(n)
With n("carousel_media").XmlIfNothing With n("carousel_media").XmlIfNothing
@@ -1165,6 +1222,7 @@ NextPageBlock:
Dim qStr$, r$ Dim qStr$, r$
Dim i% = -1 Dim i% = -1
Dim jj As EContainer Dim jj As EContainer
Dim section As Sections = IIf(GetUserStory, Sections.UserStories, Sections.Stories)
ThrowAny(Token) ThrowAny(Token)
If StoriesList.ListExists Or GetUserStory Then If StoriesList.ListExists Or GetUserStory Then
If Not GetUserStory Then tmpList = StoriesList.Take(5) If Not GetUserStory Then tmpList = StoriesList.Take(5)
@@ -1181,7 +1239,7 @@ NextPageBlock:
Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
If j.Contains("reels") Then If j.Contains("reels") Then
ProgressPre.ChangeMax(j("reels").Count) ProgressPre.ChangeMax(j("reels").Count)
For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token) : Next For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token, section) : Next
End If End If
End Using End Using
End If End If
@@ -1189,7 +1247,8 @@ NextPageBlock:
End If End If
End If End If
End Sub End Sub
Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean, ByVal Token As CancellationToken) Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean,
ByVal Token As CancellationToken, Optional ByVal Section As Sections = Sections.Stories)
If Not Node Is Nothing Then If Not Node Is Nothing Then
With Node With Node
ProgressPre.Perform() ProgressPre.Perform()
@@ -1210,7 +1269,7 @@ NextPageBlock:
pid = storyID & s.Value("id") pid = storyID & s.Value("id")
If Not _TempPostsList.Contains(pid) Then If Not _TempPostsList.Contains(pid) Then
ThrowAny(Token) ThrowAny(Token)
ObtainMedia(s, pid, sFolder) ObtainMedia(s, pid, sFolder,,,,,,, Section)
_TempPostsList.Add(pid) _TempPostsList.Add(pid)
End If End If
Next Next

5
SCrawler/API/OnlyFans/Declarations.vb
View File

@@ -11,6 +11,11 @@ Namespace API.OnlyFans
Friend Module Declarations Friend Module Declarations
Friend ReadOnly DateProvider As New ADateTime("O") Friend ReadOnly DateProvider As New ADateTime("O")
Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue) Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue)
Friend ReadOnly FilesSources As New List(Of Object()) From {
{{"source", "source"}},
{{"files", "source", "url"}},
{{"files", "full", "url"}}
}
Friend Property Rules As DynamicRulesEnv Friend Property Rules As DynamicRulesEnv
End Module End Module
End Namespace End Namespace

21
SCrawler/API/OnlyFans/UserData.vb
View File

@@ -394,6 +394,14 @@ Namespace API.OnlyFans
Loop While Not _complete Loop While Not _complete
End Sub End Sub
#End Region #End Region
Private Function GetMediaURL(ByVal m As EContainer) As String
Dim v$
For Each node As Object() In FilesSources
v = If(m.ItemF(node)?.Value, String.Empty)
If Not v.IsEmptyString Then Return v
Next
Return String.Empty
End Function
Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing, Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False, Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing, Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing,
@@ -405,11 +413,14 @@ Namespace API.OnlyFans
With n("media") With n("media")
If .ListExists Then If .ListExists Then
For Each m In .Self For Each m In .Self
If IsHL Then postUrl = GetMediaURL(m)
postUrl = m.Value({"files", "source"}, "url") 'If IsHL Then
Else ' 'postUrl = m.Value({"files", "source"}, "url")
postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full")) ' postUrl = GetMediaURL(m)
End If 'Else
' 'postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
' postUrl = GetMediaURL(m)
'End If
postUrlBase = String.Empty postUrlBase = String.Empty
Select Case m.Value("type") Select Case m.Value("type")
Case "photo" : t = UTypes.Picture : ext = "jpg" Case "photo" : t = UTypes.Picture : ext = "jpg"

1
SCrawler/API/Twitter/UserData.vb
View File

@@ -90,6 +90,7 @@ Namespace API.Twitter
GifsPrefix = .GifsPrefix GifsPrefix = .GifsPrefix
UseMD5Comparison = .UseMD5Comparison UseMD5Comparison = .UseMD5Comparison
RemoveExistingDuplicates = .RemoveExistingDuplicates RemoveExistingDuplicates = .RemoveExistingDuplicates
If RemoveExistingDuplicates Then StartMD5Checked = False
DownloadModel = DownloadModels.Undefined DownloadModel = DownloadModels.Undefined
DownloadModelForceApply = .DownloadModelForceApply DownloadModelForceApply = .DownloadModelForceApply
MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets

2
SCrawler/API/YouTube/UserData.vb
View File

@@ -203,7 +203,7 @@ Namespace API.YouTube
If IsMusic Or DownloadYTVideos Then If IsMusic Or DownloadYTVideos Then
maxDate = Nothing maxDate = Nothing
LastDownloadDateVideos = nDate(LastDownloadDateVideos) LastDownloadDateVideos = nDate(LastDownloadDateVideos)
url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}" url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/videos"
container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True) container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False) applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now) If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)

13
SCrawler/Download/Feed/DownloadFeedForm.vb
View File

@@ -550,7 +550,7 @@ Namespace DownloadObjects
End Sub End Sub
Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo, Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo,
ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean
Const MsgTitle$ = "Copy/Move checked files" Dim MsgTitle$ = "Copy/Move checked files"
Try Try
Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO) Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO)
Dim moveOptions As FeedMoveCopyTo = Nothing Dim moveOptions As FeedMoveCopyTo = Nothing
@@ -591,7 +591,18 @@ Namespace DownloadObjects
data = {FeedMediaData.Media} data = {FeedMediaData.Media}
data_files = {FeedMediaData.File} data_files = {FeedMediaData.File}
End If End If
MsgTitle = $"{IIf(isCopy, "Copy", "Move")} {IIf(Not FeedMediaData Is Nothing Or GetChecked, "checked", "ALL")} files"
If data.ListExists Then If data.ListExists Then
If (FeedMediaData Is Nothing And Not GetChecked And Not isCopy) AndAlso
MsgBoxE({$"YOU ARE TRYING TO MOVE ALL FEED/SESSION DATA.{vbCr}EVERY FILE WILL BE MOVED, NOT JUST THE SELECTED ONES.", MsgTitle},
vbExclamation,,, {"Process", "Cancel"}) = 1 Then
ShowOperationCanceledMsg(MsgTitle)
Return False
End If
If MCTOptions.Destination.IsEmptyString Then If MCTOptions.Destination.IsEmptyString Then
Using f As New FeedCopyToForm(data_files, isCopy) Using f As New FeedCopyToForm(data_files, isCopy)
f.ShowDialog() f.ShowDialog()

2
SCrawler/MainFrame.vb
View File

@@ -247,7 +247,7 @@ CloseResume:
BTT_DOWN_AUTOMATION.PerformClick() BTT_DOWN_AUTOMATION.PerformClick()
ElseIf e.Alt And e.KeyCode = Keys.P Then ElseIf e.Alt And e.KeyCode = Keys.P Then
BTT_PR_INFO.PerformClick() BTT_PR_INFO.PerformClick()
ElseIf e.Alt And e.KeyCode = Keys.F Then ElseIf (e.Alt And (e.KeyCode = Keys.F Or e.KeyCode = Keys.U)) Or (e.Control And e.KeyCode = Keys.U) Then
MySearch.FormShow() MySearch.FormShow()
Else Else
b = False b = False

4
SCrawler/My Project/AssemblyInfo.vb
View File

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below: ' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")> ' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2024.8.10.0")> <Assembly: AssemblyVersion("2024.9.2.0")>
<Assembly: AssemblyFileVersion("2024.8.10.0")> <Assembly: AssemblyFileVersion("2024.9.2.0")>
<Assembly: NeutralResourcesLanguage("en")> <Assembly: NeutralResourcesLanguage("en")>

21
SCrawler/UserImage.vb
View File

@@ -13,6 +13,25 @@ Friend Class UserImage : Inherits ImageRenderer
Friend Const ImagePostfix_Small As String = "_Small" Friend Const ImagePostfix_Small As String = "_Small"
Private _LargeAddress As SFile Private _LargeAddress As SFile
Private _SmallAddress As SFile Private _SmallAddress As SFile
Private _ForceSaveOrig As Boolean = False
Friend Shared Function NewUserPicture(ByVal ImageOrig As SFile, ByVal Destination As SFile,
Optional ByVal Save As Boolean = True, Optional ByVal GetInstance As Boolean = False) As UserImage
Dim uImg As New UserImage(ImageOrig, Destination)
With uImg
._ForceSaveOrig = ImageOrig.Extension.IsEmptyString OrElse ImageOrig.Extension.ToLower = "gif" OrElse Not {"jpg", "jpeg", "png"}.Contains(ImageOrig.Extension.ToLower)
If Not ._ForceSaveOrig Then
If .Address.Exists AndAlso Not .Address.Delete(SFO.File,, EDP.ReturnValue) Then ._ForceSaveOrig = True
If Not ._ForceSaveOrig AndAlso Not ImageOrig.Copy(.Address) Then ._ForceSaveOrig = True
End If
If Not ._ForceSaveOrig Then
._SmallAddress.Extension = .Address.Extension
._LargeAddress.Extension = .Address.Extension
End If
If Save Then .Save()
End With
If Not GetInstance Then uImg.Dispose() : uImg = Nothing
Return uImg
End Function
Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True) Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True)
MyBase.New(_ImgOriginal) MyBase.New(_ImgOriginal)
Dim f As SFile = Destination Dim f As SFile = Destination
@@ -71,7 +90,7 @@ Friend Class UserImage : Inherits ImageRenderer
End With End With
End Function End Function
Public Overrides Sub Save() Public Overrides Sub Save()
MyBase.Save() If _ForceSaveOrig Then MyBase.Save()
Small.Save(_SmallAddress) Small.Save(_SmallAddress)
Large.Save(_LargeAddress) Large.Save(_LargeAddress)
End Sub End Sub

6
Tools/ArchiveSCrawlerUsersDataFiles.bat
View File

@@ -4,4 +4,8 @@ REM Replace 'd:\Downloads\SocialNetworks\' with the path to your SCrawler data f
REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS. REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS.
REM But I believe 7Zip also has CLI commands REM But I believe 7Zip also has CLI commands
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "d:\Downloads\SocialNetworks\SCrawlerBackup.rar" "d:\Downloads\SocialNetworks\" REM This line archives SCrawler settings files.
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl "D:\MyPrograms\SCrawler\Backup\Settings.rar" "D:\MyPrograms\SCrawler\Settings\"
REM This line archives SCrawler users' settings files.
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "D:\MyPrograms\SCrawler\Backup\SCrawlerBackup.rar" "D:\MyPrograms\SCrawler\Data\"