diff --git a/Changelog.md b/Changelog.md
index 0e7bc93..862f78d 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,3 +1,19 @@
+# 2024.9.2.0
+
+*2024-09-02*
+
+- Added
+ - Instagram: options to enable/disable image extraction from video
+ - Feed: **prompt before moving entire feed/session**
+ - Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
+ - Minor improvements
+- Updated
+ - gallery-dl up to version **1.27.3**
+- Fixed
+ - **OnlyFans**: data is not downloading
+ - YouTube (SCrawler): incorrect parsing of video page
+ - Minor bugs
+
# 2024.8.10.0
*2024-08-10*
diff --git a/FAQ.md b/FAQ.md
index d26eb50..b2f1b34 100644
--- a/FAQ.md
+++ b/FAQ.md
@@ -28,22 +28,24 @@ I strongly recommend you to **regularly** create backup copies of the settings f
**Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
+*How to use: find your problem in the list and read the answer.*
+
## General questions
- **PROFILES**
- - I added a profile but **nothing downloaded**: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so.
- - User downloading failed: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). Don't forget to attach the LOG.
+ - I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
+ - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). Don't forget to attach the LOG.
- [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
- - How to **add profile** to download: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
+ - How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
- [How to report a problem](#how-to-report-a-problem)
-- I want you to **add the site** to SCrawler: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
-- What language is SCrawler written in: vb.net
-- I don't know vb.net and I can't write a plugin. You can write a plugin in `C#`
-- I have a suggestion, will it be added: maybe if it interested me.
-- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`): **there is no such functionality and there are no such plans**.
+- I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
+- What language is SCrawler written in :arrow_forward: vb.net
+- I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
+- I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
+- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
-- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing**: restore user settings from [backup](#backup).
+- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
- Installation, update and configuration
- How to install: https://github.com/AAndyProgram/SCrawler#installation
- How to update: https://github.com/AAndyProgram/SCrawler#updating
@@ -52,12 +54,15 @@ I strongly recommend you to **regularly** create backup copies of the settings f
- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
- [Video how to configure](#video-how-to-configure)
- **Antivirus**
- - **Antivirus detects SCrawler as a virus**: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
- - **Antivirus detects gallery-dl as a virus**: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
+ - **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
+ - **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
## Sites questions
-- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials)
-- **META** (Instagram, Threads, Facebook): you need **cookies** and fill in **all fields**
+
+*How to use: find the site you need in the list and read the answer.*
+
+- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
+- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
- Porn sites: **COOKIES**!
diff --git a/ProgramScreenshots/SettingsSiteInstagram.png b/ProgramScreenshots/SettingsSiteInstagram.png
index 241c61c..d3eca2d 100644
Binary files a/ProgramScreenshots/SettingsSiteInstagram.png and b/ProgramScreenshots/SettingsSiteInstagram.png differ
diff --git a/SCrawler/API/Base/UserDataBase.vb b/SCrawler/API/Base/UserDataBase.vb
index 1bd1d2e..100421d 100644
--- a/SCrawler/API/Base/UserDataBase.vb
+++ b/SCrawler/API/Base/UserDataBase.vb
@@ -410,9 +410,7 @@ Namespace API.Base
End Function
Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture
Try
- If f.Exists Then
- Using p As New UserImage(f, MyFile) : p.Save() : End Using
- End If
+ If f.Exists Then UserImage.NewUserPicture(f, MyFile)
Catch
End Try
End Sub
@@ -451,11 +449,7 @@ BlockPictureScan:
New ErrorsDescriber(EDP.ReturnValue) With {
.ReturnValue = New List(Of SFile),
.ReturnValueExists = True}).FirstOrDefault
- If NewPicFile.Exists Then
- p = New UserImage(NewPicFile, MyFile)
- p.Save()
- GoTo BlockReturn
- End If
+ If NewPicFile.Exists Then p = UserImage.NewUserPicture(NewPicFile, MyFile,, True) : GoTo BlockReturn
BlockDeletePictureFolder:
On Error GoTo BlockReturn
If DelPath Then
@@ -654,6 +648,7 @@ BlockNullPicture:
End Sub
Protected ReadOnly _TempMediaList As List(Of UserMedia)
Protected ReadOnly _TempPostsList As List(Of String)
+ Private ReadOnly _MD5List As List(Of String)
Friend Function GetLastImageAddress() As SFile
If _ContentList.Count > 0 Then
Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File
@@ -679,6 +674,7 @@ BlockNullPicture:
Protected MyFileSettings As SFile
Protected MyFileData As SFile
Protected MyFilePosts As SFile
+ Private MyMD5File As SFile
Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists
Friend Overridable Property DataMerging As Boolean
Get
@@ -856,6 +852,7 @@ BlockNullPicture:
LatestData = New List(Of UserMedia)
_TempMediaList = New List(Of UserMedia)
_TempPostsList = New List(Of String)
+ _MD5List = New List(Of String)
Labels = New List(Of String)
UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler)
@@ -1037,6 +1034,8 @@ BlockNullPicture:
If _ContentList.Count > 0 Then x.AddRange(_ContentList)
x.Save(MyFileData)
End Using
+ If Not MyMD5File.IsEmptyString And _MD5List.Count > 0 Then _
+ TextSaver.SaveTextToFile(_MD5List.ListToString(Environment.NewLine), MyMD5File, True,, EDP.None)
Catch ex As Exception
LogError(ex, "history saving error")
End Try
@@ -1131,6 +1130,7 @@ BlockNullPicture:
TokenPersonal = Nothing
ProgressPre.Reset()
UpdateDataFiles()
+ _MD5Loaded = False
_DownloadInProgress = True
_DescriptionChecked = False
_DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
@@ -1212,7 +1212,7 @@ BlockNullPicture:
ProgressPre.Done()
ThrowAny(Token)
- If UseMD5Comparison And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
+ If RemoveExistingDuplicates And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then
If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd)
@@ -1315,6 +1315,11 @@ BlockNullPicture:
MyFilePosts = MyFileSettings
MyFilePosts.Name &= "_Posts"
MyFilePosts.Extension = "txt"
+ If Not IsSavedPosts Then
+ MyMD5File = MyFileSettings
+ MyMD5File.Name &= "_MD5"
+ MyMD5File.Extension = "txt"
+ End If
Else
Throw New ArgumentNullException("User.File", "User file not detected")
End If
@@ -1438,81 +1443,94 @@ BlockNullPicture:
End Sub
#End Region
#Region "MD5 support"
- Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
+ Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
Friend Property UseMD5Comparison As Boolean = False
Protected Property StartMD5Checked As Boolean = False
Friend Property RemoveExistingDuplicates As Boolean = False
- Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken)
+ Private ReadOnly ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
+ Private _MD5Loaded As Boolean = False
+ Private Sub LoadMD5()
+ Try
+ If Not _MD5Loaded Then
+ _MD5Loaded = True
+ _MD5List.Clear()
+ If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(c) c.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
+ If MyMD5File.Exists Then _MD5List.ListAddList(MyMD5File.GetLines, LAP.NotContainsOnly, EDP.ThrowException)
+ End If
+ Catch ex As Exception
+ ErrorsDescriber.Execute(EDP.SendToLog, ex, "LoadMD5")
+ End Try
+ End Sub
+ Private Function ValidateMD5_GetMD5(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
+ Try
+ Dim ImgFormat As Imaging.ImageFormat = Nothing
+ Dim hash$ = String.Empty
+ Dim __isGif As Boolean = False
+ If __data.Type = UTypes.GIF Then
+ ImgFormat = Imaging.ImageFormat.Gif
+ __isGif = True
+ ElseIf Not __data.File.IsEmptyString Then
+ ImgFormat = GetImageFormat(__data.File)
+ End If
+ If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
+ If IsUrl And Not __isGif Then
+ hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
+ ElseIf IsUrl And __isGif Then
+ hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
+ Else
+ hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+ End If
+ If hash.IsEmptyString And Not __isGif Then
+ If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
+ If IsUrl Then
+ hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
+ Else
+ hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+ End If
+ End If
+ Return hash
+ Catch
+ Return String.Empty
+ End Try
+ End Function
+ Private Sub ValidateMD5(ByVal Token As CancellationToken)
Try
Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
- If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
+ If RemoveExistingDuplicates Then
+ RemoveExistingDuplicates = False
+ _ForceSaveUserInfo = True
+ LoadMD5()
Dim i%
Dim itemsCount% = 0
Dim limit% = If(DownloadTopCount, 0)
Dim data As UserMedia = Nothing
- Dim hashList As New Dictionary(Of String, SFile)
Dim f As SFile
- Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
- Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
- Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
- Try
- Dim ImgFormat As Imaging.ImageFormat = Nothing
- Dim hash$ = String.Empty
- Dim __isGif As Boolean = False
- If __data.Type = UTypes.GIF Then
- ImgFormat = Imaging.ImageFormat.Gif
- __isGif = True
- ElseIf Not __data.File.IsEmptyString Then
- ImgFormat = GetImageFormat(__data.File)
- End If
- If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
- If IsUrl And Not __isGif Then
- hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
- ElseIf IsUrl And __isGif Then
- hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
- Else
- hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
- End If
- If hash.IsEmptyString And Not __isGif Then
- If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
- If IsUrl Then
- hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
- Else
- hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
- End If
- End If
- Return hash
- Catch
- Return String.Empty
- End Try
- End Function
+
If Not StartMD5Checked Then
StartMD5Checked = True
- If _ContentList.Exists(missingMD5) Then
- Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
- Dim eIndx%
- Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
- If RemoveExistingDuplicates Then
- RemoveExistingDuplicates = False
- _ForceSaveUserInfo = True
- If existingFiles.Count > 0 Then
- Dim h$
- ProgressPre.ChangeMax(existingFiles.Count)
- For i = existingFiles.Count - 1 To 0 Step -1
- ProgressPre.Perform()
- h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
- If Not h.IsEmptyString Then
- If hashList.ContainsKey(h) Then
- MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
- existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
- existingFiles.RemoveAt(i)
- Else
- hashList.Add(h, existingFiles(i))
- End If
- End If
- Next
+ Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
+ Dim eIndx%
+ Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
+
+ If existingFiles.Count > 0 Then
+ Dim h$
+ ProgressPre.ChangeMax(existingFiles.Count)
+ For i = existingFiles.Count - 1 To 0 Step -1
+ ProgressPre.Perform()
+ h = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(i)}, False)
+ If Not h.IsEmptyString Then
+ If _MD5List.Contains(h) Then
+ MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate)"
+ existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
+ existingFiles.RemoveAt(i)
+ Else
+ _MD5List.Add(h)
+ End If
End If
- End If
+ Next
+ End If
+
+ If _ContentList.Count > 0 AndAlso _ContentList.Exists(missingMD5) Then
ProgressPre.ChangeMax(_ContentList.Count)
For i = 0 To _ContentList.Count - 1
data = _ContentList(i)
@@ -1522,61 +1540,34 @@ BlockNullPicture:
ThrowAny(Token)
eIndx = existingFiles.FindIndex(eFinder)
If eIndx >= 0 Then
- data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
+ data.MD5 = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
End If
End If
existingFiles.RemoveAll(eFinder)
End If
Next
- If existingFiles.Count > 0 Then
- ProgressPre.ChangeMax(existingFiles.Count)
- For i = 0 To existingFiles.Count - 1
- f = existingFiles(i)
- ProgressPre.Perform()
- data = New UserMedia(f.File) With {
- .State = UStates.Downloaded,
- .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
- .File = f
- }
- ThrowAny(Token)
- data.MD5 = __getMD5(data, False)
- If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
- Next
- existingFiles.Clear()
- End If
End If
- End If
- If _ContentList.Count > 0 Then
- With _ContentList.Select(Function(d) d.MD5)
- If .ListExists Then .ToList.ForEach(Sub(md5value) _
- If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
- End With
- End If
-
- ProgressPre.ChangeMax(_TempMediaList.Count)
- For i = _TempMediaList.Count - 1 To 0 Step -1
- ProgressPre.Perform()
- If limit > 0 And itemsCount >= limit Then
- _TempMediaList.RemoveAt(i)
- Else
- data = _TempMediaList(i)
- If missingMD5(data) Then
+ If existingFiles.Count > 0 Then
+ ProgressPre.ChangeMax(existingFiles.Count)
+ For i = 0 To existingFiles.Count - 1
+ f = existingFiles(i)
+ ProgressPre.Perform()
+ data = New UserMedia(f.File) With {
+ .State = UStates.Downloaded,
+ .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
+ .File = f
+ }
ThrowAny(Token)
- data.MD5 = __getMD5(data, True)
- If Not data.MD5.IsEmptyString Then
- If hashList.ContainsKey(data.MD5) Then
- _TempMediaList.RemoveAt(i)
- Else
- hashList.Add(data.MD5, New SFile)
- _TempMediaList(i) = data
- itemsCount += 1
- End If
- End If
- End If
+ data.MD5 = ValidateMD5_GetMD5(data, False)
+ If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
+ Next
+ existingFiles.Clear()
End If
- Next
+ End If
+
+ If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(d) d.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
End If
Catch iex As ArgumentOutOfRangeException When Disposed
Catch ex As Exception
@@ -1614,6 +1605,7 @@ BlockNullPicture:
Source.Progress.Done()
End Sub
End Class
+ Protected Const VideoFolderName As String = "Video"
Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
Try
Dim i%
@@ -1622,6 +1614,7 @@ BlockNullPicture:
If _ContentNew.Count > 0 Then
_ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
If _ContentNew.Count > 0 Then
+ If UseMD5Comparison Then LoadMD5()
MyFile.Exists(SFO.Path)
Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
Dim MyDir$ = DownloadContentDefault_GetRootDir()
@@ -1630,6 +1623,7 @@ BlockNullPicture:
Dim __interrupt As Boolean
Dim f As SFile
Dim v As UserMedia
+ Dim __fileDeleted As Boolean
Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String)
Try
@@ -1641,9 +1635,21 @@ BlockNullPicture:
ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex)
End Try
End Sub
+ Dim updateDownCount As Action = Sub()
+ Dim __n% = IIf(__fileDeleted, -1, 1)
+ If __isVideo Then
+ v.Type = UTypes.Video
+ DownloadedVideos(False) += __n
+ ElseIf v.Type = UTypes.GIF Then
+ DownloadedPictures(False) += __n
+ Else
+ v.Type = UTypes.Picture
+ DownloadedPictures(False) += __n
+ End If
+ End Sub
Using w As New OptionalWebClient(Me)
- If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
+ If vsf Then CSFileP($"{MyDir}\{VideoFolderName}\").Exists(SFO.Path)
Progress.Maximum += _ContentNew.Count
If IsSingleObjectDownload Then
If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
@@ -1671,6 +1677,8 @@ BlockNullPicture:
If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
+ __fileDeleted = False
+
If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
Try
__isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
@@ -1691,7 +1699,7 @@ BlockNullPicture:
End If
If __isVideo And vsf Then
If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then
- f.Path = $"{f.PathWithSeparator}Video"
+ f.Path = $"{f.PathWithSeparator}{VideoFolderName}"
If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
End If
End If
@@ -1715,19 +1723,26 @@ BlockNullPicture:
End If
End If
- If __isVideo Then
- v.Type = UTypes.Video
- DownloadedVideos(False) += 1
- ElseIf v.Type = UTypes.GIF Then
- DownloadedPictures(False) += 1
- Else
- v.Type = UTypes.Picture
- DownloadedPictures(False) += 1
- End If
+ updateDownCount()
v.File = ChangeFileNameByProvider(f, v)
v.State = UStates.Downloaded
DownloadContentDefault_PostProcessing(v, f, Token)
+ If UseMD5Comparison And (v.Type = UTypes.GIF Or v.Type = UTypes.Picture) Then
+ If v.File.Exists Then
+ v.MD5 = ValidateMD5_GetMD5(v, False)
+ If Not v.MD5.IsEmptyString Then
+ If _MD5List.Contains(v.MD5) Then
+ __fileDeleted = v.File.Delete(SFO.File, SFODelete.DeletePermanently, EDP.ReturnValue)
+ If __fileDeleted Then dCount -= 1 : updateDownCount()
+ Else
+ _MD5List.Add(v.MD5)
+ End If
+ End If
+ Else
+ dCount -= 1
+ End If
+ End If
dCount += 1
Catch woex As OperationCanceledException When Token.IsCancellationRequested
__deleteFile.Invoke(f, v.URL_BASE)
@@ -1745,7 +1760,7 @@ BlockNullPicture:
Else
v.State = UStates.Skipped
End If
- _ContentNew(i) = v
+ If Not __fileDeleted Then _ContentNew(i) = v
If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then
Progress.Perform(_ContentNew.Count - dTotal)
Exit Sub
@@ -2240,6 +2255,7 @@ BlockNullPicture:
LatestData.Clear()
_TempMediaList.Clear()
_TempPostsList.Clear()
+ _MD5List.Clear()
TokenPersonal = Nothing
If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
If Not Responser Is Nothing Then Responser.Dispose()
diff --git a/SCrawler/API/Instagram/EditorExchangeOptions.vb b/SCrawler/API/Instagram/EditorExchangeOptions.vb
index 6171a0e..7de2501 100644
--- a/SCrawler/API/Instagram/EditorExchangeOptions.vb
+++ b/SCrawler/API/Instagram/EditorExchangeOptions.vb
@@ -9,6 +9,7 @@
Imports SCrawler.Plugin.Attributes
Namespace API.Instagram
Friend Class EditorExchangeOptions
+#Region "Download"
Friend Property GetTimeline As Boolean
@@ -19,6 +20,21 @@ Namespace API.Instagram
Friend Property GetStoriesUser As Boolean
Friend Property GetTagged As Boolean
+#End Region
+#Region "Extract image"
+
+ Friend Property GetTimeline_VideoPic As Boolean
+
+ Friend Property GetReels_VideoPic As Boolean
+
+ Friend Property GetStories_VideoPic As Boolean
+
+ Friend Property GetStoriesUser_VideoPic As Boolean
+
+ Friend Property GetTagged_VideoPic As Boolean
+#End Region
+
+ Friend Property PutImageVideoFolder As Boolean
Friend Sub New(ByVal u As UserData)
With u
GetTimeline = .GetTimeline
@@ -26,6 +42,14 @@ Namespace API.Instagram
GetStories = .GetStories
GetStoriesUser = .GetStoriesUser
GetTagged = .GetTaggedData
+
+ GetTimeline_VideoPic = .GetTimeline_VideoPic
+ GetReels_VideoPic = .GetReels_VideoPic
+ GetStories_VideoPic = .GetStories_VideoPic
+ GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
+ GetTagged_VideoPic = .GetTaggedData_VideoPic
+
+ PutImageVideoFolder = .PutImageVideoFolder
End With
End Sub
Friend Sub New(ByVal s As SiteSettings)
@@ -35,6 +59,14 @@ Namespace API.Instagram
GetStories = CBool(.GetStories.Value)
GetStoriesUser = CBool(.GetStoriesUser.Value)
GetTagged = CBool(.GetTagged.Value)
+
+ GetTimeline_VideoPic = CBool(.GetTimeline_VideoPic.Value)
+ GetReels_VideoPic = CBool(.GetReels_VideoPic.Value)
+ GetStories_VideoPic = CBool(.GetStories_VideoPic.Value)
+ GetStoriesUser_VideoPic = CBool(.GetStoriesUser_VideoPic.Value)
+ GetTagged_VideoPic = CBool(.GetTagged_VideoPic.Value)
+
+ PutImageVideoFolder = CBool(.PutImageVideoFolder.Value)
End With
End Sub
End Class
diff --git a/SCrawler/API/Instagram/SiteSettings.vb b/SCrawler/API/Instagram/SiteSettings.vb
index 039594a..cebff7e 100644
--- a/SCrawler/API/Instagram/SiteSettings.vb
+++ b/SCrawler/API/Instagram/SiteSettings.vb
@@ -57,6 +57,7 @@ Namespace API.Instagram
#End Region
#Region "Categories"
Private Const CAT_DOWN As String = "Download data"
+ Private Const CAT_UserDefs_VIDEO As String = DN.CAT_UserDefs & ": extract image from video"
#End Region
#Region "Authorization properties"
Friend Const Header_IG_APP_ID As String = "x-ig-app-id"
@@ -187,14 +188,28 @@ Namespace API.Instagram
Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
Friend ReadOnly Property GetTimeline As PropertyValue
+
+ Friend ReadOnly Property GetTimeline_VideoPic As PropertyValue
Friend ReadOnly Property GetReels As PropertyValue
+
+ Friend ReadOnly Property GetReels_VideoPic As PropertyValue
Friend ReadOnly Property GetStories As PropertyValue
+
+ Friend ReadOnly Property GetStories_VideoPic As PropertyValue
Friend ReadOnly Property GetStoriesUser As PropertyValue
-
+
+ Friend ReadOnly Property GetStoriesUser_VideoPic As PropertyValue
+
Friend ReadOnly Property GetTagged As PropertyValue
+
+ Friend ReadOnly Property GetTagged_VideoPic As PropertyValue
+
+ Friend ReadOnly Property GetSavedPosts_VideoPic As PropertyValue
+
+ Friend ReadOnly Property PutImageVideoFolder As PropertyValue
@@ -203,19 +218,19 @@ Namespace API.Instagram
Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
#End Region
#Region "Download ready"
-
+
Friend ReadOnly Property DownloadTimeline As PropertyValue
Private ReadOnly Property DownloadTimeline_Def As PropertyValue
-
+
Friend ReadOnly Property DownloadReels As PropertyValue
Private ReadOnly Property DownloadReels_Def As PropertyValue
-
+
Friend ReadOnly Property DownloadStories As PropertyValue
Private ReadOnly Property DownloadStories_Def As PropertyValue
-
+
Friend ReadOnly Property DownloadStoriesUser As PropertyValue
Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue
-
+
Friend ReadOnly Property DownloadTagged As PropertyValue
Private ReadOnly Property DownloadTagged_Def As PropertyValue
#End Region
@@ -425,10 +440,17 @@ Namespace API.Instagram
SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
GetTimeline = New PropertyValue(True)
+ GetTimeline_VideoPic = New PropertyValue(True)
GetReels = New PropertyValue(False)
+ GetReels_VideoPic = New PropertyValue(True)
GetStories = New PropertyValue(False)
+ GetStories_VideoPic = New PropertyValue(True)
GetStoriesUser = New PropertyValue(False)
+ GetStoriesUser_VideoPic = New PropertyValue(True)
GetTagged = New PropertyValue(False)
+ GetTagged_VideoPic = New PropertyValue(True)
+ GetSavedPosts_VideoPic = New PropertyValue(True)
+ PutImageVideoFolder = New PropertyValue(False)
TaggedNotifyLimit = New PropertyValue(200)
TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker
diff --git a/SCrawler/API/Instagram/UserData.GQL.vb b/SCrawler/API/Instagram/UserData.GQL.vb
index a436501..ddccb8b 100644
--- a/SCrawler/API/Instagram/UserData.GQL.vb
+++ b/SCrawler/API/Instagram/UserData.GQL.vb
@@ -194,7 +194,7 @@ Namespace API.Instagram
With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"})
If .ListExists Then
ProgressPre.ChangeMax(.Count)
- For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token) : Next
+ For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token, Sections.Stories) : Next
End If
End With
End If
@@ -217,7 +217,7 @@ Namespace API.Instagram
Using j As EContainer = JsonDocument.Parse(r)
If j.ListExists Then
Dim i% = -1
- GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token)
+ GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token, Sections.UserStories)
End If
End Using
End If
diff --git a/SCrawler/API/Instagram/UserData.vb b/SCrawler/API/Instagram/UserData.vb
index b00fa96..12573d1 100644
--- a/SCrawler/API/Instagram/UserData.vb
+++ b/SCrawler/API/Instagram/UserData.vb
@@ -26,10 +26,16 @@ Namespace API.Instagram
Private Const Name_LastCursor As String = "LastCursor"
Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
Private Const Name_GetTimeline As String = "GetTimeline"
+ Private Const Name_GetTimeline_VideoPic As String = "GetTimeline_VideoPic"
Private Const Name_GetReels As String = "GetReels"
+ Private Const Name_GetReels_VideoPic As String = "GetReels_VideoPic"
Private Const Name_GetStories As String = "GetStories"
+ Private Const Name_GetStories_VideoPic As String = "GetStories_VideoPic"
Private Const Name_GetStoriesUser As String = "GetStoriesUser"
+ Private Const Name_GetStoriesUser_VideoPic As String = "GetStoriesUser_VideoPic"
Private Const Name_GetTagged As String = "GetTaggedData"
+ Private Const Name_GetTagged_VideoPic As String = "GetTaggedData_VideoPic"
+ Private Const Name_PutImageVideoFolder As String = "PutImageVideoFolder"
Private Const Name_TaggedChecked As String = "TaggedChecked"
Private Const Name_NameTrue As String = "NameTrue"
#End Region
@@ -79,10 +85,32 @@ Namespace API.Instagram
Private LastCursor As String = String.Empty
Private FirstLoadingDone As Boolean = False
Friend Property GetTimeline As Boolean = True
+ Friend Property GetTimeline_VideoPic As Boolean = True
Friend Property GetReels As Boolean = False
+ Friend Property GetReels_VideoPic As Boolean = True
Friend Property GetStories As Boolean
+ Friend Property GetStories_VideoPic As Boolean = True
Friend Property GetStoriesUser As Boolean
+ Friend Property GetStoriesUser_VideoPic As Boolean = True
Friend Property GetTaggedData As Boolean
+ Friend Property GetTaggedData_VideoPic As Boolean = True
+ Friend Property PutImageVideoFolder As Boolean = False
+ Private Function ExtractImageFrom(ByVal Section As Sections) As Boolean
+ Select Case Section
+ Case Sections.Timeline : Return GetTimeline_VideoPic
+ Case Sections.Reels : Return GetReels_VideoPic
+ Case Sections.Tagged : Return GetTaggedData_VideoPic
+ Case Sections.Stories : Return GetStories_VideoPic
+ Case Sections.UserStories : Return GetStoriesUser_VideoPic
+ Case Sections.SavedPosts
+ Try
+ If Not HOST Is Nothing AndAlso HOST.Key = InstagramSiteKey Then Return MySiteSettings.GetSavedPosts_VideoPic.Value
+ Catch
+ End Try
+ Return True
+ Case Else : Return True
+ End Select
+ End Function
Protected _NameTrue As String = String.Empty
Friend ReadOnly Property NameTrue As String
Get
@@ -98,20 +126,32 @@ Namespace API.Instagram
LastCursor = .Value(Name_LastCursor)
FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
- GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value)
+ GetTimeline_VideoPic = .Value(Name_GetTimeline_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline_VideoPic.Value))
+ GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels.Value))
+ GetReels_VideoPic = .Value(Name_GetReels_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels_VideoPic.Value))
GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
- GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value)
+ GetStories_VideoPic = .Value(Name_GetStories_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories_VideoPic.Value))
+ GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser.Value))
+ GetStoriesUser_VideoPic = .Value(Name_GetStoriesUser_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser_VideoPic.Value))
+ PutImageVideoFolder = .Value(Name_PutImageVideoFolder).FromXML(Of Boolean)(CBool(MySiteSettings.PutImageVideoFolder.Value))
GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
+ GetTaggedData_VideoPic = .Value(Name_GetTagged_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged_VideoPic.Value))
TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
_NameTrue = .Value(Name_NameTrue)
Else
.Add(Name_LastCursor, LastCursor)
.Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
.Add(Name_GetTimeline, GetTimeline.BoolToInteger)
+ .Add(Name_GetTimeline_VideoPic, GetTimeline_VideoPic.BoolToInteger)
.Add(Name_GetReels, GetReels.BoolToInteger)
+ .Add(Name_GetReels_VideoPic, GetReels_VideoPic.BoolToInteger)
.Add(Name_GetStories, GetStories.BoolToInteger)
+ .Add(Name_GetStories_VideoPic, GetStories_VideoPic.BoolToInteger)
.Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
+ .Add(Name_GetStoriesUser_VideoPic, GetStoriesUser_VideoPic.BoolToInteger)
.Add(Name_GetTagged, GetTaggedData.BoolToInteger)
+ .Add(Name_GetTagged_VideoPic, GetTaggedData_VideoPic.BoolToInteger)
+ .Add(Name_PutImageVideoFolder, PutImageVideoFolder.BoolToInteger)
.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
.Add(Name_NameTrue, _NameTrue)
End If
@@ -130,6 +170,14 @@ Namespace API.Instagram
GetStories = .GetStories
GetStoriesUser = .GetStoriesUser
GetTaggedData = .GetTagged
+
+ GetTimeline_VideoPic = .GetTimeline_VideoPic
+ GetReels_VideoPic = .GetReels_VideoPic
+ GetStories_VideoPic = .GetStories_VideoPic
+ GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
+ GetTaggedData_VideoPic = .GetTagged_VideoPic
+
+ PutImageVideoFolder = .PutImageVideoFolder
End With
End If
End Sub
@@ -809,7 +857,7 @@ NextPageBlock:
With j("items")
For Each jj In .Self
before = _TempMediaList.Count
- ObtainMedia(jj, PostsToReparse(i).ID, specFolder)
+ ObtainMedia(jj, PostsToReparse(i).ID, specFolder,,,,,,, IIf(IsTagged, Sections.Tagged, Sections.Timeline))
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
Next
@@ -911,7 +959,7 @@ NextPageBlock:
End Select
End If
before = _TempMediaList.Count
- ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts)
+ ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts,, Section)
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False
End If
@@ -950,6 +998,7 @@ NextPageBlock:
Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_AllowAbstract As Boolean = False
+ Private Const ObtainMedia_NoSection As Integer = -10
Protected Sub ObtainMedia_SetReelsFunc()
ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
If ss.Value("url").IsEmptyString Then
@@ -971,7 +1020,8 @@ NextPageBlock:
Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
Optional ByVal PostOriginUrl As String = Nothing,
Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0,
- Optional ByVal TryExtractImage As Boolean = False)
+ Optional ByVal TryExtractImage As Boolean = False,
+ Optional ByVal Section As Sections = ObtainMedia_NoSection)
Try
Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
@@ -1018,6 +1068,12 @@ NextPageBlock:
If TryExtractImage Then
t = 1
abstractDecision = True
+ If Not SpecialFolder.IsEmptyString AndAlso PutImageVideoFolder Then
+ Dim endsAbs As Boolean = SpecialFolder.EndsWith("*")
+ If endsAbs Then SpecialFolder = SpecialFolder.TrimEnd("*")
+ If Not SpecialFolder.IsEmptyString Then SpecialFolder = $"{SpecialFolder.TrimEnd("\")}\{VideoFolderName}{IIf(Not endsAbs, "*", String.Empty)}"
+ If endsAbs Then SpecialFolder &= "*"
+ End If
ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then
If n.Contains(vid) Then
t = 2
@@ -1064,7 +1120,8 @@ NextPageBlock:
End If
End With
End If
- If Not TryExtractImage Then ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True)
+ If Not TryExtractImage And Not Section = ObtainMedia_NoSection And ExtractImageFrom(Section) Then _
+ ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True, Section)
Case 8 'gallery
DateObj = mDate(n)
With n("carousel_media").XmlIfNothing
@@ -1165,6 +1222,7 @@ NextPageBlock:
Dim qStr$, r$
Dim i% = -1
Dim jj As EContainer
+ Dim section As Sections = IIf(GetUserStory, Sections.UserStories, Sections.Stories)
ThrowAny(Token)
If StoriesList.ListExists Or GetUserStory Then
If Not GetUserStory Then tmpList = StoriesList.Take(5)
@@ -1181,7 +1239,7 @@ NextPageBlock:
Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
If j.Contains("reels") Then
ProgressPre.ChangeMax(j("reels").Count)
- For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token) : Next
+ For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token, section) : Next
End If
End Using
End If
@@ -1189,7 +1247,8 @@ NextPageBlock:
End If
End If
End Sub
- Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean, ByVal Token As CancellationToken)
+ Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean,
+ ByVal Token As CancellationToken, Optional ByVal Section As Sections = Sections.Stories)
If Not Node Is Nothing Then
With Node
ProgressPre.Perform()
@@ -1210,7 +1269,7 @@ NextPageBlock:
pid = storyID & s.Value("id")
If Not _TempPostsList.Contains(pid) Then
ThrowAny(Token)
- ObtainMedia(s, pid, sFolder)
+ ObtainMedia(s, pid, sFolder,,,,,,, Section)
_TempPostsList.Add(pid)
End If
Next
diff --git a/SCrawler/API/OnlyFans/Declarations.vb b/SCrawler/API/OnlyFans/Declarations.vb
index a7938f3..b697206 100644
--- a/SCrawler/API/OnlyFans/Declarations.vb
+++ b/SCrawler/API/OnlyFans/Declarations.vb
@@ -11,6 +11,11 @@ Namespace API.OnlyFans
Friend Module Declarations
Friend ReadOnly DateProvider As New ADateTime("O")
Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue)
+ Friend ReadOnly FilesSources As New List(Of Object()) From {
+ {{"source", "source"}},
+ {{"files", "source", "url"}},
+ {{"files", "full", "url"}}
+ }
Friend Property Rules As DynamicRulesEnv
End Module
End Namespace
\ No newline at end of file
diff --git a/SCrawler/API/OnlyFans/UserData.vb b/SCrawler/API/OnlyFans/UserData.vb
index 777e623..050641d 100644
--- a/SCrawler/API/OnlyFans/UserData.vb
+++ b/SCrawler/API/OnlyFans/UserData.vb
@@ -394,6 +394,14 @@ Namespace API.OnlyFans
Loop While Not _complete
End Sub
#End Region
+ Private Function GetMediaURL(ByVal m As EContainer) As String
+ Dim v$
+ For Each node As Object() In FilesSources
+ v = If(m.ItemF(node)?.Value, String.Empty)
+ If Not v.IsEmptyString Then Return v
+ Next
+ Return String.Empty
+ End Function
Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing,
@@ -405,11 +413,14 @@ Namespace API.OnlyFans
With n("media")
If .ListExists Then
For Each m In .Self
- If IsHL Then
- postUrl = m.Value({"files", "source"}, "url")
- Else
- postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
- End If
+ postUrl = GetMediaURL(m)
+ 'If IsHL Then
+ ' 'postUrl = m.Value({"files", "source"}, "url")
+ ' postUrl = GetMediaURL(m)
+ 'Else
+ ' 'postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
+ ' postUrl = GetMediaURL(m)
+ 'End If
postUrlBase = String.Empty
Select Case m.Value("type")
Case "photo" : t = UTypes.Picture : ext = "jpg"
diff --git a/SCrawler/API/Twitter/UserData.vb b/SCrawler/API/Twitter/UserData.vb
index b492385..a8987b5 100644
--- a/SCrawler/API/Twitter/UserData.vb
+++ b/SCrawler/API/Twitter/UserData.vb
@@ -90,6 +90,7 @@ Namespace API.Twitter
GifsPrefix = .GifsPrefix
UseMD5Comparison = .UseMD5Comparison
RemoveExistingDuplicates = .RemoveExistingDuplicates
+ If RemoveExistingDuplicates Then StartMD5Checked = False
DownloadModel = DownloadModels.Undefined
DownloadModelForceApply = .DownloadModelForceApply
MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets
diff --git a/SCrawler/API/YouTube/UserData.vb b/SCrawler/API/YouTube/UserData.vb
index fab709d..39b1d68 100644
--- a/SCrawler/API/YouTube/UserData.vb
+++ b/SCrawler/API/YouTube/UserData.vb
@@ -203,7 +203,7 @@ Namespace API.YouTube
If IsMusic Or DownloadYTVideos Then
maxDate = Nothing
LastDownloadDateVideos = nDate(LastDownloadDateVideos)
- url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
+ url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/videos"
container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)
diff --git a/SCrawler/Download/Feed/DownloadFeedForm.vb b/SCrawler/Download/Feed/DownloadFeedForm.vb
index d1df807..334a65e 100644
--- a/SCrawler/Download/Feed/DownloadFeedForm.vb
+++ b/SCrawler/Download/Feed/DownloadFeedForm.vb
@@ -550,7 +550,7 @@ Namespace DownloadObjects
End Sub
Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo,
ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean
- Const MsgTitle$ = "Copy/Move checked files"
+ Dim MsgTitle$ = "Copy/Move checked files"
Try
Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO)
Dim moveOptions As FeedMoveCopyTo = Nothing
@@ -591,7 +591,18 @@ Namespace DownloadObjects
data = {FeedMediaData.Media}
data_files = {FeedMediaData.File}
End If
+
+ MsgTitle = $"{IIf(isCopy, "Copy", "Move")} {IIf(Not FeedMediaData Is Nothing Or GetChecked, "checked", "ALL")} files"
+
If data.ListExists Then
+
+ If (FeedMediaData Is Nothing And Not GetChecked And Not isCopy) AndAlso
+ MsgBoxE({$"YOU ARE TRYING TO MOVE ALL FEED/SESSION DATA.{vbCr}EVERY FILE WILL BE MOVED, NOT JUST THE SELECTED ONES.", MsgTitle},
+ vbExclamation,,, {"Process", "Cancel"}) = 1 Then
+ ShowOperationCanceledMsg(MsgTitle)
+ Return False
+ End If
+
If MCTOptions.Destination.IsEmptyString Then
Using f As New FeedCopyToForm(data_files, isCopy)
f.ShowDialog()
diff --git a/SCrawler/MainFrame.vb b/SCrawler/MainFrame.vb
index 69c8bd2..a01c52e 100644
--- a/SCrawler/MainFrame.vb
+++ b/SCrawler/MainFrame.vb
@@ -247,7 +247,7 @@ CloseResume:
BTT_DOWN_AUTOMATION.PerformClick()
ElseIf e.Alt And e.KeyCode = Keys.P Then
BTT_PR_INFO.PerformClick()
- ElseIf e.Alt And e.KeyCode = Keys.F Then
+ ElseIf (e.Alt And (e.KeyCode = Keys.F Or e.KeyCode = Keys.U)) Or (e.Control And e.KeyCode = Keys.U) Then
MySearch.FormShow()
Else
b = False
diff --git a/SCrawler/My Project/AssemblyInfo.vb b/SCrawler/My Project/AssemblyInfo.vb
index 5016e2e..f20c41e 100644
--- a/SCrawler/My Project/AssemblyInfo.vb
+++ b/SCrawler/My Project/AssemblyInfo.vb
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
'
-
-
+
+
diff --git a/SCrawler/UserImage.vb b/SCrawler/UserImage.vb
index 776bb4e..b115e4e 100644
--- a/SCrawler/UserImage.vb
+++ b/SCrawler/UserImage.vb
@@ -13,6 +13,25 @@ Friend Class UserImage : Inherits ImageRenderer
Friend Const ImagePostfix_Small As String = "_Small"
Private _LargeAddress As SFile
Private _SmallAddress As SFile
+ Private _ForceSaveOrig As Boolean = False
+ Friend Shared Function NewUserPicture(ByVal ImageOrig As SFile, ByVal Destination As SFile,
+ Optional ByVal Save As Boolean = True, Optional ByVal GetInstance As Boolean = False) As UserImage
+ Dim uImg As New UserImage(ImageOrig, Destination)
+ With uImg
+ ._ForceSaveOrig = ImageOrig.Extension.IsEmptyString OrElse ImageOrig.Extension.ToLower = "gif" OrElse Not {"jpg", "jpeg", "png"}.Contains(ImageOrig.Extension.ToLower)
+ If Not ._ForceSaveOrig Then
+ If .Address.Exists AndAlso Not .Address.Delete(SFO.File,, EDP.ReturnValue) Then ._ForceSaveOrig = True
+ If Not ._ForceSaveOrig AndAlso Not ImageOrig.Copy(.Address) Then ._ForceSaveOrig = True
+ End If
+ If Not ._ForceSaveOrig Then
+ ._SmallAddress.Extension = .Address.Extension
+ ._LargeAddress.Extension = .Address.Extension
+ End If
+ If Save Then .Save()
+ End With
+ If Not GetInstance Then uImg.Dispose() : uImg = Nothing
+ Return uImg
+ End Function
Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True)
MyBase.New(_ImgOriginal)
Dim f As SFile = Destination
@@ -71,7 +90,7 @@ Friend Class UserImage : Inherits ImageRenderer
End With
End Function
Public Overrides Sub Save()
- MyBase.Save()
+ If _ForceSaveOrig Then MyBase.Save()
Small.Save(_SmallAddress)
Large.Save(_LargeAddress)
End Sub
diff --git a/Tools/ArchiveSCrawlerUsersDataFiles.bat b/Tools/ArchiveSCrawlerUsersDataFiles.bat
index 6e7d187..2b7f23d 100644
--- a/Tools/ArchiveSCrawlerUsersDataFiles.bat
+++ b/Tools/ArchiveSCrawlerUsersDataFiles.bat
@@ -4,4 +4,8 @@ REM Replace 'd:\Downloads\SocialNetworks\' with the path to your SCrawler data f
REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS.
REM But I believe 7Zip also has CLI commands
-"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "d:\Downloads\SocialNetworks\SCrawlerBackup.rar" "d:\Downloads\SocialNetworks\"
\ No newline at end of file
+REM This line archives SCrawler settings files.
+"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl "D:\MyPrograms\SCrawler\Backup\Settings.rar" "D:\MyPrograms\SCrawler\Settings\"
+
+REM This line archives SCrawler users' settings files.
+"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "D:\MyPrograms\SCrawler\Backup\SCrawlerBackup.rar" "D:\MyPrograms\SCrawler\Data\"
\ No newline at end of file