mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-14 15:52:18 +00:00
2024.9.2.0
Instagram: add options to enable/disable image extraction from video OnlyFans: update to the changed API YouTube: videos are parsed from the 'featured', not from the 'videos' page Feed: add prompt before moving entire feed/session MainFrame: add 'Alt+U' and 'Ctrl+U' to open the user search form UserImage: user image creation update
This commit is contained in:
Andy
16
Changelog.md
16
Changelog.md
@@ -1,3 +1,19 @@
|
||||
# 2024.9.2.0
|
||||
|
||||
*2024-09-02*
|
||||
|
||||
- Added
|
||||
- Instagram: options to enable/disable image extraction from video
|
||||
- Feed: **prompt before moving entire feed/session**
|
||||
- Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
|
||||
- Minor improvements
|
||||
- Updated
|
||||
- gallery-dl up to version **1.27.3**
|
||||
- Fixed
|
||||
- **OnlyFans**: data is not downloading
|
||||
- YouTube (SCrawler): incorrect parsing of video page
|
||||
- Minor bugs
|
||||
|
||||
# 2024.8.10.0
|
||||
|
||||
*2024-08-10*
|
||||
|
||||
31
FAQ.md
31
FAQ.md
@@ -28,22 +28,24 @@ I strongly recommend you to **regularly** create backup copies of the settings f
|
||||
|
||||
**Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
|
||||
|
||||
*How to use: find your problem in the list and read the answer.*
|
||||
|
||||
## General questions
|
||||
- **PROFILES**
|
||||
- I added a profile but **nothing downloaded**: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so.
|
||||
- User downloading failed: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
|
||||
- I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
|
||||
- User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
|
||||
- [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
|
||||
- How to **add profile** to download: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
|
||||
- How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
|
||||
- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
|
||||
- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
|
||||
- [How to report a problem](#how-to-report-a-problem)
|
||||
- I want you to **add the site** to SCrawler: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
|
||||
- What language is SCrawler written in: vb.net
|
||||
- I don't know vb.net and I can't write a plugin. You can write a plugin in `C#`
|
||||
- I have a suggestion, will it be added: maybe if it interested me.
|
||||
- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`): **there is no such functionality and there are no such plans**.
|
||||
- I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
|
||||
- What language is SCrawler written in :arrow_forward: vb.net
|
||||
- I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
|
||||
- I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
|
||||
- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
|
||||
- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
|
||||
- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing**: restore user settings from [backup](#backup).
|
||||
- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
|
||||
- Installation, update and configuration
|
||||
- How to install: https://github.com/AAndyProgram/SCrawler#installation
|
||||
- How to update: https://github.com/AAndyProgram/SCrawler#updating
|
||||
@@ -52,12 +54,15 @@ I strongly recommend you to **regularly** create backup copies of the settings f
|
||||
- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
|
||||
- [Video how to configure](#video-how-to-configure)
|
||||
- **Antivirus**
|
||||
- **Antivirus detects SCrawler as a virus**: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
|
||||
- **Antivirus detects gallery-dl as a virus**: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
|
||||
- **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
|
||||
- **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
|
||||
|
||||
## Sites questions
|
||||
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials)
|
||||
- **META** (Instagram, Threads, Facebook): you need **cookies** and fill in **all fields**
|
||||
|
||||
*How to use: find the site you need in the list and read the answer.*
|
||||
|
||||
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
|
||||
- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
|
||||
- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
|
||||
- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
|
||||
- Porn sites: **COOKIES**!
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB After Width: | Height: | Size: 46 KiB |
@@ -410,9 +410,7 @@ Namespace API.Base
|
||||
End Function
|
||||
Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture
|
||||
Try
|
||||
If f.Exists Then
|
||||
Using p As New UserImage(f, MyFile) : p.Save() : End Using
|
||||
End If
|
||||
If f.Exists Then UserImage.NewUserPicture(f, MyFile)
|
||||
Catch
|
||||
End Try
|
||||
End Sub
|
||||
@@ -451,11 +449,7 @@ BlockPictureScan:
|
||||
New ErrorsDescriber(EDP.ReturnValue) With {
|
||||
.ReturnValue = New List(Of SFile),
|
||||
.ReturnValueExists = True}).FirstOrDefault
|
||||
If NewPicFile.Exists Then
|
||||
p = New UserImage(NewPicFile, MyFile)
|
||||
p.Save()
|
||||
GoTo BlockReturn
|
||||
End If
|
||||
If NewPicFile.Exists Then p = UserImage.NewUserPicture(NewPicFile, MyFile,, True) : GoTo BlockReturn
|
||||
BlockDeletePictureFolder:
|
||||
On Error GoTo BlockReturn
|
||||
If DelPath Then
|
||||
@@ -654,6 +648,7 @@ BlockNullPicture:
|
||||
End Sub
|
||||
Protected ReadOnly _TempMediaList As List(Of UserMedia)
|
||||
Protected ReadOnly _TempPostsList As List(Of String)
|
||||
Private ReadOnly _MD5List As List(Of String)
|
||||
Friend Function GetLastImageAddress() As SFile
|
||||
If _ContentList.Count > 0 Then
|
||||
Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File
|
||||
@@ -679,6 +674,7 @@ BlockNullPicture:
|
||||
Protected MyFileSettings As SFile
|
||||
Protected MyFileData As SFile
|
||||
Protected MyFilePosts As SFile
|
||||
Private MyMD5File As SFile
|
||||
Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists
|
||||
Friend Overridable Property DataMerging As Boolean
|
||||
Get
|
||||
@@ -856,6 +852,7 @@ BlockNullPicture:
|
||||
LatestData = New List(Of UserMedia)
|
||||
_TempMediaList = New List(Of UserMedia)
|
||||
_TempPostsList = New List(Of String)
|
||||
_MD5List = New List(Of String)
|
||||
Labels = New List(Of String)
|
||||
UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
|
||||
UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler)
|
||||
@@ -1037,6 +1034,8 @@ BlockNullPicture:
|
||||
If _ContentList.Count > 0 Then x.AddRange(_ContentList)
|
||||
x.Save(MyFileData)
|
||||
End Using
|
||||
If Not MyMD5File.IsEmptyString And _MD5List.Count > 0 Then _
|
||||
TextSaver.SaveTextToFile(_MD5List.ListToString(Environment.NewLine), MyMD5File, True,, EDP.None)
|
||||
Catch ex As Exception
|
||||
LogError(ex, "history saving error")
|
||||
End Try
|
||||
@@ -1131,6 +1130,7 @@ BlockNullPicture:
|
||||
TokenPersonal = Nothing
|
||||
ProgressPre.Reset()
|
||||
UpdateDataFiles()
|
||||
_MD5Loaded = False
|
||||
_DownloadInProgress = True
|
||||
_DescriptionChecked = False
|
||||
_DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
|
||||
@@ -1212,7 +1212,7 @@ BlockNullPicture:
|
||||
ProgressPre.Done()
|
||||
ThrowAny(Token)
|
||||
|
||||
If UseMD5Comparison And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
|
||||
If RemoveExistingDuplicates And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
|
||||
|
||||
If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then
|
||||
If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd)
|
||||
@@ -1315,6 +1315,11 @@ BlockNullPicture:
|
||||
MyFilePosts = MyFileSettings
|
||||
MyFilePosts.Name &= "_Posts"
|
||||
MyFilePosts.Extension = "txt"
|
||||
If Not IsSavedPosts Then
|
||||
MyMD5File = MyFileSettings
|
||||
MyMD5File.Name &= "_MD5"
|
||||
MyMD5File.Extension = "txt"
|
||||
End If
|
||||
Else
|
||||
Throw New ArgumentNullException("User.File", "User file not detected")
|
||||
End If
|
||||
@@ -1438,23 +1443,25 @@ BlockNullPicture:
|
||||
End Sub
|
||||
#End Region
|
||||
#Region "MD5 support"
|
||||
Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
|
||||
Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
|
||||
Friend Property UseMD5Comparison As Boolean = False
|
||||
Protected Property StartMD5Checked As Boolean = False
|
||||
Friend Property RemoveExistingDuplicates As Boolean = False
|
||||
Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken)
|
||||
Private ReadOnly ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
|
||||
Private _MD5Loaded As Boolean = False
|
||||
Private Sub LoadMD5()
|
||||
Try
|
||||
Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
|
||||
If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
|
||||
Dim i%
|
||||
Dim itemsCount% = 0
|
||||
Dim limit% = If(DownloadTopCount, 0)
|
||||
Dim data As UserMedia = Nothing
|
||||
Dim hashList As New Dictionary(Of String, SFile)
|
||||
Dim f As SFile
|
||||
Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
|
||||
Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
|
||||
Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
|
||||
If Not _MD5Loaded Then
|
||||
_MD5Loaded = True
|
||||
_MD5List.Clear()
|
||||
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(c) c.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
|
||||
If MyMD5File.Exists Then _MD5List.ListAddList(MyMD5File.GetLines, LAP.NotContainsOnly, EDP.ThrowException)
|
||||
End If
|
||||
Catch ex As Exception
|
||||
ErrorsDescriber.Execute(EDP.SendToLog, ex, "LoadMD5")
|
||||
End Try
|
||||
End Sub
|
||||
Private Function ValidateMD5_GetMD5(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
|
||||
Try
|
||||
Dim ImgFormat As Imaging.ImageFormat = Nothing
|
||||
Dim hash$ = String.Empty
|
||||
@@ -1486,33 +1493,44 @@ BlockNullPicture:
|
||||
Return String.Empty
|
||||
End Try
|
||||
End Function
|
||||
If Not StartMD5Checked Then
|
||||
StartMD5Checked = True
|
||||
If _ContentList.Exists(missingMD5) Then
|
||||
Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
|
||||
Dim eIndx%
|
||||
Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
|
||||
Private Sub ValidateMD5(ByVal Token As CancellationToken)
|
||||
Try
|
||||
Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
|
||||
If RemoveExistingDuplicates Then
|
||||
RemoveExistingDuplicates = False
|
||||
_ForceSaveUserInfo = True
|
||||
LoadMD5()
|
||||
Dim i%
|
||||
Dim itemsCount% = 0
|
||||
Dim limit% = If(DownloadTopCount, 0)
|
||||
Dim data As UserMedia = Nothing
|
||||
Dim f As SFile
|
||||
|
||||
If Not StartMD5Checked Then
|
||||
StartMD5Checked = True
|
||||
Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
|
||||
Dim eIndx%
|
||||
Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
|
||||
|
||||
If existingFiles.Count > 0 Then
|
||||
Dim h$
|
||||
ProgressPre.ChangeMax(existingFiles.Count)
|
||||
For i = existingFiles.Count - 1 To 0 Step -1
|
||||
ProgressPre.Perform()
|
||||
h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
|
||||
h = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(i)}, False)
|
||||
If Not h.IsEmptyString Then
|
||||
If hashList.ContainsKey(h) Then
|
||||
MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
|
||||
If _MD5List.Contains(h) Then
|
||||
MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate)"
|
||||
existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
|
||||
existingFiles.RemoveAt(i)
|
||||
Else
|
||||
hashList.Add(h, existingFiles(i))
|
||||
_MD5List.Add(h)
|
||||
End If
|
||||
End If
|
||||
Next
|
||||
End If
|
||||
End If
|
||||
|
||||
If _ContentList.Count > 0 AndAlso _ContentList.Exists(missingMD5) Then
|
||||
ProgressPre.ChangeMax(_ContentList.Count)
|
||||
For i = 0 To _ContentList.Count - 1
|
||||
data = _ContentList(i)
|
||||
@@ -1522,13 +1540,15 @@ BlockNullPicture:
|
||||
ThrowAny(Token)
|
||||
eIndx = existingFiles.FindIndex(eFinder)
|
||||
If eIndx >= 0 Then
|
||||
data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
|
||||
data.MD5 = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
|
||||
If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
|
||||
End If
|
||||
End If
|
||||
existingFiles.RemoveAll(eFinder)
|
||||
End If
|
||||
Next
|
||||
End If
|
||||
|
||||
If existingFiles.Count > 0 Then
|
||||
ProgressPre.ChangeMax(existingFiles.Count)
|
||||
For i = 0 To existingFiles.Count - 1
|
||||
@@ -1540,43 +1560,14 @@ BlockNullPicture:
|
||||
.File = f
|
||||
}
|
||||
ThrowAny(Token)
|
||||
data.MD5 = __getMD5(data, False)
|
||||
data.MD5 = ValidateMD5_GetMD5(data, False)
|
||||
If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
|
||||
Next
|
||||
existingFiles.Clear()
|
||||
End If
|
||||
End If
|
||||
End If
|
||||
|
||||
If _ContentList.Count > 0 Then
|
||||
With _ContentList.Select(Function(d) d.MD5)
|
||||
If .ListExists Then .ToList.ForEach(Sub(md5value) _
|
||||
If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
|
||||
End With
|
||||
End If
|
||||
|
||||
ProgressPre.ChangeMax(_TempMediaList.Count)
|
||||
For i = _TempMediaList.Count - 1 To 0 Step -1
|
||||
ProgressPre.Perform()
|
||||
If limit > 0 And itemsCount >= limit Then
|
||||
_TempMediaList.RemoveAt(i)
|
||||
Else
|
||||
data = _TempMediaList(i)
|
||||
If missingMD5(data) Then
|
||||
ThrowAny(Token)
|
||||
data.MD5 = __getMD5(data, True)
|
||||
If Not data.MD5.IsEmptyString Then
|
||||
If hashList.ContainsKey(data.MD5) Then
|
||||
_TempMediaList.RemoveAt(i)
|
||||
Else
|
||||
hashList.Add(data.MD5, New SFile)
|
||||
_TempMediaList(i) = data
|
||||
itemsCount += 1
|
||||
End If
|
||||
End If
|
||||
End If
|
||||
End If
|
||||
Next
|
||||
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(d) d.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
|
||||
End If
|
||||
Catch iex As ArgumentOutOfRangeException When Disposed
|
||||
Catch ex As Exception
|
||||
@@ -1614,6 +1605,7 @@ BlockNullPicture:
|
||||
Source.Progress.Done()
|
||||
End Sub
|
||||
End Class
|
||||
Protected Const VideoFolderName As String = "Video"
|
||||
Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
|
||||
Try
|
||||
Dim i%
|
||||
@@ -1622,6 +1614,7 @@ BlockNullPicture:
|
||||
If _ContentNew.Count > 0 Then
|
||||
_ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
|
||||
If _ContentNew.Count > 0 Then
|
||||
If UseMD5Comparison Then LoadMD5()
|
||||
MyFile.Exists(SFO.Path)
|
||||
Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
|
||||
Dim MyDir$ = DownloadContentDefault_GetRootDir()
|
||||
@@ -1630,6 +1623,7 @@ BlockNullPicture:
|
||||
Dim __interrupt As Boolean
|
||||
Dim f As SFile
|
||||
Dim v As UserMedia
|
||||
Dim __fileDeleted As Boolean
|
||||
Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
|
||||
Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String)
|
||||
Try
|
||||
@@ -1641,9 +1635,21 @@ BlockNullPicture:
|
||||
ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex)
|
||||
End Try
|
||||
End Sub
|
||||
Dim updateDownCount As Action = Sub()
|
||||
Dim __n% = IIf(__fileDeleted, -1, 1)
|
||||
If __isVideo Then
|
||||
v.Type = UTypes.Video
|
||||
DownloadedVideos(False) += __n
|
||||
ElseIf v.Type = UTypes.GIF Then
|
||||
DownloadedPictures(False) += __n
|
||||
Else
|
||||
v.Type = UTypes.Picture
|
||||
DownloadedPictures(False) += __n
|
||||
End If
|
||||
End Sub
|
||||
|
||||
Using w As New OptionalWebClient(Me)
|
||||
If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
|
||||
If vsf Then CSFileP($"{MyDir}\{VideoFolderName}\").Exists(SFO.Path)
|
||||
Progress.Maximum += _ContentNew.Count
|
||||
If IsSingleObjectDownload Then
|
||||
If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
|
||||
@@ -1671,6 +1677,8 @@ BlockNullPicture:
|
||||
|
||||
If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
|
||||
|
||||
__fileDeleted = False
|
||||
|
||||
If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
|
||||
Try
|
||||
__isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
|
||||
@@ -1691,7 +1699,7 @@ BlockNullPicture:
|
||||
End If
|
||||
If __isVideo And vsf Then
|
||||
If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then
|
||||
f.Path = $"{f.PathWithSeparator}Video"
|
||||
f.Path = $"{f.PathWithSeparator}{VideoFolderName}"
|
||||
If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
|
||||
End If
|
||||
End If
|
||||
@@ -1715,19 +1723,26 @@ BlockNullPicture:
|
||||
End If
|
||||
End If
|
||||
|
||||
If __isVideo Then
|
||||
v.Type = UTypes.Video
|
||||
DownloadedVideos(False) += 1
|
||||
ElseIf v.Type = UTypes.GIF Then
|
||||
DownloadedPictures(False) += 1
|
||||
Else
|
||||
v.Type = UTypes.Picture
|
||||
DownloadedPictures(False) += 1
|
||||
End If
|
||||
updateDownCount()
|
||||
|
||||
v.File = ChangeFileNameByProvider(f, v)
|
||||
v.State = UStates.Downloaded
|
||||
DownloadContentDefault_PostProcessing(v, f, Token)
|
||||
If UseMD5Comparison And (v.Type = UTypes.GIF Or v.Type = UTypes.Picture) Then
|
||||
If v.File.Exists Then
|
||||
v.MD5 = ValidateMD5_GetMD5(v, False)
|
||||
If Not v.MD5.IsEmptyString Then
|
||||
If _MD5List.Contains(v.MD5) Then
|
||||
__fileDeleted = v.File.Delete(SFO.File, SFODelete.DeletePermanently, EDP.ReturnValue)
|
||||
If __fileDeleted Then dCount -= 1 : updateDownCount()
|
||||
Else
|
||||
_MD5List.Add(v.MD5)
|
||||
End If
|
||||
End If
|
||||
Else
|
||||
dCount -= 1
|
||||
End If
|
||||
End If
|
||||
dCount += 1
|
||||
Catch woex As OperationCanceledException When Token.IsCancellationRequested
|
||||
__deleteFile.Invoke(f, v.URL_BASE)
|
||||
@@ -1745,7 +1760,7 @@ BlockNullPicture:
|
||||
Else
|
||||
v.State = UStates.Skipped
|
||||
End If
|
||||
_ContentNew(i) = v
|
||||
If Not __fileDeleted Then _ContentNew(i) = v
|
||||
If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then
|
||||
Progress.Perform(_ContentNew.Count - dTotal)
|
||||
Exit Sub
|
||||
@@ -2240,6 +2255,7 @@ BlockNullPicture:
|
||||
LatestData.Clear()
|
||||
_TempMediaList.Clear()
|
||||
_TempPostsList.Clear()
|
||||
_MD5List.Clear()
|
||||
TokenPersonal = Nothing
|
||||
If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
|
||||
If Not Responser Is Nothing Then Responser.Dispose()
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
Imports SCrawler.Plugin.Attributes
|
||||
Namespace API.Instagram
|
||||
Friend Class EditorExchangeOptions
|
||||
#Region "Download"
|
||||
<PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
|
||||
Friend Property GetTimeline As Boolean
|
||||
<PSetting(Caption:="Get reels", ToolTip:="Download user reels")>
|
||||
@@ -19,6 +20,21 @@ Namespace API.Instagram
|
||||
Friend Property GetStoriesUser As Boolean
|
||||
<PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")>
|
||||
Friend Property GetTagged As Boolean
|
||||
#End Region
|
||||
#Region "Extract image"
|
||||
<PSetting(Caption:="Extract image from video: timeline")>
|
||||
Friend Property GetTimeline_VideoPic As Boolean
|
||||
<PSetting(Caption:="Extract image from video: reels")>
|
||||
Friend Property GetReels_VideoPic As Boolean
|
||||
<PSetting(Caption:="Extract image from video: stories")>
|
||||
Friend Property GetStories_VideoPic As Boolean
|
||||
<PSetting(Caption:="Extract image from video: stories: user")>
|
||||
Friend Property GetStoriesUser_VideoPic As Boolean
|
||||
<PSetting(Caption:="Extract image from video: tagged posts")>
|
||||
Friend Property GetTagged_VideoPic As Boolean
|
||||
#End Region
|
||||
<PSetting(Caption:="Place the extracted image into the video folder")>
|
||||
Friend Property PutImageVideoFolder As Boolean
|
||||
Friend Sub New(ByVal u As UserData)
|
||||
With u
|
||||
GetTimeline = .GetTimeline
|
||||
@@ -26,6 +42,14 @@ Namespace API.Instagram
|
||||
GetStories = .GetStories
|
||||
GetStoriesUser = .GetStoriesUser
|
||||
GetTagged = .GetTaggedData
|
||||
|
||||
GetTimeline_VideoPic = .GetTimeline_VideoPic
|
||||
GetReels_VideoPic = .GetReels_VideoPic
|
||||
GetStories_VideoPic = .GetStories_VideoPic
|
||||
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
|
||||
GetTagged_VideoPic = .GetTaggedData_VideoPic
|
||||
|
||||
PutImageVideoFolder = .PutImageVideoFolder
|
||||
End With
|
||||
End Sub
|
||||
Friend Sub New(ByVal s As SiteSettings)
|
||||
@@ -35,6 +59,14 @@ Namespace API.Instagram
|
||||
GetStories = CBool(.GetStories.Value)
|
||||
GetStoriesUser = CBool(.GetStoriesUser.Value)
|
||||
GetTagged = CBool(.GetTagged.Value)
|
||||
|
||||
GetTimeline_VideoPic = CBool(.GetTimeline_VideoPic.Value)
|
||||
GetReels_VideoPic = CBool(.GetReels_VideoPic.Value)
|
||||
GetStories_VideoPic = CBool(.GetStories_VideoPic.Value)
|
||||
GetStoriesUser_VideoPic = CBool(.GetStoriesUser_VideoPic.Value)
|
||||
GetTagged_VideoPic = CBool(.GetTagged_VideoPic.Value)
|
||||
|
||||
PutImageVideoFolder = CBool(.PutImageVideoFolder.Value)
|
||||
End With
|
||||
End Sub
|
||||
End Class
|
||||
|
||||
@@ -57,6 +57,7 @@ Namespace API.Instagram
|
||||
#End Region
|
||||
#Region "Categories"
|
||||
Private Const CAT_DOWN As String = "Download data"
|
||||
Private Const CAT_UserDefs_VIDEO As String = DN.CAT_UserDefs & ": extract image from video"
|
||||
#End Region
|
||||
#Region "Authorization properties"
|
||||
Friend Const Header_IG_APP_ID As String = "x-ig-app-id"
|
||||
@@ -187,14 +188,28 @@ Namespace API.Instagram
|
||||
Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
|
||||
<PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(23), PClonable>
|
||||
Friend ReadOnly Property GetTimeline As PropertyValue
|
||||
<PropertyOption(ControlText:="From timeline", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(23), PClonable>
|
||||
Friend ReadOnly Property GetTimeline_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(24), PClonable>
|
||||
Friend ReadOnly Property GetReels As PropertyValue
|
||||
<PropertyOption(ControlText:="From reels", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(24), PClonable>
|
||||
Friend ReadOnly Property GetReels_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(25), PClonable>
|
||||
Friend ReadOnly Property GetStories As PropertyValue
|
||||
<PropertyOption(ControlText:="From stories", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(25), PClonable>
|
||||
Friend ReadOnly Property GetStories_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(26), PClonable>
|
||||
Friend ReadOnly Property GetStoriesUser As PropertyValue
|
||||
<PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
|
||||
<PropertyOption(ControlText:="From stories: user", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(26), PClonable>
|
||||
Friend ReadOnly Property GetStoriesUser_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="Get tagged posts", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
|
||||
Friend ReadOnly Property GetTagged As PropertyValue
|
||||
<PropertyOption(ControlText:="From tagged posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(27), PClonable>
|
||||
Friend ReadOnly Property GetTagged_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="From saved posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(28), PClonable>
|
||||
Friend ReadOnly Property GetSavedPosts_VideoPic As PropertyValue
|
||||
<PropertyOption(ControlText:="Place the extracted image into the video folder", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(29), PClonable>
|
||||
Friend ReadOnly Property PutImageVideoFolder As PropertyValue
|
||||
<PropertyOption(ControlText:="Tagged notify limit",
|
||||
ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
|
||||
"-1 to disable"), PXML, ControlNumber(27), PClonable>
|
||||
@@ -203,19 +218,19 @@ Namespace API.Instagram
|
||||
Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
|
||||
#End Region
|
||||
#Region "Download ready"
|
||||
<PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
|
||||
<PropertyOption(ControlText:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
|
||||
Friend ReadOnly Property DownloadTimeline As PropertyValue
|
||||
<PXML> Private ReadOnly Property DownloadTimeline_Def As PropertyValue
|
||||
<PropertyOption(ControlText:="Download reels", ControlToolTip:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
|
||||
<PropertyOption(ControlText:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
|
||||
Friend ReadOnly Property DownloadReels As PropertyValue
|
||||
<PXML> Private ReadOnly Property DownloadReels_Def As PropertyValue
|
||||
<PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
|
||||
<PropertyOption(ControlText:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
|
||||
Friend ReadOnly Property DownloadStories As PropertyValue
|
||||
<PXML> Private ReadOnly Property DownloadStories_Def As PropertyValue
|
||||
<PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
|
||||
<PropertyOption(ControlText:="Download stories: user", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
|
||||
Friend ReadOnly Property DownloadStoriesUser As PropertyValue
|
||||
<PXML> Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue
|
||||
<PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
|
||||
<PropertyOption(ControlText:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
|
||||
Friend ReadOnly Property DownloadTagged As PropertyValue
|
||||
<PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue
|
||||
#End Region
|
||||
@@ -425,10 +440,17 @@ Namespace API.Instagram
|
||||
SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
|
||||
|
||||
GetTimeline = New PropertyValue(True)
|
||||
GetTimeline_VideoPic = New PropertyValue(True)
|
||||
GetReels = New PropertyValue(False)
|
||||
GetReels_VideoPic = New PropertyValue(True)
|
||||
GetStories = New PropertyValue(False)
|
||||
GetStories_VideoPic = New PropertyValue(True)
|
||||
GetStoriesUser = New PropertyValue(False)
|
||||
GetStoriesUser_VideoPic = New PropertyValue(True)
|
||||
GetTagged = New PropertyValue(False)
|
||||
GetTagged_VideoPic = New PropertyValue(True)
|
||||
GetSavedPosts_VideoPic = New PropertyValue(True)
|
||||
PutImageVideoFolder = New PropertyValue(False)
|
||||
TaggedNotifyLimit = New PropertyValue(200)
|
||||
TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker
|
||||
|
||||
|
||||
@@ -194,7 +194,7 @@ Namespace API.Instagram
|
||||
With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"})
|
||||
If .ListExists Then
|
||||
ProgressPre.ChangeMax(.Count)
|
||||
For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token) : Next
|
||||
For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token, Sections.Stories) : Next
|
||||
End If
|
||||
End With
|
||||
End If
|
||||
@@ -217,7 +217,7 @@ Namespace API.Instagram
|
||||
Using j As EContainer = JsonDocument.Parse(r)
|
||||
If j.ListExists Then
|
||||
Dim i% = -1
|
||||
GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token)
|
||||
GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token, Sections.UserStories)
|
||||
End If
|
||||
End Using
|
||||
End If
|
||||
|
||||
@@ -26,10 +26,16 @@ Namespace API.Instagram
|
||||
Private Const Name_LastCursor As String = "LastCursor"
|
||||
Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
|
||||
Private Const Name_GetTimeline As String = "GetTimeline"
|
||||
Private Const Name_GetTimeline_VideoPic As String = "GetTimeline_VideoPic"
|
||||
Private Const Name_GetReels As String = "GetReels"
|
||||
Private Const Name_GetReels_VideoPic As String = "GetReels_VideoPic"
|
||||
Private Const Name_GetStories As String = "GetStories"
|
||||
Private Const Name_GetStories_VideoPic As String = "GetStories_VideoPic"
|
||||
Private Const Name_GetStoriesUser As String = "GetStoriesUser"
|
||||
Private Const Name_GetStoriesUser_VideoPic As String = "GetStoriesUser_VideoPic"
|
||||
Private Const Name_GetTagged As String = "GetTaggedData"
|
||||
Private Const Name_GetTagged_VideoPic As String = "GetTaggedData_VideoPic"
|
||||
Private Const Name_PutImageVideoFolder As String = "PutImageVideoFolder"
|
||||
Private Const Name_TaggedChecked As String = "TaggedChecked"
|
||||
Private Const Name_NameTrue As String = "NameTrue"
|
||||
#End Region
|
||||
@@ -79,10 +85,32 @@ Namespace API.Instagram
|
||||
Private LastCursor As String = String.Empty
|
||||
Private FirstLoadingDone As Boolean = False
|
||||
Friend Property GetTimeline As Boolean = True
|
||||
Friend Property GetTimeline_VideoPic As Boolean = True
|
||||
Friend Property GetReels As Boolean = False
|
||||
Friend Property GetReels_VideoPic As Boolean = True
|
||||
Friend Property GetStories As Boolean
|
||||
Friend Property GetStories_VideoPic As Boolean = True
|
||||
Friend Property GetStoriesUser As Boolean
|
||||
Friend Property GetStoriesUser_VideoPic As Boolean = True
|
||||
Friend Property GetTaggedData As Boolean
|
||||
Friend Property GetTaggedData_VideoPic As Boolean = True
|
||||
Friend Property PutImageVideoFolder As Boolean = False
|
||||
Private Function ExtractImageFrom(ByVal Section As Sections) As Boolean
|
||||
Select Case Section
|
||||
Case Sections.Timeline : Return GetTimeline_VideoPic
|
||||
Case Sections.Reels : Return GetReels_VideoPic
|
||||
Case Sections.Tagged : Return GetTaggedData_VideoPic
|
||||
Case Sections.Stories : Return GetStories_VideoPic
|
||||
Case Sections.UserStories : Return GetStoriesUser_VideoPic
|
||||
Case Sections.SavedPosts
|
||||
Try
|
||||
If Not HOST Is Nothing AndAlso HOST.Key = InstagramSiteKey Then Return MySiteSettings.GetSavedPosts_VideoPic.Value
|
||||
Catch
|
||||
End Try
|
||||
Return True
|
||||
Case Else : Return True
|
||||
End Select
|
||||
End Function
|
||||
Protected _NameTrue As String = String.Empty
|
||||
Friend ReadOnly Property NameTrue As String
|
||||
Get
|
||||
@@ -98,20 +126,32 @@ Namespace API.Instagram
|
||||
LastCursor = .Value(Name_LastCursor)
|
||||
FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
|
||||
GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
|
||||
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value)
|
||||
GetTimeline_VideoPic = .Value(Name_GetTimeline_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline_VideoPic.Value))
|
||||
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels.Value))
|
||||
GetReels_VideoPic = .Value(Name_GetReels_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels_VideoPic.Value))
|
||||
GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
|
||||
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value)
|
||||
GetStories_VideoPic = .Value(Name_GetStories_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories_VideoPic.Value))
|
||||
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser.Value))
|
||||
GetStoriesUser_VideoPic = .Value(Name_GetStoriesUser_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser_VideoPic.Value))
|
||||
PutImageVideoFolder = .Value(Name_PutImageVideoFolder).FromXML(Of Boolean)(CBool(MySiteSettings.PutImageVideoFolder.Value))
|
||||
GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
|
||||
GetTaggedData_VideoPic = .Value(Name_GetTagged_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged_VideoPic.Value))
|
||||
TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
|
||||
_NameTrue = .Value(Name_NameTrue)
|
||||
Else
|
||||
.Add(Name_LastCursor, LastCursor)
|
||||
.Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
|
||||
.Add(Name_GetTimeline, GetTimeline.BoolToInteger)
|
||||
.Add(Name_GetTimeline_VideoPic, GetTimeline_VideoPic.BoolToInteger)
|
||||
.Add(Name_GetReels, GetReels.BoolToInteger)
|
||||
.Add(Name_GetReels_VideoPic, GetReels_VideoPic.BoolToInteger)
|
||||
.Add(Name_GetStories, GetStories.BoolToInteger)
|
||||
.Add(Name_GetStories_VideoPic, GetStories_VideoPic.BoolToInteger)
|
||||
.Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
|
||||
.Add(Name_GetStoriesUser_VideoPic, GetStoriesUser_VideoPic.BoolToInteger)
|
||||
.Add(Name_GetTagged, GetTaggedData.BoolToInteger)
|
||||
.Add(Name_GetTagged_VideoPic, GetTaggedData_VideoPic.BoolToInteger)
|
||||
.Add(Name_PutImageVideoFolder, PutImageVideoFolder.BoolToInteger)
|
||||
.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
|
||||
.Add(Name_NameTrue, _NameTrue)
|
||||
End If
|
||||
@@ -130,6 +170,14 @@ Namespace API.Instagram
|
||||
GetStories = .GetStories
|
||||
GetStoriesUser = .GetStoriesUser
|
||||
GetTaggedData = .GetTagged
|
||||
|
||||
GetTimeline_VideoPic = .GetTimeline_VideoPic
|
||||
GetReels_VideoPic = .GetReels_VideoPic
|
||||
GetStories_VideoPic = .GetStories_VideoPic
|
||||
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
|
||||
GetTaggedData_VideoPic = .GetTagged_VideoPic
|
||||
|
||||
PutImageVideoFolder = .PutImageVideoFolder
|
||||
End With
|
||||
End If
|
||||
End Sub
|
||||
@@ -809,7 +857,7 @@ NextPageBlock:
|
||||
With j("items")
|
||||
For Each jj In .Self
|
||||
before = _TempMediaList.Count
|
||||
ObtainMedia(jj, PostsToReparse(i).ID, specFolder)
|
||||
ObtainMedia(jj, PostsToReparse(i).ID, specFolder,,,,,,, IIf(IsTagged, Sections.Tagged, Sections.Timeline))
|
||||
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
|
||||
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
|
||||
Next
|
||||
@@ -911,7 +959,7 @@ NextPageBlock:
|
||||
End Select
|
||||
End If
|
||||
before = _TempMediaList.Count
|
||||
ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts)
|
||||
ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts,, Section)
|
||||
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
|
||||
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False
|
||||
End If
|
||||
@@ -950,6 +998,7 @@ NextPageBlock:
|
||||
Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
|
||||
Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
|
||||
Protected ObtainMedia_AllowAbstract As Boolean = False
|
||||
Private Const ObtainMedia_NoSection As Integer = -10
|
||||
Protected Sub ObtainMedia_SetReelsFunc()
|
||||
ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
|
||||
If ss.Value("url").IsEmptyString Then
|
||||
@@ -971,7 +1020,8 @@ NextPageBlock:
|
||||
Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
|
||||
Optional ByVal PostOriginUrl As String = Nothing,
|
||||
Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0,
|
||||
Optional ByVal TryExtractImage As Boolean = False)
|
||||
Optional ByVal TryExtractImage As Boolean = False,
|
||||
Optional ByVal Section As Sections = ObtainMedia_NoSection)
|
||||
Try
|
||||
Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
|
||||
Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
|
||||
@@ -1018,6 +1068,12 @@ NextPageBlock:
|
||||
If TryExtractImage Then
|
||||
t = 1
|
||||
abstractDecision = True
|
||||
If Not SpecialFolder.IsEmptyString AndAlso PutImageVideoFolder Then
|
||||
Dim endsAbs As Boolean = SpecialFolder.EndsWith("*")
|
||||
If endsAbs Then SpecialFolder = SpecialFolder.TrimEnd("*")
|
||||
If Not SpecialFolder.IsEmptyString Then SpecialFolder = $"{SpecialFolder.TrimEnd("\")}\{VideoFolderName}{IIf(Not endsAbs, "*", String.Empty)}"
|
||||
If endsAbs Then SpecialFolder &= "*"
|
||||
End If
|
||||
ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then
|
||||
If n.Contains(vid) Then
|
||||
t = 2
|
||||
@@ -1064,7 +1120,8 @@ NextPageBlock:
|
||||
End If
|
||||
End With
|
||||
End If
|
||||
If Not TryExtractImage Then ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True)
|
||||
If Not TryExtractImage And Not Section = ObtainMedia_NoSection And ExtractImageFrom(Section) Then _
|
||||
ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True, Section)
|
||||
Case 8 'gallery
|
||||
DateObj = mDate(n)
|
||||
With n("carousel_media").XmlIfNothing
|
||||
@@ -1165,6 +1222,7 @@ NextPageBlock:
|
||||
Dim qStr$, r$
|
||||
Dim i% = -1
|
||||
Dim jj As EContainer
|
||||
Dim section As Sections = IIf(GetUserStory, Sections.UserStories, Sections.Stories)
|
||||
ThrowAny(Token)
|
||||
If StoriesList.ListExists Or GetUserStory Then
|
||||
If Not GetUserStory Then tmpList = StoriesList.Take(5)
|
||||
@@ -1181,7 +1239,7 @@ NextPageBlock:
|
||||
Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
|
||||
If j.Contains("reels") Then
|
||||
ProgressPre.ChangeMax(j("reels").Count)
|
||||
For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token) : Next
|
||||
For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token, section) : Next
|
||||
End If
|
||||
End Using
|
||||
End If
|
||||
@@ -1189,7 +1247,8 @@ NextPageBlock:
|
||||
End If
|
||||
End If
|
||||
End Sub
|
||||
Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean, ByVal Token As CancellationToken)
|
||||
Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean,
|
||||
ByVal Token As CancellationToken, Optional ByVal Section As Sections = Sections.Stories)
|
||||
If Not Node Is Nothing Then
|
||||
With Node
|
||||
ProgressPre.Perform()
|
||||
@@ -1210,7 +1269,7 @@ NextPageBlock:
|
||||
pid = storyID & s.Value("id")
|
||||
If Not _TempPostsList.Contains(pid) Then
|
||||
ThrowAny(Token)
|
||||
ObtainMedia(s, pid, sFolder)
|
||||
ObtainMedia(s, pid, sFolder,,,,,,, Section)
|
||||
_TempPostsList.Add(pid)
|
||||
End If
|
||||
Next
|
||||
|
||||
@@ -11,6 +11,11 @@ Namespace API.OnlyFans
|
||||
Friend Module Declarations
|
||||
Friend ReadOnly DateProvider As New ADateTime("O")
|
||||
Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue)
|
||||
Friend ReadOnly FilesSources As New List(Of Object()) From {
|
||||
{{"source", "source"}},
|
||||
{{"files", "source", "url"}},
|
||||
{{"files", "full", "url"}}
|
||||
}
|
||||
Friend Property Rules As DynamicRulesEnv
|
||||
End Module
|
||||
End Namespace
|
||||
@@ -394,6 +394,14 @@ Namespace API.OnlyFans
|
||||
Loop While Not _complete
|
||||
End Sub
|
||||
#End Region
|
||||
Private Function GetMediaURL(ByVal m As EContainer) As String
|
||||
Dim v$
|
||||
For Each node As Object() In FilesSources
|
||||
v = If(m.ItemF(node)?.Value, String.Empty)
|
||||
If Not v.IsEmptyString Then Return v
|
||||
Next
|
||||
Return String.Empty
|
||||
End Function
|
||||
Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
|
||||
Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
|
||||
Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing,
|
||||
@@ -405,11 +413,14 @@ Namespace API.OnlyFans
|
||||
With n("media")
|
||||
If .ListExists Then
|
||||
For Each m In .Self
|
||||
If IsHL Then
|
||||
postUrl = m.Value({"files", "source"}, "url")
|
||||
Else
|
||||
postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
|
||||
End If
|
||||
postUrl = GetMediaURL(m)
|
||||
'If IsHL Then
|
||||
' 'postUrl = m.Value({"files", "source"}, "url")
|
||||
' postUrl = GetMediaURL(m)
|
||||
'Else
|
||||
' 'postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
|
||||
' postUrl = GetMediaURL(m)
|
||||
'End If
|
||||
postUrlBase = String.Empty
|
||||
Select Case m.Value("type")
|
||||
Case "photo" : t = UTypes.Picture : ext = "jpg"
|
||||
|
||||
@@ -90,6 +90,7 @@ Namespace API.Twitter
|
||||
GifsPrefix = .GifsPrefix
|
||||
UseMD5Comparison = .UseMD5Comparison
|
||||
RemoveExistingDuplicates = .RemoveExistingDuplicates
|
||||
If RemoveExistingDuplicates Then StartMD5Checked = False
|
||||
DownloadModel = DownloadModels.Undefined
|
||||
DownloadModelForceApply = .DownloadModelForceApply
|
||||
MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets
|
||||
|
||||
@@ -203,7 +203,7 @@ Namespace API.YouTube
|
||||
If IsMusic Or DownloadYTVideos Then
|
||||
maxDate = Nothing
|
||||
LastDownloadDateVideos = nDate(LastDownloadDateVideos)
|
||||
url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
|
||||
url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/videos"
|
||||
container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
|
||||
applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
|
||||
If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)
|
||||
|
||||
@@ -550,7 +550,7 @@ Namespace DownloadObjects
|
||||
End Sub
|
||||
Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo,
|
||||
ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean
|
||||
Const MsgTitle$ = "Copy/Move checked files"
|
||||
Dim MsgTitle$ = "Copy/Move checked files"
|
||||
Try
|
||||
Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO)
|
||||
Dim moveOptions As FeedMoveCopyTo = Nothing
|
||||
@@ -591,7 +591,18 @@ Namespace DownloadObjects
|
||||
data = {FeedMediaData.Media}
|
||||
data_files = {FeedMediaData.File}
|
||||
End If
|
||||
|
||||
MsgTitle = $"{IIf(isCopy, "Copy", "Move")} {IIf(Not FeedMediaData Is Nothing Or GetChecked, "checked", "ALL")} files"
|
||||
|
||||
If data.ListExists Then
|
||||
|
||||
If (FeedMediaData Is Nothing And Not GetChecked And Not isCopy) AndAlso
|
||||
MsgBoxE({$"YOU ARE TRYING TO MOVE ALL FEED/SESSION DATA.{vbCr}EVERY FILE WILL BE MOVED, NOT JUST THE SELECTED ONES.", MsgTitle},
|
||||
vbExclamation,,, {"Process", "Cancel"}) = 1 Then
|
||||
ShowOperationCanceledMsg(MsgTitle)
|
||||
Return False
|
||||
End If
|
||||
|
||||
If MCTOptions.Destination.IsEmptyString Then
|
||||
Using f As New FeedCopyToForm(data_files, isCopy)
|
||||
f.ShowDialog()
|
||||
|
||||
@@ -247,7 +247,7 @@ CloseResume:
|
||||
BTT_DOWN_AUTOMATION.PerformClick()
|
||||
ElseIf e.Alt And e.KeyCode = Keys.P Then
|
||||
BTT_PR_INFO.PerformClick()
|
||||
ElseIf e.Alt And e.KeyCode = Keys.F Then
|
||||
ElseIf (e.Alt And (e.KeyCode = Keys.F Or e.KeyCode = Keys.U)) Or (e.Control And e.KeyCode = Keys.U) Then
|
||||
MySearch.FormShow()
|
||||
Else
|
||||
b = False
|
||||
|
||||
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
|
||||
' by using the '*' as shown below:
|
||||
' <Assembly: AssemblyVersion("1.0.*")>
|
||||
|
||||
<Assembly: AssemblyVersion("2024.8.10.0")>
|
||||
<Assembly: AssemblyFileVersion("2024.8.10.0")>
|
||||
<Assembly: AssemblyVersion("2024.9.2.0")>
|
||||
<Assembly: AssemblyFileVersion("2024.9.2.0")>
|
||||
<Assembly: NeutralResourcesLanguage("en")>
|
||||
|
||||
@@ -13,6 +13,25 @@ Friend Class UserImage : Inherits ImageRenderer
|
||||
Friend Const ImagePostfix_Small As String = "_Small"
|
||||
Private _LargeAddress As SFile
|
||||
Private _SmallAddress As SFile
|
||||
Private _ForceSaveOrig As Boolean = False
|
||||
Friend Shared Function NewUserPicture(ByVal ImageOrig As SFile, ByVal Destination As SFile,
|
||||
Optional ByVal Save As Boolean = True, Optional ByVal GetInstance As Boolean = False) As UserImage
|
||||
Dim uImg As New UserImage(ImageOrig, Destination)
|
||||
With uImg
|
||||
._ForceSaveOrig = ImageOrig.Extension.IsEmptyString OrElse ImageOrig.Extension.ToLower = "gif" OrElse Not {"jpg", "jpeg", "png"}.Contains(ImageOrig.Extension.ToLower)
|
||||
If Not ._ForceSaveOrig Then
|
||||
If .Address.Exists AndAlso Not .Address.Delete(SFO.File,, EDP.ReturnValue) Then ._ForceSaveOrig = True
|
||||
If Not ._ForceSaveOrig AndAlso Not ImageOrig.Copy(.Address) Then ._ForceSaveOrig = True
|
||||
End If
|
||||
If Not ._ForceSaveOrig Then
|
||||
._SmallAddress.Extension = .Address.Extension
|
||||
._LargeAddress.Extension = .Address.Extension
|
||||
End If
|
||||
If Save Then .Save()
|
||||
End With
|
||||
If Not GetInstance Then uImg.Dispose() : uImg = Nothing
|
||||
Return uImg
|
||||
End Function
|
||||
Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True)
|
||||
MyBase.New(_ImgOriginal)
|
||||
Dim f As SFile = Destination
|
||||
@@ -71,7 +90,7 @@ Friend Class UserImage : Inherits ImageRenderer
|
||||
End With
|
||||
End Function
|
||||
Public Overrides Sub Save()
|
||||
MyBase.Save()
|
||||
If _ForceSaveOrig Then MyBase.Save()
|
||||
Small.Save(_SmallAddress)
|
||||
Large.Save(_LargeAddress)
|
||||
End Sub
|
||||
|
||||
@@ -4,4 +4,8 @@ REM Replace 'd:\Downloads\SocialNetworks\' with the path to your SCrawler data f
|
||||
REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS.
|
||||
REM But I believe 7Zip also has CLI commands
|
||||
|
||||
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "d:\Downloads\SocialNetworks\SCrawlerBackup.rar" "d:\Downloads\SocialNetworks\"
|
||||
REM This line archives SCrawler settings files.
|
||||
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl "D:\MyPrograms\SCrawler\Backup\Settings.rar" "D:\MyPrograms\SCrawler\Settings\"
|
||||
|
||||
REM This line archives SCrawler users' settings files.
|
||||
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "D:\MyPrograms\SCrawler\Backup\SCrawlerBackup.rar" "D:\MyPrograms\SCrawler\Data\"
|
||||
Reference in New Issue
Block a user