Compare commits

..

6 Commits

Author SHA1 Message Date
Andy
00a06d3e9a 2024.9.2.0
Instagram: add options to enable/disable image extraction from video
OnlyFans: update to the changed API
YouTube: videos are parsed from the 'featured', not from the 'videos' page
Feed: add prompt before moving entire feed/session
MainFrame: add 'Alt+U' and 'Ctrl+U' to open the user search form
UserImage: user image creation update
2024-09-02 18:22:11 +03:00
Andy
2055461829 Update FAQ.md 2024-08-14 13:12:25 +03:00
Andy
723155e20c Update FAQ.md 2024-08-14 12:27:05 +03:00
Andy
effaa3b65b Update FAQ 2024-08-14 11:45:00 +03:00
Andy
e285de10f6 2024.8.10.0
YT
Fix bug when video is parsed using cookies but not downloaded

SCrawler
Feed: add a button to open file folder
2024-08-10 13:36:31 +03:00
Andy
26db0e3e24 2024.8.1.0
Feed: add the ability to set PostUrl for data when moving a file and/or adding to a feed
TDownloader: add 'PostUrl' property to 'UserMediaD'
2024-08-01 20:41:51 +03:00
28 changed files with 555 additions and 286 deletions

View File

@@ -1,17 +1,14 @@
# Contributor's Guide
I welcome requests! Follow these steps to contribute:
Follow these steps to contribute:
1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
1. Let me know you are working on it by posting a comment on the issue.
1. If you find an error in the code, please provide a link to the file and the line number.
1. Let me know you're working on this by posting a comment on this issue.
1. If you find a bug in the code, please provide a link to the file and line number.
1. If you have a code change suggestion, you can post a replacement code block.<!-- I also accept pull requests.-->
# How to report a problem
1. Attach the **profile URLs or links** that you cannot download.
1. Attach the **LOG** if it exists.
1. **Attach the environment information copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text into the issue).**
1. *Add screenshots to illustrate the problem (**optional**)*
**[Read here](https://github.com/AAndyProgram/SCrawler/blob/main/FAQ.md#how-to-report-a-problem)**
# How to build from source
1. Delete the `PersonalUtilities` project from the solution.
@@ -30,7 +27,7 @@ I welcome requests! Follow these steps to contribute:
**I'm currently not accepting requests to develop new sites.**
1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
1. If you don't find anything, create a new issue with your request.
# Requirements for new site requests

View File

@@ -1,3 +1,40 @@
# 2024.9.2.0
*2024-09-02*
- Added
- Instagram: options to enable/disable image extraction from video
- Feed: **prompt before moving entire feed/session**
- Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
- Minor improvements
- Updated
- gallery-dl up to version **1.27.3**
- Fixed
- **OnlyFans**: data is not downloading
- YouTube (SCrawler): incorrect parsing of video page
- Minor bugs
# 2024.8.10.0
*2024-08-10*
- Added
- Feed: button to open file folder
- Updated
- yt-dlp up to version **2024.08.06**
- gallery-dl up to version **1.27.2**
- Fixed
- YouTube (standalone app): **video is being parsed using cookies but is not downloading** *(Issue #205)*
# 2024.8.1.0
*2024-08-01*
- Added
- Minor improvements
- Updated
- yt-dlp up to version **2024.08.01**
# 2024.7.24.0
*2024-07-24*

162
FAQ.md
View File

@@ -1,120 +1,98 @@
# Frequently asked questions
**Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
**Join our Discord server**: https://discord.gg/uFNUXvFFmg
<br/>*You can get help faster there!*
**Also read [here](README.md) for basic information.**
# Docs
- Basic info: https://github.com/AAndyProgram/SCrawler/blob/main/README.md
- **GUIDE**: https://github.com/AAndyProgram/SCrawler/wiki/
- Settings: https://github.com/AAndyProgram/SCrawler/wiki/Settings
- Discord: https://discord.gg/uFNUXvFFmg
Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
Any other questions I will keep in this file.
# Backup
I strongly recommend you to **regularly** create backup copies of the settings files. **An [example script](https://github.com/AAndyProgram/SCrawler/blob/main/Tools/ArchiveSCrawlerUsersDataFiles.bat) for this** on GitHub (you **should adapt** it to your environment, and you can use it when [SCrawler is closed](https://github.com/AAndyProgram/SCrawler/wiki/Settings#behavior)).
----
**This way you'll always have the latest backup of your settings files and can restore it if something goes wrong!**
#### Q: **HOW TO SETUP COOKIES**
A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
----
#### Q: **Does this program have GUI or CLI.**
A: This is a GUI program.
----
#### Q: **Will CLI be added in the future?**
A: NO.
----
#### Q: **I want to add "...." site. How to request.**
<!---A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-a-new-site)--->
**I'm currently not accepting requests to develop new sites.**
----
#### Q: **Site download failed.**
A: Check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
**You also can join our Discord server**: https://discord.gg/uFNUXvFFmg
<br/>*You can get help faster there!*
# How to report a problem
1. **Post your problem [here](https://github.com/AAndyProgram/SCrawler/issues) or in the [help channel](https://discord.com/channels/1124032649682493462/1124281838056259614) on our Discord server**
2. Attach the **profile URLs or links** that you cannot download.
3. Attach the **LOG** if it exists.
4. Attach **the environment information** copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text into the message).
5. *Add screenshots to illustrate the problem (**optional**)*
**ATTENTION! Issues without URLs will be closed without a response!**
----
# Most frequently questions about SCrawler
#### Q: **I have set credentials but still nothing is downloading**
**Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
A: Click the `Start downloading` button or press `F5`
*How to use: find your problem in the list and read the answer.*
----
## General questions
- **PROFILES**
- I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
- User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
- [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
- How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
- [How to report a problem](#how-to-report-a-problem)
- I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
- What language is SCrawler written in :arrow_forward: vb.net
- I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
- I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
- Installation, update and configuration
- How to install: https://github.com/AAndyProgram/SCrawler#installation
- How to update: https://github.com/AAndyProgram/SCrawler#updating
- What file executes the program: **`SCrawler.exe`**
- Where to find binaries: https://github.com/AAndyProgram/SCrawler/releases/latest
- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
- [Video how to configure](#video-how-to-configure)
- **Antivirus**
- **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
- **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
#### Q: **Where can I find the release?**
## Sites questions
A: https://github.com/AAndyProgram/SCrawler/releases/latest
*How to use: find the site you need in the list and read the answer.*
----
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
- Porn sites: **COOKIES**!
- ThisVid: https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid-faq
- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq)
- **JustForFans**: **THE VIDEO ISN'T DOWNLOADING AT THE MOMENT** ([Issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))
#### Q: **How to run the program?**
## Other questions
A: Double-click `SCrawler.exe`
### Does the program remember the last download and check for new posts, downloading only new posts, or does the program download the entire profile every time
The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
----
### Does this program have a GUI or CLI, and will a CLI be added in the future
This is a GUI program and **NO**, <u>CLI will not be added</u>
#### Q: **Where to find binaries?**
### How to remove the label
There is no functionality to remove an individual label. You can open the `Labels.txt` file in the program settings folder and delete any label you want. You also can delete this file (`Labels.txt`). In this case, when SCrawler is launched, the list of labels will be populated only with existing labels (from the user data files).
A: https://github.com/AAndyProgram/SCrawler/releases/latest
### How to remove a user from the blacklist
Just add that user back to the program. In the dialog box that opens, click the `Add and remove from blacklist` button.
----
### You lost me. Your program is too complicated.
**I'm fine with that**. If the program is too complicated for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I develop SCrawler for myself and publish it on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm okay with it.
#### Q: **Does the program remember the last download and check for new posts, downloading only new posts? Or does the program download the entire profile every time?**
### Add a step-by-step guide or video on how to use the program
**NO!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I'll add it. All options and their purposes are described on the wiki. The wiki also contains a description of all the settings and how to configure them. For complex settings there is a step-by-step guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I've developed a program with an intuitive interface. There is a `Settings` button, download buttons, a context menu that appears when you right-click on a user, and other controls. Anyone can use it.
A: The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
**There is already a [video](#video-how-to-configure) example of how to configure a site.**
----
#### Q: **How to redownload all data**
A: https://github.com/AAndyProgram/SCrawler/wiki#redownload-user
----
#### Q: **How to remove the label**
A: There is no functionality to remove an individual label. You can open the `Labels.txt` file in the program settings folder and delete any label you want. You also can delete this file (`Labels.txt`). In this case, when the program starts, the list of labels list will be updated with only existing labels (from the user data files).
----
#### Q: **How to remove a user from the blacklist**
A: Just add that user back to the program. In the dialog box that opens, click on the `Add and remove from blacklist` button.
----
#### Q: **Why don't you answer how it works**
A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
----
#### Q: **You lost me. Your program is too complicated.**
A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
----
#### Q: **I can't configure something**
A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
----
#### Q: **Can you add a step-by-step guide or video on how to use the program?**
A: **NO!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.
# Video how to configure
**The following video was recorded by a user who loves SCrawler and demonstrates how to add credentials using Instagram as an example:**

Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 46 KiB

View File

@@ -82,7 +82,7 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
- **Instagram**
- **Threads**
- **Facebook**
- JustForFans *(partial support)*[^1]
- JustForFans *(partial support) ([video issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))*[^1]
- Mastodon *(out of support)*
- TikTok
- RedGifs

View File

@@ -32,6 +32,7 @@ Namespace API.YouTube.Controls
Private Initialization As Boolean = True
Private ReadOnly InheritsFromContainer As Boolean
Private ReadOnly M3U8Files As List(Of SFile)
Friend Property UseCookies As Boolean = False
Private ReadOnly Property M3U8FilesFull As List(Of SFile)
Get
Return ListAddList(Nothing, M3U8Files, LAP.NotContainsOnly).ListAddValue(CMB_PLS.Text, LAP.NotContainsOnly)
@@ -65,6 +66,7 @@ Namespace API.YouTube.Controls
CNT_PROCESSOR = New TableControlsProcessor(TP_CONTROLS)
Me.InheritsFromContainer = InheritsFromContainer
MyFieldsChecker = New FieldsChecker
UseCookies = MyYouTubeSettings.DefaultUseCookies
End Sub
#End Region
#Region "Form handlers"
@@ -227,7 +229,7 @@ Namespace API.YouTube.Controls
Dim data As IEnumerable(Of Control)
If .HasElements Then
data = .Elements.Select(Function(ee) New MediaItem(ee, True) With {.Dock = DockStyle.Fill, .Checked = ee.Checked})
data = .Elements.Select(Function(ee) New MediaItem(ee, True) With {.Dock = DockStyle.Fill, .Checked = ee.Checked, .UseCookies = UseCookies})
Else
data = (From m As MediaObject In .Self.MediaObjects
Where m.Type = __contentType

View File

@@ -437,7 +437,7 @@ Namespace DownloadObjects.STDownloader
Else
RaiseEvent BeforeOpenEditor(Me, MyContainer)
End If
Using f As New VideoOptionsForm(MyContainer, initProtected Or isFull)
Using f As New VideoOptionsForm(MyContainer, initProtected Or isFull) With {.UseCookies = UseCookies}
f.ShowDialog()
.Protected = IIf(f.DialogResult = DialogResult.OK, True, initProtected)
End Using
@@ -465,12 +465,12 @@ Namespace DownloadObjects.STDownloader
If Not MyContainer Is Nothing Then
Dim f As Form = Nothing
Select Case MyContainer.ObjectType
Case Base.YouTubeMediaType.Single : f = New VideoOptionsForm(MyContainer, True)
Case Base.YouTubeMediaType.Single : f = New VideoOptionsForm(MyContainer, True) With {.UseCookies = UseCookies}
Case Base.YouTubeMediaType.Channel, Base.YouTubeMediaType.PlayList
If MyContainer.IsMusic Then
f = New MusicPlaylistsForm(MyContainer)
Else
f = New VideoOptionsForm(MyContainer, True)
f = New VideoOptionsForm(MyContainer, True) With {.UseCookies = UseCookies}
End If
End Select
If Not f Is Nothing Then

View File

@@ -132,7 +132,8 @@ Namespace DownloadObjects.STDownloader
#End Region
#Region "Controls"
Protected Sub ControlCreateAndAdd(ByVal Container As IYouTubeMediaContainer, Optional ByVal DisableDownload As Boolean = False,
Optional ByVal PerformClick As Boolean = True, Optional ByVal IsLoading As Boolean = False)
Optional ByVal PerformClick As Boolean = True, Optional ByVal IsLoading As Boolean = False,
Optional ByVal UseCookies As Boolean = False)
ControlInvokeFast(TP_CONTROLS, Sub()
With TP_CONTROLS
.SuspendLayout()
@@ -142,7 +143,7 @@ Namespace DownloadObjects.STDownloader
.RowStyles.Insert(0, New RowStyle(SizeType.Absolute, 60))
.RowCount = .RowStyles.Count
OffsetControls(0, True)
Dim cnt As New MediaItem(Container) With {.Dock = DockStyle.Fill, .Margin = New Padding(0)}
Dim cnt As New MediaItem(Container) With {.Dock = DockStyle.Fill, .Margin = New Padding(0), .UseCookies = UseCookies}
AddHandler cnt.FileDownloaded, AddressOf MediaControl_FileDownloaded
AddHandler cnt.Removal, AddressOf MediaControl_Removal
AddHandler cnt.DownloadAgain, AddressOf MediaControl_DownloadAgain
@@ -333,19 +334,19 @@ Namespace DownloadObjects.STDownloader
If Not c Is Nothing Then
Dim f As Form
Select Case c.ObjectType
Case YouTubeMediaType.Single : f = New VideoOptionsForm(c)
Case YouTubeMediaType.Single : f = New VideoOptionsForm(c) With {.UseCookies = useCookies}
Case YouTubeMediaType.Channel, YouTubeMediaType.PlayList
If c.IsMusic Then
f = New MusicPlaylistsForm(c)
Else
f = New VideoOptionsForm(c)
f = New VideoOptionsForm(c) With {.UseCookies = useCookies}
End If
Case Else : c.Dispose() : Throw New ArgumentException($"Object type {c.ObjectType} not implemented", "IYouTubeMediaContainer.ObjectType")
End Select
If Not f Is Nothing Then
If TypeOf f Is IDesignXMLContainer Then DirectCast(f, IDesignXMLContainer).DesignXML = DesignXML
f.ShowDialog()
If f.DialogResult = DialogResult.OK AndAlso ValidateContainerURL(c) Then ControlCreateAndAdd(c, disableDown)
If f.DialogResult = DialogResult.OK AndAlso ValidateContainerURL(c) Then ControlCreateAndAdd(c, disableDown,,, useCookies)
f.Dispose()
End If
End If

View File

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2024.7.24.0")>
<Assembly: AssemblyFileVersion("2024.7.24.0")>
<Assembly: AssemblyVersion("2024.8.10.0")>
<Assembly: AssemblyFileVersion("2024.8.10.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2024.7.24.0")>
<Assembly: AssemblyFileVersion("2024.7.24.0")>
<Assembly: AssemblyVersion("2024.8.10.0")>
<Assembly: AssemblyFileVersion("2024.8.10.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -410,9 +410,7 @@ Namespace API.Base
End Function
Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture
Try
If f.Exists Then
Using p As New UserImage(f, MyFile) : p.Save() : End Using
End If
If f.Exists Then UserImage.NewUserPicture(f, MyFile)
Catch
End Try
End Sub
@@ -451,11 +449,7 @@ BlockPictureScan:
New ErrorsDescriber(EDP.ReturnValue) With {
.ReturnValue = New List(Of SFile),
.ReturnValueExists = True}).FirstOrDefault
If NewPicFile.Exists Then
p = New UserImage(NewPicFile, MyFile)
p.Save()
GoTo BlockReturn
End If
If NewPicFile.Exists Then p = UserImage.NewUserPicture(NewPicFile, MyFile,, True) : GoTo BlockReturn
BlockDeletePictureFolder:
On Error GoTo BlockReturn
If DelPath Then
@@ -654,6 +648,7 @@ BlockNullPicture:
End Sub
Protected ReadOnly _TempMediaList As List(Of UserMedia)
Protected ReadOnly _TempPostsList As List(Of String)
Private ReadOnly _MD5List As List(Of String)
Friend Function GetLastImageAddress() As SFile
If _ContentList.Count > 0 Then
Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File
@@ -679,6 +674,7 @@ BlockNullPicture:
Protected MyFileSettings As SFile
Protected MyFileData As SFile
Protected MyFilePosts As SFile
Private MyMD5File As SFile
Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists
Friend Overridable Property DataMerging As Boolean
Get
@@ -856,6 +852,7 @@ BlockNullPicture:
LatestData = New List(Of UserMedia)
_TempMediaList = New List(Of UserMedia)
_TempPostsList = New List(Of String)
_MD5List = New List(Of String)
Labels = New List(Of String)
UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler)
@@ -1037,6 +1034,8 @@ BlockNullPicture:
If _ContentList.Count > 0 Then x.AddRange(_ContentList)
x.Save(MyFileData)
End Using
If Not MyMD5File.IsEmptyString And _MD5List.Count > 0 Then _
TextSaver.SaveTextToFile(_MD5List.ListToString(Environment.NewLine), MyMD5File, True,, EDP.None)
Catch ex As Exception
LogError(ex, "history saving error")
End Try
@@ -1131,6 +1130,7 @@ BlockNullPicture:
TokenPersonal = Nothing
ProgressPre.Reset()
UpdateDataFiles()
_MD5Loaded = False
_DownloadInProgress = True
_DescriptionChecked = False
_DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
@@ -1212,7 +1212,7 @@ BlockNullPicture:
ProgressPre.Done()
ThrowAny(Token)
If UseMD5Comparison And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
If RemoveExistingDuplicates And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then
If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd)
@@ -1315,6 +1315,11 @@ BlockNullPicture:
MyFilePosts = MyFileSettings
MyFilePosts.Name &= "_Posts"
MyFilePosts.Extension = "txt"
If Not IsSavedPosts Then
MyMD5File = MyFileSettings
MyMD5File.Name &= "_MD5"
MyMD5File.Extension = "txt"
End If
Else
Throw New ArgumentNullException("User.File", "User file not detected")
End If
@@ -1438,81 +1443,94 @@ BlockNullPicture:
End Sub
#End Region
#Region "MD5 support"
Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
Friend Property UseMD5Comparison As Boolean = False
Protected Property StartMD5Checked As Boolean = False
Friend Property RemoveExistingDuplicates As Boolean = False
Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken)
Private ReadOnly ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
Private _MD5Loaded As Boolean = False
Private Sub LoadMD5()
Try
If Not _MD5Loaded Then
_MD5Loaded = True
_MD5List.Clear()
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(c) c.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
If MyMD5File.Exists Then _MD5List.ListAddList(MyMD5File.GetLines, LAP.NotContainsOnly, EDP.ThrowException)
End If
Catch ex As Exception
ErrorsDescriber.Execute(EDP.SendToLog, ex, "LoadMD5")
End Try
End Sub
Private Function ValidateMD5_GetMD5(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
Try
Dim ImgFormat As Imaging.ImageFormat = Nothing
Dim hash$ = String.Empty
Dim __isGif As Boolean = False
If __data.Type = UTypes.GIF Then
ImgFormat = Imaging.ImageFormat.Gif
__isGif = True
ElseIf Not __data.File.IsEmptyString Then
ImgFormat = GetImageFormat(__data.File)
End If
If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl And Not __isGif Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
ElseIf IsUrl And __isGif Then
hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
If hash.IsEmptyString And Not __isGif Then
If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
End If
Return hash
Catch
Return String.Empty
End Try
End Function
Private Sub ValidateMD5(ByVal Token As CancellationToken)
Try
Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
If RemoveExistingDuplicates Then
RemoveExistingDuplicates = False
_ForceSaveUserInfo = True
LoadMD5()
Dim i%
Dim itemsCount% = 0
Dim limit% = If(DownloadTopCount, 0)
Dim data As UserMedia = Nothing
Dim hashList As New Dictionary(Of String, SFile)
Dim f As SFile
Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
Try
Dim ImgFormat As Imaging.ImageFormat = Nothing
Dim hash$ = String.Empty
Dim __isGif As Boolean = False
If __data.Type = UTypes.GIF Then
ImgFormat = Imaging.ImageFormat.Gif
__isGif = True
ElseIf Not __data.File.IsEmptyString Then
ImgFormat = GetImageFormat(__data.File)
End If
If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl And Not __isGif Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
ElseIf IsUrl And __isGif Then
hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
If hash.IsEmptyString And Not __isGif Then
If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
If IsUrl Then
hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
Else
hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
End If
End If
Return hash
Catch
Return String.Empty
End Try
End Function
If Not StartMD5Checked Then
StartMD5Checked = True
If _ContentList.Exists(missingMD5) Then
Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
Dim eIndx%
Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
If RemoveExistingDuplicates Then
RemoveExistingDuplicates = False
_ForceSaveUserInfo = True
If existingFiles.Count > 0 Then
Dim h$
ProgressPre.ChangeMax(existingFiles.Count)
For i = existingFiles.Count - 1 To 0 Step -1
ProgressPre.Perform()
h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
If Not h.IsEmptyString Then
If hashList.ContainsKey(h) Then
MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
existingFiles.RemoveAt(i)
Else
hashList.Add(h, existingFiles(i))
End If
End If
Next
Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
Dim eIndx%
Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
If existingFiles.Count > 0 Then
Dim h$
ProgressPre.ChangeMax(existingFiles.Count)
For i = existingFiles.Count - 1 To 0 Step -1
ProgressPre.Perform()
h = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(i)}, False)
If Not h.IsEmptyString Then
If _MD5List.Contains(h) Then
MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate)"
existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
existingFiles.RemoveAt(i)
Else
_MD5List.Add(h)
End If
End If
End If
Next
End If
If _ContentList.Count > 0 AndAlso _ContentList.Exists(missingMD5) Then
ProgressPre.ChangeMax(_ContentList.Count)
For i = 0 To _ContentList.Count - 1
data = _ContentList(i)
@@ -1522,61 +1540,34 @@ BlockNullPicture:
ThrowAny(Token)
eIndx = existingFiles.FindIndex(eFinder)
If eIndx >= 0 Then
data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
data.MD5 = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
End If
End If
existingFiles.RemoveAll(eFinder)
End If
Next
If existingFiles.Count > 0 Then
ProgressPre.ChangeMax(existingFiles.Count)
For i = 0 To existingFiles.Count - 1
f = existingFiles(i)
ProgressPre.Perform()
data = New UserMedia(f.File) With {
.State = UStates.Downloaded,
.Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
.File = f
}
ThrowAny(Token)
data.MD5 = __getMD5(data, False)
If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
Next
existingFiles.Clear()
End If
End If
End If
If _ContentList.Count > 0 Then
With _ContentList.Select(Function(d) d.MD5)
If .ListExists Then .ToList.ForEach(Sub(md5value) _
If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
End With
End If
ProgressPre.ChangeMax(_TempMediaList.Count)
For i = _TempMediaList.Count - 1 To 0 Step -1
ProgressPre.Perform()
If limit > 0 And itemsCount >= limit Then
_TempMediaList.RemoveAt(i)
Else
data = _TempMediaList(i)
If missingMD5(data) Then
If existingFiles.Count > 0 Then
ProgressPre.ChangeMax(existingFiles.Count)
For i = 0 To existingFiles.Count - 1
f = existingFiles(i)
ProgressPre.Perform()
data = New UserMedia(f.File) With {
.State = UStates.Downloaded,
.Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
.File = f
}
ThrowAny(Token)
data.MD5 = __getMD5(data, True)
If Not data.MD5.IsEmptyString Then
If hashList.ContainsKey(data.MD5) Then
_TempMediaList.RemoveAt(i)
Else
hashList.Add(data.MD5, New SFile)
_TempMediaList(i) = data
itemsCount += 1
End If
End If
End If
data.MD5 = ValidateMD5_GetMD5(data, False)
If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
Next
existingFiles.Clear()
End If
Next
End If
If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(d) d.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
End If
Catch iex As ArgumentOutOfRangeException When Disposed
Catch ex As Exception
@@ -1614,6 +1605,7 @@ BlockNullPicture:
Source.Progress.Done()
End Sub
End Class
Protected Const VideoFolderName As String = "Video"
Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
Try
Dim i%
@@ -1622,6 +1614,7 @@ BlockNullPicture:
If _ContentNew.Count > 0 Then
_ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
If _ContentNew.Count > 0 Then
If UseMD5Comparison Then LoadMD5()
MyFile.Exists(SFO.Path)
Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
Dim MyDir$ = DownloadContentDefault_GetRootDir()
@@ -1630,6 +1623,7 @@ BlockNullPicture:
Dim __interrupt As Boolean
Dim f As SFile
Dim v As UserMedia
Dim __fileDeleted As Boolean
Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String)
Try
@@ -1641,9 +1635,21 @@ BlockNullPicture:
ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex)
End Try
End Sub
Dim updateDownCount As Action = Sub()
Dim __n% = IIf(__fileDeleted, -1, 1)
If __isVideo Then
v.Type = UTypes.Video
DownloadedVideos(False) += __n
ElseIf v.Type = UTypes.GIF Then
DownloadedPictures(False) += __n
Else
v.Type = UTypes.Picture
DownloadedPictures(False) += __n
End If
End Sub
Using w As New OptionalWebClient(Me)
If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
If vsf Then CSFileP($"{MyDir}\{VideoFolderName}\").Exists(SFO.Path)
Progress.Maximum += _ContentNew.Count
If IsSingleObjectDownload Then
If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
@@ -1671,6 +1677,8 @@ BlockNullPicture:
If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
__fileDeleted = False
If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
Try
__isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
@@ -1691,7 +1699,7 @@ BlockNullPicture:
End If
If __isVideo And vsf Then
If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then
f.Path = $"{f.PathWithSeparator}Video"
f.Path = $"{f.PathWithSeparator}{VideoFolderName}"
If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
End If
End If
@@ -1715,19 +1723,26 @@ BlockNullPicture:
End If
End If
If __isVideo Then
v.Type = UTypes.Video
DownloadedVideos(False) += 1
ElseIf v.Type = UTypes.GIF Then
DownloadedPictures(False) += 1
Else
v.Type = UTypes.Picture
DownloadedPictures(False) += 1
End If
updateDownCount()
v.File = ChangeFileNameByProvider(f, v)
v.State = UStates.Downloaded
DownloadContentDefault_PostProcessing(v, f, Token)
If UseMD5Comparison And (v.Type = UTypes.GIF Or v.Type = UTypes.Picture) Then
If v.File.Exists Then
v.MD5 = ValidateMD5_GetMD5(v, False)
If Not v.MD5.IsEmptyString Then
If _MD5List.Contains(v.MD5) Then
__fileDeleted = v.File.Delete(SFO.File, SFODelete.DeletePermanently, EDP.ReturnValue)
If __fileDeleted Then dCount -= 1 : updateDownCount()
Else
_MD5List.Add(v.MD5)
End If
End If
Else
dCount -= 1
End If
End If
dCount += 1
Catch woex As OperationCanceledException When Token.IsCancellationRequested
__deleteFile.Invoke(f, v.URL_BASE)
@@ -1745,7 +1760,7 @@ BlockNullPicture:
Else
v.State = UStates.Skipped
End If
_ContentNew(i) = v
If Not __fileDeleted Then _ContentNew(i) = v
If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then
Progress.Perform(_ContentNew.Count - dTotal)
Exit Sub
@@ -2240,6 +2255,7 @@ BlockNullPicture:
LatestData.Clear()
_TempMediaList.Clear()
_TempPostsList.Clear()
_MD5List.Clear()
TokenPersonal = Nothing
If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
If Not Responser Is Nothing Then Responser.Dispose()

View File

@@ -9,6 +9,7 @@
Imports SCrawler.Plugin.Attributes
Namespace API.Instagram
Friend Class EditorExchangeOptions
#Region "Download"
<PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
Friend Property GetTimeline As Boolean
<PSetting(Caption:="Get reels", ToolTip:="Download user reels")>
@@ -19,6 +20,21 @@ Namespace API.Instagram
Friend Property GetStoriesUser As Boolean
<PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")>
Friend Property GetTagged As Boolean
#End Region
#Region "Extract image"
<PSetting(Caption:="Extract image from video: timeline")>
Friend Property GetTimeline_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: reels")>
Friend Property GetReels_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: stories")>
Friend Property GetStories_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: stories: user")>
Friend Property GetStoriesUser_VideoPic As Boolean
<PSetting(Caption:="Extract image from video: tagged posts")>
Friend Property GetTagged_VideoPic As Boolean
#End Region
<PSetting(Caption:="Place the extracted image into the video folder")>
Friend Property PutImageVideoFolder As Boolean
Friend Sub New(ByVal u As UserData)
With u
GetTimeline = .GetTimeline
@@ -26,6 +42,14 @@ Namespace API.Instagram
GetStories = .GetStories
GetStoriesUser = .GetStoriesUser
GetTagged = .GetTaggedData
GetTimeline_VideoPic = .GetTimeline_VideoPic
GetReels_VideoPic = .GetReels_VideoPic
GetStories_VideoPic = .GetStories_VideoPic
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
GetTagged_VideoPic = .GetTaggedData_VideoPic
PutImageVideoFolder = .PutImageVideoFolder
End With
End Sub
Friend Sub New(ByVal s As SiteSettings)
@@ -35,6 +59,14 @@ Namespace API.Instagram
GetStories = CBool(.GetStories.Value)
GetStoriesUser = CBool(.GetStoriesUser.Value)
GetTagged = CBool(.GetTagged.Value)
GetTimeline_VideoPic = CBool(.GetTimeline_VideoPic.Value)
GetReels_VideoPic = CBool(.GetReels_VideoPic.Value)
GetStories_VideoPic = CBool(.GetStories_VideoPic.Value)
GetStoriesUser_VideoPic = CBool(.GetStoriesUser_VideoPic.Value)
GetTagged_VideoPic = CBool(.GetTagged_VideoPic.Value)
PutImageVideoFolder = CBool(.PutImageVideoFolder.Value)
End With
End Sub
End Class

View File

@@ -57,6 +57,7 @@ Namespace API.Instagram
#End Region
#Region "Categories"
Private Const CAT_DOWN As String = "Download data"
Private Const CAT_UserDefs_VIDEO As String = DN.CAT_UserDefs & ": extract image from video"
#End Region
#Region "Authorization properties"
Friend Const Header_IG_APP_ID As String = "x-ig-app-id"
@@ -187,14 +188,28 @@ Namespace API.Instagram
Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
<PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(23), PClonable>
Friend ReadOnly Property GetTimeline As PropertyValue
<PropertyOption(ControlText:="From timeline", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(23), PClonable>
Friend ReadOnly Property GetTimeline_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(24), PClonable>
Friend ReadOnly Property GetReels As PropertyValue
<PropertyOption(ControlText:="From reels", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(24), PClonable>
Friend ReadOnly Property GetReels_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(25), PClonable>
Friend ReadOnly Property GetStories As PropertyValue
<PropertyOption(ControlText:="From stories", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(25), PClonable>
Friend ReadOnly Property GetStories_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(26), PClonable>
Friend ReadOnly Property GetStoriesUser As PropertyValue
<PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
<PropertyOption(ControlText:="From stories: user", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(26), PClonable>
Friend ReadOnly Property GetStoriesUser_VideoPic As PropertyValue
<PropertyOption(ControlText:="Get tagged posts", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
Friend ReadOnly Property GetTagged As PropertyValue
<PropertyOption(ControlText:="From tagged posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(27), PClonable>
Friend ReadOnly Property GetTagged_VideoPic As PropertyValue
<PropertyOption(ControlText:="From saved posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(28), PClonable>
Friend ReadOnly Property GetSavedPosts_VideoPic As PropertyValue
<PropertyOption(ControlText:="Place the extracted image into the video folder", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(29), PClonable>
Friend ReadOnly Property PutImageVideoFolder As PropertyValue
<PropertyOption(ControlText:="Tagged notify limit",
ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
"-1 to disable"), PXML, ControlNumber(27), PClonable>
@@ -203,19 +218,19 @@ Namespace API.Instagram
Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
#End Region
#Region "Download ready"
<PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
<PropertyOption(ControlText:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
Friend ReadOnly Property DownloadTimeline As PropertyValue
<PXML> Private ReadOnly Property DownloadTimeline_Def As PropertyValue
<PropertyOption(ControlText:="Download reels", ControlToolTip:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
<PropertyOption(ControlText:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
Friend ReadOnly Property DownloadReels As PropertyValue
<PXML> Private ReadOnly Property DownloadReels_Def As PropertyValue
<PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
<PropertyOption(ControlText:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
Friend ReadOnly Property DownloadStories As PropertyValue
<PXML> Private ReadOnly Property DownloadStories_Def As PropertyValue
<PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
<PropertyOption(ControlText:="Download stories: user", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
Friend ReadOnly Property DownloadStoriesUser As PropertyValue
<PXML> Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue
<PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
<PropertyOption(ControlText:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
Friend ReadOnly Property DownloadTagged As PropertyValue
<PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue
#End Region
@@ -425,10 +440,17 @@ Namespace API.Instagram
SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
GetTimeline = New PropertyValue(True)
GetTimeline_VideoPic = New PropertyValue(True)
GetReels = New PropertyValue(False)
GetReels_VideoPic = New PropertyValue(True)
GetStories = New PropertyValue(False)
GetStories_VideoPic = New PropertyValue(True)
GetStoriesUser = New PropertyValue(False)
GetStoriesUser_VideoPic = New PropertyValue(True)
GetTagged = New PropertyValue(False)
GetTagged_VideoPic = New PropertyValue(True)
GetSavedPosts_VideoPic = New PropertyValue(True)
PutImageVideoFolder = New PropertyValue(False)
TaggedNotifyLimit = New PropertyValue(200)
TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker

View File

@@ -194,7 +194,7 @@ Namespace API.Instagram
With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"})
If .ListExists Then
ProgressPre.ChangeMax(.Count)
For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token) : Next
For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token, Sections.Stories) : Next
End If
End With
End If
@@ -217,7 +217,7 @@ Namespace API.Instagram
Using j As EContainer = JsonDocument.Parse(r)
If j.ListExists Then
Dim i% = -1
GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token)
GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token, Sections.UserStories)
End If
End Using
End If

View File

@@ -26,10 +26,16 @@ Namespace API.Instagram
Private Const Name_LastCursor As String = "LastCursor"
Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
Private Const Name_GetTimeline As String = "GetTimeline"
Private Const Name_GetTimeline_VideoPic As String = "GetTimeline_VideoPic"
Private Const Name_GetReels As String = "GetReels"
Private Const Name_GetReels_VideoPic As String = "GetReels_VideoPic"
Private Const Name_GetStories As String = "GetStories"
Private Const Name_GetStories_VideoPic As String = "GetStories_VideoPic"
Private Const Name_GetStoriesUser As String = "GetStoriesUser"
Private Const Name_GetStoriesUser_VideoPic As String = "GetStoriesUser_VideoPic"
Private Const Name_GetTagged As String = "GetTaggedData"
Private Const Name_GetTagged_VideoPic As String = "GetTaggedData_VideoPic"
Private Const Name_PutImageVideoFolder As String = "PutImageVideoFolder"
Private Const Name_TaggedChecked As String = "TaggedChecked"
Private Const Name_NameTrue As String = "NameTrue"
#End Region
@@ -79,10 +85,32 @@ Namespace API.Instagram
Private LastCursor As String = String.Empty
Private FirstLoadingDone As Boolean = False
Friend Property GetTimeline As Boolean = True
Friend Property GetTimeline_VideoPic As Boolean = True
Friend Property GetReels As Boolean = False
Friend Property GetReels_VideoPic As Boolean = True
Friend Property GetStories As Boolean
Friend Property GetStories_VideoPic As Boolean = True
Friend Property GetStoriesUser As Boolean
Friend Property GetStoriesUser_VideoPic As Boolean = True
Friend Property GetTaggedData As Boolean
Friend Property GetTaggedData_VideoPic As Boolean = True
Friend Property PutImageVideoFolder As Boolean = False
Private Function ExtractImageFrom(ByVal Section As Sections) As Boolean
Select Case Section
Case Sections.Timeline : Return GetTimeline_VideoPic
Case Sections.Reels : Return GetReels_VideoPic
Case Sections.Tagged : Return GetTaggedData_VideoPic
Case Sections.Stories : Return GetStories_VideoPic
Case Sections.UserStories : Return GetStoriesUser_VideoPic
Case Sections.SavedPosts
Try
If Not HOST Is Nothing AndAlso HOST.Key = InstagramSiteKey Then Return MySiteSettings.GetSavedPosts_VideoPic.Value
Catch
End Try
Return True
Case Else : Return True
End Select
End Function
Protected _NameTrue As String = String.Empty
Friend ReadOnly Property NameTrue As String
Get
@@ -98,20 +126,32 @@ Namespace API.Instagram
LastCursor = .Value(Name_LastCursor)
FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value)
GetTimeline_VideoPic = .Value(Name_GetTimeline_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline_VideoPic.Value))
GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels.Value))
GetReels_VideoPic = .Value(Name_GetReels_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels_VideoPic.Value))
GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value)
GetStories_VideoPic = .Value(Name_GetStories_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories_VideoPic.Value))
GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser.Value))
GetStoriesUser_VideoPic = .Value(Name_GetStoriesUser_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser_VideoPic.Value))
PutImageVideoFolder = .Value(Name_PutImageVideoFolder).FromXML(Of Boolean)(CBool(MySiteSettings.PutImageVideoFolder.Value))
GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
GetTaggedData_VideoPic = .Value(Name_GetTagged_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged_VideoPic.Value))
TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
_NameTrue = .Value(Name_NameTrue)
Else
.Add(Name_LastCursor, LastCursor)
.Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
.Add(Name_GetTimeline, GetTimeline.BoolToInteger)
.Add(Name_GetTimeline_VideoPic, GetTimeline_VideoPic.BoolToInteger)
.Add(Name_GetReels, GetReels.BoolToInteger)
.Add(Name_GetReels_VideoPic, GetReels_VideoPic.BoolToInteger)
.Add(Name_GetStories, GetStories.BoolToInteger)
.Add(Name_GetStories_VideoPic, GetStories_VideoPic.BoolToInteger)
.Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
.Add(Name_GetStoriesUser_VideoPic, GetStoriesUser_VideoPic.BoolToInteger)
.Add(Name_GetTagged, GetTaggedData.BoolToInteger)
.Add(Name_GetTagged_VideoPic, GetTaggedData_VideoPic.BoolToInteger)
.Add(Name_PutImageVideoFolder, PutImageVideoFolder.BoolToInteger)
.Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
.Add(Name_NameTrue, _NameTrue)
End If
@@ -130,6 +170,14 @@ Namespace API.Instagram
GetStories = .GetStories
GetStoriesUser = .GetStoriesUser
GetTaggedData = .GetTagged
GetTimeline_VideoPic = .GetTimeline_VideoPic
GetReels_VideoPic = .GetReels_VideoPic
GetStories_VideoPic = .GetStories_VideoPic
GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
GetTaggedData_VideoPic = .GetTagged_VideoPic
PutImageVideoFolder = .PutImageVideoFolder
End With
End If
End Sub
@@ -809,7 +857,7 @@ NextPageBlock:
With j("items")
For Each jj In .Self
before = _TempMediaList.Count
ObtainMedia(jj, PostsToReparse(i).ID, specFolder)
ObtainMedia(jj, PostsToReparse(i).ID, specFolder,,,,,,, IIf(IsTagged, Sections.Tagged, Sections.Timeline))
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
Next
@@ -911,7 +959,7 @@ NextPageBlock:
End Select
End If
before = _TempMediaList.Count
ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts)
ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts,, Section)
If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False
End If
@@ -950,6 +998,7 @@ NextPageBlock:
Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
Protected ObtainMedia_AllowAbstract As Boolean = False
Private Const ObtainMedia_NoSection As Integer = -10
Protected Sub ObtainMedia_SetReelsFunc()
ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
If ss.Value("url").IsEmptyString Then
@@ -971,7 +1020,8 @@ NextPageBlock:
Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
Optional ByVal PostOriginUrl As String = Nothing,
Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0,
Optional ByVal TryExtractImage As Boolean = False)
Optional ByVal TryExtractImage As Boolean = False,
Optional ByVal Section As Sections = ObtainMedia_NoSection)
Try
Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
@@ -1018,6 +1068,12 @@ NextPageBlock:
If TryExtractImage Then
t = 1
abstractDecision = True
If Not SpecialFolder.IsEmptyString AndAlso PutImageVideoFolder Then
Dim endsAbs As Boolean = SpecialFolder.EndsWith("*")
If endsAbs Then SpecialFolder = SpecialFolder.TrimEnd("*")
If Not SpecialFolder.IsEmptyString Then SpecialFolder = $"{SpecialFolder.TrimEnd("\")}\{VideoFolderName}{IIf(Not endsAbs, "*", String.Empty)}"
If endsAbs Then SpecialFolder &= "*"
End If
ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then
If n.Contains(vid) Then
t = 2
@@ -1064,7 +1120,8 @@ NextPageBlock:
End If
End With
End If
If Not TryExtractImage Then ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True)
If Not TryExtractImage And Not Section = ObtainMedia_NoSection And ExtractImageFrom(Section) Then _
ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True, Section)
Case 8 'gallery
DateObj = mDate(n)
With n("carousel_media").XmlIfNothing
@@ -1165,6 +1222,7 @@ NextPageBlock:
Dim qStr$, r$
Dim i% = -1
Dim jj As EContainer
Dim section As Sections = IIf(GetUserStory, Sections.UserStories, Sections.Stories)
ThrowAny(Token)
If StoriesList.ListExists Or GetUserStory Then
If Not GetUserStory Then tmpList = StoriesList.Take(5)
@@ -1181,7 +1239,7 @@ NextPageBlock:
Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
If j.Contains("reels") Then
ProgressPre.ChangeMax(j("reels").Count)
For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token) : Next
For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token, section) : Next
End If
End Using
End If
@@ -1189,7 +1247,8 @@ NextPageBlock:
End If
End If
End Sub
Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean, ByVal Token As CancellationToken)
Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean,
ByVal Token As CancellationToken, Optional ByVal Section As Sections = Sections.Stories)
If Not Node Is Nothing Then
With Node
ProgressPre.Perform()
@@ -1210,7 +1269,7 @@ NextPageBlock:
pid = storyID & s.Value("id")
If Not _TempPostsList.Contains(pid) Then
ThrowAny(Token)
ObtainMedia(s, pid, sFolder)
ObtainMedia(s, pid, sFolder,,,,,,, Section)
_TempPostsList.Add(pid)
End If
Next

View File

@@ -11,6 +11,11 @@ Namespace API.OnlyFans
Friend Module Declarations
Friend ReadOnly DateProvider As New ADateTime("O")
Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue)
Friend ReadOnly FilesSources As New List(Of Object()) From {
{{"source", "source"}},
{{"files", "source", "url"}},
{{"files", "full", "url"}}
}
Friend Property Rules As DynamicRulesEnv
End Module
End Namespace

View File

@@ -394,6 +394,14 @@ Namespace API.OnlyFans
Loop While Not _complete
End Sub
#End Region
Private Function GetMediaURL(ByVal m As EContainer) As String
Dim v$
For Each node As Object() In FilesSources
v = If(m.ItemF(node)?.Value, String.Empty)
If Not v.IsEmptyString Then Return v
Next
Return String.Empty
End Function
Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing,
@@ -405,11 +413,14 @@ Namespace API.OnlyFans
With n("media")
If .ListExists Then
For Each m In .Self
If IsHL Then
postUrl = m.Value({"files", "source"}, "url")
Else
postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
End If
postUrl = GetMediaURL(m)
'If IsHL Then
' 'postUrl = m.Value({"files", "source"}, "url")
' postUrl = GetMediaURL(m)
'Else
' 'postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
' postUrl = GetMediaURL(m)
'End If
postUrlBase = String.Empty
Select Case m.Value("type")
Case "photo" : t = UTypes.Picture : ext = "jpg"

View File

@@ -90,6 +90,7 @@ Namespace API.Twitter
GifsPrefix = .GifsPrefix
UseMD5Comparison = .UseMD5Comparison
RemoveExistingDuplicates = .RemoveExistingDuplicates
If RemoveExistingDuplicates Then StartMD5Checked = False
DownloadModel = DownloadModels.Undefined
DownloadModelForceApply = .DownloadModelForceApply
MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets

View File

@@ -203,7 +203,7 @@ Namespace API.YouTube
If IsMusic Or DownloadYTVideos Then
maxDate = Nothing
LastDownloadDateVideos = nDate(LastDownloadDateVideos)
url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/videos"
container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)

View File

@@ -550,7 +550,7 @@ Namespace DownloadObjects
End Sub
Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo,
ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean
Const MsgTitle$ = "Copy/Move checked files"
Dim MsgTitle$ = "Copy/Move checked files"
Try
Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO)
Dim moveOptions As FeedMoveCopyTo = Nothing
@@ -574,6 +574,7 @@ Namespace DownloadObjects
Dim sesFilesReplaced As Boolean = False
Dim filesReplace As New List(Of KeyValuePair(Of SFile, SFile))
Dim updateFileLocations As Boolean = Settings.FeedMoveCopyUpdateFileLocationOnMove
Dim postUrl$
Dim result As Boolean = False
If FeedMediaData Is Nothing Then
@@ -590,7 +591,18 @@ Namespace DownloadObjects
data = {FeedMediaData.Media}
data_files = {FeedMediaData.File}
End If
MsgTitle = $"{IIf(isCopy, "Copy", "Move")} {IIf(Not FeedMediaData Is Nothing Or GetChecked, "checked", "ALL")} files"
If data.ListExists Then
If (FeedMediaData Is Nothing And Not GetChecked And Not isCopy) AndAlso
MsgBoxE({$"YOU ARE TRYING TO MOVE ALL FEED/SESSION DATA.{vbCr}EVERY FILE WILL BE MOVED, NOT JUST THE SELECTED ONES.", MsgTitle},
vbExclamation,,, {"Process", "Cancel"}) = 1 Then
ShowOperationCanceledMsg(MsgTitle)
Return False
End If
If MCTOptions.Destination.IsEmptyString Then
Using f As New FeedCopyToForm(data_files, isCopy)
f.ShowDialog()
@@ -678,7 +690,11 @@ Namespace DownloadObjects
mm_data = mm.Data
mm_data.File = df
__isSavedPosts = mm.IsSavedPosts And moveOptions.ReplaceUserProfile_Profile Is Nothing
mm = New UserMediaD(mm_data, If(moveOptions.ReplaceUserProfile_Profile, mm.User), mm.Session, mm.Date) With {.IsSavedPosts = __isSavedPosts}
postUrl = mm.PostUrl(True)
mm = New UserMediaD(mm_data, If(moveOptions.ReplaceUserProfile_Profile, mm.User), mm.Session, mm.Date) With {
.IsSavedPosts = __isSavedPosts,
.PostUrl = postUrl
}
If __isSavedPosts Then mm.UserInfo = __user
Downloader.Files(indx) = mm
downloaderFilesUpdated = True
@@ -709,7 +725,11 @@ Namespace DownloadObjects
mm_data = mm.Data
mm_data.File = df
__isSavedPosts = mm.IsSavedPosts And moveOptions.ReplaceUserProfile_Profile Is Nothing
mm = New UserMediaD(mm_data, If(moveOptions.ReplaceUserProfile_Profile, mm.User), mm.Session, mm.Date) With {.IsSavedPosts = __isSavedPosts}
postUrl = mm.PostUrl(True)
mm = New UserMediaD(mm_data, If(moveOptions.ReplaceUserProfile_Profile, mm.User), mm.Session, mm.Date) With {
.IsSavedPosts = __isSavedPosts,
.PostUrl = postUrl
}
If __isSavedPosts Then mm.UserInfo = __user
sessionData(indx) = mm
sesFilesReplaced = True

View File

@@ -27,7 +27,6 @@ Namespace DownloadObjects
Dim TP_LBL As System.Windows.Forms.TableLayoutPanel
Dim CONTEXT_SEP_3 As System.Windows.Forms.ToolStripSeparator
Dim CONTEXT_SEP_4 As System.Windows.Forms.ToolStripSeparator
Me.CONTEXT_SEP_2 = New System.Windows.Forms.ToolStripSeparator()
Me.CH_CHECKED = New System.Windows.Forms.CheckBox()
Me.LBL_INFO = New System.Windows.Forms.Label()
Me.CONTEXT_DATA = New System.Windows.Forms.ContextMenuStrip(Me.components)
@@ -37,6 +36,7 @@ Namespace DownloadObjects
Me.BTT_CONTEXT_OPEN_USER = New System.Windows.Forms.ToolStripMenuItem()
Me.BTT_CONTEXT_OPEN_USER_URL = New System.Windows.Forms.ToolStripMenuItem()
Me.BTT_CONTEXT_OPEN_USER_POST = New System.Windows.Forms.ToolStripMenuItem()
Me.CONTEXT_SEP_2 = New System.Windows.Forms.ToolStripSeparator()
Me.BTT_COPY_TO = New System.Windows.Forms.ToolStripMenuItem()
Me.BTT_MOVE_TO = New System.Windows.Forms.ToolStripMenuItem()
Me.BTT_FEED_ADD_FAV = New System.Windows.Forms.ToolStripMenuItem()
@@ -52,6 +52,7 @@ Namespace DownloadObjects
Me.ICON_SITE = New System.Windows.Forms.PictureBox()
Me.TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
Me.LBL_TITLE = New System.Windows.Forms.Label()
Me.BTT_CONTEXT_OPEN_FILE_FOLDER = New System.Windows.Forms.ToolStripMenuItem()
CONTEXT_SEP_1 = New System.Windows.Forms.ToolStripSeparator()
TP_LBL = New System.Windows.Forms.TableLayoutPanel()
CONTEXT_SEP_3 = New System.Windows.Forms.ToolStripSeparator()
@@ -67,11 +68,6 @@ Namespace DownloadObjects
CONTEXT_SEP_1.Name = "CONTEXT_SEP_1"
CONTEXT_SEP_1.Size = New System.Drawing.Size(302, 6)
'
'CONTEXT_SEP_2
'
Me.CONTEXT_SEP_2.Name = "CONTEXT_SEP_2"
Me.CONTEXT_SEP_2.Size = New System.Drawing.Size(302, 6)
'
'TP_LBL
'
TP_LBL.ColumnCount = 3
@@ -114,9 +110,9 @@ Namespace DownloadObjects
'
'CONTEXT_DATA
'
Me.CONTEXT_DATA.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_CONTEXT_DOWN, Me.CONTEXT_SEP_0, Me.BTT_CONTEXT_OPEN_MEDIA, Me.BTT_CONTEXT_OPEN_USER, CONTEXT_SEP_1, Me.BTT_CONTEXT_OPEN_USER_URL, Me.BTT_CONTEXT_OPEN_USER_POST, Me.CONTEXT_SEP_2, Me.BTT_COPY_TO, Me.BTT_MOVE_TO, CONTEXT_SEP_3, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_ADD_FAV_REMOVE, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_ADD_SPEC_REMOVE, Me.BTT_FEED_REMOVE_FAV, Me.BTT_FEED_REMOVE_SPEC, CONTEXT_SEP_4, Me.BTT_CONTEXT_FIND_USER, Me.BTT_CONTEXT_INFO, Me.CONTEXT_SEP_5, Me.BTT_CONTEXT_DELETE})
Me.CONTEXT_DATA.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_CONTEXT_DOWN, Me.CONTEXT_SEP_0, Me.BTT_CONTEXT_OPEN_MEDIA, Me.BTT_CONTEXT_OPEN_USER, Me.BTT_CONTEXT_OPEN_FILE_FOLDER, CONTEXT_SEP_1, Me.BTT_CONTEXT_OPEN_USER_URL, Me.BTT_CONTEXT_OPEN_USER_POST, Me.CONTEXT_SEP_2, Me.BTT_COPY_TO, Me.BTT_MOVE_TO, CONTEXT_SEP_3, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_ADD_FAV_REMOVE, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_ADD_SPEC_REMOVE, Me.BTT_FEED_REMOVE_FAV, Me.BTT_FEED_REMOVE_SPEC, CONTEXT_SEP_4, Me.BTT_CONTEXT_FIND_USER, Me.BTT_CONTEXT_INFO, Me.CONTEXT_SEP_5, Me.BTT_CONTEXT_DELETE})
Me.CONTEXT_DATA.Name = "CONTEXT_PIC"
Me.CONTEXT_DATA.Size = New System.Drawing.Size(306, 414)
Me.CONTEXT_DATA.Size = New System.Drawing.Size(306, 436)
'
'BTT_CONTEXT_DOWN
'
@@ -160,6 +156,11 @@ Namespace DownloadObjects
Me.BTT_CONTEXT_OPEN_USER_POST.Size = New System.Drawing.Size(305, 22)
Me.BTT_CONTEXT_OPEN_USER_POST.Text = "Open post"
'
'CONTEXT_SEP_2
'
Me.CONTEXT_SEP_2.Name = "CONTEXT_SEP_2"
Me.CONTEXT_SEP_2.Size = New System.Drawing.Size(302, 6)
'
'BTT_COPY_TO
'
Me.BTT_COPY_TO.Image = Global.SCrawler.My.Resources.Resources.PastePic_32
@@ -289,6 +290,13 @@ Namespace DownloadObjects
Me.LBL_TITLE.Size = New System.Drawing.Size(140, 25)
Me.LBL_TITLE.TabIndex = 1
'
'BTT_CONTEXT_OPEN_FILE_FOLDER
'
Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Image = Global.SCrawler.My.Resources.Resources.FolderPic_32
Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Name = "BTT_CONTEXT_OPEN_FILE_FOLDER"
Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Size = New System.Drawing.Size(305, 22)
Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Text = "Open file folder"
'
'FeedMedia
'
Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -335,5 +343,6 @@ Namespace DownloadObjects
Private WithEvents BTT_MOVE_TO As ToolStripMenuItem
Private WithEvents CONTEXT_SEP_5 As ToolStripSeparator
Private WithEvents CONTEXT_SEP_2 As ToolStripSeparator
Private WithEvents BTT_CONTEXT_OPEN_FILE_FOLDER As ToolStripMenuItem
End Class
End Namespace

View File

@@ -178,6 +178,7 @@ Namespace DownloadObjects
BTT_CONTEXT_DOWN.Visible = True
CONTEXT_SEP_0.Visible = True
BTT_CONTEXT_OPEN_USER.Visible = False
BTT_CONTEXT_OPEN_FILE_FOLDER.Visible = False
CONTEXT_SEP_5.Visible = False
BTT_CONTEXT_DELETE.Visible = False
@@ -416,8 +417,12 @@ Namespace DownloadObjects
End Sub
#End Region
#Region "Open media, folder"
Private Sub BTT_CONTEXT_OPEN_MEDIA_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_MEDIA.Click
File.Open()
Private Sub BTT_CONTEXT_OPEN_MEDIA_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_MEDIA.Click, BTT_CONTEXT_OPEN_FILE_FOLDER.Click
If Not sender Is Nothing AndAlso sender Is BTT_CONTEXT_OPEN_FILE_FOLDER Then
GlobalOpenPath(File)
Else
File.Open()
End If
End Sub
Private Sub BTT_CONTEXT_OPEN_USER_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_USER.Click
If Not UserKey.IsEmptyString Then
@@ -455,6 +460,8 @@ Namespace DownloadObjects
Dim url$ = String.Empty
If IsSubscription Then
url = Post.URL_BASE
ElseIf Not Media.PostUrl.IsEmptyString Then
url = Media.PostUrl
Else
If Not UserKey.IsEmptyString And Not Post.Post.ID.IsEmptyString Then
Dim u As IUserData

View File

@@ -195,6 +195,7 @@ Namespace DownloadObjects
#Region "Add"
Friend Overloads Function Add(ByVal Item As UserMediaD, Optional ByVal AutoSave As Boolean = True) As Boolean
If Not Items.Contains(Item) Then
Item.PostUrl = Item.PostUrl(True)
Items.Add(Item)
If AutoSave Then Save()
Return True
@@ -242,6 +243,7 @@ Namespace DownloadObjects
Dim m As UserMedia
Dim f As SFile = Nothing
Dim ff As SFile
Dim postUrl$ = String.Empty
Dim user As IUserData
Dim processRemove As Boolean
Dim userArr As New List(Of IUserData)
@@ -254,7 +256,8 @@ Namespace DownloadObjects
m = d.Data
m.File = ff
d = New UserMediaD(m, If(replaceUser, d.User), d.Session, d.Date) With {
.IsSavedPosts = If(replaceUser Is Nothing, d.IsSavedPosts, DirectCast(replaceUser, UserDataBase).IsSavedPosts)
.IsSavedPosts = If(replaceUser Is Nothing, d.IsSavedPosts, DirectCast(replaceUser, UserDataBase).IsSavedPosts),
.PostUrl = postUrl
}
Items(i) = d
ri += 1
@@ -268,6 +271,7 @@ Namespace DownloadObjects
For i = Count - 1 To 0 Step -1
If p.Invoke(Items(i)) Then
d = Items(i)
postUrl = d.PostUrl(True)
f = Nothing
ff = Nothing
processRemove = True

View File

@@ -36,6 +36,7 @@ Namespace DownloadObjects
Private Const Name_Session As String = "Session"
Private Const Name_File As String = "File"
Private Const Name_IsSavedPosts As String = "IsSavedPosts"
Private Const Name_PostUrl As String = "PostUrl"
#End Region
Friend ReadOnly User As IUserData
Friend ReadOnly Data As UserMedia
@@ -43,6 +44,48 @@ Namespace DownloadObjects
Friend ReadOnly [Date] As Date
Friend Session As Integer
Friend IsSavedPosts As Boolean
Private _PostUrl As String
Friend Property PostUrl(Optional ByVal Generate As Boolean = False) As String
Get
Try
If Not _PostUrl.IsEmptyString Then
Return _PostUrl
ElseIf Generate Then
Dim url$ = String.Empty
With UserInfo
If Not .Plugin.IfNullOrEmpty(.Site).IsEmptyString And Not .Name.IsEmptyString And Not Data.Post.ID.IsEmptyString Then
Dim u As IUserData
If IsSavedPosts Then
If Not .Plugin.IsEmptyString Then
Dim host As SettingsHostCollection = Settings(.Plugin)
If Not host Is Nothing Then
u = host.Default.GetInstance(Download.SavedPosts, UserInfo, False, False)
If Not u Is Nothing AndAlso Not u.HOST Is Nothing Then
With DirectCast(u, UserDataBase)
.IsSavedPosts = True
.HostStatic = True
End With
Try : url = u.HOST.Source.GetUserPostUrl(u, Data) : Catch : End Try
u.Dispose()
End If
End If
End If
Else
u = Settings.GetUser(UserInfo)
If Not u Is Nothing Then url = UserDataBase.GetPostUrl(u, Data)
End If
End If
End With
Return url
End If
Catch
End Try
Return String.Empty
End Get
Set(ByVal _PostUrl As String)
Me._PostUrl = _PostUrl
End Set
End Property
Friend Sub New(ByVal Data As UserMedia, ByVal User As IUserData, ByVal Session As Integer)
Me.Data = Data
Me.User = User
@@ -78,6 +121,7 @@ Namespace DownloadObjects
Data = New UserMedia(e(Name_Media), User)
[Date] = AConvert(Of Date)(e.Value(Name_Date), DateTimeDefaultProvider, Now)
Session = e.Value(Name_Session).FromXML(Of Integer)(0)
_PostUrl = e.Value(Name_PostUrl)
Dim f As SFile = e.Value(Name_File)
If f.Exists Then Data.File = f
End If
@@ -106,7 +150,8 @@ Namespace DownloadObjects
New EContainer(Name_Date, AConvert(Of String)([Date], DateTimeDefaultProvider, String.Empty)),
New EContainer(Name_Session, Session),
New EContainer(Name_File, Data.File),
New EContainer(Name_IsSavedPosts, IsSavedPosts.BoolToInteger)},
New EContainer(Name_IsSavedPosts, IsSavedPosts.BoolToInteger),
New EContainer(Name_PostUrl, _PostUrl)},
If(IsSavedPosts, UserInfo.ToEContainer, If(Not User Is Nothing, DirectCast(User, UserDataBase).User.ToEContainer, Nothing)), LAP.IgnoreICopier)
End Function
End Structure

View File

@@ -247,7 +247,7 @@ CloseResume:
BTT_DOWN_AUTOMATION.PerformClick()
ElseIf e.Alt And e.KeyCode = Keys.P Then
BTT_PR_INFO.PerformClick()
ElseIf e.Alt And e.KeyCode = Keys.F Then
ElseIf (e.Alt And (e.KeyCode = Keys.F Or e.KeyCode = Keys.U)) Or (e.Control And e.KeyCode = Keys.U) Then
MySearch.FormShow()
Else
b = False

View File

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2024.7.24.0")>
<Assembly: AssemblyFileVersion("2024.7.24.0")>
<Assembly: AssemblyVersion("2024.9.2.0")>
<Assembly: AssemblyFileVersion("2024.9.2.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -13,6 +13,25 @@ Friend Class UserImage : Inherits ImageRenderer
Friend Const ImagePostfix_Small As String = "_Small"
Private _LargeAddress As SFile
Private _SmallAddress As SFile
Private _ForceSaveOrig As Boolean = False
Friend Shared Function NewUserPicture(ByVal ImageOrig As SFile, ByVal Destination As SFile,
Optional ByVal Save As Boolean = True, Optional ByVal GetInstance As Boolean = False) As UserImage
Dim uImg As New UserImage(ImageOrig, Destination)
With uImg
._ForceSaveOrig = ImageOrig.Extension.IsEmptyString OrElse ImageOrig.Extension.ToLower = "gif" OrElse Not {"jpg", "jpeg", "png"}.Contains(ImageOrig.Extension.ToLower)
If Not ._ForceSaveOrig Then
If .Address.Exists AndAlso Not .Address.Delete(SFO.File,, EDP.ReturnValue) Then ._ForceSaveOrig = True
If Not ._ForceSaveOrig AndAlso Not ImageOrig.Copy(.Address) Then ._ForceSaveOrig = True
End If
If Not ._ForceSaveOrig Then
._SmallAddress.Extension = .Address.Extension
._LargeAddress.Extension = .Address.Extension
End If
If Save Then .Save()
End With
If Not GetInstance Then uImg.Dispose() : uImg = Nothing
Return uImg
End Function
Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True)
MyBase.New(_ImgOriginal)
Dim f As SFile = Destination
@@ -71,7 +90,7 @@ Friend Class UserImage : Inherits ImageRenderer
End With
End Function
Public Overrides Sub Save()
MyBase.Save()
If _ForceSaveOrig Then MyBase.Save()
Small.Save(_SmallAddress)
Large.Save(_LargeAddress)
End Sub

View File

@@ -4,4 +4,8 @@ REM Replace 'd:\Downloads\SocialNetworks\' with the path to your SCrawler data f
REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS.
REM But I believe 7Zip also has CLI commands
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "d:\Downloads\SocialNetworks\SCrawlerBackup.rar" "d:\Downloads\SocialNetworks\"
REM This line archives SCrawler settings files.
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl "D:\MyPrograms\SCrawler\Backup\Settings.rar" "D:\MyPrograms\SCrawler\Settings\"
REM This line archives SCrawler users' settings files.
"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "D:\MyPrograms\SCrawler\Backup\SCrawlerBackup.rar" "D:\MyPrograms\SCrawler\Data\"